Test-Time Adaptation via Self-Training with Nearest Neighbor Information
Authors: Minguk Jang
Sae-Young Chung
Hye Won Chung
Venue: ICLR 2023 poster (2023)
Keywords: test-time adaptation
domain adaptation
domain shift

Abstract: Test-time adaptation (TTA) aims to adapt a trained classifier using online unlabeled test data only, without any information related to the training procedure. Most existing TTA methods adapt the trained classifier using the classifier's prediction on the test data as pseudo-label.
However, under test-time domain shift, accuracy of the pseudo labels cannot be guaranteed, and thus the TTA methods often encounter performance degradation at the adapted classifier. To overcome this limitation, we propose a novel test-time adaptation method, called Test-time Adaptation via Self-Training with nearest neighbor information (TAST), which is composed of the following procedures: (1) adds trainable adaptation modules on top of the trained feature extractor; (2) newly defines a pseudo-label distribution for the test data by using the nearest neighbor information; (3) trains these modules only a few times during test time to match the nearest neighbor-based pseudo label distribution and a prototype-based class distribution for the test data; and (4) predicts the label of test data using the average predicted class distribution from these modules. The pseudo-label generation is based on the basic intuition that a test data and its nearest neighbor in the embedding space are likely to share the same label under the domain shift. By utilizing multiple randomly initialized adaptation modules, TAST extracts useful information for the classification of the test data under the domain shift, using the nearest neighbor information. TAST showed better performance than the state-of-the-art TTA methods on two standard benchmark tasks, domain generalization, namely VLCS, PACS, OfficeHome, and TerraIncognita, and image corruption, particularly CIFAR-10/100C.


Published as a conference paper at ICLR 2023
TEST-TIME ADAPTATION VIA SELF-TRAINING WITH NEAREST NEIGHBOR INFORMATION
Minguk Jang. Sae-Young Chung, Hye Won Chung
School of Electrical Engineering
Korea Advanced Institute of Science and Technology (KAIST)
Daejeon, Republic Korea
{mgjang, schung, bwchungjkkaist.cek
ABSTRACT
Test-time adaptation (TTA) aims to adapt trained classifier using online unla- beled test data only, without any information related the training procedure. Most existing TTA methods adapt the trained classifier using the classifier's pre- diction on the test data as pseudo-label. However, under test-time domain shift, accuracy of the pseudo labels cannot be guaranteed, and thus the TTA methods often encounter performance degradation at the adapted classifier. To overcome this limitation, we propose novel test-time adaptation method, called Test-time Adaptation via Self-Training with nearest neighbor information (TAST), which is composed of the following procedures: (I) adds trainable adaptation modules on top of the trained feature extractor; (2) newly defines a pseudo-label distribution for the test data by using the nearest neighbor information; (3) trains these mod- ules only a few times during test time to match the nearest neighbor-basee pseudo label distribution and a prototype- proset based class distribution for the test data; and (4) predicts the label of test data using the average predicted class distribution from these modules. The pseudo-label generation based on the basic intuition that test data and its nearest neighbor in the embedding space are likely share the same label under the domain shift. By utilizing multiple randomly initialized adaptation modules, TAST extracts useful information for the classification of the test data under the domain shift, using the nearest neighbor information. TAST showed better performance than the state-of-the-at arr TTA methods on two standard benchmark tasks, domain generalization, namely VLCS, PACS, OfficeHome, and Terralncognita, and image corruption, particularly CIFAR-10/100C. Our code is available at https:: http://tw..uu.ccooo//ooo.uuuuaa/ com/maingukjaag/TA439
INTRODUCTION
Deep neural networks often encounter significant performance degradations under domain shift (i.e., distribution shift). This phenomenon has been observed in various tasks including classification (Taori al. 22200] Wang al.) 2021b), visual recognition (Saenko et al. [2010] Csurkal 2017). and reinforcement learning (Cobbe et al. 2019 Mendonca et al.) 2020 Lee and Chung 20221b) There are two broad classes of domain adaptation methods that attempt solve this problem: supervised
domain adaptation (SDA) (Tzeng et al. 2013 Motilan et al. 2017) and unsupervised domain adapta- tion (UDA) (Ganin and Lempitsky [015] Long et al. 2016 Sener al.) 2016). Both SDA and UDA representations by aligning the representations of training and test data closely in the embedding space. While testing, UDA methods require the training dataset and SDA methods additionally require labeled data of the test domain. However, in practice, it is often difficult to access training datasets or labeled data in the test domain during test time, due to data security labeling cost.
methods aim to obtain domain-invarian
Wang et al. 2021a) is a prominent ap- proach to alleviate the problems caused by the domain shift. TTA methods aim to adapt the trained model to the test domain without a labeled dataset in the test domain and any information related to the training procedure (e.g., training dataset, feature statistics of training domain (Sun et al. 2020
Test-time adaptation (TTA) Twasawaa and Matsuo 2021

---

Published as a conference paper at ICLR 2023
Liu et al. 2021 Eastwood et al.] 2022) TTA methods have access to the online unlabeled test data only, whereas domain adaptation methods assume access to the whole (i.e., offline) test data.
There are three popular categories for TTA: normalization- based method (Schneider et al.) 2020) entropy minimization (Liang et al. 2020 Wang et al. 2021a) and prototype-based methods (Iwa- sawa and Matsuo, p Normalization method replaces the batch normalization (BN) statistics of the trained model with the BN statistics estimated on test data, and does not update model parame- ters except for the BN layers. Entropy minimization methods fine-tune the trained feature extractor, which is the trained classifier except the last linear layer, by minimizing the prediction entropy of test data. These methods force the classifier to have over-confident predictions for the test data, and thus have a risk of degrading model calibration (Guo et al.) 2017 Mukhoti et al. (2020), a measure of model interpretability and reliability. One form of entropy minimization is self-training (Rosenberg et al. pseudo labels for the test data and fine-tune the classifier to make fit to the pseudo labels. These methods have a limitation that the fine-tuned classifier can overfit to the inaccurate pseudo labels, resulting in confirmation bias (Arazo et al. 2020). This limitation can be harmful when the perfor- mance of the trained classifier is significantly degraded due to the domain shift. On the other hand, Iwasawa and Matsuo (2021) proposed a prototype- -bssed TTA method, named T3A, that simply modifies a trained linear classifier (the last layer) by using the pseudo-prototype representations of each class and the prototype-
2005 Lee| 2013; Xie et al., 2020). Self-training methods use predictions from the classifier as
classification for test data, where the prototypes are constructed by previous test data and the prediction for the data from trained classifier. T3A does not update the trained feature extractor at test time. T3A is simple but it brings marginal performance gain (Table [and 3.
In this work, we propose a new test-time adaptation method, which is simple yet effective in mit- igating the confirmation bias problem of self self-training, by adding adaptation modules on top of the feature extractor, which are simply trainable during test time. We use the prototype-base based classifier as T3A, but not in the embedding space of the original feature extractor but in the embedding space of the adaptation modules, trained with nearest neighbor information, to achieve higher per- formance gains than the original simple prototype-basee classifier method. Our method, named Test-time Adaptation via Self-Training with nearest neighbor information (TAST), is composed of the following procedures: (1) adds randomly initialized adaptation modules on top of the feature extractor at the beginning of test time (Figure I (2) generates pseudo label distribution for test data considering the nearest neighbor information; (3) trains the adaptation modules only a few times during test time match the nearest neighbor-base -based pseudo label distribution and prototype- prosetype-based class distribution for the test data; and (4) predicts the label of test data using the average predicted class distribution from the adaptation modules. Specifically, in (1), we add the trainable adaptation modules to obtain new feature embeddings that are useful for classification in the test domain. In (2), TAST assigns the mean of the labels of the nearby examples in the embedding space as the pseudo label distribution for the test data based on the idea that test data and its nearest neighbors are more likely to have the same label. In (3), TAST trains the adaptation modules to output the pseudo label distribution when the test data is fed into (Figure Right.. And in (4), we average the predicted class distributions from adaptation modules for the prediction of test data (Figure Left).
We investigate the effectiveness of TAST on two standard benchmarks, domain generalization and image corruption. We demonstrate that TAST outperforms the current tatte-of-thcaar test-time adaptation methods such as Tent (Wang et al. 202Ia), T3A, and TTT++ (Liu al. 202T) on the two benchmarks. For example, TAST surpasses the current state-of-the-aar algorithm by 1.01% on average with ResNet- 18 learned by Empirical Risk Minimization (ERM) the domain generaliza- tion benchmarks. Extensive ablation studies show that both the nearest neighbor information and the adaptation module utilization contribute to the performance increase. Moreover, we experimentally found that the adaptation modules adapt feature extractor outputs effectively although the adaptation modules are randomly initialized at the beginning of test time and trained with few gradient steps per test batch during test time.
2 PRELIMINARIES
Test-time domain shift Consider labeled dataset Dirain {1;; 31) F drawn from a distri- bution purain, where E Rd and y y := {1,2,... ,K} for K-clas classification problem.
2

---

Published as a conference paper at ICLR 2023
(5) T3L
Festune estractor fo
Procotepe Rassed Class fler
Support.uet
(b) TAST
Wearest
supportset
[ar
Adustation Pale
Adastation modois
festure extrcttor
Adaptationn modiles Pallii N
Nevereh- Noghbor Clasifier
hosttape based Cussfirr
Supportsct
Protoope
Cacufter
Support oet
Label Approgation
g
C2 loss
Figure I: Overview of TAST. Left: A schematic of TAST compared to T3A. The dashed class indi- cates the ground-truth class. (a) T3A constructs prototypes that represent classes in the embedding space of feature extractor fo using support set S. Then T3A predicts the label of the test data x as the class of the nearest prototype. (b) TAST adds trainable adaptation modules [ha,} } on top of fo and computes the estimated class distributions of by aggregating the pseudo labels of the nearest support examples of æ in the embedding space of adaptation modules. Right: Overview of TAST training. Based on the intuition that a test data and its nearest neighbors N N(x;; S) are likely to share the same label, we use the mean of prototype-bart based predictions of the support examples in N M(x; S) as the pseudo label of z. We train the adaptation modules predict the pscudo labels when the test data is fed into. Notice that the feature extractor fo is frozen during test time.
A number of classifiers have been proposed that easily classify unseen test data under the i.i.d. assumption that unseen test data Diest is drawn from the same distribution as training data, i.e. pusin = piess We assume the classifier is deep neural network composed of two parts: feature extractor fo Rdd + I&d and a linear classifier 9w Rdd + Y, where 0 and w are the neural network parameters. ERM optimizes O and w to obtain good classifier for future samples in Diest by min- imizing the objective function C(0, w) = E Duin [[(gw (ff (z)). y)]. where is a loss function such cross-entropy loss. However, under the test-time domain shift (i.e., distribution shift), the i.i.d. assumption between the training and test distributions does not hold, i.e., ptrain + prst, and the trained classifiers often show poor classification performance for the test data.
Prototype-based classification in test-time adaptation Prototype-basel classification refers to a method that obtains prototype representations, which represent each class in the embedding space, and then predicts the label of an input as the class of the nearest prototype. Since labeled data is not available in the TTA setting, T3A (Iwasawa and Matsuo| (021) utilizes a support set that is composed of previous test data and their predictions for the test data by the trained classifier. T3A does not modify parameters of the classifier. Since the embedding space of the feature extractor is unchanged during the test time, T3A constructs the support set using the feature representations for
test data instead the data itself. Specifically, support set S, =S] sy. ... S } is a set of test samples until time l. The support set is initialized with the weight of the last linear classifier, i.e., knkk where w is the parts of w related to k-th class for A: = 1, K. At time I, the
||=
support set updated as
ah sk S
fo(2)) if arg max, Pe =
(I)
otherwise,
where Pk represents the likelihood that the classifier assigns xa to the k-th class. Using the sup- port set St, can obtain the class prototype for class l by taking the centroid of the representa- tions in the support set. Formally, the prototype μą for class k is computed as pa [87 Exeet for l: = 1....... K. Then, the prediction for input 24, is made by comparing the distances between the embedding of I2 and the prototypes, i.e., 9 arg min, d(fe(zz), (c) with a pre- defined metric d such as Euclidean distance or cosine similarity ! Since the wrongly pseudo la- beled examples can degrade the classification performance, the support examples with unconfi- dent pseudo labels are regarded as unreliable examples and filtered out, i.e. at time stamp t,
We use the cosine similarity as distance metric for experiments throughout this paper.
3

---

Published as a conference paper at ICLR 2023
Algorithm Test-time Adaptation via Self- Self-Training with nearest neighbor information (TAST)
Require:
Feature extractor fs. number of adaptation modules Ne, adaptation modules ( J test batch B, support set 8, number of gradient steps per adaptation T, number support examples per each class M,
number of nearby support examples N,, learning rate a
Ensure: Prodictions for all z: B
Update the support set & with eq. (T) and the entropy-based filtering Retrieve nearest neighbors (z; 8) for all E with eq. (2)
for = 1 do 1: do for E B do Obtain the nearest neighbor- based pseudo label p, (-|xx
for 2
0., of with eq.
Compute the prototype-base based class distribution p. (-lar) of x with eq. (
end for Φ; - ti a Eeea CE(p, (A (+12). p (-(x)))))) (+1x))
end for end for Compute the predictions for all x E with eq.
8
f E Sk ((((((()))) S as). where az the M-th largest prediction entropy of the samples from Sy H is Shannon entropy (Lin 1991), and a is the softmax function. T3A mod- ifies only the support set configuration and does not update the trained model parameters test time. Thus, T3A cannot effectively mitigate the classification performance degradation caused by test-time domain shift. To address this issue, we extract useful information for classification of the test data by utilizing multiple randomly initialized adaptation modules that are trained using nearest neighbor-basee pseudo labels.
3
METHODOLOGY
In this section, describe two main components of method TAST: adaptation module uti- lization (Section 3.1 and pseudo-label generation considering nearest neighbor information (Sec-
tion 3.2
3.1 ADAPTATION MODULE
We first discuss the parts to be fine-tuned in the trained classifier before explaining our test-time adaptation method. One possible choice is to fine-tune the whole network parameters in the classi- fier during test time, but this approach can unstable and inefficient (Wang et al.) 2021a Kumar et al. 2022). Another choice is to fine-tune only the parameters of batch normalization (BN) layers in the classifier as in Wang et al. (2021a). Although it achieves effective test-time adaptation, has BN layers in the trained classifier. The other
limitation that it can be utilized only if there choice is to train a new classifier added on top of the frozen feature extractor during test time as in Lee and Chung aa) We construct the new classifier by adding randomly initialized adaptation module as illustrated in Figure [ During the test time, we train the adaptation module and predict the label of the test data using prototype-based class distributions from the adaptation module. The random initialization of the adaptation module may cause performance degradation of trained clas- ensemble scheme (Wen et al. 2020 YM. al.) 20200 Mesbah al.)
sifier. Thus, we consider 2021) to alleviate the issues caused by the random initialization of the adaptation modules to obtain more robust and accurate predictions. We train the adaptation modules independently and predict the label of the test data using the average predicted class distribution from the adaptation modules.
3.2 SELF-TRAINING WITH NEAREST NEIGHIBOR INFORMATION
TAST generates pseudo label distributions for unlabeled test data with the nearest neighbor informa- tion and fine-tunes the adaptation modules with the pseudo label distributions. The whole adaptation procedure of TAST is deseribed in Algorithm [ We first update the support set S and filter out the unconfident examples from the support set as in Iwasawa and Matsuo (202T). Then, we find N, nearby support examples of test data in the embedding space of Jo. We denote N (ar;S) as the set
4

---

Published as a conference paper at ICLR 2023
of nearby support examples of
(x; = {= Sld(fo(z)) 3z),
(2)
where 3, is the distance between I and the N2-th nearest neighbor of a from $ in the embedding space of fe. Each adaptation module is trained individually during test time. For the i-th adaptation module he. [ we compute the prototype representations Ms,1, Pi, in the embedding space of he. o fe with a support set 1, ..,,
.3........ ,S*. ie., 14,6 = |o Eees ha, (z), for = ,,....,,. K. With the prototypes, we compute the prototype-based predicted class distribution of the nearby support examples in the embedding space of ha, o fo. i.e., for 2 E N (Nx:;) B), the likelihood that the prototype-basee classifier assigns to the k-th class is computed as
proto (k|2) = exp(-d (z), Σ. exp(-d(hg) (2), Pt.c)/T²
P2
where is the softmax temperature With the nearest neighbor information, TAST generates pseudo label distribution p. TTSST of x by aggregating prototype-base based predicted class distribution of the nearby support examples in N(x 8) as
TAST (k|=) = N. p.
Σ =EN(
1 [arg max p. ((
(c|2) = k)),
(4)
for k 1,2, .... K. Specifically, we use the one-hot class distributions for pseudo label generation as in Lee (2013); Sohn et al. (2020). Then, we fine-tune the adaptation modules by minimizing the cross-entropy loss between the predicted class distribution of the test example and the nearest neighbor- based pseudo label distribution:
CTAST (o.)
CE TAST (-(p))TT( proto (-,,,)))))))))))).))))))
(5)
D
#E DEs
d(p(
p proto (k(x)) = E. exp(-d(ha. (fo(z)), (o(())).i.c)/T))
k = 10=1,2 1,2, 1,2...., K,
(6)
where CE denotes the standard cross-entropy loss. We iterate the pseudo labeling and fine-tuning processes for T steps per batch. We note that our method does not propagate gradients into the pseudo labels as in Laine and Aila (2017); Berthelot et al. (2019). Finally, we predict the label using the average predicted class distribution is from the adaptation modules, i.e.,
TAST [AAT(( = N Pt
p. proto (k|z)
(7)
(x:V(
N₂ Σ N.
y JIAST- = arg max TAST (c)x) arg max
TAST (c/z)
(8)
Additionally, we consider a variant of TAST, named TAST-BN, that fine-tunes the BN layers instead of adaptation modules. The support set stores the test data itself instead of the feature representa- tions since the embedding space of the feature extractor steadily changes during the test time. The pseudocode for TAST-BN is presented in Appendix B
EXPERIMENTS
In this section, show the effectiveness of method compared to the state-of-the-ard test-time adaptation methods on two standard benchmarks, i.e., domain generalization and image corruption.
We compare TAST with the following baseline methods: (1) Pseudo Labeling (PL) (Lee) 2013) fine-
tunes the trained classifier using confident pseudo labels based on classifier predictions: (2) PLCIf is
a modified version of PL that fine-tunes only the last linear classifier; (3) Tent (Wang al. 2021al la)
2 Detailed explanation about adaptation modules is described in Section 4.1.1. and Appendix A. We set 7 manually 0.1 inspired by Oreshkin et al. (2018) for experiments throughout this paper. More experimental results with different 7 summarized in Appendix
5

---

Published as a conference paper at ICLR 2023
Table 1: Average accuracy (%) using classifiers learned by ERM on the domain generalization benchmarks. We R8 and ResNet-50 as backbone networks. Bold indicates the best performance for each benchmark. Underline indicates the best performance among the baseline methods for each benchmark. Most of the baseline methods degrade the classification performance of the trained classifiers on the benchmarks. However, our method consistently outperforms all the baselines on all of the benchmarks.
Memory usage
Method
Backbone
VLCS
PACS
OfficeHomme
Terralncognita
Avg
74.88.10.46
79.29=0.77
62.10:1.31
10.62.11.19
64.22
+Tent
72.88+0082
83.89=0.54
60.86+0.39
13.90++1.09
62.83
+TentAdapter
67.02.11.16
80.7511.01
62.6410.38
39,91100.76
62.58
+TentCIf
72.96+11448
78.57=1.78
59.33++062
38.30+3.444
62.29
+SHOT
65,2412..29
82.36-0.63
62.5810.39
33.57=1.004
60.94
+SHOTIM
ResNet-18 18
64.86:1.2.22
82.33=0061
62.57++..99
3..3
60.78
+PL.
62.97=22.72
70.98=1.78
58.20+3321
37.44=4.220
57,40
+PLCIf
74.89:10.61
78.11-2.30
61.92.10.41
1..78.11.94
64.18
+T3A
17.26++1.49
80.83-0.677
63.21+00500
40.20++0.60
65.38
+TAST (Ours)
77.27-00677
81.94=0.44
63.70:0.522
42.64:10.72
66,39
+TAST-BN (Ours)
75.21+2..36
87.07=0.53
62.79+0041
39.43+-2.22
66.13
ERM
7..71
83.21=1.14
67.13+10.99
15.93
68.25
Tent
1..96:11.27
0..16
66.29:10.77
37.08:12.04
65.37
*TnttAdpteer
69.65+1.11
83.69=1.16
67.91+0.89
43.89+#1.25
66.29
+TentCIf
7..80+0068
82.66=1.59
66.79++0.98
43.64+2.559
67.22
+SHOT
67.07=0.90
84.07=1.23
67.65:00722
35.20-10.82
63.50
+SHOTIM
Re
66.9311084
84.14.11.25
67.65.10.77
34.37.11.07
63.27
+PL.
69.41=3.12
81.72=4.61
62.85+33005
38.09+22.35
63.02
+PLCIf
75.65
13.33-1.59
6..01111000
46.66.12.12
68.16
+T3A
07.29++0.39
1.192
68.26++0.84
45.61
68.77
+TAST (Ours)
77.66=20.48
8..11=1.22
68.63-+0.70
47.43+22.09
69.46
+TAST-BN (Ours)
73.52.11.37
89.16
68.88.10,50
41.47.2.88
68.26
fine-tunes only the parameters of the BN layers to minimize the prediction entropy of test data; (4) TentAdapter modified version of Tent that adds BN layer between the feature extractor and the last linear classifier, and fine-tunes only the added BN layer; (5) TentClf is a modified version of Tent that fine-tunes only the last linear classifier instead of the BN layers; (6) SHOTIM (Liang et al.) 2020) updates the feature extractor to maximize the mutual information between an input and its prediction; (7) SHOT a method that adds pseudo-label loss SHOTIM; (8) T3A predicts the label of the test data by comparing distances between test data and the generated pscudo- prototypes. Originally, SHOT is one of source-free domain adaptation methods which focus on the offline setting, but compare method with the online version of SHOT for fair comparison.
4.1 DOMAIN GENERALIZATION
The domain generalization benchmarks designed evaluate the generalization ability of the trained classifiers to the unseen domain. The evaluation performed by a leave-one-domang procedure, which domain as test domain and the remaining domains training domains. We use the publicly released code of T3A for the domain generalization benchmarks.
4.1.1 EXPERIMENTAL SETUP
Training setup We test TAST on four domain generalization benchmarks, specifically VLCS (Fang et al. 2013). 2013), PACS (Li al. 2017). OfficeHome /Venkateswara all,2017), and TerraIncog- nita (Beery et al. 2018), For fair comparison, we follow the training setup including dataset splits and hyperparameter selection method used in T3A. We use residual networks (He et al. 2016) in- cluding batch normalization layers with 18 and 50 layers (hereinafter referred to as ResNet-18 and ResNet-50, respectively), which are widely used for classification tasks. We train the networks with various learning algorithms such as ERM and CORAL (Sun and Saenko 2016). Details about the learning algorithms are explained in Appendix
The backbone networks are trained with the
http//tttttth..c /

---

Published as a conference paper at ICLR 2023
Table 2: Ablation studies evaluate the effects of the number of adaptation module and the near- est neighbor information. We use ResNet-18 trained by ERM. TAST- is method that removes adaptation modules from TAST.
Method
N₂
VLCS
PACS
OfficeHome
Terralncognita
Avg
ERM
0.
79.29=0077
12.10+0.31
40.62:#1.19
64.22
+T3A
77.26:+1.49
80.83++0.67
63.21
40.20=10.60
65.38
+TAST-N (Ours)
76.20++1.87
81.62+0.52
63.544+0.3
1..8
65.81
75.20:10.77
81.23+10.70
62.09410.64
42.59:10.41
65.28
5
76.68+0.77
81.81+0.13
63.51+0.59
42.68+00.00
66.
+TAST (Ours)
10
77.43+0.62
81.56+00.5
63.39+00566
42.60++0.63
66.25
20
77.27+0.67
81.94+0.44
63.70+0052
42.64
66.39
default hyperparameters introduced in Gulrajani and Lopez-Paz (2021). We use a BatchEnsemble
(Wen al., 2020). which is an efficient ensemble method that reduces the computational cost by weight-sharing, for the adaptation modules of TAST. The output dimension of each adaptation mod- ule is set to a quarter of the output dimension of the feature extractor e.g., 128 for ResNet- 18. We use Kaiming normalization (He et al., (2155) for initializing the adaptation modules at the beginning of test time. We run experiments using four different random seeds. More details on the bench- marks and the training setups can be found in Appendix A Moreover, discussion on computation complexity such as runtime comparison summarized in Appendix A
Hyperparameters For a fair comparison, the baseline methods use the same hyperparameters as in Jwasawa and Matsuo ). TAST uses the same set of possible values for each hyperparameter with baseline methods. TAST involves four hyperparameters: the number of gradient steps per adaptation T, the number of support examples per each class M, the number of nearby support examples Nº. and the number of adaptation modules Ne. We define a finite set of possible values for each hyperparameter, N. (1, 4,8), T (1,3), and M {1, 20, 50, 100, -1), where -1 means to storing all samples without filtering. Ne is set to 20. We use Adam optimizer with a learning rate of 0.001. More details on the hyperparameters
be found in Appendix Moreover, refer to Appendix C for the sensitivity analysis on hyperparameters including the test batch sizes.
4.1.2 EXPERIMENTAL RESULTS
In [ we summarize the experimental results of test-time adaptation methods using classifiers trained by ERM. Our method consistently improves the performance of the trained classifiers by 2.17% for ResNet-18 and 1.21% for ResNet-50 average, respectively. TAST also outperforms the baseline methods including the state-of-the-arr test-time adaptation method T3A. Compared to T3A, TAST shows better performance by 1.01% for ResNet-18 18 and 0.69% for ResNet-50 on average, respectively. Especially, we find that our method significantly improves the performance of the trained classifiers in the Terralncognita benchmark, which is challenging benchmark in that the trained classifier shows the lowest prediction accuracy. We observe that the performance of the baseline methods, which fine-tune the feature extractors, is lower than that of the classifiers without adaptation, whereas TAST-BN improves the performance of the trained classifiers. Refer to Appendix C for the experimental results of test-time adaptation methods using classifiers trained by different learning algorithms such as CORAL (Sun and Saenko 2016) and MMD (Li et al. 2018).
Effeet of nearest neighbor information To understand the effect of nearest neighbor information, we compare Tent and TAST-BN, both of which fine-tine the BN layers. To adjust the BN layers, Tent uses entropy minimization loss, whereas TAST-BN uses the pseudo-labe loss using the nearest neighbor information. As shown in Table [ the performances of TAST-BN is better than those of Tent by 3.3% for ResNet-18 18 and 2.89% for ResNet-50, respectively. In addition, we consider an ablated variant of TAST, named TAST-N, that removes adaptation modules from TAST. TAST-N optimization- tfrne and has the same support set configuration T3A. T3A uses the prototype- bassdd based prediction of the test data itself, whereas TAST-N uses the aggregated predicted class distribution
Mooe experimental results with different output dimensions are summarized in Appendix

---

Published as a conference paper at ICLR 2023
of the nearby support examples. As shown in Table 2 the prediction using the nearest neighbor information leads to a performance gain of 0.43% on average.
Effect of adaptation modules TAST adds randomly initialized adaptation modules on top of the trained feature extractor as illustrated in Figure and trains the adaptation modules during test time. For each test batch, we update the adaptation modules T times using pseudo label distributions considering nearest neighbor information. We set 7 to or 3 throughout all experiments. To verify that the few step updates are sufficient to train the adaptation modules, we conduct experiments with different T {0, 1, 4, 8). We test on PACS using classifiers learned by ERM while M and N, set to -1 and one of [1, 2, 4, 8}. We summarize the experimental results in Figure 2 observe that the performance of the adapted classifier is better than that of the non-adapted classifier (i.e., = 0) and robust to changes T. Hence, we conjecture that we can obtain a sufficiently good adaptation module with few-step updates similar to Lee and Chung (2021ap.
In addition, to investigate the effect of adaptation modules, we test TAST with a varying number of adaptation modules, e.g., N₂ E {1,5, 10, 20). In Table 2 we find that utilizing single adaptation module leads to degraded performance than TAST-N. However, TAST with multiple adaptation modules shows improvement over TAST-N and T3A on average.
0.810
Method
CIFAR-IOC
CIFAR-10OC
Accursyy
No adaptation
29.14
60.35
+SHOT
15.32
41.54
0.800
+Tent
13.95
39.04
+PI.
22.34
40.06
0.795
N₂
+T3A
26.68
58.28
+TAST (Ours)
26.61
60.74
0.790
+TAST-BN (Ours)
13.08
37.82
+TTT+
14.33
42.38
Figure 2: Sensitivity analysis about N,. the num- ber of nearby support examples, and T, the num- ber of gradient steps per adaptation. Average ac- curacy on test environment A using classifiers learned by ERM on PACS when M is set 1.
Table 3: Average error rate (%) on CIFAR-
10C/100C. We test the highest level of image corruption. Bold indicates the best performance for each image corruption.
4.2 IMAGE CORRUPTION
The image corruption benchmark is designed to evaluate the robustness of classifier to unseen corrupted samples when the classifier is trained using clean samples. We use the publicly released code 6 of TTT+ (Liu et 2021) for the image corruption benchmark. For a fair comparison, we compare our method with the online version of TTT++ which fine-tuner tunet the feature extractor using the instance diserimination task along with matching the feature statistics of training and test time.
4.2.1 EXPERIMENTAL SETUP
We test the robustness of TAST to image corruption on CIFAR-10/100 (Krizhevsky and Hinton, 2009), which is composed of generic images consisting of 10/100 classes, respectively. To make a corrupted test dataset, we apply 15 types of common image corruptions (c.g., Gaussian noise, shot noise) to the test dataset. We call the corrupted dataset CIFAR-IOC/I0N (Hendrycks and Dietterich 2019). We use the highest level (i.e., level-5) of image corruption for this experiment. We use ResNet-50 50 backbone network. For fair comparison, Lf et al. (202T) and the same byperparameters whenever possible. The number of nearby support examples N. adaptation modules N₂ is set to 20, the number of support examples per each class M is set to 100, and the test batch size
the released trained model
1, the number of gradient steps per adaptation T is set to the number of
128. More experimental results with other hyperparameter combinations are summarized in Appendix
httpps:/ tht./gttthhhh...
8

---

Published as a conference paper at ICLR 2023
4.2.2 EXPERIMENTAL RESULTS
The overall experimental results on CIFAR-10C/100G are summarized in Table 3 We note that the best TTA method which achieves effective adaptation in the image corruption benchmarks can be different from that of the domain generalization benchmarks, since the two benchmarks deal with very different types of domain/distribution shifts. From Table | the test-time adaptation algorithms using the frozen feature extractor such as T3A and TAST show poor performance for image corruption benchmarks but better performance for domain generaliza- tion benchmarks, compared to those using the adapted feature extractor such as Tent and TAST-BN. Specifically, TAST-BN outperforms all the TTA methods and TTT++, and it achieves performance gains 1. 1.25% for CIFAR- 10C and 4.56% for CIFAR- 100C on average, compared to Tent, respec- tively. Refer to Appendix for for the detailed experimental results on 15 types of image corruptions.
and 3 we can observe that
RELATED WORKS
Test-time training methods Test-time training methods fine-tune trained classifiers by the self- supervised learning task used at training time. [Sun et (2020) uses a rotation prediction task (Feng et al. al. 2019), which predicts the rotation angle of the rotated images. Liu et al. (2021) use instance discrimination task (Chen et al. 2020). However, TTA methods, our focus in this paper, have no access to any information related to the training procedure. We empirically demonstrated that our
method outperforms the existing test-time training methods on the image corruption benchmark even without the knowledge of the self-supervised learning task.
Source-free domain adaptation methods Source-Free Domain Adaptation (SFDA) methods (Liang et al. 2020 Ishi and Sugiyama 2021] Ych et al. 2021 Eastwood et al. 2022) aim to adapt trained classifiers to unseen test domains without training dataset. SFDA methods mainly focus on the setting that they can access the whole unlabeled test data, whereas TTA methods can access the online unlabeled test data only. Recently, several SFDA methods using nearest neighbor informa- tion (Tang et al.) 2021 Yang et al.] 2021) have achieved good performances in domain adaptation benchmarks. Especially, NRC (Yang al. (222T) is built on the similar intuition that a test data and its nearest neighbors share the same label under domain shift. However, unlike NRC, TAST utilizes adaptation module structures and prototype-bas based classification.
Ensemble scheme in test-time adaptation BACS (Zhou and Levine 202 ). which incorporates
a Bayesian inference framework into the TTA setting, adapts the trained model to an unseen test domain with a regularization term induced by posterior approximated at training time. BACS constructs the ensemble of predictive models to obtain diverse labeling for uncertainty estimates at the beginning of training time and trains the models independently during training time. During test time, BACS averages the predictions of the adapted ensemble members. On the other hand, TAST builds ensemble of adaptation modules alleviate the issues caused by the random initialization of the modules at the beginning of test time.
6 DISCUSSION
We proposed TAST to effectively adapt trained classifiers during test time considering nearest neigh- bor information. We demonstrated the efficiency and effectiveness of our method by conducting experiments on domain generalization and image corruption benchmarks. To the best of our knowl- edge, our work is the first one that utilizes an ensemble scheme that is built at test time for test-time adaptation. We expect that adaptation using the ensemble scheme can be combined with the other methods source-free domain adaptation or test-time training.
One of the limitations of TAST is the extension to large-scale benchmarks. TAST and TAST-BN require good prototypes in the embedding space for prediction and pseudo-labeling. To obtain good prototypes, TAST and TAST-BN construct and update the prototypes using the encountered pseudo- labeled data during the test time. This prototype construction/uudaate, however, can be ineffective for the large-scale benchmarks especially for too many classes and small batch sizes. Detailed discussion of TAST/TAST-BN on large-scale benchmarks and possible improvement of TAST-BN for large-scale benchmarks is described in Appendix D

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This research was supported by the National Research Foundation of Korea under grant 202IRICICI and by the Ministry of Science and ICT, Korea, under the ITTP (Institute for Information and Communications Technology Panning and Evaluation) grant 2020000--.
REFERENCES
E. Arazo, D. Ortego, Albert, N. E. O'Connor, and K. McGuinness, Pseudo-labeling and confir- mation bias in deep semi-supervised learning, 2020.
Beery, G. Van Horn, and Perona. Recognition in terra incognita. Proceedings of the European Conference Computer Vision (ECCV), September 2018.
D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel. Mixmatch: A holistic approach semi-supervised learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019.
B. Chen, J. Jiang, X. Wang, P. Wan, J. Wang. and M. Long. Debiased self-training for semi- supervised learning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
T. Chen, S. Kormblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. H. D. and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020.
M. J. Choi, J. Lim, A. Torralba, and A. Willsky, Exploiting hierarchical context on a large database of object categories. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 129-136, 2010. doi: 00.1109/CVPR.200055550000
K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying generalization in rein- forcement learning. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1282-1289. PMLR, 09-15 Jun 2019.
G. Csurka. Domain adaptation for visual applications:
comprehensive survey. CoRR,
Abs/1702.053344 2017.
C. Eastwood, I. Mason, C. Williams, and B. Schölkopf. Source-free adaptation to measurement shift via bottom-up feature restoration. In International Conference Learning Representationss 2022.
M. Everingham, I. Gool, C. K. Williams, Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. Int. J. Comput. Vision, 88(2):303-338, jun 2010. ISSN 0920-5691. doi: 10.10077/112223-009--775-5
C. Fang, Y. Xu, and D. N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In 2013 IEEE International Conference on Computer Vision, pages 1657-1664, 2013. doi: 10.1109/CCV2.2133.2200
L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Comput. Vis. Image Underst., 106(1):59-70, apr 2007. ISSN 1077-3142. doi: (1.016//.ciuu2200.0099
Z. Feng, C. Xu, and D. Tao. Self-supervised representation learning by rotation feature decou- pling. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10356-10366 10366, 2019. doi: 10.1009/CV..2209.0000
Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1180-1189, Lille, France, 07- 09 Jul 2015. PMLR.
10

---

Published as a conference paper at ICLR 2023
1. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks, In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR, 06-11 Aug 2017.
K. He, X. Zhang. S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026-10344 2015. doi: 10.1 109/11C/.22015.22..
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1109/CVPR.20165
D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
M. Ishii and M. Sugiyama. Source-free domain adaptation via distributional alignment by matching batch normalization statistics. ArXiv, abs/2101.10842, 2021.
Y. Iwasawa and Y. Matsuo. Test-time classifier adjustment module for aggoosic domain gen- cralization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 2427-2440. Curran As- sociates, Inc., 2021.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2014. cite Published conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
A. Kumar, A. Raghunathan, R. M. Jones, T. Ma, and P. Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Repre- sentations, 2022.
Laine and T. Aila. Temporal ensembling for semi-supervised learning. In International Confer- on Learning Representations, 2017.
D.-H. Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop Challenges in Representation Learning (WREPL), 07 2013.
D. H. Lee and S.-Y. Chung. Unsupervised embedding adaptation via early-stage feature recon- struction for few-shot classification. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6098 6108. PMLR, 18 24-24J Jul 2021a.
Lee and S.-Y. Chung. Improving generalization meta-rl with imaginary tasks from latent dynamics mixture. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 27222-27235. Curran Associates, Inc., 2021b.
D. Li, Y. Yang, Y. Song, and T. M. Hospedales. Deeper, broader and artier domain generalization. In IEEE International Conference Computer Vision. ICCV 2017, Venice, Italy, October 22-29, 2017, pages 5543-5551. IEEE Computer Society, 2017. doi: 10. 109//CCV.2017.99.
H. Li, S. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
11

---

Published as a conference paper at ICLR 2023
J. Liang, D. Hu, and J. Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), pages 6028-6039, 2020.
J. Lin. Divergence measures based the shannon entropy. IEEE Transactions Information Theory, 1511::44--151,11 J Jan. 1991. ISSN 0018-9448. doi: 10.1109/18.61115.
Y. Liu, P. Kothari, B. van Delft, B. Bellot-Gurlet. T. Mordan, and A. Alahi. Ttt++:
When does self- supervised test-time training fail thrive? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 11808-21820 Curran Associates, Inc., 2021.
M. Long, H. Zhu, J. Wang, and M. I. Jordan. Unsupervised domain adaptation with residual transfer networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
R. Mendonca, X. Geng, C. Finn, and S. Levine. Meta-reinforcement learning robust distributional shift via model identification and experience relabeling. CoRR, abs/2006.07 07100 2020.
Y. Mesbah, Y. Y. Ibrahim, and A. M. Khan. Domain generalization using ensemble learning. CoRR, abs/2103. 10257, 2021.
S. Motiian, M. Piccirilli, D. A. Adjeroh, and G. Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Mukhoti, V. Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania. Calibrating deep neural networks using focal loss. In Advances in Neural Information Processing Systems, volume 33, pages 15288-15299, Curran Associates, Inc., 2020.
B. Oreshkin, P. Rodríguez López, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
C. Rosenberg, M. Hebert, and H. Schneiderman. Semi superviie self-training of object detection models. 2005 Seventh IEEE Workshops on Applications of Computer Vision WACVMOTIONN'O Volume 1, 1:29-36, 2005.
B. Russell, A. Torralba, K. Murphy, and W. Freeman. Labelme: A database and for image annotation. International Journal of Computer Vision, 77(1-3):157-173, 2008. ISSN 0920-5691. doi:
tool
K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In Proceedings of the 11th European Conference on Computer Vision: Part IV, ECCV' 10, page 213-226, Berlin, Heidelberg, 2010. Springer- Verlag. ISBN 364215560X.
S. Schneider, E. Rusak, I. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robust- ness against common corruptions by covariate shift adaptation. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1539--1551.. Curran Associates, Inc., 2020.
O. Sener, H. O. Song, A. Saxena, and Savarese. Learning transferrable representations for un- supervised domain adaptation. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk. A. Kurakin. H. Zhang, and C. Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07760 2020.
12

---

Published as a conference paper at ICLR 2023
B. Sun and K. Saenko. Deep CORAL: correlation alignment for deep domain adaptation. CoRR, abs/1607.01777 2016.
Y. Sun, X. Wang, L. Zhuang, J. Miller, M. Hardt, and A. A. Efros, Test-time training with self- supervision for generalization under distribution shifts. In ICML, 2020.
Tang, Y. Yang. Z. Ma, N. Hendrich, F. Zeng, S. S. Ge, C. Zhang, and J. Zhang. Near- est neighborhoodd-asee deep clustering for source data-absent unsupervised domain adaptation. CoRR, abs/2107. 12585, 2021.
R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18583 18599. Curran Associates, Inc., 2020.
E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko. Simultaneous deep transfer across domains and tasks. In ICCV, 2015.
H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsu- pervised domain adaptation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5385-5394, 2017.
D. Wang. E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021a
J. Wang, C. Lan, C. Liu, Y. Ouyang, and T. Qin. Generalizing to unseen domains: A survey on domain generalization. In Z.-H. Zhou, editor, Proceedings of the Thirtieth International Joinr Conference on Artificial Intelligence, ISCAI-21, pages 4627-46 1627-4635. International Joint Confer- ences on Artificial Intelligence Organization, 8 2021b. Track.
doi: 10.24963/jjcii.222211 20211/628.. Survey
Y. Wen, D. Tran, and J. Ba. Batchensemble: alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
M. Xu, J. Zhang, B. Ni, T. Li, C. Wang, Q. Tian, and W. Zhang, Adversarial domain adaptation with domain mixup. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 6502- 6509. AAAI Press, 2020.
S. Yang. Y. Wang. J. de weijer, L. Herranz, and S. JUI. Exploiting the intrinsic neighborhood structure for source-free domain adaptation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
H.-W. Yeh, B. Yang. P. C. Yuen, and T. Harada. Sofa: Source-data-free feature alignment for unsupervised domain adaptation. In 202/ IEEE Winter Conference on Applications of Computer Vision (WACV), pages 474 483, 2021. doi: 10.1 11110///AV8633..20000
.00052. 202.0005...
A. YM., R. C., and V. A. Self-labelling via simultancous clustering and representation learning. In International Conference Learning Representations, 2020.
A. Zhou and S. Levine. Bayesian adaptation for covariate shift. In A. Beygelzimer, Y. Dauphin,
P. Liang, and J. W. Vaughan, editors, Advances in Newral Information Processing Systems, 2021.
13

---

Published as a conference paper at ICLR 2023
A BENCHMARK AND IMPLEMENTATION DETAILS
A.1 DOMAIN GENERALIZATION BENCHMARKS
We test on four domain generalization benchmarks, specifically VLCS (ang et al.; 2013). PACS
(Li et al. 2017). OfficeHome (Venkateswara al. 2017). and TerraIncogmita (Beery et al. 2018).
VLCS is composed of photographic images from four different datasets (PASCAL VOC207 (Ever-
ingham et al. 2010), LableMe (Russell et al.) 2008), Caltech 101 (Fei-Fei et al.. 2007), and SUN09
[Choi et al. 2010)), consisting of 10,729 examples of 5 categories (bird, car, chair, dog, and per- son). PACS composed of images of objects from four different domains (photo, art, cartoon, and sketch), consisting of 9,991 examples of 7 categories (dog, elephant, giraffe, guitar, horse, house, and person). OfficeHome is composed of images of objects in the office and home from 4 different domains (artistic images, clip art, product, and real-world images), consisting of 15,588 examples of 65 categories (e.g., alarm clock, backpack, and batteries). Terralncognita is composed of wild animal images taken from 4 different locations (I.100, 1.88, L43,, and 1.46), consisting of 24,788 examples of 10 classes.
A.2 IMPLEMENTATION DETAILS ON DOMAIN GENERALIZATION BENCHMARKS
We follow the dataset splits and the hyperparameter selection method used in T3A. We split each dataset of training domains into training and validation sets. The training and validation sets are used for network training and hyperparameter selection, respectively. Specifically, we split each dataset into 80% and 20% and use the smaller set as the validation set. We choose the hyperparameters that maximize the validation accuracy of the adapted classifier. This hyperparameter selection method is called the training- domain validation. We train backbone networks using four different learning algorithms: ERM, CORAL, MMD, and Mixup. ERM is explained in Section 2 of the manuscript; CORAL aims to obtain domain-invariant representations by aligning covariance matrices of train- ing data and test data; MMD tries match the training and test data distributions using the MMD measure; Mixup trains classifiers using mixed images/features and mixed labels created by linear in- terpolation of examples from the training domains. We run experiments using four different random seeds: 0, 1, 2, and 3.
All the hyperparameters for training and test-time adaptation are taken from T3A and DomainBed We train the network with Adam optimizer with default hyperparameters introduced in DomainBed, e.g., a learning rate of O. 0.00005, weight decay of 0, a dropout rate of O, and a batch size of 32. In ad- dition to the hyperparameters for test-time adaptation described in Section 4.1.1 of the manuscript, there is one more hyperparameter 3 for the bascline methods. The learning rate for test-time adap- tation is obtained by multiplying 3 to the learning rate used in training time. We set the confidence threshold for PL and PLCIf to 0.9. The possible values for / are set to 0.1, 1.0, and 10.0. For TAST-BN, we restrict the size of the whole support set to 150 due effective memory usage and reduced runtime since the test data and the support examples are fed into the classifier for every test batch.
A.3 IMPLEMENTATION DETAILS ON IMAGE CORRUPTION BENCHMARKS
We use the same hyperparameters introduced in TTT++. We train ResNet-50 for 1000 epochs using the classification and instance discrimination tasks jointly. The weight on the instance discrimination task for balancing the two tasks is set to O.I. For the instance discrimination task, we use the same data augmentation schemes of TTT++, e.g., RandomResizzeCrpp RundomHorizontalCrop HorizontalFlip, ColorJitter, RandomGrayscale, and Normalization. We set the batch size for training the networks to 256. At test time, Pl., SHOT, and TTT++ use SGD optimizer with learning rate of 0.001 and a momentum of 0.9. On the other hand, Tent, TAST, and TAST-BN use Adam optimizer with a learning rate of 0.001. We set the batch size to 128 during the test time due to effective memory usage. We run experiments using four different random seeds: 0, 1, 2, and 3. We set the confidence threshold for PL and PLCIf to 0.9. For PL. we adjust only the BN layers in the trained model as in Tent. For TAST-BN, we restrict the size of the whole support set to 200. However, even in -EFAR-100C experiments, we can store only two support examples per class if the support set size is fixed 200. Thus, we do not restrict the size of support set for TAST-BN on CIFAR- 100C.
14

---

Published as a conference paper at ICLR 2023
A.4 RUNTIME COMPARISON
Table 4: Mean runtime (sec) to adapt classifiers that use ResNet-18 as a backbone network with a single hyperparameter combination (I' 1, = 8, M -)).
Method
VLCS
PACS
OfficeHome
Terralncognita
Tent
53.32
15.17
50.56
76.48
TentAdapter
0.59
0.43
0.64
0.92
TentCIf
0.52
0.40
0.60
0.81
SHOT
55.10
20.97
54.66
77.08
SHOTIM
54.82
20.73
54.63
77.00
PL.
55.03
20.75
54.53
77.02
PLCI
0.57
0.43
0.62
0.92
T3A
0.62
0.58
1.61
TAST (Ours)
7.71
6.92
12.74
23.69
TAST-BN (Ours)
81.54
73.93
114.33
179.48
We conduct our experiments on TITAN XP. We report the average runtime spent to adapt classifiers that use ResNet- 18 as a backbone network in Table note that TAST, which updates the support set and the adaptation modules, requires only 1/3 to 1/4 running time compared to the methods that update the entire feature extractors, e.g. SHOT or SHOTIM. On the other hand, TAST-BN, which updates the support set as well as the BN layer, requires more running time (about 2x) compared to SHOT or SHOTIM. The overhead is not significant though due to the online setting.
A.5 DETAILS ABOUT ADAPTATION MODULES
We use BatchEnsemble (BE) for the adaptation modules of our method. BE is simple and efficient ensemble method that greatly reduces computational cost by weight-sharing. Each ensemble member of BE is composed of two layers with a shared weight and rank-one factors. Specifically, the weight matrix of j-th ensemble member the rank-one factor of j-th ensemble member. Although the existing deep ensemble (DE) methods do not share any weights, all ensemble members share W, and thus BE reduces the number of parameters compared DE. Moreover, unlike DE, only the last layer of all ensemble members of BE are different, and thus it can be easily vectorized and trained simultaneously. Therefore, BE greatly reduces the computation cost.
o rrs" where W is a shared weight and rys is
The adaptation module structure is used in many fields such as self-supervised learning (which is often called "projection head"). Although the existing methods mainly focus on training time, TAST focuses on test time. For example, SimCLR (Chen et al. 2020) adds a projection head on the top of a feature extractor the beginning of training time and trains the feature extractor and the projection head with an instance discrimination loss. After the training time, for downstream tasks, SimCLR uses feature extractor outputs rather than projection head ones. However, TAST adds adaptation modules at the beginning of test time and trains the modules with the nearest neighbor-based pseudo- label distribution. To predict the label of test data, we use the averaged predicted class distribution from the adaptation modules.
B PSEUDOCODE FOR TAST-BN
We present the pseudocode for TAST-BN in Algorithm TAST-BN fine-tunes the BN layers the feature extractor instead of adaptation modules. Since the embedding space of the feature extractor steadily changes, the support set stores the test data itself instead of the feature representations. Formally, support set S, (S), ... SK is set of test samples until time t. The support set is initialized as an empty set. At the time t, the support set is updated as
{xt), if arg max, Pe = k otherwise,
st =
(9)
sk
15

---

Published as a conference paper at ICLR 2023
Algorithm 2 TAST-BN
Require:
Feature extractor fe, test batch B, support set 8, number of gradient steps per adaptation T, number of support examples each class M, number of nearby support examples N., learning rate
Ensure: Predictions Vz for all
B
Update the support set & with eq.
(9) Section
Retrieve nearest neighbors.
7 S) for all x E with eq. (10) in Section
for = do
Compute prototypes (1u: ) using the support set in the embedding space of fo for 2 (ar; do exp( Sxp(-d(folee (px=="(([()))ee))
p p=(()( (k))2)
1,2,
Ee expl (xp(-ddia(Shuee))2))
end for
for
IB
T (k(zz)
1
N, ExeN( [arg max, (f = k), 1.2,...,K K k = 1,2 1.2...,,
prom (A()z)
expl di fo))
Σ. expl (lopll-dlols))eeee///
end for
VV 13
EEER CE(P (11x), (1(x))
end for
for
do JAST (Mxx) N, EEE podl (M|2), 1.,,...,
p JAST
Uc - arg max, p TAST (AST(c()))
end for
where pk is the likelihood the classifier assigns 2, to the class Using the support set, we retrieve N, nearby support examples of in the embedding space of fo, i.e.,
(x: = {a S)d(fe(z). fo())) B,).
(10)
where B. is the distance between x and the N,-th nearest neighbor of x from S in the embedding space of fa. Then, generate a pseudo label distribution for the test data and fine-tune the BN layers to match the nearest neighbor- pseudo label and prototype-based class distributions for the test data with the same procedure described in Section 3 of the manuscript.
C ADDITIONAL EXPERIMENTS
C.1 EXPERIMENTAL RESULTS USING CLASSIFIERS TRAINED BY DIFFERENT LEARNING ALGORITHMS
In Table 5 we show the results of test-time adaptation methods using classifiers trained by three different learning algorithms, namely CORAL, MMD, and Mixup. TAST consistently enhances the performance of the trained classifiers on the benchmarks by 1.73%. 1.81%. and 2.30% on average using the classifiers trained by CORAL, MMD, and Mixup, respectively. We find that TAST has a minor performance gain compared to the results in Table 1 of manuscript, whereas it surpasses T3A on most of benchmarks. Compared to T3A, TAST shows better performance on the benchmarks by 0.21%. 0.14%, and 0.40% average with the classifiers trained by CORAL, MMD, and Mixup, respectively. Refer to Section 4 in Appendix for the experimental results of the other baseline methods.
C.2 FINE-TUNING BOTH ADAPTATION MODULES AND BN LAYERS SIMULTANROUSLY
We consider a method, named TAST-both, that fine-tunes both the attached adaptation modules and the BN layers in the feature extractor simultaneously. Table reports the experimental results using classifiers learned by ERM on domain generalization benchmarks. We use ResNet-18 as a backbone network. As shown in Table 6 TAST-both shows worse performance than TAST-BN and TAST. We conjecture that the random initialization of adaptation modules and the changes feature representation due to BN layer training negatively affect the learning of the other layers.
16

---

Published as a conference paper at ICLR 2023
Table 5: Average accuracy (%) on domain generalization benchmarks using classifiers trained by different learning algorithms, namely CORAL, MMD, and Mixup. We use ResNet-18 18 back- bone network. Bold indicates the best performance for each benchmark. TAST and TAST-BN consistently improve the performance of the trained classifiers and they outperform T3A on most of benchmarks.
Method
VLCS
PACS
OfficeHome
Terralncognita
Avg
CORAL
1..3
81.00.10.79
62.78.10.06
36.51
63.57
+T3A
1.67
82.75=0.51
63.72:+0.32
38.39:1..0 1.39
65.09
+TAST (Ours)
74,82#2.44
83.16=00.81
64.00:10.25
1.72
65,30
+TAST-BN (Ours)
7..01
87.21=0.57
62.98:00.23
37.45=t.111
66.16
MMD
74.90=1..50
81.06=0..2
62,20:10,48
35.73+2..70
63.47
+T3A
77.28:10.45
82.52=0.53
03.34
37.40
65.14
+TAST (Ours)
76.21
83.2910.26
63,4911.49
38.12::2.47
65.28
+TAST-BN (Ours)
76.06+00.89
86.35+0.76
63.22++0.26
39.46++1.63
66.27
Mixup
7..9
78.2910.88
61.83100.8
41.04:1.001
64.03
+T3A
78.43:10.76
81.9110.54
63.49110.86
39.89:10,90
65.93
+TAST (Ours)
77.19=00.80
82.85+0.36
63.83++0.74
1..44+1.67
66.33
+TAST-BN (Ours)
76.8
87.14=0.56
62.09+00.86
42.70++1.90
67.21
Table 6: Average accuracy (%) using classifiers trained by ERM on domain generalization bench- marks. We use ResNet-1e 18 as a backbone network. TAST-both is method fine-tunes both the attached adaptation modules and the BN layers simultaneously. TAST-both shows worse perfor- mances than TAST-BN and TAST.
Method
N.
VLCS
PACS
OfficHHome
TerraIncognita
avg
ERM
04.8
79.30=0.77
62.0941.31
10.63+1.119
64.22
+T3A
77.26.t 1.49
80.83:10.67
63.2140.50
40.20:10.60
65.38
+TAST-N
1.87
81.62+00.22
63.54400.3
1..8
65.81
+TAST-BN
2..21
87.07
62.79+0.41
39.43+4..22
66.13
75.20
01.23+0.70
62.09
42.59+0044
65.28
76.68+00.77
81.81
63.51+0.59
42.68++0.80
66.17
+TAST
10
77.43-0.62
81.56+0.885
63.39+0.56
42.60++0.63
66.25
20
77.27-0.67
81.94.1.44
63.70.00.52
42.64.1.772
66.39
73.35+00.57
84.85
61.70
39.27+2.05
64.79
5
73.88+0.35
84.99+0.63
+0.41
1.516
64,96
+TAST-both
10
73.66.11.57
85.13
62.03
38.50.11.25
64.83
20
75.16.10.17
85.52:10.05
62.01
38.54.L1.52
65.31
C.3 EXPERIMENTAL RESULTS USING DIFFERENT HYPERPARAMETER ON CIFAR-10C
In Table of the manuscript, we report the experimental results when N₂ and M are set to and 100 on the CIFAR-IOC, respectively. In Table 7 summarize the experimental results using different combinations of N, and M on the CIFAR-TOC. There are two observations in Table (1) T3A has shown the best performances when M is set 100; and (2) TAST and TAST-BN perform better with smaller N..
C.4 SENSITIVITY ANALYSIS ON HYPERPARAMETERS
We follow the hyperparameter selection method used in T3A. We split the dataset of training do- mains into training and validation sets. The validation set is used to select hyperparameters that maximize the validation accuracy of the adapted classifier. On the other hand, for the image cor- ruption benchmark, we use manually determined hyperparameters as Tent. Thus, we summarized experimental results other combinations of hyperparameters in Table
Additionally, we investigate the sensitivity of two hyperparameters which are set manually through- out all experiments, the softmax temperature and the output dimension of adaptation modules
17

---

Published as a conference paper at ICLR 2023
Table Average error rate (%) in the online setting on CIFAR-IOC with different hyperparameters.
Method
M
N.
guuss
brit
Jefoc
class
Eog
frost
glass
impul
JPe2
sbot
adaptation
13:27
23.38
21.93
10:84
+TA
41.56
8.28
13.27
13:45
22. 18
28.59
27.18
46.43
55.11
18.96
22.59
42.92
10.32
21.77
10.53
+T3A
44.65
7.94
14.11
13.34
22.60
29.00
78.57
45.92
56.03
19:67
24.16
40.18
40.55
22.61
12.02
+TA
20
44.26
7.72
13.82
13.35
22.24
28.71
28.36
45.69
55.87
19.34
23.76
39.82
40.38
22.44
12.20
+TA
50
12.82
7.43
13.65
12.36
22.15
28.54
27.69
41.42
54,91
19.13
22.83
38.35
38.53
22.09
11.15
13.61
26.52
37.84
21.97
10.72
+T3A
7.33
13.56
11.63
11
29.07
27.56
46.79
55.16
22.77
41
39. 59.58
22:23
10:34
-TAST
47.24
8.68
12.93
16.74
22.31 31
28.66
27.23
48.76
55.97
19.21
22.63
48.32
42,80
21.57
10:34
+TAST
47.19
9.78
15.88
15.58
24.30
30.94
29.70
48.66
58.05
21.49
27.57
41
44.21
24.58
14.75
+TAST
48.08
11.34
17.73
17.10
25.99 93
31.99
30.54
49.77
58.54
23.72
29.55
44.09
45,55
26.23
16.41
-TAST
16.82
25.55
31.78
45.35
25.63
-TAST
14.01
12.97
21.90
25.73
21.76
19.78
23.63
58.77
39.22
22.73
12:13
+TAST
20
44.38
8.22
14.51
13:73
22.28
29.07
28.66
46.30
55.92
20.49
24.39
40
40.38
23:37
12.26
-TAST
20
44.41
$.19
14,26
13,60
22.22
29.05
28.81
46.17
$6.05
20.09
24.24
40.33
40.39
23.18
12.67
-TAST
12.47
7.57
13.65
12.14
21.48
28.50
26.88
41,99
54.52
19.26
22.76
57.5
37.73
22.06
11.03
-TAST
22.43
11.25
TAST
13,90
12:15
28.51
45.10
54.93
19.71
22.87
38.21
22.29
11.02
TAST
100
42,02
7.34
13:55
11.86
21. 38
28.58
26.51
44,99
$4,19
18.96
22.55
$7.08
37.62
21.84
10.64
-TAST
100
42.35
7.61
13.89
11.95
21.50
28.75
27.06
45.15
54.55
19 56
22.77
38. 38.09
37.64
21.99
10.73
TTST
100
42.34
7.50
13,80
11.72
21.45
26.32
76.89
44.75
$4.46
19 19.18
22.54
58.00
37.61
21.97
10.67
-TSTT
45.20
14.05
22.87
30.19
23.87
41.35
22.65
10.20
TAST
13.88
11.62
22.03
29.38
42.67
22.48
10:30
+TAST
44.93
7.36
13:64
11.17
21.57
28.82
28.17
49.63
58.74
19.64
22.41
43.38
41.02
22.15
9.86
-TAST-BN
19.16
11.95
11.02
12.84
21.51
19.38
16.64
26.03
27.26
17.93
15.29
14.01
19
17.94
11.78
-TAST-BN
16.21
8.4
8:48
9.56
17.92
15.59
13.06
23.1
23.68
13.58
12.74
11.06
14,85
14,02
8.36
-TAST-BN
19.51
16.17
8:95
-TAST-BN
11.05
20.66
26.1
17.11
10.22
-TAST-BN
20
1491
7.68
7,81
8.62
16.81
15.10
12.25
21,82
22.54
12.38
11.67
10.34
13.77
12.99
7.57
-TAST-NN
20
15.11
7.85
7.96
8.75
17.00
15.02
12.32
22.07
22.54
12.57
11.99
10.50
13.98
13.12
7.70
-TAST-BN
20
15.00
7.98
8.00
8.71
16.87
14.89
12.24
21,88
22
12.58
11.89
10.47
14.06
13,06
7.59
dø- We set T and de to 0.1 and dz/4, where d, is the output dimension of the feature extractor. In Table 8 we report the average accuracy of the adapted classifier by TAST with the different combinations of J and dæ. In the experiments, we use ResNet-18 18 as backbone network trained by ERM on PACS, which is one of the domain generalization benchmarks. We experimentally show that the performance of TAST is robust to changes in 7 and dg. We especially think that the clas- sification performance of TAST is not significantly affected by changes in T because T affects both the prototype-basee based predicted class distribution of test data and the new pscudo-label distribution using nearest neighbor information and then we train the adaptation modules with the cross-entropy loss affected by only few times per each test batch during test time. Moreover, a similar classification performance regardless of the dimension of adaptation modules similar to Chen et al. 2022)).
observe
Table 8: Sensitivity analysis about the softmax temperature and the output dimension of adap- tation modules dø- Average accuracy on test environment A using classifiers learned by ERM on PACS.
de dz/4 (used)
testenv:A
d=
d.
dz/8
d=/16
10
0.8025
0.8028
0.8024
0.8023
0.8026
1
0.8034
0.8038
0.8029
0.8034
0.8034
T
0.1 (used)
0.8031
0.8038
0.8056
0.8038
0.8034
0.01
0,8026
0.8018
0,8025
0,8028
0,8023
0.001
0.8020
0.8030
0.8023
0.8019
0,8001
We used the test batch size as in T3A and Tent for domain generalization and image corruption benchmarks, respectively, as described in Appendix and Section 4 of the manuscript. We summa- rize experimental results using different test batch size. We conduct experiments using classifiers, which have ResNet-18 backbone networks, learned by ERM on PACS. As shown in Table [2] can find that Tent and PL show reduced performance in experiments using smaller test batch size, but T3A, TAST, and TAST-BN are robust to changes in test batch size.
18

---

Published as a conference paper at ICLR 2023
Table 9: tation modules dø. Average accuracy on test environment C using classifiers learned by ERM on PACS.
Sensitivity analysis about the softmax temperature and the output dimension of adap-
de da/4 4 (used)
testenv:C.
d2
dz/2
dz/8
dz/16
10
0.7842
0.7838
0.7841
0.7838
0.7835
0.7830
0.7836
0.7829
0.7837
0.7836
0.1 (used)
0.7817
0.7815
0.7826
0.7816
0.7824
0.01
0.7810
0,7816
0.7842
0.7832
0.7825
0.001
0.7811
0.7796
0.7802
0.7806
0.7803
Table 10: Sensitivity analysis about the softmax temperature and the output dimension of adap- tation modules dz. Average accuracy on test environment using classifiers learned by ERM on PACS.
dø d=/4 (used)
testenv:P
d2
d=f2
d=/8
dz/16
10
0.9611
0.9611
0.9614
0.9613
0.9609
0.9614
0.9614
0.9611
0.9614
0.9616
T 0.1 (used)
0.9615
0.9620
0.9644
0,9613
0.9606
0.01
0,9621
0,9609
0,9613
0.9606
0.9604
0.001
0.9611
0.9615
0.9607
0.9606
0.9613
Table 11: Sensitivity analysis about the softmax temperature and the output dimension of adap- tation modules dæ. Average accuracy on test environment using classifiers learned by ERM on PACS.
de dz/4 (used)
testenv:S
d,
d./2
dz/8
dz/16
10
0.7180
0.7208
0.7186
0.7155
0.7185
1
0.7175
0.7214
0.7180
0.7211
0,7202
0.1 (used)
0.7211
0.7222
0.7252
0.7216
0.7220
0.7236
0.7232
0.7223
0.7225
0.7209
0.001
0.7246
0.7196
0.7251
0.7235
0.7225
Table 12: Ablation studies to evaluate the effects of the test batch size.
Batch size 32 (used)
Methods
16
64
128
ERM
79.31 0.75
79.30=0.76
79.29:10.77
09.79
79.28=0.70
+Tent
49.52:00.4
81.16:0..46
83.89=0.54
83.90=0.54
83.85=0.12
+SHOT
81.44+t0.32
0.7
82.36++0.63
83.18+0.34
82.95-0.33
+PI.
67.90+44.226
70.33+13.53
70.98+1..78
77.52+22.99
78.90=0.39
+ +T3A
81.21++0.76
81.22+0.69
80.83+0.667
81.20+0.73
81.27+0.6
+ +TAST (ours)
81.81:10.35
1..52-11044
81.9410.44
81.92-0.87
81.69=0.64
+ TAST-BN (ours)
86.78-10.78
86.66111224
87.07-0.533
86.90-1049
86.92
D TAST ON IMAGENET-C
ImageNet-C is an image corruption benchmark such CIFAR-10/100C, but it is large-scale benchmark composed of larger images from more diverse classes. ImageNet-C challenging for the existing test-time adaptation/training methods including TTT++ Also TAST and TAST-BN may struggle with ImageNet-C, since TAST and TAST-BN require prototypes to represent each class in the embedding space. To obtain good prototypes, sufficient amount of data per class is required, but we have no access to any labeled data due to TTA settings. Pseudo-labeling alleviates this issue CIFAR- PEFAR-10/1000 10/100C, but ImageNet-C due the following concerns:
19

---

Published as a conference paper at ICLR 2023
The prototype updates of TAST and TAST-BN are based the estimated labels of test data by the classifier, not the ground-truth labels. Under test-time domain shift, classifier bias may occur, which may result in assigning most test data only to subset of classes. As observed in [Chnn et al. (2022), the classifier hias often occurs under the covariate shift such image corruption and style transfer. Then, even after large number of hatch updates which cover all the ground-truth classes by at least one sample, some prototypes may have not been updated since no previous test data has been classified to those classes. For example, we found that for the experiments with Gaussian noise, it took 768 batches out of 782 batches until all the prototypes were updated least by once.
Since the number of classes (1000) is much larger than the test batch size (64), few proto- types for our method are updated per each test batch while the remaining prototypes remain unupdated. It might affect the performance of the prototype-based classification. To ad- dress this issue, it might require a batch size larger than 1000, which is impossible due to the hardware cost.
When the number of classes (1000) is much larger than the test batch size (64), obtaining good prototypes for TAST-BN can be difficult especially the early stage of test time explained above. To alleviate the concerns, we consider a variant of TAST-BN, in which the prototypes are initialized with the weight of the last linear classifier as in TAST and fixed during the test time. We call this variant TAST-BN (w/ fixed prototypes). In Table 13 we report the experimental results (test accuracy) ImageNet-C with severity level when we set (N,, M, T to (1, 1).
Table 13: Accuracy of TAST-BN (w/ fixed prototypes) on ImageNet-C
Nansdag TAST-BN tw/ Exod proteyppo
0.300
0.152
0103
0.50
0021
01582
0013
0,116
01000
01030
0.701
0.6498
0.1906
0.1620
0.4495
0.460
0.3422
0.185
0.1645
6.7422
6.4133
0.2856
0,5940
0.72
0.3615
0.4014
0.3295
Of course, one can still update the prototypes over the test time, but the performance gain from the updating may not as significant as before. Nonetheless, from the result of Table we can see that the effective adaptation ImageNet-C can be achieved with the combination of the prototype approach and self self-training (entropy minimization) method of TAST-BN (w/ fixed prototypes).
E FULL RESULTS
Table 14: Full results using classifiers trained by ERM for Table of the manuscript on VLCS. We use ResNet-18 18 as backbone network.
Method
Avg
ERM
94.70k 1.33
63.79 1.30
67.00+1997
73.154
*Tent
29.82-22.89
-TentAdapter
79.80-4.74
58.51
44
616221992
68.14::174
67.02
*TantCI
1.45
63.74
67.92-2222
65:40-6991
72.6
+SHUT
91.45-6.83
48.26
24.75--2999
66.51:1125
65.24
+SHOTTM
00.78:7.000
47.961144
66.57:1.19
4PL.
23.57=2.24
59.222511
50.8--9550
23.01
62.97
+PL,CII
94.67=1.38
63.64223311
67.00:2::21
73.342::00
74,89
97.52
65.3222.04
70.70-13444
1.75
-TAST (Ours)
99.17:00.00
6..87:11190
6..13:11.76
755
77.27
+TAST-NN (Ouus)
92.60-88.66
64.75
67.27++3.14
7623343773
75.21
20

---

Published as a conference paper at ICLR 2023
Table 15: Full results using classifiers trained by ERM for Table of the manuseript on PACS. We use ResNet-18 as a backbone network.
Method
P
ERM
7..78=0881
75.0==13 1.22
95. 194:0.29
00.11:1.22
79.29
82.21=1.00
81.20=0051
95.32400.8
768241199
Tenttddapeer
95.7740.40
2.78
TTnnCCT
78. 18.16=1.00
75.01 1.53
95.50=00335
65.0015.006
78.57
-SHOT
01.09-0886
09.68=0.91
96. 18:10.27
72.48:2.04
82.36
+SHOTIM
01.10=0.00
09.6620005
18:0.77
22.15:2.001
82.13
76.42:4.99
95.70+0.56
-PECII
79.09 1.41
75.6612..3
95.43:00333
62.48:17.31
78,
-T3A
7..81
77,1421.200
95.02:0:36
7.44421663
80.83
+TAST (Ours)
80,56:10.53
0..96:10.99
96.41:0:.20
72.52:100777
$1.94
HAST-EN
36.49-0.20
83.70
97.23-00111
8085::1142
Table 16: Full results using classifiers trained by ERM for Table of the manuscript on OfficeHome. We use ResNet-18 18 as a backbone network.
Method
R
FRM
55.19:00.49
47 76:2 1.02
72:22:00.55
73.21:089
62.10
03.39=0.61
70.50::068
71.20:0.72
*TantAdatter
55.53=20.33
7301+1123
TTahCHI
58.17=20.67
36.73 1:94
72:01 1.55
73.22.097
59.11
ASHOT
55.14=00.77
50..7= 1.18
71:09:0455
73211091
62.58
*SHOTIM
55.08:00.56
50.294
71:11
7321-0000
$4.49:11006
14.66:13.13
72.00:0065
.PLCIE
55.14:20:70
47.20
72.21:20.54
72.62:096
61.92
+T3A
55.10:20.74
49.5 49.56211.14
74.10:20:55
1..0721.188
63.21
+TAST (Qurs)
36.15:10.68
$0.04::1.31
74.30--0.28
74,28
63.70
+TAST-BN (Oun)
55.11:10.58
51.35::0.85
72.58::0.00
72.13:10.78
62.79
Table 17: Full results using classifiers trained by ERM for Table of the manuscript TerraIncog- nita. We use ResNet-18 18 as a backbone network.
Method
L100
L38
L43
1/16
ERM
37.18-2:2.6
4.212:24.20
53. 18-118-:100 1:27
36.02:11.31
40.62
38.20-20.8
3.91
41 11:51::559
<TearAdapter
+TeaKCIf
14.33::13.31
34.19-5.766 5:76
52.71
$1.86:22.26
38.30
+SHOT
13.87-0.66
28 2.10
40 2.09
30.83:11.26
33.57
+SHOTIM
33.83-21.29
28.13=2.50
20.11
30.64:2::46
$1.92-1..19
22.26:2-.21
.PI CIr
45.22:12.45
36.03-5..81
52. 76 1:54
33. 3.:20722.227
41.78
+T3A
36.22ch 1.89
40.088:1::5
50.72:::000
13.79:11.25
40.20
*TAST (Ours)
43.67:22.83
39.24:23.79
52.6411:000
35.01:2::000
42.64
+EAST BN (Ours)
51.06:67.31
32.74::7.54
41.70:12.86
32.21:3:.05
39.43
Table 18: Full results using classifiers trained by ERM for Table of the manuscript on VLCS. We use ResNet-50 as a backbone network.
Mishod
Avg
ERM
92.66 D.66
63.87 1.71
71.21+11522
76.71
92.36-22444
58.46 3.29
673442245
73.19:42.08
TentAAdatter
.TeunCaf
97.61=0.58
63.67
6..77
73.16: 1:31
75,80
+SHOT
98. 98.72=
46.82 2.57
15.70-1778
07.04-2-288
67.07
+SHOTIM
98.652144
2.5522
55.112233
66.73:282
66.93
59.45-19.34
*PLCIE
97.63-00664
63.36
69.74-6-0778
71.66:44.53
75.65
~T3A
09.17-0.38
64.78:1611
73.01
72.30-22.44
77.29
-TAST (Ours)
99.35:20.30
65.61:1:.78
73.63:13.58
72.01:1::66
77.66
-TAST-BN (Oun)
96.09-2:.40
60.22266.08
65.78:16.511
71.99:1..00
73.52
21

---

Published as a conference paper at ICLR 2023
Table 19: Full results using classifiers trained by ERM for Table of the manuscript PACS. We use ResNet-50 as a backbone network.
Method
C
ERM
82.92:11.66
38.05
96. 16.5011:.332
33.38:13331
83.21
TTent
83.54:2 1.32
84.90 1.35
05:55:00:31
77.74 77.742:11 1.36
85.16
-TentAdapter
TTmiClf
83.00:11.87
77.86:24220
96.53:1:0.36
73.25:2:114
82.66
SHOTT
84.67
80.1721.39
96. 06.582:0.52
24:86:2295
84.07
+SHOTIM
$4.62:11..9
80.24:: 1.41
96.54:10.46
75.16:12288
$4.14
76.35
69 11.24
+PLCIE
$3.88-42.000
78.93- 3.68
96.53-2-0.40
73.96-1-6008
$3.33
+T3A
83.56:12..3
3.14
96.9911:0244
75.26:133.77
83.92
ITAST (Uurr)
83.85=22.055
96.93-b0.27
76.49::.113
84.11
-TAST-RN (Chus)
27.11=22.04
88. 1.93
97.79::0.47
83.23:2144
89.16
Table 20: Full results using classifiers trained by ERM for Table of the manuscript on OfficeHome. We use ResNet-50 backbone network.
Method
Avg
68.32=#0.69
77:0++092
TTat
60:98:00:7
53.94
04
75.75
66.29
TTantAdatter
02.63:20.45
$4.90
76.20-000
77.92
67.91
*TanC'r
61.35:20.75
75:23-1-05
61.9120.33
77:00:800
ASHOTIM
61.84:0.322
53.63-0922
75.56-2
77.57:0.79
67.65
+PI.
59.42:2
42.40 12:31
73.80:2.06
75.77:1.500
62.85
61.35:20.40
$2.87211.6
77.9421.110
08.91:00.59
77.39
78667:0.61
(Qurs)
7883-4093
+TAST DN (Ouns)
63.22:10.85
58.20
77.14:1:.10
7691:100.9
68.88
Table 21: Full results using classifiers trained by ERM for Table of the manuscript on Terralncog- nita. We use Re as a backbone network.
Method
L100
L38
1.43
1.46
ERM
46.84:6 1.96
43.26-6-251
53.32:11.92
40,30:1
45.93
41.20:12.71
29.72:1..59
41.35:1:2.2
36.03-1..55
-TemtAdapter
46.64:1.1 1.17
31.11
49.31:1:105
38.52:22204
43.89
*TenCCIf
49.87-13.880
13.31
53.01
68.40:46.19
43.64
+SIOOT
36.17:13 2.770
29.80 2.92
41.00:1:0:30
33.83-1
35.20
+PL
56.75 5.578
46.12:1:11 1.03
29.04-1:0.14
20006:14.66
25.09
-PLCII
32.28::33.95
43.76:22.96
52.78.1.2.15
37.81:12.19
46.66
45.13:b1.26
44.67-1-2350 2.56
52.52-40078
40.13-6-2.11
*TAST (Durs)
53.01:2:1955
43.27
53.70-22.722
39.66-2:366
47.43
-TAST-NN
33.92299.6
43.87-44770
32.33-2-4440
22

---

Published as a conference paper at ICLR 2023
Table 22: Average accuracy(%) using classifiers trained by CORAL on the domain generalization benchmarks for T namely VLCS, PACS, OfficeHome, and Terralncognita. We use ResNet-18 and ResNet-50 as backbone networks. Bold indicates the best performance for each benchmark. Our proposed method TAST outperforms all the baselines most of the benchmarks.
Mathod
Backbone
OfficeHome
Teralncognita
Avg
CORAL
74.00:11113
62.78::0.06
36.51:22.35
63.57
Tent
71:13:145
84.17:00.11
62.37:20.09
06.71
63.60
-RmmAdaptee
1..66:1:1.66
0..36
63.37.10.11
31,89: 1.25
62.30
-TenCC
72.27:1..229
15.71
62.65:00.08
30.27:16.34
60.23
+SHOT
660111.75
84.67.0.47
03.54:1..23
01:20:20449
+SHOTIM
ResNet
65.75:1:3.70
84.63:20.49
63.53-1:021
13.10:00.42
61.75
+PL
66.58:11.99
76.46:13.23
61.1911:1 1.52
29.32:55.57
58.39
+PLCH
73.20--0.39
76,16=22.44
62.68
14.29:33.90
02.75:0.551
-TAST (Ouns)
74.82:1:2443
83.16:20.81
64.00:00.25
39.21:2::75
65.30
+ISSEBN (Ouss)
77.01-10.36
87.21 0.57
62.98 02.2
31.45 1..5::1111
66.16
CORAL.
1.0
83.5220.67
16.99:0.30
42.79 1.27
67.40
Tent
74.43:00.98
86,50
06.30
42.15=22811
67.35
04:44
76.45
64.03:0.56
2.744
+SHOT
64.11::0.79
85 19 1.05
67.75
03.96:20.59
62.72
ASHOITM
ResNer-50
63.63:20.00
85 0.990.9
07.7220..29
14. 14.17=0:90
62.65
72.74
75.96.6.46
60.7422.91
36.69=14477
*PLCH
75.68-4
03.56-0800
44.93=3.76
+TRA
77.31-20.77
84 84.54
68.06 94
41 41.50:20.19 0.19
68.
-TST (Chus))
77.23-2 1.25
85 04.44
68.3920.554
44.22 1.33
68.72
-TSTTBN (0m)
79.13:10.43
69.4110.36
Table 23: Full results using classifiers trained by CORAL for Table as backbone network.
on VLCCS. We use ResNet-
Method
Avg
CORAL
98.31=3.755
61.11
70.62 +:0.5
70.95-00.33
74,00
TTait
06.78=1.20
59.24+0.85
13:09
71.13
-TastAdagter
79.89-5.76
54.2 34.29
62.72:1066
68.76
63.66
*TanCH
94.96-22.77
5..42
71.01 100:::110
64.11-5442
72.27
+SHOT
88.21211.8
50.29-22911
58.00
67.55:20511
*SHUTIM
87.74=11. 11.38
29.89422774
67.61
+PL
95.78-11.34 1.54
54.09444.51
55.00 2:209
61.42:19.04
66.38
+PLCTT
06.04=2.59
37.67
71.00:133
71.09:0.44
73.70
97.10=3.422
63.61
67:9050.778
73.34:::266
+TAST (Ours)
95.36-12.76
62.95-4-3.23
69:06-t:1.05
71.894 1.95
7482
(Oun)
98.90:20.58
61.01:1:.411
69.74:1:2.57
78:10:1:.115
Table 24: Full results using classifiers trained by CORAL for Table 22 as backbone network.
on PACS. We use ResNet-18
Mathol
C
Avg
CORAL
78.74 1.79
74.57=1.79
92.48::0.00
1.51
$1.00
*Tant
82.11=00:55
81.22 1.02
95.15-2-.04
78.20--1.3 1.33
84.17
*TmnAAdapter
0..06:20.66
77.14 1.35
94.05-10.31
77.7440.00
$2.28
*TanCH
77.14-0:99
63.00-6.22
+SHOT
1..92:1..35
1.18
95.26--0.00
19.59411005
84.67
ASHOTIM
83.92 1.24
81.062 1.17
95.3022057
70.23-4:110
84.63
83.42 1.79
67.36
94.2042:.99
62.22411265
79.72=11.00
62.95-8.19
76.16
T33
80.68:2:11 1.15
77.5220.54
93.25
70.59::0070
$2.75
+TAST (Qurs)
30.80 50.88:11.333 1.33
77.86::0.92
94.28.10056
19.60:1..60
83.16
-TAST-BN (Ouns
36 06666.6666
83.58 1.33
96. 5.:5000622
81.69:11.16
87.21
23

---

Published as a conference paper at ICLR 2023
Table 25: Full results using classifiers trained by CORAL for Table 22 ResNet-16 18 as a backbone network.
OfficeHome. We use
Method
P
CORAL
55. 0.27
50.09=00009
72.09=00.32
73.16
62.78
55.33=0.33
5..79:00.51
71.08-0.522
72.29:00.11
62.37
TentAdapter
06.5920.34
0:05
72.1910.10
73.1740.00
TTnnCCT
55.56=0.400
49 49.9620.116
72.0000444
73:00:0444
62.65
+SHOT
55.77=00299
52.6720444
72.22:1:..4
73.50:10.28
63.54
+SHOTIM
55.72-0.28
02.6422055
72.03:1031
73.52:40.99
54:17=1.85
4.774
71.53+0.40
72.34+0099
-PECIT
53.77
50.04::044
71.00:10.377
73.00:1::42
62.68
-T3A
55.83=00336
51,6820.51
75.704:043
73.66:0.55
63,72
+TAST (Ours)
56.22:10.57
51.73:00448
74.05:10.73
74.00:10.52
64.00
HAST-EN
54.11=0.40
52477:061
72.04:065
72.33:b0.49
62.98
Table 26: Full results using classifiers trained by CORAL for Table 22 on TerraIncognita. We use ResNet-18 as a backbone network.
Method
L100
L
143
1,46
Avg
CORAL.
28.41=:2.79
25 25.96-66.55
45 53
36 06082117 75
36.51
37.51=3.54
14.76:1000
75.9910.73
18.7911.78
TeanAdapter
33.9-232
20.59:33.11
37.89
TTaiCHI
12.99
18.74
43 46
31 89 89
30.27
-SHOT
0..95:0.88
25.85-1440
58 63.330009
52.67:083
33.20
SSHOUM
25.81=0.77
2..6
58.16-0775
32.70:0.445
37.37:23349
24
51.9447.577
23.842683
29.32
-PECIT
45.07=27.63
20.52-19.27
14.4-1332
27.18:2:000
34.29
-T3A
37,14:22.11
34,993477
45.00:3:911
36.911 86
38.39
+TAST (Ous)
46.01:1:.188
32.11
43.314-3.17
35.42-4:..53
29.21
-TAST (Ours)
43.04:13300
32.25-15000
42.53:12:54
31.99100811
37.45
Table 27: Full results using classifiers trained by CORAL for Table 22 as backbone network.
VLCSS. We use ResNet-50
Method
5
CORAL
96.82:1.066
62.51 0.81
71.46-t 1.71
76.39
96.53.2.00
09.5520.11
67.96-666
73
-euttAdatter
84.24-2.99
e'lenChf
96.99-1.16
61.48
12.3241177
75.00:1:.07
76.45
ASHOTT
85.07=1665
26.2222622
16.77-1
66:34:1011
64.11
+SHOTIM
03.57=0.89
2.95
56.55-13111
68.339998
63.63
98.12-0.64
67.66+1116
4PE
96.85_1.30
58.71 2.776
12.20:1166
74.0413999
75.68
=T3A
98.24=0.70
64.69:1.6
75.06:2:044
733343381
77.33
-TAST (Ours)
05:10:20
64.17:2.337
72.30:2775
73.2743.27
77.23
+TAST (Ours)
99.14::0:45
64.95
13339
79.01:2211
79.13
Table 28: Full results using classifiers trained by CORAL for Table as backbone network.
22 on PACS. We ResNet-50
Melhol
CORAL
84.40::: 1.36
79.88=22.00
95.58-20.09
74.24:30000
83.52
85.15-1.8
78.4741..2
+TanAAdapter
*TanCHI
84.68:21 888
80.32-2.809 2.89
95.9
67.59:28.18
82.14
*SHOT
35.75 1.92
82.38 2.52
9688--952
75.37-4-2.6
85.09
ASHOTIM
65.72
965440044
3.66
37.95-3.78
44.7444.0.19
75.96
*PLCIE
85.45:11.55
79.81 3.15
96.21 06.9199999
72.78:55.06
83.56
4T3A
84.11211996
81, 2.99
96.8820553
75.47:22.36
84.54
-TAST (Ours)
15.74:21.77
81.05:22.79
96.88:10.11
76.48:1:.33
$5.04
-TAST-BN (Ouns)
00.95:2: 1.20
86.762:1.33
98 13117:0
85.72:1:003
90.41
24

---

Published as a conference paper at ICLR 2023
Table 29: Full results using classifiers trained by CORAL for Table 22 on OfficeHome. ResNet-50 backbone network.
We use
Avg
+Tmmt
61.11 1:105
0..16:00.56
73.61 1:0.66
1:228
66.30
-TemiAdapter
61.91-11117
55.75
76.28 0:447
76.82:1 1.09
67.68
*TauCH
54.31
49.96
75.30+00.88
76.56-1 1.64
+SHOTIM
61.75:11.52
06.93
75.94100.81
76.24:09555
67.72
+PL
54.78=33.66
40.71 10.71=10.95 10:95
74.19-
73.28 1.811
00.74
+PLCIE'
39.80:10.855
13.40:1:.46
75.60:10.117
76.15:11.33
66.24
05.57=0.85
62.02:21.30
08.8820086
78062:00
77.60:1::44
-TAST-8N (Chus)
63:45:2
19.01
76.68-00.44
77.00:
Table 30: Full results using classifiers trained by CORAL for Table 22 on TerraIncognita. We use ResNet-50 as a backbone network.
Method
L100
L388
1.43
CORAL
45.52:20.53
29,33:22.81
48 48.98-1:2225
37.35:22.1
42.79
46.72:22.57
34.14=3.01
48.05-1:1044
10:70:1:114
+TentAdapter
48.27:12.51
38.07=1.008
44.89:12.73
34.92-1..07
41.54
*TeuCIr
42.57:15.73
38.35=3.337
44. 10:10:22.18
33.97:1:55 58
39.74
ASHOT
38.95 1.35
76. 16.99=1..76 1.76
40.03:2:2.51
28 28.98:4:777
33.96
19.52:122.40
3.94
+PL.
$1.29:1:2.49
47.8822.50
10.78
17.99
+PLCIE
53.69:15.34
43.78:16.66
47.67:1:6.30
34.57:15.188
44,93
+T3A
15.57:21.04
1.53
49.81
38.31:1:.60
43.50
+EAST (Ours)
$3.1120.58
29 29,99-:2.38
48.07-4-3.30
35.02-43.12
44.22
-TANT-BN
54.80:233222
41.56226.79
47.00:2:5588
30.38:24660 60
Table 31: Average accuracy(%) using classifiers trained by MMD for Table 5 on the domain gener- alization benchmarks, namely VLCS, PACS, OfficeHome, and TerraIncognita. We use ResNet-18 backbone network.
OfficeHomee
Terralozognita
Avg
74:00.00 50
31.0::0.09
62.20-20.48
35.73:1::70
63.47
74.99:1066
44.77:0.19
62.15-0.53
1::0772
TemiAdugeer
65.91:1:191
82 02.44.20.20 0.26
62.80-20.46
37.66:10.34
62.21
TeniClr
7:091:0000
57. 7777:::533
0.
23.94226.56 56
55.93
+SHOT
65.112203
63.2520.53
62.34:2068
61.64
+SHOTTM
65.00:1:2220
03.53:10.22
63.20:20.56
32.25:1:055
61.50
+PI.
66. 66.17:1::73
74 74-47:2:4
60 60.40:21.00
22.
22.30:18:42
55.85
4PLCH
74 74.54-20550
69 09.95:20.95
62, 0.44
$1.04:2.5677
59.72
77:25.10:455
63.34:20.55
+IAST (Ours)
76.21-40079
03.29:50.26
63.49-00.99
38.12442247
65.28
+TAST-BN (Qurs)
76.06::0.89
86.35:20.66
63.22220226
30.46::
66.27
Table 32: Full results using classifiers trained by MMD for Table 31 on VLCS. We use ResNet-18 as backbone network.
Method
V
Avg
MMD
95.51-12..9
63.32:10.68
70.09::0000
71.68:1833
74,90
TTent
91.20:20.55
61.53-0.60
69:324--.58
70.33
74,59
~TemAAdspeer
30.71:1:5.11
51.96:2331
61.88:1000
66.2511.55
65.96
=TemCTI
96.1122.166
61.8711002
15.159991
71.90:1822
+SHOT
38.10:38.59
18.99 2.74
57.14:1:1.13
67.43:10.47
65.41
+SIOTIMM
36.93-18.90
28.71
1.19
67.37.10.18
65.00
+PL
05.63:2 1.07
21.75:2.110
19.89:1:1004
57.40:16855
66.17
7..77.0.79
+T3A
99.15:10.52
61.1910.75
71.16:1:0.4
74.62:1:.66
77.28
(Uues)
99.29-10.46
63.14-0.66
61922-12233
7231-11.30
-TAST-RN (Chns)
99. 14:20.48
61.5422.74
65.03-199 1.95
78.5320.35
76.06
25

---

Published as a conference paper at ICLR 2023
Table 33: Full results using classifiers trained by MMD for Table 31 on PACS, We use ResNet-18 backbone network.
AVg
+Tent
82.81:10.97
81.22 0.885
05:49:1::40
78.32-10.005
84.47
-TemtAdapter
90,51:10.84
77.09 07.09:0.44
94.35:10.33
77.59:10.44
82.44
+TeaCCr
96.53-416.38
47.89
4-0.6
33.26-18.17
$4.50:10.13
+SHOTIM
$4.19:10.20
01.94
95.38-1:0:22
00.09:10.889
85.53
+PL
$4.59:10.48
13.96=10..2
95. 95.10-2-0118
64.04+ 11.28
74.42
+PLCIE
50.01:1:0.60
63.87
93:60:1:0.57
42.28:1..47
69,95
81.92:00.41
1.13
$1.7620.311
77.1320.51
05.3740.70
78.91
+TANT-BN (Curs)
86.23:1.188
82. 12.16:1199
97.4620010
79.55
Table 34: Full results using classifiers trained by MMD for Table 31 on OfficeHome. We use ResNet- 18 as a backbone network.
Method
MMD
54.39=0.88
49.1 49.71 1.02
71:05:00.55
72.741:041
62.20
55.73=0.47
70.75:0774
45.41:04.5
62.15
TentAdapier
05.34:20.56
51.09::100
72.11:10446
72.68:10227
62.80
TTniCI
54.39=0.72
49.28 10.28:1111 1.18
71.05:20666
72.77:1027
62.
-SHOTT
54.87=0.77
02.58=0095
72.59:0..1
731822015
63.25
+SIOTIIM
72.54:10.11
73.15100117
51.2621.114
44.63=1444
71,76120.88
72.33:2008
60.49
+PLCIE
54,46:00667
19.67:00.97
71.97:1:.65
72.57:1-.299
62.16
-T3A
54.43:11.26
51.25::0.83
73.86:10.57
7388410.47
63.34
+TAST (Ours)
15.17=11004
50.80+=1.14
74.20-H-057
73.78+0.511
63.49
(Ouus)
54.40=0.600
79.30::335
72.96:00:2
63.22
Table 35: Full results using classifiers trained by MMD for Table 31 Terralncognita. We use ResNet-18 as a backbone network.
Method
L100
L38
L43
146
Ang
MMD
34.57:43.77
26.87-26.18
26.8
18.72:11.17
35.73
*ToM
3.7042::.72
25
44 47
37. 19:2 166
36.11
-TeanAdapter
40.04:2:.45
29.
22
34.4-21.06
+TeuClf
22.97:19.25
14
30. 10.39:1::27
28.94
+SHOT
33.7921..34
2.76
37.81 20
32.07-1..71
32.34
4SHOTIM
33.48:1.144
25.77=1955 1:95
57 57.87-0611
31.8742.44
32.25
18.27:1::75
3696:28.00
17.63=18.15
36.8722.441
31.94
+T3A
35.16:22.74
24.41
43.33-2-222
36. 16.19:11.79
37.40
+TAST (Ours)
44.70:41.822
41.97744944
42 22.30-44483
33.30-5-4.40
38.12
+TAST BN (Ours)
43.81:1:3.05
31.35 2,57
44.4
32,28 1.73
39.66
Table 36: Average accuracy( () using classifiers trained by Mixup for Table 5 on the domain gener- alization benchmarks, namely VL.CS, PACS, OfficeHome, and TerraIncognita. We ResNet- as backbone network.
Method
VLCS
PACS
Officelomee
Teraannoggnia
Avg
Mixap
74.97+00.36
08.29-0088
61.33-0.88
41.04:::100 1.1
64.03
4Temt
72.71:10411
61.22-0.45
30.52:20.55 36
ATemAdageer
01.44:0027
+TenCCI
74.33-10.22
08.55:2..96
01.45
37.21
60.49
4SHOT
68.92199
84443:0033 3.
628122042
56.72-00000
63.06
4SHOTIM
08.31:1..5
04.3220.36
02.40:20.43
36.3--66111
62.92
59.00:12.1 19
68.02:12.47
60.66:20.63
30.30-1-683
4PLCH
74.19:1:077
70.94:1:.00
61.6720086
40.53-2-4.99
61.86
4T3A
7844360.76
01.91:20.54
63 03.49:00.66
39.99-2-000
65.93
+TASE (Ouns
77.19:6-.30
02.35-40.36
63.83-00744
41.44:4-100 1.67
66.33
+TAST BN (Ours)
76.89:10.86
87.14:10.56
62.09:00.86
1.970:1.1.0
67.21
26

---

Published as a conference paper at ICLR 2023
Table 37: Full results using classifiers trained by Mixup for Table 36 on VLCS, We ResNet-18 18 as backbone network.
Method
Mixup
94.73=1.35
02.52=0.79
69. 09.70-200.00
72.93+1.2 1.27
74.97
Tent
95.27=0047
59.84=1.01
6861-1104
67.19::166 1.62
TentAddpter
76.52=1.366
60001161
TTnnCCr
94.7=1.31
61.7220.75
68.44:1044
22.20:2.222
74.33
+SHOT
96,47-3.75
51.01 1:22
1..3311555
68.47:0..77
68.69
+SHOTIM
05.63=4.31
50.49
58.7111711
68.41:0035
0.38
45.877730
-PECII
94.82=1.299
09.16.2.65
69.85:10:94
72.92:1:.54
74,19
-T3A
09.11=0.76
64.7521.02
72.6012.16
77.16:00.79
78.43
+TAST (Ours)
98.97:20.87
03.19:00.77
71.0111800
75.56:1:008
77.19
HAST-EN
99.27:00322
60.76::4475
69633--5588
179
76.89
Table 38: Full results using classifiers trained by Mixup for Table 36 on PACS. We use ResNet-18 as backbone network.
Method
C
Mixup
50.28::2.31
70.69 1.19
34.36:20.85
67.0410.73
78.29
-Tant
0.882
95.58400118
78.041100
TTattAdatter
82.40=
75.98=0.79
TTaiCHI
70 1.77
6..24:4.007
1..17:1.001
13.75-2
68.95
ASHOT
0..50:=0.56
80.15-0.88
06.70
76.18:4455
84.43
*SHOTIM
05.17=0.56
80.65-0.92
96222.7777
70055-5000
62.87=2.544
66.48=8.08
05.00:::444
.PLCIE
79.32-22.88
70.61=1.26
94.26:30.99
19.58-4-1.47
70.94
+T3A
83.06:2 1.31
75.9220..58
95.8:10666
72.80:12.09
81.91
+TAST (Qurs)
$3.93:1-0.99
76.75
96.34--0.63
74.29---1.09
$2.85
+TAST-BN (Oun)
36.18:20.49
82.69:1
97.27:10.48
$2.43:1..80
87.14
Table 39: Full results using classifiers trained by Mixup for Table 36 OfficeHome. We ResNet-16 18 as a backbone network.
Method
P
Avg
Mixup
$3.92-11.21
49.17
71.66-0:552
72.56+0082
61.83
TTnt
53.22-0.92
50.75
71.00.1004
72.0::022
-TsttAdatter
54.78=1.29
e'eenChf
$3.87=1.20
48.22
71,63
7200-1084
61.45
ASHOTT
53.95=111
52.30 1.27
72.20-1036
72.79:1043
62.81
+SHOTIM
53.94=1.2 1.22
12.28-1-422
728330041
52.92-0.96
705740822
4PE
$1.8521.10
49.1 49.12
71.441477
72:06::::04
61.67
=T3A
54.3=1149
50.97:15111
74.1490063
74.00:0311
63.49
-TAST (Ours)
54.97.20.96
51.3121.844
74.88:00666
74.16::0:22
63,83
+TAST (Ours)
53.09:21.44
51,06
72/44::644
71.80:10.44
62.09
Table 40: Full results using classifiers trained by Mixup for Table 36 Terralncognita. We ResNet-18 18 as a backbone network.
Mathod
1.100
1.38
1.43
1.46
Maxup
$2.26c: 1.44
35.9024.466
41 21.23:42.04
34.77.2.11
41.04
*Tent
41.35:2:..4
33.05
44.17:4:55
+TantAdapter
*TzuCH
40.62:2..17
17.29
279
24 0.777
37.21
*SHOT
42.78:23348
31.04 0.04--2..00 2.88
39.58 1.29
31.86-t33447
36.32
ASHOTIM
42.14:2..35
11.80-1..51
52.56=0.16
34.00211665
*PLCIE
$3.30=51..9
38.52418876
40:08-1-.115
33.04-22.57
40.63
4T3A
43.05:222.28
38.5322.37
43.32:23333
34.6521.22
39,89
-TAST (Ours)
57.57-255118
36.43::3722
38:341:2338
33.40:1:.00
41.46
-TAST-BN (Ouns)
36.81:2:.34
42.44:2..33
41.01:22.50
30.5421.552
42.70
27

---

Published as a conference paper at ICLR 2023
Table 41: Full results using classifiers trained by ERM for Table 2 of manuscript on VLCS, We use
ResNet- 18 backbone network.
Arg
94 7.111333
63.79:11.00
67. 67.90=1000
73.154:1.37
+13A
97.52.11..9
65.32-1:224
70.70-3.48
15.51
77.26
+TAST-N (Ours)
95 95.31
65.62:11.79
68.9-1.22
74,9611 1.66
76.20
+IAST (Ours)
98.62-4-106
62.61+1.90
66.84-3001
12.73+11.1 1.17
75.20
(Ours)
+TAST (Oun)
99:22:1:0:55
66.21:1:.36
68.62-2..9
75.66:1..03
77.43
+TAST (Ouns)
20
99.17-4-0.00
05.87:+1.90
08.13=1.7 1.76
75.92+1.75
71.27
Table 42: Full results using classifiers trained by ERM for Table 2 of manuscript on PACS. We use ResNet- 18 backbone network.
No
Avg
77.78.1811
75.09:11.22
+13A
78.11+0977
77.14+1.20
95.92-0.36
21.44 1.03
80.33
+IAST-N N (Ours)
80.18:1:0.88
77.34
96.57 0.27
72.38
81.62
+TAST (Ouns)
80.21:10.00
77.06:11.44
96.13-0.53
71.51
81.23
+TAST (Curs)
+TAST (Ours]
10
79.95:1-1:42
78.12:10.:8
96.51
71,65 2.33
81.56
+TAST (Chur)
20
80.56-00.33
78.26:00.99
06.42=0.20
72.52:40.77
81.94
Table 43: Full results using classifiers trained by ERM for Table 2 of manuscript on OfficeHome. We ResNet-18 backbone network.
Methol
N₂
Avg
55.19:4-.49
72.22
+T3A
55 14:00:774
49.56:11.14
74. 0.555
14.07:1..118
65.21
+1AS I- (Uurs)
55 55-25-00977
50.45:1.02
74.24-0.55
1..25
63.54
+IAST (Ours)
53.53-10.00
49.46:11.39
12.84-0.69
12.53-111330 1.30
62.09
+TAST (Chur]
15.344:.04
74.23-0.43
63.51
+1AST (Ourst
10
38.76=5-068
49.52:1..5
74.17-0.49
24.11
63.39
+TAST (Ouns)
20
56.15 36.15
30.04:11.31
74.33
74.28 1.23
63.70
Table 44: Full results using classifiers trained by ERM for Table S of manuscript on Terralncognita. We use ResNet-18 backbone network.
Method
N=
L100
L38
L43
L46
Avg
37.18:4:2.66
53.18-1.27
36.02:11.37
+13A
36.22:4-1.99
40.08:1 1.99
30.72 1.02
35.791:.22
40.20
4TAST-N (Durs)
99.75:2:176
39.200:2.65
52,33-2.63
36.2421.28
41.33
(Ouur)
43.23-1:0.77
42.40:1:1.08
51.22-3.69
1.0
42.59
+TAST (Ours
43.95:12..33
1.04
35.7:11.27
+TAST (Chrs)
10
43.90---2.922
38.48111.56
53.27-2.73
34.67:1..4
42,60
+TAST (Chrs)
20
43.67-t:2.83
39.24:13.79
52.64=3.022
1.00
42.64
Table 45: Average error rate (%) on CIFAR-I0C for the highest severe corruptions. Bold indicates the best performance for each image corruption.
Method
gausss
hrit
defoc
elast
frost
glass
jpeg
shot
adaptation
48.73
7.01
13.27
11.84
23.38
29.41
28.24
50.78
57.00
19.46
23.38
47.88
44.00
21.93
10.84
*SHOT
17.09
8.64
8.57
9.83
19.53
19.72
13.93
25.60
27.15
13.98
14.01
11.68
16.02
15.39
8.22
15.91
+PL
33.56
7.54
11.53
10.60
20.21
23.86
21.78
38.36
43.64
16.88
18.72
29.83
30.43
18.75
9.43
+T3A
41.87
7.30
13.61
11.99
22.05
28.52
27.13
44.10
54.26
18.71
22.54
37.55
37.84
21.97
10.72
42.02
+TAST-BN (Ours)
14.91
7.68
7.81
8.62
16.81
15.10
12.25
21.82
22.54
12.38
11.67
10.34
13.77
12.99
7.57
16.25
7.27
7.46
9.12
18.17
17.72
12.36
25.74
26.43
13.13
12.85
11.38
15.02
14.40
7.39
28

---

Published as a conference paper at ICLR 2023
Table 46: Average error rate (%) on CIFAR-100C for the highest severe corruptions. Bold indicates the best performance for each image corruption.
Method
gauss
frost
glass
impul
ipeg:
pisel
adaptation
80.77
28.86
50.93
39.62
59.54
68.11
60.19
54.79
82.25
87.75
49.96
54.22
72.27
77.84
54.58
+SHOT
45.95
30.14
31.93
32.81
46.19
49.49
40.65
51.79
$7.02
37.99
39.22
37.57
44.33
44.08
30.97
Tent
43.02
29.65
30.52
31.48
43.88
44.03
39.21
50.91
53.10
36.22
36.31
34.10
41.58
41.85
29.73
+PL.
30.14
42.68
42.77
+T1A
76.95
29.54
48.02
39.64
55.68
65.90
58.45
78.23
86.39
48.82
53.46
66.31
74.14
55.01
37.68
+TAST (Ours)
80.13
29.40
50.85
40.43
58.13
69.24
81.94
88.
50.44
57.26
70.58
77.47
56.98
38.45
(Ours)
33.38
+ITT++
47.10
29.99
31.10
32.61
47.73
$1.74
41.37
$7.36
60.40
38.93
39.01
37.17
45.34
44.53
31.31
29


Reviews:
[{"review_id": "wvsTvvBeNT1", "reviewer": "Reviewer_tB9n", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new approach for fully test-time adaptation. The approach aims to prevent model collapse due to confirmation bias by using nearest neighbour information for pseudo-labelling. It then adapts an ensemble of adaptation modules that are used instead of the original linear classifier layer, on top of the feature representation. The approach is evaluated on four domain generalization benchmarks and on the common corruptions benchmark on CIFAR-10/100.", "summary_of_the_review": "Overall, in the current state I find clarity of the exposition lacking and would tend towards rejection. However, I would be willing to reconsider my evaluation if the issues mentioned above would be addressed.\n\n### Update after discussion period ###\nThe authors have devoted considerable work in clarifying open questions and revising the manuscript. Overall, I think the proposed TAST method can be useful in certain settings (image classification problems with few classes, high level semantic shifts) but in others, like (a) image corruptions (where only TAST-BN performs competitive) or (b) with many classes, it remains preferable to adapt the feature extractor/backbone itself. I attribute (a) to lower-level perceptual shifts being handled better by earlier features and (b) to the property that features unlike prototypes are shared over classes and thus adapting them is more data-efficient. I think the remaining \"niche\" for TAST is still useful and relevant but I would like the authors to be more transparent about the limitations and avoid broad statements like \"TAST outperforms the state-of-the-art TTA methods\" in the abstract. Assuming the discussion of strengths and weaknesses becomes more balanced in the final version, I would lean towards acceptance and I increase my score accordingly.", "strength_and_weaknesses": "Strength:\n * The paper is overall well written and presented\n * The proposed TAST approach is a novel contribution (extending T3A)\n * Figure 1 provides a useful illustration of the proposed procedure\n * The empirical evaluation on domain generalization benchmarks is extensive and strong performance of the proposed TAST is demonstrated.\n * Reasonable ablation studies on TAST are conducted.\n\nWeaknesses:\n * Clarity is insufficient, mostly because there are several inconsistencies in notation and undefined symbols (see below)\n * The ensembling of adaptation modules is not well motivated and also the trade-off such an ensemble poses in terms of increasing inference time is not discussed or quantified. Also, if one employs an ensemble, wouldn't it be natural/preferable to also employ different support sets in conjunction with different adaptation models in order to increase diversity within the ensemble?\n * The method TAST has several additional hyperparameters (N_s, T, M, N_e) and it is not entirely clear how these have been chosen in certain experiments. In particular, they should not be chosen in any way based on the target domains. In general, TTA methods should have few if any free hyperparameters, so it would be preferable to clearly state default values for hyperparameters and use these everywhere except for ablation studies.\n * Experiments on TTA on common corruptions are only conducted on CIFAR10/100-C and not ImageNet-C. Is this because 1000 classes are a challenging for TAST?\n * Runtime of different TTA methods is not reported but a relevant quantity when considering TTA in practical applications. Specifically, would TAST-BN, T3a be slower or faster than TAST? \n * The weak performance of TAST on CIFAR10/100-C is not well explained. A summary sentence like \"T3A and TAST show slight performance changes compared to existing TTA methods that fine-tune the feature extractor.\" does not reflect well that TAST/T3A are more than 20 percent points worse than TENT on CIFAR100-C. I think it is fair to say that CIFAR10/100-C require adapting the entire feature extractor and thus, only TAST-BN is a competitive method here, while TAST is not performing systematically better than \"No Adaptation\".\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pUa1rcoO8JA", "reviewer": "Reviewer_rMac", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a test time adaptation (TTA) method named TAST where a trainable projector is added at test time on top of the feature extractor and optimised via information from the top nearest neighbors.", "summary_of_the_review": "Although an interesting method with some technical novelty, intuitions on the design choices and motivation are missing, as well as discussions/related work of the use of the same techniques outside TTA. I encourage the authors to respond to the concerns above and I am willing to increase my score accordingly with clear and satisfying answers.", "strength_and_weaknesses": "### Strengths\n\nS1: The method uses \"pseudo-label distributions for test data using the nearest neighbor information\" an interesting idea and novel in a test-time adaptation setting\n\nS2: The method improves results on different benchmarks\n\n### Weaknesses\n\nW1: **Transductive vs non-transducive test setups mixed**. A support set is needed and kept updated at test time  to define (pseudo)-prototypes for the proposed method, similar to T3A but unlike other methods like TENT . This seems like a strong enough assumption since now one could call the test setup \"transductive\" and authors should explicitly mention this eg in Table 1, and separate the methods accordingly.\n\nW2: **Differences with related works are conceptually small**. The proposed approach adds adaptors a projector for TTA, something generally common in many other settings (see N1) and instead of constructing prototypes as in T3A, it uses a neiarest neighbor classifiers. TAST-BN is very similar to other TTA methods that only updated BN parameters. It is unclear what the intuitions behind this specific design is and what insights it offers.\n\n\nW4: **discussion on efficiency missing**.How much is the added compute and parameters vs other methods? \n\n### Notes and questions\n\nQ1: **architecture of adaptors**. What is the architecture of h? this should be more prominently featured\n\nN1: **Discussion on the use of projectors beside TTA** is missing. From SSL to SL projectors are being used during training o help transfer performance. SSL methods like AsimSiam, SImSiam, SimCLR and many others, as well as supervised methods like SL-MLP (Revisiting the Transferability of Supervised Pretraining: an MLP Perspective, CVPR 2022) or t-ReX (Improving the Generalization of Supervised Models, arxiv 2022).\n\nN2: **Missing refs utilizing nearest neighbor information during training**. Some missing references that learn using nearest neighbor information in similar ways and should be discussed/cited - see below. Moreover, \"self-training with nearest neighbor information\" is how most manifold learning methods operate -  some discussion on that should be added. \n\n- Iscen, Ahmet, et al. \"Learning with Neighbor Consistency for Noisy Labels.\" CVPR 2022.\n\n- Almazán, Jon, et al. \"Granularity-aware Adaptation for Image Retrieval over Multiple Tasks.\" ECCV 2022 \n\nNote that although for a different setting, Almazán, Jon, et al. also use multiple adaptors that are trained using pseudolabels, while also fused using nearest neighbor-based pseudo labels or by averaging the multiple adaptors\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C-am6Bb5Cr", "reviewer": "Reviewer_Lha6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author proposed a test-time adaptation via a self-training method (TAST). Introducing an adaptation module to generate pseudo-labels through the nearest neighbor approach. Based on the T3A method, the author incorporated a support set created using previous data and a prediction from a classifier. The support set only contains examples with a reliable pseudo label, which is determined by calculating the cosine similarity between the test sample x and its nearest neighbor in the embedding space. The multiple adaptation module is randomly initialized, trained during the test time, and then ensemble the result generated from different adaptation modules to predict the labels for test data. The method outperforms state-of-art results on domain generalization on corrupted image corruption datasets.", "summary_of_the_review": "This work delivers on its claimed contributions, although the time complexity may limit it application at test time. I anticipate it will be a valuable publication for researchers in test time adaptation.", "strength_and_weaknesses": "Strength:\n+The proposed method's main strength is introducing adaptation module(s) that were randomly initialized and fine-tuned using the nearest neighbor samples from test data. The author also introduced TAST-BN, which fine-tunes the BN layer (if it exists) instead of the adaptation module. \n+ This method outperformed other techniques in image corruption datasets. The empirical results show the benefit of the method, and some ablation studies are provided.\n+The supplementary material provides additional implementation details and experimental results that help support the paper.\n+ Also, since the codes are also provided in supplementary material, there is less concern that the results in this paper would be difficult for a reader to reproduce.  The paper includes information that would make it possible to reproduce the methods and experiments (but not with MIDRC data).\n\n\nWeaknesses:\n+ The proposed method is computationally expensive. Although, the author compares their method with others in terms of computation in an ablation study, and mentions that the proposed method (TAST) was computationally expensive. Still, their method takes 4x more time than the state-of-the-art (T3A). This performance is not ideal, especially when adapting the model on test time.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3QnwqRyVhd", "reviewer": "Reviewer_hpzb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the setting of test-time adaptation, an adaptation method tries to update a model given only online access to test data in order to reduce generalization error without altering the training process. The proposed test-time adaptation via self-training with nearest neighbors method (TAST), differs from existing methods in (1) its choice of parameters for adaptation, (2) its use of nearest neighbors in defining the targets for self-training updates, and (3) it's combination of prototype (average) and exemplar (neighbor) predictions. The parameters for adaptation are those of newly-initialized predictors on top of the trained feature extractor, including multiple predictors acting as an ensemble, in order to provide a variety of predictions to average over for robustness. TAST differs from prior prototypical methods like T3A by making use of neighbors to update the representation, while T3A keeps the representation fixed which limits the range of its improvement. Experiments on standard benchmarks for domain generalization (VLCS, PACS, OfficeHome, TerraIncognita) and image corruption (CIFAR-10/100-C) show marginal improvements of +1 point of accuracy (with the exception of a +6 point gain on PACS). This marginal improvement is above the state-of-the-art methods for test-time adaptation including entropy minimization and prototypical approaches.", "summary_of_the_review": "TAST is a technically correct and slightly novel test-time adaptation method that combines adaptation by exemplar (neighbor), prototype (cluster), and entropy minimization (optimizing). It achieves a slight improvement in accuracy (around 1 percentage point) for multiple domain generalization and image corruption datasets, but only reaches this improvement by requiring more computation time and memory for inference and adaptation. As such it is a borderline paper, but it can inform the community about the possibility of improving test-time adaptation results by combining clustering based on neighbors with entropy minimization of newly-included adaptation layers without updating the parameters from training.", "strength_and_weaknesses": "*strengths*\n\n- TAST only updates parameters of the deepest, final layers so it does not need to backpropagate gradients through the entire model, unlike methods like TENT/MEMO/SHOT that update parameters in all normalization layers/all layers/all but the last layer, respectively. This could improve computational efficiency, if it is not cancelled out by the number of new final layers that TAST adds and the number of updates needed to train them.\n- The domain generalization datasets diversely include images that are real (VLCS, OfficeHome), stylized/synthetic (PACS), and scientific (TerraIncognita) and follow the established choices of prior work on test-time adaptation (T3A). The ablation of including nearest neighbor predictions without updates (\"Effect of nearest neighbor information\") does show a slight (<1%) effect of of neighbors alone, which validates the idea of the proposed method.\n- The image corruption benchmarks are standard, but small. CIFAR-10/100-C are common, but ImageNet-C is preferred, as it is more representative of the scale of current image classifiers. prior work such as Tent and SHOT report on ImageNet-C for this reason.\n- The results are insensitive to batch size across 8-128 and show improvement over ERM at each size. However, the minimum viable batch size is not reported, which would be worth knowing to determine which methods are competitive at small batch sizes and in the limit batch size one.\n\n*weaknesses*\n\n- The effect of TAST is small. generally improves by 1 point or less, with the exception of PACS with +6 points (Table 1), and this gap is only achieved with a higher number of adapters. The ablation of $N_e$ for the number of adapters shows that the improvement with no adapters is only recovered at 5 adapters, which has greater computational cost than none.\n- More memory and computation is needed to maintain a set of neighbors, as done by TAST, than to maintain a set of prototypes, as done by T3A. \n- The results are sensitive to the choice of parameters to update, and which choice is best varies across datasets. In particular neither TAST, with updates to new modules, nor TAST-BN, with updates to normalization layers, is a clear winner. Without a model selection rule to choose between them it is not clear which to apply.\n- The method has hyperparameters to determine the threshold between reliable and unreliable pseudo-labels (eq. 2), like T3A, but there are no ablations to check the sensitivity of results to this reliability rule. If it needs tuning then it would limit the general applicability of the method.\n- There is closely related work on source-free domain adaptation with neighbors, which is cited (Tang et a. 2021 and Yang et al. 2021), but these works do nevertheless reduce the novelty of the proposed TAST.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster