Test-Time Adaptation to Distribution Shifts by Confidence Maximization and Input Transformation
Authors: Chaithanya Kumar Mummadi
Robin Hutmacher
Kilian Rambach
Evgeny Levinkov
Thomas Brox
Jan Hendrik Metzen
Venue: NeurIPS 2021 Submitted (2021)
Keywords: natural corruptions
corruption robustness
distribution shift
test time adaptation
domain shift

Abstract: Deep neural networks often exhibit poor performance on data that is unlikely under the train-time data distribution, for instance data affected by corruptions. Previous works demonstrate that test-time adaptation to data shift, for instance using entropy minimization, effectively improves performance on such shifted distributions. This paper focuses on the fully test-time adaptation setting, where only unlabeled data from the target distribution is required. This allows adapting arbitrary pretrained networks. Specifically, we propose a novel loss that improves test-time adaptation by addressing both premature convergence and instability of entropy minimization. This is achieved by replacing the entropy by a non-saturating surrogate and adding a diversity regularizer based on batch-wise entropy maximization that prevents convergence to trivial collapsed solutions. Moreover, we propose to prepend an input transformation module to the network that can partially undo test-time distribution shifts. Surprisingly, this preprocessing can be learned solely using the fully test-time adaptation loss in an end-to-end fashion without any target domain labels or source domain data. We show that our approach outperforms previous work in improving the robustness of publicly available pretrained image classifiers to common corruptions on such challenging benchmarks as ImageNet-C.

Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation
Anonymous Author(s)
Affiliation
Address
email
Abstract
Deep neural networks often exhibit poor performance on data that is unlikely under the vain-time data distribution, for instance data affected by corruptions. Previous works demonstrate that time adaptation to data shift, for instance using entropy minimization [1 effectively improves performance on such shifted distributions. This paper focuses the fully test-time adaptation setting. where only unlabeled data from the target distribution is required. This allows adapting arbitrary pre- trained networks. Specifically, adaptation by addressing both premature convergence and instability of entropy minimization. This is achieved by replacing the entropy by a non-saturating surro- gate and adding a diversity regularizer based on batch-wise entropy maximization that prevents convergence trivial collapsed solutions. Moreover, we propose to prepend an input transformation module to the network that can partially undo test-time distribution shifts. Surprisingly, this preprocessing can be learned solely using the fully test-time adaptation loss in an end-to-end fashion without any target domain labels or source domain data. We show that our approach outperforms previous work in improving the robustness of publicly available pretrained image classifiers to common corruptions on such challenging benchmarks as ImageNet-C.
2
4
6
propose novel loss that improves test-time
8
10
11
12
13
14
15
16
17
Introduction
18
Deep neural networks achieve impressive performance on test data, which has the same distribution 20 as the training data. Nevertheless, they often exhibit a large performance drop on test (target) data which differs from training (source) data; this effect known as data shift [2] and can be caused for instance by image corruptions. There exist different methods to improve the robustness of the model
19
21
22
23 during training 4 However, generalization to different data shifts is limited since infeasible to include sufficiently many augmentations during training to cover the excessively wide range of potential data shifts (6). Alternatively, in order to generalize to the data shift hand, the model can be adapted during test-time. Unsupervised domain adaptation methods such as 7 use both source and target data to improve the model performance during test-time. In general source data might not be available during inference time, e.g., due to legal constraints (privacy profit). Therefore we focus
24
25
27
29 on the fully test-time adaptation setting [ the model is adapted to the target data given only the arbitrarily pretrained model parameters and the unlabeled target data that share the same label space
31 as source data. We extend the work of Wang et al. [ by introducing a novel loss function, using diversity regularizer, and prepending a parametrized input transformation module to the network. 33 We show that our approach outperform previous works and make pretrained models robust against common corruptions image classification benchmarks as ImageNet-C g and ImageNet-R (9]
34
35 Sun et al. [10] investigate test-time adaptation using self-supervision task. Wang et al. | and Liang et al. [1]] use the entropy minimization loss that uses maximization of prediction confidence as self-supervision signal during test-time adaptation. Wang et al. M has shown that such loss performs
37
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

38
better adaptation than a proxy task [TO]. When using entropy minimization, however, high confidence predictions do not contribute to the loss significantly anymore and thus provide little supervispi. This is a drawback since high-confidence samples provide the most trustworthy self-supervin 40
39
41
We mitigate this by introducing two novel loss functions that ensure that gradients of samples with high confidence predictions do not vanish and learning based on self-supervision from these samples continues. Our losses do not focus on minimizing entropy but on minimizing the negative log likelihood ratio between classes; the two variants differ in using either soft or hard pscudo-labels. In contrast to entropy minimization, the proposed loss functions provide non-saturating gradients, even when there are high confident predictions. We refer to Figure I for an illustration of the losses and the resulting gradients. Using these new loss functions, we are able to improve the network performance under data shifts in fully test-time adaptation.
42
43
44
45
47
49 In general, sulf-suupr by confidence maximization can lead to collapsed trivial solutions, which make the network to predict only a single or a set of classes independent of the input. To overcome this issue a diversity regularizer [1 12 can be used, that acts on a batch of samples. It encourages the network to make different class predictions on different samples. We extend the regularizer by including a moving average, in order to include the history of the previous batches and show that this stabilizes the adaptation of the network to unlabeled test samples. Furthermore we also introduce parametrized input transformation module, which we prepend to the network. The module is trained in fully test-time adaptation manner using the proposed loss function, i. e. without the need of any target domain labels or source data. It aims to partially undo the data shift at hand. This helps to further improve the performance on image classification benchmark with corruptions.
50
51
53
54
55
57
59 Since our method does not change the training process, it allows to use any pretrained models. This 60 is beneficial because any good performing pretrained network can be readily reused, e.g., network trained on some proprictary data not available to the public. We show, that our method significantly improves performance on models that are trained on clean ImageNet data such ResNet50 [3] as well as robust models such as ResNet50 models trained using AeepAuumet++AggMMe 9.
61
63
64 In summary our main contributions are as follows: we propose non saturattn losses based on the negative log likelihood ratio, such that gradients from high confidence predictions still contribute to test-time adaptation. We extend the diversity regularizer that acts on a batch of samples to a moving average version, which includes the history of the previous batch samples. This prevents the network from collapsing to trivial solutions. Furthermore we also introduce an input transformation module, which partially undoes the data shift at hand. We show that the performance of different pretrained models can be significantly improved on challenging benchmarks like ImageNet-C and ImageNet-R.
65
88
67
68
69
Related work
71
72
Common image corruptions are potentially stochastic image transformations motivated by real- world effects that can be used for evaluating model's robustness. One such benchmark, ImageNet-C
73
74 []) contains simulated corruptions such as noise, blur, weather effects, and digital image transforma- tions. Additionally, Hendrycks et al. [9] proposed three data sets containing real-world distribution shifts, including Imagenet-R. R. The ImageNet-C have been further extended to MNIST [14), several
76
77
object detection datasets [15), and image segmentation (16), reflecting the interest of the robustness community. Most proposals for improving robustness involve special training protocols, requiring
78
79 time and additional resources. This includes data augmentation like Gaussian noise [17 [8 9
CutMix 19). AugMix 44. training on stylized images 3 20 or against adversarial noise distribu-
80
81 tions 21). Mintun al. [22) pointed out that many improvements
ImageNet-C are due to data augmentations which are too similar to the test corruptions, that is: overfitting to ImageNet-C occurs.
82
83
Thus, the model might be less robust to corruptions not included in the test set of ImageNet-C.
84
Unsupervised domain adaptation methods train a joint model of the source and target domain by cross-domain losses, with the hope to find more general and robust features. These losses optimize
86 feature alignment 23 24 between domains, adversarial invariance 25 5 26 27), shared proxy tasks 28 or adapting the entropy minimization via an adversarial loss 77. While these approaches
88 are effective, they require explicit access to source and target data at the same time, which may not always be feasible. Our approach works with any pretrained model and only needs target data.
89
90
Test-time adaptation (also termed source free adaptation in some literature) is a setting, when training (source) data is unavailable at test-time. Several works use generative models (29 30 31 32
91
2

---

92 for the source free adaptation and require several thousand epochs to adapt to the target data 30 32 Besides, there is another line of work [TO
35 I that interpret the common corruptions as 94 data shift and aim to improve the model robustness against these corruptions with efficient test-time adaptation strategy to facilitate online adaptation. Such setting refrain the usage of generative models 96 or methods that require larger number of adaptation steps. Our work also falls in this line of research and aims to test-time adapt the model to common corruptions with less computational overhead.
93
33
34
95
97
98
Sun et al. 10] update feature extractor parameters at test-time via a self-supervised proxy task (predicting image rotations). However, Sun et al. [T0] alter the training procedure by including the proxy loss into the optimization objective as well, hence arbitrary pretrained models cannot be used
99
100
101
directly for test-time adaptation. Inspired by the domain adaptation strategies [36 (37,, several works
102
B4 335 replace the estimates of Batch Normalization (BN) activation statistics with the statistics of the corrupted test images. Fully test time adaptation, studied by Wang et al. []] (TENT) uses
B3
103
104
entropy minimization to update the channel-wise affine parameters of BN layers on corrupted data along with the batch statistics estimates. SHOT [T] also uses entropy minimization and a diversity regularizer to avoid collapsed solutions. SHOT modifies the model from the standard setting by adopting weight normalization at the fully connected classifier layer during training to facilitate their pseudo labeling technique. Hence, SHOT is not readily applicable to arbitrary pretrained models.
105
107
109
We show that pure entropy minimization [T results in vanishing gradients for high confidence predictions, thus inhibiting learning. Our work addresses this issue by proposing a novel non- saturating loss, that provides non-vanishing gradients for high confidence predictions. We show that our proposed loss function improves the network performance after test-time adaptation. In particular, performance on corruptions of higher severity improves significantly. Furthermore, we add and extend the diversity regularizer [11 [2] to avoid collapse to trivial, high confidence solutions.
110
111
112
113
114
115
Note that the existing diversity regularizers [| [2] act on a batch of samples, hence the number of classes has to be smaller than the batch size. We mitigate this problem by extending the regularizer
117
to a running average version. Prior work 55] 38 39] transformed inputs by an additional module
to overcome domain shift, obtain robust models, and also to learn to resize. In our work, we also prepend an input transformation module to the model, but in contrast to former works, this module is trained purely at test-time to partially undo the data shift at hand and thus aids the adaptation.
119
120
3 Method
122
We propose novel method for fully test-time adaption. For this, we assume that neural network fo with parameters is available that was trained on data from some distribution D, as well a set of (unlabeled) samples X ~ D' from target distribution D' * D (importantly, no samples from D are required). We frame fully test-time adaption as a two-step process: (i) Generate a novel network Jo based on fo. where & denotes the parameters that are adapted. A simple variant for this is g = f and & C 0 [. However, we propose a more expressive and flexible variant in Section 3.1 (ii) Adapt the parameters á of g on X using an unsupervised loss function L. We propose two novel losses Lgtr and Lntr in Section 3.2 that have non-vanishing gradients for high-confidence self-supervision.
123
125
126
127
128
129
130
3.1 Input Transformation
We propose to define the adaptable model as 9 f d. That is: we preprend trainable network d to f. The motivation for the additional component d is to increase expressivity of such that it can learn to (partially) undo the domain shift D - D'.
132
133
134 Specifically, we choose d(x)= = 7 [rz + (1- (1=r)r((x)] + 8, where T E R. (8,7) E R with nnn being the number of input channels, re being network with identical input and output shape, and denoting elementwise multiplication. Specifically, / and 7 implement a channel-wise affine transformation and 7 implements convex combination of unchanged input and the transformed input
136
138 Ty (z). By choosing T = 1. 7 = 11, and B = O, we ensure d(z)) = x and thus g f at initialization. In principle, re can be chosen arbitrarily. In this work, we choose T' as simple stack of 3 x 3 convolutions, group normalization, and ReLUs (for details, we refer to the appendix). However, exploring other choices would be interesting avenue for future work.
140
142
Importantly, while the motivation for is to learn to partially undo a domain shift D - D', we train d end-to-end in the fully test-time adaptation setting on data X ~ D', without any access to samples
3

---

144
from the source domain D, based on the losses proposed in Section 3.2 The modulation parameters of I are @ = (3. y, T, v,, 8*), where 0' 8. That is, we adapt only a subset of the parameters 0 of the pretrained network f. We largely follow Wang et al. []] in adapting only the affine parameters of normalization layers in f while keeping parameters of convolutional kernels unchanged. Additionally, batch normalization statistics (if any) are adapted to the target distribution.
145
146
147
148
149
Please note that the proposed method applicable to any pretrained network that contains normaliza- tion layers with a channel-wise affine transformation. Even for networks that do not come with such affine transformation layers, one can add affine transformation layers into f that are initialized to identity as part of model augmentation.
150
151
152
153
3.2
Adaptation Objective
154
We propose a loss function L = Laiv eoo for fully test-time network adaptation that consists of two components: (i) a term Laiv that encourages predictions of the network over the adaptation dataset
155
156
X that match a target distribution pr> (y). This can help avoiding test-time adaptation collapsing
157
to too narrow distributions such as always predicting the same or very few classes. If Pp (v) is (close to) uniform, it acts as diversity regularizer. (ii) A term Leatt that encourages high confidence prediction on individual datapoints. We note that test-time entropy minimization (TENT) [1 fits into
158
159
180
this framework by choosing Ldiv and Connf as the entropy.
161
3.2.1
Class Distribution Matching Laiee
162
Assuming knowledge of the class distribution ppr (y) on the target domain D', we propose to add term the loss that encourages the empirical distribution of (soft) predictions of go on X to match this distribution. Specifically, let Pg. (t) be an estimate of the distribution of (soft) predictions of gø- divergence Laiv = D (Pg. (v)l (D)()) as loss term. In a special case of PD (y) being a uniform distribution over the classes, this corresponds to maximizing the entropy H (ps. (v)). Similar assumption has been made in SHOT [11] to circumvent the collapsed solutions.
163
164
185
We use the Kullback-
166
167
168
Since the estimate Pg. (v) depends on o, which is continuously adapted, it needs to be re-estimated on a per-batch level. Since re-esti re-estimating Poo (v) from scratch would be computational expensive, we propose to use a running estimate that tracks the changes of o as follows: let pt-1 (y) be the estimate at n2
169
170
iteration --1 and p3 emp 1 E g((k,, where (((t) are the predictions (confidences) of gs on a mini-
171
batch of n inputs ((k) ~ X. We update the running estimate via Pt (v) = K Pt (1) + + (1-x).2 pp:"p. The loss becomes Laiv = DKL (P) PDr (v)) accordingly. We use K = 0.9 in the experiments.
172
173
174
3.2.2
Confidence Maximization Leons
175 We motivate our choice of Leont step-by-step from the (unavailable) supervised cross cross-entropy loss: for this, let ğ = (x(2) be the predictions (confidences) of model Io and H (ū)""))= Ee ye log de be the cross- between prediction ÿ and some reference y". Moreover, let the last layer of be
178
177
178
a softmax activation layer softmax. That is ÿ = softmax(()) where o are the network's logits. We note that we can rewrite the cross-entropy loss in terms of the logits o and a one-hot reference y" as follows: (ssftmaax(() y²) - + log Eis ed where c" is the index of the I in y" and na is
179
180
the number of classes.
182
In the case of labels being available for the target domain (which we do not assume) in the form of ooe encoded reference for data T2, one could use the supervised cross-entropy loss by setting
183
194
y" = Ye and using Luu y²) = H y²) H (ū. ye). Since fully test-time adaptation assumes no label information being available, the supervised cross-entropy loss is not applicable and other options for y" need to be used.
185
186
187
One option are (hard) pseudo-labels. That is, one defines the reference y" based on the network pre- dictions ý via y" onchot(j), where onchot creates one-hot reference with the corresponding to the class with maximal confidence in ÿ. This results in L,x(0) = HŪ. onchot()) (g)) = log yee, with arg max ÿ. One disadvantage with this loss is that the (hard) pseudo-labels ignore uncertainty in the network predictions during self-supervision. This results in large gradient magnitudes with 2Ln
188
189
190
181
192
being generated in situations where the network is highly unconfident (see respect to the logits dog-

---

0.0
0.4
0..2
Logit Gradient 0L
80.
0.2
Loos
0.0
-0.6
Loss I.
-0.2
-0.8
Likilnood Rabio
Entroy Hard Penablatek
-0.4
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6 0.8 1.0
Confidence
Confidence yn
Figure 1: Illustration of different losses confidence maximization. Losses (left, shifted such that maxima of all losses are 0) and the resulting gradients with respect to the first logit (right) as function of the first classes confidence are shown for the case of a binary classification problem. Both entropy and hard pseulo-labels have vanishing gradients for high confidence predictions. Accordingly, both have maximum gradient amplitude for low-confidence self-supervision, with this effect being stronger for the hard pscudo-labels. Hard Likelihood Ratio has constant gradient amplitude for any confidence and thus takes into account low- and high-confidence self-supervision equally. Soft Likelihood Ratio also shows non- van-isiiiin (albeit mon-maximum) gradients for high- confidence self-supervision and additionally produces small gradient amplitudes from low cow-confidence self-supervision. Since the likelihood ratio-based losses unbounded, the design of the model needs to ensure that logits cannot grow unbounded.
Figure D. This is undesirable since it corresponds the network being affected most by data points where the network's s self-supervision is least reliable.
188
194
An alternative is to use soft pseudo-labels, that is y" ÿ. This takes uncertainty in network predictions into account during self-labelling and results in the entropy minimization loss of TENT D: (ū) = ( D) = H (9) = -e te De. However, also for the entropy the logits' gradient magnitude اسساا goes to 0 when one of the entries in ý goes to 1 (see Figure 1p. For a binary classification task, for instance, the maximal logits' gradient amplitude is obtaiedd for 9 x (0.82, 0.18). This implies that during later stages of test-time adaptation where many predictions
196
198
199
200
201
typically already have very high confidence, i. e. above 0.82, gradients are also dominated by datapoints with relative low confidence in self-supervision.
202
203
While both hard and soft pseudo-labes are clearly motivated, they are not optimal in conjunction with gradient- based optimizer since the supersuunn from low confidence predictions dominates (at least during later stages of training). To address this issue, we propose two losses that are analogous Tpp and Trent, but are not based the cross-entropy H but instead on the negative log likelihood ratios
205
206
207
R(O. Eve log
Ge
- ve (log de - log Σ0)
0.) = -(.v""))"")
Σ ve log Σ
U:
Eito 9s
igc
208
Note that while the entropy H is lower bounded by 0, A can get arbitrary small if ye + 1 and the sum Eitc D: + 0 and thus log Eitc D: -00 This property will induce non-vanishing gradients for high confidence predictions.
209
210
211
The first loss we consider is the hard likelihood ratio loss that is defined similarly to the hard pseudo-labels loss Lpt:
Lii- (ū) R(g, onehot(j)) loog Dc-
e Eijer cor ) -ct + log Σ c
Enje- D1 log(
where c* = arg max ÿ. We note that OLaie = 1, thus also high-confidence self-supervision contributes equally to the maximum logits' gradients. This loss was also independently proposed as negative log likelihood ratio loss by Yao et al. 44 as a replacement to the fully-supervised cross entropy loss for classification task. However, to the best of our knowledge, we are the first to motivate and identify the advantages of this loss for self-supervised learning and test-time adaptation due to its non-saturating gradient property.
213
Zo,-
214
215
217
5

---

219 In addition to Lutr, we also account for uncertainty in network predictions during self seff-labelling in
220 similar way as for the entropy loss Lents and propose the soft likelihood ratio loss:
Latr = R) = Σ De log(
Dc
- Σ Je log(
Σife Di
=
Σ enl
= =Σ uol-00 + log Σel E)
ipfle
221
We note that as Dc- 1, (1) + (ū). Thus the asymptotic behavior of the two likelihood ratio losses for high confidence predictions is the same. However, the soft likelihood ratio loss creates lower amplitude gradients for low confidence self-supervision. We provide illustrations of the discussed losses and the resulting logits' gradients in Figure
222
223
225
We note that both likelihood ratio losses would typically encourage the network simply scale its logits larger and larger, since this would reduce the loss even if the ratios between the logits remain constant. However, when finetuning an existing network and restricting the layers that are adapted such that the logits remain approximately saaee-normalized, these losses can provide a useful and non-vanishing gradient signal for network adaptation. We achieve this appproximate scale normalization by freezing the top layers of the respective networks. In this case, normalization layers such as batch normalization prohibit "logit explosion". However, predicted confidences can presumably become overconfident; calibrating confidences in a self-supervised test-time adaptation setting is an open and important direction for future work.
226
227
228
229
231
232
233
Experimental settings
234
235
Datasets We evaluate our method on image classification datasets for corruption robustness and domain adaptation. We evaluate on the challenging benchmark ImageNet-C [8), which includes a wide variety of 15 different synthetic corruptions with 5 severity levels that attribute to data shift. This benchmark also includes 4 additional corruptions as validation data. For domain adaptation, we choose ImageNet trained models to adapt ImageNet-R proposed by Hendrycks et al. 19]. This dataset contains various naturally occurring artistic renditions of object classes from the original ImageNet. ImageNet-R R comprises 30,000 image renditions for 200 ImageNet classes. Please refer Sec. A.5 for the experiments on other domain adaptation datasets ViSDA-C [41), Office-Home [42]
236
238
240
242
Models Our method operates in fully test-time adaptation setting that allows us to use any arbitrary pretrained model. We use publicly available ImageNet pretrained models ResNet50, DenseNet1 Res5eXt50. MobileNetV2 from torchvision 43
244
We also test robust ResNet50 model trained
246
using PecpAugment+AAugMMe
[9]
247
Baseline for fully test-time adaptation Since TENT from Wang et al. [ outperformed competing methods and fits the fully test-time adaptation setting, we consider as a baseline and compare our results to this approach. Similar to TENT, we also adapt model features by estimating the normalization statistics and optimize only the channel-wise affine parameters on the target distribution.
248
249
250
251
Settings We conduct test-time adaptation on a target distribution for 5 epochs with batch size 64 and use the Adam optimizer with cosine decay scheduler of the learning rate with initial value 0.0006. We set the weight of Lconf in our loss function to ó = 0.025 and k 0.9 in the running estimate Py(y) of Laiv (we investigate the effect of in the Sec. A.3). Similar to SHOT (... we also choose the target distribution pD (y) in Ldiv as uniform distribution over the available classes. We found that the models converge during to 5 epochs and do not improve further.
252
253
254
255
256
257
For TENT, we use SGD with momentum 0.9 at constant learning rate 0.00025 with batch size 64. These values correspond to the ones of Wang et al. [I: alternative settings of optimizer and learning rates for TENT did not improve performance. TENT is originally optimized only for epoch. For a fair comparison to our method, we optimize TENT also for 5 epochs. Similar to Wang al. (1. also control for ordering by data shuffling and sharing the order across the methods.
259
260
261
Note that all the hyperparameter settings are tuned solely on the validation corruptions of ImageNet-C that are disjoint from the test corruptions. As discussed in Section 3.2.2, we freeze all trainable parameters in the top layers of the networks to prohibit "logit explosion" Note that normalization
263
From https://gitubb..o//eenntt
Owner permitted to use it for research/commerciall purposes.

---

Table 1: Test-time adaptation of ResNet50 on ImageNet- at highest severity level 5. Ground truth
labels are used to adapt the model in supervised manner to obtain empirical upper bound performance.
Mothod
Gaess
Shot
Impulse
Defocus
Glass
Moxion
Zoom
Snow
Frust
Fog
Bright
Contrast
Elastic
Pixel
JPBG
No Adaptation
2.44
2.99
1.96
17.92
9.82
14.78
22.50
16.89
23.31
24.43
58.93
5.43
16.95
20.51
31.65
Pseudo Labels
2,44
2.99
1.96
17.92
9.82
14.78
22.50
16.89
23.31
21.43
58.93
5.43
16.95
20.61
31.65
Epoch
TENT
32,70
35.34
<1.82
68.
27.53
55.98
TENT-
33.96
36.66
35.75
33.70
33.33
47.73
53.22
52.16
44.79
60.62
68.91
35.60
58.81
61.82
56.23
HLR (ours)
38.39
41.11
40.28
38.25
38. 18
51.63
55.55
$5.45
48.
62. 19
68.17
49.47
60.34
62.51
57.
SER (ouns)
39,51
42,09
41.58
39.35
39,02
52.67
55,80
55.92
49.64
62.62
68.47
50.27
60.80
63.01
$7,80
Epoch 5
TENT
59.25
56.17
TENT-
33.97
37.95
51.42
54.33
02
74.08
661.6
63:10
57.31
HIR (ours)
41.37
44.04
43.68
41.74
41.09
54.26
56.45
57.03
50.81
03.05
68.29
50.98
61.15
63.08
58.13
SLR (ouns)
41.52
42,90
44.07
41.69
40.78
54.76
$6.59
$7.38
51.01
63.53
68.72
50.65
61.49
63,46
58.32
Grundtruth
55,68
58.10
61.27
55.84
55.08
65.83
67.22
67.56
62.60
72.49
76.97
65.04
70.85
72.51
68.55
Table 2: SSIM and SLR-adapted ResNet50 accuracy without and with input transformation (IT).
Comupttin
Gauss
Shot
Impulse
Defocus
Motion
Zoom
Snow
Frost
Fog
Bright
Contrast
Elastic
Pixel
JPBG
SSIM SSIMaIT
0.123
0.147
0.135
0.623
0.648
0.622
0,676
0.517
0.575
0.619
0.653
0.545
(.625
0.786
0.800
0.173
0.188
0.347
0.605
0.658
0.608
0.670
0.580
0.628
0.626
0.676
0.765
0.616
0.776
0.795
41.70
51.06
SLR-IT
43.09
44.39
64.05
41.98
40.99
55.73
56.75
58.56
$1.68
63.64
68.85
$5.01
61.32
63.39
58.24
265
statisties are still updated in these layers. Please refer Sec. A.2 for more details regarding which layers are frozen in different networks.
266
Furthermore, we prepend trainable input transformation module d (cf. Sec. 3.1 to the network to partially counteract the data-shift. Note that the parameters of this module discussed in See. B... are trainable and subject optimization. This module is initialized to operate as identity function prior to adaptation on a target distribution by choosing T = 1, 7 1, and / = O. We adapt the parameters of this module along with the channel-wist affine transformations and normalization statistics in an end-to-end fashion, solely using our proposed loss function along with the optimization details mentioned above. The architecture of this module is discussed in Sec. A.I.
288
269
270
272
273
274
Since Lav is independent We denote this as TENT+ and also set 0.9 here. Note that TENT optimizes all channel-wise affine parameters in the network (since entropy is saturating and does not cause logit explosion). For fair comparison to our method, we also freeze the top layers of the networks in TENT+. We show that adding Taiv and freezing top layers significantly improves the networks performance over TENT. Note that SHOT [1] is the combination of TENT, batch-level diversity regularizer, and their pseudo labeling strategy. TENT+ can be seen as a variant of SHOT but without their pseudo labeling technique. Please refer to Sec. A.4 for the test-time adaptation of pretrained models with SHOT.
Leontt we also propose to combine Lay with TENT, L Laiv + Lent-
276
278
279
280
281
282
Note that each corruption and each severity ImageNet-C is treated as a different target distribution and in all settings we reset model parameters to their pretrained values before every adaptation. We run our experiments for three times with different random seeds (2020, 2021, 2022) in PyTorch and report the average accuracies.
284
5
Results
286
287
Evaluation on ImageNet-C We adapt different models on the ImageNet-C benchmark using TENT.
288
TENT+, and both hard likelihood ratio (HLR) and soft likelihood ratio (SLR) losses. Figure 2 (top row) depicts the mean corruption accuracy (mCA% %) of each model computed across all the corruptions and severity levels. can be observed that TENT+ improves over TENT, showcasing the importance of diversity regularizer Lav- Importantly, our methods HLR and SLR outperform TENT
289
290
291
292
and TENT+ across DenseNet121. MobileNerV2, ResNet50, Xt50 and perform comparable Augmix model. This shows that the mCA% of robust DeepAugment+ Augmix model can be further increased from 58% (before adaptation) to 68.6% using test-time adaptation techniques. Here, the average of mCA obtained from three different
with TENT+ on robust ResNeet50DeepAuugee
294
7

---

Decuefle1121
Mobi 1ebll
Beskwti50
ResNee50
Imagdhet Acurscy
62.1
64.4
66.8
53.0
58.1
55.4
58.0
35.8
31,4
30
53.3 52.8
50.5
50.3
$5.1
Imagah Accurcy
43.6
16.0
43.3
45.3
38.2
38.2.
36.2-
31.0
31.5
30
23,8
Imapohes Accursay
77.6 71.3
76.1
76.2
75.8
74
74.4
73,6
725
74.5
74.0
73.6
73.9
69.
TENT
MLR
T2NT TBVT-
1L.
TEVT
SLA
MA
TENT
SLR
MA
TENT
TENT
1L.
Figure 2: Test-time adaptation results (top row) ImageNet-C. averaged across all 15 corruptions and severities, (middle row) ImageNet-R, (bottom row) clean ImageNet. NA refers to "No Adaptation".
cornuption mrost spilt by: class
corruption: giass pAur spiit by: class
corruption contrast spilt by:
cornupter: gaussen, poise split by:: class
30
10
Adeptation
split by: samples
split by: samples
spilt by samples
split by: samples
30
10
0.2
0.6 Spactiraction
0.4
0.8
1.0
0.2
0.4
0.5
0.8
1.0
0.2
0.4 0.6
0.5
1.0
0.2
0.4
0.5
Split Fraction
SpactFrattion
SplitFraction
Figure 3: Test-time adaptation of ResNet50 using (top row) a subset of classes, and (bottom row) a subset of samples per class on 4 different corruptions at severity 5. Accuracy is computed based on the evaluation of adapted model on the entire target data. Note that error bars are smaller visualize.
random seeds are depicted along with the error bars. These smaller error bars represent that the test-time adaptation results are not sensitive to the choice of random seed.
297
298
We also illustrate the performance of ResNet50 on the highest severity level across all 15 test corruptions of ImageNet-C Table Here, the adaptation results after epoch I and 5 reported. It can be seen that single epoch of test-time adaptation improves the performance significantly and makes minor improvements until epoch 5. TENT adaptation for more than one epoch result in reduced performance and TENT with Lai (TENT+) prevents this behavior. We note that both HLR and SLR clearly and consistently outperform TENT and TENT+ on the ResNet50. We also
289
300
301
302
303
304
compare our results with the hard pseudo-labels (PL) objective and also with an oracle setting where target data are used for adapting the model in a supervised manner (GT). Note that this oracle setting is not of practical importance but illustrates the empirical upper bound on fully test-time adaptation performance under the chosen modulation parametrization. The reported
305
the groundtruth labels
307
numbers in the table the average of three random seeds.

---

309
ImageNet-R We evaluate different adapted models on ImageNet-R and depict the results in Figure 2 (middle row). Results show that our methods significantly improve performance of all the models. including the model pretrained with DeepAugment+ Augmixx Moreover, both HLR and SLR clearly outperform TENT and TENT+
310
312
313
Evaluation with data subsets In the above experiments, the model is evaluated on the same data that is also used for the test-time adaptation. Here, we test model generalization by adapting on subset of target data and evaluate the performance on the whole dataset, which also includes unseen data that is not used for adaptation. We conduct two case studies: (i) adapt on the data from a subset of ImageNet classes and evaluate the performance on the data from all the classes. (ii) Adapt only on
314
315
316
317
318
a subset of data from each class and test on all seen and unseen samples from the whole dataset.
319
Figure 3 illlustrates generalization of a ResNet50 adapted on different proportions of the data across different corruptions, both in terms of classes and samples. We observe that adapting model on a small subset of samples and classes is sufficient to achieve reasonable accuracy on the whole target data. This suggests that the adaptation actually learns to compensate the data shift rather than overfitting to the adapted samples or classes. The performance of TENT decreases as the number of classes/samples increases, because Lent can converge to trivial collapsed solutions and more data corresponds to more updates steps during adaptation. Adding Lav such as in TENT+ stabilizes the adaptation process and reduces this issues. Reported are the average of random seeds with error bars.
320
321
322
323
324
325
326
327
Input transformation We investigate whether the input transformation (IT) module, trained end-to- end with ResNet50 and SLR loss on data of the respective distortion without seeing any source (undistorted) data, can partially undo certain domain shifts of ImageNet-C: and also increase accuracy on corrupted data. We measure domain shift via the structural similarity index measure (SSIM) [44 between the clean image (unseen by the model) and its distorted version/the output of IT on the distorted version. Table[]shows that IT increases the SSIM considerably on certain distortions such as Impulse, Contrast, Snow, and Frost. IT increases SSIM also for other types of noise distortions, while it slightly reduces SSIM for the blur distortions, Elastic, Pixelate, and JPEG. When combined with SLR, IT considerably increases accuracy on distortions for which also SSIM increased significantly
328
329
330
331
332
333
334
335
336
(for instance +20 percent points on Impulse, +4 percent points on Contrast) and never reduces accuracy by more than 0. 1 percent points. We provide illustrations of effect of IT in the appendix.
337
338
Clean images As sanity check, we investigate the effect of test-time adaptation when target data comes from the same distribution as training data. For this, we adapt pretrained models on clean validation data of ImageNet. The results in Figure 2 (bottom row) depict that the performance of
339
341
SLR/HLR adapted models drops by 1.5 to 2.5 percent points compared to the pretrained model,
342
We attribute this drop self-supervision being less reliable than the original full supervision on in- distribution training data. The drop is smaller for TENT and TENT+, presumably because predictions
343
344
on in- distirribution target data are typically highly confident such that there little gradient and thus little change to the pretrained networks by TENT. In summary. while self self-supervision by confidence maximization is powerful method for adaptation to domain shift, the observed drop when adapting
345
348
347
to data from the source domain indicates that there is "no free lunch" in test-time adaptation.
348
6
Conclusion
We propose a method to improve corruption robustness and domain adaptation of models in a fully test-time adaptation setting. Unlike entropy minimization, our proposed loss functions provide non-vanishing gradients for high confident predictions and thus attribute to improved adaptation in a supersuue manner. We also show that additional diversity regularization on the model predictions is crucial to prevent trivial solutions and stabilize the adaptation process. Lastly, we introduce a trainable input transformation module that partially refines the corrupted samples to support the adaptation. We show that our method improves corruption robustness on ImageNet-C and domain adaptation to ImageNet-R R on different ImageNet models. We also show that adaptation on a small fraction of data and classes is sufficient to generalize to unseen target data and classes.
350
352
354
356
358
Ethical and Societal Impact Our non-saturating loss increases accuracy but might result in over- confident predictions, which can cause harm in safety-critical downstream applications when not properly calibrated. At the same time, self-supervised confidence maximization might amplify bias in pretrained models. We hope that the diversity regularizer in the loss partially compensates this issuc.
360
9

---

References
362
363
[1] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 107222 2020.
364
385
[2] Joaquin AuiioneooCaandele Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset Shift in Machine Learning. MIT Press, 2009.
367
[3] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
368
369
370
[4] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: 200022222222,011111
372
373
374
[5] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7167-7176, 2017.
376
377
[6] Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and corruptions in natural corruption robustness. arXiv preprint arXiv:2102.1127,, 2021.
378
379
[7] Tuan- Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Ad- versarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2517-2526, 2019.
381
382
383
[8]
Dan Hendryeks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations (ICLR), 2019.
384
385
386
99]
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization, 2020.
387
388
389
390
[10] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Muchine Learning, pages 9229-9248. PMLR, 2020.
392
393
[1] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6028-6039. PMLR, 13-18 Jul 2020.
394
395
398
397
[12] Xiaofu Wu, Quan Zhou, Zhen Yang, Chumming Zhao, Longin Jan Latecki, et al. Entropy mini- mization diversity maximization for domain adaptation. arXiv preprint arXiv: 01600, 2020.
399
400
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
401
402
[14] Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprins arXiv:/906.00 903377 2019.
404
405
[15] Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arxiv:1907.07788. 2019.
406
407
408
[16] Christoph Kamann and Carsten Rother. Benchmarking the robustness of semantic segmenta- tion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8828-88888888 8838. 2020.
409
410
411
[17] Nic Ford, Justin Gilmer, Nicolas Carlini, and Dogus Cubuk. Adversarial examples are a natural consequence of test error in noise. arXiv preprint arXiv:1901.10513 2019.
10

---

413
[18]
Raphael Gontijo Lopes, Dong Yin. Ben Poole, Justin Gilmer, and Ekin D Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint arXiv: 1ww.1/9066001111 2019.
414
416
[19]
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6023-6032, 2019.
417
419
420
[20]
Christoph Kamann, Burkhard Güssefeld, Robin Hutmacher, Jan Hendrik Metzen, and Carsten Rother. Increasing the robustness of semantic segmentation models with painting-by seattinggb--nunn numbers... arXiv preprini arXiv:2010.05495, 2020.
422
423
[21] Evgenia Rusak, Lukas Schott, Roland Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. Increasing the robustness of dnns against image corruptions by playing the game of noise. arXiv preprint arXiv:2001.06057, 3, 2020.
424
425
428
[22] Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and conrptiinn in natural corruption robustness, 2021. URI http:/attpss//peeneeieeeeee
427
428
id=zbEupOtJJFF
[23] Joaquin Quifioncro-Candel Masashi Sugiyama, Anton Schwaighofer, and N Lawrence. Co- variate shift and local learning by distribution matching, 2008
430
431
[24]
Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adaptation. In Domain Adaptation in Computer Vision Applications, pages 153-171. Springer, 2017.
432
433
434
[25]
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
435
436
[26] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain adversaraa training of neural networks. The journal of machine learning research, 17(1):209--2030, 2016.
437
438
439
[27] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. PMLR, 2018.
440
441
442
[88] Yu Sun, Eric Tzeng. Trevor Darrell, and Alcxii A Efros. Unsupervised domain adaptation through self--uupevision. arXiv preprint arXiv: /909. / /825, 2019.
299 Jogendra Nath Kundu, Naveen Venkat, Rahul M V, and R. Venkatesh Babu. Universal source- free domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
445
446
447
[30] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsuper- vised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
448
449
450
[31] Vinod K. Kurmi, Venkatesh K. Subramanian, and Vinay P. Namboodiri. Domain impression: source data free domain adaptation method. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), January 2021.
452
453
[32] Hao- Wei Ych, Baoyao Yang. Pong C. Yuen, and Tatsuya Harada. Sofa: Source-data-free feature aligument for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision January 2021.
455
456
[33] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33, 2020.
457
458
459
[34] Zachary Nado, Shreyas Padhy, D. Sculley, Alexander D' D'Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction time batch normalization for robustness under covariate shift, 2021.
460
461
462
[35] Philipp Benz, Chaoning Zhang, Adil Karjauv, and In So Kweon. Revisiting batch normalization for improving corruption robustness. In Proceedings of the IEEF/CVE Winter Conference on Applications of Computer Vision (WACV), pages 494-503, January 2021.
463
11

---

485
[36] Fabio Maria Carlucci, Lorenzo Porzi, Barbara Caputo, Elisa Ricci, and Samuel Rota Bulo. Autodial: Automatic domain alignment layers. In Proceedings of the IEEE International Conference on Computer Vision, pages 5067-5075. 2017.
486
467
468
[37] Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprini arXiv: 0003.60033799 2016.
469
[38] Evgenia Rusak, Lukas Schott, Roland S. Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV, 2020.
471
472
473
474
[39]
Hossein Talebi and Peyman Milanfar. Learning to resize images for computer vision tasks. arXiv preprint arXiv: 103.09900 2021.
475
478
[40]
Hengshuai Yao, Dong-lai Zhu, Bei Jiang, and Peng Yu. Negative log likelihood ratio loss for deep neural network classification. In Kohei Arai, Rahul Bhatia, and Supriya Kapoor, editors, Proceedings of the Future Technologies Conference (FTC), pages 276-282, Cham,
477
478
479
2020. Springer International Publishing. ISBN 778-3-030-32520-
480
[41] Xingehao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang. and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv: 2017.
481
482
[42] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018-5027, 2017.
484
485
[43] Borch-Contributors. Torchvision models, 2020. URL attpps:/pyttorch.orr//// stable//oddels.hrg .html
486
487
[44]
Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: From error visibility structural similarity. Trans. Img. Proc., 13(4):600 612, April 2004. ISSN
488
489
1057-7149.
Checklist
490
491
1. For all authors...
492
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
494
(b) Did you describe the limitations of your work? [Yes] We discuss that our class distribution matching term Laiv requires knowledge of the class distribution PD (v) (3.2t) We also discuss that confidence maximization using a non-saturating loss might result in overconfident predictions (Section (.2.2). We also discuss the small drop of accuracy when applying our method to adaptation on data from the source domain (part "Clean images" in Section 4
485
496
497
498
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please refer to the "Ethical and Societal Impact" paragraph in Section
501
502
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
503
2. If you are including theoretical results...
505
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We do not provide theoretical results
506
507
(b) Did you include complete proofs of all theoretical results? [N/A] We do not provide theoretical results
509
3. If you ran experiments...
510
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We include the code and instructions as a part of supplementary material.
511
512
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We report training details and hyperparameters and how they are chosen in Section 4 part "Settings".
514
12

---

518
(c) Did you report error bars (e.g., with respect to the random seed after running experi- For the results in the ments multiple times)? [Yes] We provide error bars in Figure 2 Tables, we report error bars in the appendix.
517
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] We can not report the total amount of compute since did not track it. However, that all its activities including compute on the GPU clusters on which the experiments have been performed do no longer leave carbon footprint.
620
organization is carbon neutral so
522
523
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
525
(a) If your work uses existing assets, did you cite the creators? [Yes] Our work builds upon pre-trained models. We cite the creators (see Section 4 part "Models"). Our work uses
526
527
several openly available datasets. We cite their creators (see Section part "Datasets").
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or URL? [No]
530
We do not provide new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
532
533
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] To the best of our knowledge data used in this project does not contain any personally identifiable information or offensive content.
535
536
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A] No crowdsourcing was used and no research with human subjects was conducted
538
539
540
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals. if applicable? [N/A] No crowdsourcing was used and no research with human subjects was conducted
542
543
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] No crowdsourcing was used and research with human subjects was conducted
545
13


Reviews:
[{"review_id": "ShKALzt9Ozl", "reviewer": "Reviewer_XAwq", "summary": "This paper presents a method for test time adaptation based on several techniques. These include a self-supervised adaptation objective based on log likelihood ratios, an additional regularizing objective to encourage diverse predictions, and an input transformation module that is also trained with the aforementioned objectives. Together, these techniques lead to better performance on ImageNet-C and ImageNet-R compared to Tent, a recent and similar test time adaptation method based on entropy minimization.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n---\nThe paper itself notes that most of the techniques constituting the proposed method (diverse prediction regularization, input transformation module) are not novel. The self-supervised log likelihood ratio objective appears novel, as far as I am aware. I do not view the lack of originality as a substantial mark against this paper, as it is empirically focused and simply focuses on devising a better method for test time adaptation. And certainly, for this problem setting, the combination of the aforementioned techniques is novel and does lead to stronger empirical results than what has been previously reported.\n\n\nQuality\n---\nThe paper is of varying quality, and some improvements should be made as to this regard.\n\nFirst, the paper could do a better job at motivating the proposed log likelihood ratio objective. Currently, there are two threads of motivation: intuitive, in the \"non-saturating gradient\" property, and empirical, in the improved results. But in their current form, these seem insufficient. For the intuitive motivation, it is not immediately clear why non-saturating gradients are desirable. If a model is already very confident about its prediction on a certain point, why and how should it derive any self-supervised signal to adapt? The idea behind entropy minimization methods like Tent is that confidence correlates with accuracy, so increasing confidence is desirable. This idea, of course, has flaws, but certainly it is intuitively understandable. It is not clear what the idea is behind the log likelihood ratio objective, or the general idea that adaptation can and should be performed when the model is confident. Furthermore, the gradient plot in Fig 1 does not seem to align well with the intuitive explanation for two reasons: the gradient of the entropy (green) is still non-trivially large for fairly high confidence, e.g., 0.9, and the caption says that entropy has \"maximum gradient amplitude for low-confidence\" prediction when this seems clearly false, e.g., the gradient around 0.5 is close to 0?\n\nAs for the empirical motivation that the objective simply performs better, this is nice but cannot fully substitute for better exposition of the idea itself. It is also unclear how important the input transformation module is, because I believe the paper does not include the baseline of Tent with this module (if I understand correctly, Tent+ only adds the diversity regularization). If such a baseline were to work very well, then it would turn out that the input transformation module is actually the most important piece, rather than changing the adaptation objective. Without results from this baseline, this possibility cannot be ruled out.\n\nA more minor point is that it would be nice to include results in the setting where the model predicts on a batch directly after it adapts on the batch, rather than the current \"offline\" setting where several epochs are permitted on the entire test set before any predictions are made. In some practical applications, this would simply be infeasible, and predictions would have to be made right away. The paper could follow the general \"online\" setting that Tent uses.\n\n\nClarity\n---\nThe paper is generally well written and structured. The main concern regarding clarity is that of the motivation behind the adaptation objective. Another more minor concern is whether the claim that the input transformation module can \"undo distribution shifts\" is justified, considering the mixed results in Table 2. Lastly, the exposition could be improved in 3.2.1 regarding the diversity regularization in running average mode. It seems like p_t ( y ) in L172 needs a stop gradient operator on p_t-1 ( y ), such that the gradients of the loss L_div in L173 are computed only with respect to p_t^emp. Otherwise, the method would involve storing a long history of model activations, and this seems unwieldy (and unlikely). If I am correct, it would be good to clarify this in the paper.\n\n\nSignificance\n---\nThe paper focuses on an important topic (distribution shift) and an interesting emerging methodology (test time adaptation) with a moderate set of good results to support the proposed approach. Significance could be further improved through additional testbeds. For example, SVHN to MNIST/MNIST-M/USPS would offer another direct point of comparison to Tent. Alternatively, prior work [1] has shown that batch normalization based adaptation is ineffective on other test sets such as ImageNet-A and ImageNet-v2, so positive results on such test sets would be of high interest. The paper would be greatly strengthened through these additional experiments along with addressing the quality concerns noted above.\n\n[1] Schneider et al, \"Improving robustness against common corruptions by covariate shift adaptation\". NeurIPS 2020.\n\n---\n\nEdit after reading the author response\n---\nAs noted in my response below, my empirical concerns are resolved and I am thus upgrading my recommendation. I believe that, with the additional experiments, this work is above the bar for publication.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have done so adequately, though additional discussion would also be nice, even if just in the appendix. Note that this is a minor point and does not contribute to my overall score.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "RBOHWm2GD5s", "reviewer": "Reviewer_ab5L", "summary": "Focuses on test-time adaptation to distribution shifts, and proposes: i) a likelihood-ratio based loss with non-saturating gradients, ii) a diversity regularizer based on batch-wise entropy maximization, and iii) a learned input transformation module to undo the effect of distribution shift. Results are presented on standard test-time adaptation benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n\n– This work extends prior work (TENT) to optimize only batch norm parameters in addition to another “input transformation” module, to minimize a different loss function. \n\n– The originality of the stated technical contributions is somewhat limited:\n\ni) As the authors acknowledge (L215), Yao et al. [40] were the first to propose the negative log likelihood-ratio loss \n\nii) the diversity regularizer has also been proposed in prior work [11, 12]. The authors claim that they extend this to multiple batches, but this has also been done in Li et al. [A], which regularizes diversity by maximizing entropy over a history of model predictions over batches stored in a queue. \n\n– The shortcomings of the entropy minimization loss have been studied in prior work in domain adaptation: eg. [B] which proposes a new “max squares loss” and [C] which applies a Charbonnier penalty to entropy minimization. This work i) misses citing these highly related works, ii) clarifying what the contributions over these existing works is iii) empirically showing how the proposed loss compares against these prior works.\n\n[A] Li, Bo, et al. \"Rethinking distributional matching based domain adaptation.\" arXiv preprint arXiv:2006.13352 (2020).\n\n[B] Chen, Minghao, Hongyang Xue, and Deng Cai. \"Domain adaptation for semantic segmentation with maximum squares loss.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n\n[C] Yang, Yanchao, and Stefano Soatto. \"Fda: Fourier domain adaptation for semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\nQuality\n\n– The improvements over TENT on test-time adaptation are convincing, and the authors do well to benchmark multiple architectures.\n\n– The paper lacks a clear ablation study -- only ITM appears to be ablated. What is the effect of removing L_{DIV} / L_{SLR}? Or optimizing L_{DIV} only over the current batch rather than as a moving average? \n\n– It’s also not entirely convincing that ITM helps – Table 2 shows that SSIM (SLR) > SSIM (SLR+IT) in 7/15 settings (though admittedly by narrower margins as the authors mention). In my opinion, this is inadequate evidence to claim that “ITM partially undoes the distribution shift”. The authors say that “illustrations of the effect of IT” is included in appendix, but I was unable to find this experiment.\n\nClarity\n\n– The design of input transformation module is not clearly motivated. It appears to be a channel-wise learned affine transformation applied to a convex combination of the input and its (nonlinear) transformed version – while the first part seems identical to a batch-normalization layer, what is the motivation/intuition behind the second part? Have the authors experimented with alternatives?\n\n– L162 states “assuming knowledge of the class distribution .. on the target domain”, but does not clarify how this is estimated / approximated. Lacking access to labels on the target, how is p_{D’}(y) computed? Or is it assumed to be the uniform distribution?\n\n– Minor: Why is there a separate “pseudo labels” row in Table 1? Is it not identical to the previous “no adaptation” row by definition? \n\n– In Figure 2, HLR outperforms SLR on ImageNet-C and clean ImageNet, but the opposite happens on ImageNet-R. Have the authors \ninvestigated the reason for this? When is it more appropriate to use one over the other?\n\n– Minor: Why is SHOT not directly included as a point of comparison in say Table 1, but rather in the appendix?\n\nSignificance\n\n– The paper studies an interesting and practical setting of test-time adaptation, and the proposed approach appears to improve upon the existing state-of-the-art (TENT) on standard benchmarks.\n\nOverall\n\nThe paper studies the important and practical setting of test-time adaptation, and extends prior work (TENT) via a new loss function with non-saturating gradients and a new input transformation module, which lead to improved performance. The paper is mostly well-written and easy to follow. I have concerns primarily about the originality of the paper’s stated contributions (primarily, not citing / comparing to highly relevant work in DA studying the shortcomings of entropy minimization) and the quality of its experimental section (primarily ablations), and so believe that in its current form it is below the bar for acceptance.\n\n\n---post-rebuttal---\n\nI have read the author response and other reviews. While I still believe the technical novelty is slightly limited, I think the paper is well-written and the rebuttal addresses most of my primary concerns, and recommend marginal acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors adequately addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "GnPjjD9_Htg", "reviewer": "Reviewer_mx5v", "summary": "Fully test-time adaptation is a promising approach to handling data shift and making the DNNs more robust. This paper presents a new method for fully test-time adaptation. Specifically, they propose a non-saturating surrogate of entropy and combine it with the diversity regularizer. They empirically demonstrate the efficacy of the proposed method, comparing it against a recently proposed test-time adaptation method, Tent, using various backbone networks. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "---\nOverall, the paper is very clearly written, and it was a fan to read. The proposed method might not be a revolutionary one, a combination of several (known) techniques, yet it is well described, and the experimental results are extensive enough to show benefits. They also provide results on various backbone networks, which is informative for future researches. However, several details regarding the motivation and implementation are unclear for me, making me hesitate to vote for full acceptance of the paper. \n\n---\n**Comments/Questions**\n\n1. The motivation behind the proposal of a non-saturating objective is that information about more confident samples should be affected more on test-time adaptation. However, I do not fully agree with this point, as it does not affect the training just because the model parameters are already suitable for the data point. \n2. If I correctly understand the paper, there are two main differences between the Tent and the proposed method: (1) loss function and (2) tuned parameter. However, it is not clear to me why do we need to change the latter one. Can you discuss more that? Specifically, what is the potential problem of transforming only the parameter of BN as with the Tent?\n3. Figure 2 suggests that the proposed method often negatively affects the performance on the clean dataset, compared to the Tent and Tent+. Why does it happen? Besides, it is not clear to me how exactly the performance on the clean dataset is measured. More specifically, is the model adapted on the clean dataset or adapted on the corrupted dataset? \n4. In section 4, the authors say that the experiment uses cosine decay scheduling. However, looking at the code, this doesn't seem to apply to Tent.  It seems that the comparison is unfair due to this. \n5. Learning five epochs during the test seems a bit strange to me. This is because the deployed model usually needs to make correct predictions at that moment; there is no point in going back in time and correcting the predictions. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper mentions a little about the difficulty of using the method in safety-critical downstream tasks and describes the diversity regularizer possibly compensates the issue. However, it is unclear to me how the proposed regularizer help to migrate the issue. I feel this is the most critical issue of fully test-time adaptation, and it is worth discussing more. \n\nBesides that, I can not find any statements about limitations. For example, it can not directly be used for the regression, as the non-saturating modification assumes the discrete prediction (although Tent has the same issue). Would you please discuss more about such a limitation of this work? ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for improving DNNs performance on distribution shifts via test time adaptation based on several techniques (a non-saturating loss function, diversity maximization and input transformation module). The paper requires significant modifications to be ready for publication in terms of clarity, additional comparison experiments, discussing the effectiveness of the method in more details, etc (see reviews and discussions for details). The discussions have not been enough to convince me and reviewers to accept the paper as is. I encourage authors to update the paper in light of discussions and feedback and submit to a future venue.
Decision: Reject