If your data distribution shifts, use self-learning
Authors: Evgenia Rusak
Steffen Schneider
George Pachitariu
Luisa Eck
Peter Vincent Gehler
Oliver Bringmann
Wieland Brendel
Matthias Bethge
Venue: Accepted by TMLR ()

Abstract: We demonstrate that self-learning techniques like entropy minimization and pseudo-labeling are simple and effective at improving performance of a deployed computer vision model under systematic domain shifts. We conduct a wide range of large-scale experiments and show consistent improvements irrespective of the model architecture, the pre-training technique or the type of distribution shift. At the same time, self-learning is simple to use in practice because it does not require knowledge or access to the original training data or scheme, is robust to hyperparameter choices, is straight-forward to implement and requires only a few adaptation epochs. This makes self-learning techniques highly attractive for any practitioner who applies machine learning algorithms in the real world. We present state-of-the-art adaptation results on CIFAR10-C (8.5% error), ImageNet-C (22.0% mCE), ImageNet-R (17.4% error) and ImageNet-A (14.8% error), theoretically study the dynamics of self-supervised adaptation methods and propose a new classification dataset (ImageNet-D) which is challenging even with adaptation.

Published in Transactions on Machine Learning Research (11/2022)
If your data distribution shifts, use self-learning
Evgenia Rusak" University of Tübingen
rugervia.rusaa@@eellggeedd
Steffen Schneider" University of Tübingen
stefeeusschciidder@hhhhglo....d
George Pachitariu University of Tübingen
geong..acchtarriuutue.cc
Luisa Eck University of Orford
luisa, www.eekk@phsscs.o..... oww.iic.ss..cc....
Peter Gehler Amazon Tübingen
pyehler@amazzon.co
Oliver Bringmann University of Tübingen
oliver. hetorbbrtiiguaannnouuuuuubinnnnn
Wieland Brendel:
wieland Weelandd.rendd@@@hciigge.......
Maz-Planck Institute for Intelligent Systems Tübhingen
Matthias Bethge*
University of Tübingen
Reviewed on OpenReview: https: opwnreviee. formm? id= XXXX
Abstract
We demonstrate that self-learning techniques like entropy minimization and pseudo-labeling are simple and effective at improving performance of a deployed computer vision model under systematic domain shifts. We conduct a wide range of large-scale experiments and show consistent improvements irrespective of the model architecture, the pre-training technique or the type of distribution shift. At the same time, self-learning is simple to use in practice because it does not require knowledge or access to the original training data or scheme, is robust to hyperparameter choices, is straight-forward to implement and requires only a few adaptation epochs. This makes self-lcarning techniques highly attractive for any practitioner who applies machine learning algorithmss the real world. We present stale-of-the-art adaptation results on CIFAR10-C (8.5% error), ImageNet-Ci (22.0% mCE), ImageNet-Ii (17.4% error) and ImageNct-A (14.8% error), theoretically study the dynamics of self- supervised adaptation methods and propose new classification dataset (ImageNet-D) which is challenging even with adaptation.
"Equal contribution.
Work initiated during internship Amazon Tübingen. Joint senior authors.

---

Published in Transactions on Machine Learning Research (11/2022)
Large scale (semi) supervised pre-training
Unsupervised domain adaptation
Robustification
Self-supervised pre-training
Pre-trained model
Student
Self-Learning Entropy Minimization Pseudo-Labeling
Adapted model
Systematic domain shift
Teacher
Figure 1: Robustness and adaptation to datasets has traditionally been achieved robust pre-training (with hand-selected data-driven augmentation strategies, or additional data), unsupervised domain adaptation (with access unlabeled samples from the set), or, more recently, self-supervised learning methods. We show that on top of these different pre-training lasks, it is always possible (irrespective of archilecture, model size pre-training algorithm) to further adapt models to larget domain with simple self-learning techniques.
1 Introduction
Deep Neural Networks (DNNs) can reach human-level performance in complex cognitive tasks (Brown et al., 2020; He al., 2016a; Berner et al., 2019) if the distribution of the test data is sufficiently similar to the training data. However, DNNs known to struggle if the distribution of the test. data shifted relatively to the training data (Geirhos et al., 2018; Dodge & Karam, 2017).
Two largely distinct communities aim to increase the performance of models under test-time distribution shifts: The robusiness communily generally considers ImageNet-scale datasets and evaluates models in an ad-hoc scenario. Models are trained on a clean source dataset like ImageNet (Deng et al., 2009), using heavy data augmentation (Hendrycks et al., 2020a; Rusak et al., 2020; Geirhos et al., 2019) and/or large-scale pre-training (Xie al., 2020a; Mahajan et al., 2018). The trained models are not adapted in any way test-time distribution shifts. This evaluation scenario is relevant for applications in which very different. distribution shifts encountered in unpredictable order, and hence misses out the gains of adaptation to unlabeled samples of the targel distribution.
The unsupervised domain adaptation (UDA) community often considers smaller-scale datasets and assumes that both the source and the (unlabelod) target dataset are known. Models are trained on both datascts, c.g., with an adversarial objective (Ganin et al., 2016; Tzeng al., 2017; Holfmaan et al., 2018), before evaluation on the target domain data. This evaluation scenario provides optimal conditions for adaptation, but the reliance the source datasct makes UDA more computationally expensive, more impractical and prevents the use of pre-trained models for which the source dataset is unknown simply too large. We refer the reader to Farahani et al. (2021) for a review of UDA.
In this work, we consider the source-free domain adaptation selting, a middle ground between the classical ad-hoc robustness setting and UDA in which models can adapt to the target distribution but without using the source dataset (Kundu et al., 2020; Kim al., 2021; Li et al., 2020; Liang al., 2020). This evaluation scenario is interesting for many practitioners and applications as an extension of the ad-hoc robustness scenario. It evaluates the possible performance of deployed model on systematic, unseen distribution shift inference time: embedded computer vision system in autonomous car should adapt to changes without being trained on all available training data; imagc-based quality control software may not necessarily open-source the images it has been trained on, but still has to be adapled to the lighting conditions at the operation location; a computer vision system in a hospital should perform robustly when tested on scanner different from the one used for producing the training images- intissmmmpptttyt be known development time which scanner the vision system will be tested on, and it might be prohibitod to share images from many hospitals to run UDA.
might not
2

---

Published in Transactions on Machine Learning Research (11/2022)
Can self-learning methods like pseudo-labeling and entropy-mminiizzatin also be used in this source-free domain adaptation setting?' To answer this question, we perform an extensive study of several self-learning variants, and find consistent and substantial gains in test-time performance across several robustness and out-of-domain benchmarks and a wide range of models and pro-training methods, inchuding models trained with UDA methods that do not use sell-lerrnigg, see Figure 1. We also find that self-learnigg outperforms state-of-thhcart source-free domain adaptation methods, namely Time-Time Training which is based on a self-supervised auxiliary objective and continual training (Sun et al., 2020), test-time entropy minimization (Wang et al., 2021), Meta Teme Training (Bartler et al., 2022), and (gradient-free) frree))t-free) BatchNorm adaptation (Schneider et al., 2020; Nado et al., 2020), and can be improved with additional techniques such as diversity regularization (Mummadi et al., 2021). We perform a large number of ablations to study important design choices for self-learning methods in source-free domain adaptation. Furthermore, we show that a variant of pseudo-labeling with a robust loss function consistently outperforms entropy minimization on ImageNet-scale datascts.
We begin by positioning our work in the existing literature ($2 and proceed with an overview of various self-learning variants that have been applied over the past years, and propose new technique for robust pscudo-labeling ($ We then outlinc a rigorous experimental protocol that aims to highlight the strengths (and shortcomings) of various self-learning methods. We test various model architectures, with different pre-training schemes covering the most important models in unsupervised domain adaptation, robustness, and large-scale pre-training (8 4). Using this protocol, we show the effectiveness of self-learning across architectures, models and pre-tra pre-training schemes ($)). We proceed with an in-depth analysis of self-learning, both empirical ($6) and theoretical ($7) Since the outlined results on TeageNet-CC (22.0% mCF), TmageNet-R, (17.4% error) and ImageNet-A (14.8%) approach clean performance (11.6% error for bascline), we propose ImageNet-D as new benchmark, which analyse in $ 8. We conclude by proposing a set of best practices for evaluating test-time adaptation techniques in the future to ensure scientific rigor and to enable fair model and method comparisons ($9).
2 Related Work
Test- Time Adaptation The main question we ask in this work is whether self-learning methods such as entropy minimization and different variants of pseudo-labeling can improve the performance of models when adapted at test-time to data with a distribution shift relative to the training data. Our work is most similar to test-time entropy minimization (TENT; Wang et al., 2021) since entropy minimization is one of our studied tost-time adaptation techniques. The conceptual difference between our experiments and Wang et al. (2021) is that Wang et al. (2021) compare TENT to UDA while we argue that UDA can be regarded as a pretraining step, and self-learning can be used on top of any checkpoint pretrained with UDA. Wang et al. (2021) study the effectiveness of entropy minimization across different models, datasets and tasks. We expand upon their experimental setup and show that entropy minimization is effective across a large range of model architectures (convolutional neural networks, Vision Transformers (Dosovitskiy et al., 2021; Caron et al., 2021)), sizes (ResNet50 to EfffciienNeet-.22 (Xie et al., 2020a; Tan & Le, 2019)), and are orthogonal to robustification (e.g,, DeepAugment (Hendrycks et al., 2020a)) and other pre-training schemes. Finally, we perform large hyperparameter study for test-time entropy minimization, and thereby further improve upon the results reported by Wang et al. (2021).
We also compare our self-learning results to gradient-free adaptation of batch normalization (BN; loffe & Szegedy, 2015) statistics (BN adapt; Schneider et al., 2020) who proposed re-estimating the BN statistics on the shifted data distribution.
In Test-Time Training (TTT; Sun et al., 2020), the softmax cross-entropy loss is combined with a rotation prediction task (Gidaris et al., 2018) during pretraining the source dataset. At test-time, the model is fine-tuned on the unlabeled test data with the self-supervised task. Sun et al. (2020) report results when
SSelf-learning defined by Tsypkin (1968) to denote "learning when there is no external indication concerning the correctness of the response of the automatic system to the presented patterns". We opted to use this term name the superset of pseudo-laheling, entropy minimization and self-supervised learning variants highlight the fact these methods do not require ground truth labels for adaptation.
3

---

Published in Transactions on Machine Learning Research (11/2022)
adapting to single test example, and also for online adaptation where the model successively adapts to a stream of data, where the data can either come from the same or gradually changing distribution. We added a detailed comparison to Sun et al. (2020) in Appendix C.6. Liu et al. (2021) (TTT+++) replace the rotation prediction task with SimCLR (Chen et al., 2020a), and find this modification improves performance.
Fastwood et al. (2022) propose Feature Restoration where the approximate feature distribution under the target data is realigned with the feature distribution under source data. Eastwood et al. (2022) report results on CIFAR10 (among other datasets) for ResNet18 architecture which allows us to include their method as baseline. In concurrent publication, Niu et al. (2022) propose anti-forgetting test- test-time adaptation method called EATA, which combine sample-efficient entropy minimization with anti-forgetting weight regularization. They report accuracy numbers on ImageNet-CC for a ResNet50 which we incldde as a baseline. Mummadi et al. (2021) introduce a novel loss to stabilize entropy minimization by replacing the entropy by a non-saturating surrogate and a diversity regularizer based on batch-wise entropy maximization that prevents convergence to trivial collapsod solutions. To partially undo the distribution shift at test time, they additionally propose to add an input. transformation module to the model. Mummadi et al. (2021) report results on the highest severity of ImageNet-C for ResNet50 which we include as a comparison to our results. We note that Mummadi et al. (2021) constitute concurrent unpublished work.
Meta Test-Time Training (MT3; Bartler et al., 2022) combine meta-learning, self-supervision and test-time training to adapt a model trained on clean CIFARIO (Krizhevsky et al., 2009) to CIFARIO-C (Hendrycks & Dietterich, 2019). We added a detailed comparison to Bartler et al. (2022) in Appendix C.7.
The following papers also consider the setting of test-time adaptation, but are not used as direct. baselines in our work, bocause they study other datascts tasks. Azimi et al. (2022) show performance improvements when using lest-time adaplation on video data. They show adaptation results for the popular test-time adaptation techniques of BN adaptation (Schneider et al., 2020), Test-Time Training (Sun et al., 2020) and test-time entropy minimization (Wang et al., 2021). MEMO (Zhang et al., 2021) maximizes the prediction consistency of different augmented copies regarding a given test sample. We do not consider the single-sample adaptation setting, and comparing our self-learning techniques to MEMO is not a fair setting for MEMO; unsurprisingly, techniques benefiting from multiple samples such as TENT (Wang et al., 2021) outperform MEMO. AdaContrast by Chen et al. (2022) combincs pscudo-labeling with other techniques, such as self-supervised castrastive learning on the larget domain, soft k-nearest neighbors voling to stabilize the lseudo-labels, as well as consistency and diversity regularization. A direct comparison with their results is difficult because they evaluate on VISDA-C and DomainNet, and so we would need to train our methods on either of the datasets and perform a full hyperparameter search for a fair comparison. We do have one point of comparison: in their paper, Chen et al. (2022) perform better than TENT (Wang et al., 2021). However, we note that Chen et al. (2022) used the default hyperparameters of TENT and did not perform hyperparameter tuning on the new dataset, thus, we would expect the performance of properly tuned TENT to be better than reported in the paper. We expect the additional changes of AdaContrass to further improve upon our simple self-lcarning basclines. Our work is conceptually similar to virtual adversarial domain adaptation in the fine-tuning phase of DIRT-T (Shu et al., 2018). In contrast to DIRT-T, our objective is simpler and we scale the approach to considerably larger datasets on ImageNet scale. Iwasawa & Matsuo (2021) propose a test-time adaptation algorithm for the task of domain generalization based on computing the distance of each test sample and
for each class. Kumar et al. (2020) study the setting of self-lcarning for gradual domain adaptation. They find that self-learning works better if the data distribution changes slowly. The gradual domain adaptation setting differs from ours: instead of gradual shift over time, we focus a fixed shift at test time.
Self-learning for domain adaptation Xie et al. (2020b) introduce "In-N-Out" which uses auxiliary information to boost both in- and out-of-distribution performance. AdaMatch (Berthelot et al., 2021) builds upon FixMatch (Sohn et al., 2020) and can be used for the lasks of unsupervised domain adaptation, semi-supervised learning and semi-supervised domain adaptation as a general-purpoee algorithm. Prabhu et al. (2021) propose SENTRY, an algorithm based on judging the predictive consistency of samples from the target domain under different image transformations. Zou et al. (2019) show that different types of confidence regularization can improve the performance of self-learning. A theoretically motivated framework
4

---

Published in Transactions on Machine Learning Research (11/2022)
for self-learning in domain adaptation based consistency regularization has been proposed by Wei et al. (2020) and then extended by Cai et al. (2021).
The main differences from these works to ours are that they 1) utilize both source and target data during training (i.e., the classical UDA setup) whereas we only require access to unlabeled target data (source-free freee setup), 2) train their models from scratch whereas we adapt pretrained checkpoints to the unlabeled target data, and 3) are oftentimes more complicated (also in terms of the number of hyperparameters) than our approach due to using more than one term in the objective function. We would like to highlight that utilizing source data should always result in better performance compared to not using source data. Our contribution is to show that self-learning can still be very beneficial with a small compute budget and no access to source data. Our setup largets "deployed systems"" e.g., a self-driving car or a detection algorithm in a production line which adapts to the distribution shift "on-the-fly" and cannot (or should not) be retrained from scratch for every new domain shift.
Model selection Gulrajani Lopez-Paz (2021) show that model selection for hyperparameter tuning is ton-trivial for the task of domain generalization, and propose model selection criteria under which models should be selected for this task. Following their spirit, we identify model selection criterion for test-time adaptation, and rigorously use it in all our experiments. We outperform state-of-the-arl techniques which did not disclose their hyperparameter selection protocols.
3 Self-learning for Test- Time Adaptation
Teypkin (1968) defines self-learning as "learning when there is no external indication concerning the correctness of the response of the automatic system to the presented patlerns", and thus, we use this term a supersel of different variants of pseudo-labeling and entropy minimization to highlight that these methods can be used for adaptation to unlabeled data. Different versions of self-lcarning have been used in both unsupervised domain adaptation (French et al., 2018; Shu el al., 2018), self-supervised representation learning (Caron et al., 2021), and in semi-snpervised learning (Xie et al., 2020a). In typical self-learning setting, a teacher network f trained on the source domain predicts labels on the target domain. Then, a student model f is fine-tuned on the predicted labels.
In the following, let f' (x) denote the logits for sample x and let p'(jlx = (x) denote the probability for class j obtained from a softmax function a-(-). Similarly, (x) and p'(ix) denote the logits and probabilities for the student model f*. For all techniques, one can optionally only admit samples where the probability max, p¹ x) exceeds some threshold. We consider three popular variants of self-learning Pseudo-labeling with hard or soft labels, as well as entropy minimization.
Hard Pseudo-Labeling (Lee, 2013; Galstyan & Cohen, 2007). We generate labels using the teacher and train the student pseudo-labels i using the softmax cross-entropy loss,
t (x) = log p" (ilx), i == argmax, p² (j(x)
(1)
Usually, only samples with confidence above a certain threshold are considered for training the student. We test. several thresholds but note that thresholding means discarding a potentially large portion of the data which leads to performance decrease in itself. The teacher is updated after each epoch.
Soft Pseudo-Labeling (Lee, 2013; Galstyan & Cohen, 2007). In contrast to the hard grendo-labeling variant, we here train the student on class probabilities predicted by the teacher,
-- (ixx) (ilx).
(s(x) := 1=(x)==-Aa E
Ex'
(2)
Soft pseudo-labeling is typically not used in conjunction with thresholding, since it already incorporates the certainty of the model. The teacher is updated after each epoch.
Entropy Minimization (ENT; Grandvalet & Bengio, 2004; Wang et al., 2021). This variant is similar to soft pseudo-labeling, but no longer differentiate between a teacher and student network. Tt.
5

---

Published in Transactions on Machine Learning Research (11/2022)
corresponds to an "instantancous" update of the teacher. The training objective becomes
(x(x) : - logg) (jlx).
(3)
Inluitively, self-learning with entropy minimization leads to a sharpening of the output distribution for each sample, making the model more confident in its predictions.
Robust Pscudo-Labeling (RPL). Virtually all introduced self-learning variants use the softmax cross- entropy classification objective. However, the softmax cross-entropy loss has been shown to be sensitive to label noise (Zhang & Sabuncu, 2018; Zhang et al., 2017). In the setting of domain adaptation, inaccuracies in the teacher predictions and, thus, the labels for the student, are inescapable, with severe repercussions for training stability and hyperparameter sensitivity as we show in the results.
As straight-forward solution to this problem, we propose to replace the cross-entropy loss by a robust classification loss designed to withstand certain amounts of label noise (Ghosh al., 2017; Song al., 2020; Shu et al., 2020; Zhang& & Sabuncu, 2018). A popular candidate is the Generalized Cross Entropy (GCE) loss which combines the noise-tolerant. Mean Absolute Error (MAF) loss (Ghosh et al., 2017) with the CE loss. We only consider the hard labels and use the robust GCE loss as the training loss for the student,
argmax, p (j(x), (ocE(xx == (1 p" (1)x))),
with q E (0, 1]. For the limit case + 0, the GCE loss approaches the CE loss and for q q= 1, the GCE loss is the MAE loss (Zhang & Sabuncu, 2018). We test updating the teacher both after every update step of the student (RPL) and once per epoch (RPL.P).
Adaptation parameters. Following Wang et al. (2021), we only adapt the afline scale and shift parameters y and / following the batch normalization layers (loffe & Szegedy, 2015) in most of our experiments. We verify that this type of adaptation works better than full model adaptation for large models in ablation study in Section 6.
Additional regularization in self-learning Different regularization terms have been proposed a means to stabilize entropy minimization. Niu et al. (2022) propose an aorieforg weight regularization term, Liang et al. (2020); Mummadi et al. (2021); Chen et al. (2022) add diversity regularizer, and Chen et al. (2022) use an additional consistency regularizer. These methods show improved performance with these regularization lerms over simple entropy minimization, but also introduce additional hyperparameters, the tuning of which significantly increases compute requirements. In this work, we do not experiment with additional regularization, as the main point of our analysis is to show that pure self-learning is effective at improving the performance over the unadapted model across model architectures/ /sizes and pre-training schemes. For practitioners, we note that regularization terms can further improve the performance if the new hyperparameters are tuned properly.
4 Experiment design
Datasets. ImageNet-C (IN-C; Hendrycks Dietterich, 2019) contains corrupted versions of the 50 000 images the ImageNet validation set. There are fifteen test and four hold-out corruptions, and there are five severity levels for each corruption. The established metric to report model performance on IN-C is the mean Corruption Error (mCE) where the error is normalized by the AlexNet error, and averaged over all corruptions and severity levels, see Eq. 20, Appendix C.I. ImageNet-R (IN-R: Hendryeks et al,, 2020a) contains 30 000 images with artistic renditions of 200 classes of the ImageNet datasct. ImageNet-A (IN-A; Hendrycks et al., Nes55
2019) is composed of 7500 unmodified real-world images on which standard Image (He et al., 2016b) models yield chance level performance. CIFAR10 (Krizhevsky et al., 2009) and STL10 (Coates et al., 2011) are small-scale image recognition datascts with 10 classes cach, and training sets of 50 000/5000 5000 images and test sets of 10 100000000 images, respectively. The digit datasets MNIST (Deng, 2012) and MNIST-M (Ganin et al., 2016) both have 60 000 training and 10 000 test images.
6

---

Published in Transactions on Machine Learning Research (11/2022)
Table 1: Self-learning decreases the error on ImageNet-scale robustness datasets. Robust pseudo-labeling
generally outperforms entropy minimization.
number of w/o adapt
w/ adapt (A)
w/ adapt (A)
mCE [%] on IN-C test (%)
parameters
RPL
ENT
ResNet50 vanilla (He et al., 2016b)
2.6 x 107
76.7
50.5 (-26.2)
51.6 (-25.1)
ResNet50 AM (Hendrycks et al., 2020a)
2.6 x 107
53.6
41.7 (-11.9)
42.6 (-11.0)
DenseNet161 vanilla (Huang et al., 2017)
2.8 x 10?
66.4
47.0 (-19.4)
47.7 (-18.7)
ResNeet101 vanilla (Xie et al., 2017)
8,8 x 107
66.6
43.2 (-23.4)
44.3 (-22.3)
ResNeXt10lwxxd DAug+AM (Hendrycks et al., 2020a)
8,8 x 107
44.5
34.8 (-9.7)
35,5 (-9.0)
ResNeXt1001swws IG-3.5B (Mahajan et al., 2018)
8.8 x 10
51.7
10.9 (-10.8)
10.8 (-10.9)
ElffcientNeet-.22 Noisy Student (Xie al., 2020a)
4.8 x 10
28.3
22.0 (-6.3)
23.0 (-5.3)
lopl error [%]] on IN-R (*)
ResNet50 vanilla (He et al., 2016b)
2.6 x 107
63.8
54.1 (-9.7)
56.1 (-7.7)
FfficientNet--.22 Noisy Student (Xie al., 2020a)
4.8 x 10
23.5
17.4 (-6.1)
19.7 (-3.8)
topl error [%] on ImageNet-A ('x)
ElficientNettT.22 Noisy Student (Xie al., 2020a)
4.8 10
16.5
14.8 (-1.7)
15.5 (-1.0)
Hyperparameters. The different self-learning variants have a range of hyperparameters such as the learning rate or the stopping criterion. Our goal is to give a realistic estimation on the performance to be expected in practice. To this end, we optimize hyperparameters for each variant of pseudo- set of IN-C that contains four types of image corruptions (#speckle noise", "Gaussian blur", "saturate" and "spatter") with five different strengths each, following the procedure suggested in Hendrycks & Dietterich (2019). We refer to the hold-out set of IN-C set of CIFAR1O-C for hyperparameter tuming. On all other datasets, we use the hyperparameters obtained on the hold-out sets of IN-C (for large-scale datascts) or CIFARIO-C (on small-scale datasets).
on a hold-out
dev set. On the small-scale datasets, use the hold-out
Models for ImageNet-scale datasets. We consider five popular model architectures: ResNet50 (He et al., 2016b), DenseNet161 (Huang et al., 2017), ResNeXt101 (Xie et al., 2017), EfficientNNet-22 (Tan & Le, 2019), and the Vision Transformer (ViT: Dosovitskiy et al., 2021) (sce Appendix B.1 for details on the used models). For ResNet50, DenseNet and ResNeXt101, we include a simple vanilla version trained on ImageNet only. For ResNet50 and ResNeXt101, additionally include a state-of-thee trained with DeepAugment and Augmix (DAug+ also include a version that was trained on 3.5 billion weakly labeled images (IG-3.5B; Mahajan et al., 2018). For EfficientNNet-.22 we select the current state of the art on IN-C which was trained on 300 million images from JFTT-30M (Chollet, 2017; Hinton et al., 2014) using a noisy studenn-teaccher protocol (Xie et al., 2020a). Finally, for the ViT, we use the model pretrained with DINO (Caron et al., 2021). We validate the ImageNet and IN-C performance of all considered models and match the originally reported scores (Schneider et al., 2020). For EfffciienNeet-I2, we match ImageNet top-1 accuracy up to 0.1% points, and IN-C up to 0.6% points mCE.
robust version Hendrycks et al., 2020a)². For the ResNeXt model, we
Models for CIFAR10/ MNIST-scale datasets. For CIFARIO-C experiments, we use three WideRes.Nets (WRN; Zagoruyko & Komodakis, 2016): the first one is trained on clean CIFAR10 and has a depth of 28 and a width of 10, the second one is trained with CIFAR10 with the AugMix protocol (Hendrycks et al., 2020b) and has a depth of 40 and a width of 2, and the third one has a depth of 26 layers, and is pre-trained on clean CIFAR10 using the default training code from https://githu: com cww//oomneeeeeeeee ceaannli/pyytordee cifar. We used this code-base to also train the ResNet18 and the ResNet50 models CIFAR10. The remaining small-scale models are trained with UDA methods. We propose to regard any UDA method which requires joint training with source and target data as pre-training step, similar to regular pre-training on ImageNet, and use self-learning on top of the final checkpoint. We consider two popular UDA methods: self-supervised domain adaptation (UDA-SS; Sun et al., 2019) and Domain Aaveain-dvvesaariall Training of Neural Networks (DANN; Ganin et al., 2016). In UDA-SS, the authors seek to align the representations of both domains by performing an auxiliary self-supervised task on both domains simultaneously. In all UDA-SS experiments, we use a embedding
WideResNet with depth of 26 and width of 16. In DANN, the authors learn a
see leaderboard github. wwt htmm/h/nndrycck//rruuseeees
7

---

Published in Transactions on Machine Learning Research (11/2022)
Table 2:
Self- learning decreases the error on scall-scale datasets, for models pre-trained using data augmen-
tation and unsupervised domain adaptation. Entropy minimization outperforms robust pscudo-labcling.
number of w/o adapt w/ adapt (A)
w/ adapt (A)
topl error [%] on CIFARIO-C (N)
parameters
RPL
ENT
WRN-28-10 10 vauilla (Zagoruyko & Komodakis, 2016)
3.6 x 107
26.5
13.7 (-12.8)
13.3 (-13.2)
WRN-40-2 AM (Hendrycks et al., 2020b)
2.2 x 10
11.2
9.0 (-2.2)
8.5 (-2.7)
WRN-26-1-GN (Bartler et al., 2022)
1.5 x 10
18.6
18.0 (-0.6)
18.4 (0.2)
WRN-2661-BN (Zagoruyko & Komodakis, 2016)
1.5 x 10
25.8
15.1 (-10.7)
13.1 (-12.7)
WRN-26616 vanilla (Zagoruyko & Komodakis, 2016)
9.3 x 10"
24.2
11.8 (-12.4)
11.2 (-13.0)
WRN-26616 UDA-SS (Sun et al., 2019)
9.3 x 10?
27.7
18.2 (-9.5)
16.7 (-11.0)
WRN-26616 DANN (Ganin et al., 2016)
9.3 107
29.7
28.6 (-1.1)
28.5 (-1.2)
UDA CIFAR1O -SS--,, lopl error targel [%]())
WRN-26-16 UDA-SS (Sun et al., 2019)
9.3 X 10
28.7
22.9 (-5.8)
21.8 (-6.9)
WRN-26-16 16 DANN (Ganin et al., 2016)
9.3 X 10?
25.0
24.0 (-1.0)
23.9 (-1.1)
UDA -NNIST-MNIST -NTST-M, top1 eтror on target [%[(})
WRN-26-16 UDA-SS (Sun et al., 2019)
9.3 107
1.8
2.4 (-2.4)
2.0 (-2.8)
WRN-26-2 DANN (Ganin et al., 2016)
1.5 x 10
11.4
5.2 (-6.2)
5.1 (-6.3)
by optimizing a minimax objective. For all DANN experiments except for MNNIST we use the same WRN architecture above. For the MNIST-MMNIST-M experiment, the training with the larger model diverged and we used a smaller WideResNet version with width of 2. We note that DANN training involves optimizing minimax objective and is generally harder to lune.
5 Self-learning universally improves models
Self-learning is a powerful learning scheme, and in the following section we show that it allows to perform test-time adaptation on robustified models, models obtained with large-scale pre-training, as well as (already) domain adapted models across a wide range of datasets and distribution shifts. Our main results on large-scale and scall-scale datasets are shown in Tables 1 and 2. These summary tables show final results, and all experiments use the hyperparameters we determined separately on the dev set. The self-learning loss function, i.c. soft- hard-pssundollaeeling / entropy minimization / robust psoudo-labeling, is hyperparamcter itself, and thus, in Tables and 2, we show the overall best results. Results for the other loss functions can be found in Section 6 and in Appendix C.
Self-lcarning successfully adapts ImageNet-scale models across different model architectures on IN-C, IN-A and IN-R (Table 1). We adapt the vanilla ResNet50, ResNeXt101 and DenseNet161 models to IN-C and decrease the mCE by over 19 percent points in all models. Further, self-learning works for models irrespective of their size: Self-learning substantially improves the performance of the ResNet50 and the ResNext101 trained with +M,, on IN-C by 11.9 and 9.7 percent points, respectively. Finally, we further improve the current state of the art model on IN-C- the EfficientNet-L12 Noisy Student model and report a new state-of-the result of 22% mCE (which corresponds to topl error of 17.1%) on this benchmark with test-time adaptation (compared to 28% mCE without adaptation).
Self-learning not limited to the distribution shifts in IN-C like compression artefacts or blur. On IN-R, a dataset with renditions, self-lcarning improves both the vanilla ResNet50 and the EfficientNet-L.2 model, the latter of which improves from 23.5% to a new state of the art of 17.4% top-1 error. For vanilla ResNet50, we improve the top-1 error from 63.8% (Hendrycks et al., 2020a) to 54.1%. On IN-A, adapting the EfficientNet-L22 model using self-learning decreases the top-1 error from 16.5% (Xie et al., 2020a) to 14.8% top-1 error, again constituting new state of the art with test-time adaptation on this dataset. Self-learning can also be used in an online adaptation setting, where the model inualny adapts to new samples on IN-C in Fig. 7(i) or IN-R Fig. 7(ii, Appendix C.10.
Adapting a ResNet50 on IN-A with RPL increases the error from 0% 0.13% (chance level: 0.1%). Thus, unadapted Re has 0% accuracy IN-A by design and this error "is increased" to chance-level with
8

---

Published in Transactions on Machine Learning Research (11/2022)
self-learning. Since all labels on ImageNet-A are wrong by design, predicting wrong labels pseudo-labels does not lead to improvements beyond restoring chance-level performance.
The finding that self-learning can be effective across model architectures has also been made by Wang et al. (2021) who show improved adaptation performance on CIFAR100-C for architectures based on self-attention (Zhao et al., 2020) and equilibrium solving (Bai et al., 2020), and also by Mummadi et al. (2021) who showed adaptation results for TENT and TENT+ which combines entropy minimization with a diversity regularizer for DenseNet121 (Huang et al., 2017), MobileNetV2 (Sandler et al., 2018), ResNeXt50 (Xie et al., 2017), and a robust model trained with DAug+ +MM on IN-C and IN-R.
The improvements of self-learning are very stable: In Table 30, Appendix C.9, we show the averaged results across three different seeds for a ResNet50 model adapted with ENT and RPL. The unbiased sid is roughly two orders of magnitude lower than any improvement we report in our paper, showcasing the robustness of our results to random initialization.
Self-learning improves robustified and domain adapted models on small-scale datasets (Table 2). We test common domain adaptation techniques like DANN (Ganin et al., 2016) and UDA-SS (Sun et al., 2019), and show that self-learning is effective at further tuning such models to the target domain. We suggest to view unsupervised source/targes domain adaptation as a step comparable to pre-training under corruptions, rather than an adaptation technique specifically tuned to the target set- set-indeed, we can achieve error rates using, e.g., DANN- + target adaptation previously only possible with source/target based pseudo-labeling, across different common domain adaptation benchmarks.
For the UDA-SS experiments, we additionally trained a vanilla version with the same architecture using the widely used training code for CIFARIO at https://githut: com/ stars), and find that the vanilla trained model performs better both with and without adaptation compared to the UDA-SS model. We think that the UDA-SS model would need hyperparameter tuning; we did not perform any tuning for this model, especially because the authors provided scripts with hyperparameters they found to be optimal for different setups. In addition, the clean accuracy of the vanilla model on CIFARIO (96.5%) is much higher than the average clean accuracy of the UDA-SS model (82.6%), which may explain or imply generally higher robustness under distribution shift (Miller et al., 2021). The finding that self-learning is more effective in the vanilla model compared to the UDA-SS model points towards the hypothesis that the network weights of the vanilla model trained on the source distribution are sufficiently general and can be tuned successfully using only the affine BN statistics, while the weights of the UDA-SS model are already co-adapted to both the source and the target distribution, and thus, self-learning is less effective.
(1.9k forks, 4.8k
Self-learning also decreases the error on CIFAR10-C of the Wide ResNet model trained with AugMix (AM, Hendrycks et al., 2020b) and reaches a new state of the art on CIFARIO-C of 8.5% topl error with test-time adaptation.
Self-learning also improves large pre-trained models (Table 3). Unlike BatchNorm adaptation (Schneider et al., 2020), we show that self-lcarning transfers well to models pre-trained on a large amount of unlabeled data: self-learning decreases the mCE on IN-C of the ResNe trained on 3.5 billion weakly labeled samples (IG-3.5B, Mahajan et al., 2018) from 51.7% to 40.9%.
Self-lcarning outperforms other test-time adaptation techniques on IN-C (Tables 4 and 5). The main point of our paper is showing that sell-learning is effective across datasets, model sizes and pretraining methods. Here, we analyze whether our simple techniques can compete with other state the-rrrrttttetttt adaptation methods. Overall, we find that self-lcarning outperforms several statc-of-tho-art techniques, but underperforms in some cases, especially when self-learning is combined with other techniques.
By rigorous and fair model selection, we are able to improve upon TENT (Wang et al., 2021), and find that RPL performs better than entropy minimization on IN-C. We also compare to BN adaptation (Schneider et al., 2020), and find that 1) self-learning further improves the performance upon BN adapt, and 2) self-learning improves performance of model pretrained on large amount of data (Mahajan et al., 2018) (Table 3) which is a setting where BN adapt failed.
9

---

Published in Transactions on Machine Learning Research (11/2022)
Table 3: Unlike batch norm adaptation, self-learning adapts large-scale models trained on external data. mCE, test [%] ('x
w/o adapt
BN adapt
RPI.
ResNeXt101 vanilla
66.6
56.8
43.2
ResNeXt101 IG-3.5B
51.7
51.8
40.9
Table 4: Self-learning oulperforms other rest-lime-adaplation techniques on IN-C.
mCE [%] on IN-C test ('x w/o adapt BN Adapt
TENT (ATA((ifeoong) RPL. ENT
ResNet50
76.7
62.2
53.5
51.2
50.5 51.6
Table 5: Self-learning can further be improved when combining with other techniques.
literature results
our results
w/o adapt w/ adapt w/o adapt w/ adapt (RPL) w/ adapt (ENT)
topl error % on IN-C test, sev. 5(%)
TTT, ResNet18 (Sun et al., 2020)
86.6
66.3
85.4
61.9
62.8
SLR, ResNet50 (Mummadi et al., 2021)
82.0
(46.9)
82.0
54.6
54.7
topl error [%] on CIFARIO-C ( (N)
MT3,
(Bartler et al., 2022)
35.7
24.4
18.6
18.4
18.0
TTT+++, ResNet50 (Liu et al., 2021)
29.1
9.8
24.9
14.6
12.4
BUFR, ResNet18 (Eastwood el al., 2022)
42.4
10.6
25.5
13.4
12.9
Table 6: Vision Transformers can be adapted with self-learning.
w/o adapt
w/ adapt w/ adapt w/ adapt w/ adapt
mCE IN-C test [%] (N)
affine layers bottleneck layers lin. layers all weights
ViT-S/16
62.3
51.8
46.8
45.2
43.5
ENT and RPL outperform the recently published EATA method (Niu et al., 2022). In EATA, high entropy samples are excluded from optimization, and a regularization term is added to prevent the model from forgetting the source distribution. EATA requires tuning two additional parameters: the threshold for high entropy samples to be discarded and the regularization trade-off parameter 3.
RPL, ENT and simple hard PL putperfrr TTT (Sun et al., 2020); in particular, note that TTT requires special loss function at training time, while our approach is agnostic to the pro-training phase. A detailed comparison to TTT is included in Appendix C.6. Mummadi et al. (2021) (SLR) is unpublished work and performs better than ENT and RPL on the highest severity of IN-C. SLR is an extension of entropy minimization where the entropy minimization loss replaced with a version to ensure non-vanishing gradients of high confidence samples, as well as a diversity regularizer; addition, a trainable module prepended to the network to partially undo the distribution shift. Mummadi et al. (2021) introduce the hyperparameters 8 as the trade-off parameter between their two losses, well as as the momentum in their diversity regularization term. The success of SLR over ENT and RPL shows the promise of extending self-learning methods by additional objectives, and corroborates our findings on the effectiveness of self-learning.
We also compare our approach to Meta Test-Time Training (MT3, Bartler et al., 2022), which combines meta-learning, self-supervision and test-time training for test-time adaptation. We find that both ENT and RPL perform better than MT3: using the same architecture as Bartler et al. (2022), our best error on CIFARIO-C is 18.0% compared to their best result of 24.4%. When exchanging GroupNorm layers (Wu He, 2018) for BN layers, the error further reduces to 13.1% (Table 11). We thus find that self-learning is more effective when adapting affine BN parameters instead of GN layers, which is consistent with the findings in Schneider et al. (2020). We included a detailed comparison to Bartler et al. (2022) in Appendix C.7.
TTT+++ (Liu et al., 2021) oulperforms both ENT and RPL on CIFAR1O-C. Since Liu et al. (2021) do not report results on IN-C, it is impossible to judge whether their gains would generalize, although they do report much better results compared to TENT on Visda-C, so TTT+++ might also be effective on IN-C. Similar to TTT, TTT+++ requires special loss function during pretraining and thus, cannot be used as an out-of-the-box adaptation technique top of any pretrained checkpoint.
10

---

Published in Transactions on Machine Learning Research (11/2022)
Table 8: RPL performs best without a threshold.
Table 7: RPL (ENT) performs better on IN-C (CIFARIO-C).
Uhreshold
0.0
0.5
0.9
mCE, IN-C dev
err, C10-C
mCE on IN-C dev [%]]
ResNet50
ResNeXt-101
RiNett-T22
WRN-40
no adapt
69.5
ENT
50.0 + 0.04
43.0
22.2
8.5
soft PL
60.1
RPL
48.9 + 0.02
42.0
21.3
9.0
hard PL
53.8
51.9
52.4
RPL
49.7 49.9
51.8
Bottom-Up Fealure Restoration (BUFR; Eastwood et al., 2022) outperforms self-learning on CIFAR1O-C. The authors note that BUFR applicable to dataset shifts with measurement shifts which stem from measurement artefacts, but not applicable to more complicated shifts where learning new features would be necessary.
Self-supervised methods based on self-learning allow out-of-the-box test-time adaptation (Table 6). The recently published DINO method (Caron et al., 2021) is another variant of self-supervised learning that has proven to be effective for unsupervised representation lcarning. At the core, the method uses soft pseudo-labeling. Here, we test whether a model trained with DINO on the source dataset can be test-time adapted on IN-C using DINO to further improve ont-of-disttribtiin performance. We highlight that we specifically test the self-supervised DINO objective for its practicality as a test-time adaptation method, and did not switch the DINO objective for ENT or RPL to do test-time adaptation. Since the used model is a vision transformer model, test different choices of adaptation parameters and find considerable performance improvements in all cases, yiclding an mCE of 43.5% mCE at a parameter count comparable to ResNet50 model. For adapting the affine layers, we follow Houlsby et al. (2019).
6 Understanding test-time adaptation with self-learning
In the following section, we show ablations and interesting insights of using self-learning for test-time adaptation. If not specified otherwise, all ablations are run on the hold-out corruptions of IN-C (our dev set) with a vanilla ResNet50.
Robust pseudo-labeling outperforms entropy minimization on large-scale datasets while the reverse is true small-scale datasets (Table 7). We find that robust pscudo-labeling consistently improves over entropy minimization IN-C, while entropy minimization performs better smaller scale data (CIFAR1O, STL10, MNIST). The finding highlights the importance of testing both algorithms on new datasets. The improvement is typically on the order of one percent point.
Robust pseudo-labeling allows usage of the full dataset without threshold (Table 8). Classical hard labeling needs confidence threshold (T) for best performance, thereby reducing the dataset size, while best performance for RPL is reached for full dataset training with threshold T of 0.0. We corroborate the results of Wang et al. (2021) who showed that TENT outperforms standard hard labeling with a threshold on the highest severity of CTFAR10-C and CIFAR100-C. We show that this result transfers to IN-C, for a variety of thresholds and pseudo-labeling variants.
Short update intervals are crucial for fast adaptation (Table 9). Having established that RPL gen- erally performs better than soft- and hard-labeling, we vary the update interval for the teacher. We find that instant updates are most effective. In entropy minimization, the update interval is instant per default.
Adaptation of only affine layers is important in CNNs (Table 10). On IN-C, adapting only the affine parameters after the normalization layers (i..., the rescaling and shift parameters 8 and 7) works better on ResNet50 architecture than adapting all parameters or only the last layer. We indicate the number of adapted parameters in brackets. Note that for Vision Transformers, full model adaptation works better than affine adaptation (see Table 6). We also noticed that on convolutional models with a smaller parameter count like ResNet18, full model adaptation is possible. Wang et al. (2021) also used the affine BN parameters for test-time adaptation with TENT, and report that last layer optimization can improve performance but degrades with further optimization. They suggest that full model optimization does not improve performance
11

---

Published in Transactions on Machine Learning Research (11/2022)
Table RPL performs best with instantaneous updates (ResNet50).
Update interval (RPT. w/o adapt
none epoch instant
mCE, IN-C dev [%)
69.5
54.0
49.7
49.2
Table 10: RPL performs best when affine BN parameters adapted (ResNe(50).
Mechanism
w/o adapt
last layer
full model affine
mCE, IN-C dev [%]
69.5
60.2
51.5
48.9
adapted parameters
0
2M
22.6M
5.3k
Table 11: Self-learning works better in models with BN layers compared to models with GN layers, although
un-adapted models with GN are more stable under distribution shift compared to models with BN layers.
mumber of w/o adapt w/ adapt (A) w/ adapt (A) RPT.
top1 error [%] on CIFAR10-C (N)
parameters
ENT
WRN-2661-BN (Zagoruyko & Komodakis, 2016)
1.5 x 10
25.8
15.1 (-10.7)
13.1 (-12.7)
WRN-2661-GN (Bartler et al., 2022)
1.5 x 10
18.6
18.4 (-0.2)
18.0 (-0.6)
mCE [%] on IN-C test (%)
ResNet50 Big BigTrrnnserr (Kolesnikov et al., 2020)
2.6 x 10"
55.0
54.4 (-0.6) 56.4 (+1.4)
Table 12: Our expanded analysis confirms hyperparameter choices from the literature.
Method
short updates adapt affine params use BN instead of GN adapt to full dataset
TENT (Wang et al., 2021)
EATA (Niu et al., 2022)
SRL (Mummadi et al., 2021)
/
MEMO (Zhang et al., 2021)
x
/
RPL/ENT(our))
/
at all. In contrast, we find gains with full model adaptation, but stronger gains with adaptation of only affine parameters.
Affine BN parameters work better for test-time adaptation compared to GN parameters. (Table 11). Schneider al. (2020) showed that models with batch normalization layers are less robust to distribution shift compared to models with group normalization (Wu & Hc, 2018) layers. However, after adapting BN statistics, the adapted model oulperformed the non-adapled GN model. Here, we show that these results also hold for test-time adaptation when adapting a model with GN or BN layers. We show that a WideRosNet-26-1 (WRN-26-1) vanilla model with BN layers pretrained on clean CIFAR10 has a much higher error on CIFARIO-C than the same model with GN layers, but it has a much lower error after adaptation. The full results for the WRN-26-1 model can be found Appendix C.7. Further, we test the pretrained Bignrfnsfer (Kolesnikov al., 2020) models which have GN layers, and find only small improvements with RPL and no improvements with ENT. There are pretrained weights released for the Big TigTrrnseer models which have BN layers, thus, a comparison similar to the WRN-26-1 model is not possible. A more detailed discussion on our BigTransfer results as well as hyperparameter selection study can be found in Appendix E.2.
Hyperparameters obtained on corruption datasets transfer well to real world datasets. When evaluating models, we select the hyperparameters discussed above (the learning rate and the epoch used for early stopping are the most critical ones) on the dev set (full results in Appendix C.2). We note that this technique transfers well to IN-R and -A, highlighting the practical value of corruption robustness datasets for adapting models on real distribution shifts.
Learning rate and number of training epochs are important hyperparameters We tune the learning rate as well as the number training epochs for all models, except for the EfficientNet--2 model where we only train for one epoch due computational constraints. In Tables 18, 19, 20, and 21 in Appendix C.2, we show that both ENT and RPL collapse after a certain number of epochs, showing the inherent instability of pseudo-labeling. While Wang et al. (2021) trained their model only for one epoch with one learning rate,
12

---

Published in Transactions on Machine Learning Research (11/2022)
Table 13: Self-learning leads to an increased ECE compared to the unadapted model.
adaptation
IN-C full
IN-C sev 1
LN-C sev 2
IN-C sev 3
IN-C sev 4
IN-C sev 5
w/o adapt
2.3 + 0.8
2.3 - 0.3
2.1 + 0.3
2.1 + 0.3
2.2 0.4
2.9 = 1.6
RPL
10.6 7.3
6.6 0.5
7.9 1.5
8.9 2.1
11.2
18.7 = 12.6
ENT
10.6 + 7.3
6.6 1 0.5
7.9
8.9 2.1
11.2
18.7 = 12.6
rigorously perform full hyperparameter selection on hold-out set (our dev set), and use the optimal hyperparameters on all tested datasets. We believe that this kind of experimental rigor is essential in order to be able to properly compare methods.
Our analysis confirms hyperparameter choices from the literature Having searched over a broad space of hyperparameters and self-learning algorithms, we identified ENT and RPL, as the best performing variants across different model sizes, architectures and pretraining techniques. We summarize the most important hyperparameter choices in Table 12 and compare them to those used in the literature when applying self-learning for test-time adaptation. Wang et al. (2021) identified that CIFAR-C, entropy minimization outperforms hard PL, and found that affine adaptation of BN parameters works better than full model adaptation, and adapted to the full dataset since TENT does not have a threshold in contrast. to hard PL. EATA (Niu et al., 2022) and SRL (Mummadi et al., 2021) are extensions TENT, and thus, followed their hyperparameter choices. EATA does not adapt to the full datasct as they do not adapt to very similar samples or samples with high entropy values. MEMO (Zhang et al., 2021) adapts to a single sample and adapts all model weights. It would be interesting to study whether the performance of MEMO can be improved when adapting only affine BN parameters.
Pure self-learning hurts calibration, (Table 13) Eastwood et al. (2022) show that approaches based on entropy minimization and pseudo-labeling hurt calibration due to the objective of confidence maximization. We corroborate their results and report the Expected Calibration Error (ECE; Naeini et al., 2015) when adapting a vanilla ResNet50 model with RPL and ENT. We report the mean ECE (lower better) and standard deviation across corruptions (and severities) below. We observe that both methods increase ECE compared to the unadapted model. The increase in ECE is higher for more severe corruptions which can be explained by successively stronger distribution shift compared to the source dataset.
To judge the forgetting effect on the source distribution, we calculated the accuracy on clean ImageNet for our adapted ENT and RPL checkpoints. We note that success of self-learning can partially be attributed to correcting the BN statistics of the vanilla model with respect to the distribution shift. (Schneider et al., 2020). Thus, we only wish to examine the effect of fine-tuning of the affine BN parameters to the target distribution with respect to the source distribution. Thus, we again correct the mismatched statistics to the source dataset when calculating the accuracy.
Self-learning leads to slightly decreased accuracy on the source dataset (Table 14)
We report the mean and standard deviation across corruptions (and severities) below. We observe that both ENT and RPL lead to a decrease in performance on the source dataset, an effect which has also been observed by Niu et al. (2022). The decrease in performance is higher for more severe corruptions which can be explained by a increasingly stronger distribution shift compared to the source datasct. We find that the effect of forgetting is less pronounced in RPL compared to ENT.
Additional experiments and ablation studies, well as detailed results for all models and datasets can be found in Appendix C. We discuss additional proof-of-concept implementations on the WILDS benchmark (Koh et al., 2021), BigTransfer (BiT; Chen el al., 2020b) models and on self-learning based UDA models in Appendix E. On WILDS, self-learning is effective for the Camelyon17 task with a systematic shift between train, validation and test sots (cach set comprised of different hospitals), while self-lcarning fails to improve on tasks with mixed domains, such as on the RxRxi and the FMoW tasks. These results support our claim that self-learning is effective, while showing the important limitation when applied to more diverse shifts.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Table 14: Self-learning leads slightly decreased accuracy on the source dataset (clean IN).
model
top1 accuracy on TmageNet val [%]
w/o adapt
74.2
RPL adapted IN-C (avg over corruptions, sev. 1)
74.7 + 0.6
RPL adapted to IN-C (avg over corruptions, sev. 2)
73.9 + 01.
RPL adapted to IN-C (avg over corruptions, 3)
73.0 + 1.3
RPL adapted to IN-C (avg over corruptions, sev. 4)
71.8 + 1.8
RPL adapted to IN-C (avg over corruptions, sev. 5)
69.8 + 3.0
ENT adapted to IN-C (avg over corruptions, sev.
74.3 0.6
ENT adapted to IN-C (avg over corruptions, sev. 2)
73.2 + 1.1
ENT adapted to IN-C (avg over corruptions, sev. 3)
72.3 + 1.7
ENT adapted to IN-C (avg over corruptions, sev. 4)
70.7 + 2.3
ENT adapted to IN-C (avg over corruptions, sev. 5)
68.0 + 3.9
7
A simple model of stability in self-learning
We observed that different self-learning schemes optimal for small-scale large-scale datasets and varying amount of classes. We reconsider the used loss functions, and unily them into
f'() Te
F'
f (x)
((x) = - --
log
(5)
[((x): sg(((x)),,
entropy minimization
(x(x
pseudo-labeling.
where we introduced student and teacher temperature тa and as parameters in the softmax function and the stop gradient operation sg. To study the learning dynamics, we consider a linear student network f = w e gd and a linear teacher network f = w' E gd which are trained on N data points {x,} binary cross-entropy loss function defined as
with a
N
C
Σ
[(x,) = Σ (o.(x] logo,,,, w²) ac(-x w') logo,( (-x (-x[w")))
(6)
where oz(2)
1 1-2// and (2(2) 1 1--
With stop gradient, student and teacher evolve in time according to
W - w"= (w", w²), ='=a w'),
(7)
where a is the learning rate of the teacher. Without stop gradient, student. and teacher are set equal to each other (following the instant updates we found to perform empirically best), and they evolve as
w -V-V,((W), where w w' =
(8)
We restrict the theoretical analysis to the time evolution of the components of wi in direction of two data points Xę and X 1/ x w and v0" x w All other components yf with of k, are neglected to reduce the dimensionality of the equation system. It turns out that the resulting model captures the neural network dynamics quite well despite the drastic simplification of taking only two data points into account (see Figure 2 for a comparison of the model vs. self-lcarning on the CIFAR-C dataset). We obtain the dynamics:
with stop gradient:
oi x V wD (f(x*) {(xi)), Di = x7 V (f(x2) + {(x:))
oh alv vi). if = =(vi vi),
(9)
without stop gradient: in X V (e(xx) + {(xi)),
in = - -x] V (E(xx) + E(xi))
With this setup in place, derive
14

---

Published in Transactions on Machine Learning Research (11/2022)
Two points
CIFAR-C
PL
Error
Ertor
0%
CAA SBAS
10g10 Ts
50%
100%
2
2
2
-1
0
2
1
0
ENT
T: = 2To
logi0 Ts
-2
-2
-
0
0
log10
log10 T
Figure 2: For the two point model, we show accuracy, and for the CIFARIO-C simulation, we show improvement (yellow) vs. degradation (purple) over the non-adapted baseline (BAS). An important convergence criterion for pseudo-labeling (top row) and entropy minimization (bottom row) is the ratio of student and teacher temperatures; it lies at for PL, and 2T,- T; for ENT. Despite the simplicity of the two-point model, the general convergence regions transfer to CIEARIO-C.
Proposition 7.1 (Collapse in the two-point model). The student and teacher networks w. and w. trained with stop gradient do not collapse to the trivial representation Vx x w 0, 2 w = 0 if Tn тt- The network w trained without stop gradient does not collapse if T3 T2/2. Proof. see $ A.1.
We validate the proposition on a simulated two datapoint toy dataset, as well as on the CIFAR-C dataset (Figure 2). In general, the size and location of the region where collapse is observed in the simulated model also depends on the initial conditions, the learning rate and the optimization procedure. An in depth discussion, as well additional simulations are given in Appendix A.
Entropy minimization with standard lemperatures (T. = 1) and hard pseudo-labeling (Te - 0) are hence stable. The two-point learning dynamics vanish for soft pseudo-labeling with Tr, suggesting that one would have to analyze a more complex model with more data points. While this does not directly imply that the learning is unstable at this point, we empirically observe that both entropy minimization and hard labeling outperform soft-labeling in practice.
The finding aligus with empirical work: For instance, Caron et al. (2021) fixed T, and varied Tr during training, and empirically found an upper bound for т above which the training was longer stable. It also aligns with our findings suggesting that hard-labeling tends to outperform soft-labeling approaches, and soft-labeling performs best when selecting lower teacher temperatures.
In practice, the result suggests that sludent temperatures should always exceed the teacher temperalures for pseudo-labeling, and student temperutures should always ezceed half the teacher temperuture for entropy minimization, which narrows the search space for hyperparamoter optimization considcrably.
8 Adapting models on a wider range of distribution shifts reveals limitations of robustification and adaptation methods
Robustness datasets on ImageNet-scall have so far been limited to a few selected domains (image corruptions in IN-C, image renditions in IN-R, difficult images for ResNet50 classifiers in IN-A). In order to test our approach on wider range of complex distribution shifts, we re-purpose the dataset from the Visual Domain Adaptation Challenge 2019 (DomainNet; Saenko al., 2019) additional robustness benchmark.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Table 15: Self-learnigg decreases the topl error on IN-D domains with strong initial performance, but fails
improve performance on challenging domains.
domain adapt
Real
Painting w/o
Clipart w/o
Sketch w/o
Infograph w/o
Quickdraw w/o w/
ImageNet w/o
w/o
w/
w/
w/
w/
w/
model
EffNet-L2 Noisy Student
29.2
27.9
42.7
40.9
45.0
37.9
56.4
51.5
77.9
94.3
98.4
99.4
11.6
ResNet50 DAug I AM
39.2
36.5
58.7
53.4
68.4
57.0
75.2
61.3
88.1
83.2
98.2
99.1
23.3
ResNet50 vanilla
40.1
37.3
65.1
57.8
76.0
63.6
82.0
73.0
89.6
85.1
99.2
99.8
23.9
Creation of ImageNet-D The original DomainNet dataset comes with six image styles: "Clipart", "Real", "Infograph", "Painting", "Quickdraw" and "Sketch", and has 345 classes in total, out of which 164 overlap with ImageNet. We map these 164 DomainNet classes to 463 ImageNet classes, e.g., for an image from the "bird" class in DomainNet, we accept all 39 bird classes in ImageNet as valid predictions. ImageNet also has ambiguous classes, e.g., has separate classes for "cellular telephone" and "dial phone". For these cases, we accept both predictions as valid. In this sense, the mapping from DomainNet to ImageNet is a one-to-many mapping, We refer to the smaller version of DomainNet that is now compatible with ImageNNet-traine models as ImageNet-D (IN-D). The benefit of IN-D over DomainNet is this re-mapping to ImageNet classes which allows robustness researchers to easily benchmark on this dataset, without the need of re-training a model (as is common in UDA). We show example images from IN-D in Table 15. The detailed evaluation protocol on IN-D, our label-mapping procedure from DomainNet to ImageNet along with justifications for our design choices and additional analysis are outlined in Appendix D.
The most similar robustuess dataset to IN-D is LN-R which contains renditions of ImageNet classes, such as art, cartoons, deviantart, graffiti, embroidery, graphics and others. The benefit of LN-D over LN-R is that in IN-D, the images are separated according to the domain allowing for studying of systematic domain shifts, while in IN-R, the different domains are not distinguished. ImageNet-Sketch (Wang et al., 2019) is a dataset. similar to the "Sketch" domain of IN-D.
More robust models perform better on IN-D. To test whether self-learning is helpful for more complex distribution shifts, we adapt a vanilla ResNet50, several robust IN-C models and the EfficientNNet-.22 Noisy Student model on IN-D. We use the same hyperparameters we obtained on IN-C dev for all our IN-D experiments". We show our main results in Table 15. Comparing the performance of the vanilla ResNet50 model to its robust AA variant, we find that the DAug+AM model performs better all domains, with the most significant gains on the "Clipart", "Painting" and "Skotch" domains. We show detailed results for all domains and all tested models in Appendix D.3, along with results on IN-C and IN-R for comparison. We find that the best performing models on IN-D are also the strongest ones on IN-C and IN-R which indicates good generalization capabilities of the techniques combined for these models, given the large differences between the three considered datasots. The Spearman's 's rank correlation cocfficient between IN-C and IN-D (averaged over all domains) is 0.54, and 0.73 between TN-R and IN-D. Thus, the errors on IN-R are strongly correlated to errors on IN-D which can be explained by the similarity of IN-D and IN-R. We show Spearman's 's rank correlation cofficients for the individual domains versus IN-C/IN-R in Fig. 9 in Appendix D.5, and find correlation values above 0.8 between IN-R and IN-D for all domains except for the "Real" domain where the coefficient is almost zero. Further, we find that even the best models perform 20 to 30 percentage points worse on IN-D compared to their performance on IN-C or IN-R, indicating that LN-D might be a more challenging benchmark.
All models struggle with some domains of IN-D. The EfficientNet- Noisy Student model obtains the best results on most domains. However, note that the overall error rates are surprisingly high compared to the model's 's strong performance on the other considered datasets (IN-A: 14.8% top-1 error, IN-R: 17.4% top-1
IIn regards to hyperparameter selection, we performed a control experiment where selected hyperparameters with leave-one-nut cross validation -hhi selection scheme actually performed worse than TN-C parameter selection (see Appendix D.2).
16

---

Published in Transactions on Machine Learning Research (11/2022)
error, IN-C: 22.0% mCE). Even the "Real" domain closest to clean ImageNet where the EfficientNet-L22 model has a top-1 error of 11 .66% the model only reaches a top-1 error of 29.2%. Self-learning decreases the top-1 error on all domains except for "Infograph" and "Quickdraw". We note that both domains have very high error rates from the beginning and thus hypothesize that the produced pscudo-labels are of low quality.
Error analysis on IN-D. We investigate the errors a ResNet50 model makes on IN-D by analyzing the most frequently predicted classes for different domains to reveal systematic errors indicative of the encountered distribution shifts. We find most errors interpretable: the classifier assigns the label "comic book" to images from the "Clipart" or "Painting" domains, "website" to images from the "Infograph" domain, and "envelope" to images from the "Sketch" " domain. Thus, the classifier predicts the domain rather than the class. We find no systematic errors on the "Real" domain which is expected since this domain should be similar to ImageNet. Detailed results on the most frequently predicted classes for different domains can be found in Fig. 9, Appendix D.5.
IN-D should be used an additional robustness benchmark. While the error rates on IN-C, -R and -A are at a well-acceptable level for our largest Efficient.Net-L2 model after adaptation, IN-D performance is consistently worse for all models. We propose to move from isolated benchmark settings like IN-R (single domain) to benchmarks more common in domain adaptation (like DomainNet) and make IN-D publicly available as an easy to use dataset for this purpose.
9 Best practices and evaluation in test-time adaptation
Based on our results as well as our discussion on previous work, we arrive at several proposals on how test-time adaptation should be evaluated in future work to ensure scientific rigor:
Cross- validatio:: We propose using the hold-out set of IN-C for model selection of all relevant hyperparameters, and then using these hyperparameters for testing on different datasets.
Comparison to simple baselines: With proper hyperparameter tuning, very simple baselines can perform on par with sophisticated approaches. This insight is also discussed by Guirajani & Lopez-Paz (2021) for the setting of domain generalization and by Rusak et al. (2020) for robustness to common corruptions.
3. Using more robust models: Test-time adaptation can further improve upon robust models, which were pre-trained with more data with UDA, using protocols to increase robustness. A test-lime adaptation method will be much more relevant to practitioners if it can improve upon the most robust model they can find for their task.
4. Important hyperparameters:: We identify several important hyperparameters which affect the final performance in a crucial way, and thus, should be tuned to ensure fair comparisons:
Number of adaptation epochs and learning rate: The final performance crucially depends both of these parameters for all models and all methods that we have studied.
Adaptation parameters While adaptation of affine batch normalization parameters works
best for adaptation of CNNs, full adaptation performs best for ViTs. Therefore, it is important to benchmark test-time adaptation for different model architectures and adaptation parameters.
10 Conclusion
We evaluated and analysed how self-learning, essential component in many unsupervised domain adaptation and self-supervised pre-training techniques, can be applied for adaptation to both small and large-scale image recognition problems common in robustness rescarch. We demonstrated new statc-of-the-art adaptation results with the EfficientNet-L22 model on the benchmarks ImageNet-C, -R, and -A, and introduced a new benchmark dataset (ImageNet-D) which remains challenging even after adaptation. Our theoretical analysis shows the influence of the temperature parameter the self-learning loss function on the training stability and provides guidelines how choose suitable value. Based on our extensive experiments, we formulate best practices for future research on test-time adaptation. Across the large diversity of (systematic) distribution
17

---

Published in Transactions on Machine Learning Research (11/2022)
shifts, architectures and pre-training methods we tested in this paper, we found that self-learning almost universally improved test-time performance. An important limitation of current self-learning methods is the observed instability over longer adaptation time frames. While we mitigate this issue through model selection (and show its robustness across synthetic and natural distribution shifts), this might not universally hold across distribution shifts encountered in practice. Concurrent work, e.g. Niu et al. (2022) tackle this problem through modifications of self-lcarning algorithms, and we think this direction will be important to continue to explore in future work. That being said, we hope that our results encourage both researchers and practitioners to experiment with self-learning if their data distribution shifts.
Reproducibility Statement We attempted to make our work as reprodueible as possible: We mostly used pre-trained models which are publicly available and we denoted the URL addresses of all used checkpoints; for the checkpoints that were necessary to retrain, we report the Github directories with the source code and used an official or verified reference implementation when available. We report all used hyperparameters in the Appendix and released the code.
Software and Data Code for reproducing results of this paper is available at https://github. com/ bethgelab/rooustness.
Author Contributions StS and ER conceived the project with suggestions from MB and WB. ER ran initial experiments. StS performed large scale experiments on ImageNet-C,RR.A with support from PG, ER and WB. ER performed domain adaptation and CIFAR experiments with suggestions from Sts. ER implemented all baselines that have been included as comparisons in the manuscript. GP ran DINO adaptation, WILDS, and self-cnsembling experiments with suggestions from ER & StS. ER developed the ImageNet-D dataset with suggestions from StS and WB, and performed the large scale experiments on ImageNettDD. LE and SiS developed the theoretical model, ER contributed the CIFAR-C experiments. WB, ER and StS wrote the initial manuscript with help from all authors. ER wrote an extensive related work section. ER reviewed and edited the manuscript in the course of the auvioerre discussion period.
Acknowledgements We thank Roland S. Zimmermann, Yi Zhu, Raghavan Manmatha, Matthias Kümmerer, Matthias Tangemann, Bernhard Schölkopf, Justin Gilmer, Shubham Krishna, Julian Bitterwolf, Berkay Kicanaoglu and Mohammadreza Zolfaghari for helpful discussions on pseudo labeling and feedback on our paper draft. We thank Yasser Jadidi and Alex Smola for support in the setup of our compute infrastructure. We thank the anonymous reviewers from TMLR and our Action Editor for their constructive feedback.
We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting E.R. and St.S.; St.S. acknowledges his membership in the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program. This work was supported by the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 011S18039A), by the Deutsche Forschungseenetnschaft (DFG) in the priority program 1835 under grant (1122215-22 and in the SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms (TP3), project number: 276693517. WB acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 1-1 and via the Open Philantropy Foundation funded by the Good Ventures Foundation. WB is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645.
References
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({ (ODD)) 16), pp. 265-283, 2016. Cited on p. 48.
Azimi, F., Palacio, S., Rauc, F., Hees, J., Bertinctto, L., and Dengel, A. Self-supervised test-time adaptation on video data. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3439 3448, 2022. Cited on p. 4.
Bai, S., Koltun, V., and Kolter, J. Z. Multiscale deep equilibrium models. Advonces in Neural Information Processing Systems, 33:5238- 52500, 2020. Cited on p. 9.
18

---

Published in Transactions on Machine Learning Research (11/2022)
Bartler, A., Bühler, A., Wiewel, F., Döbler, M., and Yang, B. Mt3: Meta test-time training for self-supervised test-time adaption. In International Conference on Arlificial Intelligence and Statistics, pp. 3080-3000 -0900. PMLR, 2022. Cited on pp. 3, 4, 8, 10, 12, 36, and 37
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q. Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. ArXiv preprint, (bs//992.00660 2019. URL https://aara www..rrg//aa... ataaaabs/111// 06680. Cited on p. 2.
Berthelot, D., Roelofs, R.,, Sohn, K., Carlini, N., and Kurakin, A. Adamatch: A unified approach to semi-supervised learning and domain adaptation, 2021. Cited on p. 4.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T.,, Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCancllish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Haddelll R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https: ptttp::/proceedd: /ttp///eroceennssse ietpeeeddnnn..uuuippp..... nwwrine..ccoooo cctpppprrccppppr/////peppappcsrt 2020/hash/ /ttpaaaa/...67700000000001100110
Abstractbbttat........ html. Cited p.
Cai, T., Gao, R., Lee, J. D., and Lei, Q. A theory of label propagation for subpopulation shift. or preprint arXiv:2102.11 11203, 2021. Cited on p. 5.
Caron, M., Touvron, H., Misra, 1,, Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. ArXiv preprint, abs/2104.14294, 2021. URL https: 14294. Cited on pp. 3, 5, 7, 11, 15, and 31
Chen, D., Wang, D., Darrell, T., and Ebrahimi, S. Contrastive test-time adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 295-305, 2022. Cited on pp. 4 and 6
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597 1607. PMLR, 2020a. Cited on p. 4.
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E. Big self-supervised models are strong semii- supeisupe learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (cds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 62, 2020, virtual, 2020b. URL https: httpss/:://pr http///eeceeenns.... www.:oosss5--ttttttttteee html. Cited on p. 13.
neurips. http:///e//pooooaaoaoaaccaaaaaaa
Chollet, F. Xception: Deep learning with depthwise separable convolutions. In 2017 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. URL htaps://doi.oor
1800-1807. IEEE Computer Society, 2017. doi: 10. 1109/CVPR. 2017. 195. Cited on pp. and 30
Coates, A., Ng, A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011. Cited on p. 6.
Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., and Hein, M. Robustbench: standardized adversarial robustness benchmark. ArXiv preprinl, abs abs/2000.09660,, 2020. URL https://arxivv www//bs/////eeeeeeeeee/eeee att///aas/22011 2010000000000 09670. Cited on p. 31.
Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Li, F. Imagenet: A large-scale hierarchical image database. In 2009 IEFE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248 255. IEEE Computer Society, 2009. doi: 1109 /t/0//VPR... 2009. 5206848. Cited on p. 2.
10.100//CVPR.2000.5220 52066488 URL fttps:///oi..rgg
19

---

Published in Transactions on Machine Learning Research (11/2022)
Deng, L. The mnist database of handwritten digit images for machine learning rescarch. IEEE Signal Processing Magazine, 29())111,-14,, 2012. Cited on 6.
Dodge, S. F. and Karam, L. J. study and comparison of human and deep learning recognition performance under visual distortions. In International Conference on Computer Communications and Networks, 1CCCN 2017, 2017. Cited on p. 2.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Golly, S., al. An image worth 16x16 words: Transformers for image recognition scale. In Proceedings of the 36th International Conference on Machine Learning, 2021. Cited on pp. 3 and
Eastwood, C., Mason, I,, Williams, C. K., and Schölkopf, B. Source-free adaptation to measurement shift via bottom-up up feature restoration. In International Conference on Learning Representations (ICLR), 2022. Cited on pp. 4, 10, 11, and 13
Farahani, A., Voghoci, S., Rasheed, K., and Arabmia, H. R. A bricf review of domain adaptation. Advances in dala science and information engineering, pp.
2021. Cited on p. 2.
French, G., Mackiewiez, M., and Fisher, M. H. Self-ensembling for visual domain adaptation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.nee .nc,, 2018. URL attps://openrevie..rt nett/ forum?id derummid=rppoTaxAA Cited on pp. 5, 46, and 48
Galstyan, A. and Cohen, P. R. Empirical comparison of hard and soft label propagation for relational classification. In 1 17th international conference on Inductive logic programming, 2007. Cited p. 5.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. adversaid training of neural networks. The journal of machine learning research, 17(1): 2096 2096-2030, 2016. Cited on pp. 2, 6, 7, 8, and 9
Geirhos, R., Temme, C. R. M., Rauber, J., Schütt, H. H., Bethge, M., and Wichmann, F. A. Generali- sation in humans and deep neural networks. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Bisa-Bianch,, N., and Garnett, R. (eds.). Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréol, Canada, pp. 7549-7561, 2018. URL https: https://proceeeings neurips ccc/pppr//018//aass html. Cited on p. 2.
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W. Tmagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In 7th Inter- national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.nel! 2019. URL
www./eeetfuunnoee http://wwww..ee...ooo/oi//iiggge Bid-Bygh9j0KKX. Cited on pp. and 40
Ghosh, A., Kumar, H., and Sastry, P. S. Robust loss functions under label noise for deep neural networks. In Singh, S. P. and Markovitch, S. (eds.), Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 1919 1925. AAAI Press, 2017. URL http://aaat: http://aaa...g////o///// www.ocss////oooooooooooooo.. htt//ocs/index. index.ndxx/// pttp:/AA/AAAA///ppppr///////////
Cited on p. 6.
Gidaris, S., Singh, P., and Komodakis, N. Unsupervised representation learning by predicting image rotations.
arXiv preprint Xiv: 1803. 0777277 2018. Cited on p. 3.
Grandvalet, Y. and Bengio, Y. Semi-supervised learning by entropy minimization. In Advances in Newral Information Processing Systems 17 (Neural Information Processing Systems, NIPS 2004, December 19-18, 2004, Vancowver, British Columbia, Canadaj, pp. 529 536, 2004. URL https: patee///pocceedngss ietpeeediingg..uuipp...... httppp//000044aaa////22200566a12
html. Citod on p.
Gulrajani, and Lopez-Paz, D. In search of lost domain generalization, In Proceedings of the 36th International Conference Machine Learning, 2021. Cited pp. and 17
20

---

Published in Transactions on Machine Learning Research (11/2022)
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Patlern Recognation, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770 778. IEEE Computer Society, 2016a. doi: 10.1109/CVPR.2016.900 URL https: //doi.org/10.. 1109/CVPR. 2016.90 90. Cited on p. 2.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-90, 2016, pp. 770-778. TEFE Computer Society, 2016b. doi: 10.1109/CVPR.2016.009 URL. https: //doi.org/10 10. 1109/CVPR. 2016. 90. Cited on pp. 6, 7, and 31
Hendrycks, D. and Dietterich, T. G. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview. net, 2019. URL https: http::///ppeeee..neeeeeeeeee net www..neteoooooiiidtttiiioo.. Cited on pp. 4, 6, 7, and 39
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. ArXiv 2019. URL https://arxxe wwrrg/ii/rr.oooooooooooooo abo/ 07174. Cited on p. 6.
preprint, abs/1990.0771
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. http:::///arai.orr/bb///006
ArXiv preprint, 7, 30, 31, and 40
2020a. URL pettpa//arxior wwtp:////00.........mm/// 112000106..1.....1. Cited on pp. 2, 3, 6,
Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. Augmix: A simple data processing method to improve robustness and uncertainty. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview net, 2020b. URL /ttf/orormniddmmmmearr
attps://openreviie
Cited on pp. 7, 8, 9, 31, and 40
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. In NIPS Deep Learning Workshop, 2014. Cited on pp. 7 and 30
Hoffman, J., Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A., and Darrell, T. Cycada: Cycle-consistent adversarial domain adaplation. In International conference on machine learning, pp- 1989-1998. Pmlr, 2018. Cited on p. 2.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficien transfer learning for NLP, In Proceedings of the 36th International Conference on Machine Learning, 2019. Cited on p. 11.
Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 2261-226210 2269. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017..0 https://doi.ors wttp://rrg......1//...... 1119//119...../ 223.. Cited on pp. 7, 9, 31, and 45
URL
Toffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Bach, F. R. and Blei, D. M. (eds.), Proceedings of the 32nd International Conference Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448 456. JMLR.org, 2015. URL http://proceedii Cited on pp. 3 and 6
mlr www.mrrresss.......... www.ppews/////eme press/v37/ioffeet /iofff11....1..// html.
Iwasawa, Y. and Matsuo, Y. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Newral Information Processing Systems, 34:2427-2440, 2021. Cited on p. 4.
Kim, Y., Cho, D., Han, K., Panda, P., and Hong, S. Domain adaptation without source data. IEEE Transactions Artificial Intelligence, 2021. Cited on p. 2.
Koh, P. W., Sagawa, S., Marklund, H., Xic, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., Lee, T., David, E., Stavness, I., Guo, W., Earnshaw, B. A., Haque, S., Beery, S., Leskovec, J., Kundaje, A., Pierson, F.,, Levine, S., Finn, C., and Liang, P. WILDS: A benchmark of
21

---

Published in Transactions on Machine Learning Research (11/2022)
in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021. Cited pp. 13, 45, and 48
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgore, UK, August 23 28, 2020, Proceedings, Part V 16, pp. 491-507. Springer, 2020. Cited on pp. 12 and 46
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009. Cited on pp. 4 and 6
Kumarr, A., Ma, T., and Liang, P. Understanding self-training for gradual domain adaptation. International Conference on Machine Learning, pp. 5448--549. PMLR, 2020. Cited on p.
Kundu, J. N., Venkat, N., Babu, R. V., et al. Universal source-free domain adaptation. In Proceedings of the IEEE/CVI Conference on Computer Vision and Pallern Recognition, pp. 4544 4553, 2020. Cited on p. 2.
Lee, D.-H. Pseudo-label: The simple and efficient supervipe learning method for deep neural networks. In TCML Workshop Challenges in Representation Learning (WREPL), 2013. Cited on p. 5.
Li, R., Jiao, Qu Cao, W., Wong, H.-S., and Wu, S. Model adaptation: Unsupervised domain adaptation without source data. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. Cited on p. 2.
Liang, J., Hu, D., and Feng, J. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, 2020. Cited on pp. 2 and 6
Liu, Y., Kothari, P., van Delft, B. G., Bellot-Gurlet, B., Mordan, T., and Alahi, A. Ttt+ When does self- supervised test-time training fail or thrive? In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. Cited on pp. and 10
Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and van der Maaten, L. Exploring the limits weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. Cited on pp. 2, 7, 30, and 31
Marcel, S. and Rodriguez, Y. Torchvision the machino-vision package of torch. In ACM International Conference Multimedia, 2010. Cited on pp. 31 and 48
Merkel, D. Docker: Lightweight linux containers for consistent development and deployment. Linur J., 2014 (239), 2014. ISSN 1075-3583. Cited p. 48.
Miller, J. P., Taori, R.,, Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, L. Accuracy on the line: on the strong correlation between out-of-distributio and in-distribution generalization. In International Conference on Machine Learning, pp. 7721-7735. PMLR, 2021. Cited on p. 9.
Mummadi, C. K., Hutmacher, R., Rambach, K., Levinkov, E., Brox, T., and Metzen, J. H. Test-time adaptation to distribution shift by confidence maximization and input transformation. arXio preprint arXiv:2106.1 14999, 2021. Cited on pp. 3, 4, 6, 9, 10, 12, and
Nado, Z., Padby, S., Sculley, D., D'Amour, A., Lakshminarayanan, B., and Snoek, Evaluating prediction- time batch normalization for robustness under covariate shift. ArXio preprint, abs/2006. 10000611966,, 2020. URL https://aaxxe wrggabs//////.///// 10963. Cited on p.
Nacini, M. P., Cooper, G., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning.
In Twentt-Ninth AAAI Conference Artificial Intelligence, 2015. Cited on p. 13.
22

---

Published in Transactions on Machine Learning Research (11/2022)
Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. Efficient test-time model adaptation without forgetting. In Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2022. Cited on pp. 4, 6, 10, 12, 13, and 18
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017. Cited on p. 48.
Prabhu, V., Khare, S., Kartik, D., and Hoffman, J. Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8558-8567, 2021. Cited on p.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8766 PMLR, 2021. Cited on p. 39.
Recht, B., Roclofs, R.,, Schmidt, L.,, and Shankar, V. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389 5400. PMLR, 2019. Cited p. 39.
Rusak, E.,, Schott, L., Zimmermann, R., Bitterwolf, J., Bringmann, O., Bethge, M., and Brendel, W. Increasing the robustness of dnns against image corruptions by playing the game of noise, ArXiv preprint, abs/ (bss/200100655/ 2020. URL https:/ // aaxiv.v www.aaiw..g/a.aala.laaaaaa 2tt///2000.00... Ciled on pp. 2, 17, and 40
Saenko, K., Peng, X., Usman, B., Saito, K., and Hu, P. Visual Domain Adaplation Challenge (VisDA-2019), 2019. URL http:/ //ai./aiiiiie bu. bu..bu..eeeeeee... www.ddd/isdaaa2111.... Cited on p. 15.
Sandler, M., Howard, A. G., Zhu, M., Zhmoginov, A., and Chen, L. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 5510-4520. IEEE Computer Society, 2018, doi: URL http: httpp://openaccess thecvf com/ content_cvpr.... ctt.pppr..011/mmmm//aaaaeeeeee MobileNetV2 Inverted Lee___Rsidduu doaliduas____a daaiiduuallspaaaaeeeee 201R__211_______________11111112 _acl___ppp____ html. Cited on p. 9.
Schneider, S., Rusak, E., Eck, L., Bringmann, O., Brendel, W., and Bethge, M. Improving robustness against. common corruptions by covariate shift adaptation. In Advances in neural information processing systems, 2020. Cited on pp. 3, 4, 7, 9, 10, 12, 13, 30, 34, and 36
Shu, J., Zhao, Q., Chen, K., Xu, Z., and Meng, D. Learning adaptive loss for robust learning with noisy labels. ArXiv preprint, Abs/2002.004422 2020. URL https://arxiv.. http://aaxiv...r/bbbeeaaee www.oorg///a/oooooooe 06482. Cited on p. 6.
Shu, R., Bui, H. H., Narui, H., and Ermon, S. A DIRT-T approach to unsupervised domain adaptation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openrevier net/ forum?id=H1q-Te AW. Cited pp. 4, 5, and 47
Sohn, K., Berthelot, D., Li, C.-L., Zhang, Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., and Raffel, C. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NewrIPS, 2020.
Cited on 4.
Song, H., Kim, M., Park, D., and Lee, J.-G. Learning from noisy labels with deep neural networks: A survey. Ai preprint, Abs/2007.08199, 2020. URL
201.200777............/.. Cited on 6.
Sun, Y., Tzeng, E., Darrell, T., and Efros, A. A. Unsupervised domain adaptation through self-supervision. ArXiv preprint, abs//1909.1188 11825, 2019. URL https://arxiv. httpp/:///xxxw.o///b/baa/// orwooo//////o. abs/1909 11825. Cited on pp. 7, 8, and 9
Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A., and Hardt, M. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp. 9229-9248. PMIRR, 2020. Cited pp. 3, 4, 10, 35, and 36
23

---

Published in Transactions on Machine Learning Research (11/2022)
Tan, M. and Le, Q. V. Efficientnet: Rethinking model scaling for convolutional neural networks. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114. PMLR, 2019. URL http://prroceedings mlr. wtt.sssww...r...eeeeeeeeeeeeeeee wtwpss//7/taal///0000 html. Cited on pp. 3, 7, and 31
Tange, O. Gnu parallel the command-line power tool. ;login: The USENIX Magazine, 36(1):42 47, 2011. URL http:// www..gn..a. www.gng...gggggggge httu.org///paralallll Cited p. 48.
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribution shifts image classification. Advances in Neural Information Processing Systems, 33:18583
18599, 2020. Cited on p. 39.
Tsypkin, Y. Self-learning what is it? IEEE Transactions Aulomatic Comtrol, 13(6):608 612, 1968. doi: /09//AAC.166 1099015. Cited on pp. 3 and 5
Tzeng, E., Hoffman, J., Saenko, K., and Darrell, T. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017. Citod on p.
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Jarrod Millman, K., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E.,, Carey, C., Polat, I., Feng, Y., Moore, E. W., Vand erPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., and Contributors, S. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 -222, 2020. doi: https:/ /ttsss//doooor////// 10. 1038/00/////00 //1444555 559--19-068666 Cited p. 48.
A., Harris,
Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations (ICLR), 2021. Cited on pp. 3, 4, 5, 6, 9, 11, 12, and
Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predictive power, In Advances in Newral Information Processing Systems, pp. 10506 10518, 2019. Cited on p. 16.
Wei, C., Shen, K., Chen, Y., and Ma, T. Theoretical analysis self-training with deep networks on unlabeled data. In ICLR, 2020. Cited on p.
Wightman, R. Pytorch image models. http:////tttth....r.gra////////e 2019. Cited on pp. 46 and 48
Wu, Y. and He, K. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. Cited on pp. 10 and 12
Xie, Q., Luong, M., Hovy, E. H., and Le, Q V. Self-training with noisy student improves imagenet classification. In 2020 IEE//VE Conference on Computer Vision and Pottern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 10684-10695. IEEE, 2020a. doi: 10.1109 CCV/R420P22200000 0..42200.2020.000 2020.01070. URL https://doi.com http///i..og////// 10. http://..119//////// 1199///P44200000.... 2020 01070. Cited on pp. 2, 3, 5, 7, 8, 30, 31, and 34
Xie, S., Girshick, R. B., Dollár, P., Tu, Z., and He, K. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HT, USA, July 21-26, 2017, pp. 5987-5995. IEEE Computer Society, 2017. doi: (0.1109/CVPR.200 URL https://doi org/10. 10. 1109/CVPR. 2 2017 .634. Cited on pp. 7, 9, and 31
Xic, S. M., Kumar, A., Jones, R., Khani, F., Ma, T., and Liang, P. In-n-out: Pre-training and self-training using auxiliary information for out-of-distributtin robustness. arXiv preprint S.t::2012.045500 2020b. Cited p. 4.
24

---

Published in Transactions on Machine Learning Research (11/2022)
Zagoruyko, S. and Komodakis, N. Wide residual networks. In Wilson, R. C., Hancock, E. R.., and Smith, W. A. P. (eds.), Proceedings of the Brilish Machine Vision Conference 2016, BMVC 2016, York, UK, Seplember 19-22, 2016. BMVA Press, 2016. URL http://wwet bmva. http:///0000/ppppppp/ppprrr///// httm.......ndd...m.... html. Cited on pp. 7, 8, and 12
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning roquires rethinking generalization. In 5th International Conference on Learning Represenlations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview net,,,, 2017. URL https: Cited on p. 6.
//openreview.netee hnttfoormmiddSyygdaaa
Zhang, M., Levine, S., and Finn, C. Memo: Test time robustness via adaptation and augmentation. arXiv preprint ar Xiv:21110. 09506, 2021. Cited on pp. 4, 12, 13, and 38
Zhang, Z. and Sabuncu, M. R. Generalizod cross entropy loss for training deep neural networks with noisy labels. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neurol Information Processing Systems 31: Annual Confer- ence Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Mon- tréal, Camada, pp. 8792-8802, 2018. URL cttps://proceedings. nww..nerrii.oom cttpriisscccppppe2221111 //aah/h///h//////// 7295597cccaa205aa555510eeeee AbsteaattOtttttttttaeaaa html. Cited on p. 6.
Zhao, H., Jia, J., and Koltun, V. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10076 10006, 2020. Cited p.9. 9.
Zou, Y., Yu, Z., Liu, X., Kumar, B., and Wang, Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5982-5991, 2019. Cited on p. 4.
25

---

Published in Transactions on Machine Learning Research (11/2022)
A A two-point model of self-learning
A.1 Proof of Proposition 7.1
Learning dynamics with stop gradient. explicitly yields
Computing the stop gradient evolution defined in equation
N (-xxxx))= (ae(x, w²) w')os(-xi w²) - (x(-x; w wx)) (e]]w"))))))))))))) x;
we -
Ts
(10)
w' a(w" w')
The second equality uses the well-known derivative of the sigmoid function, 0.0(2) - (())(--2).
The equation system of 2d nonlinear, coupled ODEs for w E F and w' Id in equation 10 is analytically difficult to analyze. Instead of studying the ODEs directly, act them with the data points x k == I....,,, N,, and investigate the dynamics of the components Xg w = =
xx) (or(u()os(-v)) or(-v()o.(v() 5i - Σ xx) Ta i=1
(11)
5k alyi vke).
The learning rate of each mode yi is scaled by (x, x,) which is much larger for = k than for i * & in high-dimensional spaces. In the two-point approximation, we consider only the two (in absolute value) largest terms I for given in the sum in equation 11. Any changes that yi (t) and yī" (t)( might induce in other modes yf" are neglected, and so we are left with only four ODEs:
1
1/
||* (oelvi)o - 0, (vR))
1
(xf xi) (ore(vi),,(-1i -. -
+
To
0f
1
| 2 (o
0.( (x11)
(12)
To
+ (xTxi) (o. ( vit - ael
1/. =a vi). if aui vi)-
The fixed points of equation 12 satisfy
11 = of = V/. 2/ = 0.
(13)
For a > 0, requiring oi = of 0 implies that vf = vic and yi = v/- For T. = = Tr, the two remaining equations if = i/j vanish automatically so that there are no non-trivial two-point learning dynamics. For т2 * Tr there is a fixed point at at yi since at this point, each bracket equation vanishes individually:
1 4
az(uk.)o,(-00 vk.) a.( (((-,.)))(()e)))))
(14)
the fixed point vf = y/ = 0, w and w' are orthogonal to both x; and x; and hence classification fails, If this fixed point stable, w and w' will stay the fixed point once they have reached it, ie. the model collapses. The fixed point stable when all eigenvalues of the Jacobian J of the ODE system equation 12 st
evaluated at yi vi 0 are negative. Two eigenvalues X always negative, whereas the two other
26

---

Published in Transactions on Machine Learning Research (11/2022)
cigenvalues d are positive if T3 TC
= 1xx ?
[x] xil
[x] xi)
47.
AraTe
dra
(xg
|
|x]
4r2
4
4
Mk
=0
0
0
0
0
1
(15)
A+ + V
A3 + 32ar? (220TBB(TT/F ((To/T -1) <0,
1672
-A+ + v VA + (iTaTTBB(TTTT - 1) 0 if Tt, where
16T
A+ = S + B,
= x + X ||
+
(1|x|| - ||x||)2 + x1) x1)²
To sum up, training with stop gradient and Ti avoids collapse of the two-point model to the trivial representation U/ s.t = Vi att
= 0 since the fixed point is not stable in this parameter regime.
Learning dynamics without stop gradient Without stop gradient, we set w = w = w which leads to an additional term the gradient:
w V L
(xx(x, w)oo(-x( w) - xx(-x; w))o(xx w))
i=1
(16)
N +
(x www/ w) (log (x, w) log os (-xx(xx, w)) Xi.
=log (11+ev//ro) )([te x1/-a))=us/7.
As before, we focus on the evolution of the two components yk = w Xx and y = w Xi-
(at 1 -o((w))r(--u
úu =1x+11
1
1 _o -22 22)
(xTxx) xi)
- 0r( +
TST2
(17)
(or(w)), - + -0(()))
+ (xx xi) (a.(u)) oo (vx)) (ox) (y)))
T&Tr
There is a fixed point yk yt 0 where each bracket in equation 17 vanishes individually, _ or (wa.)))( (-.
(o.(un.)) 1) a. a- (-v,,)) (yr.)) Tx
(18)
TTT
The Jacobian of the ODE system in equation 17 and its eigenvalues evaluated at the fixed point are given by
1x. 2
dr. 1
2
(x, xi)
2 1
4r,
(x xi)
2 1
1
U4 -
1r,
47.
(19)
1
\1,2
||xx||
14 +
||x||| - 2| X 2 ||| + 4(x] x,) + +||x + +|x/||
87s
Tt.
< ||x*||+*|| +||x!|?
20 with equality if x
Hence the fixed point unstable when T 7/2 and thus the model without stop gradient does not collapse onto y/k yt 0 in this regime, concluding the proof.
27

---

Published in Transactions on Machine Learning Research (11/2022)
A.2 Simulation of the two-point model
For visualization purposes in the main paper, set w = w'= -..5, T and train the model using instant gradient updates on the dataset with points and momentum 0.9. We varied student and teacher temperatures on a log-scale with 250 points from 10-3 to 10. Qualitattvely similar results can be obtained without momentum training, at higher learning rates (most likely due to the implicit learning rate scaling introduced by the momentum term).
[1, 0] and X2 [Oo -1] using SGD with learning rate 0.1
Note that the temperature scales for observing the collapse effect depend on the learning rate, and the exact training strategy lower learning rates can empirically prevent the model from collapsing and shift the convergence region. The result in Figure 2 will hence depend on the exact choice of learning rate (which is currently not considered in our continuous time evolution theory), while the predicted region without collapse is robust to details of the optimization.
To visualize the impact of different hyperparameters, we show variants of the two point model with different. learning rates using gradient descent with (Figure 3) and without momentum (Figure 4), and with different start conditions (Figure 5), which all influence the regions where the model degrades, but not the stable regions predicted by our theory.
28

---

Published in Transactions on Machine Learning Research (11/2022)
Ir: 10
0.1
0.01
0.001
PL
2T
1og10 Ts
ENT
Error
0%
log10 Te
50%
100%
-
2
-1
1
-1
0
1-2
-1
1-2
-1 0
1-2
-1
0
log10
T
log10
log1o T
log10 T
log10 7
Figure 3: Entropy minimization (top) Training two point model with momentum 0.9 and different learning rates with initialization w w
0.5
0.1
0.01
0.001
71
logio Ts
-2
ENT
Error
0%
logi0 To
50%
2
0
2
log10 Tt
log10 T
logio Tt
log10 Tt
log10 Te
Figure Training two point model without momentum and different learning rates with initialization w w' [0.5, 0.5] Note that especially for lower learning rates, longer training would increase the size of the collapsed region.
Ir:
0.1
0.01
0.001
PL
71 T5
10 Ts
2
ENT 1
Error
0%
logao To
50%
100%
2
-1
0
1
-1
1 2 1
log T
log
log T
logi0 Tr
log T
Figure Training two point model with momentum 0.9 and different learning rates with initialization w w' 0..6
29

---

Published in Transactions on Machine Learning Research (11/2022)
B Additional information on used models
B.1 Details on all hyperparameters we tested for different models
For all models except. EeeiceentNee-L2,, we adapt the batch norm statistics to the test domains following (Schneider et al., 2020). We do not expect significant gains for combining FffiiiientNet--.22 with batch norm adaptation: demonstrated in (Schneider et al., 2020), models trained with large amounts of weakly labeled data do not seem to benefit from batch norm adaptation.
ResNet50 models (IN-C) We use vanilla ResNet50 model and compare soft- and hard-labeling against entropy minimization and robust pseudo-labeling To find optimal hyperparameters for all methods, we perform extensive evaluation and test (i) three different adaptation mechanisms (ii) several learning rates 1.0 x 10-4, 1.0 x 10-3, 1.0 x 10-2 and 5.0 x 10-2, (iii) the number of training epochs and (iv) updating the teacher after each epoch or each iteration. For all experiments, we use a batch size of 128. The hyperparameter search performed on IN-C dev. We then use the optimal hyperparameters to evaluate the methods on the IN-C test set.
ResNeXt101 models The ResNeXt101 model is considerably larger than the ResNet50 model and we therefore limit the number of ablation studies we perform for this architecture. Besides a baseline, we include a state-of-the-art robust version trained with and a version that was trained on 3.5 billion weakly labeled images (IG-3.5B, Mahajan al., 2018). We only test the two leading methods the ResNeXt101 models (ENT and RPL). We vary the learning rate in same interval as for the ResNet550 model but scale it down linearly to account for the smaller batch size of 32. We only train the affine batch normalization parameters because adapting only these parameters leads to the best results on ResNet50 and is much more resource efficient than adapting all model parameters. Again, the hyperparameter search is performed only on the development corruptions of IN-C. We then use the optimal hyperparameters to evaluate the methods on the IN-C test set.
+ (DAug+AM, Hendrycks et al., 2020a)
Efficient(et-L22 models The current state of the art on IN, IN-C, IN-R and IN-A is an Efficient.Nt-L22 trained on 300 million images from JFT-300M (Chollet, 2017; Hinton et al., 2014) using a noisy sandent-teacher protocol (Xie et al., 2020a). We adapt this model for only one epoch due to resource constraints. During the hyperparameter search, we only evaluate three corruptions on the IN-C development set* and test the learning rates 4.6 x 10-2, 4.6 10-3 4.6 10-4 and 4.6 x 10-5, We use the optimal hyperparameters to evaluate ENT and RPL on the full IN-C test set (with all severity levels).
UDA-SS models We trained the models using the scripts from the official code base at github.com yueatsprograms//ud release. We used the provided scripts for the cases: (a) source: CIFAR10, target: STL10 and (b) source: MNIST, target: MNIST-M. For the case (c) source: CIFAR10, target: CIFARIO-C, used the hyperparameters from case (a) since this case seemed to be the closest match to the new setting. We think that the baseline performance of the UDA-SS models can be further improved with hyperparameter tuning.
DANN models To train models with the DANN-method, used the Pyllorch implementation of this paper at for the case (b) source: MNIST, target: MNIST-M. For the cases (a) and (c), we used the same optimizer and trained the model for 100 epochs. We think that the baseline performance of the DANN models can be further improved with hyperparameter tuning.
1.33. The code base only provides scripts and hyperparameters
Preprocessing For IN, IN-R, IN-A and IN-D, we resize all images to 256 x 256 px and take the center 224 x 224 px crop. The IN-C images are already rescaled and cropped. We center and re-scale the color values with μNGB- = [0.485, 0.456, 0.406] and RGB = [0.229, 0.224, 0.225). For the Net-.,en we follow the
We compare the results of computing the dev set on the 1, and severities versus the 1, 2, 3, 4 and severities our ResNeXt101 model in the Supplementary material.
30

---

Published in Transactions on Machine Learning Research (11/2022)
procedure in Xie et al. (2020a) and rescale all inputs to resolution of 507 x 507 px and then center-crop them to 475 x 475 px.
B.2 Full list of used models
ImageNet scale models ImageNet trained models (ResNet50, DenseNet161, ResNeXt) are taken directly from torchvision (Marcel & Rodriguez, 2010). The model variants trained with DeepAugment and AugMix The
augmentations (Hendrycks et al., 2020b;a) are taken from https://githuub.com /ttg///t...o/o//d/de/////s//ieee weakly-supervised 10 model is taken from the PyTorch Hub. For Efficient.Net (Tan & Le, 2019), we use the PyTorrc re- re-implementation available at This is a verified re-implementation of the original work by Xie et al. (2020a). We verify the performance on ImageNet, yielding a 88. 88.23% top-1 accuracy and 98.546% top-5 accuracy which is within 0.2% points of the originally reported result (Xie et al., 2020a). On ImageNet-C,, our reproduced baseline achieves 28.9% mCE vs. 28.3% mCE originally reported by Xie et al. (2020a). As noted in the re-implementation, this offset is possible due to minor differences in the pre-processing. It is possible that our adaptation results would improve further when applied on the original codebase by Xie et al.
seaiemannngee-effiietnteeee weaa---ffiiennte--yeeeeeee pytorch.
Small scale models We train the UDA-SS models using the original code base at github.com yueatsprograms/udd release, with the hyperparameters given the provided bash scripts. For our DANN experiments, we use the PyTorch implementation at rameters in the provided bash scripts.
son/uungtion/DANNN py3. We use the hyperpa-
The following Table 16 contains all models we evaluated on various datasets with references and links to the corresponding source code.
Table 16: Model checkpoints used for our experiments.
Model
Source
(2deResNNet(8.100 10) (Croce et al., 2020)
WideResNet( (00)))))eM (Croce al., 2020)
https:/ http:///ww.....oooo/o//////////e htte/////eeteee//mee///////bbeee
ResNet50 (He al., 2016b)
https:/ /ttg///uub..o//yttteeeett wwwpech//viiiooi/treeeeeeee /troon//tttt///meer///eeea httt//ttoooooooooooooooooooo /oorechwsssooooodeellll /models
ReaNeXt101, 32 x&d (He et al., 2016b)
https:/ ctddubb.oom/pytorde /itivviii///eeeeeee/eee satee/matter/tordeee vurrhviisoo/mmmdelss /models
DeuseNet (Huang al., 2017)
https:/ /ggthuub.com/pyyoteeee
wttpee/yyyoooo///teeeee vision/tree/ /master /torchvision //ooolls
ResNeXt101, 32x (Xie et al., 2017)
https:/ mnyttorch..rg//uby.. foo.chooooooeeercchhhhc WSL. Images
Images_resnextt/ resnext/
ResNet50+ DeepAugment Augguuu AugMix (Hendrycks al., 2020a)
https:/ ///////aab..o//eeedrreeeeeeeeeee heteee/ddddy///mameett... /imagenet-r
R0sNext101 (Hendrycks et al., 2020a)
https:/ http://hh..coo///////////////mmm
ResNext 101 82x8d IG-3.5B (Mahajan al., 2018)
https: http:///ttttb.coooooooooo//eeeee Images, haaob//mattr/hubbottt
Noisy Student EfficientNeet-122 (Xic al., 2020a)
https:/ http:///tthiillooo/eeggghh/////e pytorch
VIT-S/16 (Caron et al., 2021)
31

---

Published in Transactions on Machine Learning Research (11/2022)
C Detailed and additional Results on IN-C
C.1 Definition of the mean Corruption Error (mCE)
The established performance metric on IN-C the mean Corruption Error (mCE), which is obtained by normalizing the model's top-1 errors with the top-1 errors of AlexNet across the 1=15 test corruptions and S-5 severities:
1
model
mCE((ooee) =
(20)
c=1 s=1 emr Alecet
The AlexNet errors used for normalization are shown in Table 17.
Category
Corruption
topl
Gaussian Noise
0.886428
Noise
Shot Noise
0.894468
Impulse Noise
0.922640
Defocus Blur
0.819880
Glass Blur
0.826268
Blur
Motion Blur
0.785948
Zoom Blur
0.798360
Snow
0.866816
Frost
0.826572
Weather
Fog
0.819324
Brightness
0.564592
Contrast
0.853204
Elastic Transform
0.646056
Pixelate
0.717840
Digital
JPEG Compression
0.606500
Hold-out Noise
Speckle Noise
0.845388
Hold-out Digital
Saturate
0.658248
Hold-out Blur
Ganssian Blur
0. 0.787108
Hold-out Weather
Spatter
0.717512
Table 17: AlexNet topl errors ImageNet-C
C.2 Detailed results for tuning epochs and learning rates
We tune the learning rate for all models and the number of training epochs for all models except the EfficientNet-L2. In this section, we present detailed results for tuning these hyperparameters for all considered models. The best hyperparameters that we found in this analysis, are summarized in Table 22.
Table 18: mCE % on the IN-C dev set for ENT and RPL for different numbers of training epochs when adapting the affine batch norm parameters of ResNet50 model.
Table mCE (N)) in on the IN-C dev set for different learning rates for EfficientNet- L.2 We favor q 0.8 over q 0.7 due to slightly improved robustness to changes in the learning rate in the worst case error setting.
criterion
RPL 10-3
10
10
10-2
10"4
1
epoch
60.2
60.2
60.2
60.2
Ir (4.6 x) basc
1 x 10 10-3
1 10 4
10
X 10-6
54.3
72.5
57.4
52.5
52.4
50.9
96.5
55.8
49.6
57.4
ENT
25.5
87.8
25.3
22.2
24.1
51.5
51.0
112.9
54.6
49.2
RPI
25.5
60.3
21.3
23.3
n/a
51.0
52.4
124.1
53.7
49.0
71.0
RPL
25.5
58.2
21.4
23.4
n/a
50.7
53.5
131.2
52.9
48.9
76.3
32

---

Published in Transactions on Machine Learning Research (11/2022)
Table 22: The best hyperparameters for all models that found on IN-C. For all models, we fine-tune only the affine batch normalization parameters and 0.8 for RPL. The small batchsize for the EfficientNet model is due to hardware limitations.
number of epochs
Model
Method
Learning rate
batch size
vanilla ResNet50
ENT
10-3
128
vanilla ResNet50
RPL
1 1 10-3
128
5
vanilla ResNeXt101
ENT
2.5
10-4
128
1
vanilla ResNeXt101
RPL
2.5 10
128
IG-3.5B ResNeXt101
ENT
2.5 10
4
IG-3.5B ResNeXt101
RPL
2.5 10
128
2
DAug+AM ResNeXt101
ENT
2.5 10
128
AM ResNeXt101
RPL
2.5 x
10
128
4
Fef
ENT
4.6 10-5
8
NeficientNee--.22
RPL
4.6 10
Table 20: mCE in % on IN-C dev for entropy minimiza- tion for different learning rates and training epochs for ResNeXt101. (div.=diverged)
Table 21: mCE in % on IN-C dev for robust pseudo- labeling for different learning rates and training epochs for ResNeXt101. (div.=dvverged)
ENT
Baseline
IG-3.5B
DAug+AAM
RPI. 2.5x epoch
Baseline
IG-3.5B
D
2.5
1e-4
1e-3
Ge-3
le-4
le-3
5e-3
1e-4
1e-3
Ge-3
1e-4
1e-3
Ge-3
le-4
1e-3
5e-3
1e-4 1e-3
5e-3
epoch
53.6
53.6
47.4
43.0
02.2
div.
40.9
40.4
58.6
35.4
46.4
div.
43.4
51.3
div.
45.0
39.9
43.6
35.3
35.1
79.1
44.8
118.4
div.
39.8
41.5
69.5
35.5
90.8
div.
2
42.3
63.2
div.
43.4
39.3
48.2
34.9
35.6
121.2
45.4
131.9
div.
39.3
42.6
76.1
35.5
122.5
div.
42.0
72.6
div.
42.4
39.4
52.9
34.7
40.1
133.5
46.7
div.
div.
39.1
44.2
84.3
35.6
133.8
div.
42.0
72.6
div.
42.4
39.4
52.9
34.7
40.1
133.5
C.3 Detailed results for all IN-C corruptions
We outline detailed results for all corruptions and models in Table 23. Performance across the severities in the dataset is depicted in Figure 6. All detailed results presented here are obtained by following the model selection protocol outlined in the main text.
RN50
RNx101
RNx101 IG-3.5B RNx101 +eegmii Noisy Student L2
MCE [%)
RPL
Base
40
20
2
5
3
5
5
Severity
Severity
Severity
Severity
Severity
corruption (normalized using the average AlexNet bascline error for each corruption) for ResNet50 (RN50), ResNext101 (RNx101) variants and the Noisy Student L2 model. Especially for more robust models DDeepAnnmenntAAuguee and Noisy Student L2), most gains are obtained across higher severitics and 5. For weaker models, the baseline variant (Base) additionally substantially improved for smaller corruptions.
Figure 6: Severity-wise
33

---

Published in Transactions on Machine Learning Research (11/2022)
Table 23: Detailed results for each corruption along with mean corruption error (mCE) as reported Table the main paper. We show (unnormalized) top-1 error rate averaged across 15 test corruptions along with the mean corruption error (mCE: which normalized). Hyperparameter selection for both ENT and RPI. was carried out on the dev corruptions outlined in the main text. Mismatch in baseline mCE for EfficientNet-12 can be most likely attributed to pre-processing differences between original tensorflow implementation Xie al. (2020a) and the employ. We start with slightly weaker baselines for ResNet50 and ResNext than
PyTTrrc reimplementation Schneider et al. (2020): ResNet50 and ResNext 101 results are slightly worse than previously reported results (typically 0.1% points) due to the smaller batch size of 128 and 32. Smaller batch sizes impact the quality of re-estimated batch norm statistics when computation is performed on the fly Schneider et al. (2020), which is of no concern here due to the large gains obtained by pseudo-labeling
gatua
shot
impulae
defocus
glass
motion
zoom
anco
frost
Fog
bright
contrant
olastic
pixelate
jpeg
mC
ResNet50
Baseline (Schneider et al., 2020)
62.2
Baseline (ours)
57.2
59.5
60.0
61.4
62.3
51.3
49.5
54.8
54.1
39.3
46.7
41.4
38.2
41.8
62.8
ENT
45.5
45.5
16.8
48.4
48.7
40.0
40.3
12.0
46.6
33.2
28
02.4
35.2
32.2
35.1
51.6
RPL
34.4
ResNeXt101 Baseline
Baseline (Schneider al., 2020)
56.7
Baseline (ours)
52.8 54.
54.0
55.4
56.8
46.7
46.6
48.5
49.4
36.6
25.4
42.8
37.8
32.5
36.7
56.8
ENT
40.5
39.5
11.4
43.0
34.5
35.0
28.5
24.0
33.8
30.3
27.2
30.5
44.3
RPL
39.4
38.9
39.8
10.3
41.0
33.4
33.8
34.6
38.7
28.0
23.7
31.4
29.8
26.8
30.0
13.2
ResNeXt101 IG-3.5B
Baseline (Schneider et sl., 2020)
51.6
Baseline (ours)
50.7
51.5
53.1
54.2
55.5
45.5
44.7
41.7
33.8
35.4
27.8
33.9
61.8
ENT
38.6
38.3
10.4
11.4
11.5
33.8
33
32.2
31.6
21.
19.7
26.3
27
21
27.9
10.8
RPL
ResNeXt101 DeepAug+AAugin
Baseline (Schneider et al., 2020)
38.0
Baseline (ours)
30.0
30.0
30.2
32.9
35.5
28.9
31.9
33.3
32.8
29.5
22.6
28
31.2
23.0
26.5
38.1
ENT
28.7
28.5
29.0
29.8
30.9
26.9
28.0
29.3
30.5
26.2
23
26.3
28.5
23.7
26.0
35.5
RPL
28.1
27.8
28.3
29.1
30.1
26.3
27.4
28.8
29.8
25.9
22.7
25.6
27.9
23.2
25.4
31.8
Noisy Student 1,2
Baseline (Nie et al., 2020a)
28.3
Baseline (murs)
21.6 22.0
20.5
40.5
19.8
23.2
22.8
26.9
21.0
15.2
21.
24.8
18.6
28.9
ENT
18.5 18.7
17.4
18.8
23.4
16.9
18.8
17.1
19.6
16.8
14.1
16.6
19.6
15.8
16.5
23.0
RPL
34

---

Published in Transactions on Machine Learning Research (11/2022)
C.4
Detailed results for the CIFAR1O-C and UDA adaptation
Table 24: Detailed results for each corruption along with mean error on CIFARIO-C reported in Table the main paper.
gouse
ahot
impulse
defocus
gless
motiom
zoom
kRow
frost
fog
bright
contrast
elastic
pixelate
ipeg
WRN-28-10 vanilla
Baseline
53.0
11.2
41.7
16.2
28.0
BN adapt
20.8
17.6
22.7
8.1
28.4
10.9
9.2
14.2
13.0
8.7
6.8
8.5
13.5
12.1
21.0
14.4
ENT
20.6
RPL
19.1
21.4
16.3
8.1
26.4
8.9
10.9
13.7
12.9
6.9
13.1
19.7
8.2
11.4
8.7
13.7
WRN-40-2 AM
Baseline
19.1
14.0
13.3
6.3
17.1
7.9
7.0
10.4
10.6
8.5
5.9
9.7
9.2
16.8
11.9
11.2
BN adapt
14.1
13.9
7.2
17.6
8.7
10.8
ENT
10.8
9.1
10.9
6.0
13.4
7.2
6.3
8.4
7.8
7.1
5.7
7.1
9.2
7.4
11.2
8.5
RPL
11.5
11.6
9.6
6.2
14.2
6.5
7.4
8.8
8.2
9.5
11.9
7.9
8.0
7.6
9.0
WRN-26-16 UDA-SS
Baseline
26.0
24.7
19.3
22.4
56.2
31.7
26.6
15.8
20.4
26.3
BN adapt
20.5
19.0
15.6
13.5
43.1
19.
18.3
23
21.2
16.2
12.8
14.1
20.9
16.7
23. 4
19.9
ENT
16.7
12.3
11.9
RPI.
18.1
17.1
13.2
11.9
41.5
17.3
16.1
20.4
19.1
14.5
11.8
12.7
18.8
18.1
22.6
18.2
WRN-26-16 vanilla
Baseline
50.8
46.9
39.3
15.9
44.2
20.8
18.8
14.9
17.8
4.8
13.8
20.4
19.0
25.0
10.0
24.2
adapt
7.4
9.9
6.8
ENT
16.7
18.4
13.9
5.7
23.0
6.8
7.9
10.8
9.9
5.0
10.8
17.3
6.4
9.1
6.4
11.2
RPL.
16.1
13.6
23.2
9.3
11.7
17.0
9.0
7.2
Table 25: Detailed results for the UDA methods reported in Table 2 of the main paper.
Baseline
BN adapt
RPL ENT
UDA CIFARIO-STLI0 topl error target. [%](N)
WRN-26-16 16 UDA-SS
28.7
24.6 25.0
22.9
21.8
WRN-26-16 16 DANN
25.0
24.0
23.9
UDA MNIST >MNIST-M, topl error on target (%]())
WRN-26-16 UDA-SS WRN-26-2 DANN
4.8
3.9
2.4
2.0
11.4
6.2
5.2
5.1
C.5 Ablation over the hyperparameter q for RPL
For RPL., we must. choose the hyperparameter 4. We performed an ablation study over and show results in Table 26, demonstrating that RPL is robust to the choice of q, with slight preference to higher values. Note: the initial parameter sweep for this paper, we only compared q = 0.7 and g = 0.8. Given the result Table 26, could be interesting re-run the models in Table of the main paper with 0.9, which could yield another (small) improvement in mCE.
Table 26: ImageNet-C dev set mCE in %, vanilla ResNet50, batch size 96. maximum of six adaptation epochs.
We report the bost score across a
0.5
0.6
0.7
0.8
0.9
mCE (dev)
49.5
49.3
49.2
49.2
49.1
C.6 Self-learning outperforms Test- Time Training (Sun et al., 2020)
Sun et al. (2020) use a ResNet18 for their experiments on ImageNet and only evaluate their method on severity 5 of IN-C. To enable fair comparison, we trained ResNet18 with both hard labeling and RPL and compare the efficacy of both methods to Test-Time Training in Table 27. For both hard labeling and RPT.
35

---

Published in Transactions on Machine Learning Research (11/2022)
use the hyperparameters we found for the vanilla. ResNet50 model and thus, expect even better results for hyperparameters tuned on the vanilla R8sNet18 model and following our general hyperparameter search protocol.
While all methods (self-learning and TTT) improve the performance over a simple vanilla ResNet18, we note that even the very simple baseline using hard labeling already outperfoms Test-Time Training; further gains are possible with RPL. The result highlights the importance of simple baselines (like self-learning) when proposing new domain adaptation schemes. It is likely that many established DA techniques more complex than the basic self-learning techniques considered in this work will even further improve over TTT and other adaptation approaches developed exclusively in robustnoss settings.
We report belter accuracy numbers achieved with self-learning compared to online TTT, but note that they adapt to a single test sample while we make use of batches of data. Due to the single-image approach, they utilize GN instcad of BN, and this may explain the performance gap a cortain degree as we show that BN is more effective for lest-time adaptation compared to GN, even though the un-adapted BN models are less robust compared to the un-adapted GN models, also noted by Sun et al. (2020). Further, Sun et al. (2020) optimize all model parameters while we only optimize the affine BN parameters which works better.
Table 27: Comparison of hard-pseudo labeling and robust pseudo-labeling to Test-Time Training Sun et al. (2020): Top-1 error for a ResNet18 and severity 5 for all corruptions. Simple hard outperforms TTT, robust pscudo labeling over multiple epochs yields additional gains.
already
gAuse
whot
impulee
defocus
glass
motion
200m
ShOw
Frost
Rog
bright
contrast
elastic
pixelate
ipeg
Avg
vanilla ResNet18
98.8
98.2
99.0
88.6
91.3
88.8
82.4
80.1
83.5
85.7
48.7
96.6
83.2
76.9
70.4
85.4
Test-Time Training
73.7
71.4
73.1
76.3
93.4
71.3
66.6
64.4
81.3
52.4
41.7
64.7
55.7
52.2
55.7
66.3
hard PT., (1 epoch)
73.2
70.8
73.6
76.5
75.6
63.9
56.1
59.0
65.9
48.4
39.7
85.2
50.4
47.0
51.5
62.5
ENT(I epoch)
72.8
69.8
73.2
77.2
75.7
63.1
55.5
58.0
68.1
48.0
39.8
92.7
49.6
46.4
51.3
62.8
RPL (4 epochs)
71.3
68.3
71.7
76.2
75.6
61.5
54.4
56.9
67.1
47.3
39.3
93.2
48.9
45.7
50.4
61.9
C.7 Comparison to Meta Tise Training (Bartler et al., 2022)
Bartler et al. (2022) report error of 24.4% their top result on CIFAR10-C which is far worse than top results of 8.5% with an AugMix trained model, or 13.3% with vanilla trained model, but different architecture: they used a WRN-26-1 while we report results with a WRN-40-2. Thus, to make the comparison more fair, we tested our approach on their model architecture.
A direct comparison is not straight-forward since Bartler et al. (2022) trained their models using Keras while our code-base is in PyToreh. Therefore, we first trained baseline model in PyTorch on clean CIFAR10 using their architecture with the standard and widely used CIFAR10 training code available at https://githubb wtt.t//w....oo/auu//////aa wwaaangll//ytttcchhcce
(1.9k forks, 4.8k stars); the baseline test accuracy on clean CIFAR10 using the architecture of Bartler et al. (2022) is at 94.3%. As a second step, we adapted the baseline model with ENT and RPL on CIFARIO-C. Please see Table 28 for our results. BN adaptation is not possible for the model used by Bartler et al. (2022) since they use GroupNorm instead of BatchNorm. Due to the usage of Group.Norm, it not evident whether the affine GroupNorm parameters should be adapted, or all model paramcters. Thus, we tested both, and report the results for both adaptation mochanisms.
We find that adaptation of affine GN layers works better than full model adaptation, consistent with our results for the adaptation of BN layers. As the gains due to ENT and RPL seem lower than for our WRN-40-2 architecture, we hypothesize that the issue lies in the GN layers as the presence of GN instead of BN layers is the main difference between our WRN-40-2 and the new WRN-26-1 model. Therefore, we trained another WRN-2661 model with BN layers instead of GN, and adapted this model using BN, ENT and RPL. The bascline accuracy on clean CIFAR10 of WRN-26-1-BN is 95.04%. Indeod, we find that adapting the model with BN instead of GN layers leads to much larger gains, which is consistent with findings by Schneider et al. (2020) who found that BN adaptation outperforms non-adapted models trained with GN. Thus, we conclude
36

---

Published in Transactions on Machine Learning Research (11/2022)
that self-learning techniques work better with models which have BN layers and less well with models with GN layers. For both model types (with GN with BN layers), ENT works better than RPL. which is consistent with our other results on small-scale datasets.
Overall, our best result for the model architecture used by Bartler et al. (2022) (after replacing GN layers with BN layers) al. is 13.1% which much lower than their best result of 24.4%. Even when using GN layers, our best top-1 error is 18.0% which significantly lower than the best result of Bartler et al. (2022).
Table 28: Detailed results for our comparison to MT3 (Bartler et al., 2022)
gonss
shot
Imptise
defocus
glnas
motion
suon
snow
frost
fog
bright
contrast
elastic
pisclate
Spng
avg.
WRN-26-1-GN vanilla
Baseline (Bartler sl., 2022)
50.5
17.2
56.1
23.7
51.7
21.3
26.3
25.6
34.4
28.1
13.5
25.0
27.4
55.8
29.8
34.6
adapted (Bartler et al., 2022)
30.1
29.5
41.8
15.6
33.7
22.8
18.7
20.2
18.8
24.1
13.8
22.4
23.7
27.6
22.7
24.4
WRN-2G-1-GN vanilla
Baseline [ours]
32.0
30.2
ENT [GN layers, ours]
41.6
30.7
32.8
9.7
30.9
11.8
11.4
13.5
14.9
10.0
7.2
8.9
13.2
13.3
19.8
18.0
GN layers, ours]
39.3
31.6
30.6
10.6
16.1
13.9
ENT [ull model, ours
61.2
46.0
37.6
9.1
30.2
11.1
10.4
13.1
14.4
10.0
7.2
8.7
13.1
11.9
19.7
20.3
RPL [ull model, ours]
54.1
37.3
33.0
10.2
31.5
11.9
12.4
13.6
15.3
7.3
13.6
20.4
9.2
13.6
10.2
19.6
WRN-26-1-BN vanilla
Baseline [ours]
44.8
43.1
21.3
23.6
BN adept [ours]
28.0
28.9
24.2
10.5
31.5
12.7
14.0
18.4
18.1
9.4
17.1
26.9
13.2
15.3
12.7
18.7
ENT [BN layers, ours]
17.9
19.6
14.9
8.1
23.2
9.1
10.4
13.2
7.2
13.2
18.0
9.9
10.5
8.9
13.1
RPL [BN layers, ours]
21.3
22.0
17.9
9.3
26.6
10.5
11.8
15.1
15.0
7.8
14.7
20.8
12.1
11.8
10.4
15.1
C.8 Effect of batch size and linear learning rate scaling
How is self-learning performance affected by batch size constraints? We compare the effect of different batch sizes and linear learning rate scaling. In general, found that affine adaptation experiments ResNet50 scale can be run with batch size 128 on Nvidia V100 GPU (16GB), while only batch size 96 experiments are possible on RIX 2080 GPUs.
The results in Table 29 show that for a ResNet50 model, higher batch size yields a generally better performance.
Table 29: ImageNet-C dev set mCE for various batch sizes with linear learning rate scaling. All results are computed for a vanilla ResNet50 model using RPL with q 0.8, reporting the best score across maximium of six adaptation epochs.
batch size learning rate (x10-3)
16
32
64
80
96
128
0.125
0.250
0.500
0.625
0.750
1
dev mCE
53.8
51.0
49.7
49.3
49.2
48.9
C.9 Performance over different seeds in ResNet50 on ImageNet-C
To limit the amount of compute, ran RPL and ENT for our vanilla ResNet50 model three times with the optimal hyperparameters. The averaged results, displayed as "mean (unbiased std)" are:
Table 30: ImageNet-C performance for three seeds ResNet50 for ENT and RPL.
ResNet50 + sell-lerning
mCE on IN-C dev [%]
mCCE on IN-C test [%]
ENT
50.0 (0.04)
51.6 (0.04)
RPL
48.9 (0.02)
50.5 (0.03)
37

---

Published in Transactions on Machine Learning Research (11/2022)
C.10 Self-learning continuous test-test adaptation
We test our method on continuous test-time adaptation where the model adapts to continuous stream of data from the same domain. In Fig. 7, we display the error of the Noisy Student L2 model while it is being adapted to ImageNet-CC and ImageNct-R. The model performance improves as the model sees more data from the new domain. We differentiate continuous test-time adaptation from the online test-time adaptation setting (Zhang et al., 2021) where the model is adapted each test. sample individually, and reset after each test sample.
(i) ImageNet-G
(ii) ImageNet-R
28.9
26
28
Top-1 Error [%]
24
23.5
mCE [%]
Baseline
22
ENT
RPL
20
19.7
23.0
22.0
18
17.
22
0
3
Samples [x 10
Samples [x 10*]
Figure 7: Evolution of error during online adaptation for EeficientNet-L2.
38

---

Published in Transactions on Machine Learning Research (11/2022)
Table 31: Statistics of one-to-many mappings from IN-D to ImageNet.
Number of IN classes one IN-D class is mapped to Frequency these mappings
2
3
4
5
6
7
13
28
39
132
102
32
13
3
D Detailed and additional Results on IN-D
D.1 Detailed protocol for label mapping from DomainNet to ImageNet
The mapping was first done by comparing the class labels in DomainNet and the synset labels ImageNet. Afterwards, the resulting label maps were cleaned manually, because simply comparing class label strings resulted in imperfect matches. For example, images of the class "hot dog" in DomainNet were mapped to the class "dog" in IN. Another issue is that IN synset labels of different animal species do not contain the animal name in the text label, e.g., the class "orangutan, orang, orangutang, Pongo pygmaeus" does not contain the word "monkey" and we had add this class to the hierarchical class "monkey" manually. We verified the mappings by investigating the class-confusion matrix of the true DomainNet class and the predicted ImageNet classes remapped to DomainNet the main diagonal, indicaling that ImageNet classes have not been forgotten. The statistics for the mappings are shown in Table 31. Most IN-D classes (102) are mapped to one single ImageNet class, A few IN-D classes are mapped to more than 20 ImageNet classes: the IN-D classes "monkey" and "snake" are mapped to 28 ImageNet monkey and smake species classes, the IN-D class "bird" is mapped to 39 ImageNet bird species classes, and the IN-D class "dog" is mapped 132 ImageNet dog breed classes.
the "Real" domain, and checked that the predictions lay
We have considcred max-pooling predictions across all sub-classes according to the ImageNet class hicrarchy following the approach of Taori et al. (2020) for Youtube-BB and ImageNet-Vid (Recht et al., 2019). However, Radford et al. (2021) note that the resulting mappings are sometimes "much less than perfect", thus, we decided to clean the mappings ourselves to increase the mappings' quality. The full dictionary of the mappings will released alongside the code.
D.2 Evaluation protocol on IN-D
The domains in IN-D differ in terms of their difficulty for the studied models. Therefore, to calculate aggregate score, we propose normalizing the error rates by the error achieved by AlexNet on the respective domains to calculate the mean error, following the approach in Hendrycks & Dietterich (2019) for IN-C. This way, we obtain the aggregate score mean Domain Error (mDE) by calculating the mean over different domains,
ES
D
DE
mDE
Σ
El
(21)
E EAlexNet
d=1
where Ef is the top-1 error of classifier on domain d.
For all IN-D results we report in this paper, we chose the hyperparameters the IN-C dev set. We tried different model selection scheme on IN-D control experiment with "Leave out "oss-validation" (LloutCV): with round-robin procedure, choose the hyperparameters for the test domain on all other domains. We select the same hyperparameters as when tuning on the "dev" set: For the ResNet50 model, we select over the number of training epochs (with a maximum of 7 training epochs) and search for the optimal learning rate in the set [0.01, 0,001, 0.0001]. For the EfficientNet-L2 model, we train only for one epoch as before and seloct the optimal learning rate in the set [4.6 10-3 4.6 x 10-4, 4.6 10-5, 5, 4.6 10-9]. This model selection leads to worse results both for the ResNet50 and the EfficientNet--22 models, highlighting the robustness of our model selection process, see Table 32.
Leave--nne-utt-ccoo--ellldaaa
39

---

Model Published in Transactions on Machine Learning Research (11/2022)
Table 32: mDE in % on IN-D for different model selection strategies.
model
model selection
LloutCV
IN-C dev
ResNet50 RLL ResNet50 ENT
81.3
76.1
82.4
77.3
Efficient ENT
69.2
66.8
EfficientNNet-L22 RPI.g=0.8
69.1
67.2
D.3 Detailed results for robust ResNet50 models on IN-D
We show detailed results for all models on IN-D for vanilla evaluation (Table 33) BN adaptation (Table 34), L (Table 35) and ENT(aable 36). For RPLg- and ENT, we use the same hyperparameters that we chose on our IN-C 'dev' set. This means we train the models for epochs with RPL and for one epoch with ENT.
We evaluate the pre-trained and public checkpoints of SIN (Geirhos et al., 2019), ANT (Rusak et al., 2020), ANT+SIN (Rusak et al., 2020), AugMix (Hendrycks et al., 2020b), DeepAugment (Hendrycks et al., 2020a) and A (Hendrycks et al., 2020a) in the following tables.
Table 33: Top-1 error on IN-D in % as obtained by robust Re models. For reference, we also show the
mCE on IN-C and the top-1 error on IN-R. See main test for model references.
Model
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
mDE
IN-C
IN-R
vanilla
76.0
89.6
65.1
99.2
40.1
82.0
88.2
76.7
63.9
SIN
71.3
88.6
62.6
97.5
40.6
77.0
85.6
69.3
58.5
ANT
73.4
88.9
63.3
99.2
39.9
80.8
86.9
62.4
61.0
ANT+-SIN
68.4
88.6
60.6
95.5
40.8
70.3
83.1
60.7
53.7
AugMix
70.8
88.6
62.1
99.1
39.0
78.5
85.4
65.3
58.9
DeepAugment
72.0
88.8
61.4
98.9
39.4
78.5
85.6
60.4
57.8
DeepAug Augmix
68.4
88.1
58.7
98.2
39.2
75.2
83.4
53.6
53.2
Table 34: Topl error on IN-D in % obtained by state-of-the-art robust ResNet50 models and batch norm adaptation, with batch size of 128. See main text. for model references.
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
mDE
vanilla
70.2
88.2
63.5
97.8
41.1
78.3
80.2
SIN
67.3
97.2
44.0
75.2
79.6
ANT
69.2
97.5
42.9
79.5
80.7
ANT+SIN
64.9
88.2
96.8
42.6
77.8
AugMix
66.9
88.1
61.2
97.1
40.4
75.0
78.4
Augment
66.6
89.7
60.0
97.2
42.5
75.1
78.8
Aug Auggmmix Augmix
61.9
85.7
57.5
95.3
40.2
69.2
74.9
Table 35: Top-1 error on IN-D in % as obtained by stale-of-the-arl robust ResNet50 models and RPL.g See main text for model references. Model
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
mDF
vanilla.
63.6
85.1
57.8
99.8
37.3
73.0
76.1
SIN
60.8
86.4
99.0
37.8
67.0
76.8
ANT
63.4
86.3
99.2
37.7
78.1
ANT SIN
61.5
86.4
97.0
39.0
76.1
AugMix
59.7
83.4
98.2
35.6
70.1
74.6
Deep
58.1
84.6
53.3
99.0
36.2
64.2
74.8
Aug-Aug++ Augming
57.0
83.2
53.4
99.1
36.5
61.3
72.6
40

---

Published in Transactions on Machine Learning Research (11/2022)
Table 36: Top-1 error on IN-D in % as obtained by state-of-the-art robust ResNet50 models and ENT. See main text for references the used models.
Model
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
mDE
vanilla
65.1
85.8
59.2
98.5
38.4
75.8
77.3
SIN
62.1
87.0
57.3
99.1
39.0
68.6
75.5
ANT
64.2
86.9
58.7
97.1
38.8
72.8
76.5
ANT+SIN
62.2
86.8
57.7
95.8
40.1
68.7
75.2
AugMix
60.2
84.6
55.8
97.6
36.8
72.0
74.4
DeepAugment
59.5
85.7
54.4
98.0
37.1
66.4
73.3
DoepAug Augmix
58.4
84.3
54.7
98.5
38.1
63.6
72.7
Table 37: mDE IN-D in % as obtained by robust ResNet.50 models with baseline evaluation, batch norm adaptation, R.L and ENT. See main text for model references.
mDE on IN-D (
Model
Baseline BN adapt RPL,=00
ENT
vanilla
77.3
SIN
85.6
79.6
76.8
75.5
ANT
86.9
80.7
78.1
76.5
ANT+SIN
83.1
77.8
76.1
75.2
AugMix
85.4
78.4
74.6
74.4
Deep
85.6
78.8
74.8
73.3
Deep DeepAugment+Auggils
83.4
74.9
72.6
72.7
The summary results for all models are shown in Table 37.
We show the top-1 error for the different IN-D domains versus training epochs for a vanilla ResNet50 in Fig. We indicate the epochs and 5 at which we extract the crrors with dashed black lines.
clipass
Infograpb
paincing
90
85-
95-
80
RPL ENT
80
75
70-
70
65-
85-
quickdlrow
<derthh
200.00
100
99
99.0
98.5
85
80
97.5-
75
97.0
10
10
15
0
Epachs
Egorths
Epochs
Figure 8: Top-1 error for the different IN-D domains for a ResNet50 and training with RPL and ENT. We indicate the epochs which we extract the test errors by the dashed black lines (epoch for ENT and epoch 5 for RP o.)
41

---

Published in Transactions on Machine Learning Research (11/2022)
D.4 Detailed results for the EfficientNet-L2 Noisy Student model IN-D
We show the detailed results for the Fffiiiinn/Net-.22 Noisy Student model on IN-D Table 38.
Table 38: Top-1 error ( on IN-D in % for ElfficientNet-..2
Domain
Baseline
ENT
RPL
Clipart
45.0
39.8
37.9
Infograph
77.9
91.3
94.3
Painting
42.7
41.7
40.9
Quickdraw
98.4
99.4
99.4
Real
29.2
28.7
27.9
Sketch
56.4
48.0
51.5
mDE
67.2
66.8
67.2
D.5 Detailed results the error analysis IN-D
Analysing frequently predicted classes We analyze the most frequently predicted classes on IN-D by a vanilla ResNet50 and show the results Fig. The colors of the bars indicate whether the predicted class is part of the IN-D dataset: "blue" indicates that the class appear in the IN-D dataset, while "orange" means that the class is not present in IN-D.
We make several interesting observations: First, we find most errors interpretable: it makes sense that a ResNet50 assigns the label "comic book" to images from the "Clipart" or "Painting" domains, or "website" to images from the "Infograph" domain, or "envelope" to images from the "Sketch" domain. Second, on the hard domain "Quickdraw", the ResNet50 mostly predicts non-sensical classes that are not in IN-D, mirroring its almost chance performance on this domain. Third, we find no systematic errors on the "Real" domain which is expected since this domain should be similar to IN.
A Analyzing the correlation between the performance on IN-C/IN-R and IN-D We show the Spearman's rank correlation coefficients for errors on ImageNet-D correlated to crrors on ImageNet-R and ImageNet-Ci for robust ResNet50 models in Fig. 9. For this correlation analysis, we take the error numbers from Table 33. We find the correlation to be high between most domains in IN-D and IN-R which is expected since the distribution shift between IN-R and IN is similar to the distribution shift between IN-D and ImageNet. The only domain where the Spearman's rank correlation coefficient is higher for IN-C is the "Real" domain which can be explained with IN-C being closer to real-world data than IN-R. Thus, we find that the Spearman's rank correlation coefficient reflects the similarity between different datasets.
Filtering predictions on IN-D that cannot be mapped to ImageNet We perform a second analysis: We filter the predicted labels according to whether they can be mapped to IN-D and report the filtered top-1 errors as well as the percentage of filtered out inputs in Table 39. We note that for the domains "infograph" and "quickdraw", the ResNet50 predicts labels that cannot be mapped IN-D in over 70% of all cases, highlighting the hardness of these two domains.
Filtering labels and predictions on IN that cannot be mapped ImageNet-D To test for possible class-bias effects, we test the performance of ResNet50 model on ImageNet classes that can be mapped to IN-D and report the results in Table 39.
First, we map IN labels to IN-D to make the sctting as similar as possible to our experiments on IN-D and report the top-1 error (12.1%). This error is siguificantly lower compared to the top-1 error a ResNec50 obtains following the standard evaluation protocol (23.9%). This can be explained by the simplification of the task: While in IN there are bird classes, these are all mapped to the same hicrarchical class in IN-D. Therefore, the classes in IN-D are more dissimilar from each other than in IN. Additionally, there are only 164 IN-D classes compared to the 1000 IN classes, raising the chance level prediction.
42

---

Published in Transactions on Machine Learning Research (11/2022)
Table 39: top-1 error on IN and different IN-D domains for different settings: left column: predicted labels that cannot be mapped to IN-D filtered out, right column: percentage of filtered out labels.
Dataset
top-1 error on filtered labels in percentage of rejected inputs
IN val
13.4
52.7
IN-D real
17.2
27.6
IN-D clipart
59.0
59.0
IN-D infograph
59.3
74.6
IN-D painting
39.5
42.4
IN-D quickdraw
96.7
76.1
IN-D sketch
65.6
47.9
Clipart
Infograph
Painting
spearnnn (ImageNet-RRpOO.C 85 2_sppamann (ImageNor-C)=0.58
speermen [lmegeNet-Rj-O.. 82 P_spcarman (ImageNat-C)=O.s
speermen (ImageNet-R)-O 96
P_spearman (ImagoNat SaagoNat-CC--O.8e 86
Number predictinn
60
150
60
20:
100
40
20-
50-
10-
comic book
envelope
jigsaw puzzle
website
menu
envelope
comic book
book jacket
jigsaw puzzle
Quickdraw
o_speaman (ImageNet-RjeO..k
P_spearman (ImageNetRR)o0 S.eeermmnnn JImageNet-C)HD.3
P_spearman (ImageNett-)=00s
O_ s.seemmnnn (ImageNet-Cy+0.38
p_ o.spparman (ImageNet-C)-0.5g
Number predictions
100
3
75
75
50
25
25
hhook claw
chain
labyrinth
envelope
cornic book
studio couch
enivelope
labyrinth
nematode
Figure 9: Systematic predictions vanilla ResNet50 on IN-D for different domains. The colors of the bars indicate whether the predicted class part of the IN-D dataset: "blue" indicates that the class appear in the IN-D dataset, while "orange" means that the class is not present in IN-D. The errors IN-R are strongly correlated to errors IN-D for most domains, except for the "Real" domain.
If we further only accept predictions that can be mapped to IN-D, the top-1 error is slightly increased to 13.4%. In total, about 52.7% of all images in the IN validation set cannot be mapped to IN-D.
D.6 Top-1 error on IN-D for AlexNet
We report the top-1 error numbers on different IN-D as achieved by AlexNet in Table 40. We used these numbers for normalization when calculating mDE.
43

---

Published in Transactions on Machine Learning Research (11/2022)
Table 40: top-1 error on IN-D by AlexNet which was used for normalization.
Datasel
lop-1 error in %
IN-D real
54.887
IN-D clipart
84.010
IN-D infograph
95.072
IN-D painting
79.080
IN-D quickdraw
99.745
IN-D N sketch
91.189
44

---

Published in Transactions on Machine Learning Research (11/2022)
E
Additional experiments
E.1 Beyond ImageNet classes: Self-learning on WILDS
The WILDS benchmark (Koh et al., 2021) is comprised of ten tasks to test domain generalization, subpopu- lation shift, and combinations thereof. In contrast to the setting considered here, many of the datasets in WILDS mix several 10s or 100s domains during test time.
The Camelyon17 dataset in WILDS contains histopathological images, with the labels being binary indicators of whether the rentral 32x32 region contains any tumor tissue; the domain identifies the hospital that the patch was taken from. Camelyon17 contains three different test splits with different domains and varying difficulty levels. For evaluation, took the pretrained checkpoint from worksheets.codalla 4G0004444593334 1864222066424666600 (camelyon17 (Huang et al., 2017) and verified the reported baseline performance numbers. We adapt the models using ENT or RPL for a maximum of 10 epochs using learning rates {3 x 10 10-4, ..... 3 x 10-1). The best hyperparameter is selected according to OOD Validation accuracy.
worksheets/ densenet seed0) for DenseNet121 model
The RxRx1 dataset in WILDS contains RGB images of cells obtained by fluorescent microscopy, with the labels indicating which of the 1,139 genetic treatments (including no treatment) the cells received; the domain identifies the batch in which the imaging experiment test splits, however, unlike Camelyon17, in all of the splits the domains are mixed. For evaluation, we took for a ResNet50 model and verified the reported baseline performance numbers. We adapt the models using ENT or RPL for a maximum of 10 epochs using base learning rates {6.25 10-6, 6.25 10- 6.25 x 10 10-2), which are scaled to the admissible batch size for single GPU adaptation using linear scaling. The best. hyperparameter is selected according to OOD Validation accuracy.
run. The RxRx1 dataset contains three
the pretrained checkpoint from wwwesshcts.codalab.ooo minalabborgbbuddls///Oeee
Table 41: Self-learning can improve performance on WILDS if a systematic shift is present on Camelyon17, the ood validation and test sets are different hospitals, for example. On datasets like RxRx1 and FMoW, we do not see an improvement, most likely because the ood domains are shuffedd, and a limited amount of images exist for each test domain.
Top-1 accuracy [%
Validation (ID)
Validation (OOD)
Test (OOD)
Camelyon17
Baseline
81.4
88.7
63.1
BN adapt
97.8 (+16.4)
90.9 (+2.2)
88.0 (+244.))
ENT
97.6 (+16.2)
92.7 (+4.0)
91.6 (+228.5)
RPL
97.6 (+16.2)
93.0 (+4.3)
91.0 (+-27.9)
RxRx1
Bascline
35.9
19.1
29.7
BN adapt
35.0 (-0.9)
19.1 (0.0)
29.4 (-0.3)
ENT
34.8 (-1.1)
19.2 (+0.1)
29.4 (-0.3)
RPL
34.8 (-1.1)
19.2 (+0.1)
29.4 (-0.3)
FMoW
Baseline
60.5
59.2
BN adapt
59.9 (-0.6)
57.6 (-1.6)
51.8 (-1.1)
ENT
59.9 (-0.6)
58.5 (-0.7)
52.2 (-0.7)
RPL
59.8 (-0.7)
58.6 (-0.6)
52.1 (-0.8)
The FMoW dataset in WILDS contains RGB satellite images, with the labels being one of 62 building land use categories; the domain specifies the year in which the image was (aken and its geographical region (Africa, the Americas, Oceania, Asia, or Europe). The FMoW dataset contains four test splits for different time periods, for which all regions are mixed together. For evaluation, we took the pretrained checkpoint from / crrorkshetsscodaaabb wooallb.orgg//ooooooooooooooo..l Donddess00x001000 (tt.://ecee220eeeeaaloeeeeeeeee verified the reported baseline performance numbers. We adapt the models using ENT or RPI. for maximum
for 1enseNet121 model and
45

---

Published in Transactions on Machine Learning Research (11/2022)
...... x 20- of 10 epochs using learning rates (5.0 x 10-0, 10 x 1............................... according to OOD Validation accuracy.
The best hyperparameter is selected
While we see improvements on Camelyon17, neither BN adaptation nor self-learning can improve performance on RxRx1 or FMoW. Initial experiments on Pov and iWildsCam also do not show improvements with self-learning We hypothesize that the reason lies the mixing of the domains: Both BN adaptation and our self-learning methods work best on systematic domain shifts. These results support our claim that
self-learning is effective, while showing the important limitation when applied to more diverse shifts.
E.2 Small improvements on Big Transfer models with Group normalization layers
We evaluated BigTransser models (Kolesnikov et al., 2020) provided by the timm library (Wightman, 2019). A difference to the ResNet550, ResNeXt101 and NfficientNeet models is the use of group normalization layers, which might influence the optimal method for adaptation for this evaluation, we followed our typical protocol as performed on ResNet50 models, and used affine adaptation.
For affine adaptation, distilled Bigntfnsfer ResNet50 model improves from 49.6 % to 48.4 % mCE on the ImageNet-C development set, and from 55.0 % to 54.4 % mCE the ImageNet-Ci test set when using RPL (q= 0.8) for adaptation, at learning rate 7.5 x 10-4 at batch size 96 after a single adaptation epoch. Entropy minimization did not further improve results on the ImageNet-C test set. An ablation over learning rates and epochs on the dev set shown Table 42, the final results are summarized Table 43.
Table 42: mCE % on the IN-C dev set for ENT and RPL for different numbers of training epochs when adapting the affine batch norm parameters of BigTraanser ResNet50 model.
Table 43: mCE in % on the IN-C dev set for ENT and RPL for differ- ent nunbers of training epochs when adapting the affine batch norm param- eters of a Big BigTransfer ResNet50 model.
criterion 1r 7.5 x epoch
ENT
RPL.
10-6
10 10-4
10
10-6
10-4
10-3
49.63
49.63
49.63
49.63
49.63
49.63
dev mCE
test mCE
49.44
50.42
52.50
49.54
48.89
48.95
49.26
50.27
56.47
49.47
48.35
50.77
Baseline
49.63
55.03
ENT
48.80
56.36
49.08
52.18
60.06
49.39
48.93
51.45
RPL
48.35
54.41
48.91
52.03
60.50
49.31
50.01
51.53
48.80
51.97
62.91
49.24
49.96
51.34
48.83
52.10
62.96
49.16
49.71
51.19
7
48.83
52.10
62.96
49.16
49.71
51.19
E.3 Can Self- Learning improve over Self-Learning based UDA?
An interesting question is whether test-time adaptation with self-learning can improve upon self-learning based UDA methods. To investigate this question, we build upon French et al. (2018) and their released We trained the Baseline models
code base at github.com/BBrig Booom/rittffur/seeeeeeee -etprr/ssssee-emmee--iiiei vewtumbbeeviuua-ddmaiin from scratch using the provided shell scripts with the default hyperparameters and verified the reported performance. For adaptation, we tested BN adaptation, ENT, RPL, as well as continuing to train in exactly the setup of French al. (2018), but without the supervised loss. For the different losses, we adapt the models for maximun of 10 epochs using learning rates [1 10 5 10-4 ....., 10-1).
Note that for this experiment, in contrast to any other result in this paper, purposefully do not perform proper hyperparameter selection based on a validation intasot-instead we report the best accuracy across all tested epochs and learning rates to give an upper bound on the achievable performance for test-time adaptation.
As highlighted in Table 44, none of the four tested variants is able to meaningfully improve over the baseline, corroborating our initial hypothesis that searning within a full UDA setting is the optimal strategy, if dataset size and compute permits. On the other hand, results like the teacher refinement. step in DIRT-T
46

---

Published in Transactions on Machine Learning Research (11/2022)
(Shu et al., 2018) show that with additional modifications in the loss function, it might be possible to improve over standard UDA with additional adaptation at. test time.
Table 44: Test-time adaptation marginally improves over self-ensembling.
Baseline
BN adapt
ENT
RPL
Self-ensembling loss
MNIST=SVHN
MT+TF
33.88
34.44
34.87
35.09
33.27
MT+CT
32.62
34.11
34.25
34.21
33.36
MT+CT+IF
41.59
41.93
41.95
41.95
42.70
MT+CT+TFA
30.55
32.53
32.54
32.55
30.84
SVHN-spocific aug.
97.05
96.82
96.91
96.87
97.12
MSIST-USPS
MT+TF
98.01
97.91
97.96
97.91
98.16
MT+CT
88.34
88.39
88.54
88.39
88.44
MT+CT+IF
98.36
98.41
98.41
41
98.50
MT+CT+TEA
98.45
98.45
98.45
98.45
98.61
SVHN >MNIST
MT
98.49
98.47
98.49
98.47
99.40
MT+CT
88.34
88.36
88.36
88.36
89.36
MT+CT+TF
99.51
99.49
99.5
99.49
99.57
+T+CT+TEA
99.56
99.57
99.57
99.57
99.58
SVHN-specific aug.
99.52
99.49
99.5
99.49
99.65
USPS >NNIST
MT IT
92.79
92.62
92.62
92.66
93.08
MT+CT
99.11
99.13
99.14
99.13
99.21
+T+CT+TF
99.41
99.42
99.45
99.42
99.52
MT+CT+TFA
99.48
99.54
99.57
99.54
99.54
47

---

Published in Transactions on Machine Learning Research (11/2022)
F
Software stack
We use different open souree software packages for our experiments, most notably Docker (Merkel, 2014), scipy and numpy (Virtanen et al., 2020), GNU parallel (Tange, 2011), Tensorflow (Abadi et al., 2016), PyTorch (Paszke et al., 2017), timm (Wightman, 2019); Self-ensembling for visual domain adaptation (French et al., 2018), the WILDS benchmark (Koh et al., 2021), and torchvision (Marcel & Rodriguez, 2010).
48


Reviews:
[{"review_id": "tTPfG0zLp5", "reviewer": "Reviewer_iPbT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- This is an informative collection of results on how well self-training/entropy minimization/pseudo-labeling can do and how to apply it well. The experiments cover\n  - architectures for image classification that span ResNet, DenseNet, ResNeXt, EfficientNet, and ViTs;\n  - pre-training on the standard ImageNet data, on larger-scale uncurated Instagram data, and by domain adaptation schemes;\n  - parameterization of updates over all parameters, only affine parameters, or more variants for ViT architectures; and\n  - hyperparameters in general like the learning rate and epochs for optimization and the update interval and other settings specifically for RPL.\n- Self-learning does indeed work well, and the state-of-the-art results among adaptation methods are worth sharing, in particular for the ImageNet-scale experiments on IN-C/A/R/D.\n- The breadth and rigor of the experiments is relevant and necessary: prior works only include a subset of models or shifts in their experiments and they tune and evaluate in varying ways.\n  The suggestions for best practices (Sec. 9) are definite, justified, and computationally practical.\n- The thoroughness of the experimentation and documentation for reproduction and extension are commendable (see appendices B, C, and D).\n- The proposed IN-D dataset makes reasonable and justified design choices, as explained in the text and appendix D, following in the style of \"renditions\" as a type of shift (as done by IN-R).\n\n**Weaknesses**\n\n- The related works of TTT and Tent are incompletely or incorrectly summarized. This submission evidently delivers a much-expanded scope for experiments with entropy minimization and related self-learning techniques, but it should still appropriately summarize what came before.\n    - For TTT, it is worth noting that it included results on gradual adaptation (across severities of IN-C), counter to the \"future work\" left in Sec. 2 of this work.\n  In the results, TTT is likewise able to make use of the full dataset, which is for some reason underlined for RPL only in Sec. 6 of this work.\n    - For TENT, the difference with ENT in this work is left unclear, and the existing experiments for TENT are not accurately summarized.\n  When are ENT results the same or different than TENT? (For example, a row in Sec. C.4's Table 19 is labeled TENT, but tables in the main paper are labeled ENT.)\n  While TENT experiments are summarized as a \"single model [...] on ImageNet-C\", TENT experimented with multiple models (ResNet-50, ResNet-26, the SAN attention model, and the implicit DEQ model) and on multiple datasets (CIFAR-10/100 corruption datasets, VisDA-C and GTA-CityScapes simulation-to-real datasets, and the digits datasets SVHN, MNIST, USPS).\n- Prior test-time and self-learning schemes for updating models are absent from the related work and results: see [TTT++ NeurIPS'21](https://proceedings.neurips.cc/paper/2021/hash/b618c3210e934362ac261db280128c22-Abstract.html), [T3A NeurIPS'21](https://proceedings.neurips.cc/paper/2021/hash/1415fe9fea0fa1e45dddcff5682239a0-Abstract.html), [BUFR ICLR'22](https://arxiv.org/abs/2107.05446), [EATA ICML'22](https://arxiv.org/abs/2204.02610), and [confidence maximization with input transformation](https://arxiv.org/abs/2106.14999).\n  These should at least be referenced, though all need not be compared against on ImageNet, where they do not report results at ImageNet scale (T3A, BUFR).\n  Of course evaluating them on ImageNet would be a further contribution, but it is reasonable to rule them out as beyond the scope of this empirical study.\n- (Minor) The theoretical component is vastly simplified from the experiments studied, although this is admitted by the text. The real issue is that is not an explanation for the highlighted observation that \"different self-learning schemes are optimal for small-scale vs. large-scale datasets\". This weakness is minor, as the theoretical content is a bonus on top of this primarily empirical work, but the purpose of the theory could be better indicated.\n- (Minor) The proposed IN-D dataset (Sec. 8) is derived from the VisDA benchmark, so the degree of contribution is lessened. Nevertheless it is a contribution, simply because it is harder, but more importantly because it differs from IN-R in separating out the types of shift. This allows future research to compare single-target vs. multi-target adaptation by evaluating shifts separately or jointly.\n\n**Summary**\n\nThis work _delivers on its claimed contributions_, and _needs only correct the gaps and errors in its coverage of related work_ to be ready for publication.\nI expect it will serve as an informative publication for researchers in adaptation and robustness. The contributions are:\n\n- experiments of substantial scope across datasets, model architectures and parameters, and shifts;\n- a new to source-free adaptation loss, RPL, derived from the GCE from label noise robustness research;\n- a new dataset, IN-D, derived from the VisDA dataset for UDA, but with a different protocol for source-free adaptation, and error rates to verify the shifts are difficult; and\n- a rigorous experimental protocol that is strictly adhered to and is accompanied by clear advice on best practices.\n\n**Miscellaneous Feedback and Questions**\n\n- Sec. 3: Is \"standard cross entropy\" more precisely described as softmax cross entropy? There are other cross entropies, of course, but all of the deep classifiers in this study make softmax outputs.\n- Sec. 4, Table 1: What is the result for with R-50 on IN-A? Does self-learning help or harm? This is of interest, as this dataset should be adversarial to self-learning in particular.\n- Sec. 4, Table 2: It would be nice to match the architecture of the vanilla and UDA models, to assess how much UDA vs. TTA is helping. As it stands, the vanilla baseline is better than UDA.\n- Sec. 5: For the ViT result, please clarify the loss. Is it RPL or ENT, or purely DINO?\n- Sec. 6: Several classes of results were already demonstrated by TENT, and so perhaps should be identified as such? This is the case of \"usage of the full dataset without a threshold\", \"short update intervals\", and \"adaptation of only affine parameters\". To be clear, the confirmatory and broader evidence is valuable, but it is nevertheless not the first.\n- Sec. 9 Conclusions: Can we recommend self-learning to practitioners without further understanding of stability and deployment data distributions? The topic is certainly of interest, but it may not be ready to deploy. A more nuanced statement could be appropriate.\n- Sec. D.5: Consider discarding the results without the mapping of permissible classes. Leading the nonsensical/irrelevant predictions first is confusing.\n- Sec. E.1 & E.2: The limitations on mixed shifts and architecture are worth pointing to from the main paper. Consider a footnote or pointer from the results section.\n- Sec. E.3: How does this square with the digits results for TENT? To briefly mention them here, TENT shows improved test-time adaptation accuracy for 2/3 datasets, suggesting that models can perhaps do better by specializing to target than jointly training on source and target.\n\nOptional Suggestions for Experiments\n\n- ENT seems to do better on smaller models and datasets, such as on CIFAR-10-C and digit datasets. Is this specific to the data are architecture? How do ENT and RPL compare on R-18 on IN-C, for instance, to measure a smaller model on a larger dataset?\n- How do RPL and ENT compare in the online and test-time setting? Most results are for the offline and source-free setting of multi-epoch optimization, which is fine, but the most efficient setting may also be of interest.\n\nTrivial\n\n- Sec. C.6: Why is TTT labeled contrastive? It is a self-supervised method with an auxiliary task, but it is not a contrastive learning method.\n- Sec. D.3: Should GCE be RPL?\n- Typo, pg. 2: \"both in\" is written for more than two sets\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Self-learning, or the updating of a model from its own predictions, is an established technique for semi-supervised learning and domain adaptation.\nIn particular, entropy minimization and pseudo-labeling have recently been the focus of _source-free_ and _test-time_ adaptation methods, which adapt a model given only unlabeled target/test data without labeled source/train data.\nThis work provides a broad and informative empirical study of self-learning for adaptation across training schemes, model architectures, and various dataset shifts.\nThe presented study is of particular value due to (1) the scale of its models and focus on ImageNet-like data, (2) the variety of shifts including corruptions, different renditions, and adversarially filtered data, and (3) the emphasis on a clear, uniform, and rigorous experimental protocol for tuning and evaluation.\nThe self-learning techniques studied are pseudo-labeling (PL) and entropy minimization (ENT), as established by prior work, and a variant of pseudo-labeling that imports the generalized cross-entropy loss from prior work on robustness to label noise, which this work names Robust Pseudo-Labeling (RPL).\nThe new knowledge of these experiments is not in the (lack of) novelty of the losses, but in the broad and fair comparison of them in the setting of source-free adaptation.\nBenchmarking underlines the finding that self-learning can achieve state of the art results for adaptation on CIFAR-10-C, IN-C, IN-R, IN-A.\nThis finding is worth demonstrating so thoroughly, and worth broadcasting to the community, in order to highlight how self-learning can be effective, simple, and robust to hyperparameters, as this has not necessarily been appreciated while more complex and computationally-intensive methods have been developed.\nThis work proposes a new adaptation dataset, IN-D, in addition to its empirical contributions measured on existing datasets.\nIN-D is like the existing IN-R, but harder, which one could argue is needed as more progress is made on the existing IN variants.\nFinally, a last methodological contribution is made with the emphasis placed on experimental protocol: this work tunes on a single development set (the held-out corruption types of IN-C) and then evaluates on all other shifts.\nExising papers on source-free adaptation suffer from a certain amount of diversity and hence incomparability in setup, so advocating for a single and practical setup is worthwhile.\nIn summary, this work extends and standardizes empirical knowledge of self-learning for adaptation and does so thoroughly, with good agreement between the claimed scope and completed experiments.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "**Critical Changes** for Correctness:\n\n- Naming in Sec. 1 & 3: Please clarify \"self-learning\" as the superset of pseudo-labeling, entropy minimization, and RPL (or not). The current language is a bit confused, as entropy minimization is included in self-learning, but then the text and table captions (see Table 4) for instance then claims self-learning is better than entropy minimization.\n- Crediting Related Work in Sec. 2: Please discuss the related test-time adaptation methods listed above that are absent from the related work. If these are not in scope, then please explain how this is so during the discussion phase.\n- Attributing Techniques in Sec. 3 & 4: Please note the choice of parameters aligns with TENT, in updating the affine parameters of normalization layers, and not TTT/MEMO/others in the updating of all parameters.\n- Attributing Results in Sec. 2, 4, and 6: Please identify which results are new and which are expansions of prior work when reporting results. For instance, the confirmations of short update intervals and affine parametrs for convnets are presented alone, but both were already shown (for fewer architectures, datasets, ...) by TENT. Furthermore, results on gradual adaptation are shown not just by Kumar 2020 but also TTT.\n\n**Minor Changes** for Improvement:\n\n- Sec. 3: Please consider a discussion of parameterization (that is, the choice of affine parameters or not in TENT) and regularization (that is, the information maximization/diversity regularizer in SHOT). These are important choices for the quality of adaptation, as this work confirms for affine parameterization.\n- Sec. 4., Tables 1 & 2: Please consider reporting RPL and ENT side-by-side in both tables. This allows the reader to measure their difference, and would reinforce the point that self-learning generally works, even if one or the other may be best for a given evaluation.\n- Sec. 2 Related Work: Please make a contribution statement or summarize the aspects on which related work are compared and contrasted. The immediate dive into papers does not orient the reader. Perhaps consider defining source-free and test-time at the beginning of this section or in the introduction.\n- Sec. 7 Theory: Please clarify the connection of this model with the difference in small-scale vs. large-scale results or introduce\n- Sec. 6, Table 9: Please specify the architecture for a more self-contained table.\n- Sec. 7, Table 10: Please include the IN error rate to gauge the degree of shift.\n\n", "broader_impact_concerns": "There are no ethical concerns that warrant a broader impact statement. This work is about a general approach for adapting visual recognition models to reduce generalization error, and it is not particularly suited to a specific positive or negative ethical purpose.\nNevertheless, the authors are encouraged to reconsider their advice to apply self-learning in practice, as it is not guaranteed to improve accuracy.\nIt may be obvious to those with experience in adaptation research, but a caution to experiment with a given shift before deployment would not be unreasonable.\n\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9McAPSU5zM", "reviewer": "Reviewer_4Aye", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\n1. The paper addresses an important problem in machine learning, i.e., the distribution shift between training and testing.\n2. The paper has a clear argument and successfully proved it, i.e., using self-learning for test-time adaptation.\n3. The paper is well-written and easy to follow.\n4. The experimental validation is sufficient to validate the core argument.\n\n# Weakness\n1. The paper seems to rebrand unsupervised learning or self-distillation as \"self-learning\". I am not sure if this is appropriate.\n2. The idea of self-supervised learning for test-time adaptation does not seem novel. It is worthwhile to provide sufficient literature review on this topic. Some related work to consider (some do not seem cited by the paper):\n    * Azimi et al. Self-supervised Test-time Adaptation on Video Data. 2021\n    * Chen et al. Contrastive Test-Time Adaptation. 2022\n    * Bartler et al. MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaption. 2022\n    * Sun et al. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. 2020\n3. Experiments should compare with some related work\n4. The calibration of the adapted model is not studied\n5. The forgetting effect of the training distribution is not studied", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of distribution discrepancy between training and testing. The technique of interest is self-learning such as entropy minimization and pseudo labeling. The paper's purpose is to demonstrate the effectiveness of using self-learning on the testing examples, which have no label information. The experiments show the promise of self-learning, which significantly error reduction on multiple datasets.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Improve the discussion on related work\n2. Provide necessary comparison with related approaches\n3. Provide a discussion on the model calibration of the unadapted and adapted models\n4. Provide a discussion on the forgetting effect of the training distribution.", "broader_impact_concerns": "The proposed approach could introduce bias from the test data set. People usually execute actions to avoid bias during training, however, there is no known policy to restrict bias in the testing data. The model adapted to the biased testing data will lead to fairness issues if used subsequently on broader applications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "twBAESlYdv", "reviewer": "Reviewer_qrwv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This empirical study is well motivated that a thorough guideline/evaluation for the recently developed self-learning and test-time adaptation methods is desired;\n- The paper is clear and overall well written;\n- The paper included a wide range of domain shift datasets and network architectures;\n- The authors performed rich ablation studies on the characteristics of the self-learning methods;\n\nWeaknesses:\n- The hospital example for the necessity of source-free domain adaptation in the introduction section is a great motivation for this study. However, there’s no medical data has been evaluated in the experiments section;\n- Only single-run results have been reported in the paper. The comparison between baselines might be subject to randomness. Perhaps repeated experiments are necessary ;\n- In Table 1 & 2, the authors only show robust pseudo-labeling or entropy minimization -based adaptation results. Perhaps showing results in both tables can further gives us insights of the performance of these two techniques on different scales of datasets;\n- In Table 2, there’s only marginal improvements of the entropy minimization adaptation compared to the DANN baseline (29.7 -> 28.5; 25.0 -> 23.9). Does this suggest that the performance of self-learning is relatively upper bound by adversarial adaptation approaches?\n- The authors only compare self-learning methods with adversarial adaptation and test-time adaptation baselines but not domain-invariant approaches such as invariant minimization risk (IRM);\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors performed a thorough evaluation of several self-learning techniques such as entropy minimization and pseudo-labeling for source-free domain adaptation performance on a range of datasets (ImageNet-C/R, CIFAR10/STL10, etc) with different network architecture choices (ResNet, DenseNet, WRN, etc). From the experimental results, they show that self-learning methods universally improves domain adaptation performance regardless of model choices and dataset size. They also provided abundant ablation studies on the properties of self-learning and a stability study of the technique. Finally, the authors suggest best practices of self-learning for real deployment.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Include medical data and test self-learning across hospitals;\n- Perform repeated experiments especially for results that are indistinguishable;\n- Show both  robust pseudo-labeling and entropy minimization results in Table 1 & 2;\n- Compare self-learning results with invariant learning techniques;\n", "broader_impact_concerns": "No ethical concerns observed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: At the end of the discussion period, all reviewers were leaning towards acceptance. The authors meticulously responded to all of the reviewers' concerns and added large amounts of convincing experiments for each relevant concern during the rebuttal phase. Overall I believe that this submission fits TMLR well, due to its reproducible claims that are supported by a detailed and extensive set of experiments. 
Decision: Accept (poster)