Year: 2021

Abstract:
We demonstrate that self-learning techniques like entropy minimization and pseudo-labeling are simple and effective at improving performance of a deployed computer vision model under systematic domain shifts. We show consistent improvements irrespective of the model architecture, the pre-training technique or the type of distribution shift. At the same time, self-learning is simple to use in practice because it does not require knowledge or access to the original training data or scheme, is robust to hyperparameter choices, is straight-forward to implement and requires only a few adaptation epochs. This makes selflearning techniques highly attractive for any practitioner who applies machine learning algorithms in the real world. We present state-of-the art adaptation results on CIFAR10-C (8.5% error), ImageNet-C (22.0% mCE), ImageNet-R (17.4% error) and ImageNet-A (14.8% error), theoretically study the dynamics of self-supervised adaptation methods and propose a new classification dataset (ImageNet-D) which is challenging even with adaptation.

heading: 1 INTRODUCTION
text: Deep Neural Networks (DNNs) can reach human-level performance in complex cognitive tasks (Brown et al., 2020; He et al., 2016a; Berner et al., 2019) if the distribution of the test data is sufficiently similar to the training data. However, DNNs are known to struggle if the distribution of the test data is shifted relatively to the training data (Geirhos et al., 2018; Dodge & Karam, 2017).
Two largely distinct communities aim to increase the performance of models under test-time distribution shifts: The robustness community generally considers ImageNet-scale datasets and evaluates models in an ad-hoc scenario. Models are trained on a clean source dataset like ImageNet, using heavy data augmentation (Hendrycks et al., 2020a; Rusak et al., 2020; Geirhos et al., 2019) and/or large-scale pre-training (Xie et al., 2020a; Mahajan et al., 2018). The trained models are not adapted in any way to test-time distribution shifts. This evaluation scenario is relevant for applications in which very different distribution shifts are encountered in an unpredictable order, and hence misses out on the gains of adaptation to unlabeled samples of the target distribution.
The unsupervised domain adaptation (UDA) community often considers smaller-scale datasets and assumes that both the source and the (unlabeled) target dataset are known. Models are trained on both datasets (e.g., with an adversarial domain objective, Ganin et al., 2016) before evaluation on the target domain data. This evaluation scenario provides optimal conditions for adaptation, but the reliance on the source dataset makes UDA more computationally expensive, more impractical and prevents the use of pre-trained models for which the source dataset is unknown or simply too large.
In this work, we consider the source-free domain adaptation setting, a middle ground between the classical ad-hoc robustness setting and UDA in which models can adapt to the target distribution but without using the source dataset (Kundu et al., 2020; Kim et al., 2021; Li et al., 2020; Liang et al., 2020). This evaluation scenario is interesting for many practitioners and applications as an extension of the ad-hoc robustness scenario. It evaluates the possible performance of a deployed model on a systematic, unseen distribution shift at inference time: an embedded computer vision system in an autonomous car should adapt to changes without being trained on all available training data; an image-based quality control software may not necessarily open-source the images it has been trained on, but still has to be adapted to the lighting conditions at the operation location; a computer vision system in a hospital should perform robustly when tested on a scanner different from the training images—importantly, it might not be known at development time which scanner it will be tested on, and it might be prohibited to share images from many hospitals to run UDA.
Can self-learning methods like pseudo-labeling and entropy-minimization also be used in this source-free domain adaptation setting? To answer this question, we perform an extensive study of several self-learning variants, and find consistent and substantial gains in test-time performance across several robustness and out-of-domain benchmarks and a wide range of models and pretraining methods, including models trained with UDA methods that do not use self-learning. We also find that self-learning outperforms state-of-the-art source-free domain adaptation methods, namely Test-Time Training which is based on a self-supervised auxiliary objective and continual training (Sun et al., 2019b), test-time entropy minimization (Wang et al., 2020) and (gradient-free) BatchNorm adaptation (Schneider et al., 2020; Nado et al., 2020). We perform a large number of ablations to study important design choices for self-learning methods in source-free domain adaptation. Furthermore, we show that a variant of pseudo-labeling with a robust loss function consistently outperforms entropy minimization on ImageNet-scale datasets. We theoretically analyze and empirically verify the influence of the temperature parameter in self-learning and provide guidelines how this single parameter should be chosen. Our approach is visualized in Figure 1. We do not consider test-time adaptation in an online setting like is studied e.g., by Zhang et al. (2021), where the model is adapted to one example at a time, and reset after each example.
Related Work. Variants of self-learning have been used for UDA (Berthelot et al., 2021), for example using auxiliary information (Xie et al., 2020b), consistency (Wei et al., 2020; Cai et al., 2021; Prabhu et al., 2021) or confidence (Zou et al., 2019) regularization. The main difference from these works to ours is that they 1) utilize both source and target data for self-learning whereas we only require access to unlabeled target data, 2) train their models from scratch whereas we merely fine-tune pretrained checkpoints on the unlabeled target data, and 3) are generally more complicated than our approach due to using more than one term in the objective function.
Our work is conceptually most similar to virtual adversarial domain adaptation in the fine-tuning phase of DIRT-T (Shu et al., 2018)) and Test-time entropy minimization (TENT; Wang et al., 2020). In contrast to DIRT-T, our objective is simpler and we scale the approach to considerably larger datasets on ImageNet scale. TENT, on the other hand, only evaluated a single method (entropy minimization) on a single vanilla model (ResNet-50) on IN-C. We substantially expand this analysis to show that self-learning almost universally increases test-time performance under distribution shifts, regardless of the type of distribution shift, the model architecture or the pre-training method.
Self-learning has also been applied to UDA for semantic segmentation (Zou et al., 2018), for gradual domain adaptation (Kumar et al., 2020), for semi-supervised learning (Rizve et al., 2021; Mukherjee & Awadallah, 2020), for learning in biased datasets (Chen et al., 2020b) and for automated data annotation (De Sousa Ribeiro et al., 2020). Zoph et al. (2020) show that self-learning outperforms pretraining when stronger data augmentation is used and more labeled data is present. A more detailed discussion of related work alongside with the main differences to our work can be found in Appendix F. Our main contribution beyond these works is to show the effectiveness of self-learning on top of both robust, large scale, and domain adapted models, at scale.

heading: 2 SELF-LEARNING FOR TEST-TIME ADAPTATION
text: Different variants of self-learning have been used in both unsupervised domain adaptation (French et al., 2018; Shu et al., 2018), self-supervised representation learning (Caron et al., 2021), and in semi-supervised learning (Xie et al., 2020a). In a typical self-learning setting a teacher network f t trained on the source domain predicts labels on the target domain. Then, a student model fs is fine-tuned on the predicted labels.
In the following, let f t(x) denote the logits for sample x and let pt(j|x) ≡ σj(f t(x)) denote the probability for class j obtained from a softmax function σj(·). Similarly, fs(x) and ps(j|x) denote the logits and probabilities for the student model fs. For all techniques, one can optionally only admit samples where the probability maxj pt(j|x) exceeds some threshold. We consider three popular variants of self-learning: Pseudo-labeling with hard or soft labels, as well as entropy minimization.
Hard Pseudo-Labeling (Lee, 2013; Galstyan & Cohen, 2007). We generate labels using the teacher and train the student on pseudo-labels i using the standard cross-entropy loss,
`H(x) := − log ps(i|x), i = argmaxj pt(j|x) (1) Usually, only samples with a confidence above a certain threshold are considered for training the student. We test several thresholds but note that thresholding means discarding a potentially large portion of the data which leads to a performance decrease in itself. The teacher is updated after each epoch.
Soft Pseudo-Labeling (Lee, 2013; Galstyan & Cohen, 2007). In contrast to the hard pseudolabeling variant, we here train the student on class probabilities predicted by the teacher,
`S(x) := − ∑ j pt(j|x) log ps(j|x). (2)
Soft pseudo-labeling is typically not used in conjunction with thresholding, since it already incorporates the certainty of the model. The teacher is updated after each epoch.
Entropy Minimization (ENT; Grandvalet & Bengio, 2004). This variant is similar to soft pseudolabeling, but we no longer differentiate between a teacher and student network. It corresponds to an “instantaneous” update of the teacher. The training objective becomes
`E(x) := − ∑ j ps(j|x) log ps(j|x). (3)
Intuitively, self-training with entropy minimization leads to a sharpening of the output distribution for each sample, making the model more confident in its predictions.
Robust Pseudo-Labeling (RPL). Virtually all introduced self-training variants use the standard cross-entropy classification objective. However, the standard cross-entropy loss has been shown to be sensitive to label noise (Zhang & Sabuncu, 2018; Zhang et al., 2017). In the setting of domain adaptation, inaccuracies in the teacher predictions and, thus, the labels for the student, are inescapable, with severe repercussions for training stability and hyperparameter sensitivity as we show in the results.
As a straight-forward solution to this problem, we propose to replace the cross-entropy loss by a robust classification loss designed to withstand certain amounts of label noise (Ghosh et al., 2017; Song et al., 2020; Shu et al., 2020; Zhang & Sabuncu, 2018). A popular candidate is the Generalized Cross Entropy (GCE) loss which combines the noise-tolerant Mean Absolute Error (MAE) loss (Ghosh et al., 2017) with the CE loss. We only consider the hard labels and use the robust GCE loss as the training loss for the student,
i = argmaxj p t(j|x), `GCE(x, i) := q−1(1− ps(i|x)q), (4)
with q ∈ (0, 1]. For the limit case q → 0, the GCE loss approaches the CE loss and for q = 1, the GCE loss is the MAE loss (Zhang & Sabuncu, 2018). We test updating the teacher both after every update step of the student (RPL) and once per epoch (RPLep).

heading: 3 EXPERIMENT DESIGN
text: Datasets. IN-C (Hendrycks & Dietterich, 2019) contains corrupted versions of the 50 000 images in the IN validation set. There are fifteen test and four hold-out corruptions, and there are five severity levels for each corruption. The established metric to report model performance on IN-C is the mean Corruption Error (mCE) where the error is normalized by the AlexNet error, and averaged over all corruptions and severity levels, see Eq. 20, Appendix C.1. IN-R (Hendrycks et al., 2020a) contains 30 000 images with artistic renditions of 200 classes of the IN dataset. IN-A (an, 2019) is composed of 7500 unmodified real-world images on which standard IN-trained ResNet50 (He et al., 2016b) models yield chance level performance. CIFAR10 (Krizhevsky et al., 2009) and STL10 (Coates et al., 2011) are small-scale image recognition datasets with 10 classes each, and training sets of 50 000/5000 images and test sets of 10 000/8000 images, respectively. The digit datasets MNIST (Deng, 2012) and MNIST-M (Ganin et al., 2016) both have 60 000 training and 10 000 test images.
Hyperparameters. The different self-learning variants have a range of hyperparameters such as the learning rate or the stopping criterion. Our goal is to give a realistic estimation on the performance to be expected in practice.. To this end, we optimize hyperparameters for each variant of pseudolabeling on a hold-out set of IN-C that contains four types of image corruptions (“speckle noise”, “Gaussian blur”, “saturate” and “spatter”) with five different strengths each, following the procedure suggested in Hendrycks & Dietterich (2019). We refer to the hold-out set of IN-C as our dev set.
Models for ImageNet-scale datasets. We consider four popular model architectures: ResNet50 (He et al., 2016b), DenseNet161 (Huang et al., 2017), ResNeXt101 (Xie et al., 2017) and EfficientNet-L2 (Tan & Le, 2019) (see Appendix B.1 for details on the used models). For ResNet50, DenseNet and ResNeXt101, we include a simple vanilla version trained on IN only. For ResNet50 and ResNeXt101, we additionally include a state-of-the-art robust version trained with DeepAugment and Augmix (DAug+AM, Hendrycks et al., 2020a)1. For the ResNeXt model, we also include a version that was trained on 3.5 billion weakly labeled images (IG-3.5B, Mahajan et al., 2018). Finally, for EfficientNet-L2 we select the current state of the art on IN-C which was trained on 300 million images from JFT-300M (Chollet, 2017; Hinton et al., 2014) using a noisy studentteacher protocol (Xie et al., 2020a). We validate the IN and IN-C performance of all considered models and match the originally reported scores (Schneider et al., 2020). For EfficientNet-L2, we match IN top-1 accuracy up to 0.1% points, and IN-C up to 0.6% mCE.
Models for CIFAR10/MNIST-scale datasets. For CIFAR10-C experiments, we use two WideResNets (WRN, Zagoruyko & Komodakis, 2016): the first one is trained on CIFAR10 and has a depth of 28 and a width of 10 and the second one is trained with AugMix (Hendrycks et al., 2020b) and has a depth of 40 and a width of 2. The remaining small-scale models are trained with unsupervised domain adaptation (UDA) methods. We propose to regard any UDA method which requires joint training with source and target data as a pre-training step, similar to regular pretraining on IN, and use self-learning on top of the final checkpoint. We consider two popular UDA methods: self-supervised domain adaptation (UDA-SS; Sun et al., 2019a) and Domain-Adversarial Training of Neural Networks (DANN; Ganin et al., 2016). In UDA-SS, the authors seek to align the representations of both domains by performing an auxiliary self-supervised task on both domains simultaneously. In all UDA-SS experiments, we use a WideResNet with a depth of 26 and a width of 16. In DANN, the authors learn a domain-invariant embedding by optimizing a minimax objective. For all DANN experiments except for MNIST→MNIST-M, we use the same WRN architecture as above. For the MNIST→MNIST-M experiment, the training with the larger model diverged and we used a smaller WideResNet version with a width of 2. We note that DANN training involves optimizing a minimax objective and is generally harder to tune.

heading: 4 RESULTS: SELF-LEARNING UNIVERSALLY IMPROVES MODELS
text: Self-learning is a powerful learning scheme, and in the following section we show that it allows to perform test-time adaptation on robustified models, models obtained with large-scale pre-training, as well as domain adapted models across a wide range of datasets and distribution shifts. Our main results on large-scale and small-scale datasets are shown in Tables 1 and 2, respectively. These
1see leaderboard at github.com/hendrycks/robustness
summary tables show final results, and all experiments use the hyperparameters we determined separately on the dev set.
Self-learning is not limited to the distribution shifts in IN-C like compression artefacts or blur. On IN-R, a dataset with renditions, self-learning improves both the vanilla ResNet50 and the EfficientNet-L2 model, the latter of which improves from 23.5% to a new state-of-the art of 17.4% top-1 error. For a vanilla ResNet50, we improve the top-1 error from 63.8% (Hendrycks et al., 2020a) to 54.1%. On IN-A, adapting the EfficientNet-L2 model using self-learning decreases the top-1 error from 16.5% (Xie et al., 2020a) to 14.8% top-1 error, again constituting a new state of the art with test-time adaptation on this dataset.

heading: 5 UNDERSTANDING TEST-TIME ADAPTATION WITH SELF-LEARNING
text: In the following section, we show ablations and interesting insights of using self-learning for testtime adaptation. If not specified otherwise, all ablations are run on the holdout corruptions of IN-C (our dev set) with a vanilla ResNet50.
Note that for Vision Transformers, full model adaptation works better than affine adaptation (see Table 5). We also noticed that on convolutional models with a smaller parameter count like ResNet18, full model adaptation is possible.
Hyperparameters obtained on corruption datasets transfer well to real world datasets. When evaluating models, we select the hyperparameters discussed above (the learning rate and the epoch used for early stopping are the most critical ones) on the holdout set of IN-C. We note that this technique transfers well to IN-R, -A and -D, highlighting the practical value of corruption robustness datasets for adapting models on real distribution shifts.
On IN-D, we performed a control experiment where we selected hyperparameters with leave-oneout cross validation—this selection scheme actually performed worse than IN-C parameter selection (see Appendix D.1).

heading: 6 ADAPTING MODELS ON A WIDER RANGE OF DISTRIBUTION SHIFTS REVEALS LIMITATIONS OF ROBUSTIFICATION AND ADAPTATION METHODS
text: Robustness datasets on ImageNet-scale have so far been limited to a few selected domains (image corruptions in IN-C, image renditions in IN-R, difficult images for ResNet50 classifiers in IN-A). In order to test our approach on a wider range of complex distribution shifts, we re-purpose the dataset from the Visual Domain Adaptation Challenge 2019 (DomainNet, Saenko et al., 2019) as an additional robustness benchmark. This dataset comes with six image styles: Clipart, Real, Infograph, Painting, Quickdraw and Sketch. It has 345 classes in total, of which 164 overlap with IN. To benchmark robustness of IN trained models out of the box, we filter out the classes that cannot be mapped to IN and refer to the smaller version of DomainNet as ImageNet-D (IN-D). We map 463 classes in IN to these 164 IN-D classes, e.g., for an image from the “bird” class in IN-D, we accept all 39 bird classes in IN as valid predictions. We show example images from IN-D in Table 10. The detailed evaluation protocol along with justifications for our design choices and additional analysis are outlined in Appendix D.
The benefit of IN-D over DomainNet is the re-mapping to ImageNet classes which allows robustness researchers to easily benchmark on this dataset, without the need of re-training a model (as common in UDA). To test whether self-learning is helpful for more complex distribution shifts, we adapt a vanilla ResNet50, several robust IN-C models and the EfficientNet-L2 Noisy Student model on IND. We use the same hyperparameters we obtained on IN-C dev for all our IN-D experiments. We show our main results in Table 10.
More robust models perform better on IN-D. Comparing the performance of the vanilla ResNet50 model to its robust DAug+AM variant, we find that the DAug+AM model performs better on all domains, with the most significant gains on the “Clipart”, “Painting” and “Sketch” domains. We show detailed results for all domains and all tested models in Appendix D.2, along with results on IN-C and IN-R for comparison. We find that the best performing models on IN-D are also the
strongest ones on IN-C and IN-R which indicates good generalization capabilities of the techniques combined for these models, given the large differences between the three considered datasets. However, even the best models perform 20 to 30 percentage points worse on IN-D compared to their performance on IN-C or IN-R, indicating that IN-D might be a more challenging benchmark.
All models struggle with some domains of IN-D. The EfficientNet-L2 Noisy Student model obtains the best results on most domains. However, we note that the overall error rates are surprisingly high compared to the model’s strong performance on the other considered datasets (IN-A: 14.8% top-1 error, IN-R: 17.4% top-1 error, IN-C: 22.0% mCE). Even on the “Real” domain closest to clean IN where the EfficientNet-L2 model has a top-1 error of 11.6%, the model only reaches a top-1 error of 29.2%. Self-learning decreases the top1 error on all domains except for “Infograph” and “Quickdraw”. We note that both domains have very high error rates from the beginning and thus hypothesize that the produced pseudo-labels are of low quality.
Error analysis on IN-D. We investigate the errors a ResNet50 model makes on IN-D by analyzing the most frequently predicted classes for different domains to reveal systematic errors indicative of the encountered distribution shifts. We find most errors interpretable: the classifier assigns the label “comic book” to images from the “Clipart” or “Painting” domains, “website” to images from the “Infograph” domain, and “envelope” to images from the “Sketch” domain. Thus, the classifier predicts the domain rather than the class. We find no systematic errors on the “Real” domain which is expected since this domain should be similar to IN. Detailed results on the top-3 most frequently predicted classes for different domains can be found in Fig. 9, Appendix D.4.
IN-D should be used as an additional robustness benchmark. While the error rates on IN-C, -R and -A are at a well-acceptable level for our largest EfficientNet-L2 model after adaptation, IN-D performance is consistently worse for all models. We propose to move from isolated benchmark settings like IN-R (single domain) to benchmarks more common in domain adaptation (like DomainNet) and make IN-D publicly available as an easy to use dataset for this purpose.
Additional experiments and limitations. We discuss additional proof-of-concept implementations on the WILDS benchmark (Koh et al., 2021), BigTransfer (BiT; Chen et al., 2020a) models and on self-learning based UDA models in Appendix E. On WILDS, self-learning is effective for the Camelyon17 task with a systematic shift between train, validation and test sets (each set is comprised of different hospitals), while self-learning fails to improve on tasks with mixed domains.

heading: 7 A SIMPLE MODEL OF STABILITY IN SELF-LEARNING
text: We observed that different self-learning schemes are optimal for small-scale vs. large-scale datasets and varying amount of classes. We reconsider the used loss functions, and unify them into
`(x) = − ∑ j σj ( f t(x) τt ) log ( σj ( fs(x) τs )) ,
f t(x) = { f(x), entropy minimization sg(f(x)), pseudo-labeling.
(5)
We introduced student and teacher temperature τs and τt as parameters in the softmax function and the stop gradient operation sg. Caron et al. (2021) fixed τs and varied τt during training,
and empirically found an upper bound for τt above which the training was no longer stable. To better understand such behavior, we study the learning dynamics of the loss function in equation 5 theoretically in a simple two-datapoints, two-classes model with linear student and teacher networks fs(x) = x>ws and f t(x) = x>wt defined in Appendix A.1. Gradient descent with stop gradient corresponds to hard pseudo-labeling in the limit τt → 0 and to soft pseudo-labeling when τs = τt = 1. Gradient descent without stop gradient, i.e., setting ws = wt = w corresponds to entropy minimization. We obtain the following result:
Proposition 1 (Collapse in the two-point model). The student and teacher networks ws and wt trained with stop gradient does not collapse to the trivial representation ∀x : x>ws = 0,x>wt = 0 if τs > τt. The network w trained without stop gradient does not collapse if τs > τt/2. Proof. see § A.2.
We validate the proposition on a simulated two datapoint toy dataset, as well as on the CIFARC dataset and outline the results in Figure 2. In general, the size and location of the region where collapse is observed in the simulated model also depends on the initial conditions, the learning rate and the optimization procedure. An in depth discussion, as well as additional simulations are given in the Appendix. In practice, the result suggests that student temperatures should exceed the teacher temperatures for pseudo-labeling, and student temperatures should exceed half the teacher temperature for entropy minimization.
Entropy minimization with standard temperatures (τs = τt = 1) and hard pseudo-labeling (τt → 0) are hence stable. The two-point learning dynamics vanish for soft pseudo-labeling with τs = τt,
suggesting that one would have to analyze a more complex model with more data points. While this does not directly imply that the learning is unstable at this point, we empirically observe that both entropy minimization and hard labeling outperform soft-labeling in practice.

heading: 8 CONCLUSION
text: We evaluated and analysed how self-learning, an essential component in many unsupervised domain adaptation and self-supervised pre-training techniques, can be applied for adaptation to both small and large-scale image recognition problems common in robustness research. We demonstrated new state-of-the-art adaptation results with the EfficientNet-L2 model on the benchmarks ImageNet-C, -R, and -A, and introduced a new benchmark dataset (ImageNet-D) which remains challenging even after adaptation. Our theoretical analysis shows the influence of the temperature parameter in the self-learning loss function on the training stability and provides guidelines how to choose a suitable value. Self-learning universally improves test-time performance under diverse, but systematic distribution shifts irrespective of the architecture or pre-training method. We hope that our work encourages both researchers and practitioners to use self-learning if their data distribution shifts.
Reproducibility Statement We attempted to make our work as reproducible as possible: We mostly used pre-trained models which are publicly available and we denoted the URL addresses of all used checkpoints; for the checkpoints that were necessary to retrain, we report the Github directories with the source code and used an official or verified reference implementation when available. We report all used hyperparameters in the Appendix and will release our code upon acceptance of the paper.

heading: A A TWO-POINT MODEL OF SELF-LEARNING

heading: A.1 DEFINITION OF THE TWO-POINT MODEL
text: To understand the learning dynamics and properties of different loss functions and their hyperparameters, we propose a simple model of self-learning, both for entropy minimization and pseudo-labeling.
A student network ws ∈ Rd and a teacher network wt ∈ Rd are trained on N data points {xi}Ni=1 with the cross-entropy loss function L defined as
L = − N∑ i=1 `(xi) = − N∑ i=1 ( σt(x > i w t) log σs(x > i w s) + σt(−x>i wt) log σs(−x>i ws) ) ,
where σt(z) = 1
1 + e−z/τt and σs(z) =
1
1 + e−z/τs .
(6)
Here τs and τt denote the student and teacher temperature parameters. With stop gradient, student and teacher evolve in time according to
ẇs = −∇wsL ( ws,wt ) , ẇt = α(ws −wt), (7)
where α is the learning rate of the teacher. Without stop gradient, student and teacher are set equal to each other, and they evolve as
ẇ = −∇wL(w), where ws = wt = w. (8) We restrict the theoretical analysis to the time evolution of the components of ws,t in direction of two data points xk and xl, y s,t k ≡ x>k ws,t and y s,t l ≡ x>l ws,t. All other components y s,t i with i 6= k, l are neglected to reduce the dimensionality of the equation system. It turns out that the resulting model captures the neural network dynamics quite well despite the drastic simplification of taking only two data points into account (see Figure 2).
with stop gradient: ẏsk = −x>k∇ws (`(xk) + `(xl)) , ẏsl = −x>l ∇ws (`(xk) + `(xl)) , ẏtk = α(y t k − ysk), ẏtl = α(ytl − ysl ),
without stop gradient: ẏk = −x>k∇w (`(xk) + `(xl)) , ẏl = −x>l ∇w (`(xk) + `(xl)) . (9)

heading: A.2 PROOF OF PROPOSITION 1
text: Learning dynamics with stop gradient. Computing the stop gradient evolution defined in equation 7 explicitly yields
ẇs = −∇wsL = 1
τs N∑ i=1 ( σt(x > i w t)σs(−x>i ws)− σt(−x>i wt)σs(x>i ws) ) xi
ẇt = α(ws −wt) (10)
The second equality uses the well-known derivative of the sigmoid function, ∂zσ(z) = σ(z)σ(−z). The equation system of 2d nonlinear, coupled ODEs for ws ∈ Rd and wt ∈ Rd in equation 10 is analytically difficult to analyze. Instead of studying the ODEs directly, we act on them with the data points x>k , k = 1, . . . , N , and investigate the dynamics of the components x > k w s,t ≡ ys,tk :
ẏsk = 1
τs N∑ i=1 ( x>i xk ) ( σt(y t i)σs(−ysi )− σt(−yti)σs(ysi ) ) ẏtk = α(y s k − ytk).
(11)
The learning rate of each mode ysk is scaled by (x > k xi) which is much larger for i = k than for i 6= k in high-dimensional spaces. In the two-point approximation, we consider only the two (in absolute
value) largest terms i = k, l for a given k in the sum in equation 11. Any changes that ys,tk (t) and ys,tl (t) might induce in other modes y s,t i (t) are neglected, and so we are left with only four ODEs:
ẏsk = 1
τs ‖xk‖2
( σt(y t k)σs(−ysk)− σt(−ytk)σs(ysk) ) + 1
τs (x>k xl)
( σt(y t l )σs(−ysl )− σt(−ytl )σs(ysl ) ) ,
ẏsl = 1
τs ‖xl‖2
( σt(y t l )σs(−ysl )− σt(−ytl )σs(ysl ) ) + 1
τs (x>k xl)
( σt(y t k)σs(−ysk)− σt(−ytk)σs(ysk) ) ẏtk =α(y s k − ytk), ẏtl = α(ysl − ytl ).
(12)
The fixed points of equation 12 satisfy
ẏsk = ẏ s l = ẏ t k = ẏ t l = 0. (13)
For α > 0, requiring ẏtk = ẏ t l = 0 implies that y s k = y t k and y s l = y t l . For τs = τt, the two remaining equations ẏsk = ẏ s l = 0 vanish automatically so that there are no non-trivial two-point learning dynamics. For τs 6= τt, there is a fixed point at ys,tk = y s,t l = 0 since at this point, each bracket in equation 12 vanishes individually:
σt(yk,l)σs(−yk,l)− σs(−yk,l)σt(yk,l) ∣∣∣∣ yk,l=0 = 1 4 − 1 4 = 0. (14)
At the fixed point ys,tk = y s,t l = 0, w s and wt are orthogonal to both xk and xl and hence classification fails. If this fixed point is stable, ws and wt will stay at the fixed point once they have reached it, i.e. the model collapses. The fixed point is stable when all eigenvalues of the Jacobian J of the ODE system equation 12 evaluated at ys,tk = y s,t l = 0 are negative. This is the case whenever τs < τt:
J ∣∣∣∣ ys,tk =y s,t l =0 =  ‖xk‖2 4 ( 1 τt − 1τs ) (x>k xl) 4 ( 1 τt − 1τs ) 0 0 (x>k xl) 4 ( 1 τt − 1τs ) ‖xl‖2 4 ( 1 τt − 1τs ) 0 0
α 0 −α 0 0 α 0 −α  , eigenvalues: λ1 = λ2 = −α < 0,
λ3,4 = 1
8
( 1
τt − 1 τs )±√‖xk‖4 + ‖xl‖4 − 2‖xk‖2‖xl‖2 + 4(x>k xl)2︸ ︷︷ ︸ ≤‖xk‖2+‖xl‖2 +‖xk‖2 + ‖xl‖2  ︸ ︷︷ ︸
≥0 with equality if xk=±xl (15)
To sum up, training with stop gradient and τs > τt avoids a collapse of the two-point model to the trivial representation ys,tk = y s,t l = 0 since the fixed point is not stable in this parameter regime.
Learning dynamics without stop gradient Without stop gradient, we set wt = ws ≡ w which leads to an additional term in the gradient:
ẇ = −∇wL = 1
τs N∑ i=1 ( σt(x > i w)σs(−x>i w)− σt(−x>i w)σs(x>i w) ) xi
+ 1
τt N∑ i=1 σt(x > i w)σt(−x>i w) ( log σs(x > i w)− log σs(−x>i w) )︸ ︷︷ ︸ =log ((1+eyi/τs )/(1+e−yi/τs ))=yi/τs xi.
(16)
As before, we focus on the evolution of the two components yk = w>xk and yl = w>xl. ẏk =‖xk‖2 ( 1
τs (σt(yk)σs(−yk)− σt(−yk)σs(yk)) +
1 τt σt(yk)σt(−yk)yk ) + (x>k xl) ( 1
τs (σt(yl)σs(−yl)− σt(−yl)σs(yl)) +
1
τsτt σt(yl)σt(−yl)yl ) ẏl =‖xl‖2 ( 1
τs (σt(yl)σs(−yl)− σt(−yl)σs(yl)) +
1 τt σt(yl)σt(−yl)yl ) + (x>k xl) ( 1
τs (σt(yk)σs(−yk)− σt(−yk)σs(yk)) +
1
τsτt σt(yk)σt(−yk)yk
) (17)
There is a fixed point at yk = yl = 0 where each bracket in equation 17 vanishes individually,
1 τs (σt(yk,l)σs(−yk,l)− σt(−yk,l)σs(yk,l)) + 1 τsτt σt(yk,l)σt(−yk,l)yk,l ∣∣∣∣ yk,l = 0. (18)
The Jacobian of the ODE system in equation 17 and its eigenvalues evaluated at the fixed point are given by
J ∣∣∣∣ yk=yl=0 =  ‖xk‖24τs ( 2τt − 1τs) (x>k xl)4τs ( 2τt − 1τs) (x>k xl) 4τs ( 2 τt − 1τs ) ‖xl‖2 4τs ( 2 τt − 1τs ) 
λ1,2 = 1
8τs
( 2
τt − 1 τs )±√‖xk‖4 + ‖xl‖4 − 2‖xk‖2‖xl‖2 + 4(x>k xl)2︸ ︷︷ ︸ ≤‖xk‖2+‖xl‖2 +‖xk‖2 + ‖xl‖2  ︸ ︷︷ ︸
≥0 with equality if xk=±xl
.
(19) Hence the fixed point is unstable when τs > τt/2 and thus the model without stop gradient does not collapse onto yk = yl = 0 in this regime.

heading: A.3 SIMULATION OF THE TWO-POINT MODEL
text: For visualization purposes in the main paper, we set ws = wt = [0.5, 0.5]> and train the model using instant gradient updates on the dataset with points x1 = [1, 0] and x2 = [0,−1] using SGD with learning rate 0.1 and momentum 0.9. We varied student and teacher temperatures on a log-scale with 250 points from 10−3 to 10. Qualitatively similar results can be obtained without momentum training, at higher learning rates (most likely due to the implicit learning rate scaling introduced by the momentum term).
Note that the temperature scales for observing the collapse effect depend on the learning rate, and the exact training strategy—lower learning rates can empirically prevent the model from collapsing and shift the convergence region. The result in Figure 2 will hence depend on the exact choice of learning rate (which is currently not considered in our continuous time evolution theory), while the predicted region without collapse is robust to details of the optimization.
To visualize the impact of different hyperparameters, we show variants of the two point model with different learning rates using gradient descent with (Figure 3) and without momentum (Figure 4), and with different start conditions (Figure 5), which all influence the regions where the model degrades, but not the stable regions predicted by our theory.

heading: B ADDITIONAL INFORMATION ON USED MODELS

heading: B.1 DETAILS ON ALL HYPERPARAMETERS WE TESTED FOR DIFFERENT MODELS
text: For all models except EfficientNet-L2, we adapt the batch norm statistics to the test domains following (Schneider et al., 2020). We do not expect significant gains for combining EfficientNet-L2 with batch norm adaptation: as demonstrated in (Schneider et al., 2020), models trained with large amounts of weakly labeled data do not seem to benefit from batch norm adaptation.
ResNet50 models We use a vanilla ResNet50 model and compare soft- and hard-labeling against entropy minimization and robust pseudo-labeling. To find optimal hyperparameters for all methods, we perform an extensive evaluation and test (i) three different adaptation mechanisms (ii) several learning rates 1.0× 10−4, 1.0× 10−3, 1.0× 10−2 and 5.0× 10−2, (iii) the number of training epochs and (iv) updating the teacher after each epoch or each iteration. For all experiments, we use a batch size of 128. The hyperparameter search is performed on IN-C dev. We then use the optimal hyperparameters to evaluate the methods on the IN-C test set.
ResNeXt101 models The ResNeXt101 model is considerably larger than the ResNet50 model and we therefore limit the number of ablation studies we perform for this architecture. Besides a baseline, we include a state-of-the-art robust version trained with DeepAugment+Augmix (DAug+AM, Hendrycks et al., 2020a) and a version that was trained on 3.5 billion weakly labeled images (IG-3.5B, Mahajan et al., 2018). We only test the two leading methods on the ResNeXt101 models (ENT and RPL). We vary the learning rate in same interval as for the ResNet50 model but scale it down linearly to account for the smaller batch size of 32. We only train the affine batch normalization parameters because adapting only these parameters leads to the best results on ResNet50 and is much more resource efficient than adapting all model parameters. Again, the hyperparameter search is performed only on the development corruptions of IN-C. We then use the optimal hyperparameters to evaluate the methods on the IN-C test set.
EfficientNet-L2 models The current state of the art on IN, IN-C, IN-R and IN-A is an EfficientNet-L2 trained on 300 million images from JFT-300M (Chollet, 2017; Hinton et al., 2014) using a noisy student-teacher protocol (Xie et al., 2020a). We adapt this model for only one epoch due to resource constraints. During the hyperparameter search, we only evaluate three corruptions on the IN-C development set2 and test the learning rates 4.6× 10−2, 4.6× 10−3, 4.6× 10−4 and 4.6× 10−5. We use the optimal hyperparameters to evaluate ENT and RPL on the full IN-C test set (with all severity levels).
UDA-SS models We trained the models using the scripts from the official code base at github. com/yueatsprograms/uda release. We used the provided scripts for the cases: (a) source: CIFAR10, target: STL10 and (b) source: MNIST, target: MNIST-M. For the case (c) source: CIFAR10, target: CIFAR10-C, we used the hyperparameters from case (a) since this case seemed to be the closest match to the new setting. We think that the baseline performance of the UDA-SS models can be further improved with hyperparameter tuning.
DANN models To train models with the DANN-method, we used the PyTorch implementation of this paper at https://github.com/fungtion/DANN py3. The code base only provides scripts and hyperparameters for the case (b) source: MNIST, target: MNIST-M. For the cases (a) and (c), we used the same optimizer and trained the model for 100 epochs. We think that the baseline performance of the DANN models can be further improved with hyperparameter tuning.
Preprocessing For IN, IN-R, IN-A and IN-D, we resize all images to 256 × 256 px and take the center 224 × 224 px crop. The IN-C images are already rescaled and cropped. We center and re-scale the color values with µRGB = [0.485, 0.456, 0.406] and σRGB = [0.229, 0.224, 0.225]. For the EfficientNet-L2, we follow the procedure in Xie et al. (2020a) and rescale all inputs to a resolution of 507× 507 px and then center-crop them to 475× 475 px.
2We compare the results of computing the dev set on the 1, 3 and 5 severities versus the 1, 2, 3, 4 and 5 severities on our ResNeXt101 model in the Supplementary material.

heading: B.2 FULL LIST OF USED MODELS
text: ImageNet scale models ImageNet trained models (ResNet50, DenseNet161, ResNeXt) are taken directly from torchvision (Marcel & Rodriguez, 2010). The model variants trained with DeepAugment and AugMix augmentations (Hendrycks et al., 2020b;a) are taken from https: //github.com/hendrycks/imagenet-r. The weakly-supervised ResNeXt101 model is taken from the PyTorch Hub. For EfficientNet (Tan & Le, 2019), we use the PyTorch re-implementation available at https://github.com/rwightman/gen-efficientnet-pytorch. This is a verified re-implementation of the original work by Xie et al. (2020a). We verify the performance on ImageNet, yielding a 88.23% top-1 accuracy and 98.546% top-5 accuracy which is within 0.2% points of the originally reported result (Xie et al., 2020a). On ImageNet-C, our reproduced baseline achieves 28.9% mCE vs. 28.3% mCE originally reported by Xie et al. (2020a). As noted in the re-implementation, this offset is possible due to minor differences in the pre-processing. It is possible that our adaptation results would improve further when applied on the original codebase by Xie et al..
Small scale models We train the UDA-SS models using the original code base at github.com/ yueatsprograms/uda release, with the hyperparameters given in the provided bash scripts. For our DANN experiments, we use the PyTorch implementation at github.com/fungtion/DANN py3. We use the hyperparameters in the provided bash scripts.
The following Table 11 contains all models we evaluated on various datasets with references and links to the corresponding source code.

heading: C DETAILED AND ADDITIONAL RESULTS ON IN-C

heading: C.1 DEFINITION OF THE MEAN CORRUPTION ERROR (MCE)
text: The established performance metric on IN-C is the mean Corruption Error (mCE), which is obtained by normalizing the model’s top-1 errors with the top-1 errors of AlexNet across the C=15 test corruptions and S=5 severities:
mCE(model) = 1
C C∑ c=1 ∑S s=1 err model c,s∑S s=1 errAlexNetc,s . (20)
The AlexNet errors used for normalization are shown in Table 12.

heading: C.2 DETAILED RESULTS FOR TUNING EPOCHS AND LEARNING RATES
text: We tune the learning rate for all models and the number of training epochs for all models except the EfficientNet-L2. In this section, we present detailed results for tuning these hyperparameters for all considered models. The best hyperparameters that we found in this analysis, are summarized in Table 17.
Table 13: mCE in % on the IN-C dev set for ENT and RPL for different numbers of training epochs when adapting the affine batch norm parameters of a ResNet50 model.
criterion ENT RPL lr 10−4 10−3 10−2 10−4 10−3 10−2 epoch
0 60.2 60.2 60.2 60.2 60.2 60.2 1 54.3 50.0 72.5 57.4 51.1 52.5 2 52.4 50.9 96.5 55.8 49.6 57.4 3 51.5 51.0 112.9 54.6 49.2 64.2 4 51.0 52.4 124.1 53.7 49.0 71.0 5 50.7 53.5 131.2 52.9 48.9 76.3 6 50.7 53.5 131.2 52.9 48.9 76.3
Table 14: mCE (↘) in % on the IN-C dev set for different learning rates for EfficientNetL2. We favor q = 0.8 over q = 0.7 due to slightly improved robustness to changes in the learning rate in the worst case error setting.
lr (4.6 ×) base 10−3 10−4 10−5 10−6
ENT 25.5 87.8 25.3 22.2 24.1 RPLq=0.7 25.5 60.3 21.3 23.3 n/a RPLq=0.8 25.5 58.2 21.4 23.4 n/a
Table 15: mCE in % on IN-C dev for entropy minimization for different learning rates and training epochs for ResNeXt101. (div.=diverged)
ENT Baseline IG-3.5B DAug+AM lr 2.5× 1e-4 1e-3 5e-3 1e-4 1e-3 5e-3 1e-4 1e-3 5e-3 epoch
BASE 53.6 53.6 53.6 47.4 47.4 47.4 37.4 37.4 37.4 1 43.0 92.2 div. 40.9 40.4 58.6 35.4 46.4 div. 2 44.8 118.4 div. 39.8 41.5 69.5 35.5 90.8 div. 3 45.4 131.9 div. 39.3 42.6 76.1 35.5 122.5 div. 4 46.7 div. div. 39.1 44.2 84.3 35.6 133.8 div.
Table 16: mCE in % on IN-C dev for robust pseudolabeling for different learning rates and training epochs for ResNeXt101. (div.=diverged)
RPL Baseline IG-3.5B DAug+AM lr 2.5× 1e-4 1e-3 5e-3 1e-4 1e-3 5e-3 1e-4 1e-3 5e-3 epoch
BASE 53.6 53.6 53.6 47.4 47.4 47.4 37.4 37.4 37.4 1 43.4 51.3 div. 45.0 39.9 43.6 35.3 35.1 79.1 2 42.3 63.2 div. 43.4 39.3 48.2 34.9 35.6 121.2 3 42.0 72.6 div. 42.4 39.4 52.9 34.7 40.1 133.5 4 42.0 72.6 div. 42.4 39.4 52.9 34.7 40.1 133.5

heading: C.3 DETAILED RESULTS FOR ALL IN-C CORRUPTIONS
text: We outline detailed results for all corruptions and models in Table 18. Performance across the severities in the dataset is depicted in Figure 6. All detailed results presented here are obtained by following the model selection protocol outlined in the main text.
RN50
RNx101
RNx101 IG-3.5B
RNx101 DeepAug+Augmix
Noisy Student L2

heading: C.4 DETAILED RESULTS FOR THE CIFAR10-C AND UDA ADAPTATION
text: C.5 ABLATION OVER THE HYPERPARAMETER q FOR RPL
For RPL, we must choose the hyperparameter q. We performed an ablation study over q and show results in Table 21, demonstrating that RPL is robust to the choice of q, with slight preference to higher values. Note: In the initial parameter sweep for this paper, we only compared q = 0.7 and q = 0.8. Given the result in Table 21, it could be interesting to re-run the models in Table 1 of the main paper with q = 0.9, which could yield another (small) improvement in mCE.
C.6 SELF-TRAINING OUTPERFORMS CONTRASTIVE TEST-TIME TRAINING (SUN ET AL., 2019B)
Sun et al. (2019b) use a ResNet18 for their experiments on ImageNet and only evaluate their method on severity 5 of IN-C. To enable a fair comparison, we trained a ResNet18 with both hard labeling and RPL and compare the efficacy of both methods to Test-Time Training in Table 22. For both hard labeling and RPL, we use the hyperparameters we found for the vanilla ResNet50 model and thus, we expect even better results for hyperparameters tuned on the vanilla ResNet18 model and following our general hyperparameter search protocol.
While all methods (self-learning and TTT) improve the performance over a simple vanilla ResNet18, we note that even the very simple baseline using hard labeling already outperfoms Test-Time Training; further gains are possible with RPL. The result highlights the importance of simple baselines (like self-learning) when proposing new domain adaptation schemes. It is likely that many established DA techniques more complex than the basic self-learning techniques considered in this work will even further improve over TTT and other adaptation approaches developed exclusively in robustness settings.
vanilla ResNet18 98.8 98.2 99.0 88.6 91.3 88.8 82.4 89.1 83.5 85.7 48.7 96.6 83.2 76.9 70.4 85.4 Test-Time Training 73.7 71.4 73.1 76.3 93.4 71.3 66.6 64.4 81.3 52.4 41.7 64.7 55.7 52.2 55.7 66.3 hard PL, (1 epoch) 73.2 70.8 73.6 76.5 75.6 63.9 56.1 59.0 65.9 48.4 39.7 85.2 50.4 47.0 51.5 62.5 RPL (4 epochs) 71.3 68.3 71.7 76.2 75.6 61.5 54.4 56.9 67.1 47.3 39.3 93.2 48.9 45.7 50.4 61.9

heading: C.7 EFFECT OF BATCH SIZE AND LINEAR LEARNING RATE SCALING
text: How is self-learning performance affected by batch size constraints? We compare the effect of different batch sizes and linear learning rate scaling. In general, we found that affine adaptation experiments on ResNet50 scale can be run with batch size 128 on a Nvidia V100 GPU (16GB), while only batch size 96 experiments are possible on RTX 2080 GPUs.
The results in Table 23 show that for a ResNet50 model, higher batch size yields a generally better performance.

heading: C.8 PERFORMANCE OVER DIFFERENT SEEDS IN A RESNET50 ON IMAGENET-C
text: To limit the amount of compute, we ran RPL and ENT for our vanilla ResNet50 model three times with the optimal hyperparameters. The averaged results, displayed as “mean (unbiased std)” are:

heading: C.9 SELF-LEARNING AS CONTINUOUS TEST-TEST ADAPTATION
text: We test our method on continuous test-time adaptation where the model adapts to a continuous stream of data from the same domain. In Fig. 7, we display the error of the Noisy Student L2 model while it is being adapted to ImageNet-C and ImageNet-R. The model performance improves as the model sees more data from the new domain. We differentiate continuous test-time adaptation from the online test-time adaptation setting (Zhang et al., 2021) where the model is adapted to each test sample individually, and reset after each test sample.

heading: D DETAILED AND ADDITIONAL RESULTS ON IN-D

heading: D.1 EVALUATION PROTOCOL ON IN-D
text: The domains in IN-D differ in terms of their difficulty for the studied models. Therefore, to calculate an aggregate score, we propose normalizing the error rates by the error achieved by AlexNet on the respective domains to calculate the mean error, following the approach in Hendrycks & Dietterich (2019) for IN-C. This way, we obtain the aggregate score mean Domain Error (mDE) by calculating the mean over different domains,
DEfd = Efd
EAlexNetd , mDE =
1
D D∑ d=1 Efd , (21)
where Efd is the top-1 error of a classifier f on domain d.
Leave-one-out-cross-validation For all IN-D results we report in this paper, we chose the hyperparameters on the IN-C dev set. We tried a different model selection scheme on IN-D as a control experiment with “Leave one out cross-validation” (L1outCV): with a round-robin procedure, we choose the hyperparameters for the test domain on all other domains. We select the same hyperparameters as when tuning on the “dev” set: For the ResNet50 model, we select over the number of training epochs (with a maximum of 7 training epochs) and search for the optimal learning rate in the set [0.01, 0.001, 0.0001]. For the EfficientNet-L2 model, we train only for one epoch as before and select the optimal learning rate in the set [4.6× 10−3, 4.6× 10−4, 4.6× 10−5, 4.6× 10−6]. This model selection leads to worse results both for the ResNet50 and the EfficientNetL2 models, highlighting the robustness of our model selection process, see Table 25.

heading: D.2 DETAILED RESULTS FOR ROBUST RESNET50 MODELS ON IN-D
text: We show detailed results for all models on IN-D for vanilla evaluation (Table 26) BN adaptation (Table 27), RPLq=0.8 (Table 28) and ENT(Table 29). For RPLq=0.8 and ENT, we use the same hyperparameters that we chose on our IN-C ‘dev’ set. This means we train the models for 5 epochs with RPLq=0.8 and for one epoch with ENT.
We evaluate the pre-trained and public checkpoints of SIN (Geirhos et al., 2019), ANT (Rusak et al., 2020), ANT+SIN (Rusak et al., 2020), AugMix (Hendrycks et al., 2020b), DeepAugment (Hendrycks et al., 2020a) and DeepAug+Augmix (Hendrycks et al., 2020a) in the following tables.
The summary results for all models are shown in Table 30.
We show the top-1 error for the different IN-D domains versus training epochs for a vanilla ResNet50 in Fig. 8. We indicate the epochs 1 and 5 at which we extract the errors with dashed black lines.

heading: D.3 DETAILED RESULTS FOR THE EFFICIENTNET-L2 NOISY STUDENT MODEL ON IN-D
text: We show the detailed results for the EfficientNet-L2 Noisy Student model on IN-D in Table 31.

heading: D.4 DETAILED RESULTS ON THE ERROR ANALYSIS ON IN-D
text: Frequently predicted classes We analyze the most frequently predicted classes on IN-D by a vanilla ResNet50 and show the results in Fig. 9. We make several interesting observations: First, we find most errors interpretable: it makes sense that a ResNet50 assigns the label “comic book” to images from the “clipart” or “painting” domains, or “website” to images from the “infograph” domain, or “envelope” to images from the “sketch” domain. Second, on the hard domain “quickdraw”, the ResNet50 mostly predicts non-sensical classes that are not in IN-D, mirroring its almost chance performance on this domain. Third, we find no systematic errors on the “real” domain which is expected since this domain should be similar to IN.
Filtering predictions on IN-D that cannot be mapped to ImageNet-D We perform a second analysis: We filter the predicted labels according to whether they can be mapped to IN-D and report the filtered top-1 errors as well as the percentage of filtered out inputs in Table 32. We note that for the domains “infograph” and “quickdraw”, the ResNet50 predicts labels that cannot be mapped to IN-D in over 70% of all cases, highlighting the hardness of these two domains.
Filtering labels and predictions on IN that cannot be mapped to ImageNet-D To test for possible class-bias effects, we test the performance of a ResNet50 model on IN classes that can be mapped to IN-D and report the results in Table 32.
First, we map IN labels to IN-D to make the setting as similar as possible to our experiments on IN-D and report the top-1 error (12.1%). This error is significantly lower compared to the top-1 error a ResNet50 obtains following the standard evaluation protocol (23.9%). This can be explained by the simplification of the task: While in IN there are 39 bird classes, these are all mapped to the same hierarchical class in IN-D. Therefore, the classes in IN-D are more dissimilar from each other than in IN. Additionally, there are only 164 IN-D classes compared to the 1000 IN classes, raising the chance level prediction.
If we further only accept predictions that can be mapped to IN-D, the top-1 error is slightly increased to 13.4%. In total, about 52.7% of all images in the IN validation set cannot be mapped to IN-D.

heading: D.5 TOP-1 ERROR ON IN-D FOR ALEXNET
text: We report the top-1 error numbers on different IN-D as achieved by AlexNet in Table 33. We used these numbers for normalization when calculating mDE.

heading: E ADDITIONAL EXPERIMENTS

heading: E.1 BEYOND IMAGENET CLASSES: SELF-LEARNING ON WILDS
text: The WILDS benchmark (Koh et al., 2021) is comprised of ten tasks to test domain generalization, subpopulation shift, and combinations thereof. In contrast to the setting considered here, many of the datasets in WILDS mix several 10s or 100s domains during test time.
The Camelyon17 dataset in WILDS contains histopathological images, with the labels being binary indicators of whether the central 32×32 region contains any tumor tissue; the domain identifies the hospital that the patch was taken from. Camelyon17 contains three different test splits with different domains and varying difficulty levels. For evaluation, we took the pretrained checkpoint from worksheets.codalab.org/worksheets/0x00d14c55993548a1823a710642f6d608 (camelyon17 erm densenet121 seed0) for a DenseNet121 model (Huang et al., 2017) and verified the reported baseline performance numbers. We adapt the models using ENT or RPL for a maximum of 10 epochs using learning rates {3×10−5, 3×10−4, . . . 3×10−1}. The best hyperparameter is selected according to OOD Validation accuracy.
The RxRx1 dataset in WILDS contains RGB images of cells obtained by fluorescent microscopy, with the labels indicating which of the 1,139 genetic treatments (including no treatment) the cells received; the domain identifies the batch in which the imaging experiment was run. The RxRx1 dataset contains three test splits, however, unlike Camelyon17, in all of the splits the domains are mixed. For evaluation, we took the pretrained checkpoint from worksheets.codalab.org/bundles/ 0x7d33860545b64acca5047396d42c0ea0 for a ResNet50 model and verified the reported baseline performance numbers. We adapt the models using ENT or RPL for a maximum of 10 epochs using base learning rates {6.25× 10−6, 6.25× 10−5, . . . 6.25× 10−2}, which are scaled to the admissible batch size for single GPU adaptation using linear scaling. The best hyperparameter is selected according to OOD Validation accuracy.
The FMoW dataset in WILDS contains RGB satellite images, with the labels being one of 62 building or land use categories; the domain specifies the year in which the image was taken and its geographical region (Africa, the Americas, Oceania, Asia, or Europe). The FMoW dataset contains four test splits for different time periods, for which all regions are mixed together. For evaluation, we took the pretrained checkpoint from //worksheets.codalab.org/ bundles/0x20182ee424504e4a916fe88c91afd5a2 for a DenseNet121 model and verified the reported baseline performance numbers. We adapt the models using ENT or RPL for a maximum of 10 epochs
using learning rates {5.0× 10−6, 5.0× 10−5, . . . 5.0× 10−2}. The best hyperparameter is selected according to OOD Validation accuracy.
While we see improvements on Camelyon17, neither BN adaptation nor self-learning can improve performance on RxRx1 or FMoW. Initial experiments on PovertyMap and iWildsCam also do not show improvements with self-learning. We hypothesize that the reason lies in the mixing of the domains: Both BN adaptation and our self-learning methods work best on systematic domain shifts. These results support our claim that self-learning is effective, while showing the important limitation when applied to more diverse shifts.

heading: E.2 SMALL IMPROVEMENTS ON BIGTRANSFER MODELS WITH GROUP NORMALIZATION LAYERS
text: We evaluated BigTransfer models (Kolesnikov et al., 2020) provided by the timm library (Wightman, 2019). A difference to the ResNet50, ResNeXt101 and EfficientNet models is the use of group normalization layers, which might influence the optimal method for adaptation—for this evaluation, we followed our typical protocol as performed on ResNet50 models, and used affine adaptation.
For affine adaptation, a distilled BigTransfer ResNet50 model improves from 49.6 % to 48.4 % mCE on the ImageNet-C development set, and from 55.0 % to 54.4 % mCE on the ImageNet-C test set when using RPL (q = 0.8) for adaptation, at learning rate 7.5× 10−4 at batch size 96 after a single adaptation epoch. Entropy minimization did not further improve results on the ImageNet-C test set. An ablation over learning rates and epochs on the dev set is shown in Table 35, the final results are summarized in Table 36.

heading: E.3 CAN SELF-LEARNING IMPROVE OVER SELF-LEARNING BASED UDA?
text: An interesting question is whether test-time adaptation with self-learning can improve upon selflearning based UDA methods. To investigate this question, we build upon French et al. (2018) and their released code base at github.com/Britefury/self-ensemble-visual-domain-adapt. We trained the Baseline models from scratch using the provided shell scripts with the default hyperparameters and verified the reported performance. For adaptation, we tested BN adaptation, ENT, RPL, as well as continuing to train in exactly the setup of French et al. (2018), but without the supervised loss. For the different losses, we adapt the models for a maximum of 10 epochs using learning rates {1× 10−5, 1× 10−4, . . . , 1× 10−1}. Note that for this experiment, in contrast to any other result in this paper, we purposefully do not perform proper hyperparameter selection based on a validation dataset—instead we report the best accuracy across all tested epochs and learning rates to give an upper bound on the achievable performance for test-time adaptation.
As highlighted in Table 37, none of the four tested variants is able to meaningfully improve over the baseline, corroborating our initial hypothesis that self-learning within a full UDA setting is the optimal strategy, if dataset size and compute permits. On the other hand, results like the teacher
refinement step in DIRT-T (Shu et al., 2018) show that with additional modifications in the loss function, it might be possible to improve over standard UDA with additional adaptation at test time.

heading: F DETAILED DISCUSSION OF RELATED WORK
text: Self-learning for domain adaptation Xie et al. (2020b) introduce “In-N-Out” which uses auxiliary information to boost both in- and out-of-distribution performance. AdaMatch (Berthelot et al., 2021) builds upon FixMatch (Sohn et al., 2020) and can be used for the tasks of unsupervised domain adaptation, semi-supervised learning and semi-supervised domain adaptation as a generalpurpose algorithm. Prabhu et al. (2021) propose SENTRY, an algorithm based on judging the predictive consistency of samples from the target domain under different image transformations. Zou et al. (2019) show that different types of confidence regularization can improve the performance of self-learning. A theoretically motivated framework for self-learning in domain adaptation based on consistency regularization has been proposed by Wei et al. (2020) and then extended by Cai et al. (2021). Self-learning has also been used for semantic segmentation (Zou et al., 2018).
The main difference from these works to ours is that they 1) utilize both source and target data during training (i.e., the classical UDA setup) whereas we only require access to unlabeled target data (source-free setup), and 2) train their models from scratch whereas we adapt pretrained checkpoints to the unlabeled target data, 3) are oftentimes more complicated (also in terms of the number of hyperparameters) than our approach due to using more than one term in the objective function. We would like to highlight that utilizing source data should always result in better performance compared to not using source data. Our contribution is to show that self-learning can still be very beneficial with a small compute budget and no access to source data. Our setup targets “deployed systems”, e.g., a self-driving car or a detection algorithm in a production line which adapts to the distribution shift “on-the-fly” and cannot (or should not) be retrained from scratch for every new domain shift.
Kumar et al. (2020) study the setting of self-learning for gradual domain adaptation. They find that self-learning works better if the data distribution changes slowly. The gradual domain adaptation setting differs from ours; instead of a gradual shift over time, we focus on a fixed, systematic shift at test time dataset. Kumar et al. (2020) tested their method on a synthetic Gaussian dataset, MNIST and the Portraits datasets; building and evaluating ImageNet-scale datasets for a gradual domain adaptation perspective is a very interesting extension of our work, but left for future work, and would not only require changes/adaptations to the self-learning method, but also to the evaluation datasets.
Chen et al. (2020b) prove that under certain conditions, self-learning can improve performance in biased datasets where spurious features correlate with the label in the source domain but are independent of the label in the target domain. While Chen et al. (2020b) also consider the setting of source-free domain adaptation (like we do), they limit their experiments to small scale models on MNIST and Celeb-A, while we conduct a large scale empirical study on all common robustness datasets with large scale models – some of the observed and studied effects in our paper (effectiveness of different loss functions at different problem scales, adaptation mechanisms, etc.) can be attributed to this large scale evaluation setting, and extending our insights over small scale experiments.
Similar to us, Chen et al. (2020b) find that a strong source classifier is necessary for self-learning to work; however, in their case, a teacher accuracy of 72% (on CMNIST10) is already too low and leads to worse student accuracy. In contrast, in our experiments, self-learning still works for an mCE as high as 80% (cf. appendix Figure 3, severity 5) and teacher accuracies as low as 10.4% (on ImageNet-D “Infograph”), and breaks down at accuracies around 1-2% (on ImageNet-D “Quickdraw”). This discrepancy might be due to the spurious correlations that Chen et al. (2020b) introduced in their dataset leading to systematic biases, which are not present in the datasets we studied.
Self-learning in semi-supervised learning (SSL) In a different line of work which is not related to domain adaptation directly, self-learning has been used in a semi-supervised setting. Zoph et al. (2020) show that self-learning outperforms pretraining when stronger data augmentation is used and more labeled data is present. They use human labels on the target task (e.g., object detection on COCO) and pseudo-labels on an unlabeled dataset (e.g. ImageNet), and optimize the loss on both datasets, with the aim to improve performance on the task where ground truth labels are known. The work of Zoph et al. (2020) is orthogonal to ours, in the sense that we could adapt their final
checkpoint to a new domain with our method, similar to how we adapted the Noisy Student model which was also trained using self-learning.
Rizve et al. (2021) propose an uncertainty-aware pseudo-label selection (UPS) framework which outperforms other SSL methods in a few-label regime. UPS is helpful to reduce the impact of noisy pseudo-labels; in our case, we use the generalized cross-entropy loss for this purpose. Testing the UPS framework (and other means for improving the quality of pseudo-labels, or robustness against label noise) on robustness datasets would be an interesting direction for future work.
De Sousa Ribeiro et al. (2020) propose Deep Bayesian Self-Training (DBST) for automatic data annotation. Mukherjee & Awadallah (2020) suggest using self-learning in a semi-supervised setting for text classification with few labels.

heading: G SOFTWARE STACK
text: We use different open source software packages for our experiments, most notably Docker (Merkel, 2014), scipy and numpy (Virtanen et al., 2020), GNU parallel (Tange, 2011), Tensorflow (Abadi et al., 2016), PyTorch (Paszke et al., 2017), timm (Wightman, 2019), Self-ensembling for visual domain adaptation (French et al., 2018), the WILDS benchmark (Koh et al., 2021), and torchvision (Marcel & Rodriguez, 2010).

Review summaries:
The paper studies effectiveness of self-training to improve test time performance when the distribution of test data is not similar to the training data. The paper more specifically focuses on source-free domain adaptation settings where the source target data is not available. In this setup self-training has been tested as an additional step on top of  different robustness and adaptation approaches such as robust pretraining, unsupervised domain adaptation and self-supervised pretraining. The paper shows improvement on multiple ImageNet variants and CIFAR10-C, and also introduces ImageNet-D dataset as a new benchmark. ImageNet-D has been produced by matching label space of IN datasets with DomainNet data provided in Visual domain adaptation challenges. The main contribution of this paper is to perform a systematic and large study of self-training as a method to deal with distribution shifts.

This paper provides an in depth empirical evaluation of classical self-training techniques such as pseudo-labelling and entropy minimization on test performance under domain shifts. The authors stress that, although simple, these techniques consistently improve the robustness to distribution shifts regardless of model architecture or pre-training techniques used. This makes them especially useful to practitioners applying machine learning algorithms to real problems where distribution shifts are prevalent. The authors claim state-of-the-art adaptation results on a number of popular dataset corruption benchmarks, and present a new challenging dataset for evaluating the robustness of deep vision models.