If your data distribution shifts, use self-learning
Authors: Evgenia Rusak
Steffen Schneider
George Pachitariu
Luisa Eck
Peter Vincent Gehler
Oliver Bringmann
Wieland Brendel
Matthias Bethge
Venue: ICLR 2022 Submitted (2022)
Keywords: Self-Learning
Domain Adaptation
Robustness
Pseudolabeling
Entropy Minimization
Corruption Robustness

Abstract: In this paper, we demonstrate that self-learning techniques like entropy minimization or pseudo-labeling are simple, yet effective techniques for increasing test performance under domain shifts. Our results show that self-learning consistently increases performance under distribution shifts, irrespective of the model architecture, the pre-training technique or the type of distribution shift. At the same time, self-learning is simple to use in practice because it does not require knowledge or access to the original training data or scheme, is robust to hyperparameter choices, is straight-forward to implement and requires only a few training epochs. This makes self-learning techniques highly attractive for any practitioner who applies machine learning algorithms in the real world. We present state-of-the art adaptation results on CIFAR10-C (8.5% error),  ImageNet-C (22.0% mCE), ImageNet-R (17.4% error) and ImageNet-A (14.8% error), theoretically study the dynamics of self-supervised adaptation methods and propose a new classification dataset (ImageNet-D) which is challenging even with adaptation.

Under review as a conference paper at ICLR 2022
IF YOUR DATA DISTRIBUTION SHIFTS, USE SELF-LEARNING
A Anonymous authors Paper under double-bling review
ABSTRACT
We demonstrate that self-learning techniques like entropy minimization and pseudo-labeling are simple and effective at improving performance of deployed computer vision model under systematic domain shifts. We show consistent improvements irrespective of the model architecture, the pre-training technique or the type of distribution shift. At the same time, self-learning is simple to use in practice because it does not require knowledge access to the original training data or scheme, robust to hyperparameter choices, is straight-forward implement and requires only few adaptation epochs. This makes self- learning techniques highly attractive for any practitioner who applies machine learning algorithms
the real world. We present state-of-the art adaptation results on CIFAR 10-C (8.5% error), ImageNet-C (22.0% mCE), ImageNet-R (17.4% error) and ImageNet-A (14.8% error), theoretically study the dynamics of self-supervised adaptation methods and propose new classification dataset (ImageNet-D) which is challenging even with adaptation.
INTRODUCTION
Deep Neural Networks (DNNs) can reach human-level performance in complex cognitive tasks (Brown al., 2020; He et al., 2016a; Berner et al., 2019) if the distribution of the test data is sufficiently similar to the training data. However, DNNs are known to struggle the distribution of the test data is shifted relatively the training data (Geirhos al., 2018; Dodge & Karam, 2017).
Two largely distinct communities aim to increase the performance models under test-time distribution shifts: The robustness community generally considers ImageNet-scalle datasets and evaluates models in an ad-hoc scenario. Models are trained on clean source dataset like ImageNet, using heavy data augmentation (Hendrycks et al., 2020a; Rusak et al., 2020; Geirhos et al., 2019) and/or large-scale pre-training (Xie et al., 2020a; Mahajan et al.. 2018). The trained models are adapted in any way test-time distribution shifts. This evaluation scenario is relevant for applications in which very different distribution shifts are encountered in unpredictable order, and hence misses out on the gains of adaptation to unlabeled samples of the target distribution.
Large scale (semi) supervised pre-training
Unsupervised domain
Robustification
Self-supervised pre-training
adaptation
Pre-trained
Student
Self-Learning
H
Adapted
Systematic domain shift
Entropy Minimization
model
Teacher
Pseudo Pseudo-Labeling Labeling
Figure 1: Robustness adaptation to new datasets traditionally been achieved by robust pre-training (with hand--ellettedddata--ri augmentation strategies, or additional data), unsupervised domain adaptation (with access to unlabeled samples from the test set), or, more recently, self-supervised learning methods. We show that top of these different pro-training tasks, is always possible (irrespective of architecture, model size pre-training algorithm) to further adapt models the target domain with simple self-lear leamnee techniques.

---

Under review as a conference paper at ICLR 2022
The unsupervised domain adaptation (UDA) community often considers smaller-scale datasets and assumes that both the source and the (unlabeled) target dataset are known. Models are trained on both datasets (e.g., with an adversarial domain objective, Ganin et al., 2016) before evaluation the target domain data. This evaluation scenario provides optimal conditions for adaptation, but the reliance on the source dataset makes UDA more computationally expensive, more impractical and prevents the use of pre-trained models for which the source dataset is unknown or simply too large.
In this work, we consider the source-free domain adaptation setting, a middle ground between the classical ad-hoc robustness setting and UDA in which models can adapt to the target distribution but without using the source dataset (Kundu et al., 2020; Kim et al., 2021; Li et al., 2020: Liang et al., 2020). This evaluation scenario is interesting for many practitioners and applications as an extension of the ad-hoc robustness scenario. evaluates the possible performance of a deployed model on systematic, unseen distribution shift at inference time: an embedded computer vision system in an autonomous car should adapt to changes without being trained on all available training data; image-based quality control software may not necessarily open-source the images it has been trained on, but still has to be adapted to the lighting conditions at the operation location; a computer vision system in a hospital should perform robustly when tested on scanner different from the training images-importann -importantly, it might not be known at development time which scanner it will be tested on, and it might be prohibited to share images from many hospitals to run UDA.
Can self-learning methods like pseudo-labeling and entraa--minimization also be used in this source-free domain adaptation setting? To answer this question, perform extensive study of several self-lcarning variants, and find consistent and substantial gains in test-time performance across several robustness and out-of-domain benchmarks and wide range of models and pre- training methods, including models trained with UDA methods that do not use self-learning. We also find that self-learning outperforms state-of-the-art source-free domain adaptation methods, namely Test-Time Time Training which is based on a self-supervised auxiliary objective and continual training (Sun et al., 2019b), test-time entropy minimization (Wang et al., 2020) and (gradient-free BatchNorm adaptation (Schneider et al., 2020; Nado et al., 2020). We perform a large number of ablations to study important design choices for self-learning methods in source-free domain adaptation. Furthermore, we show that variant of pseudo-labeling with robust loss function consistently outperforms entropy minimization on ImageNet-scale datasets. We theoretically analyze and empirically verify the influence of the temperature parameter in self-learning and provide guidelines how this single parameter should be chosen. Our approach is visualized in Figure 1. We do not consider test-time adaptation in an online setting like is studied e.g., by Zhang et al. (2021), where the model is adapted to one example at a time, and reset after each example.
Related Work. Variants of self-learning have been used for UDA (Berthelot al., 2021), for example using auxiliary information (Xie et al., 2020b), consistency (Wei et al., 2020; Cai et al., 2021; Prabhu et al., 2021) or confidence (Zou et al., 2019) regularization. The main difference from these works to ours is that they 1) utilize both source and target data for self-learning whereas we only require access to unlabeled target data, 2) train their models from scratch whereas we merely fine-tune pretrained checkpoints on the unlabeled target data, and 3) are generally more complicated than our approach due to using more than one term in the objective function.
Our work is conceptually most similar to virtual adversarial domain adaptation in the fine-tuning phase of DIRT-T (Shu et al., 2018)) and Test-time entropy minimization (TENT; Wang et al., 2020). In contrast to DIRT-T, our objective is simpler and we scale the approach to considerably larger datasets on ImageNet scale. TENT, on the other hand, only evaluated a single method (entropy minimization) on a single vanilla model (ResNet-50)) on IN-C. We substantially expand this analysis to show that self-learning almost universally increases test-time performance under distribution shifts, regardless of the type of distribution shift, the model architecture or the pre-training method.
Self-learning has also been applied to UDA for semantic segmentation (Zou et al., 2018), for gradual domain adaptation (Kumar et al., 2020), for semi-supervised learning (Rizve et al., 2021; Mukherjee & Awadallah, 2020), for learning in biased datasets (Chen et al., 2020b) and for automated data annotation (De Sousa Ribeiro et al., 2020). Zoph al. (2020) show that self-learning outperforms pretraining when stronger data augmentation is used and more labeled data is present. A more detailed discussion of related work alongside with the main differences to our work can be found in Appendix F. Our main contribution beyond these works is to show the effectiveness of self-learning on top of both robust, large scale, and domain adapted models, at scale.
2

---

Under review as a conference paper at ICLR 2022
2 SELF-LEARNING FOR TEST-TIME ADAPTATION
Different variants of self-learning have been used in both unsupervised domain adaptation (French al., 2018: Shu al., 2018), self-supervised representation learning (Caron al., 2021), and in semi-supervised learning (Xie et al., 2020a). In a typical self-learning setting a teacher network f' trained on the source domain predicts labels on the target domain. Then, a student model f is fine-tuned on the predicted labels.
In the following, let fx)) denote the logits for sample x and let p'(j)x) a (f' (x)) denote
the probability for class j obtained from a softmax function ay(+). Similarly, (*(x and p² x) denote the logits and probabilities for the student model f*. For all techniques, one can optionally
only admit samples where the probability max; p'(]l) exceeds some threshold. We consider
three popular variants of self-learning: Pseudo-labeling with hard or soft labels, as well as entropy minimization.
Hard Pseudo-Labeling (Lee, 2013; Galstyan & Cohen, 2007). We generate labels using the teacher and train the student on pseudo-labels using the standard cross-entropy loss,
[x == - log (f)x), = argmax, p' (j(x))
(1)
Usually, only samples with a confidence above a certain threshold are considered for training the student. We test several thresholds but note that thresholding means discarding potentially large portion of the data which leads to a performance decrease in itself. The teacher is updated after each epoch.
Soft Pseudo-Labeling (Lee, 2013; Galstyan & Cohen, 2007). In contrast to the hard pseudo- labeling variant, we here train the student on class probabilities predicted by the teacher,
{s(x) : Σn (ixx) log (j(x).
(2)
Soft pseudo-labeling is typically not used in conjunction with thresholding, since it already incorporates the certainty of the model. The teacher is updated after each epoch.
Entropy Minimization (ENT; Grandvalet & Bengio, 2004). This variant is similar soft pseudo- labeling, but we no longer differentiate between teacher and student network. It corresponds to an "instantancous" update of the teacher. The training objective becomes
t (x(x) := Σv' (i'))))oogr' (jlx).
(3)
Intuitively, self-training with entropy minimization leads sharpening of the output distribution for each sample, making the model more confident in its predictions.
Robust Pseudo-Labeling (RPL). Virtually all introduced self-training variants use the standard cross-entropy classification objective. However, the standard cross-entropy loss has been shown to be sensitive to label noise (Zhang Sabuneu, 2018; Zhang et al., 2017). In the setting of domain adaptation, inaccuracies in the teacher predictions and, thus, the labels for the student, are inescapable, with severe repercussions for training stability and hyperparameter sensitivity as we show in the results.
As a straight-forward solution to this problem, we propose to replace the cross-entropy loss by a robust classification loss designed to withstand certain amounts of label noise (Ghosh et al., 2017; Song et al., 2020; Shu et al., 2020; Zhang & Sabuncu, 2018). A popular candidate is the Generalized Cross Entropy (GCE) loss which combines the noise-tolerant Mcan Absolute Error (MAE) loss (Ghosh et al., 2017) with the CE loss. We only consider the hard labels and use the robust GCE loss as the training loss for the student.
(4) with g E (0, 1]. For the limit case g + the GCE loss approaches the CE loss and for q = 1, the GCE loss is the MAE loss (Zhang & Sabuncu, 2018). We test updating the teacher both after every update step of the student (RPL) and once per epoch (RPL.P).
= argmax, xi)x),
:= (<cr(x.):=p-aappp(() (iil))",,
3

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENT DESIGN
Datasets. IN-C (Hendrycks & Dietterich, 2019) contains corrupted versions of the 000 images in the IN validation set. There are fifteen test and four hold-out corruptions, and there are five severity levels for each corruption. The established metric to report model performance on IN-C is the mean Corruption Error (mCE) where the error is normalized by the AlexNet error, and averaged over all corruptions and severity levels, see Eq. 20, Appendix C.1. IN-R (Hendrycks et al., 2020a) contains 30 000 images with artistic renditions of 200 classes of the IN dataset. IN-A (an, 2019) composed of 7500 unmodified real-world images on which standard IN-trained ResNet50 (He et al.. 2016b) models yield chance level performance. CIFARIO (Krizhevsky et al., 2009) and STL.10 (Coates et al.. 2011) are small-scale image recognition datasets with 10 classes each, and training sets of 50 000/5000 images and test sets of 10 000/8000 images, respectively. The digit datasets MNIST (Deng, 2012) and MNIST-M (Ganin et al., 2016) both have 60 000 training and 10 test images.
Hyperparameters. The different self-learning variants have a range of hyperparameters such as the learning rate or the stopping criterion. Our goal is to give realistic estimation on the performance to be expected in practice.. To this end, we optimize hyperparameters for each variant of pseudo- labeling on hold-out set of IN-C that contains four types of image corruptions ("speckle noise", "Gaussian blur", "saturate" and "spatter") with five different strengths each, following the procedure suggested in Hendrycks & Dietterich (2019). We refer the hold-out set of IN-C as our dev set.
Models for ImageNet-scale datasets. We consider four popular model architectures: ResNet50 (He et al., 2016b). DenseNet161 (Huang et al., 2017). ResNeXt101 (Xie et al., 2017) and EfficientNet-L2 (Tan & Le, 2019) (see Appendix B.1 for details on the used models). For ResNet50, DenseNet and ResNeXt101, we include a simple vanilla version trained on IN only. For ResNet50 and ResNeXt101, we additionally include a state-of-the-art robust version trained with DeepAugment and Augmix (M, Hendrycks et al., 2020a)'. For the ResNeXt model, we also include version that was trained on 3.5 billion weakly labeled images (IG-3.5B, Mahajan et al., 2018). Finally, for EfficientNet-L2 we select the current state of the art on IN-C which was trained on 300 million images from JFT-300M (Chollet, 2017; Hinton et al., 2014) using noisy student- teacher protocol (Xie et al., 2020a). We validate the IN and IN-C performance of all considered models and match the originally reported scores (Schneider al., 2020). For EfficientNet-22, we match IN top-1 accuracy up to 0. 1% points, and IN-C up to 0.6% mCE.
Models for CIFARI0/MNIST--cale datasets. For CIFARIO-C experiments, we two WedeResNets (WRN, Zagoruyko & Komodakis, 2016): the first one is trained on CIFARI0 and has depth of 28 and a width of 10 and the second one is trained with AugMix (Hendrycks al., 2020b) and has a depth of 40 and a width of 2. The remaining small-scale models are trained with unsupervised domain adaptation (UDA) methods. We propose to regard any UDA method which requires joint training with source and target data as a pre-training step, similar to regular pre- training on IN, and use self-learning on top of the final checkpoint. We consider two popular UDA methods: self-supervised domain adaptation (UDA-SS; Sun et al., 201 9a) and Aaversrrdvrrsrriall Training of Neural Networks (DANN; Ganin et al., 2016). In UDA-SS, the authors seek to align the representations of both domains by performing auxiliary self-supervised task on both domains simultaneously. In all UDA-SS experiments, we use a WideResNer with a depth of 26 and a width of embedding by optimizing a minimax objective.
16. In DANN, the authors learn a domain-invarian For all DANN experiments except for MNIST- -NIISTMM, we use the same WRN architecture as above. For the MNIST- we used a smaller WideResNet version with a width of 2. We note that DANN training involves optimizing a minimax objective and is generally harder to tune.
experiment, the training with the larger model diverged and
RESULTS: SELF-LEARNING UNIVERSALLY IMPROVES MODELS
Self-learning is a powerful learning scheme, and in the following section we show that it allows to perform test-time adaptation on robustified models, models obtained with large-scale pre-training as well as domain adapted models across a wide range of datasets and distribution shifts. Our main results on large-scale and small-scale datasets are shown in Tables 1 and 2, respectively. These
see leaderboard git com//henddtyek//roouuteeeee
4

---

Under review as a conference paper at ICLR 2022
summary tables show final results, and all experiments the hyperparameters determined separately on the dev set.
Table 1: Self-learning successfully adapts ImageNet-scale models across different model architectures on IN-C, IN-A and IN-R. We adapt the vanilla ResNet50, ResNeXt101 and 1enseNet161 models to IN-C and decrease the mCE by over 19 percent points in all models. Further, self-leaming works for models irrepecctive of their size: Self-learning substantially improves the performance of the ResNet50 and the ResNext101 trained with AAu IN-C by 11.9 and 9.7 percent points, respectively. Finally, we further improve the current state-of-thearrt model on IN-C- the EfficientNet-L2 Noisy Student model- and report new state-of the-hettt result of 22% mCE (which corresponds to topl eπrrr of 17.1%) on this benchmark with test-time adaptation (compared 28% mCE without adaptation).
number parameters
w/o adapt
w/ adapt
A
mCE [%] on IN-C test (s)
RPI.
ResNet50 vanilla (He al., 2016b)
2.6 10"
76.7
50.5
(-26.2)
ResNet50 DAug+AM (Hendrycks et al., 2020a)
2.6 x 107
53.6
41.7
(-11.9)
DenseNet161 vanilla (Huang al., 2017)
2.8 10"
66.4
47,0
(-19.4)
ResNeXt101 32xad vanilla (Xie et al., 2017)
8.8 x 107
66.6
43.2
(-23.4)
ResNeXr101 DAug+AM (Hendrycks et al., 2020a)
8.8 x 10?
44.5
34.8
(-9.7)
szxxd IG-3.5B (Mahajan al., 2018)
8.8 107
51.7
40.9
(-10.8)
EfficientNet-L.2 Noisy Student (Xie et al., 2020a)
4.8 x 10
28.3
22.0
(-6.3)
topl error [%] IN-R (*))
ResNe150 vanilla (He et al., 2016b)
2.6 x 107
63.8
54.1
(-9.7)
EfficientNettL.2 Noisy Student (Xie et al., 2020a)
4.8 10
23,5
17.4
(-6.1)
lopl error [%6 on ImageNet-A (S)
EfficientNet-L2 Noisy Student (Xie et al., 2020a)
4.8 x 10
16.5
14.8
(-1.7)
Self-learning is not limited to the distribution shifts in IN-C like compression artefacts blur. On IN-R, a dataset with renditions, self-learning improves both the vanilla ResNet50 and the EfficientNet-L.2 model, the latter of which improves from 23.5% to top-1 error For a vanilla ResNet50, we improve the top-1 error from 63.8% (Hendrycks et al., 2020a) 54.1%. On IN-A, adapting the EfficientNet-L2 model using self-learning decreases the top-1 етror from 16.5% (Xie et al.. 2020a) to 14.8% top-1 error, again constituting a new state of the art with test-time adaptation on this dataset.
state-of-the art of 17.4%
Table 2: Self-learning improves robustified and domain adapted models on small-scale datasets. We test common domain adaptation techniques like DANN (Ganin al., 2016) and UDA-SS (Sun et al., 2019a), and show that self-learning is effective at further tuning such models to the target domain. We suggest to view unsupervised source/target domain adaptation as step comparable pre-training under corruptions, rather than an adaptation technique specifically tuned to the target set-indeed, we can achieve error rates using, e.g., DANN + target adaptation previously only possible with source/target based pseudo-labeling across different common domain adaptation benchmarks. Self-lcarning also decreases the eтror on CIFARIO-C trained with AugMix (AM, Hendrycks et al., 2020b) and reaches new state of the CIFARIO. of 8.5% top1 error with test-time adaptation. "denotes preliminary results on CIFAR-C dev only, due to instabilities in training the adversarial network in DANN.
the Wide ResNet model
number parameters
w/o adapt
adapt
topl error [%] on CIFARIO-C
ENT
WRN-28- 10 vanilla (Zagoruyko & Komodakis, 2016)
3.6 x 10"
26.5
13.3
(-13.2)
WRN-40-2 (Hendrycks et al., 2020b)
10
8.5
(-2.7)
WRN-26-I1 16 UDA-SS (Sun et al., 2019a)
9.3 x 107
27.7
16.7
(-11.0)
WRN-26-16 DANN (Ganin al.. 2016)
107
129.7
128.5
(-1.2)
UDA CIFAR TAARROOSTTIE -SLIO top1 сгrоr on target [6][
WRN-26- UDA-SS (Sun al., 2019a)
9.3 107
28.7
21.8
(-6.9)
WRN-26-16 16 DANN (Gamin et al., 2016)
9.3 x 107
25.0
23.9
(-1.1)
UDA MINIST-MNIISTAN topl enror target [%](()
WRN-266166 UDA-SS (Sun et al., 2019a)
9.3 x 107
4.8
2.0
(-2.8)
WRN-26-2 DANN (Ganin et al., 2016)
1.5 x 10
11.4
5.1
(-6.3)
5

---

Under review as a conference paper at ICLR 2022
Table 3: Self-learning also improves large pre-trained models. Unlike BatchNorm adaptation (Schneider et al., 2020), we show that self-learning transfers well to models pre-trained on a large amount of unlabeled data: self-learning decreases the mCE IN-C of the ResNeXt101 trained on 3.5 billion weakly labeled samples (IG-3.5B, Mahajan et al., 2018) from 51.7% to 40.9%.
mCE on IN-C test [%]C
adaptation
BN adaptation
self-learning
aaaNextt0112xxe vanilla
66.6
56.8
43.2
ResNeXt101 32x8d IG-3.5B
51.7
51.8
40.9
Table 4: Self-learning outperforms previously published test-time adaptation approaches on IN-C. The robustness benchmark IN-C has so far mostly been regarded in the ad-hoc evaluation setting as discussed in our introduction. Thus, there are only few published methods that report numbers for test-time adaptation: BatchNorm adaptation (Schneider et al., 2020), Test-Time Training (TTT, Sun et al., 2019b), and TENT (Wang et al., 2020). In particular, note that TTT requires special loss function at training time, while our approach is agnostic to the pre-training phase. Our self-trainin training results outperforms all three baselines (also after tuning TENT with our full experimental protocol):
mCE on IN-C test [%] (S)
w/o adapt 76.7
BN adapt 62.2
TENT (ours)
self-learning 50.5
ResNet50 vanilla
53.5 (51.6)
topl error [%] on IN-C, sev. sev.
(N)
w/o adapt
BN adapt
TTT 66.3
sell-learnin
ResNet18 vanilla
85.4
72.2
61.9
Table 5: Self-supervised methods based on self-learning allow out-of-the-box test-time adaptation. The recently published DINO method (Caron al., 2021) is another variant of self- supervised learning that has proven to be effective for unsupervised representation learning. At the core, the method uses soft pseudo-labelingg Here, we test whether model trained with DINO on the source dataset can be test-time adapted on IN-C using DINO to further improve out-of of-distribution performance. Since the used model is a vision transformer model, we test different choices of adaptation parameters and find considerable performance improvements in all cases, yielding an mCE of 43.5% at a parameter count comparable a ResNer50 model. For adapting the affine layers, we follow Houlsby et al. (2019):
w/o adapt
w/ adapt affine layers
w/ adapt bottleneck layers
w/ adapt
w/ adapt
mCE on IN-C test [%] (N)
lin. layers 45.2
all weights
VIT-S/16
62.3
51.8
46.8
43.5
UNDERSTANDING TEST-TIME ADAPTATION WITH SELF-LEARNING
In the following section, show ablations and interesting insights of using self-learning for test- time adaptation. If not specified otherwise, all ablations are run on the holdout corruptions of IN-C (our dev set) with vanilla ResNet50.
Table 6: Robust pseudo-labeling outperforms entropy minimization on large-scale datasets while the reverse true on small-scale datasets. We find that robust pseudo-labeling consistently improves over entropy minimization on IN-C, while entropy minimization performs better on smaller scale data (CIFAR10, STL10, MNIST). The finding highlights the importance of testing both algorithms on new datasets. The improvement is typically on the order of one percent point:
mCE, IN-C dev
ResNet50
ResNeXt-101
EfficientNet-22
top-1 err, CIFAR-C
WRN-40
ENT RPL
50.0 + 0.04
43.0
22.2
ENT
8.5
48.9 + 0.02
42.0
21.3
RPL
9,0
Table 7: Robust pseudo-labeling allows usage of the full dataset without threshold. Classical hard labeling needs a confidence threshold (T) for best performance, thereby reducing the dataset size. while best performance for RPL is reached for full dataset training with a threshold T of 0.0:
diff. self-leaming methods no adapt soft PI. hard PL (T): 0.0
0.5
0.9
(T):
0.0 0.5 0.9
mCE on IN-C dev [%]
69.5
60.1
53.8
51.9 52.4
49.7 49.9 51.8

---

Under review as a conference paper at ICLR 2022
Table 8: Short update intervals are crucial for fast adaptation. Having established that RPL generally performs better than soft- and hard-labeling, we vary the update interval for the teacher. We find that instant updates most effective. In entropy minimization, the update interval instant per default.
Update interval for RPL
w/o adapt
no update
epoch
instant
mCC on IN-C dev [%]
69.5
54.0
49.7
49.2
Table Adaptation of only affine layers is important in CNNs. On IN-C, adapting only the affine parameters after the normalization layers (i.e., the rescaling and shift parameters B and y) works better on a ResNet50 architecture than adapting all parameters or only the last layer. We indicate the number of adapted parameters in brackets.
Adaptation mechanism
w/o adapt
last layer
full model
affine
mCE on IN-C dev [%] 69.5 [0]
60.2 (2M)
51.5 [22.6M]
48,9 [5.3k]
Note that for Vision Transformers, full model adaptation works better than affine adaptation (see Table 5). We also noticed that convolutional models with smaller parameter count like ResNet18, full model adaptation is possible.
Hyperparameters obtained corruption datasets transfer well to real world datasets. When evaluating models, we select the hyperparameters discussed above (the learning rate and the epoch used for early stopping the most critical ones) the holdout of IN-C. We note that this technique transfers well to IN-R, -A and DD, highlighting the practical value of corruption robustness datasets for adapting models on real distribution shifts.
On IN-D, we performed a control experiment where we selected hyperparameters with leave-one- out cross validation this selection scheme actually performed worse than IN-C parameter selection (see Appendix D.1).
6 ADAPTING MODELS ON A WIDER RANGE OF DISTRIBUTION SHIFTS REVEALS LIMITATIONS OF ROBUSTIFICATION AND ADAPTATION METHODS
Robustness datasets on ImageNet-scale have so far been limited to a few selected domains (image corruptions in IN-C, image renditions in IN-R, difficult images for ResNet50 classifiers in IN-A). In order test our approach on a wider range of complex distribution shifts, we re-purpose dataset from the Visual Domain Adaptation Challenge 2019 (DomainNet, Saenko et al., 2019) as an additional robustness benchmark. This dataset comes with six image styles: Clipart, Real, Infograph, Painting, Quickdraw and Sketch. It has 345 classes in total, of which 164 overlap with IN. To benchmark robustness
trained models out of the box, we filter out the classes that cannot be mapped to IN and refer to the smaller version of DomainNet as ImageNet-D (IN-D). We map 463 classes IN to these 164 IN-D classes, e.g., for an image from the "bird" class in IN-D, we accept all 39 bird classes in IN as valid predictions. We show example images from IN-D in Table 10. The detailed evaluation protocol along with justifications for our design choices and additional analysis are outlined in Appendix D.
The benefit of IN-D over DomainNet is the re-mapping to ImageNet classes which allows robustness researchers to easily benehmark on this dataset, without the need re- traininng model common in UDA). To test whether self-learning is helpful for more complex distribution shifts, we adapt vanilla ResNet50, several robust IN-C models and the EfficientNet-L.2 Noisy Student model on IN- D. We use the same hyperparameters we obtained on IN-C dev for all our IN-D experiments. We show our main results Table 10.
More robust models perform better on IN-D. Comparing the performance of the vanilla ResNer50 model to its robust DAug+AM variant, we find that the DAug+AM model performs better on all domains, with the most significant gains the "Clipart", "Painting" and "Sketch" domains. We show detailed results for all domains and all tested models Appendix D.2, along with results on IN-C and IN-R for comparison. We find that the best performing models on IN-D also the
7

---

Under review as a conference paper at ICLR 2022
Table 10: Self-lean Self-learning decreases the topi some IN-D domains but increases others.
domain
Real
Painting
Clipart
Sketch w/o
Infograph w/o
Quickdraw w/
adapt
w/o
w/
w/o
w/
w/o
w/
w/
w/
w/o
model
EfiNet-L2 Noisy Student
29.2
27.9
42.7
40.9
45.0
37.9
56.4
51.5
77.9
94.3
98.4
ResNet50 DAug+AM
39.2
36.5
58.7
53.4
68.4
57.0
75.2
61.3
88.1
83.2
98.2
99.1
ResNet50 vanilla
40.1
37.3
65.1
57.8
76.0
63.6
82.0
73.0
89.6
85.1
99.2
99.8
strongest ones on IN-C and IN-R which indicates good generalization capabilities of the techniques combined for these models, given the large differences between the three considered datasets. However, even the best models perform 20 to 30 percentage points worse on IN-D compared to their performance on IN-C or IN-R, indicating that IN-D might be a more challenging benchmark.
All models struggle with some domains of IN-D. The EfficientNet-L22 Noisy Student model obtains the best results on most domains. However, we note that the overall error rates are surprisingly high compared to the model's strong performance on the other considered datasets (IN-A: 14.8% top-1 error, IN-R: 17.4% top-1 error, IN-C: 22.0% mCE). Even on the "Real" domain closest clean IN where the EfficientNet-L2 model has top-1 error of 1 11.6%, the model only reaches a top-1 error of 29.2%. Self-learning decreases the top1 error on all domains except for "Infograph" and "Quickdraw". We note that both domains have very high error rates from the beginning and thus hypothesize that the produced pseudo-labels are of low quality.
Error analysis IN-D. We investigate the errors ResNet50 model makes IN-D by analyzing the most frequently predicted classes for different domains to reveal systematic errors indicative of the encountered distribution shifts. We find most errors interpretable: the classifier assigns the label "comic book" to images from the "Clipart" or "Painting" domains, "website" to images from the "Infograph" domain, and "envelope" to images from the "Sketch" domain. Thus, the classifier predicts the domain rather than the class. We find no systematic errors on the "Real" domain which expected since this domain should be similar IN. Detailed results on the top-3 most frequently predicted classes for different domains can be found in Fig. 9, Appendix D.4.
IN-D should used as an additional robustness benehmark. While the error rates on IN-C, -R and -A are at a well-acceptable level for our largest EfficientNet-L22 model after adaptation, IN-D performance is consistently worse for all models. We propose to move from isolated benchmark settings like IN-R (single domain) benchmarks more common in domain adaptation (like DomainNet) and make IN-D publicly available as an easy to use dataset for this purpose.
Additional experiments and limitations. We discuss additional proof-of-concept implementations on the WILDS benchmark (Koh et al., 2021), BigTransfer (BiT; Chen et al., 2020a) models and self-learning based UDA models in Appendix E. On WILDS, self-learning is effective for the Camelyon 77 task with systematic shift between train, validation and test sets (each set is comprised of different hospitals), while self-learning fails improve tasks with mixed domains.
A SIMPLE MODEL STABILITY IN SELF-LEARNING
We observed that different self-learning schemes are optimal for small-scale vs. large-scale datasets and varying amount of classes. We reconsider the used loss functions, and unify them into
f'
f'(x) Tt
f((x)
((x) Σn,
log
Ts
(5)
f((x) f(xx)
entropy minimization pseudo-labeling.
(f(x)))),
We introduced student and teacher temperature Т₃ and т₃ as parameters in the softmax function and the stop gradient operation sg. Caron et al. (2021) fixed т, and varied 7, during training.
8

---

Under review as a conference paper at ICLR 2022
and empirically found upper bound for Tr above which the training was longer stable. To better understand such behavior, we study the learning dynamics of the loss function in equation 5 theoretically in simple two-datapoints, two-classes model with linear student and teacher networks fx() x w² and f'(x) = x w' defined in Appendix A.1. Gradient descent with stop gradient corresponds
hard pseudo-labeling in the limit 7r 0 and to soft pscudo-tabeling when Ta = T 1. Gradient descent without stop gradient, i.e., setting w w w corresponds to entropy minimization. We obtain the following result:
Two points
CIFAR-C
Proposition (Collapse in the two-point model). The student and teacher networks Ws and Wi trained with stop gradient does not collapse to the trivial representation v/x x w 0,x w
PL
Error
0%
50%
BAS
log
1005
if To > Tr- The network w trained without stop
gradient does not collapse Ts > Tz/2. Proof. see $ A.2.
-2
2
ENT
We validate the proposition on a simulated two datapoint toy dataset, as well as the CIFAR- C dataset and outline the results in Figure 2. In general, the size and location of the region where collapse is observed in the simulated model also depends on the initial conditions, the learning rate and the optimization procedure. An in depth discussion, as well as additional simulations are given in the Appendix. In practice, the result suggests that student temperatures should exceed the teacher temperatures for pseudo-labeling, and student temperatures should exceed half the teacher temperature for entropy minimization.
logi0 Te
0
-2
108.0 T
1og1o T
Figure 2: For the two point model, we show error and for the CIFARIO-C simulation we show improvement (yellow) vs. degradation (purple) over the non-adapted baseline (BAS). An important convergence criterion for pscudo-labeling (top row) and entropy minimization (bottom row) is the ratio of student and teacher temperatures; it lies тo = Tr for PL., and 27, = т; for ENT. Despite the simplicity of the two-point model, the general convergence regions transfer to CIFAR10- 10-C.
Entropy minimization with standard temperatures (T. = T. 1) and hard pseudo-labeling (T: 0) are hence stable. The two-point learning dynamics vanish for soft pseudo-labeling with т3 Tτ. suggesting that one would have to analyze a more complex model with more data points. While this does not directly imply that the learning is unstable at this point, we empirically observe that both entropy minimization and hard labeling outperform soft-labeling in practice.
8 CONCLUSION
We evaluated and analysed how self-learning an essential component in many unsupervised domain adaptation and self-supervised pre- training techniques, can be applied for adaptation both small and large-scale image recognition problems common in robustness research. We demonstrated new offte-of-the-art adaptation results with the EfficientNet-L.2 model on the benchmarks ImageNet-C. -R, and -A, and introduced a new benchmark dataset (ImageNet-D) which remains challenging even after adaptation. Our theoretical analysis shows the influence of the temperature parameter in the self-learning loss function the training stability and provides guidelines how to choose suitable value. Self-learning universally improves test-time performance under diverse, but systematic distribution shifts irrespective of the architecture pre-training method. We hope that work encourages both researchers and practitioners to use self-learning their data distribution shifts.
Reproducibility Statement We attempted to make our work as reprodueible as possible: We mostly used pre-trained models which are publicly available and we denoted the URL addresses of all used checkpoints; for the checkpoints that were necessary to retrain, we report the Github directories with the source code and used an official or verified reference implementation when available. We report all used hyperparameters in the Appendix and will release our code upon acceptance of the paper.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX) Symposium on Operating Systems Design and Implementation ((OSD)) 16), pp. 265-283, 2016.
Dan Hendrycks an. Natural adversarial examples. ArXiv preprint, abs/1907.07174, 2019. URL https: /aar:v///a/ivva//a wttp://////p//..o./////
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. ArXiv preprint, abs/1912.066800, 2019. URL https://
wwwliv.org//ba. /1912. /ttp///abbb/911.660... 06680.
David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: unified approach to semi-supervised learning and domain adaptation, 2021. 2,
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πly Sutskever, and Dario Amodei. Language models are few- shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hssuan-Tie Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: http://pooceeeenn......ooo 4457c0d6bccb4990 6741671166 ffffacclcccccoccccoccccccccc -et.baaaa44afaaaaaaaaaaaaaaa Abst html.
htt..//spppppooooo.nnnnnn neurips wwtpaaiiss.cppppppr..eeepaaa
cc http:/ppp/222///aa///
Tianle Cai, Ruiqi Gao, Jason D Lee, and Qi Lei. A theory of label propagation for subpopulation shift. arXiv preprint 1rXiv:2102.11220 2021. 2, 35
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. ArXiv preprint, abs/2104, 14294, 2021. URL htttps:// htt//axiiv..../ org/
wtwpoog///////ooo///////////aaar abs 14294. 3, 6, 8, 21
Ting Chen, Simon Korblithh Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020a. URL https:/ http://poooeddi...eeeoee Febc95ccdd551dar 81 01200c0ccc 4001111-006650000 ratt 8
Hugo
cc hcc/paper/2020/haa
Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift. In NeurIPS, 2020b. 2. 35
François Chollet. Xception: Deep learning with depthwise separable convolutions. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1800-1807. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2217..195 URL https: /oii. org/10 1109
22.09/vV/ CVpR.. www................. 2017 195. 4, 20
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the Fourteenth International Conference on Artificial
Intelligence and Statistics, 2011.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag. Edoardo Debenedetti, Nicolas
Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. ArXiv preprint, abs/2010.09670, 2020. URL https: //aaxivv.rr/abb//2111.
21
10

---

Under review as a conference paper at ICLR 2022
Fabio De Sousa Ribeiro, Francesco Calivá, Mark Swainson, Kjartan Gudmundsson, Georgios Leontidis, and Stefanos Kollias Applications, 32(9):4275-477 1291, 2020. 2, 36
Deep bayesian self-training. Neural Computing and
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):111-42, 2012. 4
Samuel F. Dodge and Lina J. Karam. A study and comparison of human and deep learning recognition performance under visual distortions. In International Conference on Computer Communications and Networks, ICCCN 2017, 2017.
Geoffrey French, Michal Mackiewicz, and Mark H. Fisher. Self-ensembling for visual domain adaptation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https::p////..m / http:///eeneeeeeeeeeeeeeeeee/eep
net forum? id=rkpoTaxR 3, 3,33, 37
Aram Galstyan and Paul R. Cohen. Empirical comparison of hard and soft label propagation for relational classification. In 17th international conference on Inductive logic programming. 2007.
3
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversaare training of neural networks. The journal of machine learning research, 27(1):2096 2030, 2016. 2, 4,
Robert Geirhos, Carlos R. Medina Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. Generalisation in humans and deep neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7549-7561, 2018. URL https: / 2018/ he//aahh/O/0 http//93755555 9 abstraccthhmm
http:///cccccecccceeeeeo. meg.rrceeedngss.u.i.......... cc/paper/ ff5 Swweeeeeeeeeeeeee
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Iragenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustmess. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: // opoeneevie. www.ne/tfoorrr d= beggghjjj0 9KXX. 1, 27
Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Satinder Singh and Shaul Markovitch (eds.), Proceedings the Thirty-First AAAI Conference Artificial Intelligence, Februury 4-9, 2017, San Francisco, California, USA, pp. 1919-1925. AAAI Press, 2017. URL http:/ /aaaiaaa....../..a org/ ocs/ index.dx....... phpp Sahp/AAs AAAI/ AAAI17 httpe///pppe////// vit//vveii///////e/ 14759.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems 17 [Neural Information Processing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada), pp- http//ppe/220////ee/ Abstract html.
529-536, 2004. URL https: https:: http://pooceeeein....... wwweiiiininnnnnnnniseeeeeeeeeies www.wnis.c.wpppeer...... wwt.d/ad111133000000000222222222
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEFF Conference Computer Vision and Putlern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016a. doi: 10.1109/CVPR.2016.
URL httht:///// www.oooo..o..oooooooooooooooe.oe /ttp://.....//....//////.......
1ww./1112...../ 90.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016b. doi: 10.1109/CVP22201..900 URL ht //wii/oiii/////// /tt//0.g.........//. 1109/1100000 CVPR. 2016. 90. 4. 5, 21
11

---

Under review as a conference paper at ICLR 2022
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: / /ppenreview. forum? d=HJz 6t. CqYm. 4, 27
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. ArXiv preprint. abs/2006.16241, 2020a. URL bttps:://rxxe
org/ alss 6006. 6241. 1, 5, 20, 21,
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020b. URL https:::t///..........t / hoppneveeereeeeeeeeeeeeeee net/forum? d==S1 gmrxHFvR. gim 4, 5, 21, 27
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS Deep Learning Workshop. 2014. 4, 20
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning. 2019. 6
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 2261-2269, IEEE Computer Society, 2017. doi: 10.1109//VVR.2211..455 URL https: /oi.i/ooiiioo//oo/oo org 243. 4, 5, 21, 32
/tgg/1100 10.11011190 /CVRR 2017.
Youngeun Kim, Donghyeon Cho. Kyeongtak Han, Priyadarshini Panda, and Sungeun Hong. Domain adaptation without source data. IEEE Transactions on Artificial Intelligence, 2021.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021. 8, 37
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Purt V 16. pp. 491-507. Springer, 2020. 33
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 4
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. International Conference on Machine Learning, pp. 5468-5467777 PMLR, 2020. 2, 35
Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, al. Universal source-free domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4544 4553, 2020. 2
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop: Challenges in Representation Learning (WREPL), 2013.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In 2020 IEFE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), 2020. 2
12

---

Under review as a conference paper at ICLR 2022
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning. 2020. 2
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. Proceedings of the European Conference on Computer Vision (ECCV), 2018. 1,
4, 5, 6, 20, 21
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In ACM
International Conference on Multimedia, 2010. 21, 37
Dirk Merkel. Docker: Lightweight linux containers for consistent development and deployment.
Limux J., 2014(239), 2014. ISSN 1075-3583, 37
Subhabrata Mukherjee and Ahmed Hassan Awadallah. classification with few labels. In NeurlPS, 2020. 2, 36
Uncertainty-aware self-training for text
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D A'mour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. ArXiv preprint, abs/2006. 10963, 2020. URL https:
http//bbs/0000 10963. 2
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8558-8567, 2021. 2, 35
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An uncertainty-avee awaar pseudo-label selection framework for semi-supervised learning. In ICLR, 2021. 2, 36
Evgenia Rusak, Lukas Schott, Roland Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. Increasing the robustness of dnns against image corruptions by playing the game of noise. ArXiv preprint, abs/2001.06057, 2020. URL https: arxiv.
org/abs// abs 20100011.200 06057. 1.27
Kate Saenko, Xingchao Peng. Ben Usman, Kuniaki Saito, and Ping Hu. Visual Domain Adaptation Challenge (VisDA-2019), 2019. URL http: //ai bu.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Advances in neural information processing systems, 2020. 2, 4, 6, 20, 24
Jun Shu, Qian Zhao, Keyu Chen, Zongben Xu, and Deyu Meng. Learning adaptive loss for robust learning with noisy labels. ArXiv preprint, abs/2002.006482, 2020. URL https: arxiv. org/abs/ 2002.000222 06422.
Rui Shu, Hung H. Bui, Hirokazu Narui, and Stefano Ermon. A DIRT-T approach to unsupervised domain adaptation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: http///wwnweeeeeeee/eeeee httppss////peeeeeeieeeee net. forumpi-=:lIq-------------------
3,
Kihyuk Sohn, David Berthelot, Chun-Liang Li. Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020. 35
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. ArXiv preprint, abs/2007.0811 1999, 2020. URL https: / // //rxiiv... a 08199.
13

---

Under review as a conference paper at ICLR 2022
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised domain adaptation through self-supervision. ArXiv preprint, 1tss/1999.11255, 2019a. BtLtht:s://aiiiii org/ abs/ 1909. 4,5
Yu Sun, Xiaolong Wang. Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training for out-of-distribution generalization. ArXiv preprint, 125/1999.13233. 2019b. URL
https: /atpvv.aaavvvv/aa///// org/ abs/1909. 13231. 2, 6,
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA. volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114, PMLR, 2019. URL
www.hnng.iiii....m... fiaar.press / /97 /et///aallaa.. html. 4, 21
O. Tange. Gnu parallel the command-line power tool. :login: The USENIX Magazine, 36(1):
42-47, 2011. URL http: /www. gnu. org/ s htt//s/pareeee heppaaaallll.... 37
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1. O Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-271 2020. doi: 37
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Fully test- time adaptation by entropy minimization. ArXiv preprint, abs/2006. 10726, 2020. URL https: //arxiv.
2006 10726.06666........ 10726. 2,
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In ICLR, 2020. 2, 35
Ross Wightman. Pytorch image models. pytorch- sichh-imasn image- s, 2019. 33, 37
Qizhe Xie, Minh-Thang Luong, Eduard H. Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. doi: 10. 10.1109/CVP442000 0T0R446000200000077 2020.01070. http::/ /ttp///iiii///////// wwwo/r/////0
IEEE, 2020a.
202 0. 01 070. 1, 3, 4, 5, 20, 21, 24
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference Computer Vision and Patlern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 5987-5995. IEEE Computer Society, 2017. doi: 10.1 109/CVPR.2001..644 URL https: /ooii /10
11C9/CVPR. 2 2017.00000000 634. 4. 21
Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In- n-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness. arXiv preprini ar 2012v:2012.04: 2020b. 2,
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016. URLhttp: //wp:/ bmva. org hwwc///w////////// / 201 6/ ht/pappersss paapers/paper0855 ndex html. 1.1...... 4,5
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: opwenevview....eooooee/ee net. forum? disdmiidddSSyssx...... 3
14

---

Under review as a conference paper at ICLR 2022
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. arXiv preprint arXiv:1100.09066, 2021. 2,
Zhilu Zhang and Ment R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp- 8792 8792-8802, 2018. URL https: htt.tt:::/prooeeddnngg/ httpp//poeennnss.eeepppoop neurin..nissss cc http/paper/221//ea0/ f292 w97bc199111..0 3add133222555555555 802 feea0-eeaa----- F
html. 3
Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le. Rethinking pre- pre-training and self-training. In NeurlPS, 2020. 2, 35
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pp. 289-305, 2018. 3,
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 5982-5991, 2019. 2, 35
15

---

Under review as a conference paper at ICLR 2022
A A TWO-POINT MODEL OF SELF-LEARNING
A. DEFINITION OF THE TWO-POINT MODEL
To understand the learning dynamics and properties of different loss functions and their hyperparameters, we propose simple model of self-learning, both for entropy minimization and pseudo-labeling.
A student network w I and teacher network w' R are trained on N data points {x; / with the cross-entropy loss function defined as
N C C= Σ [(x,)= 1=1
N Σ (x, w (x w*) + ox(xx w²) (oogo((xXx ())),
-
(6)
where 0x(2) 1 and (2)) 1
Here Ta and 7, denote the student and teacher temperature parameters. With stop gradient, student and teacher evolve in time according to
W = - v C (w", w'), w' a(w"
where a is the learning rate the teacher. Without stop gradient, student and teacher are set equal each other, and they evolve as
w -=-vw[(w) where w w'
(8)
We restrict the theoretical analysis the time evolution of the components of w in direction of two data points Xę and Xt, V/ s.t x wi and yt x w All other components y with i j k, I are neglected to reduce the dimensionality of the equation system. It turns out that the resulting model captures the neural network dynamics quite well despite the drastic simplification of taking only two data points into account (see Figure 2).
with stop gradient:
iit x V w- (E(xN) + [(xi)), ii -x Vw (f(x)) + ((xi)),
3/4 alyk vi), if =a -vf),
(9)
without stop gradient: in x Vw (c(xx) ((xi)), in = V.
XVV (f(xx) l(xi)).
A.2 PROOF OF PROPOSITION
Learning dynamics with stop gradient. Computing the stop gradient evolution defined in equation explicitly yields
N (oe(x] (=x] - ox(-x; w')os(xTw")) x; (10)
w = - V Vw L
T.
v' = (w" -w")
The second equality uses the well-known derivative of the sigmoid function, 0,a(2) =
((2)0(-2)) =
The equation system of 2d nonlinear, coupled ODEs for w E Rd and w' Rd in equation 10 is analytically difficult to analyze. Instead of studying the ODEs directly, we act on them with the data points x h N and investigate the dynamics of the components x w. = y/
T. Σ xx) (o.(v))o, o.(-vl)o.(v())
(1I)
U alyi v(.)
The learning rate of each mode y is scaled by (x, x:) which is much larger for = k than for i / in high-us nigh-dimenssonal spaces. In the two-point approximation, we consider only the two (in absolute
16

---

Under review as a conference paper at ICLR 2022
value) largest terms k, I for given l in the sum in equation 11. Any changes that v (t) and yi" (t) might induce in other modes yt" "("()) (8) are neglected, and so we are left with only four ODEs:
1
Vt
|xxx 11 (orlv)) oil -f(()))oo((
Ta
1
+
(x)
0rl -
To
1 ||x|| 2 (o, 10. m1) o (-11),,,00))
(12)
T
+ - (x xi) (oe Jo v/6) orl (-
1/ -a wh): ii alyi - vi).
The fixed points of equation 12 satisfy
16 1/ 1/ 1/ 0.
(13)
For a > O, requiring 1íf = 1/ = 0 implies that yi = v and yf = yf- For Ta = the two remaining equations i/ = yf vanish automatically so that there are no non-trivial two-point learning dynamics. For T, of 7.. there fixed point at y = vi since at this point, each bracket in equation 12 vanishes individually:
1 0.
ov(Un,))o_--un.)) va(-Ve.)o
(14)
16,1=0
At the fixed point Uf vi est O. w and w are orthogonal to both X₃ and x; and hence classification fails. If this fixed point is stable, w and w' will stay at the fixed point once they have reached it, i.c. the model collapses. The fixed point is stable when all eigenvalues of the are negative. This the
a.t Jacobian J of the ODE system equation 12 evaluated 3/k yi"
case whenever < Tr:
xx
x
0
4
J 81
(x)
1
xx
0
=
=0
0
0
a
eigenvalues: d1 = A2 = aa < 0,
1
1
A3,1
+ +V/1sx /|/xx| + + |x]|
2 |xxx| 1|2 + 4(x; 2
+||28
1 | 1
8
T1
Te
< <||xa| || ||x:
20 with equality if =x
(15)
To sum up. training with stop gradient and T, > avoids collapse of the two-point model to the trivial representation yx s.f s.t yt = 0 since the fixed point is not stable in this parameter regime.
Learning dynamics without stop gradient leads to an additional term in the gradient:
Without stop gradient, we set w' w w which
1
w = -
(x. (iocTw)o((xpp] ox(-x; (x, w)) x;
To
1=1
(16)
ax(x w)oo(=x] w) (loga,(x; w) loggo (-xx w)) xs.
7
=log ((+t22/70)/(14e a(
17

---

Under review as a conference paper at ICLR 2022
As before, focus the evolution of the two components yk w xy and yr w X/-
ii =1|x=|1? - (oul
1 (yr
T
1
1
+ (xx xi)
(o)(m))s(-w)--) - (wi) + (v)))aa--11
Ts
Ts
(17)
31 = =|x1||? (
1
-
Tt
+ (x xi)
o.
(.l (y
TsT
There is fixed point at ys = yl = 0 where each bracket in equation 17 vanishes individually,
1 (or(vu.)ss( -0( -
= 0.
(18)
TaTx
ga.l
The Jacobian of the ODE system in equation 17 and its eigenvalues evaluated at the fixed point are given by
xx !| (x,
X xi
J
xi)
2
1
2
1
ye =
4T,
1
2
A1,2 =
|xx
+
||x|
1| - 2|
|
|||
4 x1)2 + |||x=|1 1 ||x/||
8Ts
20 with equality x;=
(19)
Hence the fixed point is unstable when т, 72/2 /2 and thus the model without stop gradient does not collapse onto yl: yt in this regime.
A.3 SIMULATION OF THE TWO-POINT MODEL
For visualization purposes in the main paper, we set w w 0.5, 0.5)T and train the model using instant gradient updates on the dataset with points with learning rate 0.1 and momentum 0.9. We varied student and teacher temperatures on log-scale with 250 points from 10-3 to 10. Qualitatively similar results can be obtained without momentum training, at higher learning rates (most likely due to the implicit learning rate scaling introduced by the momentum term).
[11 0) and = [0,- -1] using SGD
Note that the temperature scales for observing the collapse effect depend on the learning rate, and the exact training strategy-lower learning rates empirically prevent the model from collapsing and shift the convergence region. The result Figure 2 will hence depend on the exact choice of learning rate (which is currently not considered in our continuous time evolution theory), while the predicted region without collapse is robust to details of the optimization.
To visualize the impact of different hyperparameterrs, different learning rates using gradient descent with (Figure 3) and without momentum (Figure 4), and with different start conditions (Figure 5), which all influence the regions where the model degrades, but not the stable regions predicted by our theory.
show variants of the two point model with
18

---

Under review as a conference paper at ICLR 2022
Ir: 10
0.1
0.01
0.001
PL
7I. 2Ta
10g10 f2
0
ENT
Error
0%
10g10 7a
50%6
100%
-2
0
1
2
-1
0
1-2
-1
0
1-2
-1
0
1-2
-1
0
1
log10
log10 7t
log
7
log10
log10 Tt
Figure 3: Entropy minimization (top) Training two point model with momentum 0.9 and different learning rates with initialization w w' [0.5, 0.5]
Ir:
10
0.1
0.01
0.001
PL
2T
10g10 To
22
ENT
Error
0%
log10 Fa
50%
-2
0
2
-1
0
2
log
logio Tt
log T
log
logio Tt
Figure 4: Training two point model without momentum and different learning with initialization w w² [0.5, 0.5] Note that especially for lower learning rates, longer training would increase the size of the collapsed region.
10
0.1
0.01
0.001
PL
Ti 27
log10 f6
-2
ENT
0%
10g10 Ts
50%
100%
-2
0
2
-1
0
1
1
2
0
2
1
log T
logio Tt
log T
log T
log Tt
Figure 5: Training point model with momentum 0.9 and different learning rates with initialization w w [0.6, 0.3)
19

---

Under review as a conference paper at ICLR 2022
B ADDITIONAL INFORMATION ON USED MODELS
B.1 DETAILS ON ALL HYPERPARAMETER WE TESTED FOR DIFFERENT MODELS
For all models except EfficientNet-22, we adapt the batch norm statistics the test domains following (Schneider et al., 2020). We do not expect significant gains for combining EfficientNet-L2 with batch norm adaptation: as demonstrated in (Schneider et al., 2020), models trained with large amounts of weakly labeled data do not seem to benefit from batch norm adaptation.
ResNet50 models We use a vanilla ResNet50 model and compare soft- and hard-labeling against entropy minimization and robust heedo-labeling To find optimal hyperparameters for all methods, we perform an extensive evaluation and test (i) three different adaptation mechanisms (ii) several learning rates 1.0 10- 4, 1.0 10-3, 1.0 10-2 and 5.0 10-2, (iii) the number of training epochs and (iv) updating the teacher after each epoch or each iteration. For all experiments, we use a batch size of 128. The hyperparameter search performed on IN-C dev. We then use the optimal hyperparameters to evaluate the methods on the IN-C test set.
ResNeXt101 models The ResNeXt101 model is considerably larger than the ResNet50 model and we therefore limit the number of ablation studies we perform for this architecture. Besides a baseline, we include a state-of-the-art robust version trained with Deep:Augment-AAgmiix (DAug+AM, Hendrycks et al., 2020a) and a version that was trained on 3.5 billion weakly labeled images (IG-3,5B, Mahajan et al.. 2018). We only test the two leading methods on the ResNeXt101 models (ENT and RPI.). We vary the learning rate in same interval for the ResNet50 model but scale down linearly to account for the smaller batch size of 32. We only train the affine batch normalization parameters because adapting only these parameters leads to the best results on ResNer50 and is much more resource efficient than adapting all model parameters. Again, the hyperparameter search is performed only on the development corruptions IN-C. We then use the optimal hyperparameters to evaluate the methods on the IN-C test set.
EfficientNet-L2 models The current state of the art on IN, IN-C, IN-R and IN-A is an EfficientNet-L2 trained on 300 million images from JFT-300M (Chollet, 2017; Hinton et al., 2014) using a noisy student-teacher protocol (Xie et al., 2020a). We adapt this model for only one epoch due to resource constraints. During the hyperparameter search, only evaluate three corruptions on the IN-C development set" and test the learning rates 4.6 x 10-2, 4.6 x 10-3, $, 4.6 x 10-4 and 4.6 10-3 We use the optimal hyperparameters evaluate ENT and RPL on the full IN-C test set (with all severity levels).
UDA-SS models We trained the models using the scripts from the official code base at github. com/yuealsprograassuuaa release. We used the provided scripts for the cases: (a) source: CIFAR 10, target: STL and (b) source: MNIST, target: MNIST-M. For the case (c) source: CIFAR I0, target: CIFAR1O-C, we used the hyperparameters from case (a) since this case seemed to be the closest match to the new setting. We think that the baseline performance of the UDA-SS models can be further improved with hyperparameter tuning.
DANN models To train models with the DANN-method, we used the PyTorch implementation htt..//ottttooo/DAANNII -py33. The code base only provides scripts and
of this paper at hyperparameters for the case (b) source: MNIST, target: MNIST-M. For the cases (a) and (c), we used the same optimizer and trained the model for 100 epochs. We think that the baseline performance of the DANN models can be further improved with hyperparameter tuning.
Preprocessing For IN, IN-R, IN-A and IN-D, we resize all images to 256 x 256 px and take the center 224 x 224 px crop. The IN-C images are already rescaled and cropped. We center and re-scale the color values with JBRBB = [0.485, 0.456, 0.406) and RRGB = [0.229, 0.224, 0. 0.225). For the EfficientNet-L.2, we follow the procedure in Xie et al. (2020a) and rescale all inputs resolution of 507 x 507 px and then center-crop them to 475< x 475 px.
We compare the results of computing the dev set on the 1. 3 and 5 severities versus the 1, 2, 3, 4 and 5 severities Xes0e model in the Supplementary material.
20

---

Under review as a conference paper at ICLR 2022
B.2 FULL LIST OF USED MODELS
ImageNet scale models ImageNet trained models (ResNet50, DenseNet16l, ResNeXt) are taken directly from torchvision (Marcel & Rodriguez, 2010). The model variants trained with DeepAugment and AugMix augmentations (Hendrycks et al., 2020b;a) are taken from https: The weakly-supervised ResNeXt101 model is taken from the PyTorch Hub. For EfficientNet (Tan & Le. 2019), we use the Py Torch re-implementation available https//gghuub..o/m/iigaaaaaaga wwwpa//wwwiiiiaaaaeeeeeeeeeeeeee efficientnet-pytoreeh This is a verified re-implementation of the original work by Xie et al. (2020a). We verify the performance on ImageNet, yielding a 88.23% top-1 accuracy and 98. 98.546% top-5 accuracy which is within 0.2% points of the originally reported result (Xie et al., 2020a). On ImageNet-C. our reproduced baseline achieves 28.9% mCE vs. 28.3% mCE originally reported by Xie et al. (2020a). As noted in the re-implementation, this offset is possible due to minor differences in the pre- pre-prccessing It is possible that our adaptation results would improve further when applied on the original codebase by Xie et al..
gggitubb.om/hheddrycss/iieee
Small scale models We train the UDA-SS models using the original code base github.com/ peeetsprograms/udarreeaass with the hyperparameters given in the provided bash scripts. For our DANN experiments, we use the PyTorch implementation github.com/fungg Sattu//coooo/uggtoDD/NN..yy We use the hyperparameters in the provided bash scripts.
The following Table 11 contains all models we evaluated on various datasets with references and links to the corresponding source code.
Table 11: Model checkpoints used for our experiments.
Model
Source
WideResNet(2S 10) (Croee al., 2020)
(44,ReeNet(402))4Auppp (Cioce al., 2020)
https:/gittubbcoo https///ittub..ommRRoussff www.://wwww...oooooooooeoeeeeeoe
ResNe150
https//github.cc httpiiiu...oo/ytoooootooo www.pppywoccliisssoteeeeeseeeeee /sioo//mdelsss
ResNcX:101, x2 (He et al, 2016b)
http:://iiubb.compsttonhhh wwt.rr/cliiiiiiieeeeeeeeeeeerrie
DenseNet (Huang et al., 2017)
https//pthuu owwppp/rooowiiiisttttoeetettttee
ResNeX:101, 32x (Xie 2017)
httpss/pp porchtt wrg.huuubttccceboookreecccdd WSL-Imian -Images resnext/
ReeNadOODDepppgnnoo--AAugtttt (Hendrycks al., 2020a)
/gith/ how.be/o//ooooooo//meeeeett
ResNexti0l (Hendrycks et al., 2020a)
https://rithhh.co httggg/hhh.ooe/eennnn///imeeeeee
ResNext101 x2 IG-3.5B (Mahajan et al., 2018)
https /giph// hww.h///wwwwoobooooosscce///eeee htt.s//www.....oooooooooooooooee
Noisy Student EfficientNee-1. 1.2 (Xie et al., 2020a)
httpe//githhh.. http:///thuh..co/twigggig hwwp://wwghhaanggeefffeeeeee pytorch
V/T-S/16 (Caron et al., 2021)
https//githb www/tsceeoonreeeeeeeeeeoooooeeee
21

---

Under review as a conference paper at ICLR 2022
C DETAILED AND ADDITIONAL RESULTS ON IN-C
C.I DEFINITION OF THE MEAN CORRUPTION ERROR (MCE)
The established performance metric IN-C is the mean Corruption Error (mCE), which obtained by normalizing the model's top-1 errors with the top-1 enrors of AlexNet across the C=15 test corruptions and S=5 severities:
C Σ Σ s= model
mCE(model)
(20)
AleNNe
The AlexNet emros used for normalization are shown in Table 12.
Category
Corruption
topl error
Gaussian Noisc
0.886428
Noise
Shot Noise
0.894468
Impulse Noise
0.922640
Defocus Blur
0.819880
Glass Blur
0.826268
Blur
Motion Blur
0.785948
7oom Blur
0.798360
Snow
0.866816
Frost
0.826572
Weather
Fog
0.819324
Brightness
0.564592
Contrast
0.853204
Elastic Transform
0.646056
Pixelate
0.717840
Digital
JPEG Compression
0.606500
Hold-out Noise
Speckle Noise
0.845388
Hold-out Digital
Saturate
0.658248
Hold-out Blur
Gaussian Blur Spatter
0.787 108
Hold-out Weather
0.717512
Table 12: AlexNet top1 errors ImageNet-C
C.2 DETAILED RESULTS FOR TUNING EPOCHS AND LEARNING RATES
We tune the learning rate for all models and the number of training epochs for all models except the EfficientNer-l.2. In this section, present detailed results for tuning these hyperparameters for all considered models. The best hyperparameters that we found in this analysis, are summarized in Table 17.
Table 13: mCE in on the IN-C dev set for ENT and RPI. for different numbers of training epochs when adapting the affine batch norm parameters of ResNet50 model.
Table 14: mCE ( ()) in % on the IN-C dev set for different learning rates for EfficientNet- L2. We favor q 0.8 over slightly improved robustness to changes the learning rate the worst case error setting.
0.7 due
criterion
ENT
RPL
Ir 10
10-3
10
10
10 10m3
10 10-2
epoch
Ir (4.6 x)
base
10 3
10
10 5
10
0
60.2
60.2
60.2
60.2
60.2
60.2
54.3
50.0
72.5
57.4
51.1
52.5
ENT
25.5
22.2
2
52.4
50.9
96.5
55.8
49.6
57.4
RPL
25.5
60.3
21.3
23.3
n/a
3
51.5
51.0
112.9
54.6
49.2
64.2
RPL
25.5
58.2
21.4
23,4
n/a
4
51.0
52.4
124.1
53.7
49.0
71.0
5
50.7
53.5
131.2
52.9
48.9
76.3
50.7
53.5
131.2
52.9
48.9
76.3
22

---

Under review as a conference paper at ICLR 2022
Table 17: The best hyperparameters for all models that found IN-C. For all models, fine-tune only the affine batch normalization parameters and use for RPL. The small batchsize the EfficientNet model is due to hardware limitations.
number of epochs
Model
Method
Learning rate
batch size
vanilla ResNe150
ENT
10
vanilla ResNe150
RPL
1 10
128
5
vanilla ResNeXt101
ENT
2.5 10
128
vanilla ResNext111
RPL
2.5 10
IG-3,5B ResNeXt101
ENT
2 10
128
4
IG-3.5B ResNeXt101
2.5 10
128
DAug+AM ResNeXt101
ENT
2.5 x 10
128
AA ResNeXt101
RPL
2.5 10
128
4
EfficientNet-L22 Nef1
4.6 10-5
RPL
4.6 x 10
Table 15: mCE % IN-C dev for entropy minimization different learning rates and training epochs for ResNeXt101, (div:=diverge))
Table 16: mCE in % IN-C dev for robust pseudo- labeling different learning rates and training epochs for ResNeXt101. (div.=diverged)
ENT epoch
Baseline
IG
DAug+AM le-3
RPL epoch
Baseline
IG-3,5B
DAug-AM
5e-3
le.4
le-3
Se-3
le-4
5e-3
le-4
le-4
Se-3
BASE
53.6
53.6
53.6
47.4
47.
474
37.4
37.4
37.4
BASE
53.6
53.6
53.6
47.4
47.4
47.4
37.4
37.4
37.4
43.0
92.2
div.
40,9
40.4
38.6
35,4
46.4
div.
43.4
51.3
div.
45.0
39.9
43.6
35.3
35.1
79.1
3
45.4
131.9
div.
39.3
42.6
76.1
35.5
122.5
div.
3
42.0
72.6
div.
42.4
39.4
$2.9
34.7
40.1
133.5
133.5
C.3 DETAILED RESULTS FOR ALL IN-C CORRUPTIONS
We outline detailed results for all corruptions and models in Table 18. Performance across the severities in the dataset is depicted in Figure 6. All detailed results presented here are obtained by following the model selection protocol outlined in the main text.
RN50
RNx101
RNx101 IG-3.58 RNx101 DeepAug +Augmix
Naisy Student
ENT RPL
MCE (96]
60
Base
40
20
2
Severity
Severity
Severity
Severity
Severity
Figure 6: Severity-wise mean corruption error (normalized using the average AlexNet baseline error each corruption) for ResNet50 (RN50), ResNext101 (RNx101) variants and the Noisy Student 1.2 model. Especially for more robust models (DeepAumment-Augme and Noisy Student I.2), most gains are obtained across higher severities 4 and 5. For weaker models, the baseline variant (Base) is additionally substantially improved for smaller corruptions.
23

---

Under review as a conference paper at ICLR 2022
Table 18: Detailed results for each corruption along with corruption (mCE) reported in Table 2 main paper. We show (unnormalized) top-1 error rate averaged across 15 test corruptions along with the mean corruption ertor (mCE: which is normalized). Hyperparameter selection for both ENT and RPL was carried out on the dev corruptions as outlined in the main text. Mismatch in baseline mCE for EfficientNet- L2 can be most likely attributed to pre-processing differences between the original tensorflow implementation Xie al. (2020a) and the PyTorch reimplementation we employ. We with slightly weaker baselines for ResNet50 and ResNext101 than Schneider et al. (2020): ResNe150 than previously reported results (typically 0. % points) due to the smaller batch size of 128 and 32. Smaller batch sizes impact the quality of re-estimated batch norm statistics when computation performed on the fly Schneider et al. (2020), which is of no concern here due to the large gains obtained by eseudo-labeling
ResNext101 results are slightly worse
Bauns
shot
impulae
defocats
glaus
moticn
zaum
snow
frust
Feg
bright
contanl
elastic
piselate
jreg
mCE
ResNet50
Bascline (Schncider et al., 2020)
Baseline (ours)
57.2
59.5
60.0
61.4
62.3
51.3
49.5
54.6
54.1
39.3
29.1
46.7
41.4
38.2
41.8
62.8
ENT
45.5
45.5
46.8
48.4
48.7
40.0
40.3
42.0
46.6
33.2
28.1
42.4
35.2
32.2
35
51.6
ResNeXt101 Baseline
Baseline (Schneider et 2020)
56.7
Baseline (ours)
$2.8 54.1
54.0
55.4
56.8
46.7
46.6
48.5
49.4
36.6
25.4
42.8
37.8
32.5
36.7
56.8
40.5
39.5
41.4
41.6
43.0
34.1
34.5
35.0
39.4
28.5
24.0
33.8
30.3
27.2
30.5
44.3
RPL
39.4
38.9
39.8
40.3
41.0
334
33.8
34.6
38.7
28.0
23.7
31.4
29.8
26.8
30.0
43.2
ResNeXt101 IG-3.5B
Baseline (Schneider et 2020)
Baseline (ours)
50.7
51.5
53.1
54.2
55.5
45.5
44.7
41.7
42.0
28.1
20.1
33.8
35.4
27.8
33.9
518
ENT
38.6
38.1
40.4
414
41.5
33.8
33.6
32.2
34.6
24.1
19.7
26.3
27.6
24.2
27.9
40.8
40.8
42.1
42.4
34.7
19.6
26.1
27.5
ResNeXt101 DeepAng+Augnt
Baseline (Schneider et al., 2020)
38.0
Baseline (ours)
30.0
30.0
30.2
32.9
35.5
28.9
31.
33.3
32.8
29.5
22.
28
31
23.0
26.5
38.1
28.7
28.5
29.0
29.8
30.9
26.9
28.0
30.5
26.2
23.2
26.3
28.5
23.7
26.0
35.5
RPI.
28.1
27.8
28.3
29.1
30.1
26.3
27.4
28.8
29.8
25.9
22.7
25.6
27.9
23.2
25.4
34.8
Naisy Student L.2
Baseline (Xie al., 2020s)
28.3
Baseline (ours)
21.6
22.0
20.5
23,9
40.5
19.8
23.2
22.8
26.9
21.0
15.2
21.2
24.8
18.6
28.9
ENT
18.5
18.7
17.4
18.8
23.4
16.9
18.8
17.1
19.6
16.
14.1
16.6
19.6
15.8
16.5
23.0
17.8
18.0
17.0
18.1
21.4
16.4
16.4
18.7
15.7
136
15.6
15.6
22.0
C.4 DETAILED RESULTS FOR THE CIFARIO-C AND UDA ADAPTATION
Table 19: Detailed results for each corruption along with mean error on CIFARIO-C as reported Table in the main paper.
Fauss
alell
impulee or
defocus
plass
mation
Zrom
MROW
frsst
Fog
bripht
comtrant
elasic
pisclate
Jpeg
Nvg
WRN-18. vanilla
Baseline
53.0
41.2
44.7
18.5
22.3
24.4
18.1
25.0
11.2
17.4
16.2
28.0
22.4
26.5
BN adapt
20.8
17.6
22.7
8.1
28.4
10.9
9.2
14.2
13.0
8.7
6.8
8.5
13.5
12.1
21.0
14.4
12.6
RPI.
19.6
16.7
21.9
8.1
27.1
10.9
8.9
13.9
13.0
8.7
6.9
8.4
13.2
11.7
20.1
13.9
WRN 40-2 AM
Baseline
19.1
14.0
13.3
6.3
17.1
7.9
7.0
10.4
10.6
8.5
5.9
9.7
9.2
16.8
11.9
11.2
adapt
TENT
10.8
9.1
10.9
6.0
13.4
7.2
6.3
8.4
7.8
7.1
5.7
7.1
9.2
7.4
11.2
8.5
RPL.
12.4
9.5
8.2
WRN. 26 UDA
Baseline
26.0
24.7
19.3
22.4
56.2
32.4
32.1
31.7
31.2
26.6
15.8
20.4
26.3
21.5
28.9
27.7
BN adapt
20.5
19.0
15.6
13.5
43.1
19.4
18.3
23.1
21.2
16.2
12.8
14.1
20.9
16.7
23.4
19.9
RPL
18.1
17.1
13.2
11.9
41.5
17.3
16.1
20.4
19.1
14.5
11.8
12.7
18.8
18.1
22.6
18.2
24

---

Under review as a conference paper at ICLR 2022
Table 20: Detailed results for the UDA methods reported in Table 2 of the main paper.
Bascline
BN adapt
RPL
ENT
UDA IIFIIO--STLO,, topl error on target (%]())
WRN-26-16 16 UDA-SS WRN-26-16 DANN
28.7 25.0
24.6 25.0
22.9
21.8
24.0
23.9
UDA MINIST-MNIST-M, topl error target [%](
WRN 26- 16 UDA-SS
WRN-26-168 WRN-26-2 DANN
4.8
3.9
2.4
2.0
11.4
6.2
5.2
5.1
C.5 ABLATION OVER THE HYPERPARAMETER 9 FOR RPL
For RPL, we must choose the hyperparameter q. We performed an ablation study over and show results in Table 21, demonstrating that RPL. is robust to the choice y, with slight preference to higher values. Note: In the initial parameter sweep for this paper. only compared g 0.7 and q = 0.8. Given the result Table 21, could be interesting to re-run the models in Table 1 of the main paper with q 0.9, which could yield another (small) improvement in mCE.
Table ImageNet-C dev set mCE %, vanilla ResNet50, batch size 96. We report the best score across maximum of six adaptation epochs.
0.5
0.6
0.7
0.8
0.9
mCE (dev)
49.5
49.3
49.2
49.2
49.1
C.6 SELF-TRAINING OUTPERFORMS CONTRASTIVE TEST-TIME TRAINING (SUN ET AL.. 2019B)
Sun al. (2019b) use ResNet18 for their experiments ImageNet and only evaluate their method on severity 5 of IN-C. To enable a fair comparison, we trained a ResNet18 with both hard labeling and RPL and compare the efficacy of both methods Test-Time Training in Table 22. For both hard labeling and RPL, we use the hyperparameters we found for the vanilla ResNet50 model and thus, we expect even better results for hyperparameters tuned on the vanilla ResNet18 model and following our general hyperparameter search protocol.
While methods (self-learnng and TTT) improve the performance over a simple vanilla ResNet 18, we note that even the very simple baseline using hard labeling already outperfoms Test-Time Training; further gains are possible with RPL. The result highlights the importance of simple baselines (like self-learning) when proposing new domain adaptation schemes. It is likely that many established DA techniques more complex than the basic self-lcarning techniques considered this work will even further improve over TTT and other adaptation approaches developed exclusively in robustness settings.
Table 22: Comparison of hard- hard-pseudo labeling and robust escudo-labeling to Test-Time Training Sun et al. (2019b): Top-1 1 error for ResNet18 and severity for all corruptions. Simple hard ascudo-labeling already outperforms TTT, robust pseudo labeling over multiple epochs yields additional gains.
gauss
shot
impulse
defocus
glass
moton
200m
SnOW
frost
fog
bright
contrast
elastic
pixelate
ipeg
Avg
vanilla ResNet18
98.8
98.2
99.0 88.6 91.3 88.8 82,4
89 83.5 85.7 48.7 96.6 83.2 76.9 70.4
85.4
Test-Time Training
73.7
71.4
73.1
76.3
93.4
71.3
66.6
64.4
81.3
52.4
41.7
64.7
55.7
52.2
55.7
66.3
hard PL. (1 epoch)
73.2
70.8
73.6
76.5
75.6
63.9
56.1
59.0
65.9
48.4
39.7
85.2
50.4
47.0
51.5
62.5
RPL (4 epochs)
71.3 68.3
71.7
76.2
75.6
61.5
54.4
56.9
67.1
47.3
39.3
93.2
48.9
45.7
50.4
61.9
25

---

Under review as a conference paper at ICLR 2022
C.7 EFFECT OF BATCH SIZE AND LINEAR LEARNING RATE SCALING
How is self-learning performance affected by batch size constraints? We compare the effect of different batch sizes and linear learning rate scaling. In general, we found that affine adaptation experiments on ResNet50 scale can be run with batch size 128 on a Nvidia V100 GPU (16GB), while only batch size 96 experiments are possible on RTX 2080 GPUs.
The results in Table 23 show that for ResNe150 model, higher batch size yields generally better performance.
Table 23: ImageNet-C C dev set mCE for various batch sizes with lincar learning rate scaling. All results are computed for vanilla ResNet50 model using RPL with 0.8, reporting the best score across maximium of six adaptation epochs.
batch size learning rate (x10-3)
16
32
80
128
0.125
0.250
0.500
0.625
0.750
1
dev mCE
53.8
51.0
49.7
49.3
49.2
48.9
C.8 PERFORMANCE OVER DIFFERENT SEEDS IN A RESNET5O ON IMAGENET-C
To limit the amount of compute, we ran RPL and ENT for our vanilla ResNet50 model three times with the optimal hyperparameters. The averaged results, displayed as "mean (unbiased std)" are:
Table 24: ImageNet-C. performacce for three seeds ResNet50 for ENT and
ResNet50 + self-lcarnng
mCE on IN-C dev [%]
mCE on IN-C test [%]
ENT RPL
50.0 (0.04)
51.6 (0.04)
48.9 (0.02)
50.5 (0.03)
C.9 SELF-LFARNING AS CONTINUOUS TEST-TEST ADAPTATION
We test our method on continuous test-time adaptation where the model adapts to a continuous stream of data from the same domain. In Fig. 7, we display the eπror of the Noisy Student L2 model while is being adapted to ImageNet-C and ImageNet-R. The model performance improves as the model sees more data from the new domain. We differentiate continuous test-time adaptation from the online test-time adaptation setting (Zhang et al., 2021) where the model is adapted to each test sample individually, and reset after each test sample.
ImageNet-C
(ii) ImageNet-R
28.9
28
Top-1 Error [%]
24
23.5
mCE [%6]
Baseline
26
22
ENT
24
RPL
23.0
20
19.7
22.0
18
17.4
4
2
3
Samples [x109]
Samples [x10*]
Figure 7: Evolution of error during online adaptation for EfficientNet-L2.
26

---

Under review as a conference paper at ICLR 2022
D DETAILED AND ADDITIONAL RESULTS ON IN-D
D.1 EVALUATION PROTOCOL ON IN-D
The domains in IN-D differ in terms of their difficulty for the studied models. Therefore, to calculate an aggregate score, we propose normalizing the eπror rates by the error achieved by AlexNet on the respective domains calculate the mean error, following the approach in Hendrycks & Dietterich (2019) for IN-C. This way, obtain the aggregate score mean Domain Error (mDE) by calculating mean over different domains,
E
D Σ
DE
mDF
(21)
EAlexNet
d=1
where E/ is the top-1 error of classifier domain d.
For all IN-D results we report in this paper, we chose the hyperparameters on the IN-C dev set. We tried a different model selection scheme on IN-D as control experiment with "Leave one out cross-validation" (LloutCV): with round-robin procedure, we choose the hyperparameters for the test domain on all other domains. We select the same hyperparameters when tuning the "dev" set: For the ResNet50 model, we select over the number of training epochs (with a maximum of training epochs) and search for the optimal learning rate in the set [0.01, 0.001, 0.0001]. For the EfficientNNt-L.2 model, we train only for one epoch as before and select the optimal learning rate in the set [4.6 x 10-3 4.6 x 10 4.6 x 10-5, 4.6 10-6) This model selection leads worse results both for the ResNet50 and the EfficientNet- L2 models, highlighting the robustness of our model selection process, see Table 25.
Leave-one-ot-crrss--allaaaa
Table 25: mDE in % on IN-D for different model selection strategies.
model
model selection LloutCV
IN-C dev
ResNet50 RPL
81.3
76.1
ResNet50 ENT EfficientNet-L2 ENT
82.4
77.3
69.2
66.8
FfffcientNet--.2 RPI,
69.1
67.2
D.2 DETAILED RESULTS FOR ROBUST RESNET50 MODELS ON IN-D
We show detailed results for all models IN-D for vanilla evaluation (Table 26) BN adaptation (Table 27), R. (Table 28) and ENT(eable 29). For RPL and ENT, we use the same hyperparameters that chose
IN-C 'dev' set. This means we train the models for 5 epochs with RPLG=0.8 and for one epoch with ENT.
We evaluate the pre-trained and public checkpoints of SIN (Geirhos et al., 2019), ANT (Rusak et al., 2020), ANT+SIN (Rusak et al., 2020), AugMix (Hendrycks et al., 2020b), DeepAugment (Hendrycks al., 2020a) and DeepAug+Augmix (Hendrycks et al., 2020a) in the following tables.
Table 26: Top-1 error on IN-D in % obtained by robust ResNet50 models. For reference, we also show the mCE on IN-C and the top-1 error on IN-R. See main test for model references.
Model
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
mDE
IN-C
IN-R
vanilla
76.0
89.6
65.1
99.2
40.1
82.0
88.2
76.7
63.9
SIN
71.3
88.6
62.6
97.5
40.6
77.0
85.6
69.3
58.5
ANT
73.4
88.9
63.3
99.2
39.9
80.8
62.4
61.0
ANT+SIN
68.4
88.6
60.6
95.5
40.8
70.3
83.1
60.7
$3.7
AugMix
70.8
88.6
62.1
99.1
39,0
78.5
85.4
65.3
58,9
DuepAugment
72.0
88.8
61.4
98.9
39.4
78.5
85.6
60.4
57.8
DsepAug+Augmix
68.4
88.1
58.7
98.2
39.2
75.2
83.4
53.6
53.2
27

---

Under review as a conference paper at ICLR 2022
Table 29: Top-1 error IN-D in % obtained by state-of-the ar- robust ResNet50 models and ENT. See main text for references the used models.
Model
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
mDE
vanilla
65.1
85.8
59.2
98.5
38.4
75.8
77.3
SIN
62.1
87.0
57.3
99.1
39.0
68.6
75.5
ANT
64.2
86.9
58.7
97.1
38.8
72.8
76.5
ANT+SIN
62.2
57.7
95.8
40.1
68.7
75.2
AugMix
60.2
84.6
55.8
97.6
36.8
72.0
74.4
DeepAugment
59.5
85.7
54,4
98.0
37.1
66.4
73.3
DuepAug+Augmix 58.4
84.3
54.7
98.5
38.1
63.6
72.7
Table 30: mDE IN-D in % obtained by robust ResNet50 models with baselinc evaluation, batch norm adaptation, R.L and ENT. See main text for model references.
mDE on IN-D (
Model
Baseline BN adapt RPI 000 ENT
vanilla
88.2
80.2
76.1
77.3
SIN
85,6
79.6
76.8
75.5
ANT
86.9
80.7
78.1
76.5
ANT+SIN
77.8
76.1
75.2
AugMix
85,4
78.4
74.6
74.4
DeepAugment
85.6
78.8
74.8
73.3
DeepAugment-AAugmix
83.4
74.9
72.6
72.7
Table 27: Top1 emrrr on IN-D in as obtained by state-of-the -he-rrt robust ResNet50 models and batch norm adaptation, with batch size of 128. See main text for model references.
Model
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
vanilla
70.2
88.2
63.5
97.8
41.1
78.3
80.2
67.3
89.7
62.2
97.2
44.0
75.2
79.6
ANT
69.2
89.4
63.0
97.5
42.9
79.5
80.7
ANT+SIN
64.9
88.2
60.0
96.8
42.6
73.0
77.8
AugMix
66.9
88.1
61.2
97.1
40.4
75.0
78.4
DeepAugment
89.7
60.0
42.5
78.8
DeepAug+ Augmix
61.9
85.7
57.5
95.3
40.2
69.2
74.9
Table 28: Top-1 error on IN-D as obtained by state-of ffate-of-theart the-art robust ResNet50 models and KPL. See main text for model references.
Model
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
vanilla
63.6
85.1
57.8
99.8
37.3
73.0
76.1
SIN
60.8
86.4
56.0
99.0
37.8
67.0
76.8
ANT
63.4
86.3
57.7
99.2
37.7
71.0
78.1
ANT+SIN
61.5
86.4
56.8
97.0
39.0
67.1
76.1
AugMix
59.7
83.4
54.1
98.2
35.6
70.1
74.6
DeepAugment
58.1
84.6
53.3
99.0
36.2
64.2
DeepAug+Augnmt
57.0
83.2
53.4
99.1
36.5
61.3
72.6
The summary results for all models are shown in Table 30.
We show the top-1 error for the different IN-D domains versus training epochs for vanilla ResNet50 in Fig. 8. We indicate the epochs and 5 at which we extract the emrrs with dashed black lines.
28

---

Under review as a conference paper at ICLR 2022
dipam
Isbogapph
82-
53-
70
100.0
10.0
92-
16.5
06.
87-
07.5
Epochs
Figure 8: Top-1 error for the different IN-D domains for a ResNet50 and training with RPLg=0.s and ENT. We indicate the epochs at which we extract the test errors by the dashed black lines (cpoch for ENTand epoch 5 for RPLg=0.8).
D.3 DETAILED RESULTS FOR THE EFFICIENNEET-2 NOISY STUDENT MODEL ON IN-D
We show the detailed results for the EfficientNet-L.22 Noisy Student model on IN-D in Table 31.
Table 31: Top-1 error on IN-D in % for EfficientNet-L2
Domain
Baseline
ENT
RPL
Clipart
45.0
39.8
37.9
Infograph
77.9
91.3
94.3
Painting
42.7
41.7
40.9
Quickdraw
98.4
99.4
99.4
Real
29.2
27.9
Sketch
56.4
48.0
51.5
mDE
67.2
66.8
67.2
D.4 DETAILED RESULTS ON THE ERROR ANALYSIS ON IN-D
Frequently predicted classes We analyze the most frequently predicted classes on IN-D by vanilla ResNet50 and show the results in Fig. 9. We make several interesting observations: First, we find most errors interpretable: it makes sense that a ResNet50 assigns the label "comic book" to images from the "clipart" or "painting" domains, "infograph" domain, or "envelope" to images from the "sketch" domain. Second, on the hard domain "quickdraw", the ResNet50 mostly predicts non-sensical classes that not in IN-D, mirroring its almost chance performance on this domain. Third, we find no systematic errors on the "real" domain which is expected since this domain should be similar IN.
"website" to images from the
Filtering predictions on IN-D that cannot be mapped to ImageNet-D We perform second analysis: We filter the predicted labels according to whether they can be mapped to IN-D and report the filtered top-1 errors well the percentage of filtered out inputs in Table 32. We note that for the domains "infograph" and "quickdraw", the ResNet50 predicts labels that cannot be mapped to IN-D in over 70% of all cases, highlighting the hardness of these two domains.
29

---

Under review as a conference paper at ICLR 2022
Table 32: top-1 emror IN and different IN-D domains for different settings: left column: default evaluation, middle column: predicted labels cannot mapped to IN-D are filtered out, right column: percentage filtered out labels.
Dataset
top-1 error in %
top-1 error filtered labels
percentage of rejected inputs
IN val
12.1
13.4
52.7
IN-D real
40.2
17.2
27.6
IN-D clipart
59.0
59.0
IN-D infograph
89.7
59.3
74.6
IN-D painting
65.2
39,5
42.4
IN IN-D quickdraw
99.3
96.7
76.1
IN-D sketch
82.1
47.9
Filtering labels and predictions on IN that cannot be mapped ImageNet-D To test for possible class-bias effects, we test the performance of a ResNet50 model on IN classes that can be mapped IN-D report the results Table 32.
First, we map IN labels to IN-D to make the setting as similar possible to our experiments on IN-D and report the top-1 error (12. This error significantly lower compared to the top-1 error a ResNet50 obtains following the standard evaluation protocol (23.9%). This can be explained by the simplification the task: While in IN there are 39 bird classes, these are all mapped to the same hierarchical class in IN-D. Therefore, the classes in IN-D more dissimilar from each other than in IN. Additionally, there are only 164 IN-D classes compared to the 1000 IN classes, raising the chance level prediction.
we further only accept predictions that can be mapped to IN-D, the top-1 error is slightly increased to 13.4%. In total, about 52.7% of all images in the IN validation set cannot be mapped to IN-D.
clpott
Infogreph
prstictan
10
73-
53-
25
bools
0
webeitt
esoelcpe
painting
25.
20
books
book claw
lnfertuch
seal
dlotc
103
Mandored
43-
look
Figure 9: Systematic predictions of a vamilla ResNet50 on IN-D for different domains.
30

---

Under review as a conference paper at ICLR 2022
D.5
TOP-1 ERROR ON IN-D FOR ALEXNET
We report the top-1 error numbers on different IN-D as achieved by AlexNet in Table 33. We used these numbers for normalization when calculating mDE.
Table 33: top-1 error on IN-D by AlexNet which was used for normalization.
Dataset
top-1 error in %
IN-D real
54,887
IN-D clipart
84.010
IN-D infograph
95.072
IN-D painting
79.080
IN-D quickdraw
99.745
IN-D sketch
91.189
31

---

Under review as a conference paper at ICLR 2022
E ADDITIONAL EXPERIMENTS
E. BEYOND IMAGENET CLASSES: SELF-LEARNING ON WILDS
The WILDS benchmark (Koh al., 2021) is comprised of ten tasks to domain generalization, subpopulation shift, and combinations thereof. In contrast to the setting considered here, many of the datasets in WILDS mix several 10s or 100s domains during test time.
The Camelyon17 7 dataset in WILDS contains histopathological images, with the labels being binary indicators of whether the central x2x32 region contains any tumor tissue; the domain identifies the hospital that the patch was taken from. Camelyon17 contains three different test splits with different domains and varying difficulty levels. For evaluation, we took the pretrained checkpoint wooalab.org/workssheetsbbb 4c5599955444 1 1823a7 10647166662266688 (camelyon17- erm densenet12) (eeeed) for a DenseNet121 model (Huang et al., 2017) and verified the reported baseline performance numbers. We adapt the models using ENT or RPL for maximum of 10 epochs using learning rates {3 x 10 3 x 1 10- The best hyperparameter is selected according to OOD Validation accuracy.
from
The RxRx1 dataset in WILDS contains RGB images of cells obtained by fluorescent microscopy, with the labels indicating which of the 1,139 genetic treatments (including treatment) the cells received; the domain identifies the batch in which the imaging experiment was run. The RxRx1 dataset contains three test splits, however, unlike Camelyon17, in all of the splits the domains are mixed. For evaluation, we took the pretrained checkpoint from worksheets. worksheets.codalabb.co gudalab.org/bundles (4033360055b444c55447744444422
for ResNet50 model and verified the reported baseline performance numbers. We adapt the models using ENT or RPL for maximum of 10 epochs using base learning rates {6.25 x 10-6 6.25 10-5, -0... 6.25 x 10 10-2). 2). which are scaled to the admissible batch size for single GPU adaptation using linear scaling. The best hyperparameter is selected according to OOD Validation accuracy.
Table 34: Self-learning can improve performance on WILDS if systematic shift is present - on Camelyon 17, the ood validation and test sets different hospitals, for example. On datasets like RxRx1 and FMoW, we do not improvement, most likely because the ood domains shuffled, and limited amount of images exist for each test domain.
Top-1 accuracy [%]
Validation (ID)
Validation (OOD)
Test (OOD)
Camelyon17
Baseline
81.4
88.7
63.1
BN adapt
97.8 (+16.4)
90.9 (+2.2)
88.0 (+24.99
ENT
97.6 (+16.2)
92.7 (+4.0)
91.6 (428.5)
RPL
97.6 (+16.2)
93.0 (+4.3)
91.0 (+27.9)
RxRx1
Baseline
35.9
19.1 19.1 (0.0)
29.7
BN adapt
(-0.9)
29.4 (-0.3)
ENT
34.8 (-1.1)
19.2 (+0.1)
29.4 (-0.3)
RPL
34.8 (-1.1)
19.2 (+0.1)
29.4 (-0.3)
FMoW
Baseline
60.5
59.2
52.9
BN adapt
59.9 (-0.6)
(-1.6)
51.8 (-1.1)
ENT
59.9 (-0.6)
58.5 (-0.7)
52.2 (-0.7)
RPL
59.8 (-0.7)
58.6 (-0.6)
52.1 (-0.8)
The FMoW dataset in WILDS contains RGB satellite images, with the labels being one of 62 building or land use categories; the domain specifies the year in which the image was taken and its geographical region (Africa, the Americas, Oceania, Asia, or Europe). The FMow dataset contains four test splits for different time periods, for which all regions are mixed together. For evaluation, we took the pretrained checkpoint from //worksheets wwwrksheets.codaaab.oooo codalab.org! (wfee6e9eeaad555 for DenseNet121 model and verified the reported baseline performance numbers, We adapt the models using ENT or RPL for a maximum 10 epochs
32

---

Under review as a conference paper at ICLR 2022
using learning rates {5.0 x 10"9, 5.0 X x 10 10"5.......0 10020-00 2). The best hyperparameter is selected according to OOD Validation accuracy.
While improvements Camelyon17, neither BN adaptation nor self-learning improve performance on RxRx1 or FMoW. Initial experiments on Mav and iWildsCam also do not show improvements with self-learming. We hypothesize that the reason lies in the mixing of the domains: Both BN adaptation and our self-learning methods work best on systematic domain shifts. These results support our claim that self-learning effective, while showing the important limitation when applied to more diverse shifts.
E.2 SMALL IMPROVEMENTS ON BIGTRANSFER MODELS WITH GROUP NORMALIZATION LAYERS
We evaluated Big Transfer models (Kolesnikov al., 2020) provided by the timm library (Wightman, 2019). A difference to the ResNet50, ResNeXt101 and EfficientNet models is the use of group normalization layers, which might influence the optimal method for adaptation for this evaluation, we followed our typical protocol as performed on ResNet50 models, and used affine adaptation.
For affine adaptation, distilled Big Transfrr ResNet50 model improves from 49.6 % to 48.4 % mCE on the ImageNet-C development set, and from 55.0 % to 54.4 % mCE on the ImageNet-C test set when using RPL (q 0.8) for adaptation, learning rate 7.5 10-4 at batch size 96 after single adaptation epoch. Entropy minimization did not further improve results on the ImageNet-C test set. An ablation over learning rates and epochs the dev set shown in Table 35, the final results summarized in Table 36.
Table 35: mCE in on the IN-C dev set for and for different numbers training epochs when adapting the affine batch norm parameters of ResNet50 model.
Table 36: mCE in on the IN- dev set for ENT and RPL for different numbers of training epochs when adapting the affine batch norm parameters of ResNet50 model.
criterion
ENT
RPL 10-4
Ir. 7.5
10
10
10-3
10
10
epoch
dev mCE
test mCE
49.63
49.63
49.63
49.63
49.63
49.63
Baseline
49.63
$5.03
49.44
50.42
52.59
49.54
48.89
48.95
ENT
48.80
56.36
49.26
50.27
56.47
49.47
48.35
50.77
RPL
48.35
54.41
49.08
52.18
60.06
49.39
48.93
51.45
48.91
52.03
60.50
49.31
50.01
51.53
48.80
51.97
62.91
49.24
49.96
51.34
48.83
52.10
62.96
49.16
49.71
51.19
48.83
52.10
62.96
49.16
49.71
51.19
E.3 CAN SEIF-LEARRIIGG IMPROVE OVER SELF-LEARNING BASED UDA?
An interesting question is whether test-time adaptation with self-learning can improve upon self- learning based UDA methods. To investigate this question, we build upon French et al. (2018) and waaruub.oo//Brttffrugee
their released code base the Baseline models from scratch using the provided shell scripts with the default hyperparameters and verified the reported performance. For adaptation, tested BN adaptation, ENT, RPL, well as continuing to train in exactly the setup of French et al. (2018), but without the supervised loss. For the different losses, we adapt the models for maximum of 10 epochs using learning rates {1 X 10 5,1 10 ....4...,11.............. 10 10-1].
sisemmblevisualldoneee adapt. We trained
Note that for this experiment, in contrast any other result in this paper, we purposefully do not perform proper hyperparameter selection based on a validation Mataset-instest we report the best accuracy across all tested epochs and learning rates to give an upper bound on the achievable performance for test-time adaptation.
As highlighted in Table 37, none of the four tested variants is able meaningfully improve over the baseline, corroborating our initial hypothesis that self-learning within a full UDA setting is the optimal strategy, if dataset size and compute permits. On the other hand, results like the teacher
33

---

Under review as a conference paper at ICLR 2022
refinement step in DIRT-T (Shu et al., 2018) show that with additional modifications in the loss function, might be possible to improve over standard UDA with additional adaptation at test time.
Table 37: Test-time adaptation marginally improves over self-ensembling.
Baseline
adapt
ENT
RPL
Self-ensembling loss
MNIST-SVHN
MT+TF
33,88
34,44
34.87
35.09
33,27
MT+CT
32.62
34.11
34.25
34.21
33.36
MT+CT+TF
41.59
41.93
41.95
41.95
42.70
MT+CT+TFA
30.55
32.53
32.54
32.55
30.84
SVHN-specific aug.
97.05
96.82
96.91
96.87
97.12
MNIST
MT+TF
98.01
97.91
97.96
97.91
98.16
MT+CT
88.34
88.39
88.54
88.39
88.44
MT+CT+TF
98.36
98.41
98.41
98.41
98.50
MT+CT+TFA
98.45
98.45
98.45
98.45
98.61
SVHN-MNIST
MT+TF
98.49
98.47
98.49
98.47
99.40
MT+CT*
88.34
88.36
88.36
88.36
89.36
MT+CT+TF
99.51
99.49
99.5
99.49
99.57
MT+CT+TFA
99.56
99.57
99.57
99.57
99.58
SVHN specific aug.
99.52
99,49
99.5
99.49
99.65
UNIST
MT+IF
92.79
92.62
92.62
92.66
93.08
MT+CT*
99.11
99.13
99.14
99.13
99.21
MT+CT+TE
99.41
99.42
99.45
99.42
99.52
MT+CT+TFA
99.48
99.54
99.57
99.54
99.54
34

---

Under review as a conference paper at ICLR 2022
F DETAILED DISCUSSION OF RELATED WORK
Self-learning for domain adaptation Xie et al. (2020b) introduce "In-N-Out" which uses auxiliary information boost both in- and out-of-distribution performance. AdaMatch (Berthelot et al., 2021) builds upon FixMatch (Sohn et al., 2020) and can be used for the tasks of unsupervised domain adaptation, semi-supervised learning and semi-supervised domain adaptation as a general- purpose algorithm. Prabhu et al. (2021) propose SENTRY, an algorithm based on judging the predictive consistency of samples from the target domain under different image transformations. Zou et al. (2019) show that different types of confidence regularization can improve the performance of self-learning. theoretically motivated framework for self-learning in domain adaptation based on consistency regularization has been proposed by Wei et al. (2020) and then extended by Cai et al. (2021). Self-learning has also been used for semantic segmentation (Zou et al., 2018).
The main difference from these works to ours is that they 1) utilize both source and target data during training (i.e., the classical UDA setup) whereas we only require access to unlabeled target data (source-free setup), and 2) train their models from scratch whereas we adapt pretrained checkpoints to the unlabeled target data, 3) are oftentimes more complicated (also in terms of the number of hyperparameters) than our approach due to using more than one term in the objective function. We would like to highlight that utilizing source data should always result in better performance compared not using source data. Our contribution is show that self-learning can still be very beneficial with a small compute budget and no access to source data. Our setup targets "deployed systems", e.g., a self-driving car or a detection algorithm in production line which adapts to the distribution shift "on-the-fly -ly" and cannot (or should not) be retrained from scratch for every new domain shift.
Kumar et al. (2020) study the setting of self-learning for gradual domain adaptation. They find that self-learning works better if the data distribution changes slowly. The gradual domain adaptation setting differs from ours; instead of a gradual shift over time, we focus on a fixed, systematic shift at test time dataset. Kumar et al. (2020) tested their method on a synthetic Gaussian dataset, MNIST and the Portraits datasets; building and evaluating ImageNet-scale datasets for a gradual domain adaptation perspective is a very interesting extension of our work, but left for future work, and would not only require changes/adaptations the self-leamning method, but also to the evaluation datasets.
Chen al. (2020b) prove that under certain conditions, self-learning can improve performance in biased datasets where spurious features correlate with the label in the source domain but are independent of the label in the target domain. While Chen et al. (2020b) also consider the setting of source-free domain adaptation (like we do), they limit their experiments to small scale models on MNIST and Celeb-A, while we conduct a large scale empirical study on all common robustness datasets with large scale models some of the observed and studied effects in our paper (effectiveness of different loss functions at different problem scales, adaptation mechanisms, etc.) can be attributed to this large scale evaluation setting, and extending our insights over small scale experiments.
Similar to us, Chen et al. (2020b) find that a strong source classifier is necessary for self-learning to work; however, in their case, a teacher accuracy of 72% (on CMNISTIO) is already too low and leads to worse student accuracy. In contrast, in our experiments, self-learning still works for an mCE as high as 80% (cf. appendix Figure 3, severity 5) and teacher accuracies as low as 10.4% (on ImageNet-D "Infograph"), ), and breaks down accuracies around 1-2% (on ImageNet-D "Quickdraw"). This discrepancy might be due to the spurious correlations that Chen et al. (2020b) introduced in their dataset leading to systematic biases, which are not present in the datasets studied.
Self-learning in semi-supervised learning (SSL) In a different line of work which is not related to domain adaptation directly, self-learning has been used in semi-supervised setting. Zoph et al. (2020) show that self-learning outperforms pretraining when stronger data augmentation used and more labeled data is present. They use human labels on the target task (c.g., object detection on COCO) and pseudo-labels on an unlabeled dataset (e.g. ImageNet), and optimize the loss on both datasets, with the aim to improve performance the task where ground truth labels are known.
The work of Zoph et al. (2020) is orthogonal to ours, in the sense that we could adapt their final
35

---

Under review as a conference paper at ICLR 2022
checkpoint to domain with method, similar to how adapted the Noisy Student model which was also trained using self-learming.
Rizve et al. (2021) propose uncertainty -warr aware pscudo-label selection (UPS) framework which outperforms other SSL methods few-label regime. UPS is helpful to reduce the impact of noisy pseudo-labels; in case, use the generalized cross-entropy loss for this purpose. Testing the UPS framework (and other means for improving the quality of pseudo-labels, or robustness against label noise) on robustness datasets would be interesting direction for future work.
De Sousa Ribeiro et al. (2020) propose Deep Bayesian Self-Training (DBST) for automatic data annotation. Mukherjee & Awadallah (2020) suggest using self-leaming in semi-supervised setting for text classification with few labels.
36

---

Under review as a conference paper at ICLR 2022
G SOFTWARE STACK
We use different open source software packages for our experiments, most notably Docker (Merkel, 2014), scipy and numpy (Virtanen et al., 2020), GNU parallel (Tange, 2011), Tensorflow (Abadi et al., 2016), PyTorch (Paszke et al., 2017), timm (Wightman, 2019), Self-ensembling for visual domain adaptation (French et al., 2018), the WILDS benchmark (Koh al., 2021), and torchvision (Marcel & Rodriguez, 2010).
37


Reviews:
[{"review_id": "RMnXMQq_BZM", "reviewer": "Reviewer_s85k", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-The message of the paper is very clear, and it is generally well written.\n\n-I believe this paper is of interest to the research community, as improving\nthe robustness of deep vision models to distributional shifts is of critical importance going forward.\n\n-The empirical evaluation is thorough, significant and convincing, demonstrating \nthe advantages of using self-training on top of existing approaches to \nimprove the robustness of deep models on a variety of standard benchmarks.\n\nWeaknesses/Suggestions:\n\n-The insights from this paper are mainly empirical, and there is little\nalgorithmic/theoretical novelty. The self-training algorithms evaluated here\nare well established, even in a deep learning setting (see below). \nThe exception to this is section 8 which comes across as an attempt to address \nthis deficiency late on, nonetheless it is useful to know what temperatures \nought to be used in student/teacher training. All this is not to say that \nempirical demonstrations are less important, but from an algorithmic perspective \nit makes the results not as surprising. At the risk of stating the obvious, \nsince we're using self-training to adapt to new domains, we would expect the \nmodels to be more robust to them. With that said, demonstrating this \nconvincingly in practice is non-trivial and demands significant engineering\neffort as evidenced in this paper.\n\n-Because there are so many results, the paper is quite dense and a bit hard \nto follow at times since the text is broken up a lot. This isn't a major \nissue but something the authors could consider improving for the final \nversion.\n\n-Self-training using deep learning as presented here has gained popularity \nrecently, and the paper is missing some related references (see below to name a few).\n\n-I think including some results on model calibration would add a lot\nof value to the exposition, especially if the authors could demonstrate that \nself-learning also calibrates predictions, thereby improving not only accuracy \nbut also uncertainty under domain shift.\n\n-Reporting standard deviations of results would strengthen the author's claims\n\n-------------------\n\nReferences:\n\n[1] Zou, Yang, et al. \"Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n\n[2] Rizve, Mamshad Nayeem, et al. \"In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning.\" International Conference on Learning Representations. 2020.\n\n[3] De Sousa Ribeiro, Fabio, et al. \"Deep bayesian self-training.\" Neural Computing and Applications 32.9 (2020): 4275-4291.\n\n[4] Zou, Yang, et al. \"Confidence regularized self-training.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n\n[5] Zoph, Barret, et al. \"Rethinking Pre-training and Self-training.\" Advances in Neural Information Processing Systems 33 (2020).\n\n[6] Mukherjee, Subhabrata, and Ahmed Awadallah. \"Uncertainty-aware self-training for few-shot text classification.\" Advances in Neural Information Processing Systems 33 (2020).\n\n[7] Wei, Colin, et al. \"Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data.\" International Conference on Learning Representations. 2020.", "summary_of_the_paper": "This paper provides an in depth empirical evaluation of classical self-training techniques such as pseudo-labelling and entropy minimization on test performance under domain shifts. The authors stress that, although simple, these techniques consistently improve the robustness to distribution shifts regardless of model architecture or pre-training techniques used. This makes them especially useful to practitioners applying machine learning algorithms to real problems where distribution shifts are prevalent. The authors claim state-of-the-art adaptation results on a number of popular dataset corruption benchmarks, and present a new challenging dataset for evaluating the robustness of deep vision models.\n\n\n", "summary_of_the_review": "Overall I like the paper. I think it would bring value to the research community and it serves as a reminder that the simplest methods often work very well in practice.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8iANtp4NwiV", "reviewer": "Reviewer_b5KJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The paper is very well written and organized and easy to follow. They also explain the target and overarching goal very clearly. \n- The empirical study is relevant and beneficial for the research community.\n- The proposed suggestion is easy to use/implement, and there are interesting results supporting the main claim.\n\n\nMain concerns and comments to improve the paper:\n- Real world evaluation beyond natural images is missing which helps to solidify claims of the paper: The paper focuses on improvement on curated datasets such as IN-family. There are prior works that show similar results including [Barret’20, Chen’20, and etc]  and premises of self-training have been established previously.  However the paper does not consider real-world applications such as test time performance drop in medical data or satellite images and only limited to curated datasets which limits the future impact of the work.\n- Definition of mCE (mean Corruption Error) and the calculation procedure is required. \n- The method is of an incremental nature and not novel.\n\nSuggestion to improve the paper:\n- Consider adding/studying other domain dataset and tasks. There are multiple open source datasets available.   You can check the WILDS benchmark. \n- Adding model calibration and statistical analysis of the results can boost the validity of the result section. \n- Consider using Big Transfer (BiT) model performance as one family of Models for ImageNet-scale datatsets. \n", "summary_of_the_paper": "The paper studies effectiveness of self-training to improve test time performance when the distribution of test data is not similar to the training data. The paper more specifically focuses on source-free domain adaptation settings where the source target data is not available. In this setup self-training has been tested as an additional step on top of  different robustness and adaptation approaches such as robust pretraining, unsupervised domain adaptation and self-supervised pretraining. The paper shows improvement on multiple ImageNet variants and CIFAR10-C, and also introduces ImageNet-D dataset as a new benchmark. ImageNet-D has been produced by matching label space of IN datasets with DomainNet data provided in Visual domain adaptation challenges. The main contribution of this paper is to perform a systematic and large study of self-training as a method to deal with distribution shifts.", "summary_of_the_review": "The paper ran a large scale study to establish the benefits of self-training for data distribution shift. The technical contributions of the paper are only marginal and more of incremental nature, however the study itself is valuable and can be beneficial for the community. This study can significantly get boosted by diversifying the range of datasets and tasks under study. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "19vgZuQ2h05", "reviewer": "Reviewer_PUq6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper shows robust results where adding self-learning on top of existing methods (by starting from the checkpoint of the previous methods) improves performance, even when the existing method has already seen the unlabeled target data (UDA methods). One interesting test to add here would be where self-learning helps on top of using self-learning for UDA (like applying Noisy student on source and target unlabeled data). It seems to be complementary to doing self-learning (Noisy student) on the ImageNet tasks.\n- There are interesting results on how self-learning improves large pre-trained models beyond BN adaptation. However, a comparison to other methods is missing here.\n- As a pretty generic method, self-learning also seems to improve over test-time adaptation approaches like Test-time Training which had a more similar setting in mind. This shows that the gains are not purely from the setup of test-time adaptation.\n- The paper is pretty comprehensive in methods and architectures, also testing on recent self-supervised methods (DINO). \n- There are some interesting experiments on the proposed variant of DomainNet they call ImageNet-D, which tests transfer from ImageNet to overlapping classes in DomainNet. DomainNet allows for domain-level insights, breaking down where self-learning fails. However, I am left wondering whether ImageNet to ImageNet-D is different from the Real -> any other domain part of DomainNet. Perhaps the main gains here are that the scale of the Real data is much larger and that we can test models that are trained on ImageNet rather than a different special dataset. \n- The paper gives an interesting preliminary analysis of the self-learning dynamics in a simple two-point setting, with some results for setting student and teacher temperatures in self-learning that seem to bear out in practice as well. The result seems general beyond test-time adaptation.\n\nWeaknesses\n- There have been quite a few works that focus on how self-training / self-learning improves distribution shift and how self-training and pre-training stack together: https://arxiv.org/abs/2012.04550, http://proceedings.mlr.press/v139/cai21b.html (in the experimental section), https://arxiv.org/abs/2002.11361, https://arxiv.org/abs/2106.04732, https://arxiv.org/abs/2006.10032, https://arxiv.org/abs/2012.11460, https://arxiv.org/abs/2006.06882 (this one is just about stacking them, not distribution shift). In some sense, the main premise of the paper (for example the title) is somewhat well-established. Discussion about these and your distinction from them would improve the positioning of the paper.\n- The setting is a little confusing at first - it would seem that from an \"access\" point of view, we should always have access to the source dataset, while we may not be able to know the target dataset. I think this comes from the way that it's explained in the context of UDA. From what I understand, the main thing with the \"source-free\" setting is that the source dataset is too large so we need to do pre-training. I think its a bit more straightforward to start with the OOD generalization setting (what you call ad-hoc) without any knowledge of the target, then add the knowledge of the target data. Also, the \"test-time adaptation\" term has strong connotations of the online setting, which I believe is not being considered here, but should be distinguished clearly.\n- Fig 1 clarity: Overall, the figure doesn't say very much about the source-free setting (it just looks like an option, but pictorially it doesn't look different from the other options). Some picture-level aspects were unclear, like what the different between the icons in the gray (pre-training) box are, and what the image + 3 gray shapes in the top represent, as well as what the orange stripe in the 3 gray shapes in the adaptation phase denote.  \n- Table 1,2 seem to use different self-learning algorithms. How were the methods for each table chosen? In Section 6, the paper supports Robust pseudo labeling, but Table 2 uses ENT for the main results.\n- In Table 4, why is one comparison against TENT while the other against TTT? \n- The takeaway in Table 8 that updating all affine layers is important is perhaps misleading, since in Table 5, tuning all affine layers is the worst option. It is also not fully specified what tuning all affine layers means?\n- It may be easier in terms of presentation to talk about ImageNet-D right before the results in section 7.\n- The results on ImageNet-D so far do not seem unique to test-time adaptation with self-learning; they would seem to occur in other settings too. Are there any aspects of ImageNet-D that test aspects of test-time adaptation methods?\n- As the paper suggests ImageNet-D as a robustness benchmark, it's probably worth discussing whether we would expect our models to generalize to such disparate domains - for some of the domains, the paper reports close to 0% accuracy. ", "summary_of_the_paper": "The paper considers using self-learning / self-training as a test-time adaptation step for adapting to distribution shifts, showing that it is complementary with pre-training methods, domain adaptation methods used for training the model. The test-time adaptation setting (source-free domain adaptation setting) is taken to be the setting where models can use target data to adapt but cannot use the source data (which was used for pre-training). ", "summary_of_the_review": "The paper tests self-learning as a complementary addition to improve robustness. However, the premise that self-learning improves robustness is already somewhat well-established - the main contribution here is a systematic application to different methods and datasets. The restriction to the pre-training + test-time adaptation setting has also been considered to some extent, but not as systematically. The value of the proposed dataset ImageNet-D is unclear, whether it gives insights beyond DomainNet itself, and whether it is a worthwhile goal to generalize to such disparate domains. Finally, the analysis of self-learning dynamics seems interesting and predicts some empirical behaviors nicely. I think the paper could have a good message solidifying self-learning methods for robustness, but could use some tightening up in the story/clarity of the paper, and some inconsistencies in the experimental reporting. I'd be happy to raise my score if the issues are addressed in the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main contribution of the paper is to perform a systematic and large study of self-training as a method to deal with distribution shifts. Reviewers have appreciated the clarity in the overall writing of the paper, and rigor in the empirical analysis. However the main concern from two of the reviewers is that the technical contributions of the paper are only marginal and incremental in nature. The premise that self-learning improves robustness is already somewhat well-established (Reviewer PUq6 has pointed out papers that focus on how self-training / self-learning improves distribution shift and how self-training and pre-training stack together), and the main contribution of the paper is a systematic application to different datasets. Given the existing work on the relevance of self-training in distribution shift, the paper falls below the acceptance bar for ICLR in my opinion.
Decision: Reject