Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies effectiveness of self-training to improve test time performance when the distribution of test data is not similar to the training data. The paper more specifically focuses on source-free domain adaptation settings where the source target data is not available. In this setup self-training has been tested as an additional step on top of  different robustness and adaptation approaches such as robust pretraining, unsupervised domain adaptation and self-supervised pretraining. The paper shows improvement on multiple ImageNet variants and CIFAR10-C, and also introduces ImageNet-D dataset as a new benchmark. ImageNet-D has been produced by matching label space of IN datasets with DomainNet data provided in Visual domain adaptation challenges. The main contribution of this paper is to perform a systematic and large study of self-training as a method to deal with distribution shifts.
main_review: Strength:
- The paper is very well written and organized and easy to follow. They also explain the target and overarching goal very clearly. 
- The empirical study is relevant and beneficial for the research community.
- The proposed suggestion is easy to use/implement, and there are interesting results supporting the main claim.


Main concerns and comments to improve the paper:
- Real world evaluation beyond natural images is missing which helps to solidify claims of the paper: The paper focuses on improvement on curated datasets such as IN-family. There are prior works that show similar results including [Barret’20, Chen’20, and etc]  and premises of self-training have been established previously.  However the paper does not consider real-world applications such as test time performance drop in medical data or satellite images and only limited to curated datasets which limits the future impact of the work.
- Definition of mCE (mean Corruption Error) and the calculation procedure is required. 
- The method is of an incremental nature and not novel.

Suggestion to improve the paper:
- Consider adding/studying other domain dataset and tasks. There are multiple open source datasets available.   You can check the WILDS benchmark. 
- Adding model calibration and statistical analysis of the results can boost the validity of the result section. 
- Consider using Big Transfer (BiT) model performance as one family of Models for ImageNet-scale datatsets. 

summary_of_the_review: The paper ran a large scale study to establish the benefits of self-training for data distribution shift. The technical contributions of the paper are only marginal and more of incremental nature, however the study itself is valuable and can be beneficial for the community. This study can significantly get boosted by diversifying the range of datasets and tasks under study.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers using self-learning / self-training as a test-time adaptation step for adapting to distribution shifts, showing that it is complementary with pre-training methods, domain adaptation methods used for training the model. The test-time adaptation setting (source-free domain adaptation setting) is taken to be the setting where models can use target data to adapt but cannot use the source data (which was used for pre-training). 
main_review: Strengths
- The paper shows robust results where adding self-learning on top of existing methods (by starting from the checkpoint of the previous methods) improves performance, even when the existing method has already seen the unlabeled target data (UDA methods). One interesting test to add here would be where self-learning helps on top of using self-learning for UDA (like applying Noisy student on source and target unlabeled data). It seems to be complementary to doing self-learning (Noisy student) on the ImageNet tasks.
- There are interesting results on how self-learning improves large pre-trained models beyond BN adaptation. However, a comparison to other methods is missing here.
- As a pretty generic method, self-learning also seems to improve over test-time adaptation approaches like Test-time Training which had a more similar setting in mind. This shows that the gains are not purely from the setup of test-time adaptation.
- The paper is pretty comprehensive in methods and architectures, also testing on recent self-supervised methods (DINO). 
- There are some interesting experiments on the proposed variant of DomainNet they call ImageNet-D, which tests transfer from ImageNet to overlapping classes in DomainNet. DomainNet allows for domain-level insights, breaking down where self-learning fails. However, I am left wondering whether ImageNet to ImageNet-D is different from the Real -> any other domain part of DomainNet. Perhaps the main gains here are that the scale of the Real data is much larger and that we can test models that are trained on ImageNet rather than a different special dataset. 
- The paper gives an interesting preliminary analysis of the self-learning dynamics in a simple two-point setting, with some results for setting student and teacher temperatures in self-learning that seem to bear out in practice as well. The result seems general beyond test-time adaptation.

Weaknesses
- There have been quite a few works that focus on how self-training / self-learning improves distribution shift and how self-training and pre-training stack together: https://arxiv.org/abs/2012.04550, http://proceedings.mlr.press/v139/cai21b.html (in the experimental section), https://arxiv.org/abs/2002.11361, https://arxiv.org/abs/2106.04732, https://arxiv.org/abs/2006.10032, https://arxiv.org/abs/2012.11460, https://arxiv.org/abs/2006.06882 (this one is just about stacking them, not distribution shift). In some sense, the main premise of the paper (for example the title) is somewhat well-established. Discussion about these and your distinction from them would improve the positioning of the paper.
- The setting is a little confusing at first - it would seem that from an "access" point of view, we should always have access to the source dataset, while we may not be able to know the target dataset. I think this comes from the way that it's explained in the context of UDA. From what I understand, the main thing with the "source-free" setting is that the source dataset is too large so we need to do pre-training. I think its a bit more straightforward to start with the OOD generalization setting (what you call ad-hoc) without any knowledge of the target, then add the knowledge of the target data. Also, the "test-time adaptation" term has strong connotations of the online setting, which I believe is not being considered here, but should be distinguished clearly.
- Fig 1 clarity: Overall, the figure doesn't say very much about the source-free setting (it just looks like an option, but pictorially it doesn't look different from the other options). Some picture-level aspects were unclear, like what the different between the icons in the gray (pre-training) box are, and what the image + 3 gray shapes in the top represent, as well as what the orange stripe in the 3 gray shapes in the adaptation phase denote.  
- Table 1,2 seem to use different self-learning algorithms. How were the methods for each table chosen? In Section 6, the paper supports Robust pseudo labeling, but Table 2 uses ENT for the main results.
- In Table 4, why is one comparison against TENT while the other against TTT? 
- The takeaway in Table 8 that updating all affine layers is important is perhaps misleading, since in Table 5, tuning all affine layers is the worst option. It is also not fully specified what tuning all affine layers means?
- It may be easier in terms of presentation to talk about ImageNet-D right before the results in section 7.
- The results on ImageNet-D so far do not seem unique to test-time adaptation with self-learning; they would seem to occur in other settings too. Are there any aspects of ImageNet-D that test aspects of test-time adaptation methods?
- As the paper suggests ImageNet-D as a robustness benchmark, it's probably worth discussing whether we would expect our models to generalize to such disparate domains - for some of the domains, the paper reports close to 0% accuracy. 
summary_of_the_review: The paper tests self-learning as a complementary addition to improve robustness. However, the premise that self-learning improves robustness is already somewhat well-established - the main contribution here is a systematic application to different methods and datasets. The restriction to the pre-training + test-time adaptation setting has also been considered to some extent, but not as systematically. The value of the proposed dataset ImageNet-D is unclear, whether it gives insights beyond DomainNet itself, and whether it is a worthwhile goal to generalize to such disparate domains. Finally, the analysis of self-learning dynamics seems interesting and predicts some empirical behaviors nicely. I think the paper could have a good message solidifying self-learning methods for robustness, but could use some tightening up in the story/clarity of the paper, and some inconsistencies in the experimental reporting. I'd be happy to raise my score if the issues are addressed in the rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides an in depth empirical evaluation of classical self-training techniques such as pseudo-labelling and entropy minimization on test performance under domain shifts. The authors stress that, although simple, these techniques consistently improve the robustness to distribution shifts regardless of model architecture or pre-training techniques used. This makes them especially useful to practitioners applying machine learning algorithms to real problems where distribution shifts are prevalent. The authors claim state-of-the-art adaptation results on a number of popular dataset corruption benchmarks, and present a new challenging dataset for evaluating the robustness of deep vision models.



main_review: Strengths:

-The message of the paper is very clear, and it is generally well written.

-I believe this paper is of interest to the research community, as improving
the robustness of deep vision models to distributional shifts is of critical importance going forward.

-The empirical evaluation is thorough, significant and convincing, demonstrating 
the advantages of using self-training on top of existing approaches to 
improve the robustness of deep models on a variety of standard benchmarks.

Weaknesses/Suggestions:

-The insights from this paper are mainly empirical, and there is little
algorithmic/theoretical novelty. The self-training algorithms evaluated here
are well established, even in a deep learning setting (see below). 
The exception to this is section 8 which comes across as an attempt to address 
this deficiency late on, nonetheless it is useful to know what temperatures 
ought to be used in student/teacher training. All this is not to say that 
empirical demonstrations are less important, but from an algorithmic perspective 
it makes the results not as surprising. At the risk of stating the obvious, 
since we're using self-training to adapt to new domains, we would expect the 
models to be more robust to them. With that said, demonstrating this 
convincingly in practice is non-trivial and demands significant engineering
effort as evidenced in this paper.

-Because there are so many results, the paper is quite dense and a bit hard 
to follow at times since the text is broken up a lot. This isn't a major 
issue but something the authors could consider improving for the final 
version.

-Self-training using deep learning as presented here has gained popularity 
recently, and the paper is missing some related references (see below to name a few).

-I think including some results on model calibration would add a lot
of value to the exposition, especially if the authors could demonstrate that 
self-learning also calibrates predictions, thereby improving not only accuracy 
but also uncertainty under domain shift.

-Reporting standard deviations of results would strengthen the author's claims

-------------------

References:

[1] Zou, Yang, et al. "Unsupervised domain adaptation for semantic segmentation via class-balanced self-training." Proceedings of the European conference on computer vision (ECCV). 2018.

[2] Rizve, Mamshad Nayeem, et al. "In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning." International Conference on Learning Representations. 2020.

[3] De Sousa Ribeiro, Fabio, et al. "Deep bayesian self-training." Neural Computing and Applications 32.9 (2020): 4275-4291.

[4] Zou, Yang, et al. "Confidence regularized self-training." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

[5] Zoph, Barret, et al. "Rethinking Pre-training and Self-training." Advances in Neural Information Processing Systems 33 (2020).

[6] Mukherjee, Subhabrata, and Ahmed Awadallah. "Uncertainty-aware self-training for few-shot text classification." Advances in Neural Information Processing Systems 33 (2020).

[7] Wei, Colin, et al. "Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data." International Conference on Learning Representations. 2020.
summary_of_the_review: Overall I like the paper. I think it would bring value to the research community and it serves as a reminder that the simplest methods often work very well in practice.
review_title: null

Meta-review: The main contribution of the paper is to perform a systematic and large study of self-training as a method to deal with distribution shifts. Reviewers have appreciated the clarity in the overall writing of the paper, and rigor in the empirical analysis. However the main concern from two of the reviewers is that the technical contributions of the paper are only marginal and incremental in nature. The premise that self-learning improves robustness is already somewhat well-established (Reviewer PUq6 has pointed out papers that focus on how self-training / self-learning improves distribution shift and how self-training and pre-training stack together), and the main contribution of the paper is a systematic application to different datasets. Given the existing work on the relevance of self-training in distribution shift, the paper falls below the acceptance bar for ICLR in my opinion.
Decision: Reject