Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose to improve generalization and robustness under distribution shift by cloning the network and training the clone for on multiple masked copies of each incoming example at evaluation time using an auxiliary masked reconstruction loss.
strengths_and_weaknesses: Strengths:

* The paper is easy to follow and well written.
* The experimental protocol is built around multiple large scale datasets.
* The method is novel, simple, and effective.

Weaknesses:
* An explicit mathematical formulation of the training and test-time training losses would make the paper easier to read.
* Compares reconstructions solely to a rotation prediction self-supervised loss.
questions: The paper would greatly benefit from including more self-supervised losses as baselines.
limitations: The authors have clearly outlined the additional computational cost incurred by the use of their approach.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Summary:
The paper performs training using test set images using the idea of masked autoencoder to further improve the performance. Experiments show that this approach improves the performance on out-of-distribution images. They also present a theoretical analysis suggesting the reasons for the observed improvements under distribution shifts. 
strengths_and_weaknesses: Strengths and Weakness:

- The paper leverages a relatively new idea of self-supervised pre-training to obtain good results on out of distribution image classification tasks.

- I feel that the paper falls short in the contributions part. Additionally, the novelty seems to be quite limited in the paper. The idea of masked autoencoders is quite known and the paper just applies it to the pre-existing task of test-time training.  As such, there are very limited technical contributions that the paper makes. In my opinion, the paper is looking more suitable for a workshop.

- I felt the theoretical analysis part may not be adding much value and can be moved to the appendix. That would help in saving space, which could be used to write up more interesting analysis or new experiments. Some suggestions to make the contributions stronger include doing ablations with different types of MAE architectures showcasing the pros and cons of each, impact of masking ratio in MAE on the test-time training training.

- Instead of always using the same pre-trained MAE checkpoint, the authors can train or use different MAE models trained on varied datasets apart from ImageNet and then perform experiments as to which kind of datasets are more robust to test-time training and offer more improvements.

- The code is not submitted with the paper, raising reproducibility concerns.

questions: Please see the strengths and weakness section

** Update after the authors response **

Updating my scores to lean towards acceptance.
limitations: Yes, the limitations have been included in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to use test time training on the test images to improve generalization under distribution shifts. Using the establishe MAE method, the method add an additional classifier branch on the encoder for classification. To train the model, the paper studies three ways: 1) finetuning classifier and encoder, 2) ViT probing 3) joint training MAE and classifier, where ViT probing works the best. The test time training only trains the MAE part. Results are shown on ImageNet-C, R, and A.


strengths_and_weaknesses: Strengths:
1. Using MAE as test-time training improves generalization on shifted distributions is effective from the presented results. Results from 4 datasets have been shown, with significant improvement.

2. The paper is well presented.

3. The reviewer appreciates the theoretical explanation, which is interesting.

Weakness:
1. On ImageNet-C's level 1-4, baseline results are missing. What is the standard mCE number on ImageNet-C following the standard metric? Would the approach's number better than other non-test-time training ViT method? Such as the Discrete ViT [1], which uses VQVAE for encoding, similar to the MAE task for reconstruction, but does not require test time training.

2. Should add more intuition for theoretical justification part, such that without reading the equation this part is also clear.

[1] Discrete Representations Strengthen Vision Transformer Robustness. ICLR 2022. https://arxiv.org/abs/2111.10493
questions: 1. Does the proposed work on single test images? Or it has to be trained on many test images from the same domain, with batchsize>1.

2. The theoretical part is not linked to the accuracy, why the accuracy gets better? Why the proposed MAE outperforms past method, would simple PCA reconstruction also improve accuracy based on the theory? 
limitations: None.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a test-time training algorithm to address the degradation of classification performance typically seen in deep learning models. The paper combines two ideas— self-supervised learning via masked autoencoders, and test-time training (TTT) using a self-supervised objective. Specifically, the paper applies the recently proposed masked auto-encoder (MAE) framework for self-supervised training, towards test-time training. To do so, the authors propose to freeze the feature network of this model and learning a classifier head on top during training, and then fine-tuning the feature network to minimize the reconstruction loss on the test sample to address distribution shift. The authors show a significant improvement in performance across different datasets using their approach over the existing state-of-the-art using test-time training method.
strengths_and_weaknesses: Strengths:

- simple approach.
- The experimental results show significant improvement across different datasets in the test-time training paradigm.
- ablation analysis on test-time training optimizer AdamW vs SGD.

Weaknesses:

- theoretical justification is largely disconnect from the proposed method. I do not see how the data matrix affine transform as a proxy for distribution shift applies in the real world setting. It does not provide any insight into why TTT really helps, or how the specific strategy chosen in this work is better than the existing TTT algorithm that uses the rotation prediction task.
- The bias-variance terminology is confusing and should be removed, as it s trying to convey a different meaning from what it traditionally means.
- The writing is not clear in a couple of places, and needs improvement.

questions: Questions and suggestions:

- Line 53: “One way to turn extrapolation into interpolation again is to make the training distribution wider, and the training set larger, to cover the test distribution.”. This statement is misleading because it assumes that it is always possible for a single hypothesis function to perform well on both training and test distributions. But this may not be necessarily true (e.g. label shift where p(y|x) changes). This needs to be either clarified as the scope of this statement, or removed all together.

- Line 177: Not very clear. My understanding is that the baseline is the pre-trained MAE model which is kept fixed, and only a classifier head is trained using labels.

- Foot note #3 below Line 196: You mean much worse? Fig 3 shows TTT-Rot is worse than ViT baseline.

- Paragraph of Line 199: Not clearly stated what is the difference between Probing and TTT-MAE in table 2. If the goal is to talk about training-time training, and the 1st 3 rows are without TTT, then it should be explicitly mentioned, because mentioning TTT-MAE results alongside is confusing.
limitations: Limitations:

- As mentioned above, the theoretical justification is too simplistic and disconnected from the proposed method and the real world setting. I think it is better to remove it from the paper.

- Some ablation studies would be great that provide some insight into why the proposed framework works better compared to the existing rotation prediction based TTT framework. Some intuition is discussed by authors along the lines that rotation prediction is not a well defined task for certain class of images (e.g. top or bottom view images where rotation has no semantic meaning). But a clearer ablation in terms of the differences in the features learned by the two models would be more helpful.
review_title: null

Meta-review: metareview: This paper performs test-time (unsupervised) adaptation to improve generalization performance (e.g., under distribution shift). The reviewer's concerns were mostly about clarification, both in experimentation as well as overall contribution (e.g., concerns about novelty). The discussion was concise and easy to follow, and it seems that the authors addressed most of the outstanding concerns of the reviewers to the point that there's a clear consensus.

I therefore recommend acceptance of the paper.

As far as reviews, overall the discussion was light so there isn't a great deal of signal. ZRRX had the most substantial review in terms of initial content, but did not participate in further discussion.
confidence: Certain
Decision: Accept