Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors propose a test time training method using contrastive loss based SSL (Self Supervised Learning) task and feature alignment to address the domain shift between training and test data. Authors give lower bound for the performance during test under some assumptions. 
main_review: 
1) Authors clearly describe the background of the work and their proposed method. Proposed method is a novel combination of well known techniques. The proposed method improves performance by 2.5% on VisDA-C dataset and around 23% for the CIFAR 10 dataset compared to the baseline. 

2) Can authors compare the results on the Imagenet-C dataset as previous papers in the literature have done [1]? If that can not be done, can authors justify why?

3) It is not clear from paper how hyperparameters are chosen for Tent and SHOT baselines? It is possible that the baseline method's hyperparameters won't work for this data if the data processing is different. In that case, was there any grid search performed on baseline methods' hyperparameters?

4) It would be good to have a graph of classification error as test time training processes to understand if performance is increasing gradually as training progresses similar to figure 2 from [1].

5) Assumption that test feature distributions are aligned seems a bit strong. Based on what type of corruption applied on test data, the distribution can change. Can authors provide empirical data to back up this claim?

6) It would be useful to show how the method performs on different domain shifts in test data. Does the method perform comparably l with Elastic Transform domain shift as it does with Fog?

7) In equation 3, authors match the first and second moment of the data. Would it be possible to match all the moments of the distribution instead of only 2 moments like in [2]. 

8) Fundamentally the approach seems very similar to TTT where only the pretext task has changed from rotation to a new self supervised task. Is there any other fundamental change apart from first and second moment matching of the data? 


[1] Sun, Yu, et al. "Test-time training with self-supervision for generalization under distribution shifts." International Conference on Machine Learning. PMLR, 2020.

[2] Blanchard, Gilles, et al. "Domain Generalization by Marginal Transfer Learning." J. Mach. Learn. Res. 22 (2021): 2-1.
limitations_and_societal_impact: For limitations please check the main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focus on improving the generalization of deep neural networks in the presence of distributional shift at test time i.e., unsupervised source-free network adaptation on data distribution that is different from training. They follow the footsteps of a prior work by Sun et al. [8] and leverage an auxiliary task at test time. However, this work points out that test time training with auxiliary tasks like rotation prediction need not necessarily encourage the network to improve its generalization on the main task. Thus, the authors propose to transfer the benefits of self-supervised learning (SSL) to the main task by introducing an online feature alignment strategy that utilizes offline feature summarization statistics collected on the original training (source) data and also choosing more suitable SSL task like contrastive learning instead of rotation prediction. Tthey denote the resulting approach as TTT++. Results and comparison to existing techniques are shown on popular datasets like cifar10-c, cifar100-c, cifar10.1 and VisDA-C.
main_review: ## Originality ##
The online feature alignment strategy is inspired by distribution matching measures from the domain adaptation literature, however proposed firstly in this work for test time adaptation. Additionally, the authors employ the popular contrastive task as the SSL task. It is clearly distinguished from the previous works and sufficiently cited. In general, this work introduces concepts  from other fields into test-time adaptation, that are not novel in general. Originality is thus relatively small.

## Quality ##
Technical details of this work are clear. They provide theoretical analysis about the relationship between SSL and main task. The method is tested on popular benchmarks. This is a complete piece of work. Authors are honest about strengths and weaknesses. 

The authors motivate the online feature alignment strategy by stating that simply applying SSL based adaptation results in poor accuracy. However, results on cifar100-c and cifar10.1 show that such strategy included in TTT++ is almost comparable to network adaptation with contrastive based SSL task and slightly better for cifar10-c and little prominent on VisDA-C. The improvements from such alignment strategy on cifar10-c and VisDA-C, but not on cifar100-c raises a question whether this technique scales to dataset with larger number of classes (e.g. ImageNet). Overall, results suggest that the impact of the online feature alignment strategy is not as pronounced as strong as authors claim.

## Clarity ##
The paper is nicely written, well organized and easy to read. 

Comments: 
* Section 3.1 provides an illustrative example of a failure case. Though this section is easy to understand, it is not sufficient to reason about a failure case. A synthetic problem is provided in appendix. I suggest authors may consider moving parts of appendix to main paper to provide sufficient rationale on the argument.
* Figure 4 illustrates the encoder feature distribution. Despite authors state that TTT+ displays improved per-class separation, it is not clear to infer this conclusion from the figure. It appears that SHOT and TTT++ show similar per-class feature distribution and feature alignment. I suggest the authors to consider a different illustration or representation to emphasize their claim from Figure 4, ideally provide quantitative numbers rather than only a visualization.

## Significance ##
The results indicate that the proposed approach outperforms previous works on cifar10-c, cifar100-c and cifar10.1. However, the major improvements come from replacing the rotation prediction task from Sun et al. [8] with the contrastive learning and this choice of SSL task seems straightforward given its recent popularity.  The improvements from the online feature alignment are relatively small and removing it would not result in "Test-Time Training Failing"; this theme from Section 3 is actually not observed anywhere in the results.

Moreover, the numbers reported from SHOT in Table 2 are not consistent with the results from the original paper Liang et al. [9]. The per-class error rate is reported as 17.1% on VisDA-C in Liang et al. [9], whereas it is reported as 25.04% in the Table 2 comparisons. Taking the error rate reported in original paper into account, TTT++ does not improve upon SHOT[9].

It is not clear how the hyperparameters are chosen for the test time adaptation. Are they chosen based on the performance on the test corruptions of the respective benchmark or the validation corruptions? Do the hyperparameters vary across the corruptions?

I appreciate the results obtained with contrastive learning as an auxiliary task that outperforms the previous works. However, based on my above comments, I see this as a simple extension of Sun et al. [8] with small novelty. I consider this submission as marginally below acceptance threshold and would be willing to reconsider my score based on the author responses and other reviews.

## Update after author response ##
The authors have addressed my comments and questions and I am raising my score to 6, that is: I see the paper slightly above the acceptance threshold. 
The main limitations that prevent me from going to a score of 7 or higher is that (a) the proposed feature alignment strategy in its current form is limited to problems with few classes and (b) most (but not all) performance gains compared to prior work TTT come from using a stronger and well established self-supervised contrastive learning objective.

limitations_and_societal_impact: I appreciate the authors for addressing the limitations and societal impact sufficiently.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper is about test-time adaptation (TTA), in particular about improving the recently published test-time training (TTT) approach. It provides an analysis about conditions that can lead to TTT failures, and overcome them with a method that relies on i) feature alignment and ii) more sophisticated self-supervised learning (SSL) objectives (when compared with the TTT's seminal work). The method is compared against other recent TTA approaches, showing good performance on standardized benchmarks.
main_review: This is a well written paper, concerning an important, recently introduced topic. The proposed method is properly located in the literature, and the ideas are well argumented and presented. Furthermore, the proposed method is very reasonable.

TTA has the potential to revolutionize the deployment of machine learning systems in the real world, but only a bunch of algorithms exists as of today. Improving one of the leading algorithms in this niche topic (TTT, by Sun et al. [8]), this paper brings valuable contributions to the table; indeed, while effective, TTT relies on an SSL approach that is currently far from the state of the art: in this work, the importance of using more effective SSL objectives is shown, and some more methodological contributions are explored.

To improve the paper, I believe more ablation studies will allow to properly assess the efficacy of the proposed strategy. In particular:

1) One of the powerful facet of TTA is the possibility to adapt on the fly to single test samples, or at least to a few samples. In this regard, I could not understand in which of the following two ways test-time adaptation is performed: A) adapt on all test samples via SGD (with the indicated batch size) and at the end evaluate performance on all test samples, or B) adapt on a batch (with the indicated batch size) and evaluate the model on that batch (if this is the case, the adaptation process starts from scratch at every step, of from the point reached by adapting to the previous batch?). I believe that the setting B) is the most interesting for TTA, because, as mentioned above, it would allow adapting on-the-fly to single samples. In my opinion, this work should address this point, by assessing how performance changes - also with respect to competing algorithms - as the batch size varies (in my understanding, in this work the batch size at test time is kept fixed - see Sec. C). For a reference, see the analyses performed in Schneider et al. [42].

2) Related to point 1), I believe that some more ablation studies to understand the trade-off between the hyper-parameters in Eq.4 should be provided. Related to this, I found curious to see in Tables D.1 and D.2 that using the covariance matrix alone helps more than using the mean vectors alone: the covariance matrix alone only carries information about how feature vectors are spread in their space, not about their location. What would happen without the SSL objective (that is, by setting $\lambda_s=0$ in Eq.4)?

3) Minor - I did not find the analysis in Sec. 3.1. very enlightening, because the main task and the SSL task are very artificial. An example where it is shown that the proposed method avoids degenerate solutions would be significantly more helpful, in my opinion.

Note: I am not able to properly assess the usefulness and the correctness of the theoretical results in Sec. 4.1 (as well as their proofs), since they are beyond my expertize.

I believe this is a good paper, with some parts still lacking a bit of work. I look forward reading the Author response, discussing with other Reviewers and updating my score accordingly.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The work proposed a test time training method to address the distribution shift between training and testing using online feature alignment and contrastive loss based SSL. Reviewers overall see the paper well written and are content with the empirical evaluation. The rebuttal was able to convince some reviewers to raise their scores. I do agree with reviewer qJr7 that the contrastive loss based SSL task does not specifically address test time training, but rather is a generic approach to improve representation learning/generalization, which is well established. Focusing more of the discussion around the online feature alignment might shed more light on the new contribution of the work. As suggested by another reviewer, on-the-fly adaptation might be the more interesting setup to further distinguish the TTT setup from other setups facing distribution shift, such as domain adaptation.  
Decision: Accept (Poster)