Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Test-time adaptation updates a model during testing to improve its generalization to different data.
A popular choice of test-time optimization is entropy minimization or pseudo-labeling, which trains on the model's own predictions.
This work seeks to prepare a model for such test-time optimization by meta-learning (1) the model parameters for a better initialization and (2) the pseudo-labeling inference to provide better targets for the loss.
As a first step, a probabilistic alternative to deterministic pseudo-labels is proposed, and then a variational distribution is defined to condition on the test inputs and their relationships.
The probabilistic pseudo-labels simply sample new pseudo-labels given the pseudo-label probabilities.
The variational distribution for pseudo-labels conditions on the given input and the other samples in the batch through a latent weighting $w$.
Training has three phases: meta-source optimization learns the source model parameters, meta-adaptation infers the variational pseudo-labels and optimizes the target model parameters, and then meta-target optimization updates the source model parameters by the meta-loss to achieve better meta-adaptation optimization.
At the same time, the variational pseudo-labeling parameters are jointly updated alongside the model parameters, according to the meta-loss and a cross-entropy loss between the pseudo-labels and true labels.
Testing proceeds as usual for test-time adaptation, with alternating predictions and updates, as shown by Eqs. 10 and 14.
That is, given the meta-learned model and pseudo-labeling inference, the model parameters are updated during testing according to the cross-entropy loss on the predicted pseudo-labels.
Experiments evaluate adaptation on PACS, a common domain generalization dataset, along with toy datasets based on MNIST with artificial domains made by rotating images, while comparing with recent test-time adaptation methods.
Accuracy improves by 1-2 points with the proposed probabilistic pseudo-labels and variational pseudo-labels (Table 1, Table 3, Table 4).
Analysis experiments show that the variational pseudo-labels reliably improve a little bit on standard pseudo-labels: overall (Figure 2, left), across iterations (Figure 2, right), and across different amounts of data for adaptation (Figure 3).
strength_and_weaknesses: *Strengths*

- The topic of pseudo-label generation is relevant and applicable to the improvement of test-time adaptation, as a set of existing methods adapt by entropy minimization and/or pseudo-label optimization.
- The meta-learning approach makes sense, and while there is generality in not requiring specialized training (as other methods like BN and Tent do not), if better accuracy can be achieved with different training then it is worthwhile to pursue it.
- The exposition of the main idea and its technical details is clear (but see weaknesses for missing implementation detail).
- The related work provides and apt and concise summarization of published test-time adaptation methods.
- The evaluation experiments with the standard ResNet-50 and the smaller ResNet-18 to show that results generalize beyond a single architecture (although these two are related).
- It is nice to see that the proposed variational pseudo-labels improve the accuracy at most steps of adaptation (Figure 2, right) and not only at the end of optimization (Figure 2, left).

*Weaknesses*

- Related work is missing, specifically confidence regularized self-training https://arxiv.org/abs/1908.09822 that likewise addresses the uncertainty and error in pseudo-labeling.
- The method is not fully described in its implementation: what are the batch sizes, number of updates, and the choice of parameters?
  (The parameters likely follow those of Tent and other methods, which update the affine parameters in normalization layers, but this should be specified.)
- The amount of improvement is limited. On the main evaluation dataset of PACS, the improvement is about +2 points for online or offline adaptation against Tent (a common baseline).
- The evaluation is narrower than most work on test-time adaptation. In particular, the works compared against include more benchmarks, such as ImageNet-C for image corruption (Tent, TAST, SHOT), and VLCS + OfficeHome + Terra Incognita for domain generalization (T3A). Relative to these established options, the chosen Fashion-MNIST and MNIST are toy datasets.
- The process for model and hyperparameter selection is not explained. How were the learning rates for optimization chosen, for example? Are all of these tuned to the accuracy of adaptation on the test set?
- The training could be more computationally intensive per step, and may require more steps overall, due to the multiple phases of meta-learning. This is not discussed or quantified.
  The meta-learning approach is likely more expensive than standard supervised training, but the question is how much? This should be measured empirically.


clarity,_quality,_novelty_and_reproducibility: *Clarity*:

The exposition is clear, the writing straightforward, and the organization of the work is easy to navigate and digest.
The setting and method are clearly explained by the notation of Sections 2.1 and 2.2 and Figure 1.
The intuition for why sampling and variational pseudo-labels should help is communicated with a toy example.
However, the implementation details leave some points to be desired, especially given that test-time adaptation methods can be sensitive to hyperpareters like the batch size and number of updates.


*Quality*:

- The experiments and exposition are well-executed under the chosen scope, but that scope is limited. There are more possible benchmarks, and these benchmarks are common in the prior papers on this subject.
- The overall accuracy improvements are small, and in line with improvements from better tuning of learning rates and batch sizes for methods like Tent. See https://arxiv.org/abs/2104.12928 and results for "ENT (ours)".
  The improvement due to the meta-learning of variational pseudo-labels, which is the main technical contribution of the paper, is smaller still (see Table 1).

*Novelty*:

- Probabilistic sampling of pseudo-labels is straightforward but nevertheless novel. However, it is not quite correct to say that this is the first way to leverage uncertainty. The updates of soft pseudo-labeling and entropy minimization in the style of Tent both change as a function of the confidence of a pseudo-label.
- Variational pseudo-labeling as defined and implemented here is novel. Other methods like SHOT (by prototypes) and TAST (by neighbors and prototypes) also make use of other points in defining their pseudo-labels, but do not meta-learn a pseudo-labeling inference network as done by the proposed work.
- Variational pseudo-labeling approach is not the first to make use of other points in the batch. In fact, most test-time methods do, because their batch-wise statistics or parameter updates couple inference across inputs.

*Reproducibility*:

There are some gaps in the implementation details that would take some effort and experimentation to determine. There is no statement about providing the code.


summary_of_the_review: This work brings a probabilistic perspective to pseudo-labeling and contributes a meta-learning scheme for training the model parameters and a pseudo-label predictor.
The proposed stochastic and variational pseudo-labels do seem to improve marginally on PACS, but the improvement is only marginal, and the evaluation does not cover other standard choices of test-time adaptation benchmark.
The small effect and narrow evaluation dampen the significance of the proposed techniques even if they are novel variations on pseudo-labeling.
As there is already work experimenting with exactly how to define pseudo-labels—SHOT, RPL, TAST, and conjugate pseudo-labels—more is needed to advance beyond what has already been done and inform future methods for test-time adaptation.
I encourage the authors to more fully evaluate the proposed method and explore if its improvements compound with other techniques to achieve more significant accuracy improvements.

**Update after response**: The response and revision provided experiments for domain generalization to cover more datasets and more baselines. Furthermore, an ablation justifies the contribution of meta-learning and not merely sampling the pseudo-labels, and the computational cost has been measured. These results reinforce the empirical significance of the work. However, the significance for test-time adaptation is still an issue, because this work only addresses multi-domain/multi-source training, and so does not evaluation on standard benchmarks for adaptation like the ImageNet variants (-C, -R, -V2, ...) or VisDA-C. As such the recommendation is raised to 5 but not higher.

*Points for Rebuttal*

1. Please clarify the implementation of the test-time adaptation step. How many updates are made per batch? What is the batch size?
2. Please measure the training computation relative to other methods. This could be measured by some subset of time, FLOPs per step, or number of steps, for example.
3. Please comment on the lack of results for image corruptions (ImageNet-C, CIFAR-10/100-C) which are provided by most prior papers on test-time adaptation. These experiments would provide an informative and more thorough comparison of the proposed method with the state-of-the-art.
4. Please relate the novelty and claims of this work to those of confidence regularized self-training (see link under Weaknesses).
5. Please relate the proposed method and degree of improvement to the conjugate pseudo-labels method (Goyal et al. 2022). This is not a prior published method, so it does not count against this submission, but comparing to it would make this submission more comprehensive.

*Miscellaneous Feedback*

- 1. Introduction
 - "these two settings" would be clearer to a broader audience if it identifies domain adaptation (which requires targets during training) and generalization (which does not harness targets during testing).
 - The "pseudo-labels are more accurate" are they? are the pseudo-labels more correct in themselves, or better calibrated, or what? they can be more effective, if the adaptation is more accurate, but that does not mean the pseudo-labels are in themselves more accurate
- 4.3 Comparison
  - What about consistency regularization/DIRT-T and VAT? Such methods sample virtual "neighbors" of each point by penalizing changes to the output on small changes to the input. MEMO likewise updates on augmentations of the input. Do the proposed pseudo-labels compound with these methods and give more improvement?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a test-time adaptation (TTA) method from probabilistic perspective. Specifically, they analyze first analyze the pseudo labeling, which is a naive approach in TTA, from probabilistic point of view. Based on the analysis, they propose variational pseudo labels,　and meta-learning based algorithms. The empirical validation is conducted on PACS, rotated MNIST and Fashion-MNIST. 

strength_and_weaknesses: [Strength]
(1) Test-time adaptation is timely and relevant topic for the conference. 
(2) The paper is well written and easy to follow. 
(3) Meta-learning based algorithms with pseudo label is novel and interesting. 


[Weaknesses]
(1) Empirical validation is limited to small-scale dataset. While I agree PACS and rotated MNIST are standard benchmarks, but there are lot of more benchmark dataset (e.g., DomainNet, OfficeHome, TerraIncognita, etc. [1]). Also, it lacks comparison with standard DG methods. Since the standardized evaluation is regarded as one of the big issue in DG papers [1] and this paper does not provide theoretical results, I strongly recommend adding more empirical evaluations to fully show the benefit of the proposal. 

[1] Gulrajani, Ishaan and David Lopez-Paz. “In Search of Lost Domain Generalization.” ICLR2021

(2) There are some missing literatures about pseudo labeling both inside [2, 3] and outside [4] the field of the test-time adaptation. Since the pseudo label is the core of the paper, authors should discuss the differences between these studies. Besides, it is also preferable to add these methods as baselines. 

[2] Goyal, Sachin et al. “Test-Time Adaptation via Conjugate Pseudo-labels.” NeurIPS2022
[3] Wang, Qin, Olga Fink, Luc Van Gool, and Dengxin Dai. 2022. “Continual Test-Time Domain Adaptation.” CVPR2022
[4] Rizve, Mamshad Nayeem et al. “In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning.” ICLR2021
clarity,_quality,_novelty_and_reproducibility: [Clarity and Quality] 
This paper is generally well written. However, there are some confusing parts:
- What is the test-time adaptation in table 1?
- Contents of Table 1 and Table 2 seems to be largely overwrapped. 

[Originality] 
The analysis and derived meta-learning based algorithms to compute variational pseudo label is novel. 

[Reproducibility]
Since the paper is mainly based on empirical results, it is preferable to provide full access to experimental code.  

summary_of_the_review: While I agree that the pseudo-label in TTA is worth investigating and derived meta-learning based algorithms is interesting, the paper in current form lacks comprehensive empirical results and discussion about the relationships with prior works.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper discusses a probabilistic view of the adaptation process. Based on that, it introduces a meta-adaptation setting with a variational pseudo-labeling distribution.
strength_and_weaknesses: **Strength**: The paper discusses an interesting view of the online adaptation process (the variational meta-learning setting).

**Weakness**:
- I have several concerns about the probabilistic view of the adaptation process used in the paper. 
- The empirical evaluation is not weak.

Details of these weaknesses can be found below.
clarity,_quality,_novelty_and_reproducibility: The formulation is original. However, I have several concerns with this formulation.

This also hinders the clarity of the paper. I suggest the authors to address these concerns about the probabilistic formulation to improve both the correctness and clarity of the paper.
summary_of_the_review: **Concerns about the formulation**
- I have some concerns about the probabilistic formulation of the paper. For example, the authors suggest that the distributions $p(\theta_t|x_t,\theta_s)$ or $p(\theta_t|\hat{y}_tx_t,\theta_s)$ are posterior distributions. Can the authors clarify and elaborate? What is the prior distribution (of $\theta_t$)? We can't really have a posterior distribution without talking about the prior.
- I suggest the authors discuss in detail the generating process (all the priors and likelihood). Meaning, to discuss a joint distribution over $\theta_t,\theta_s,x_t,x_s,y_s,y_t$ as well as any other latent variables such as $w_t$, and all the terms that this joint distribution factorized into. Then, condition on the observed variables, we can discuss the posterior distributions and predictive distribution. This would make the paper much clearer and help us understand the formulation more deeply.

**Concerns about the evaluation**
The evaluation in the paper seems incomprehensive. 
- First of all, it is beneficial to validate the method on standard online domain adaptation datasets, such as Cifar10/Cifar100/ImageNet Corruption and the VisDA17 dataset. This will help us to have an idea of how the method compares against the literature.
- I also suggest the authors include stronger baselines. For example, EATA [1], TTT[2] or TTT++[3]


[1]  Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P. &amp; Tan, M.. (2022). Efficient Test-Time Model Adaptation without Forgetting. Proceedings of the 39th International Conference on Machine Learning
[2] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pages 9229–9248. PMLR, 2020.
[3] Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi.TTT++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 2021.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addressed test-time adaptation by incorporating uncertainty. This is inspired by the fact that pseudo labels could be confidently wrong, thus pseudo labels are treated as a distribution and test-time training is carried out on the pseudo labels sampled from the distribution. A meta-learning approach is further proposed to better handle domain shifts between source and target domains.

strength_and_weaknesses: Strength:

1. Allow pseudo label sampling is a novel idea in TTA.

Weakness:

1. Experiments are not carried out on commonly adopted TTA datasets, e.g. CIFAR10-C, CIFAR100-C, ImageNet-C, VisDA. I would like to know why benchmarking on the above datasets are missing.

2. Meta learning requires a held-out validation set. Does it require the validation set to mimic the potential corruptions or distribution shift in the real target domain? If such knowledge is available does it mean the domain shift is a known priori? I am afraid this violates the assumptions of TTA where the information on target domain should be kept unknown before inference.

3. Since the benefit is potentially contributed by sampling pseudo label from a distribution. It is recommended to evaluate directly sampling pseudo label from the posterior, i.e. the probablistic output of unlabeled data, on the presented datasets.

4. I am wondering how integrating the features of neighboring target samples can be achieved in an online TTA mode. If test samples arrive in a stream neighboring samples can not be collected before they are seen.

5. Is the meta-learning adaptation a necessary step? If not, it is recommended to separately evaluate the importance of incorporating the meta-learning adaptation.

6. One important concern for test-time training is the inference speed. How much extra computation overhead does this variational pseudo label introduce?



clarity,_quality,_novelty_and_reproducibility: The overall idea is relatively clearly presented. But some key ingredients in meta-learning are missing which makes it hard judge whether these operations are fair. For example, how is the domain shift simulated on source domain data is a very important concern. The idea of sampling pseudo labels from This method can be hardly reproduced if source code is not released.

summary_of_the_review: Overall, this paper presents some interesting ideas for test-time adaptation. In particular, sampling pseudo labels from categorical distribution is interesting, but awaits more empirical evidence. The meta-learning approach is not clearly explained thus potentially raising the question of whether domain shift must be a prior knowledge. Some commonly adopted experiment settings are missing in this work which limits evaluating the proposed method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The proposed method was deemed enough novel, well justified and clearly presented, with some exceptions.
Main issues relate to the validation phase: limited scope of the experiments, missing comparative analysis (e.g., with domain generalization methods), and the need of validation on more standard datasets (since it is limited to small scale datasets). As for the latter, it was also noted only small increases in performance, not all consistent. Finally, some related work is not quoted, as well as an evaluation of the computational cost.
Some reviewers argued about missing details on some stages of the method asking clarifications, including the probabilistic formulation, and about implementation details for reproducibility. 

Authors provide some answers to these concerns, also providing some additional results (a common remarks from all reviewers), but they did not succeed to fully satisfy them, who still remained with ratings below threshold.
For these reasons, this paper cannot be accepted to ICLR 23 in this form.
Decision: Reject