Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an approach to ensure a model trained on a set of source domains generalizes well to an unseen target domain based on a single unlabeled target sample. Unlike the standard domain generalization (DG) setting, where there is no scope for adaptation in the target domain, and domain adaptation (DA), where the model has access to source and unlabeled target data during training, the paper falls somewhere in between where no access to target data is assumed during training and a single unlabeled target sample is allowed for quick adaptation. In principle, the proposed setting is somewhat similar to source-free domain adaptation with the distinction that only one sample (and not the entire target dataset) and one quick adaptation pass on the data is allowed. The authors leverage a meta-learning paradigm to mimic domain shift by defining meta-source and meta-target domains within the source domain vocabulary (has also been considered in prior DG settings). The approach adopted by the authors relies on modeling a conditional distribution over the model parameters given a meta-target domain sample and source domain representative features. The authors set this up within a variational inference framework that allows them to explicitly parameterize this conditional distribution to infer model parameters in a target domain. The authors further explore the utility of different instantiations and alternative formulations of their objective and show that their proposed version has the best performance. When compared with prior DG and test-time adaptive methods, the proposed approach leads to competitive or improved performance. Further ablations demonstrate the utility of the proposed approach over other test-time adaptive approaches in terms of number of target samples available for unsupervised adaptation.
main_review: I will now highlight the strengths and weaknesses of the paper.

**Strengths**

1. The paper is generally well-written and easy to follow. With the exception of a few minor points raised under weaknesses, the authors generally do a good job of situating the problem setup and motivating the proposed approach. I particularly appreciate how the authors build up from the most rudimentary version of the proposed approach to adding more relevant terms to the objective that might further improve out-of-domain performance. Furthermore, in this space of test-time adaptation, when compared to prior work which either rely on single or multiple passes over the entirety of target data, I think the paper is trying to address a relatively realistic and timely problem setting that the community might find useful.

2. Unlike prior test-time adaptation approaches, I think a big positive point in support of the proposed approach is that it doesn’t require — (1) fine-tuning on target data, (2) more than a few target samples, and still ends up either being competitive or outperforming the same. Obtaining an adapted model on target data based on a single or few source samples with a single forward pass offers significant benefits in terms of data and computational constraints.

3. Out-of-domain generalization results obtained via the proposed approach seem to demonstrate the utility of the same (although marginally in some cases). Further ablations on rotated-MNIST highlight the stability and utility of the same when compared to TENT. Additionally, I like that the authors' highlight failure cases of the proposed approach accompanied with hypotheses surrounding things that fail on particular samples.

**Weaknesses**

1. If I understand correctly, one of the crucial bits the proposed approach relies on during training is the availability of domain labels on source data — source instances belong to different domains because the domain labels say so, which in turn influences how the meta-source and meta-target splits are constructed (also noted under conclusions in the submission). While this is a perfectly fine assumption, the paper would benefit if the authors included an experiment validating the extent to which the proposed approach relies on this. Essentially, a setting where the source domain labels are randomized — as in, (say) if there are 3 source domains, data at training time is grouped randomly into 3 groups instead of by domain and meta-source & meta-target splits are constructed based on these three groups. Considering such a setting would highlight how important mimicking the domain shift within a source is important. If the out-of-domain obtained results are worse, then there is additional validation that constructing appropriate meta-source & meta-target shifts matters. If not, then the utility of the proposed approach lies somewhere else. I would encourage the authors to include such an experiment.

2. [Minor Points] Including the model architecture diagram from the appendix to the main paper would make it easier for the reader to follow the supporting description in Section 3  (page 5) and under implementation details. While I understand the need to include the target sample in the variational posterior (as considered in equation 3), I don’t completely follow the supporting justification (on page 4, the paragraph above “single-test sample generalization”) in terms of the adaptivity gap. Adaptivity gap (as discussed in Dubey et al. 2021) is a general statement about DG settings. I’m not entirely certain what extra insights it provides in the supporting justification. When finally evaluating on target data, is an adapted model obtained for every target sample, or is it the case that it is obtained from a few samples. If it’s the latter, how are these target samples chosen? 
summary_of_the_review: The points highlighted under strengths and weaknesses form the basis of my rating. I am generally supportive of the paper and think that it addresses the test-time adaptation problem from a relatively constrained yet realistic standpoint. The paper is well-written and generally easy to follow. The most positive bit in support of the paper in my opinion is the lack of reliance on the entirety of target data or multiple passes on the same to adapt a model. Regarding weaknesses, since the paper relies heavily on the meta-source and meta-target split construction, I think it’s important to address the extent to which this reliance is true (in accordance with the suggested experiments in 1). The minor points in 2 are addressable, I think.

**Thoughts post author responses**

As stated in my follow-up reply to the authors, I think my primary concerns regarding domain-split construction were sufficiently addressed by the authors. Similar concerns were also shared by other reviewers and the authors provided sufficient experimental evidence to address the same. Additionally, concerns surrounding results with different backbones have also been sufficiently addressed (in my opinion) by the new experimental results. Given these and the fact that the paper makes an interesting contribution and studies a timely experimental setting, I continue to recommend acceptance of the paper (and would like to stick to my original rating).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose to learn to generalize across different target domains with single samples by using a meta-learning paradigm. Specifically, during training, the course domains are divided into several meta-source domains and meta-target domains to explore the adaptivity of the model by incorporating information of the meta-target sampling when learning the model parameters. 
main_review: Strength:
* The paper proposes to use meta-learning scheme to do one-shot learning. The idea is heuristic to the research community
* The paper provides several empirical study and analysis to demonstrate the performance of the proposed approach
* the proposed method can benefit real application scenario of extreme few shot cases


Weakness:
* It is unclear how training on each instance of the source domain would enable a large enough domain shift to benefit a larger domain shift encountered during test time. 
* even though most experiments in the paper are considering multi-source domain adaptation. It is unclear why not testing on the single-source setting, since training the proposed method does not require multiple source domains
* It seems that the proposed method underperforms the method proposed by Zhou et al. (2020a)  in all of the Resnet-18 experiments. and For some reason, for all Resnet-50 Experiments only for this baseline method is not conducted
* it would be helpful to show how an ablation study on how exactly the proposed meta-learning scheme improves performance vs non-meta-learning based one-shoot learning, such as [1, 2], 

[1] Luo, Yawei and Liu, Ping and Guan, Tao and Yu, Junqing and Yang, Yi. Adversarial Style Mining for One-Shot Unsupervised Domain Adaptation. Advances in Neural Information Processing Systems, 2020. 

[2] Dong, Nanqing and Eric P. Xing. Domain adaption in one-shot learning. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2018. 573--588. 
summary_of_the_review: As the discussed above, due to weak performance, and lack of more empirical demonstration of the proposed method. I recommend marginal reject on this paper. I would like to adjust my rating after discussing with other reviewers and AC
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studied the problem of single test sample generalization. Its goal is to adapt a pre-trained model to unseen target domains without extra fine-tuning. The paper formulated the single test sample generalization problem as a variational inference problem and proposed a meta-learning framework. To bypass extra fine-tuning, a single test sample was used as a conditional to generate model parameters. Experiments and ablation studies on common-used benchmarks demonstrate the effectiveness of the proposed method. 
main_review: Pros. 

1. The proposed method is more flexible than the previous generalization- and adaptation-based methods. Compared to generalization-based methods, the model can better adapt to target domains via conditional generated parameters. Compared to previous adaptation-based methods, the proposed method does not rely on batches of samples from the target domain with extra fine-tuning operations or extra networks.

2. The paper systematically summarizes previous related works including domain adaptation/generalization, domain meta-learning, and test-time adaptation. Table 1 clearly shows the difference between this work and previous works in terms of training and test-time settings. 

Cons.

1. The proposed method requires multiple source domains during training to support the meta-learning scheme. However, previous test-time adaptation models do not have such limitations and can be trained on one single domain. How to guarantee fairness in comparison to previous test-time adaptation methods? What is the minimal number of source domains required for training?

2. During training, the paper proposed to approximate the model function with the accessible source data by the variational inference. Is there any underlying assumption about the distribution similarity between the source and target domain? Can the "adaptivity gap" still be minimized when the target domain is significantly different from the source? 

3. The paper proposed to construct an adapted classifier during inference. As indicated in the main paper, "both the meta-prior distribution and the variational posterior distribution of the classifier are generated by amortized inference using the amortization technique", but why the amortized inference is applied?  More elaboration is required to justify the motivation.

summary_of_the_review: This paper addressed "single test sample generalization", which is an interesting and important problem in the field of domain generalization. The method is technically sound and novel. Although I have some concerns regards the method and experiments as indicated in the main review, I overall remain positive towards this paper. I hope the authors can address my concerns in the rebuttal.

After rebuttal: I have read the response and comments of other reviewers. All of my concerns have been addressed in the response. I recommend acceptance for this paper. I highly suggest that the authors should widely discuss the remaining weaknesses raised by the reviewers in the final version.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper deals with the problem of domain generalization, specifically performing inference on a single test example.  The key idea is to use meta-learning to train the model on the source data -- in the training stage, the source data is divided into meta-source and meta-target, and the model is trained to adapt to the meta-target domain.  This mimics domain shift within the source data.  The method is seamlessly transferred to the target domain, where inference is performed on a single test example, without finetuning model parameters or using additional networks. 
main_review: # Strengths
1. The idea of artificially creating meta-source and meta-target domain within the source domain is interesting.  This exposes the classifier to the notion of distribution shift and the learnt abilities of the model to handle such distribution shifts are shown to transfer to the unseen target domain.
2. The problem an interesting angle for viewing domain generalization. The key question of "how to expose models to distribution shift" when no target samples are available, is an interesting one.

# Weaknesses
1. It is not clear why a random division of source $\mathcal{S}$ into $\mathcal{S}^\prime$ and $\mathcal{T}^\prime$ will always simulate distribution shift. Does a more sophisticated division (perhaps using clustering or some notion of feature distance) make more sense to create $\mathcal{S}^\prime$ and $\mathcal{T}^\prime$?   The distribution shift between meta-source and meta-target has not been quantified or even visualized.  This limits the modification of this idea to other tasks since _why a random split works_  isn't clear.
2. While the multi-source domain generalization setting has been explored, it is not clear to me whether the same approach will apply to cases where only one source is available (for instance SVHN as source and MNIST/USPS/... as target, as shown by TENT (Wang et al. )).  Domain generalization is largely an unsolved problem in the digits classification task -- which is considered a trivial task in terms of in-domain accuracy (near-100% performance on MNIST).  It is unclear to me how the proposed approach would be able to simulate a meta-source and meta-target if the source is a largely homogenous dataset such as MNIST.
3. It is stated in the introduction that existing test-time adaptation methods such as TENT perform finetuning, while this method does not.  Unless I am mistaken, model parameters are updated for every new test sample even in this method (please correct me if I'm wrong).  So how is it not finetuning?  Sure, additional networks or proxy self-supervision tasks are not used (like in TTT), but there does seem to be adaptation at test time.

# Questions
1. In the PACS experiments (Table 3), the method surpasses prior art with ResNet50, but not with ResNet18. What might be the reason for this?
2. In Figure 2 (and similar figures in the Appendix) I don't understand the underlying space of the visualization.  How is a classifier and a test sample visualized in the same space? Or does the blue symbol indicate the subspace which the classifier predicts as the respective classes? Please provide details of how this visualization is computed.  

# Feedback 
- On page 1, existing test-time-training methods mentioned (Sun et al, Wang et al) are mentioned, with the statement that "these methods typically rely on batches of samples from the target domain".  This is not entirely true
    - TTT (Sun et al.) have two variants of their method, one which operates on a single sample, and an online method that uses a batch of training samples.
    - There is also a recent method in NLP (Banerjee et al. NAACL 2021 https://arxiv.org/abs/2103.11263 which is largely inspired from TTT) that makes predictions on single test instances.
- Table 1 is useful for understanding how this work differs from prior art.  However TTT (Sun et al.) isn't listed here -- it should be listed as a method that uses single samples $x_t$  and performs fine-tuning/adapting.
- Table 1 could also include a column that states whether extra networks / models are needed.
- The model architecture figure from supplementary material should be moved to the main paper.
- I would also recommend adding an algorithm / pseducode to complement the equations in Section 3. The methods section is more than 2 pages long, and not having a succinct pseudocode hampers the readability of this paper.
- Related to my statement in Strengths #2, a recent paper approaches this problem by proposing a data augmentation method that exposes classifiers to "novel views" of a source image: see Chai et al. CVPR 2021 https://arxiv.org/abs/2104.14551 . However, they target adversarial robustness (and do not have results on DG benchmarks).  It might be interesting to compare their method of test-time data augmentation, with your method of test-time variational inference, in related work.

# Review Summary
The idea is interesting and the paper is well-structured in general.  However there are a few confusing aspects that I have pointed out as well as some inconsistencies with the claims in the paper.  That being said, the empirical results suggest that this method is superior to TTT, TENT and other baselines on benchmarks used in this paper.  I am giving a weak reject at this point, but will be happy to receive clarification from the authors before the final decision.

## update after discussion period
The authors have sufficiently addressed concerns raised by me and other reviewers and new experiments / analyses / comparisons have strengthened the paper.  After these changes, I am increasing my rating to "accept".

summary_of_the_review: interesting idea, but confusing/inconsistent claims; unclear why a random split of source into meta-source and meta-target would create a distribution shift.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: - The paper describes a method for domain generalization that performs test-time adaptation using a single test example at a time (as opposed to a transductive setting used in other works where a whole batch of test examples are used).
- The method is cast as a meta learning task. The training data is split into meta-training/test domains to mimic the adaptation process during training.
main_review: - (+) Interesting setting that has not received much attention: test-time adaptation from a *single* test example.
- (-) However, there is little intuition given why test-time adaptation using a *single* example makes sense. It's not clear why this should work, unlike adaptation using a batch of examples, which makes much more sense intuitively (to me, at least).
- (+) Elegant formulation of the method as variational Bayesian inference.
- (+) Computationally-efficient method: a single forward pass necessary to adapt the model
- (-) I'm not sure the method is really an instance of "meta-learning" (as in "learning to learn"). I understand that the method is simply mimicking the adaptation process at training time so as to optimize the parameters of the adaptation process.
- (-) There is little analysis of the results to show why the method performs quantitatively so well. It's difficult for me to intuitively grasp why adaptation using a single test example outperforms methods that also use other test examples from the same domain (Wang et al. 2021, Dubey et al. 2021).

Additional advice to improve the paper.

- Illustrating figures (as early as the introduction) would be helpful to convey more efficiently the gist of the method, and the motivation for the setting (as mentioned above, it's not clear to me why adaptation to a single example makes sense).
- Additional possible reference of a method for test-time adaptation: [Learning to Generalize One Sample at a Time with Self-Supervision](https://arxiv.org/abs/1910.03915)
summary_of_the_review: Preliminary recommendation: unsure. The paper describes an elegant method with good empirical performance, but it does not provide intuition/theoretical explanations/empirical analysis why it should/does work. I feel that this paper brings little new scientific knowledge that future research could build on, hence its impact may be limited.

Requests/questions to the authors.
- Please address the negative points raised above.
- How are the source/target domains generated during training ? I understand they are generated at random, but since you have multiple domains in the datasets used (say, PACS) I imagine you do use the annotations of these domains at some point ?

---
**Final summary after discussion with other reviewers**

I am fine seeing this paper accepted given that this is the advice of the other reviewers. However I still see important weaknesses in this paper that the authors may want to fix in the final version and/or in future work.

1. There is no intuitive or strong theoretical support for adaptation from a single test example. The proposed method does produce non-trivial empirical results that means that it must rely on specific information/assumptions, which are not made clear. The reason it matters is that these assumptions must have limits of applicability. These **limitations** should be explored, discussed, or at the very least acknowledged in the paper. My impression is that the variational and meta learning aspects of the proposed implementation are eclipsing the more fundamental points on which the method relies.


2. The authors responded to the lack of **illustrations** by including fig. 1/2 but I don't find these satisfatory. Figure 1 provides almost no information regarding (1) above. As for Figure 2, it's not even clear what it represents ("architecture of our method"). Training time ? Test time ? The boxes are a mix of operations/layers, variables, and distributions. The colors are not defined. Etc.


3. I still think that calling of the approach "**meta learning**" is a stretch. Another reviewer made me realize that there is a precedent in the literature for using the term when merely mimicking the adaptation at training time, but I'm not sure that perpetuating a bad choice is a valid reason.
review_title: null

Meta-review: This paper proposes a new method for domain generalization by adopting a single test example. Authors formulate the problem using a variational bayesian framework which ends up in an adaptation technique requiring a single feed-forward computation. The provided empirical results indicate that the proposed method has comparable performance to techniques which require more data.

Reviewers all acknowledge the novelty and significance of this work. The paper is well-written and the related work is adequately discussed. Moreover, the proposed method is computationally efficient and empirical results provide strong evidence in its favor. While I am recommending acceptance, I tend to agree with reviewer xA1m about the main weaknesses of this work and I recommend authors to improve them for the final version:

- Lack of proper discussion or intuition about under what conditions the proposed method works well. This may be using theoretical analysis, using toy examples, trying to break the method, motivate using prior work or just simply providing intuitive arguments. Also, as reviewers pointed, Figure 1 is currently very confusing.
- Lack of analysis or ablation study allows a better understanding of the proposed method
Decision: Accept (Poster)