Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new source-free domain adaptation method that can work without back-propagation and is capable of handling a target dataset containing both domain-relevant and domain-irrelevant samples (called LDA setting). The proposed method firstly embeds the target dataset into features and conduct adaptive inference for target samples using cross-attention mechanism with the embedded features. The modules for embedding, cross-attention, and classification are jointly trained via a scheme of meta-learning. Experimental results with several popular datasets show that the proposed method works well especially in the LDA setting.
strength_and_weaknesses: <Strength>

- Since the proposed method does not require any training process with back-propagation operations, it should be easy to use and can be applied to a wide range of applications. 
- The proposed method is also useful in terms of robustness against domain-irrelevant samples in the target dataset.
- The proposed method consistently achieves good performance in several benchmark datasets.
- This paper is well-written and easy to follow. I enjoyed reading it.

<Weakness>

- I do not have any major concern.
- Shown below are minor things (almost just out of curiosity).
	- Do we need to fix the size of the support set? As a mechanism, the cross-attention module can process a variable size of the support set. Allowing this might be useful, because it makes it possible to handle a various size of a target dataset as is.
	- Can we use the proposed method in an online manner like test-time adaptation? Since LN(f(x_s)) can be stored for the subsequent inference process once we compute it, we can conduct online adaptation by using the proposed method, if we have a strategy to properly update the stored feature with LN(f(x_q)). It would be great if we can use the proposed method as an unified method to solve both source-free domain adaptation and test-time adaptation. (I understand that the above makes the proposed method much similar to ARM, though)



clarity,_quality,_novelty_and_reproducibility: <Clarity>

This paper is well-written and easy to follow. I enjoyed reading it. 
Minor concerns are as follows:
- it might be better to change the notation of the classifier in Eq. (1) from f to any other letter. 
- z represents a domain in Eq. (1) but extracted features at the end of Section 3.3. It is also encouraged to change either notation.

<Quality>

As far as I understand, this submission is technically sound.

<Novelty>

Although the proposed method follows a similar flow to ARM, there are two critical differences as steted in Section 3.1. Especially, the second one, which is robustness against domain-irrelevant samples in the target dataset, is practically important and is effectively tackled by utilizing a cross-attention mechanism, which provides sufficient originality to this study.

<Reproducibility>

The paper provides sufficient implementation details in Section 4.3 and the appendix.

summary_of_the_review: The proposed source-free domain adaptation is sufficiently novel and empirically works well. The manuscript is well-written. I vote for "accept."
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors focus on source free latent domain adaptation. Here the content or label of  target data overlaps with the one of the source data. And authors doest not get access to the source data when using target data. Furthermore, authors also strict setting where the network is not updated which is backward-free. To achieve this goal, authors explore the probability of using meta-learning, and propose to use cross-attention to select relevant examples for adaptation. The experiment results demonstrate the effectiveness of the proposed method.   




strength_and_weaknesses: 1) This paper explore new setting about latent domain adaptation: source-free and forward-pass, which is a new highly practical setting.

2) Authors try to give clear explanation about this setting, and show why it is interesting in practical application. 

3) The paper is well-written and easy to follow.

4) The experiment is sufficient to support the proposed method. 

clarity,_quality,_novelty_and_reproducibility: 1) The paper explores a new setting which is interesting.  The proposed method is based on meta-learn mechanism, which seems to be less interesting. 

2) Authors present the difference about the latent domain adaptation and standard domain adaptation, which is clear. However, the used dataset is suitable for this setting, since it is standard dataset. 

3) The result (e.g., table 1) shows the proposed method has less advantage, which seems to be hard to support the effectiveness of the proposed method. 


summary_of_the_review: This paper use the meta-learning method to explore source-free latent domain adaptation, which sounds interesting. However, for forward-pass, it is less convincing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to meta-learn a network capable of embedding the mixed-relevance target dataset and dynamically adapting inference for target examples using cross-attention. It further analysis the method performance in the domain-supervised adaption. 
strength_and_weaknesses: Strength: 
the writing of this papers seems easy to follow and it demonstrates its method from several perspectives. 

Weakness:
1.	There are several implementation details associated with the proposed method such as the augmentation, cross attention, meta learning. It would be better to have  the ablation studies on decomposing the each components. 
2.	I am concerned the performance boost with the relatively large computational time. Although in table 1, the authors shows the improvements but coming with larger computations. 
3.	The author claims privacy in the paper. However, there are not many privacy baselines are used as the baselines. I would recommend the authors to compare with some privacy related domain adaption such as black box related domain adaption. 
4.	I also have the questions regarding the scalability of the proposed method. With the meta-training, how does the proposed method formulate on the larger dataset or settings? And what are there performance? It would be better to extend the proposed method on the larger dataset and with the detailed running time. I suspect that the running time will be much larger compared to the baselines. 

clarity,_quality,_novelty_and_reproducibility: See the weakness. 
summary_of_the_review: ----------------------------Updates---------------------------------
After reading the response and other reviewers' comments, I would like to keep my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors proposed the Latent Domain Adaptation setting where the a source model should be adapted to a target dataset that contains a mixture of unlabelled domain-relevant and domain-irrelevant examples. The “latent'' means that we are not given the domain label in the training phase. To handle this problem, the authors proposed a feed-forward method which incorporates the cross-attention module with an episodic meta-learning framework. The authors have carried out extensive experiments to verify the performance of the proposed method. Furthermore, the authors analysed the training and inference time to show the strength of the proposed feed-forward method.
strength_and_weaknesses: Strength: 
1. The authors proposed a new setting for source-free domain adaptation. This setting seems more practical in real-world scenarios compared with the conventional SFDA.

2. The writing of this manuscript is clear and easy to follow. The overall presentation is nice, including smooth sentences, nice figures, and organised paragraphs.

3. The pipeline of the method is technically sound and provides detailed implementation of the proposed method.

Weakness:

1. In Section 3.1, the authors mentioned that “While the conventional domain adaptation setting assumes that xq and xs are all drawn from a common distribution…”. In the conventional domain adaptation, however, they usually belong to different domains.

2. Compared with ARM, the proposed method seems a bit incremental, it is just the simple combination of ARM and cross-attention module. The authors didn’t provide strong motivation for combining these techniques. In addition, neither ARM nor cross-attention module was proposed by the authors. 

3. Typo in Section 3.1: “To train a model than can be used as described above”. “than” should be “that”.

4. In Sect. 3.3, the author mentioned that “image-to-image attention is more suitable for domain adaptation than patch-based option because the overall representation should better capture the nature of the domain rather than a patch.” Is there any theory that supports this conclusion?

5. In my opinion, it is necessary to compare the proposed method with TENT and SHOT even though they are based on back-propagation. Although the proposed method does not require back-prop, this is a detail of the specific implementation and is not relevant to the task.

6. Even with the feed-forward, the proposed method still requires several iterations for training. So, what are the advantages of using feed-forward?

7. As the training time is heavily affected by the number of iterations of the method, the numbers in Table 3 are not convincing enough. Which method performs best with the same training time? Which method requires less training time to achieve the same performance? Furthermore, comparisons with TENT and SHOT are still missing from Table 3.



clarity,_quality,_novelty_and_reproducibility: There are some typos in the manuscript. The authors didn’t clearly demonstrate the strength of the feed-forward training. The technical novelty is very limited.
summary_of_the_review: Considering the limited technical novelty and insufficient comparison, e.g., with TENT and SHOT, the reviewer would not be able to recommend an acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a new problem setting: They assume a model is pretrained on source data, then the model is used to predict data in a target domain. They also assume that the resources are low and the method cannot update the parameters—i.e., cannot use backprop.

The proposed model is based on meta-learning. It is pretrained by taking batches as tasks and predicting target labels. The core computation part is to find the target image, in the given target set, that is most similar to the test image using a mechanism similar to self-attention in transformers. Then summing up the embedded vector of this image by the embedded vector of the test image to be used for prediction.

The model is evaluated in four datasets and shows some improvements.
strength_and_weaknesses: Reasons to accept:

The paper proposes a new research subject, and the results are good.

Reasons to reject:

The paper is extremely verbose, it is full of unnecessary information, it has some missing would-be useful information, and it lacks some visualizations to facilitate the convey of information. All together these have made the article extremely boring to read. As a person who has read many domain adaptation papers, in my opinion this paper is not worth reading, given the amount of information that it offers. Just a few examples (not a full list and not ordered): 
- You use too many mini-passages separated by a bold-faced phrase. These confuse the reader, particularly when each one focuses on a specific subject, and occasionally drift from the subject of the section.
- You used these mini-passages in the incorrect sections. For example I don’t expect to see mini-passages called “Latent domain adaptation” or “Feed-forward domain adaptation” in the introduction section, when you already have a full section called “BACKGROUND AND RELATED WORK”. This continues all over your paper. For example, you have a mini-passage called “Preliminaries” in the section called methods, but weren’t you supposed to cover these in the section called background? Even worse, when you read this mini-passage, you would find that it is not actually preliminaries, it is a mix of background and notations.
- You have a mini-passage called “Deployment phase”, aren’t you supposed to cover the model itself, then explain its deployment?
- You have a section called “OBJECTIVE”, weren’t you supposed to explain the model first, and then explain its objective?

   You see, this list goes on and on.

I am not sure about the applicability of the proposed problem setting. This is important because failing to fully justify and evaluate this part, makes me think that the authors have just tried to come up with an unrealistic problem setting to publish a paper for. The authors assume the model cannot use backprop, but cannot we replace the encoder with a smaller one to use a smaller compute? Also based on the examples that you are using in you paper, I am not convinced that why we cannot use source data. You are already using target data, so memory usage shouldn’t be an issue.

In the mini-passage called “Training phase”, you have assumed that the domain of the query image appears in the target set. Why?
clarity,_quality,_novelty_and_reproducibility: See the above section
summary_of_the_review: The paper proposes a new problem setting. But the presentation is horrible, and the problem setting needs further experiments to be justified.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In the context of episodic meta-learning, this paper introduces a new domain adaptation (DA) setup where domains are unknown ("latent") and test samples are not grouped to any explicit domain. This is motivated by their work with real DA, where test samples were not cleanly tagged into the new domains.  They propose a meta-learning approach that uses cross attention between test instances and adaptation sets.  As a second novelty, they look into an approach that does not involve back-prop optimization at adaptation time, to reduce computation at inference. The approach is tested on four benchmarks, yielding small improvement that is statistically significant in CIFAR-C and tiny imagenet.

Reviewers had split views about this paper, three reviewers recommended clear rejection and two recommended acceptance.The main concerns were around the empirical evaluations and scalability, and also around technical novelty. Authors submitted a rebuttal, but all reviewers decided to keep their scores. 

Two reviewers were concerned that training time may be prohibitively large due to the meta-learning procedure. Author claimed that training time is not important, because it is amortized across tasks. They also point out that they experiment with a large dataset iWildCam is >10GB. Authors state: "we do not consider the source model training time since this is not relevant to the actual adaptation". The point is that meta-learning approaches spend much more time at training time to yield models that can be adapted better or more quickly. They therefore tradeoff test-time adaptation with training time. 

One reviewer was concerned with the quality of presentation. This AC does not see any unusual issues with that aspect. 

Regarding the new setup of latent DA. In their motivating examples, it seems natural to group samples into domains based on metadata (location, time of acquisition), or by inferring a latent domain (clustering or manifold learning). Even their Figure 1 makes it clear that it would be easy to assign samples to domains (camera 1, night). For a new learning setup, one would expect to see examples where it is not possible to group samples into domains. In this context, iWildCam does not seem like the best datasets for this task, because the "domain" (camera ID) is not necessarily correlated with visual properties of the sample (as the authors point out). A very natural and important baselines to compare with, would be to infer a domain at test time. In a way, CXDA does exactly that in an implicit way at test time, and I would expect to see comparisons with approaches that infer the domain explicitly.

A potential benefit of the method is in terms of robustness to samples with misclassified domains. Perhaps CXDA yields stronger improvement over standard DA under such domain label noise? 

Based on these technical concerns, the paper is not ready for publication in ICLR. 




justification_for_why_not_higher_score: The learning setup was not sufficiently motivated, and some simple and natural baselines are missing. The empirical results are not strong, and scalability may be an issue.
Decision: Reject