Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work builds on the previous work[1] and extends it for a more general scenario, where per sample intermediate distribution shift is hard to quantify. They propose gradual feature interpolation for the case where samples from intermediate distribution are missing. Iterative self-training fails in such cases. The work presents results on synthetic and natural distributions to evaluate their claim.

[1] Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. arXiv preprint arXiv:2002.11361, 2020.
main_review: + The problem is well motivated and illustrated through different experiments and toy examples.
+ paper is well written and easy to follow

Clarification need:
- In Algorithm 2: matching scores are 1 for the same class, so if there are false positives(which happens a lot in real datasets), then the alignment will be biased towards them leading to poorer student network training?
- the choice of target examples to be mixed may create difficult examples early on, it is not clear how random assignments can overcome this situation? Also, the Align function along with the \lambda value will together decide, the hardness of the intermediate examples created.
- lambda_scheduler tuning is not discussed properly, will it depend on the gap between the two domains? if yes, then how should be handled?
- mismatch in lambda_scheduler step size notation \delta in Algorithm 1 and \sigma on page 3
- figure 1 has "gradual self-training" as a caption in one of the subplots but this method is not mentioned or cited.

summary_of_the_review: The work puts forwards an interesting work to gradually align two domains. Though different experiments are shown to support their claim, there are few clarifications needed as mentioned in the main review.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes an iterative self-training approach for unsupervised domain adaptation. In particular, the authors aim at gradually adapting a model trained on a source domain to the target domain. Based on the claim that previous work on this setting assumed that samples from distributions that represent gradual changes from the source to the target domain are available when adapting the model, the authors proposed a strategy to generate such intermediate samples for cases where they are not available. The introduced approach, named GIFT, consists in performing manifold mix-up between representations of examples from the source and target domains considering an increasing value of the hyperparameter that accounts for the weight of the representation of the target domain example. By doing so, the authors claim an automatic curriculum is introduced in the training. Moreover, since labels from the target domain examples are not available, the authors also introduced a heuristic to pair the examples that are mixed. GIFT was empirically validated on experiments with synthetic domain shifts on the CIFAR-10 dataset and was shown to outperform iterative self-training in terms of target accuracy. Experiments on two datasets presenting natural domain shifts were also performed.  
main_review: This work considers the problem of gradual domain adaptation, where a model should be gradually adapted from the source to the target domain. I found the idea of generating data (at a lower-dimensional space) from intermediate distributions interesting and appealing from a practical perspective. In the following, I present my major concerns, questions, and suggestions. 


- It is not clear from the manuscript why gradual domain adaptation is a setting that should be considered in cases where no samples from intermediate distributions are available. As far as I understand, gradual domain adaptation is a setting to be considered when the distribution yielding the data is evolving over time (Kumar et al. 2020) and the goal is to incorporate this knowledge about the problem structure in the model adaptation process. Given that, it is not clear to me why considering this setting in a case where the data distribution is not gradually shifting would make sense. If this is the case, why not directly use any other domain adaptation approach?


 - A major claim of this contribution is that by mixing-up representations of both source and target domains, examples from “intermediate” distribution would be generated. Although I can get a rough intuition of what an intermediate distribution could be (e.g. a mixture of source and target domains), there is no definition or discussion regarding this in the manuscript. I suggest the authors include in the manuscript a clear definition of what such distributions are, as well as include evidence that manifold mix-up is capable of generating samples from them. 


- Experiments
  - The authors mentioned that the decrease in confidence/accuracy observed on examples with higher levels of perturbation observed in 
Figure 2 confirms that the reason behind the success of iterative self-training is the implicit curriculum strategy. Although I understand that this observation indicates the existence of an implicit curriculum, it is not clear to me why it indicates it is the reason behind the success of iterative self-training.
  - It is not clear to me whether the results presented in Tables 1 and 2 are good. Even though I understand that the goal of this work isn’t to achieve state-of-the-art results in benchmarks, it is hard to assess the merit of the improvements reported in these tables without knowing how established baselines would perform in such test cases. I strongly encourage the authors to include for comparison at least DANN and CDAN.
  - In Table 3, the authors presented results obtained on two datasets. They mentioned that in the case of Camelyon17, hospitals 0, 1, and 2 were considered as source domains while hospital 3 was the target. However, in Table 3, it seems that only results with domain 0 as the source were reported. Why is it the case? Please clarify this and modify the text accordingly.
   - It is hard to tell whether the results presented in Table 3 indicate a relevant improvement of self-training approaches and the proposed GIFT in comparison to the considered baselines. The authors did not report if more than one run was performed. In case only one run was performed, I don’t think it is possible to draw conclusions from these experiments since models adapted via DANN, for example, are known to be quite sensitive to the initialization. 


- In addition to the aforementioned concerns, this manuscript presents several presentation and clarity issues that make it difficult to understand. In the following, I outline the major issues:
  - In Section 2, the authors introduced the considered setting but there are several missing points regarding the introduced notation. Please properly define $n_s$, $d$, $k$, and $n_t$. Moreover, the authors did not specify the underlying assumptions of the proposed approach, i.e., is the covariate assumption required? Is label shift allowed? 
  - Furthermore, in Section 2, the authors mentioned that “The goal is to bridge the domain difference and learn a good classifier for the target domain.” What exactly does “a good classifier” mean in this sentence? I believe the authors are considering the risk minimization setting from Ben-David et al. 2010, but please let this clear in the text.
  - It is quite difficult to parse the information contained in Algorithms 1 and 2. Several variables were not introduced, and there is no comment to explain what each line is doing. Summing this up with the fact that there is barely any explanation about GIFT training procedure and the Label-based Random Alignment throughout the text, it is hard to properly understand what the main contributions of this work are really doing. Moreover, the clarity of the Algorithms and the aforementioned contributions should be improved to facilitate the reproduction of the reported results. 


- Minor
  - It is hard to compare the results reported in Tables 1 and 2 because they are placed too far apart in the manuscript. I think it is possible to merge both into a single table.
  - The symbols $P_s$ and $P_t$ are used to denote different objects in the text. In Section 2, they are referred to as the source and target domains, respectively, while in Algorithm 1 they denote source and target datasets, respectively. 

summary_of_the_review: This work proposes an iterative self-training approach for unsupervised domain adaptation. In particular, the authors aim at gradually adapting a model trained on a source domain to the target domain. The main contribution is a strategy to generate examples from the so-called intermediate distribution between the source and target domains. Despite the reported empirical improvement over the considered baselines, I found the motivation of the proposed approach unclear, and I found that the main claim is not well-supported (i.e. is GIFT indeed capable of generating examples from intermediate distributions? Also, what are such intermediate distributions?). Moreover, the manuscript lacks clarity in several aspects and I have concerns regarding the significance of the reported results since it seems only a single run was considered in each test case. All in all, my initial assessment is that this manuscript is not ready for publication yet.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work adddresses domain adaptation (DA) by a GIFT method. GIFT consists of a manifold mixup technique, which generates virtual samples by mixing up the features of source and target domains samples. The mixup coefficient is annealed over time to bias towards the target domain. Another ingredient of GIFT is a co-teaching strategy that lets two networks teach each other. On a few small synthetic as well as natural image datasets, GIFT outperforms a few baseline methods.
main_review: 1. This work seems to have certain novelty. I haven't seen papers that anneal the mixup weights. However, it's unclear to me how much this strategy helps DA. I didn't find an ablation study that use a fixed mixup weight.
2. The technical novelty is a bit limited, as it's a simple extension of manifold mixup. To convince readers of its practical value, the authors should do more extensive experiments as well as ablation studies. In natural images, the two used datasets FMoW and Camelyon17 are not typically used to evaluate DA. The authors also should compare with many more existing methods. One important related method is Tent [1].

[1] Tent: Fully Test-time Adaptation by Entropy Minimization, ICLR 2021.

summary_of_the_review: The technical novelty of GIFT is limited. Hence, the authors should focus on showing the empirical benefits of GIFT. However, the experiments are highly insufficient, with missing ablations and baselines. Moreover, the evaluation datasets are not so popularly used. Therefore, it's hard to judge how much empirical value this work brings.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to address the gradual domain adaptation problem by creating virtual samples from intermediate distributions by interpolating representations of examples from source and target domains.
main_review: Strengths:
1. This paper is well-motivated since the gradual domain adaptation is still an under-explored problem.
2. The motivation and the overall idea of the paper is easy to follow.

Weaknesses:
1. novelty concerns:

(a) The main concern for this paper is the novelty. The main idea to address Gradual Domain Adaptation in this paper is to create synthetic data from source and target domains. There are already several previous papers working on this idea, such as DLOW [1], GVB [2], FixBi [3]. Although the above methods are not proposed to directly solve the gradual domain adaptation problem, they can be adapted to this problem. To prove that the proposed GIFT is better, the above methods should be compared after applying to the same problem.

References:

[1] Rui Gong, et al. "DLOW: Domain Flow for Adaptation and Generalization", CVPR, 2019.

[2] Shuhao Cui, et al. "Gradually Vanishing Bridge for Adversarial Domain Adaptation", CVPR, 2020.

[3] Jaemin Na, et al. "FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation", CVPR, 2021.

2. technical detail concerns:

(a) lambda scheduler: It should be critical to find the best way to transfer the intermediate data smoothly from source to target. However, the proposed lambda scheduler is a linear function with a fixed step size. The step size could be different when facing different datasets, making the current scheduler heuristic and not generalized.

(b) alignment method: it is still unclear which method is adopted for the final model according to the context. Pseudo-random alignment or cost-based alignment? This part is confusing.

3. experiment concerns:

(a) Table 1: The number of self-training iterations is different for iterative self-training and GIFT (5 and 20). This would make the comparison unfair. Why not keep the same training setup?

(b) Separating Tables 1 and 2 is confusing: It is unclear what the difference is between these two Tables. Is the main difference in the total number of training steps (1000 vs. 500)? Moreover, it is not clear why we can claim GIFT is more “robust” than iterative self-training. For Translated (50%-100%) CIFAR-10, iterative self-training raises the accuracy from 0.477 to 0.658 when increasing the training step number from 500 to 1000. However, GIFT drops the accuracy from 0.832 to 0.729. In this case, it is weird to claim GIFT is more robust.

(c) Figure 3: It is weird to compare with different numbers of teacher updates. It would be great to provide an explanation about this.

(d) Table 3: With the same baseline (i.e., Best (A, B, C, D)), GIFT does not have a clear improvement over iterative self-training.

4. other concerns:

(a) The publications in the Reference Section should be updated:

(1) Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. “The many faces of robustness: A critical analysis of out-of-distribution generalization”. ICCV, 2021.

(2) Ananya Kumar, Tengyu Ma, and Percy Liang. “Understanding self-training for gradual domain adaptation”. ICML, 2020.

(3) Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. “What is being transferred in transfer learning?”. NeurIPS, 2020.

(4) Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. “WILDS: A Benchmark of in-the-Wild Distribution Shifts”. ICML, 2021.

(5) Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. “Self-training with Noisy Student improves ImageNet classification”. CVPR, 2020.

(6) Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D. Cubuk, and Quoc V. Le. “Rethinking Pre-training and Self-training”, NeurIPS, 2020.


summary_of_the_review: The proposed method in this paper is not novel although creating virtual examples is a reasonable direction for addressing gradual domain adaptation. I suggest the authors can still work on this direction but propose a more novel way to create virtual examples that contain better intermediate information. The experimental results do not support what the paper claims and do not show significant improvement over previous methods. Therefore, I recommend rejecting this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces how to deal with the situation when intermediate distributions are unavailable. The basic idea is to create virtual samples by interpolating source and target representations. The effectiveness of the proposed method is evaluated and results in several interesting conclusions.
main_review: Strength:

This paper is well written and easy to understand. The proposed method is simple but effective. The analyses of how, why, and when iterative self-training is helpful can benefit the community, which is important.

Weakness:
1.	Generating intermediate data in domain adaptation is not a new thing, it is recommended to discuss with related works, such as [1,2].
2.	As shown in Table 3, the proposed method does not bring significant improvement over iterative self-training. It is recommended to perform evaluations on more datasets with natural distribution shifts.

[1] DLOW: Domain Flow for Adaptation and Generalization

[2] Unsupervised Adaptation Across Domain Shifts by Generating Intermediate Data Representations

summary_of_the_review: My major concern is the effectiveness of the proposed method on datasets with natural distribution shifts. Although this method works well on synthetic data, its applicability to real datasets should be further justified and evaluated.
review_title: null

Meta-review: Most reviewers agree that the paper addresses a relevant problem. However, they also  believe that
the paper lacks in several points : not-well supported claim, sometimes clarity, incremental in term of novelty.
Decision: Reject