Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper propose, IDOL, a method for gradual domain adaptation. The novelty of IDOL is that it can learn to discover the sequence of intermediate domains.
main_review: Originality: 
---
This paper touches a novel problem, that is doing gradual domain adaptation with unlabeled data without pre-defined intermediate domains. The solution is technically novel. Specifically, the author leverages the idea of cycle-consistency and meta-learning to learn the domain index. I find this methodology is original and may benefit to other tasks. 

Quality & Clarity: 
---
For the method section, in general, the authors do a good job. However I feel there is some details about the specific design missing. I understand, at high level, the author is using a meta-learning style approach. From my understanding, meta-learning has a large computation overhead. I am curious about the algorithm complexity of the proposed method as well as experimental comparison of the learning efficiency between the fine stage algorithm against other coarse stage baselines. In appendix algorithm 3, the author uses mini-batch to update the \theta(q) for T times. I am concern the efficiency of the meta-learning when T goes large.
Another place I feel not well explained is why set q = max{q,0} instead of q = clip(q,min=0,max=1). The author says during the optimization they relax the binary vector q to be real value. I am not sure whether making a bounded value q to be unbounded will causes instability during the optimization. Can author give any theoretical verification about such relaxation?

For the experiment section, the paper delivers nice results. Table 1 shows the proposed coarse-steps are better than random indexing, while with refinement, IDOL can reach similar performance as the pre-defined index. I like the analysis in section 4.2. The visualization of per-step accuracy, relationships between learned and pre-defined indices, and class imbalance analysis provide a nice understanding of the IDOL. That being said, I still have a few question:
1. In figure 4, the author shows the target accuracy stably improves after each adaptation using IDOL sequences. Figure 4(b) shows IDOL sequences do not have much correlation with years. I am curious about what information does IDOL sequences captures. 
2. The IDOL seems have worse performance than pre-defined (rotation) index in the Rotated MNIST dataset. I am assuming in this task, rotation is the best domain index. Can authors analysis how close the learned IDOL align with the rotation? If IDOL is not perfectly align with rotations, is there any potential reasons that hinders it learn the best domain index?

Significance:
---
The paper propose a new problem of doing GDA with undefined intermediate domains. The problem is important. The result presented is significant and could be a nice baseline for the future research. 

In summary, I am voting for the acceptance due to its novelty and nice experiments. My score is not final and will depend on how the author address my questions. 
limitations_and_societal_impact: The author gives a brief discussion of the limitations of this work in supplementary. Maybe the author can talk about something in more detail like when graduate domain adaptation is not applicable or when learning domain indexing is really hard. For example, in health care applications, some domain changes are caused by binary (categorical) attributes such as races, genders. Can we still apply gradual DA?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper is concerned with gradual domain adaptation, when we have access to data from intermediate distributions between source and target and use them to gradually adapt the model through self-training. To address the challenge of lack of intermediate data annotated and grouped based on their distance to the source domain, the paper proposes IDOL (Intermediate Domain Labeler) to index unlabeled available data that presumably covers the gap between source and target  domains based on their distance to source.

IDOL consists of two steps:
1. A progressively trained domain discriminator is used to assign a score to each example. (a higher/lower score indicates that the example is closer to the source/target domain).
2. A model based on a cycle-consistency loss is applied to refine the coarse scores progressively. The main idea is to group examples into intermediate domains such that enough discriminative information is preserved that we could approximately regain the accuracy on the source domain by consuming the intermediate domains in the reverse direction.. 

**Empirical results**:
Experiments on  Rotated MNIST and Portraits indicate that with IDOL we can achieve comparable performance on gradual domain adaptation as when we have the ground truth (pre-defined) domain sequences. Furthermore the experiments indicate that applying IDOL on pre-defined domain sequences, treating them as the outputs from step 1, and refining them through step 2 can lead to domain sequences that are better suited for gradual domain adaptation. In general, it seems that applying IDOL in many cases with partial or noisy annotation of the intermediated domain sequences can be beneficial.  Finally, IDOL is also compared with UDA on CIFAR10 to STL tasks, indicating superiority of both (GDA + confidence) and (GDA + IDOL) to UDA in this setting.

main_review: **Originality**: The proposed idea builds up on top of  existing works on gradual domain adaptation that study how leveraging samples from intermediate distributions can help in the unsupervised domain adaptation process. The idea is novel and theoretically motivated. It is a two step process of forming a sequence of domains given a set of unlabeled data points.  

**Quality and Clarity**: I find the paper well written and very readable. The experiments and analysis provided in the paper validate the success of the IDOL in labeling intermediate sequence both qualitatively and quantitatively, but in a limited scope.

**Significance**: Based on the empirical results provided in the paper, the method seems to lead to significant improvements over the baselines it is compared with. However, both the baselines and the benchmarks are not selected generously. Even though the experiments are limited to a few small datasets, I can understand if this is mainly due to lack of benchmarks that would be applicable for the purpose of this study. There might be some other applications for the proposed approach to make sequences of domains  from a set of unlabeled examples. E.g., for analytical studies.

**Question for the authors:**
- Have you looked into the variations in the domain sequences formed in different trials? and maybe different factors that would affect them? e.g. number of intermediate samples? nature of the shift in distribution? (e.g., comparing rotated MNIST with the Face dataset).

**Main weakness(es)**:
- Limited scope of the experiments (both in terms of baselines and datasets).

limitations_and_societal_impact: I don't think the authors have adequately addressed the limitations of the proposed method (or about negative societal impact ).
I think there might be some room for discussion there. E.g., could this gradual process results in some unintentional biases in the models?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper is addressing the gradual domain adaptation (GDA) problem without the access of intermediate domain indices. This paper proposes a coarse-to-fine method to predict the indices of the intermediate domain sequence. The results are shown to be comparative to pre-defined methods. 
main_review: 1. claim concerns:
a) One of the main concerns is the definition of “intermediate domain” in this paper. It highly depends on how the intermediate domains are generated. This paper only briefly mentions that the assumption is that the intermediate domains are under the data distribution between source and target domains. It would be great to have more analysis about: 1) What if the distributions of intermediate domains are not gradually/smoothly underlying between source and target domains, but closer to on side instead? 2) What if the intermediate domains are under very different distributions? With the above analyses, it will be easier to clarify in what situation the proposed method works.

2. technical detail concerns:
a) The descriptions of “progress training for the domain discriminator” are not clear (lines 169-180). It would be clearer if there are pseudo codes or algorithms like the one in Sec. 3.4 (line 220-225).

3. experiment concerns:
a) In Table 1, does “Refined?” mean the fine stage as described in Sec. 3.4? If so, why does this refinement also work for the “pre-defined” case? It would be great to analyze the pre-defined case with and without refinement (i.e., visualized Table 1 using a similar way as Figure 4).
b) Sec. 4.5 partially answers the question about “What if the intermediate domains are under very different distributions?”. From Table 2, it shows that the proposed method does not work well for this case. It would be better to have more analysis and explanation here. 
Moreover, the result of “UDA (T)” is weird. The numbers in Table 2 are not consistent with those in RCA [4], which shows a 81.6 accuracy as “UDA (T)” and also outperforms “Source only”.

4. other concerns:
a) The proposed method is like an additional step to generate predicted domain indices, so it would be better to have time analysis to show the computation cost. The paper only briefly mentions that using short descriptions. It would be better to have more details (e.g., a Table with numbers).

limitations_and_societal_impact: Yes, but it would be better to discuss more limitations as mentioned above.
review_title: null

Meta-review: The paper considers gradual domain adaptation without access to intermediate sequence of distributions between source and target. This is a relatively new problem and a very important one. Author discussions helped address the reviewer concerns and therefore I suggest the paper to be accepted. I ask the authors to please include the additional information discussed in the rebuttal period in the camera ready.  I also strongly recommend the authors add more explanations about the limitations of the current experiment setting (GDA) and discuss the possible ways to actively address these issues. Although the paper is interesting and the results are a good first step to tackle the problem of gradual domain adaptation, the experiment are limited, limitations of the proposed method are not investigated and the baselines are simple. Therefore, the paper does not qualify for a spotlight.
Decision: Accept (Poster)