Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the domain adaptation problem, i.e., transferring a model trained on a source domain to a new target domain for which unlabeled data is available. It does this using self-training (pseudo-labeling, noisy student) where the source-trained model is used to generate labels for the target domain. The paper builds on theoretical insights which suggest self-training works best when the input data shifts gradually from the source to target domain (gradual self-training). This theory suggests that the ideal case would involve walking the shortest path between all the joint data distributions of the intermediate domains.

The authors transform this theoretical insight into a practical algorithm by:

* making a covariate shift assumption which allows them to operate in a latent space, rather than the joint distribution of the data
* generating intermediate domains that lie close to the Wasserstein geodesic between the source and target domain
* adding a regularization term to the optimal transport problem to reduce computational complexity, in combination with thresholding the values in the transport plan to minimize space complexity
strength_and_weaknesses: I like this paper.

The idea seems well-grounded in theory. It seems to me that the authors are minimizing a lower bound though (the right-hand side of equation 6). I am also curious to know what cost function ($c$ in proposition 2). Is it L1 or (squared) L2-distance? Did the authors consider evaluating these different cost functions?

I appreciate the simplicity of the resulting algorithm: Pick a feature space to interpolate in, map both the source and target domain to this space, solve the optimal transport problem, and now just self-train your network as you slowly walk this path. This makes a lot of intuitive sense from the perspective of neural networks learning smooth "manifolds" of data, and GOAT exploits this well.

Experimentally, one concern I have is that GOAT does not seem very effective when there are no intermediate domains available (which seems to me the most common setting in practice). In particular, in table 1 the best approach for going from 0 to 45 degrees rotated MNIST seems to be to not generate any intermediate domains, and the gain for portraits is within margin of error. On the other hand, these tasks seem small and artificial enough (especially rotated MNIST) that I don't want to draw too strong conclusions from it.

Since GOAT does not require any intermediate domains, I would love to see GOAT evaluated using a few more domain adaptation datasets (e.g., the WILDS 2.0 dataset). This would give a much clearer signal as to how this algorithm performs in practice on real-world datasets. Is there a reason the authors did not do this? Does the EMD solver not scale to this size?
clarity,_quality,_novelty_and_reproducibility: I think the paper is well written and clearly presented. As a reader I would have preferred a higher-level, intuitive description of the algorithm before being presented with equations such as 4 and 5 (especially given that many of the details of equations 4 and 5 don't seem necessary to understand the main idea). However, this is partly a matter of taste.

One minor issue: I appreciate the fact that confidence intervals were given for the numbers in table 1 and 2. However, it seems to me that there are several case in which multiple columns should be bolded, because they are within margin of error (e.g., between 74.0±1.3 and 74.2±2.5 it makes little sense to bold the latter as being the "best").

The method seems novel, although I would argue that it seems a relatively straightforward idea given the existing theory and algorithms out there.

The models used are simple and clearly described in the appendix, so I imagine this work should be straightforward to reproduce.
summary_of_the_review: I think this is a good paper. It's main shortcoming is the experimental evaluation, which is limited to very small networks on small and artificial datasets. If such experiments are added, I would be able to support acceptance of this paper more strongly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address the task of gradual domain adaptation (GDA) which aims to address large distribution shifts via adapting models along a sequence of intermediate domains. To overcome domain shifts (especially when the given intermediate domains are missing or scarce), the authors propose a generative gradual domain adaptation with optimal transport (GOAT). Specifically, the authors start by generating intermediate domains along the Wasserstein geodesic, then adapt the source-trained classifier to target by considering the intermediate domains. Experimental results demonstrate the effectiveness of the proposed GOAT. However, I still have concerns about this paper. My detailed comments are as follows. 
strength_and_weaknesses: *Positive points

The task, gradual domain adaptation, which the authors focus on, is truly important and practical in reality since it mitigates the limitation of UDA whenever the distribution shift between the source and the target is large.

The proposed GOAT adapts a progressive manner for shifting between different distributions along Wasserstein geodesic, which is guaranteed by a theory from wang et al[1].

The visualization results in figure.2 are intuitive for the semantic meaning of the generated intermediate domains, which helps readers to understand this paper better.


*Negative points

How to decide the number of generated domains of the proposed method? More discussions are needed.

1. In table 1, it is observed that the performance of GOAT suffers performance degradation with a large # Generated domains (i.e., # Given Domains=4, # Generated domains=4). Besides, when given 3 domains GOAT achieves the best performance with 2 generated domains, why is it the case? Why more intermediate domains would not boost the performance of GOAT? 

2. But in Figure 4 (b), with the increase of generated domains, the test accuracy tends to increase, including Random, Uniform and OT settings. It seems that the contradictions in the experimental results.

It is necessary for the authors to provide more comprehensive and fair results.

1. The experiments are only conducted on two datasets (4 datasets are used in [1], 3 datasets are used in [2]), I would like to see the performance of the proposed GOAT in more datasets for further verification of the effectiveness of GOAT.

2. This paper only considers one baseline (GST, published in 2 years ago) for the comparison experiments. More recent state-of-the-art such as [1, 2] should be considered. 

**Reference

[1] Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. In ICML 2022.

[2] Kumar, A., Ma, T., & Liang, P. (2020, November). Understanding self-training for gradual domain adaptation. In ICML 2020.

clarity,_quality,_novelty_and_reproducibility: 
The Idea of generating intermediate domains, in my opinion, is good but requires more verification by conducting sufficient experiments. 

The proposed method is verified on two datasets and compared with only one existing method, which seems insufficient.

summary_of_the_review: The idea of generating intermediate domains for unsupervised domain adaptation is interesting to me. Moreover, the proposed GOAT is backed by a theory guarantee. However, the datasets and baselines for verifying the GOAT are insufficient for me. Thus, I recommend the weak rejection of this paper. If the authors convince me of the effectiveness of the GOAT, I would be glad to change the reviews.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: When the distribution gap between the source domain and the target domain is very large, some intermediate domains are usually used to make the model gradually adapt to the target domain. The author proposes the GOTA method to generate intermediate domains based on Wasserstein geodesic. Empirically, this GOAT framework can improve the performance of standard GDA(Gradually Domain Adaptation) when the given intermediate domains are scarce, significantly broadening the real-world application scenarios of GDA
strength_and_weaknesses: Strengths:
(1)	This paper gives a clearly theoretical analysis which explanation of the definition and generation method of the intermediate domains

Weaknesses:
(1)	Is there some problem with Equation 10. How does it become the mixup of samples? please give a detailed proof of this problem.
(2)	The experiment is insufficient. The authors mentioned [1] in related work, which also relies on intermediate domain generation, however, is compared in this work. 
(3)	While the generated intermediate domains by the proposed GOAT method is demonstrate in Figure 2 for the rotated MINIST data, other more complicated scenarios are not shown. 
(4)	In Table, given domains=0, GOAT does not bring improvement, why? 
(5)	Some papers have also achieved very good results in constructing intermediate domain at the input level [2,3]. Discussion and comparisons are needed.
[1] Gong R, Li W, Chen Y, et al. Dlow: Domain flow for adaptation and generalization[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 2477-2486.
[2] Na J, Jung H, Chang H J, et al. Fixbi: Bridging domain spaces for unsupervised domain adaptation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 1094-1103.
[3] Na J, Han D, Chang H J, et al. Contrastive Vicinal Space for Unsupervised Domain Adaptation[J]. arXiv preprint arXiv:2111.13353, 2021.

clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and novelty are only marginally significant. Not sure if the paper results can be reproduced.
summary_of_the_review: The generation theory of the intermediate domain is interesting, but there are some issues with the derivation. In addition, the author's experiment is relatively simple and lacks justification for the existing DA datasets
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the setting of unsupervised domain adaptation when the shift between the source and target is large and thus gradual domain adaptation is needed. They propose a gradual domain adaptation algorithm that creates intermediate domains for the gradual adaptation and then applies (standard) gradual self training on these created domains. Authors propose to create domains in extracted feature space (e.g. on the embedding level). They show that it does better than creating domains using the original input space. The idea of creating intermediate domains spans from theoretical bounds: synthetic domains minimize the average Wasserstein distance between the pairs of intermediate domains.  

strength_and_weaknesses: Strength:
- clarity: well written and easy to follow
- good idea inspired by the theoretical bounds - find the optimal intermediate domains
Weaknesses
- Experiments don't compare with UDA baselines
- Experiments are not convincing when there are no intermediate domains provided. May be more intermediate domains needed? 
- The method is expensive
clarity,_quality,_novelty_and_reproducibility: Clarity: excellent. Really well written, provides enough background on all the methods needed
Quality: theoretically justified idea
Novelty: novel way of generating intermediate domains
Reproducibility: authors provide enough of the details to be able to reproduce
summary_of_the_review: I read authors response and am increasing my score to accept
Overall very well written paper with somewhat weak experimental results. I am willing to be convinced otherwise if authors are able to provide additional results (please see below)

Method:
1) Is the number of domains generated a hyperparameter? Why do you go only up to 4 (or rather 4*number of intermediate domains provided)? Can you instead find the optimal value of it using the bounds in (2)? (e.g. min over T (1/T + T/sqrt(n) + 1/sqrt(nT))?
2) How many data points do you geenerate for each intermediate domain - is it the number of non zero elements in optimal transport matrix (so m+n approx, where m and n are source and target number of points?)
3) This method sounds extremely expensive. Apart from OT which you can solve approximately, you train additional T models to convergence. Can you add wall time comparison to Tables 1 and 2, I am trying to understand the bulk part of it (is it 4 times slower for 4 intermediate domains? Or more?)

Experiments:
1)  A bit surprised that only two datasets are used, but seems that a lot of papers on GDA use those 2 datasets. Though this popular paper https://arxiv.org/pdf/2002.11361.pdf also has experiments on Gaussian synthetic data. Do you have experiment results on such data by any chance?
2) Why not to add UDA like DANN or MMD to Table 1 and 2. You seem to say that they don't make sense since the shift is large. Do they underperform the 0-self training entry?
3) is in Table 1 and 2 "0 domains" entry correspond to self training? 
Looking at table 1 it seems that when there are 0 given intermediate domains, you don't imporve over the self training at all apart from one case (portrait on 4). All drastic improvements are only when there are some intermediary domains already provided. Why is this?
4) Also i do find that you report # of generated domains this way confusing - for 3 given domains and 3 generated, you actually generated 9 domains right?

Minor:
- (6) treats T delta as the length, but it was the AVERAGE (so 1/T) in (3). 
- intro: performance degrade => performance degradation
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper  proposes to use a wasserstein distance based method for generating intermediate domains for gradual
domain adaptation (DA). The proposed method is evaluated on a toy(ish) problem (MNIST) and one gradual
domain adaptation problem. From my point of view, the contribution of this paper is essentially
about applying wasserstein interpolating measures equation to gradual domain adaptation 


After discussion, it becomes clear that the theoretical/methodological contribution is a bit weak and the contribution stands mostly in applying well known "interpolating measure" notions in optimal transport to DA. As such, we would have expected the experimental analyses to be stronger with more real-world gradual DA datasets. Hence, at this point, we believe that the paper deserves some further improvements before acceptance.
justification_for_why_not_higher_score: we would have expected
* clarification of the theoretical contributions : it should be made clear that most of the theoretical parts are
well known results in OT and relations with existing works should be discussed and analyzed in details.

* and more experimental results with gradual DA datasets. Since gradual DA is an important problem, we believe
that there are more datasets/problems on which it can applied.
Decision: Reject