Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary
This paper aims to provide a unified principle for multi-source transfer learning under label shifts. Based on this principle, this paper claims that a unified algorithm is proposed for various multi-source label shift transfer scenarios: learning with limited target data, unsupervised domain adaptation and label partial unsupervised domain adaptation. The proposed algorithm is validated on three benchmark datasets. The proof seems correct via combining existing single-domain DA theory and the theory regarding Wasserstein distance. The main theorem (Theorem 1) assumes that we can get the label information in the target domain, which is not realistic in many DA problem settings (e.g., UDA or multi-source UDA in this paper). In many DA problem settings, we have to use pseudo labels to replace with true labels in the target domain, which should be analysed in the proposed theorem. However, this paper does not make any efforts to theoretically analyse the effect of pseudo labels, which results in that this paper has very limited impacts on the DA field. Besides, there are some misleading conclusions in this paper. 

Pros:
1.	The proposed theorems seem correct.
2.	The proposed algorithm has good results after using the label-distribution ratio. The proposed method can address various DA problems and some machine learning problems. 

Cons:
1.	The main theorem (Theorem 1) assumes that we can get the label information in the target domain, which is not realistic in many DA problem settings (e.g., UDA or multi-source UDA in this paper).
2.	In many DA problem settings, we have to use pseudo labels to replace with true labels in the target domain. If the quality of pseudo labels is low, can we still obtain good adaptation results in the target domain using the proposed method? If not, how does the quality of pseudo labels affects the performance of DA methods?
3.	In Theorem 1, this paper claims that Comp() decreases with larger observation numbers. However, it is still unknown if the Rademacher complexity (regarding deep networks) will converge to zero when we have infinite observations. This means that the claim made in the main content is not true, which will mislead broader ICLR readers. This claim is true under a fixed hypothesis with finite VC dimension (as said at the top of Page 20), which is not linked with your deep-network-based algorithm.
4.	The motivation of this paper is unclear. Why should we use theory based on Wasserstein distance? How about directly using pseudo labels (like SHOT in ICML20)? There are many questions regarding the motivation of this paper.
5.	The proposed method can only be used to address three transfer scenarios (presented in this paper)? Are there difficulties to apply the proposed method to address other DA problems? The motivation (testing the proposed method in the presented three problem settings) is unclear. 
6.	The format of this paper is poor, which should be revised before submission.
review_title: The Contribution of this paper is limited.

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors focus on the label shift problem in multi-source transfer learning and derive new generic principles to control the target generalization risk. They propose a framework that unifies the principles of conditional feature alignment, label distribution ratio estimation, and domain relation weights estimation. A WADN algorithm is proposed for 3 multi-source label shift transfer scenarios:  learning with limited target data, unsupervised DA, and label partial unsupervised DA. The proposed WADN algorithm is validated on different scenarios on common benchmark datasets (Digits, HomeOffice, Amazon Review), and results indicate that it can outperform related SOTA methods for these scenarios. 

- Although the paper is well written, and all the key concepts are described in some detail, I found some parts of Section 2 difficult to follow.  The supplementary material in appendices provides much additional information for the reader: proofs, results, etc. In terms of the organization, the authors present their review and analysis of the SOTA literature in Appendix A (no in the main paper). Therefore, it is not immediately clear in the main paper how their framework and algorithm are motivated by challenges in literature.  

- The experimental validation is limited in some respects. The authors do present averaged results over independent replications, using some cross-validation process.  There should be further analysis of the impact on the performance of growing: the number and size of the sources, the degree of shift, and diversity among source and target domains? The experimental section should be expanded to compare results on different backbone networks. Their model should be also compared with SOA methods in terms of time and/or memory complexity.

- Tables 2-7 shows the lower-bound (source or source + tar), but does not show upper-bound results, like when training a DL model on the source and target data that are labeled (in the UDA scenario).  These tables also show DA results with an average value (last column). This is common in DA papers, but I still fail to see the point of averaging across different problems.

- It seems like their code is not made available, so there is a concern that the results in this paper would be very difficult for a reader to reproduce.
review_title: Unified Principles For Multi-Source Transfer Learning Under Label Shifts

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper has made a good attempt to provide a unified approach for unsupervised domain adaptation. The proposed approach is applicable to three scenarios which have been traditionally  treated as three separate problems. The three problems that are treated in a unified way are Unsupervised Domain Adaptation (UDA), limited target labels and partially unsupervised domain adaptation. Another feature of the proposed approach is that it deals with target shift without assuming that conditional distributions are identical, a more realistic assumption for real-world problems.  Results on three different benchmark datasets are provided. Results show uniform improvements in the range of 2-6% over methods compared in various tables. The paper can be improved by providing comparisons with recent UDA and domain generalization methods from Balaji, Sankaranarayanan (CVPR 2018, NIPS 2018) and Balaji and Feizi (ICCV 2019). The ICCV paper also uses the Wasserstein distance for unsupervised domain adaptation. Given that one of the problems that is considered is UDA, I am not sure why the authors have not compared their approach on the Office dataset that is used in UDA papers. A recent paper from Saenko and Trevor Darrell [Saito, K., Kim, D., Sclaroff, S., Darrell, T., & Saenko, K. (2019). Semi-supervised Domain Adaptation via Minimax Entropy. arXiv preprint arXiv:1904.06487] has considered the problem of small source. small target and large unlabeled target data as a domain adaptation problem. Since the authors consider the limited target labels problem as a one of the cases, comparisons with this paper should also be provided.
review_title: The paper has good theoretical result supported by experiments.

final_score: rating: 7: Good paper, accept
review_content: Summary:
The paper is concerned with label shift in multi-source transfer learning setup. In particular authors look into target shift without assuming conditional distributions to be the same in source and target. They propose a unified frameworkthat can be used for learning with no target labels and limited target labels.
They show that the performance on target depends on how well we are able to estimate the ratio of label distrib between source and target, and the gap between real and estimated ratios,. It also depends on the weights assigned to each source tasks. Authors then proceed to show how to learn a model in various settings using the knowledge about the bounds.

Overall, this paper CAN be really impactful, but as of now, it is really hard to follow and understand how it compares to other methods (not just empirically). 
1) The structure of the paper is strange, it is impossible to just read the paper and get enough information on any of the subsections,  everything is in the appendix. Related work is missing (see appendix), the theorem with bounds (that is used through the paper) appears without at least a sketch/idea of how authors got to it and where wasserstein distance comes from. Since it is not intuitive, and since it is not clear how these bounds compare to DANN bounds (https://arxiv.org/pdf/1505.07818.pdf or Ben-David bounds for that matter), it is really hard to believe in the algorithm authors derive, even though it seems to give good empirical improvements
Also K section in appendix is essential for understanding and should be moved to the main paper.
2) Related work absolutely needs to be in the main paper It is impossible to understand what connection your work has with respect to established models like DANN and CDANN. Additionally Wassertein distance was used in a number of papers already,  https://arxiv.org/pdf/2009.02831.pdf, https://arxiv.org/pdf/1707.01217.pdf, https://arxiv.org/pdf/1909.08675.pdf. Again not clear on connection if any and comparison
3) Experiments:
- For DANN, did i understand correctly that you take all source tasks as one "source" domain and all the target as another domain. Why not to consider all the source tasks as separate domains and target as an additional domain, DANN allows for it easily (CE instead of log loss in adv head)
- You should be comparing to CDAN https://arxiv.org/pdf/1705.10667.pdf which has been shown to better than DANN
-  Also didn't understand your setup for experiments. For example on digits, what are your sources and what is your target task? A table shows only target, does it mean that source is all other digits datasets?

Minor:
- in Into, you mention "without assuming that conditional distributions are the same". I think it is most common to assume p(y|x) is the same, not p(x|y) as you show.
- what exactly is "label partial unsupervised domain adaptation"
review_title: Interesting and potentially impactful, but raw/strangely organized/hard to follow paper

Meta-review: 
The question the authors address is relevant and interesting mostly in the UDA setting. However, there exists several recent works that have 
highlighted the importance of label distribution ratio in DA (Wu et al., Combes et al. etc.), hence the main contribution of the
paper is to propose a novel analysis and results in the multi-source setting. That said, the paper has mixed reviews and
after going through the paper, the reviews and the discussion, I tend to agree with some of the reviewers that while 
the idea is interesting, the paper lacks in several points that makes it unsuitable to publication, for now.

Here are the main points leading to the decision.


A) UDA is usually the most frequent situation that occurs in domain adaptation and the most difficult to handle. 
The theoretical novelty of the bound comes only from the multi-source aspect that seems to be original

B) there is a strong contradiction in the paper. In the intro, they state that the paper addresses situations where conditional distributions differ. However in 4.2 they assume that they are finally equal.

In section 4.1, the authors show that for optimizing their problem, they need to have labels, mostly for estimating the class-conditional distributions. When these labels are available in the target domain, the problem is pretty simple and there exists many simple baselines that can handle this problem. However, in a UDA setting, they do not have label and proposes a method for estimation label proportion by assuming S_t(z|y) = T(z|y), which is in contradiction with their initial hypotheses S_t(z|y) != T(z|y). Hence under their assumption, the left hand side of Lemma 1 is zero and the equality is useless.
Hence, I would suggest the authors to avoid such a contradiction.

Under equality of S_t(z|y) = T(z|y), the approach proposed by the authors bears strong similarity with the work of Redko et al 2019 (cited in their paper). So I would highly to recommend them to compare with that algorithm. .


C) the authors use a lot a trick related to filtering, moving average.... I guess those parts is important for making the approach works and they are not properly analyzed.

D) The paper is  confusing in its writing and somehow this confusion makes the theoretical details hard to understand.
For instance, in section 3 the loss function is defined as having two variables but used one line after with only 1. In the theorem, it is not clear whether the true labelling function intervenes or how the y in h(x,y) is related to the true
labels. I guess a clarification is needed here for making the soundness of the theoretical results.
Decision: Reject