Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the multi-source domain adaptation (MSDA) problem. The authors argue that the existing MSDA solutions (1) do not explicitly consider distribution conditioned on labels of each domain, (2) rely on limited feature extraction based on one extractor, (3) do not well explore target data due to the absence of label. Correspondingly, Multi-EPL is proposed based on moment matching.  

Although the design of the proposed method seems reasonable, its novelty is marginal. Additionally, the evaluation in the experiments is somewhat unfair. I vote for rejection.

The paper is well organized. The technical details are clearly presented. A few comments are summarized below.
-In Eq.1, the authors minimize the discrepancy between every two distinct domains. It is unclear to me why not minimize the pairs of each source domain and target domain, (N-pairs). The goal is to align the distributions between source and target. Please clarify the motivation of aligning two source domains. 

Besides, it would be good to have one baseline only considering label-wise moment matching losses for only between N source and 1 target pairs.

-In page 5, the motivation of diversifying features from different extractors is unclear to me. Please clarify the benefit of classifying feature according to extractor ID labels. Moreover, the ablation study presented in section 5.3, does not show a clear improvement by introducing the diversifying loss. I would encourage the authors to design another analytical experiment to show its effectiveness.

-The performance improvement compared to the state of the arts is limited. Specifically, for Digit-Five dataset, a missing recent work [ref-1] reports an average performance of 91.8. To show the consistent performance improvement over this strong baseline, I’d encourage the authors cite and compare it under the same setting.
[ref-1] Hang Wang et al., Learning to Combine: Knowledge Aggregation for Multi-Source Domain Adaptation, ECCV 2020

-Besides [ref-1], there are several other recent MSDA papers are missing, including but not limited to
[ref-2] Chuang Lin et al., Multi-source Domain Adaptation for Visual Sentiment Classification, https://arxiv.org/abs/2001.03886 
[ref-3] Haotian Wang et al., Tmda: Task-specific multi-source domain adaptation via clustering embedded adversarial training. ICDM 2019.

-Other minor point. A typo in page 7, “threshoold” -> “threshold”

Updates: Thanks for the authors' response. Some of my queries (1st and 3rd) were clarified. However, unfortunately, I still think more needs to be done to show the superiority of the results. I retain my original decision.
review_title: Easy to read, but unclear on a few points.

final_score: rating: 4: Ok but not good enough - rejection
review_content: - Summary and contributions
    - In this work, the authors proposed an algorithm for multi-source domain adaptation. While the results seem promising, the technical contribution is incremental and limited. Meanwhile, more empirical results are needed to validate the effectiveness of the framework. 
- Strengths:
    - The paper is well written and easy to follow.
    - The problem investigated in this paper, i.e., multi-source domain adaptation, is of significance.
- Weaknesses:
    - The technical contribution of this work is limited. Label-wise moment matching or adversarial training (e.g., [1]) has been a common practice in single-source domain adaptation. The authors simply applies this idea of multi-mode aware domain adaptation to multi-source domain adaptation. Moreover, comparing the first line, i.e., MULTI-0, in Table 2 with M3SDA in Table 1(a), we find that this label-wise moment-matching makes almost no contribution.
    - The empirical results, especially the ablation studies, do not hold. Pseudo-labeling the unlabeled data in the target domain and using multiple feature extractors can also be easily used by M3SDA and DCTN. Expecting a definite performance boost, I expect the authors to provide such results. Moreover, pseudo-labeling and ensemble learning, however, are not novel; they are widely adopted techniques and can be easily incorporated into any algorithm, e.g., M3SDA, for performance improvement. 
    - Even on the dataset Office-Caltech and Amazon Reviews, the performance improvement of the proposed algorithm is minor. 
		
		
[1] Pei, Z., Cao, Z., Long, M., & Wang, J. (2018). Multi-adversarial domain adaptation. AAAI 2018.
review_title: Limited novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: --Paper summary--

The authors propose a novel method for multi-source domain adaptation (MSDA). For effective adaptation, the proposed method adopts three techniques: (1) label-wise moment matching, (2) pseudo-labeling target data, and (3) ensembling multiple feature extractors. Experimental results show that the proposed method outperforms several state-of-the-art methods in both image and text domains.

--Review summary--

Although the design of the proposed method seems reasonable, its novelty is marginal. Additionally, the evaluation in the experiments is somewhat unfair. I vote for rejection.

--Details--

Strength

- This paper is well-organized and is easy to follow. I believe that the proposed method can be easily implemented without any obstacles.
- Good performance in both image and text domains is appealing. Such results should be highly appreciated especially in machine learning community.

Weakness and concerns

- Marginal novelty. The three techniques that the proposed method adopted are all similar to those already proposed in the literature. I could not find any novel and specific design or strategy to combine them specialized for MSDA.
	- Class-wise distributional alignment is a common technique in recent domain adaptation methods, e.g. [R1] and [R2].   
[R1] "A DIRT-T Approach to Unsupervised Domain Adaptation," ICLR 2018.  
[R2] "Unsupervised Domain Adaptation via Regularized Conditional Alignment," ICCV 2019.
	- Pseudo-labeling is also a common technique in recent domain adaptation methods, e.g. [R1] and [R3].  
[R3] "Asymmetric Tri-training for Unsupervised Domain Adaptation," ICML 2017.
	- Using multiple feature representations is not so common but is presented in [R4] and [R5].  
[R4] "Domain Adaptation with Ensemble of Feature Groups," IJCAI 2011.  
[R5] "Domain Separation Networks," NeurIPS 2016.
- The design of the feature diversifying loss is not reasonable. It can be minimized by just making feature representations to be easy to discrminate their extractors, which does not necessarily increase the diversity of the representations. For example, given two extractors that share the same parameters, adding a large offset to outputs from one extractor leads to high performance of the extractor classifier but does not increase diversity of the feature representations. 
- The exprimental setting is somewhat unfair. Since the proposed method utilizes n feature extractors, the model complexity in the proposed method should be n times larger than that in existing methods. 

review_title: Although the design of the proposed method seems reasonable, its novelty is marginal.

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors propose a MULTI-EPL for multi-source domain adaptation. The key idea includes two folds: (1) to align label-wise moment, and (2) to ensemble multiple feature extractor. Experimental studies on 3 datasets are done to verify the proposed MUTL-EPL.

Overall, the paper is well-written. The technical approach is simple and sound. My major concern is on the technical significance of the method. Here are the detailed comments:

(1)	One motivation of the paper is that current methods fail to consider the shifts among sources. However, there are some multiple source transfer methods explicitly modelling the inter-domain similarities, e.g., [ref1].  The paper also misses some important multiple-source references. Please refer to the survey [ref2] for different types of multiple source transfer methods. It would be better to have a comprehensive discussion on the related works.

(2)	The proposed label-wise moment matching is not new in transfer. Early subspace based work, e.g., JDA [ref3], and latest semantic deep learning based transfer methods, e.g., [ref4] share the similar idea. 

(3)	The threshold \taw is used to obtain the good target labels.  On the one hand, it is unclear how this parameter should be set for different transfer tasks. On the other hand, high confidence score does not imply correct target label prediction. Error reinforcement may happen even with a well-tuned \taw.

(4)	The usage of ensemble feature extractor is actually using high complexity to enhance prediction accuracy. The scalability could be an issue of the proposed method, especially considering that multiple source may have extremely large data size. 

(5)	Sensitivity analyses on the balancing parameters \alpha and \beta should be done. 

(6)	Why data augmentation is done on office-caltech10 datasets? There are many datasets containing multiple domains with sufficient data, e.g., office-home, Please use these datasets instead of constructing `artificial’ real-world dataset. 

(7)	The baseline methods can be further improved (please consider [ref2] for more baselines). Regarding the current results, MUTLT-EPL performs larger improvements over M3SDA on the 2nd, 3rd, and 4th tasks in Digit –five dataset, while only achieves marginal improvements in other tasks, e.g.. tasks in office-caltech10 dataset. More analyses on the performance difference of different tasks should be discussed. 

(8)	Based on the ablation study, Mutil-epl-r achieves comparable results with multi-epl, which indicates that the extractor classifier and feature diversifying loss have less importance in the overall objective. 

[ref1] Source-target similarity modelings for multi-source transfer gaussian process regression

[ref2] Multi-source Domain Adaptation in the Deep Learning Era: A Systematic Survey

[ref3] Transfer Feature Learning with Joint Distribution Adaptation

[ref4] Deep Transfer Learning with Joint Adaptation Networks

Update: Thanks for the authors' response. However, I am not convinced on several points, e.g., (3) - (7). Considering the other reviewers' comments, I think the paper needs to be further improved. Thus, I will keep my score.
review_title: Official Blind Review #1

Meta-review: In this paper, the authors proposed a solution to the problem of multi-source domain adaptation. All the reviewers have two concerns: 1) the technical contribution/novelty is limited, and 2) the experimental results are not convincing. Therefore, this paper does not meet the standard of being published in ICLR. The authors are encouraged to improve this work by addressing the issues raised by the reviewers. 
Decision: Reject