Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposed an aggregation algorithm (DARN) for the multi-source domain adaptation problem which is highly useful in real-world applications.
The proposed method is based on the theoretical extension of the single-source domain discrepancy measure proposed by Mansour et al. 2009 to the multi-source setting.
This paper also showed the effectiveness of the proposed method on some real-world datasets.

Strengths
The paper introduces new technical insights to understand their bound, e.g. effective sample size.
The paper proposed the way to estimate coefficient, optimal \alpha, with theoretical justification, and I think this is the biggest contribution of this paper and is interesting.
The proposed method is also able to be used in the regression task since it is based on the disc which can be estimated in the regression task.

Weakness
The main theorem of this paper is an extension of existing methods, so the novelty of theoretical analysis is somewhat limited.
A naive approach to estimate coefficient with single-source domain discrepancy measures such as [1]Mansour (2009), [2,3] Ben-David(2007, 2010), [4] Kuroki et al (2019), and W1-distance is not considered.
Experimental results itself are fine but not complete.
  - Although disc can be easily estimated in the regression task (differently from d_A distance which is a special case of disc), there are no experimental results of the regression task even in the synthetic data.
  - It would be also better to show the coefficient of existing methods that have no theoretical justification.
  - It would be better to compare with a naive approach that uses domain discrepancy between each source and target as (fixed) coefficient since this approach such as Mansour (2009), Ben-David(2007, 2010) and Kuroki et al (2019) which explicitly consider the hypothesis class has theoretical justification in the form of generalization error bound in the target domain.

Overall, I like the approach of the proposed method, especially tuning coefficient during training procedure although novelty in the theoretical analysis is somewhat limited.
So this work has to be supported with more detailed experimental results to express the potential of this approach fully.
For this reason, I think it is okay but not good enough at this time.

****After the authors' response****
Increase rating.

[1] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and
algorithms. In COLT, 2009.
[2] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for
domain adaptation. In NeurIPS, 2007.
[3] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman
Vaughan. A theory of learning from different domains. Machine Learning, 2010.
[4] Seiichi Kuroki, Nontawat Charoenphakdee, Han Bao, Junya Honda, Issei Sato, and Masashi Sugiyama.
Unsupervised domain adaptation based on source-guided discrepancy. In AAAI, 2019.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper studies multi-source domain adaptation problem. First this paper proposes a new theory for this domain that extends generalized discrepancy theory to multi-source setting. After derive a new generalization bound, this paper also proposes a new method based on the theory. Evaluation on real world datasets are proposed to show the efficiency of the proposed method.

+ The theory in this paper improve bounds for multi-source DA in previous paper. The new bound provides new insight and helps the design of algorithm.
+ This paper proposes elegant method to tackle new terms in the loss function and gives its complexity analysis. 
+ The evaluation results show that the algorithm is efficient.

Comments:
- The main contribution of the proposed theory is the alpha term. Is \eta_{\mathcal{H}} is a better estimation that previous adaptability term in multi-source DA? What is the role of \eta_{\mathcal{H}} in the algorithm design? How to control it in empirical algorithm?
- The evaluation of the proposed method is not complete. Some baseline DA methods [A, B] and datasets [C, D] are not considered. 

[A] S. Sankaranarayanan, Y. Balaji, C. D. Castillo, and R. Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. 
[B] Saito, Kuniaki, et al. "Maximum classifier discrepancy for unsupervised domain adaptation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[C] K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision (ECCV), 2010. 
[D]H.Venkateswara, J.Eusebio, S.Chakraborty, and S.Panchanathan. Deep hashing network for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 

###Summary###
This paper tackles the multi-source domain adaptation by aggregate multiple source domains dynamically during the training phase. The observation is that in many real-world applications, we want to exploit multiple source datasets of similar tasks to learn a model for a different but related target datasets.

Firstly, the paper derives a multiple-source domain adaptation upper-bound from single-to-single domain adaptation generalization bound, based on the theoretical work from Cortes et al (2019). The idea is similar to Zhao et al (2019), which introduces a weighted parameter \alpha to combine the source domains together. 

Secondly, based on the theoretical result, the paper proposes an algorithm to minimize the upper bound of the theoretical result. The upper bound can be simplified as the quartic form (Eq. 4) and can be optimized with the Lagrangian form. Since no closed-form expression for the optimal v can be derived, the authors propose to use binary search to find it. 

Based on the theoretical results and the algorithm, the paper introduces Domain AggRegation Network (DARN), which contains a base network for feature extraction, h_y to minimize the task loss and h_d to evaluate the discrepancy between each source domain and target domain.  The loss is aggregation with the parameter \alpha.

Finally, the paper conduct experiments on sentimental analysis benchmark, Amazon Review and digit datasets. The paper selects MDAN, DANN, MDMN as the baselines. On the amazon review dataset, the performance of the proposed DARN model is comparable with the MDMN baseline. On the digit dataset, the model can outperform the baselines. 


### Novelty ###

The theoretical results in this paper are extended from Cortes et al (2019) and Zhao et al (2018).  Thus, the theoretical contribution of this paper is limited. 

The algorithm proposed in this paper is interesting. However, the motivation of the proposed method is to minimize the upper bound, not the loss itself, i.e. L_T(h, f_T). Intuitively, when the upper bound of the loss is minimized, it will be beneficial to minimize the loss itself. But it's not guaranteed as the upper bound contains other variables, such as the number of training samples and model complexity. If the training samples and model complexity (think about the parameters in the deep models) are significantly large, the upper bound of the loss might be also very large. 

As for the experimental results, the paper only provides results on the sentimental analysis results and digit datasets, which are small benchmarks. The selected baselines are not sufficient. The improvement from the baselines is also limited. 



###Clarity###

Overall, the paper is well organized and logically clear. The images are well-presented and well-explained by the captions and the text. 

The derivation of the algorithm in Sec 3.2 is logically clear and easy to follow. 

###Pros###

1) The paper proposes a new theoretical upper-bound based on the prior works, the upper-bound and its derivation are interesting and heuristic to the domain adaptation research community. 
2) The paper is applicable to many practical scenarios since the data from the real-world application is typically collected from multiple sources.
3) The paper is overall well-organized and well-written. The claims of the paper are verified by the experimental results.

###Cons###

1) The critical issue of this paper is that the algorithm is designed to minimize the upper bound. The idea is intuitive when the upper bound is small. However, the proposed upper bound in the paper involves other parameters, such as the model complexity and the number of training samples. 
It's an intuitive idea to weight different source domains in multi-source domain adaptation. The paper derives the weight by the Lagrangian form to minimize the upper bound. While another trivial trick is to evaluate \alpha by the domain closeness between each source domain with the target domain. 
2) The experimental results provided in this paper are weak. In the abstract and introduction,  the paper motivates the multi-source domain adaptation (MSDA) problem by arguing that the MSDA has a lot of real applications.  But the paper only provides empirical results on sentimental analysis and digit recognition.  Besides, the results on the sentimental analysis are comparable with the compared baselines. 

It will be also interesting to see how does the proposed method perform on large-scale datasets such as DomainNet and Office-Home dataset:
DomainNet: Moment Matching for Multi-Source Domain Adaptation, ICCV 2019. http://ai.bu.edu/DomainNet/
Office-Home: Deep Hashing Network for Unsupervised Domain Adaptation, CVPR 2017. http://hemanthdv.org/OfficeHome-Dataset/

3) The novelty of this paper is incremental as the theoretical results are extended from Cortes et al (2019) and Zhao et al (2018). 

Based on the summary, cons, and pros, the current rating I am giving now is weak reject. I would like to discuss the final rating with other reviewers, ACs.
To improve the rating, the author should explain the following questions:
1). Why minimizing the upper bound in this scenario would help to minimize the loss on the target domain, when the upper bound can be substantially huge? How about just evaluate \alpha by the closeness of the source domain with the target domain?
2). In the introduction, the paper motivates the multi-source domain adaptation (MSDA) problem by arguing that the MSDA has a lot of real applications. While the experiments are only performed on sentimental analysis and digit recognition. How about evaluating the proposed methods on real image recognition such as DomainNet or Office-Home? 


review_title: Official Blind Review #1

Meta-review: Thanks for the detailed replies to the reviewers.
Their score was slightly improved, this paper is still below the bar given high competition of ICLR2020.
For this reason, we decided not to accept this paper.
Decision: Reject