Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Quality:
The paper appears to be correct.

Clarity:
The paper is very clear

Originality:
The theoretical contribution extends the seminal work of Ben-David et al., the idea of using adversarial learning is not new, the novelty is mediaum

Significance:
The theoretical analysis is interested but for me limited, the idea of the algorithm is not new but as far as I know the first explicitly presented for multi-source. 

Pros:
-new theoretical analysis for multisource problem
-paper clear
-smoothed version is interesting
Cons
-Learning bounds with worst case standpoint is probably not the best analysis for multisource learning
-experimental evaluation limited in the sense that similar algorithms in the literature are not compared
-Extension a bit direct from the seminal work of Ben-David et al.


Summary:
This paper presents a multiple source domain adaptation approach based on adversarial learning.
The setting considered contains multiple source domains with labeled instances and one target domain with unlabeled instances. The authors propose learning bounds in this context that extend seminal work of Ben-David and co-authors(2007,2010) where they essentially consider the max source error and the max divergence between target and source with the presence of empirical estimate.
Then, they propose an adversarial algorithm to optimize this bound, with another version optimizing a smoothed version, following the approach of Ganin et al.(2016). 
An experimental evaluation on 3 known tasks is presented.

Comments:
Comments:

-I am not particularly convinced that the proposed theory explains best multi-source learning. In multi-source, you expect that one source may compensate the others when needed for classification of particular instances. The paper considers a kind of worst case by taking the max error over the sources and the max divergence between target and source, but not really representative of what happens for real problems in the sense that you do not take into account how the different sources interact.
The experimental results confirm this aspect actually.
Maybe, the authors could propose a learning bound that correspond to the smoothed version proposed in the paper and that works best.

The Hard version of the algorithm seems here to comply with the bound, while the algorithm that is really interesting is the smoothed version.

-Experimental evaluation is a bit limited, there is no comparison with other (deep learning methods) tackling multi-source scenarios (or equivalent), while I think it is easy to find related approaches :
-E. Tzeng, J. Hoffman, T. Darrell, K. Saenko. Simultaneous Deep Transfer Across Domains and Tasks. ICCV 2015.
-I-H Jhuo, D Liu, D.T. Lee, and S-Fu. Chang. Robust visual domain adaptation with low-rank reconstruction. In IEEE CVPR, 2012.
-Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In IEEE International Conference on Computer Vision (ICCV), 2015.
-Chuang Gan, Tianbao Yang, and Boqing Gong. Learning attributes equals multi-source domain generalization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
-R. Gopalan,R. Li,and R. Chellappa. Unsupervised Adaptation Across Domain shifts by generating intermediate data representations. PAMI, 36(11), 2014.

Note also this paper at CVPR'17: about Domain adversarial adaptation.
E. Tzeng, J. Hoffman, K. Saenko, T. Darrell.  Adversarial Discriminative Domain Adaptation, CVPR 2017.


-Nothing is said about the complexity of applying the algorithm on the different datasets (convergence, tuning, ...)
For the smoothed version, it could be interesting to see if the weights w_i associated to each source are related to each (original) source error and see how the sources are complementary. 

--
After rebuttal
--
The new results and experimental evaluation have improved the paper. I increased my score.
review_title: The theoretical analysis is new and interesting but for me its worst case aspect does not reflect real application of multisource learning and the experimental results are good but lack of comparisons with other multi-source approaches.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper builds on the previous work of Ganin et al. (2015, 2016), that introduced a domain adversarial neural network (DANN) for single source domain adaptation. Whereas Ganin et al. (2016) were building directly on the (single source) domain adaptation theorem of Ben-David et al., the authors prove a similar result for the multiple sources case. 

This new result appears to be a simple extension of the single source theorem. A similar result to Theorem 3.1 can be obtained by considering the maximum over the k bounds obtained by considering the k pairs source-target one by one, using Theorem 2.1 of Blitzer et al. (2008). In fact, the latter bound might even be tighter, as Theorem 3.1 considers the maximum over the three components of the domain adaptation bound separately (the source error, the  discrepancy and the lambda term). The same observation holds for Theorem 3.4, which is very similar to Theorem 1 of Blitzer et al. (2008).  This made me doubt that the derived theorem is studying multi-source domain adaptation in an optimal way. 
That being said, the authors show in their experiments that their multiple sources network (named MDAN), which is based on their theoretical study, generally achieves better accuracy than the best single source DANN algorithm. This succeeds in convincing me that the proposed approach is of interest. At least, these empirical results could be used as non-trivial benchmarks for further development. 

Note that the fact that the "smoothed version" of MDAN performs better than the "hard version", while the latter is directly backed by the theory, also suggests that something is not captured by the theorem. The authors suggest that it can be a question of "data-efficiency performance": "We argue that with more training iterations, the performance of Hard-Max can be further improved" (page 8). This appears to me to be the weakest claim of the paper, since it is not backed by an empirical or a theoretical study. 

Pros:
- Tackle an important problem that is not studied as it deserves.
- Based on a theoretical study of the multi-source domain adaptation problem.
- The empirical study is exhaustive enough to show that the proposed algorithm actually works.
- May be used as a benchmark for further multi-source domain adaptation research.

Cons:
- The soft-max version of the algorithm - obtaining the best empirical study - is not backed by the theory.
- It is not obvious that the theoretical study and the proposed algorithm is actually the right thing to do.

Minor comment:
- Section 5: It seems that the benchmarks named "sDANN" and "cDANN" in 5.1 are the same as "best-Single-DANN" and "Combine-DANN" in 5.2. If I am right, the nomenclature must be uniformized. 

review_title: A modest but honest extension of single source adversarial network to multi-source domain adaptation

final_score: rating: 6: Marginally above acceptance threshold
review_content: The generalization bounds proposed in this paper is an extension of Blitzer et al. 2007. The previous bounds was proposed for single domain single target setting, and this paper extends it to multiple source domain setting. 

The proposed bound is presented in Theorem 3.4, showing some interesting observations, such as the performance on the target domain depends on the worst empirical error among multiple source domains.  The proposed bound reduces to Blitzer et al. 2007’s when there is only single source domain. 

Pros 
+ The proposed bound is of some interest.
+ The bound leads to an efficient learning strategy using adversarial neural networks.

Cons:
- My major concern is that the baselines evaluated in the experiments are quite limited. There are other publications working on the multi-source-domain setting, which were not mentioned/compared in the submission.

review_title: A new generalization bound and an extension of using adversarial learning for multiple source domain adaptation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work presents a bound to learn from multiple source domains for domain adaptation using adversarial learning. This is a simple extension to the previous work based on a single source domain. The adversarial learning aspect is not new.

The proposed method (MDAN) was evaluated on 3 known data sets. Overall, the improvements from using MDAN were consistent and promising.

The bound used in the paper accounts for the worst case scenario, which may not be a tight bound when some of the source domains are very different from the target domain. Therefore, it does not completely address the problem of learning from multiple source domains. The fact that soft-max performs better than hard-max suggest that some form of domain selection or weighting might lead to a better solution. The empirical results in the third experiment (Table 4) also suggest that the proposed solution does not generalize well to domains that are less similar to the target domain.

Some minor comments:
- Section 2: "h disagrees with h" -> "h disagrees with f".
- Theorem 3.1: move the \lambda term to the end to be consistent with equation 1.
- Last line of Section 3: "losses functions" -> "loss functions".
- Table 1 and 2: shorthands sDANN, cDANN, H-Max and S-Max are used here are not consistent with those used in subsequence experiments.  It's good to be consistent.
- In Section 5.2, it was conjectured that the poorer performance of MDAN on SVHN is due to its dissimilarity to the other domains. However, given that the best-single results are close to the target only results, SVHN should be similar to one or more of the source domains. MDAN is probably hurt by the worst case bound.
- In Table 4, the DANN performance for S=6 and T=A is off compared to the rest. Any idea?


review_title: An extension of domain adaptation using adversarial learning to multiple source domains.

Meta-review: Pros
-- Lays out bounds for multi-domain adaptation based on earlier work on a single source-target domain pair.
-- Shows gains over choosing the best source domain for a target domain, or naively combining domains.

Cons
-- The reviewers agree that the extensions are relatively straightforward extensions to single source-target pair.
-- Hard-max doesn’t consider the partial contribution of multiple source domains, and considers the worst-case scenario.
-- Soft-max addresses some of these issues; the authors provide reasonable justification for the algorithm but it’s not clear that the specific choice of \alphas leads to the tightest bound.

The reviewers noted that the authors significantly improved the paper during the revision process. The AC feels that the presented techniques would be of interest to the community and would help lead discussions towards theoretically optimal ways to do domain adaptation given multiple domains. The authors are therefore encouraged to submit to the workshop track.

Decision: Invite to Workshop Track