Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: - Just to make sure, in Section 4.3:
(a) the teacher-only baseline is the combination of 3 teachers (one for each of the left-out domains), with a winner-takes-all strategy based on Jensen-Shannon. Am I correct?
(b) the source-domain specific teacher strategy implements Multiple Teacher Student model of Section 3.3?
 

- How does the performance of TS-MCD is impacted if n random points are chosen instead of the top n according to the teacher confidence?
review_title: Questions

final_score: rating: 7: Good paper, accept
review_content: This paper studies the problem of transfer learning in the context of domain adaptation. They propose to study it in the framework of knowledge distillation. Several settings are presented along with experiments on the Amazon Reviews dataset.

The paper is nicely written and the problem studied is very important towards progress in AI. The results of the experiments could be improved but still justify the validity of applying distillation for transfer learning. Of course, the experimental setting is rather limited but the benchmarks are competitive enough to be meaningful.

I had concerns regarding discussion of previous work but the extensive responses helped clarify this point (the authors should turn the arguments used in this thread into an appendix).

I think this paper would make an interesting ICLR paper.


review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Paper describes technique for leveraging multiple teachers in teacher-student framework and performing unsupervised and semi-supervised domain adaptation. The central idea relies on using similarity measure between source and target domains to weight the corresponding trustfulness of particular teachers, when using their prediction as soft targets in student training. Authors provide an experimental validation on a single benchmark corpora for sentiment analysis.

What exactly constitutes the learned representation h used in MCD measure? I assume those are the top level pre-softmax activations - is this the case? Those tend to be typically more task related, would not the intermediate ones work better?

One not entirely clear aspect concerns types of distributions applicable to proposed learning - it assumes the vocabulary (or decision space) between tasks spans same categories, as otherwise one cannot derive the KL based objective, often used in TS framework. As such, approach is rater constrained in scope.
 
Authors shall refer in the related-work to similar ideas proposed in the related field of acoustic modelling (and adaptation of acoustic models), in particular, works of Yu et al. [1] and the follow up work of Li et al. [2] which to an extent are a prior the work on knowledge distillation. Reasonably related is also work on deep relationship networks [3], where MT generative approach is proposed to avoid negative transfers, something of central role in this paper.

Minors: 
The student S similarly has an output probability -> models an output probability

[1] KL-Divergence Regularized Deep Neural Network Adaptation For Improved Large Vocabulary Speech Recognition, Dong Yu, Kaisheng Yao, Hang Su, Gang Li, Frank Seide
[2] Learning Small-Size DNN with Output-Distribution-Based Criteria, Jinyu Li, Rui Zhao, Jui-Ting Huang and Yifan Gong
[3] Learning Multiple Tasks with Deep Relationship Networks, Mingsheng Long, Jianmin Wang


review_title: review

final_score: rating: null
review_content: Could the authors provide more insights on the three measurements used to determine domain similarity? Another commonly used metric is the A-distance introduced in Ben-David, Shai, et al. 2007. Have the authors tried this method?
review_title: Measuring domain similarity

final_score: rating: 5: Marginally below acceptance threshold
review_content: The work extends knowledge distillation to domain adaptation scenario, the student model (for the target domain) is learned to mimic the prediction of the teacher model, learned on the source domain. The authors extends the idea to multi-source domain settings, proposing to weight predictions of teacher model using several domain similarity measurements. To improve the performance of proposed method when only a single source domain is available, the authors propose to use maximum cluster difference to inject pseudo-supervised examples labeled by the teacher model to train the student model. 

The paper is well written and easy to follow. The idea is straight-forward, albeit fairly heuristic in several cases. It is not clear what is the advantage of the proposed method vs existing feature learning techniques for domain adaptation, which also does not require re-train source  models, and performs comparable to the proposed method. 
Questions: 
1. Why did you choose to use different combination schemes in equation (3) and (5)? For example, in equation (5), what if minimizing H( (1-\lambda) y_teacher + \lambda P_t, P_s) instead? 
2. how will you extend the MCD definition to multi-class settings?  
review_title: review

Meta-review: The paper introduces a reasonable but somewhat heurstic and straightforward approach to domain adaptation. Especially, since the approach is not so principled, it does not seem sufficient to evaluate it on a single benchmark (document classification, specifically sentiment polarity prediction). 
 
 + the results on the sentiment dataset are strong
 + the paper is easy to follow
 
 - relatively straightforward
 - the novel aspects are a bit heuristic
 - extra evaluation is needed
Decision: Reject