Title: Knowledge Adaptation: Teaching to Adapt
Authors: Sebastian Ruder
Parsa Ghaffari
John G. Breslin
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning
Transfer Learning
Unsupervised Learning

Abstract: Domain adaptation is crucial in many real-world applications where the distribution of the training data differs from the distribution of the test data. Previous Deep Learning-based approaches to domain adaptation need to be trained jointly on source and target domain data and are therefore unappealing in scenarios where models need to be adapted to a large number of domains or where a domain is evolving, e.g. spam detection where attackers continuously change their tactics.

To fill this gap, we propose Knowledge Adaptation, an extension of Knowledge Distillation (Bucilua et al., 2006; Hinton et al., 2015) to the domain adaptation scenario. We show how a student model achieves state-of-the-art results on unsupervised domain adaptation from multiple sources on a standard sentiment analysis benchmark by taking into account the domain-specific expertise of multiple teachers and the similarities between their domains.

When learning from a single teacher, using domain similarity to gauge trustworthiness is inadequate. To this end, we propose a simple metric that correlates well with the teacher's accuracy in the target domain. We demonstrate that incorporating high-confidence examples selected by this metric enables the student model to achieve state-of-the-art performance in the single-source scenario.

TL;DR: We propose a teacher-student framework for domain adaptation together with a novel confidence measure that achieves state-of-the-art results on single-source and multi-source adaptation on a standard sentiment analysis benchmark.

Review discussion:
- Just to make sure, in Section 4.3:
(a) the teacher-only baseline is the combination of 3 teachers (one for each of the left-out domains), with a winner-takes-all strategy based on Jensen-Shannon. Am I correct?
(b) the source-domain specific teacher strategy implements Multiple Teacher Student model of Section 3.3?
- How does the performance of TS-MCD is impacted if n random points are chosen instead of the top n according to the teacher confidence?
Thank you for your questions.
re a): That is correct.
re b): Yes. I've updated the manuscript to make this clearer.
re the performance impact of choosing random n points: I've made this modification and obtained the average of 10 runs with the same conventions as in the paper. The performance of TS-MCD with random points is lower than the performance of TS-MCD with the top MCD-weighted points for all domain pairs. In most cases, the performance is similar to the performance of TS. This can be explained as training on the teacher's "hard" predictions on random examples reinforces the bias that is already imparted to the student by training on the teacher's soft predictions and doesn't allow the student to distinguish between predictions that are likely true or false.
Could the authors provide more insights on the three measurements used to determine domain similarity?
Another commonly used metric is the A-distance introduced in Ben-David, Shai, et al. 2007. Have the authors tried this method?
Thank you for the thoughtful question. We kept this part succinct to stay within the page limitation, but will extend it in an updated version.
Jensen-Shannon divergence and Renyi divergence are both based on Kullback-Leibler divergence. Jensen-Shannon divergence is a smoothed, symmetric variant of KL divergence. The Jensen-Shannon divergence between two different probability distributions $P$ and $Q$ can be written as $D_{JS}(P||Q) = 1/2 [D_{KL}(P||M) + D_{KL}(Q||M)]$ where $D_{KL}$ is the KL divergence and $M = 1/2 (P + Q)$, i.e. the average distribution of $P$ and $Q$.
Renyi divergence similarly generalizes KL divergence by assigning different weights to the probability distributions of the source and target domain. Renyi divergence is defined as: $D_R(P||Q) = \frac{1}{\alpha-1} log(\sum_{i=1}^n \frac{p_i^\alpha}{q_i^{\alpha-1}})$. If $\alpha=1$, Renyi divergence reduces to KL divergence, i.e. $D_{KL} = \sum_{i=1}^n p_i \frac{p_i}{q_i}$. We set $\alpha=0.99$ as in [1].
We utilised these two domain similarity information exemplarily as they have been successfully used in previous research.
In particular, Asch and Daelemans [1] compare six different domain similarity measures and find Renyi divergence to yield the best results, while recent work [2] employs Jensen-Shannon divergence.
These domain similarity measures are typically based on the term distributions of the source and target domains, i.e. the probability distribution $P$ of a domain is the term distribution $t \in \mathcal{R}^{|V| \times 1}$ where $t_i$ is the relative probability of word $w_i$ appearing in the domain and $|V|$ is the size of the vocabulary of the domain. The intuition behind using term distributions is that similar domains usually have more terms in common than dissimilar domains.
As term distributions only capture shallow occurrence statistics, we reasoned that using hidden representations that capture semantic meaning as the foundation for a domain similarity measure might yield better results.
For this reason, we use Maximum Mean Discrepany (MMD), which measures the distance between a source and target distribution with respect to a particular representation $\phi$. We can define MMD between the source data $X_S$ and the target data $X_T$ as follows: $MMD(X_S, X_T) = \| \frac{1}{ |X_S| } \sum\_{x_S \in X_S} \phi(x_S) - \frac{1}{ |X_T| } \sum\_{x_T \in X_T} \phi(x_T) \| $.
The representation $\phi$ is usually obtained by embedding the source data and target data in a Reproducing Kernel Hilbert Space via a specifically chosen kernel. We could have done this and used e.g. a linear combination of RBF kernels as in [3], but we were interested in how well the learned representation of our teacher is able to capture difference in domain.
For this reason, we follow [4] and use a hidden representation of the neural network as basis for $\phi$, in our case the hidden representation of the teacher model.
In our experiments, MMD based on deep representations does not outperform the more traditional term distribution-based domain similarity measure. We attribute this to two reasons: 1) Due to the limited amount of data, our teacher model might not be deep enough to capture the difference in domain in its single hidden layer; compare this to [4], where the authors identify the fully-connected layer fc7 in the AlexNet architecture as the layer minimizing MMD; 2) In contrast to other deep learning-based approaches, our teacher is only trained on the source domain data. Its representation is thus not sensitive to detect the domain shift to the target domain.
To mitigate this, we could have trained a separate network that minimizes MMD between source and target domain and used its representation for measuring domain similarity.
However, this would have incurred additional computational costs compared to the easy-to-compute term distribution-based domain similarity measures and -- more importantly -- would have required training on the source data during adaptation, which we set out to avoid to enable efficient adaptation. We intend to explore the utility of deep representations for measuring domain similarity more thoroughly in subsequent work.
For similar reasons, we decided to follow earlier work in foregoing the use of A-distance as a measure of domain similarity. Ben-David et al. [7] show that computing the A-distance between two domains reduces to minimizing the empirical risk of a classifier that tries to discriminate between the examples in those domains. Previous work [6] uses the Huber loss and a linear classifier for computing the A-distance. We initially tried using A-distance as a domain similarity measure but found that it didn't outperform Jensen-Shannon divergence.
Furthermore, its reliance on training a classifier is a downside in our scenario with multiple and/or changing target domains, where we would prefer more efficient measures of domain similarity.
We hope these insights were helpful. They will be added to the updated version. Please let us know if you have any further/follow-up questions.
[1] Van Asch, V., & Daelemans, W. (2010). Using Domain Similarity for Performance Estimation. Computational Linguistics, (July), 31–36. Retrieved from http://eprints.pascal-network.org/archive/00007014/
[2] Wu, F., & Huang, Y. (2016). Sentiment Domain Adaptation with Multiple Sources. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), 301–310. Retrieved from https://pdfs.semanticscholar.org/09f0/885d1727a0b82300e94856e0be2f2f72561c.pdf
[3] Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., & Erhan, D. (2016). Domain Separation Networks. NIPS.
[4] Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., & Darrell, T. (2014). Deep Domain Confusion: Maximizing for Domain Invariance. CoRR. Retrieved from http://arxiv.org/abs/1412.3474\npapers3://publication/uuid/DF662E85-CB89-4654-8946-CD387134C975
[5] Ben-David, S., Blitzer, J., Crammer, K., & Pereira, F. (2007). Analysis of representations for domain adaptation. Advances in Neural Information Processing Systems, 19, 137–144.
[6] Blitzer, J., Dredze, M., & Pereira, F. (2007). Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. Annual Meeting-Association for Computational Linguistics, 45(1), 440. http://doi.org/10.1109/IRPS.2011.5784441
Paper describes technique for leveraging multiple teachers in teacher-student framework and performing unsupervised and semi-supervised domain adaptation. The central idea relies on using similarity measure between source and target domains to weight the corresponding trustfulness of particular teachers, when using their prediction as soft targets in student training. Authors provide an experimental validation on a single benchmark corpora for sentiment analysis.
What exactly constitutes the learned representation h used in MCD measure? I assume those are the top level pre-softmax activations - is this the case? Those tend to be typically more task related, would not the intermediate ones work better?
One not entirely clear aspect concerns types of distributions applicable to proposed learning - it assumes the vocabulary (or decision space) between tasks spans same categories, as otherwise one cannot derive the KL based objective, often used in TS framework.
As such, approach is rater constrained in scope.
Authors shall refer in the related-work to similar ideas proposed in the related field of acoustic modelling (and adaptation of acoustic models), in particular, works of Yu et al. [1] and the follow up work of Li et al. [2] which to an extent are a prior the work on knowledge distillation. Reasonably related is also work on deep relationship networks [3], where MT generative approach is proposed to avoid negative transfers, something of central role in this paper.
Minors:
The student S similarly has an output probability -> models an output probability
[1] KL-Divergence Regularized Deep Neural Network Adaptation For Improved Large Vocabulary Speech Recognition, Dong Yu, Kaisheng Yao, Hang Su, Gang Li, Frank Seide
[2] Learning Small-Size DNN with Output-Distribution-Based Criteria, Jinyu Li, Rui Zhao, Jui-Ting Huang and Yifan Gong
[3] Learning Multiple Tasks with Deep Relationship Networks, Mingsheng Long, Jianmin Wang
Thank you a lot for your time and for your valuable feedback, which will be very helpful in improving the paper.
re the learned representation h for the MCD measure: Yes, we use the top level pre-softmax activations in this case as this is the only hidden layer of our model. We are using a one hidden layer-MLP in our experiments to ensure comparability to previous work (cf. [1]). For a deeper model, we would likely prefer to use intermediate ones in line with previous work. We will evaluate this in a future experiment.
re scope of approach: We are not sure we fully understand your comment. We do not assume that the source and target vocabularies are exactly the same, but we assume some overlap in words is present (as do most approaches to domain adaptation in NLP). As we only require the models' output probabilities for training, the student and teacher are free to create different representations from the inputs. This might be a problem if we chose to constrain intermediate layers as in [2]. In our experiments, teacher and student models use different vocabularies (as the teacher has not seen some target domain words during training) and the student achieves state-of-the-art in this scenario. Please clarify if you are referring to a different setting.
re related work: Thanks a lot for pointing us to the very relevant references. All of these are very valuable and they will be added to the related works section.
[1] Glorot, X., Bordes, A., & Bengio, Y. (2011). Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach. Proceedings of the 28th International Conference on Machine Learning, (1), 513–520. Retrieved from http://www.icml-2011.org/papers/342_icmlpaper.pdf
[2] Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). Fitnets: Hints for Thin Deep Nets. ICLR, 1–13. Retrieved from http://arxiv.org/pdf/1412.6550.pdf
Thanks for your response. Clarifying the vocabularies aspect and my initial question, how exactly is the cost H defined in (2) and (5) - I initially assumed you use soft-targets (derived from teacher) and KL divergence between teacher- and student-produced distributions.
However, in case where vocabularies are not exactly matched, you would have to somehow renormalise student distribution.
Unless you map the predictions of teachers into hard-targets again?
Thank you a lot for taking the time to clarify your initial question. We really appreciate it.
To elaborate, in most scenarios, source and target domain vocabularies are not exactly matched. If the student were using the same vocabulary as the teacher (the source domain vocabulary), it would be restricted to learning a distribution only over the subset of words shared between domains and thus unable to transfer knowledge to the new domain. Using the target domain vocabulary allows the student to learn positive and negative words that are shared across domains and to transfer this knowledge to positive and negative words that only occur in the new domain. As our model is discriminative, we do not need to renormalise the student distribution. Did this help to clarify this issue? If not, would you mind to elaborate briefly why the student distribution should be renormalised in the case of mismatched vocabularies and how you would perform this renormalisation?
thanks for the answers, already updated my score. Good luck!
The work extends knowledge distillation to domain adaptation scenario, the student model (for the target domain) is learned to mimic the prediction of the teacher model, learned on the source domain. The authors extends the idea to multi-source domain settings, proposing to weight predictions of teacher model using several domain similarity measurements. To improve the performance of proposed method when only a single source domain is available, the authors propose to use maximum cluster difference to inject pseudo-supervised examples labeled by the teacher model to train the student model.
The paper is well written and easy to follow. The idea is straight-forward, albeit fairly heuristic in several cases. It is not clear what is the advantage of the proposed method vs existing feature learning techniques for domain adaptation, which also does not require re-train source  models, and performs comparable to the proposed method.
Questions:
1. Why did you choose to use different combination schemes in equation (3) and (5)?
For example, in equation (5), what if minimizing H( (1-\lambda) y_teacher + \lambda P_t, P_s) instead?
2. how will you extend the MCD definition to multi-class settings?
Thank you a lot for your time and for your valuable feedback!
- As far as we are aware, all existing Deep Learning-based approaches to domain adaptation as well as most non-Deep Learning-based approaches require joint training on source and target data. Our method outperforms existing Deep Learning techniques as well as the state-of-the-art and does so without re-training. Would you mind referencing the papers on feature learning techniques for domain adaptation that require no joint training so that we can compare against them, as we are not sure to which you are referring?
re 1: That is a good point. The combination schemes in equations (3) and (5) are indeed equivalent. We uploaded the derivation here (https://postimg.org/image/shwj3x0zj/) for ease of use. We will update equation (5) in the paper with the scheme of equation (3) for better readability.
re 2: We can extend MCD to the multi-class setting using pair-wise cluster differences as follows: For $n$ classes, we compute $n$ cluster centroids for the clusters whose members have been assigned the same class by our model. We then create a set containing all $n(n-1)/2$ unique pairs of cluster centroids.
Finally, we compute the sum of pair-wise differences of the model's representations with regard to the cluster centroid pairs: $MCD_{multi} = \sum_{c_1, c_2 \in \text{c_pairs}} | cos(c_1, h) - cos(c_2, h) |$.
Dear AnonReviewer2,
We are sorry to reach out to you again, but we would like to enquire if our reply has helped ameliorate some of your concerns. If not, we would tremendously appreciate it if you could elaborate what aspect of your concern we failed to address.
In particular, it would greatly help us if you could concretise which feature learning techniques for domain adaptation you are referring to that do not require re-training source models and perform comparable to our method.
We intend to do our best to improve this manuscript and thank you a lot for your time to help us with this endeavour.
This paper studies the problem of transfer learning in the context of domain adaptation. They propose to study it in the framework of knowledge distillation. Several settings are presented along with experiments on the Amazon Reviews dataset.
The paper is nicely written and the problem studied is very important towards progress in AI. The results of the experiments could be improved but still justify the validity of applying distillation for transfer learning.
Of course, the experimental setting is rather limited but the benchmarks are competitive enough to be meaningful.
I had concerns regarding discussion of previous work but the extensive responses helped clarify this point (the authors should turn the arguments used in this thread into an appendix).
I think this paper would make an interesting ICLR paper.
Thank you a lot for your time and for your feedback!
We are glad our responses helped to clarify unclear points. We really appreciate your suggestion.
We have updated the paper with an appendix containing arguments of this thread. We have also clarified unclear points that were raised in previous questions and incorporated the relevant related work mentioned by AnonReviewer3.
We are looking forward to more fruitful discussions that will allow us to improve other aspects of the paper.
Dear reviewers,
We hope you've spent pleasant holidays. We regret to bother you with this comment, but we would like to invite AnonReviewer2 and AnonReviewer3 to express if our replies to their reviews have helped to mitigate some of their concerns. If not, we would tremendously appreciate it if you could elaborate what aspect of your concern we failed to address. We intend to do our best to improve this manuscript and thank you a lot for your time to help us with this endeavour.

Meta-review:
role: PC
data: comment: The paper introduces a reasonable but somewhat heurstic and straightforward approach to domain adaptation. Especially, since the approach is not so principled, it does not seem sufficient to evaluate it on a single benchmark (document classification, specifically sentiment polarity prediction). 
 
 + the results on the sentiment dataset are strong
 + the paper is easy to follow
 
 - relatively straightforward
 - the novel aspects are a bit heuristic
 - extra evaluation is needed

Decision: Reject