Title: Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
Authors: Hakan Inan
Khashayar Khosravi
Richard Socher
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: Recurrent neural networks have been very successful at predicting sequences of words in tasks such as language modeling. However, all such models are based on the conventional classification framework, where the model is trained against one-hot targets, and each word is represented both as an input and as an output in isolation. This causes inefficiencies in learning both in terms of utilizing all of the information and in terms of the number of parameters needed to train. We introduce a novel theoretical framework that facilitates better learning in language modeling, and show that our framework leads to tying together the input embedding and the output projection matrices, greatly reducing the number of trainable variables. Our framework leads to state of the art performance on the Penn Treebank with a variety of network models.


Review discussion:
- Have you tried your loss on a different dataset than PTB? (maybe now considered small for this task)
- There are many hyper-parameters (tau, beta, etc) in the proposed loss; how are they chosen (and what are the values for the reported results)?
- How important is tying with respect to training set size?
Dear reviewer,
Thank you very much for the questions. We list below the three answers to the three questions.
Q1) Have you tried your loss on a different dataset than PTB? (maybe now considered small for this task)
A1)Yes, we did do limited experiments on a different dataset which is not reported in the paper:
We tested our 2-layer LSTM model with 200 units on Wikitext 2 (>2x larger dataset, >3x larger vocabulary compared to PTB), very recently published in [1]. We used the same configuration as the one reported in our paper, only further tuning the dropout weights. Below were the word level perplexities on the validation and test sets, using the same naming convention as in our paper:
--Validation--
VD-LSTM : 128.9
VD-LSTM+AL:125.3
VD-LSTM+RE:122.7
VD-LSTM+REAL:119.6
--Test--
VD-LSTM : 122.7
VD-LSTM+AL:119.8
VD-LSTM+RE:116.1
VD-LSTM+REAL:114.1
We would like to acknowledge that for VD-LSTM and VD-LSTM+AL we might not have tuned dropout weights perfectly, judging from a large observed gap between training and validation perplexities during training.
Although we weren't able to get much better performance by tuning them further, this fact might still account for some of the perplexity gap between models that use RE (i.e. tying) and those that don't. It should, however, still be a fair comparison for VD-LSTM vs VD-LSTM+AL, and VD-LSTM+RE vs VD-LSTM+REAL.
The reasons we presented our work on PTB are the following: (i) PTB is a well recognized dataset which majority of prominent work (especially the most recent RNNLM models) on word level language modeling has extensively used for benchmark.
Although we agree that it is not a large dataset, its extensive utilization allows to include most of the previous work and make adequate comparisons. (ii) Keeping in mind the length limitation for a paper, we wanted to present our work in depth and in a coherent manner, while keeping it compact. These said, we would be happy to include more results if requested.
Q2) There are many hyper-parameters (tau, beta, etc) in the proposed loss; how are they chosen (and what are the values for the reported results)?
A2) Firstly, we agree that we were negligent with some of the details on paramtere choices, which we will improve in a future revision. Thank you very much for pointing this out.
We have observed that setting the weight of the augmented loss (called alpha in the paper) to a constant multiple of the temperature (tau) works for all the model sizes we have experimented with, and this constant is set to 0.5 in our experiments. Temperature (tau) is set to 20. With this said, we have not observed sudden deteriorations in the performance with respect to moderate variations in tau and alpha.
However, setting the the weight of the augmented loss to very large values impede learning since the embedding space is formed throughout optimization, and forcing a wrong metric space from the beginning through a large augmented loss leads to poor optimization. The augmented loss exerts its effect after a rough estimate of the embedding space is learned, and gradients from the conventional loss become smaller.
Beta is a parameter mentioned in Section 6.2 which is explicitly used for the empirical validation experiments, and its value is swept from 0 to 1 in Figure 1(a).
Q3) How important is tying with respect to training set size?
A3) This is a very interesting question, and it could perhaps be best addressed with the following experiment: Randomly subsample several times from the training set and compare the resulting averaged perplexities for tied vs. non-tied cases at different sample sizes. Unfortunately, this is not within the scope of our work, so we currently can't offer a very rigorous answer to this.
On the other hand, to give an answer in a broader sense, we can argue given the performance on datasets of different sizes that tying is a useful mechanism of our framework regardless of the size of the dataset. It offers gains due to (1) enforcing the metric similarity between word vectors, as investigated in our work (2) Reducing the model complexity by eliminating a large number of parameters, at the same time retaining the representational power according to our framework. To comment on the significance of tying compared to the augmented loss with respect to the training set size, we believe that tying is more significant for larger training sets. An important explicit mechanism of the augmented loss especially for small networks is to improve supervision from the dataset (as explained in Section 3), and large datasets partly mitigate the supervision problem. Tying, on the other hand, is still important because of the aforementioned two bullet points.
References:
[1] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843, 2016
This paper gives a theoretical motivation for tieing the word embedding and output projection matrices in RNN LMs. The argument uses an augmented loss function which spreads the output probability mass among words with close word-embedding.
I see two main drawbacks from this framework:
The augmented loss function has no trainable parameters and is used for only for regularization. This is not expected to give gains with large enough datasets.
The augmented loss is heavily “engineered” to produce the desired result of parameter tying. It’s not clear what happens if you try to relax it a bit, by adding parameters, or estimating y~ in a different way.
Nevertheless the argument is very interesting, and clearly written.
The simulated results indeed validate the argument, and the PTB results seem promising.
Minor comments:
Section 3:
Can you clarify if y~ is conditioned on the t example or on the entire history.
Eq. 3.5: i is enumerated over V (not |V|)
Thank you very much for your kind review.
We address your major and minor comments below.
Response to the major comments:
The augmented loss is meant to be a light machinery on top of inner products in the euclidean space of word embeddings.
Provided that we stick to the notion of word similarity for estimating a target distribution, we believe that this method (i.e. computing inner products and normalizing via softmax) is a simple and natural way to obtain a y~. Having obtained a distribution, we believe that minimizing a KL divergence term is a natural next step. We would like to also note here that this method was used by authors in [1] for obtaining an augmented loss term, although their method is proposed for a different problem.
Despite being simple and not having many parameters, we believe it is an expressive term. We certainly agree that augmented loss might not be as effective on large datasets, but we argue that this is also due to large datasets having enough information to mitigate the supervision problem (we discuss this also below in response to a reviewer question). As a final remark regarding the efficacy of the augmented loss, in addition to PTB we observe non-trivial improvements with the augmented loss on Wikitext-2, which is a larger dataset (results of which we will include soon in a forthcoming revision).
To comment on different ways for introducing the augmented loss, we believe that this is a very well made point and that it is definitely worth pursuing given the improvement which we are able to achieve by investigating it in a certain direction. This would potentially be pursued in two steps:
(1) Decide how to find a y~.
Although we believe that computing inner products and applying softmax is a natural way for exploiting the metric similarity, one can proceed in other ways. Frankly, we did not initially intend to reach tying by selecting a loss function (we did not build this link in our earlier work);
however our feeling is that any method which uses euclidean distance between words could easily lead to tying. The reasoning is the following: The classification step of the language model is essentially a computation of inner products between the output hidden state and the classifier weights. Choosing classifier weights as word vectors is a compelling choice to enforce similarity since the loss function aims to distribute its predictions according to word similarity.
Of course, to show this mathematically, one needs to be able to show something similar to eqn. 4.5 in our paper (i.e. L =~ W). Based on our derivation, one can possibly attain such an identity provided that the mapping from the word embeddings to y~ is differentiable and could be well represented by its first order approximation (e.g. its derivative is Lipschitz with a controllable constant). The idea, also exploited in our derivation, is to be able to obtain a linear mapping from L to W.
(2) Using a loss other than KL or cross entropy. This could be very interesting;
in fact one can directly construct a loss function with the metric space (word embeddings) built into it (a very simple example is generalized euclidean distance).
However, practically speaking, one should be careful to control the interaction between the two loss terms.
For instance, depending on the phase in training one can take over and cause instabilities or null results. My collaborators and I have also been interested in exploring different options, and we would be very glad to see other works researching losses which exploit metric similarities in language processing.
Responses to minor comments:
Comment:Can you clarify if y~ is conditioned on the t example or on the entire history.
Response: y~ is indeed conditioned on the history, and not just the t'th example. This is a recurring mistake in the paper, and we have made the necessary corrections in our internal version.
Comment: Eq. 3.5: i is enumerated over V (not |V|)
Response: Also corrected, thank you for pointing this out.
References:
[1] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015
This paper provides a theoretical framework for tying parameters between input word embeddings and output word representations in the softmax.
Experiments on PTB shows significant improvement.
The idea of sharing or tying weights between input and output word embeddings is not new (as noted by others in this thread), which I see as the main negative side of the paper. The proposed justification appears new to me though, and certainly interesting.
I was concerned that results are only given on one dataset, PTB, which is now kind of old in that literature. I'm glad the authors tried at least one more dataset, and I think it would be nice to find a way to include these results in the paper if accepted.
Have you considered using character or sub-word units in that context?
Thank you very much for your comments.
We address your comments/questions separately below.
Comment: The idea of sharing or tying weights between input and output word embeddings is not new (as noted by others in this thread), which I see as the main negative side of the paper. The proposed justification appears new to me though, and certainly interesting.
Response: We're pleased to be able to offer a convincing mechanism for tying the embeddings and the classifier;
indeed, this framework is our main contribution. Our work investigates this new framework in three directions, leading to the following: (1) Tying is a natural outcome of applying this framework (2) For relatively small models , using an additional loss function improves learning (3) Tying alone leads to significant improvements in a broader context.
Regarding the third point: Although we certainly agree that using a shared representation for inputs and outputs is not a new idea, we find it worth mentioning that the possibility of improvement by doing so was not particularly explored previously.
Rather, shared representations were employed as constituent elements of the underlying models.
Comment: I was concerned that results are only given on one dataset, PTB, which is now kind of old in that literature. I'm glad the authors tried at least one more dataset, and I think it would be nice to find a way to include these results in the paper if accepted.
Response: We would like to thank you for raising this point previously; we have since been working on optimizing all our models (hyper-parameters) for the wikitext-2 dataset, and we have achieved satisfactory results which are in line with our predictions. Hopefully we will add the results without causing much additional reading burden for the reader. We really appreciate your feedback on this matter.
Question: Have you considered using character or sub-word units in that context?
Answer: This is a very good question;
however we predict that the framework won't provide as significant gains in character based models. Our reasoning is as follows: Our loss framework aims to exploit the metric attached to the elements in the vocabulary. In the case of characters, the vocabulary is smaller and it perhaps carries less meaning.
For instance, one might expect "terrific" and "great" to be in very close proximity in English language, but it's not as reasonable to expect a particular pair of characters to be very close together, with certain exceptions. And even in the case where it might be possible (some languages might have rules with regard to character ordering), there usually isn't enough volume in the set of characters to offer very serious gains.
Also, the small vocabulary size will possibly reduce the effect of tying significantly owing to losing the advantage of reducing the redundant complexity.
Finally, the large effective increase in the training set by reducing to characters will potentially eliminate the advantage of using the augmented loss (since AL is most effective when model can use better estimated output statistics). Most arguments in this paragraph are also valid for sub-word units, but of course, the situation could be different with languages which contain many prefixed and suffixed words, and when the tokenization is done according to the language rules.
This work offers a theoretical justification for reusing the input word embedding in the output projection layer. It does by proposing an additional loss that is designed to minimize the distance between the predictive distribution and an estimate of the true data distribution. This is a nice setup since it can effectively smooth over the labels given as input.
However, the construction of the estimate of the true data distribution seems engineered to provide the weight tying justification in Eqs. 3.6 and 3.7.
It is not obvious why the projection matrix L in Eq 3.6 (let's rename it to L') should be the same as that in Eq. 2.1.
For example, L' could be obtained through word2vec embeddings trained on a large dataset or it could be learned as an additional set of parameters. In the case that L' is a new learned matrix, it seems the result in Eq 4.5 is to use an independent matrix for the output projection layer, as is usually done.
The experimental results are good and provide support for the approximate derivation done in section 4, particularly the distance plots in figure 1.
Minor comments:
Third line in abstract: where model -> where the model
Second line in section 7: into space -> into the space
Shouldn't the RHS in Eq 3.5 be \sum \tilde{y_{t,i}}(\frac{\hat{y}_t}{\tilde{y_{t,i}}} - e_i) ?
Thank you very much for your kind assessment.
We list below the responses to your major and minor comments:
Major comments:
Comment: The construction of the estimate of the true data distribution seems engineered to provide the weight tying justification in Eqs. 3.6 and 3.7.
Response: Although the analysis in the paper may naturally lead one to believe that this is the case, we would like to note that this work was not initiated with the intention of finding an explanation for sharing the embeddings and the classifier.
In fact, in our early report in [1], we did not establish such a tight analytical link between the augmented loss and reusing word embeddings although we introduced them both. We also certainly hope that our work could be primarily appreciated for the particular framework which exploits metric similarity of words which it introduces. At this point, would like to also explain our motives for eqns 3. 6 and 3. 7, namely the calculation of y~. Eqn 3. 6 is meant to select the word vector corresponding to the target token.
Eqn 3. 7 is a very elementary way to capture the word similarities through inner products and then normalize the similarity vector to be interpreted as a probability distribution; this normalization was in fact used in [2] followed by a KL distance term for a different problem. From our perspective (which certainly includes some authorship bias), the fact that this simple and plausible method leads to tying (even if seemingly obviously) points to the intuitiveness of shared embeddings in language processing tasks.
Comment: It is not obvious why the projection matrix L in Eq 3.6 (let's rename it to L') should be the same as that in Eq. 2.1.
For example, L' could be obtained through word2vec embeddings trained on a large dataset or it could be learned as an additional set of parameters.
Response: This is exactly right; and in fact, we do not claim that metric used for the augmented loss should clearly be the word embeddings of the model. Our method is simply by choice, guided by observations. As a matter of fact, very early on in our work (this is when we did not have tying in mind, and were experimenting with the augmented loss idea) and inspired by the work in [3], we used both word2vec and gloVe embeddings (both 300 dimensional) as static metrics.
However, the performance was significantly inferior to that when using the word embeddings of the model. In the interest of full disclosure, we should mention that we did not meticulously tune the models with static embeddings after not being able to see improvement (and we only tried them on PTB);
however, the results did not seem promising.
Instead, we decided to adopt the idea of information transfer within the model, and this lead to the current framework. The experimental performance of tying alone on large networks might slightly overshadow this core idea, but it was this idea that enabled us to reach tying, and link our two improvements.
Comment: In the case that L' is a new learned matrix, it seems the result in Eq 4.5 is to use an independent matrix for the output projection layer, as is usually done.
Response: Actually, not quite - although the above is not an incorrect conclusion. We will address this comment in the context of sharing embeddings. One takeaway from Section 4 is that the output projection matrix, W, will tend to the matrix used for capturing the word metric (in the subspace sense). In this case, this matrix is L'.
Therefore, under the same assumptions, one would expect W to get close to L', and it could be tempting to consider using W=L' -this is the "not quite" part-.
However, this may not be feasible from a practical perspective, because we potentially wish the classifier to be relevant to the current dataset, and not too much to an externally learned metric, for compatibility reasons -this is the "not incorrect" part-. You are certainly right in that in order to tie the embeddings and the classifier, one would need to use the the word embeddings of the model itself, and not an external embedding matrix.
Then again, the former is what our framework is based on, and it is the setting where we can make such plausible technical arguments.
Minor comments:
Comment: Third line in abstract: where model -> where the model, Second line in section 7: into space -> into the space
Response: Just corrected this in our internal version. We appreciate it, thank you.
Comment: Shouldn't the RHS in Eq 3.5 be \sum \tilde{y_{t,i}}(\frac{\hat{y}_t}{\tilde{y_{t,i}}} - e_i) ?
Response: That is correct;
in fact y~ should not have been factored out. We fixed it as a direct multiplication with e_i. Thank you very much for pointing this out.
References:
[1] Hakan Inan and Khashayar Khosravi. Improved learning through augmenting the loss, 2016
[2] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015
[3] Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya, and Tomaso A Poggio. Learning with a wasserstein loss, 2015
**This comment was slightly edited after initial posting for clarity**
Thanks for your reply and clarifications. Not factoring out y~ in Eq 3.5 seems to mean the intuition in the paragraph below it is no longer valid.
Again, it seems that to provide the justification for weight tying, L in Eq 3.6 must be fixed to the same L in Eq 2.1. If these L's are not tied together then from my reading, you will get a standard RNNLM with untied input embedding and classifier parameters with the rest of the derivation (Eqs 2.1 to 3.5) still holding. No theoretical reason seems to be given for tying these two L's so I assume the motivation is empirical.
I will be keeping my rating.
--Response about the issue regarding eqn. 3.5--
We're sorry that we did not provide an update, but actually our initial formulation was correct, and our response to the review was incorrect. Note that y~ sums up to 1 , and the factor in front of y^ becomes 1 when summed over all the i's. We are sorry we were negligent about following up with this issue, although minor.
But in conclusion, the arguments in the paragraph are valid.
-- Response to comments about the motivation behind using same L matrices in eqns 2.1 and 3.6 --
*We assume throughout that it is acknowledged that L matrices in eqns 3.6 and 3.7 are the same*
We feel that there might be a slight miscommunication. The use of the embeddings of the model itself to improve learning in a self contained manner is in fact the core of our work, and motivation behind it is as explained in our introductory paragraphs and section 3. We do not claim to further theoretically motivate this idea in our work, and we actually agreed with your comments in our first reply above.
Overall, our main theoretical contribution is the results we present assuming this framework, which is in fact motivated in section 3, based on previous work such as [1],[2], as well as the widely recognized metric features of trained word vectors. We would be more than happy to further elaborate on this point should there be any remaining concerns.
At this point, for the sake of completeness of our argument, we would like to elaborate once more on the alternative scenario where L in 2.1 and L in 3.6 are not the same (Let's call the one in eqn 3.6 L' keeping the convention from our previous discussion). There are two cases:
1) L' is static, and not learned. In this case, the output classifier matrix, W, will get close to L'.
As a result, in the extreme case where W is actually L', this means it would make sense to tie W and L' together. We have a discussion of this case above which explains why this may not be a good idea.
2) L' is learned with other parameters. This actually is probably a rather boring case because of the following: L' comes into the problem only through the augmented loss term, and this term forces W and L' to be close together, as argued above. With no other force acting on L', the optimizer will probably want to choose L'=W (that is, L' is set to whatever W is) with very little effort, since this choice will minimize the augmented loss term.
Overall, although this would exactly to lead to a scenario which you pointed out, it would not be worth pursuing for the reason above.
[1] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015
[2] Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya, and Tomaso A Poggio. Learning with a wasserstein loss, 2015

Meta-review:
role: PC
data: comment: pros:
 - nice results on the tasks that justify acceptance of the paper
 
 cons:
 - In my opinion its a big stretch to describe this paper as a novel framework. The reasons for using the specific contrived augmented loss is based on the good results it produces. I view it more as regularization.
 - The "theoretical justification" for coupling of the input and output layers is based on the premise that the above regularization is the correct thing to do. Since that's really not justified by some kind of theory, I think its questionable to call this simple observation a theoretical justification.
 - Tying weights on the inputs and output layers is far from novel.

Decision: Accept (Poster)