Proactive Sequence Generator via Knowledge Acquisition
Authors: Qing Sun
James Cross
Dmitriy Genzel
Venue:  (2020)
Keywords: neural machine translation
knowledge distillation
exposure bias
reinforcement learning

Abstract: Sequence-to-sequence models such as transformers, which are now being used in a wide variety of NLP tasks, typically need to have very high capacity in order to perform well. Unfortunately, in production, memory size and inference speed are all strictly constrained.  To address this problem, Knowledge Distillation (KD), a technique to train small models to mimic larger pre-trained models, has drawn lots of attention.  The KD approach basically attempts to maximize recall, i.e., ranking Top-k”tokens in teacher models as higher as possible, however, whereas precision is more important for sequence generation  because of exposure bias. Motivated by this, we develop Knowledge Acquisition (KA) where student models receive log q(y_t|y_{<t},x) as rewards when producing the next token y_t given previous tokens y_{<t} and the source sentence x.   We demonstrate the effectiveness of our approach on WMT’17 De-En and IWSLT’15 Th-En translation tasks, with experimental results showing that our approach gains +0.7-1.1 BLEU score compared to token-level knowledge distillation.

Under review as a conference paper at ICLR 2020
PROACTIVE SEQUENCE GENERATOR VIA KNOWLEDGE ACQUISITION
Anonymous authors Paper under double-blind review
ABSTRACT
Sequence-to-sequenee models such as transformers, which are now being used wide variety of NLP tasks, typically need to have very high capacity in order perform well. Unfortunately. in production, memory size and inference speed are all strictly constrained. To address this problem, Knowledge Distillation (KD), a technique to train small models to mimic larger pre-trained models, has drawn lots of attention. The KD approach basically attempts maximize recall, i.e. ranking Top-k" tokens in teacher models as higher as possible, however, whereas precision is more important for sequence generation because of exposure bias. Motivated by this, we develop Knowledge Acquisition (KA) where student mod- els receive log q(y.ly <t. x) as rewards when producing the next token y, given previous tokens yer and the source sentence x. We demonstrate the effective- ness of our approach on WM De-En and IWSLT' Th-En translation tasks, with experimental results showing that approach gains +0.7-1.1 BLEU score compared to token-level knowledge distillation.
INTRODUCTION
We have recently witnessed rapid progress sequence models for natural language processing (NLP). In particular, transformers (Vaswani al., 2017; Dai et al., 2019) and BERT (Devlin et al., 2018: Liu et al., 2019) have been established as state-of-the-art approaches for sequence generation and language modeling respectively. However, the resulting sequence models are often exceedingly large. For example, a big transformer with 6 layers, 1024 dim and 16 heads has ~ 100M parameters, which can make production deployment impractical, especially on mobile devices.
Unfortunately, learning small sequence models is very challenging because of the nature of deep learning- curve fitting. Training deep neural networks is data hungry and needs bigger "brains". To deal with the challenges, Kim & Rush (2016) consider applying Knowledge Distillation approaches (Hinton et al., 2015) to match the distribution the teacher model. At teken- KD approaches minimize (9 dent] is probability distribution over tokens in the vocabulary conditioned on previous tokens Yca and the source sentence In other words, the KD approaches maximize recall the frac- tion of the top-k" tokens in q(y:ly<r, x) have been retrieved among the top-k' in p(yely<t. x). However, sequence modeling actually cares about precision the fraction of the top-k' tokens in p(y,yye x) are included in the top-k" of q(yely<e, x), because imperfect predictions in the top-k' cause exposure bias at inference time.
pyyll(yylly x)) where q(yily<(, [teacher] pyy,y x) [stu-
To optimize precision, we attempt to minimize KL-divergence in the reverse order, that is, ( x) x))l((yll
Our approach can be thought of as kind of Actor-Critic ap- proach where the "Actor" is a student model asking for advice when learning to generate every single token and the "Critic" is a teacher model that gives log q(yrly <t, x)) as rewards. After re- ceiving the feedback, "Actor" updates it's belief accordingly, as shown in Fig. 1.
Unlike task-specific metrics such as BLEU score, teacher models summarize training data points and structure knowledge in an abstract, hierarchical manner. Therefore, they are able to offer more useful advice than matching n- D-grams with a finite number of human references. For example, assume there is a training example (SRC: "Die Landschaft am Meer ist wunderschn" TRG: "Amazing view along the sea" J. During learning, given the first token "Amazing", student models might ask "How about saying amazing view?", teacher models most likely give high rewards since it's expected go with

---

Under review as a conference paper at ICLR 2020
Student
Teacher
Student
Teacher
S: view?
dob Poo
Log 0.4
Lele,
boat sicemn view
boat scene view
S: bird?
L.g
deb
y1
Y
(a) Token-lever Matching.
(b) Learning strategy.
Figure 1: Student-Teacher learning system. (a) Given previous tokens (yellow), the student model asks feedback for producing token, e.g., "view". and the teacher model gives rewards. egg., log """"mii"""""a x). (h) The difference in learning strategy matters! We ignore y and x for sim- plicity. We flip the sign of gradients and demonstrate them the 2nd When q(v.) dominates p(ye), the gradients less than (magenta) and p(y:) 1. contrast, when p(y+) dominates q(y.). the gradients greater than 0 (cyan) and p(y:) J. More importantly, KA pulls the probability of "view" and "scene" more gently but push the probability of "bird" and "boat"" much harder compared to KD. In other words, KD attempts to rank "view", "scene" higher increase recall, while KA aims reducing the chance of "bird" and "boat" being picked to maximize precision.
"Amazing view
of the seaside, of the beach,
of the boats in the sea from the boats
along the sea, of the birds flying over the water
", where most of them express similar meanings with the human references even though they have mismatched n-grams.
Contributions. To the best of our knowledge, we are the first to provide depth-analysis of the effects of minimizing x (p/lg) alleviating exposure bias. In addition, enable stable inforcement training via replacing task-specific metries with high-capacity models. Finally, we improved +1.1 BLEU score WMT' 17 de-en task and +0.7 BLEU score IWSLT" 15 th-en task over level- KD with up to relative 200% improvement.
SEQUENCE GENERATION
Let (x E (X) an example translation pair, where x the source sentence and y is the target sentence. y: is the token at position & and y-ı is tokens before position t. Sutskever et al. (2014) propose sequence-to-sequeme
models which typically factorize the joint probability p(y x) over sequence of conditional probabilities with parameter 0:
T
p(ylx)
II po yet,)) t=
where, Po(y: Yer, x) is the probability of the token y: given previous tokens y and the source
sentence x. Basically, the preceding tokens y<t are encoded into the hidden states via a state tran- sition function
h, = (h:
By substituting (2) for (q). we have
p(ylx)
po he) 1=
II
(3)
Policy
This tells us that the sequence models, e.g., RNNs, acts like stochastic policy which picks a discrete action, í.e., producing a token y, V. running on a world model M with transition function f. Training. We minimize the cross-entropy loss
LCE
Σ
log Po poppooyy (yref Iyref ,))
(4)
2

---

Under review as a conference paper at ICLR 2020
where, yraf denotes human references. At each position, the models are conditioned on the ground- truth tokens annotated by humans no matter what tokens are predicted by themselves.
Inference. In sequence generation tasks, exact inference is intractable due to exponentially large search space. Instead, we apply an approximation inference algorithm Beam Search (BS). BS is greedy heuristic search that maintains the top-B most likely partial sequences through the search tree, where is referred to as the beam size. At each position, BS expands these B partial sequences all possible beam extensions and then selects the highest scoring among the expansions. Unlike training. ground-truth tokens are not available. The models have to use their own predictions in decoding.
Evaluation. To evaluate the quality of generated sequences, we typically use metrics such as BLEU score to measure their n-grams overlap with human references. However, the standard metries are problematic and none of them correlate strongly with human evaluation at the level of individual sentences. For example, given human reference "Amazing view along the sea", , the sequence "The scenery of the seaside is beautiful" gets low BLEU score because there is no matching n-grams of order 2, 3, or 4. addition, human references are few sentences lacking in diversity. For example, when asking more people, they might say "A nice beach" or what amazing view of the seashore".
2.1 DISCREPANCY AMONG PROCEDURES
Exposure bias. During training, the models only explore the training data distribution, but never get exposed to their own predictions. Searching in under-explored space causes errors. More im- portantly, such errors accumulate over time because of greedy search. For the example in Sec. 1, assume there are no sentences in training data starting with "Amazing".
At position pick token from p(y "AAmazigg""" x) = "cup"
At position 2: pick token from ply| ""Amalimag cup", xxp" = "on"
position 3: pick token from p(y p AAmazing cup on", x) + "table"
We can see that the poor token "cup" caused by the noisy distribution pyy| l'Amazingg, x) makes the situation even worse. The distributions become more and more noisy and the generation goes far away.
Sub-optimal models. The training objective is different with the metrics used in evaluation. To address this issue, some works attempt to directly optimize the metrics. It definitely helps improving the score, but suspect that this might hurt the models because poor metrics discourage learning the semantic meanings. For example, the low score of "The scenery of the seaside is beautiful" makes the meanings of "Amazing" and "beautiful" or "scaside" and "sea" far from the other.
3 KNOWLEDGE ACQUISITION
Unlike knowledge distillation, we attempt to minimize
D Poly: b.)llaa(re h.)
(5)
Eqn.6 can be explained Eyy-poyy log ( h:)] + [po he]] expected rewards with an entropy term, where
(6)
Actor-Critie algorithm (Konda & Tsitsiklis, 2000) that maximizes the
"Critic": (aka teacher models) estimate the rewards of producing token y. given preceding tokens Yer and the source sentence x, i.e.,
Q(a = Yt, s = hy) = log (ye hy) "Actor": updates the policy (y: h:) guided by the "Critic"
As opposed to behavior cloning, student models ask teacher models to evaluate their behaviors and then update accordingly when producing the next token. By comparing with KD that attempts to minimize D k (llp)) our approach actually a kind of reverse KD, which we call "Knowledge Acquisition" (KA).
3.1 LEARNING STRATEGY
Without loss of generality, we simplify the problem to a 1-D problem where the student model p(x) single-modal Gaussian and the teacher model (zx) is mixture of two Gaussians, shown in
3

---

Under review as a conference paper at ICLR 2020
0.4
KL(qIIp)
0.3
Gradient
KL(p/lq)
Pdf
0.2
p(x)
0.1
q(x)
0.0
2
-2
2
x
X
(a) p(z) and q
(b) Gradients
Figure 2: Matching two 1-D distributions. (a) p(z) and q(r). (b) The derivatives of Day wan p(x) in both orders. (xx(((x)) (p))) pulls more on where q(z) > p(z) (blue region) while De (p(x))] pushes
where (xx) q(x) (red region).
Fig. Let's take a anpth-asalysis of their objective functions and their derivatives w.r.t. p(x) to see what changes after swapping p(x) and q(x).
3.1.1 OBJECTIVE FUNCTIONS
The entropy term in (6) goes away in KD because q(x) is a fixed value and never be optimized during training. It's clear that the entropy term helps to avoid over-fitting the model to human references and generalize well unseen test data.
3.1.2 DERIVATIVES
Based on Lagrangian relaxation (see Appendix A.1), the derivatives of DEL w.r.t p(x) in both orders are
8T (x<(((())lpxz) 0p(x)
=1 q(x)
(<<(p(=)l((()) p(x)
log q(z)
(8)
p(z) θp(x)
(al
Gpla(1)
(8). tells us that
(a) (()) p(x) Gallp (x) < 0 and Gplla 0
(b) q(x) < p (z) Callp (x) 0 and (pllq(r) > 0
That is, the probability x is pulled up when q(x) dominates p(x), and pushed down otherwise. However, they take efforts at different levels (see Appendix A.2).
In (a). |G p (x)) > (Gpiq(x)) while in (b), [g (p)]a())
In other words, minimizing D (q(x))p((z)) attempts to pull up the probability of x that under- estimated by the student models while minimizing Dx (p())[(((r) attempts to push down the the probability of x that was ever-estimated by the student models. If we define
Recally (p(x), k())
Precisiona (p(2),
p's Top-k n q's Top-A"
p's Top-k n Top-A*1
(10)
k' where, p's Top-k' a set including k' items having the highest rankings p(x) and q's Top-k" is set including k" items having the highest rankings in (zx). Clearly, minimizing Der (x(((()))) (q(x))[((()) is to maximize recall because its role is increase the chance that x's in q's Top-k" are truly picked. By contrast. minimizing Dx (p(2) (px))((()))) (z(())) is to maximize precision since its role is to avoid x's that are over-estimated being picked by mistake.
h"
Overall, sequence modelling cares about precision because at test time, the generation is sensitive to the model's 's predictions. The errors caused by inaccurate predictions in the top-k' accumulate and make the generation brittle.
4

---

Under review as a conference paper at ICLR 2020
3.1.3 DOES THE DISTINCTION MATTER?
One might reasonably ask whether the distinction matters, given that both learning strategies aim to eventually reach the same equilibrium point, i.e., p(r) ( In practice, the answer is yes due to the following reasons:
Low-capacity p(z). In practice, student models are of limited capacity due to memory and inference speed restrictions, especially in production. For example, running a big transformer on mobile device with real-time response is impractical. In this case, the equilibrium state p(x) q(z) is unreachable. Thus, different learning strategies lead to different local optima, and result in different performance.
Imperfect q(z). The teacher models are generally better approximations of the real underlying distributions from which the training data is, but they still not 100% accurate. In particular, the exponentially-laree search space makes sequence models easily over-fit to more frequently occurring data points, while being uncertain about the data points that are less likely observed. Kim Rush (2016) also point out a mixture of Dx (qllp) and cross-entropy loss reaches the best performance. In summary, with the limited capacity and noisy q(x),, picking what to learn smartly turns out be important.
3.2 TO BE PROACTIVE WITH RICH ADVICE
To further study the capability in exploration, we explicitly restrict the knowledge to be partially observed by allowing only the top-k to be available. The 1K term becomes
((x)(xogp(x)) log (xx))
XETop-k p(x)
where, Top-k p(x) is a set of items with the highest ranking. Only p(x) and q(x) according to the in the top-k are caleulated while the rest is discarded. Assume is discrete variable and takes value 0, 1, 2, and 3. We build a quite simple model with single soft-max layer to produce a distribution over x, and set top-k=2. The real distribution is
qo 0.4, q1 = 0.3, q2 = 0.2, q3 = 0.1
Fig. 3 shows that KD drives Po and P1 to the real values very quickly, but almost ignore P2 and p3 since they are not directly optimized. In contrast, KA first optimizes po and p2 which are the top-2. After 100 epochs, when Po and pi become the top-2, the loss drops very fast and even lower than that of KD. Moreover, KA drives P2 and Py much closer to the real values since the model has explored states, i.e., x = 1,
In sequence generation tasks, if student models receive rewards at each position, the entire distribu- tion is fully visible. KA implicitly explores in the above way.
In theory, generalize approach multi-step
By - (h.) [lo 4o (Yeer hy) + H (Po(eee h)]
(11)
ie., minimizing D (po (h,) (Vo hy))). Yeur are tokens starting with position t and end- ing position L'. By increasing 1'-1, the student models can capture long-term dependency and are much more capable of handling exposure bias. However, the training is expensive even with ='-==2 since the search space explodes exponentially, In this case, picking top-k candidates is solution. However, in practice we only model the reward for a single step (but use the signal from all steps in each batch update).
RELATED WORK
Our work closely relates to two lines of work: knowledge transfer and dealing with the mismatch between training and inference procedures. Both of them have already been explored in the litera- ture.
Learning / inference. To handle the mismatch between training and inference, previous works attempt to directly optimize the task-specific metric at test time. Ranzato et al. (2015) propose sequence-level training algorithm in reinforcement learning. The models receive rewards until the completion of the entire sequence. Considering that the search space in sequence generation is exponentially large, ie., O(D) where V is set of tokens in the vocabulary (~ 10K more) and
5

---

Under review as a conference paper at ICLR 2020
0.45
0.45
0.40
0.40
0.3
0.3
0.2
0.2
0.25
0.25
0.20
0.10
0.10
0.08
0.08
0
100 Epoch
200
100 Epoch
200
(a) KD
(b) KA
Figure 3: ((r) evolves over epochs. pu. pl, and pa are shown from top bottom. Purple lines denote p(x) and blue lines denote ground-truth (æx.. Po and Pı in KD converge qu and qi in ~ 30 epochs, while P2 pa stay away from ground-truth qu and qa with respect to those KA.
Table 1: Model configurations for WMT' 17 De-En and IWSLT 15 Th-En
encoder dim
decoder dim
layer
head
layer
head
student
6
512
4
1
512
4
De-En
teacher
6
1024
16
6
1024
16
student
128
Th-En
teacher
3
256
1
256
2
T is the length of the sequence (~ 20 or more), the rewards are extremely sparse. This makes the training unstable. To alleviate the sparse rewards problem, Liu et al. (2017) use Monte Carlo rollouts estimate immediate rewards at each position. Unfortunately, the estimation very noisy and of high variance due to the exponentially large search space. Moreover, the training is computationally expensive. this paper, we attempt to use Actor-Critic algorithms, where the "Critic" is teacher model used to estimate action-value function, i.e., the rewards of producing the current token given previous tokens, and the "Actor" is a student model that learns to produce token at each position.
To deal with exposure bias, Bengio et al. (2015) propose to gradually replace tokens from human references to generated tokens during training. The training starts with tokens from human ref- erences and ends up with using generated tokens. However, the rewards which are the matching n-gr n-grams with a few human references limits the capability of the models in exploration. this pa- per, we equip the models with knowledgeable teachers which offer smart advice based semantic meanings.
Transfer learning. In the context of sequence generation, pre-training with monolingual data (Rad- ford et al., a;b) has had significant success in language understanding. Another line of work is knowledge distillation (KD) (Hinton et al., 2015), in particular when models of different archi- tectures and sizes. Kim & Rush (2016) propose two levels of knowledge distillation. At sequence- level, student models are trained on the augmented dataset with outputs of running beam search with teacher models. At token-level, they get the conditional probability of each token given pre- ceding tokens closer to that of teacher models. In other words, this is equivalent to minimizing (xz(gllp), where is the distribution of teacher models and is the distribution of student models. Yu et al. (2017) use adversarial training to encourage the models producing human- sequences by learning sequence-level discriminator man references. In fact, this is equivalent to minimizing 0.5 x (Dkz(lllp) 0.5 Dx (p))))))) but sequence-leve level. In this paper, offer thorough analy- sis of the effects minimizing D xr. (pllq) on avoiding exposure bias, and we find that minimizing Dee (p]lq) performs the best among D 1 (qllp) and JSD.
distinguish generated sequence and hu- Shannon Divergence (JSD, i.e.,

---

Under review as a conference paper at ICLR 2020
37
3000
Shared words
36.5
No. New Words
2500
KD-unique
BLEU
KA
KD
KA-unique
36
1500
35.5
1000
35
500
0
5
Top-K words
10
30
50
70
Epoch
(a) Accuracy vs. Top-k
(b) No. of novel tokens
Figure 4: Ablation study WMT' De-En task. (a) Accuracy validation with variable k. Larger means larger part distribution observed. (b) Number novel tokens emerge in top-16, High number indicates strong exploration in search space. We noticed that ~800 probability mass are put on the top-16 tokens.
Table 2: BLEU scores on WMT 17 De- En and IWSLT' 15 Th-En. (On test set)
Student
Teacher
KD
KA
1/2KA+1/2KD
30.23 Th-En 12.37
De-En
34.10 17.55
31.13 12.70
32.24
31.51
13.36
12.93
EXPERIMENTS
5.1 DATASET
We test approach WMT 2017 German-English with 4M sentence pairs, validate new- stest2016 and test on newstest2017. All the sentences are first tokenized with moses tokenizer and then segmented into 40K joint source and target byte pair encoding (Sennrich et al., 2015). Another Thai-English dataset comes from IWSLT 2015. There are 90k sentence pairs in training and we take 2010/2011 the dev and 2012/2013 the set. Byte pair encoding is of size 10K.
5.2 TRAINING
Without a good starting point, the performance of minimizing Dx (pllq) degrades because the models are more than likely stuck on the current best tokens and unlikely to explore. We therefore pre-train student models and fine-tune them in all the experiments by minimizing
SAL = (1 A) x, 0) D DKL. po h.)lla h:)
(12)
(xu KD
where, À is trade-off parameters. Basically, "Critic" and "Actor" are optimized simultaneously. However, in this paper, we simply freeze teacher models and leave joint training to future work. Model. Our teacher models and students models all use the transformer architecture, which has achieved state-of-thh-aar accuracy and is widely used in recent NMT research. Model configurations listed in 1. We train all transformer models using the implementation in Fairseq (Ott et al., 2019). We use Adam optimizer (Kingma & Ba, 2014) with So = 0.9, 81 0.98, = - Learning rate 0.0001 and dropout rate is 0.3. At inference time, we use beam search with beam size of and length penalty 1.0.
5.3 RESULTS
Our results on NMT tasks are reported in Table.2. We tune hyper-parameters À and top-k on vali- dation set where À 0.5 for KD which is consistent with that in Kim & Rush (2016). We observed that KA consistently outperforms
on both De-En (high-resource) and Th-En (low-resource) tasks by 0.7 1.1 BLEU score. In addition, alent to GAN. The accuracy lies between KA and KD. The results say that KA does help avoiding exploration bias and further close the gap between training and inference.
also test JSD (i.e., 1/ KA 1 KD), which is equiv-
Exploration. Similar to Sec.3.2, we evaluate the performance by allowing only top-k tokens to be available. We vary & from 2 to 16 and conduct the experiments validation set. In Fig. 4a,

---

Under review as a conference paper at ICLR 2020
Table 3: Top- 10 tokens in KD and KA epochs. ([A] denotes the token the models are trying produce the current position and is the unique token included in one approach but not the other)
SRC: werde
dieser Wache TRG: I will end] to this week
SRC:Er kante sich nicht frei hewegen.
TRG: He could [not] move froely
look, be, tike, deall do, work care, consiler, address, mike
not, he, do, move, T. nover, handlyy avoid, have, romain avoid,
look, be, lake, deal, work, make, consider. gel
be, do,
stup, have, also
deal,
consider, concern
move, nover, slop, have
ok, be, take, deal, make, work, do, care, get concern look, be, take, deal. work, do, make, care, get, concern
not be, move, do, never, no, avoid, have, make. stop not be, move, do, avuid, no, make, never, have, also
look, be, take, deel. do, work, care. consider, address, make look, tike. be, deall, care, woek, do, make, try, consider
not, be, do, move, never, hardly, avoid. have, remain not be, mowe, aooid, never, do, no, fail, have, resist
KA
look, deal.
consider, do, give
be, never, hardly, avoid, makie, also not, move, be, avoid, never, hardly, no, da, act, move, be, avoid never, no, hardly, do, make: go
look, be, take, deal, work, look, be, take, disll, make, work, see. do, get, consider
consider. do, care. try
see that the distribution is noisy because the accuracy of either KD or KA eventually goes down and when using the full information, i.e., k [V], KD achieves 35.41 BLEU while KA achieves 35.79 BLEU, which are far away from the best. Moreover, KA is able to learn more from noise because KD directly goes down as l increases, while the accuracy of KA goes up first and then drops after k = 8.
To further analyze the capability in exploration, we tempt to count the tokens in the top-k which have never been included in previous epochs.
Entropy
4
(yr E Top-k (y:((h)))
& $ Top-k (y(y) Vj <
KA
-D-KD
0
50
100
150
200
where the superscript denotes epoch. We randomly sam- ple 2000 sentence pairs from WMT' 17 De-En training data. In Fig. 4b, we see that there are much more novel to- kens in KA than that in KD. Table.3 demonstrates two ex- amples. a) Given the prefix tokens "I will" and the source sentence, KA probably imagines
Epoch
Figure 5: Entropy over epochs.
try to attend? give attention? see to?
which provide different ways to express the meaning of attend, while KD just try to explore concern. b) when predicting the token no, KA proposes a set of negative tokens
fail, resist st, refrain
and even go which may relate to the next token move.
Fig. 5 shows that the entropy KA goes up while the entropy of KD dropping during fine-tuning. This also indicates that KA motivates exploration more than KD.
6 CONCLUSION
We proposed learning principle where student models actively ask for advice from teacher models. The theoretical analysis proves that it helps to alleviate exposure bias in sequence generation and improve the capability of generalization on unseen data. In general, our approach is extendable to phase-level or sequence-level, but we simply focus on token-level due to practical issues. How- ever, our token-level KA approach is complementary to sequence-level KD by using the outputs of teacher models as extra data. Experimental results on benchmark translation data sets show that our approach improves over previous methods of leveraging large models (or ensembles) to improve the quality of smaller, more efficient models.
REFERENCES
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Sys- fems, pp. 1171-1179, 2015.
8

---

Under review as a conference paper at ICLR 2020
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoe V. Le, and Ruslan Salakhut- dinov. Transformer-xl xl: Attentive language models beyond a fixedleentth context. CoRR, abs//101.008800 2019. URL http: httpii.ivvvv..../........//.. www.ooogg/g...o.ooooo...oss.eese 1901. 02860.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04800 2018. URL http://aa /arxivv owg/abs/ga//oeoooooo...oo.. abs/1810 04805.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 02531, 2015.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. 16Xiv:1606.000 07947, 2016.
arXiv preprint
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 1412.699,, 2014.
arXiv preprint
Vijay R Konda and John N Tsitsiklis, Actor-critic algorithms. In Advances in neural information processing systems, pp. 1008-1014, 2000.
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image cap- tioning via policy gradient optimization of spider. In Proceedings of the IEEE international conference on computer vision, pp. 873-881, 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining abs / 1907 1929..
approach. CoRR, abs/1907.11699 1692, 2019. URL http: htttt::/aaxiv.oooo
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv: 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language under- standing by generative pre- pre-training. a.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level train- ing with recurrent neural networks. arXiv preprint arXiv:1511.0673: 2015.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint Xiiv:1500.0090)0 2015.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances neural information processing systems, pp. 3104-3112, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762. 2017. URL
org/ abs/1706 03762.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
A APPENDIX
A.1 DERIVATIVES
(xz(q(=)lppz))) The objective is
( s.t. Σp(#) (
Minimize Eg(x) log(
(x))
(13)
9

---

Under review as a conference paper at ICLR 2020
We write the Lagrangian for Eqn. 13 as
((x)
Lipp (r, A) = Σ ((x) log(
A
Σ p(x) -
(14)
(
Method of Lagrangian multipliers involves setting the derivative of Lil w.r.t ((2) p.r. to
aLqlp Öp(z)
9(x)
Gallp
(15)
p(z)
Using the fact that Σ p(z) 1, we can show that X 1.
+xz(p(=))lq(z)). The objective is
((x)
Minimize Ep(x) log(
log
s.t. Ep(x)=1 =1
(16)
q(x)
We write the Lagrangian for Eqn. as
L (z, Σ Method of Lagrangian multipliers involves setting the derivative of L "lu w.r.t p(x) t,
Ep(z) log(
P
X Σ p(z) 1)
(17)
dLpli 11+ + + log ((r)
Gplqq 0p(x)
= 0
(18)
Using the fact that pxx) = 1, we can show that = -1
A.2 PROPERTY
We'll 'll prove the key properties in S.c.3.1.2.
(a) When qzx) > p(r), we have 1
q(w) and log p(z) p(x)
< 0. And, when q(x) < p(x),
q(z)
we have 1 q(a) 0 and log p(a)
0.
p
q(x)
(b) Let's first consider the function 2 - log z
if =
log 2
(19)
-1
It's easy to prove because when < 1, the gradient 1 1 < 0 and when > 1, the gradient 1 ! 0. Thus, the function reaches the minimum value
When q(x) > p(z), we have
[Gllp(z)] - (Gp]ia(z))) = q(x))
p(
log
(20)
p(z)
((x)
((x))
log
(21)
(
p(x)
When qxx) < <(z), we have
((x)
(
[Gllp(=)] [x)] =11
log
(22)
p(x)
q(r)
q(1)
((r)
=1
log
(23)
((x)
((x)
10


Reviews:
[{"review_id": "SylRvuv6KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of training small models to mimic large models but, in constrast to knowledge distillation, minimize the reverse-KL between the teacher and the model instead of the forward-KL.\n\nThe authors notice that there is an interesting interaction between beam search (i.e. focusing only on the top-k tokens) and distillation. By minimizing the forward-KL, distillation focuses the student on having the non-negative mass on all the words selected by the teacher. However, the authors argue that minimizing the reverse-KL makes more sense: to only include tokens in the students that are present in the teacher.\n\nThe paper then spends time explaining the difference between minimizing the KL (KD) or the reverse KL (their proposed KA) and show some experiments validating their methods.\n\nI think the paper is well-written, but can be sometimes difficult to follow. For example, they introduce the notation p_\\theta and q_\\phi without specifying who is the teacher and who is the student (I assumed that q was the teacher and p the student as in the introduction). The paper spend a bit of time explaining the qualitative difference between minimizing the KL or the reverse-KL. Even though it is useful, I believe it is well known in the community and can be found in multiple standard references (e.g. (Murphy, 2012) or this online class on graphical models: https://ermongroup.github.io/cs228-notes/inference/variational/). I don't think this constitutes a contribution yet the authors spend a fair amount of the paper on that particular topic.\n\nThe idea is quite simple but seems to be effective (up to +1.9 BLEU on German to English and +0.6 BLEU on Thai to English). I think it would have been useful to say how each model were tuned for fair comparison (e.g. how was the learning rate chosen?). I would have also like to see more tasks, like language modelling, question answering or text summarization.\n\nI also think the use of the term `actor-critic' is misleading given that, as far as I understand, there is no reinforcement learning in this paper. Section 3.1.2 is really confusing: are you referring to the derivative of the KL between two finite-dimensional vectors? Is there a lagrangian because you are somewhat taking the derivative on the simplex?\n\nOverall, this paper proposes an interesting trick that seems to work in practice but the novelty remains limited.\n\n(Murphy, 2012) Machine Learning: a Probabilistic Perspective. Murphy, Kevin. 2012.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkggMGYTtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper introduces Knowledge Acquisition (KA), i.e., KL-divergence in the reverse order as the loss function to train student models for sequence-to-sequence tasks, and experiments were done on WMT’17 De-En and IWSLT’15 Th-En translation tasks. \n\nPros: \nThe paper is clearly written. The authors clearly show the reason to use KL-divergence in the reverse order. They provide a concrete analysis of the effects of minimizing the proposed KA loss function to alleviating exposure bias.\nSome figures like Fig. 1 and Fig. 2 helps to show the paper.\n\nCons:\n As the analysis of the authors, KL-divergence in the reverse order is an alternative of KL-divergence for training student models on sequence generation tasks. However, I have a little concern that the contribution is relatively limited since it is known that KL-divergence is not symmetric and the proposed KA (KL-divergence in the reverse order) may be thought a little straightforward. In addition, KA is only investigated on token-level, as the authors said, “due to practical issues”. \nThe experiments in this paper may be thought insufficient. Specifically, the authors only establish teacher and student model baselines by themselves. It is reasonable that the proposed method could be compared with other similar KD methods, like [1] mentioned in the paper. Another concern is whether the proposed KA method can be applied in current sequence-to-sequence state-of-the-art teacher models, such as GPT-2 [2] and XLM [3], and whether it is still effective. \n\n\nReference\n[1] Yoon Kim, Alexander M. Rush. Sequence-Level Knowledge Distillation. EMNLP 2016\n[2] Alec, Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog 2019.\n[3] Guillaume Lample, Alexis Conneau. Cross-lingual Language Model Pretraining. CoRR abs/1901.07291\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgFPO-W5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Overview]\n\nIn this paper, the authors proposed a new method called knowledge acquisition (KA) for distilling the learned knowledge from the teacher model to the student model. Unlike the conventional KL-divergence based knowledge distillation method, the authors take the reverse version which learns the student to increase the precision. The paper gave a thorgough analysis on the proposed KA strategy and compared with other strategies like KL and JS divergence. On the sequence generation task (translation), the authors showed that the proposed KA strategy achieved better performance compared with KD based methods when distilling the knowledge from a teacher model to a student model.\n\n[Pros]\n\n1. The authors proposed a new strategy to perform the knowledge distillation from a teacher model to student model for sequence generator. To improve the precision of the student model, the authors proposed to invert the formula of KL divergence, i.e., the position of prediction probability from teacher and student models.\n\n2. The authors presented a thorough analysis on the proposed KA strategy and compared it with KL strategy in terms of the precision and recall for the student models. I think it is very readable and understandable. This analysis align with those put on generative adversarial network.\n\n3. The authors performed the experiments on the translation tasks showing that the proposed KA strategy outperforms both KL and JS strategy in terms of the generation performance. Also, the authors ablated the number of top reference tokens from the teacher model and showed that using a reasonable number of top tokens is important to help alleviate the noised in the teacher model.\n\n[Cons]\n\nThe main concern about the proposed method is whether it can be used as a generic strategy for transferring the knowledge from the teacher model to student model.\n\n1. First, I have a doubt on the stability of the proposed strategy. In my opinion, the improvements on the sequence generation tasks are mainly due to the tuned hyper parameters for the training, especially the lambda in Eq(12), which is tuned at the validation set. It controls how much to modulate the prediction distribution of student model toward that of teacher model. However, a less tuned lambda would cause either over curve fitting or under curve fitting. As a result, the authors should: 1) first show how the performance would be affected by varying the lambda in the formula; 2) from the reading, I did not see whether the lambda was tuned as well for KD or (KD + KA) / 2. If not, then for fair comparison, the authors should tune the lambda for the KD and (KD + KA) / 2 strategy as well. At some point, I would think the combination of KD and KA would be better than either of them.\n\n2. Second, some experimental results are somewhat counter-intuitive to me. These are two folds: a) In Figure 4(b), we can see that the KA strategy has learned to generate more new tokens compared with KD. This is a bit strange to me because, KA will focus on the precision instead of recall. To me, pushing the precision will high likely sacrifice the recall and thus the number of novel tokens generated by the model. b) similarly, in Figure 5, it is shown that KA has generally higher entropy than KD. This is also a bit counter intuitive. In Eq(6), it is obvious that the proposed strategy has a entropy term which will be reduced when we want to reduce the KL divergence during the training time. From Figure 5, KA and KD start from the same point (I guess it is because the same pre-trained student model) is used. However, for KA, the entropy start to increase and then converge to a stable number which is consistently higher than KD strategy.\n\n3. Third, Figure 4(a) also indicates some thing. When only the top few tokens are used to transfer the knowledge from teacher model to student model, KA focus on the precision of a small subspace, which tends to have few modes. However, when the number of tokens is increased, the mode number would also increase drastically. i guess that’s why the both strategies finally become very close to each other, and the minor gap between them is probably due to the benefit from hyper-parameter fine-tuning.\n\nBesides the above comments. there are some minor points which are missed in the paper:\n\n1. As pointed above, it is not clear whether the same tuning is also applied to KD and (KD + KA) / 2. the authors should mention this in the experiment section.\n\n2. It is also not clear how many tokens is used for reporting the numbers in Table 2. Is it the whole vocabulary side? If this is the case, the gap between KA and KD on validation set are pretty close while more significant on test set.\n\n3. In Eq (11), should there be a minus sign before the expectation?\n\n4. Also, is there any more comment on why it is hard to train the student model joint from scratch? what will happen in this case?\n\n[Summary]\n\nIn this paper, the authors proposed a new strategy called Knowledge Acquisition which is used for distilling the knowledge learned from  teacher model to the student model. Different from KD strategy, it inverted the position of probability distributions for teacher and student models. By this way, the KA strategy learns a student model which can achieves higher precision. The proposed strategy is evaluated on sequence generation, particularly translation task.  However, as pointed above, in my opinion, there are some counter-intutive observations in the experimental results. It would be good if the authors can address these concerns in the rebuttal.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows a nice idea to transfer knowledge from larger sequence models to small models. However, all the reivewers find that the contribution is too limited and the experiments are insufficient. All the reviewers agree to reject.
Decision: Reject