Proactive Sequence Generator via Knowledge Acquisition

1 INTRODUCTION . We have recently witnessed rapid progress sequence models for natural language processing ( NLP ) . In particular , transformers ( Vaswani et al. , 2017 ; Dai et al. , 2019 ) and BERT ( Devlin et al. , 2018 ; Liu et al. , 2019 ) have been established as state-of-the-art approaches for sequence generation and language modeling respectively . However , the resulting sequence models are often exceedingly large . For example , a big transformer with 6 layers , 1024 dim and 16 heads has∼ 100M parameters , which can make production deployment impractical , especially on mobile devices . Unfortunately , learning small sequence models is very challenging because of the nature of deep learning - curve fitting . Training deep neural networks is data hungry and needs bigger “ brains ” . To deal with the challenges , Kim & Rush ( 2016 ) consider applying Knowledge Distillation approaches ( Hinton et al. , 2015 ) to match the distribution of the teacher model . At token-level , KD approaches minimize DKL ( q ( yt|y < t , x ) ‖p ( yt|y < t , x ) ) where q ( yt|y < t , x ) [ teacher ] or p ( yt|y < t , x ) [ student ] is a probability distribution over tokens in the vocabulary conditioned on previous tokens y < t and the source sentence x . In other words , the KD approaches maximize recall - the fraction of the top-k ” tokens in q ( yt|y < t , x ) have been retrieved among the top-k′ in p ( yt|y < t , x ) . However , sequence modeling actually cares about precision - the fraction of the top-k′ tokens in p ( yt|y < t , x ) are included in the top-k ” of q ( yt|y < t , x ) , because imperfect predictions in the top-k′ cause exposure bias at inference time . To optimize precision , we attempt to minimize KL-divergence in the reverse order , that is , DKL ( p ( yt|y < t , x ) ‖q ( yt|y < t , x ) ) . Our approach can be thought of as a kind of Actor-Critic approach where the “ Actor ” is a student model asking for advice when learning to generate every single token and the “ Critic ” is a teacher model that gives logq ( yt|y < t , x ) ) as rewards . After receiving the feedback , “ Actor ” updates it ’ s belief accordingly , as shown in Fig . 1 . Unlike task-specific metrics such as BLEU score , teacher models summarize training data points and structure knowledge in an abstract , hierarchical manner . Therefore , they are able to offer more useful advice than matching n-grams with a finite number of human references . For example , assume there is a training example { SRC : “ Die Landschaft am Meer ist wunderschn ” , TRG : “ Amazing view along the sea ” } . During learning , given the first token “ Amazing ” , student models might ask “ How about saying amazing view ? ” , teacher models most likely give high rewards since it ’ s expected to go with “ Amazing view of the seaside , of the boats in the seaof the beach , from the boats· · · , · · · along the sea , of the birds flying over the water ” , where most of them express similar meanings with the human references even though they have mismatched n-grams . Contributions . To the best of our knowledge , we are the first to provide a depth-analysis of the effects of minimizing DKL ( p‖q ) on alleviating exposure bias . In addition , we enable stable reinforcement training via replacing task-specific metrics with high-capacity models . Finally , we improved +1.1 BLEU score on WMT ’ 17 de-en task and +0.7 BLEU score on IWSLT ’ 15 th-en task over token-level KD with up to relative 200 % improvement . 2 SEQUENCE GENERATION . Let ( x , y ) ∈ ( X , Y ) be an example translation pair , where x is the source sentence and y is the target sentence . yt is the token at position t and y < t is tokens before position t. Sutskever et al . ( 2014 ) propose sequence-to-sequence models which typically factorize the joint probability p ( y | x ) over a sequence of conditional probabilities with parameter θ : p ( y|x ) = T∏ t=0 pθ ( yt | y < t , x ) ( 1 ) where , pθ ( yt | y < t , x ) is the probability of the token yt given previous tokens y < t and the source sentence x . Basically , the preceding tokens y < t are encoded into the hidden states via a state transition function ht = f ( ht−1 , yt−1 ; x ) ( 2 ) By substituting Eqn . ( 2 ) for Eqn . ( 1 ) , we have p ( y|x ) = T∏ t=0 pθ ( yt | ht ) ︸ ︷︷ ︸ Policy π ( 3 ) This tells us that the sequence models , e.g. , RNNs , acts like a stochastic policy which picks a discrete action , i.e. , producing a token yt ∈ V , running on a world modelM with transition function f . Training . We minimize the cross-entropy loss LCE = − ∑ t logpθ ( yreft | y ref < t , x ) ( 4 ) where , yref denotes human references . At each position , the models are conditioned on the groundtruth tokens annotated by humans no matter what tokens are predicted by themselves . Inference . In sequence generation tasks , exact inference is intractable due to exponentially large search space . Instead , we apply an approximation inference algorithm - Beam Search ( BS ) . BS is a greedy heuristic search that maintains the top-B most likely partial sequences through the search tree , where B is referred to as the beam size . At each position , BS expands these B partial sequences to all possible beam extensions and then selects the B highest scoring among the expansions . Unlike training , ground-truth tokens are not available . The models have to use their own predictions in decoding . Evaluation . To evaluate the quality of generated sequences , we typically use metrics such as BLEU score to measure their n-grams overlap with human references . However , the standard metrics are problematic and none of them correlate strongly with human evaluation at the level of individual sentences . For example , given a human reference “ Amazing view along the sea ” , the sequence ” The scenery of the seaside is beautiful ” gets low BLEU score because there is no matching n-grams of order 2 , 3 , or 4 . In addition , human references are a few sentences lacking in diversity . For example , when asking more people , they might say “ A nice beach ” or “ what amazing view of the seashore ” . 2.1 DISCREPANCY AMONG PROCEDURES . Exposure bias . During training , the models only explore the training data distribution , but never get exposed to their own predictions . Searching in under-explored space causes errors . More importantly , such errors accumulate over time because of greedy search . For the example in Sec . 1 , assume there are no sentences in training data starting with “ Amazing ” . At position 1 : pick a token from p ( y| “ Amazing ” , x ) ⇒ “ cup ” At position 2 : pick a token from p ( y| “ Amazing cup ” , x ) ⇒ “ on ” At position 3 : pick a token from p ( y| “ Amazing cup on ” , x ) ⇒ “ table ” We can see that the poor token “ cup ” caused by the noisy distribution p ( y| “ Amazing ” , x ) makes the situation even worse . The distributions become more and more noisy and the generation goes far away . Sub-optimal models . The training objective is different with the metrics used in evaluation . To address this issue , some works attempt to directly optimize the metrics . It definitely helps improving the score , but we suspect that this might hurt the models because poor metrics discourage learning the semantic meanings . For example , the low score of “ The scenery of the seaside is beautiful ” makes the meanings of “ Amazing ” and “ beautiful ” or “ seaside ” and “ sea ” far from the other . 3 KNOWLEDGE ACQUISITION . Unlike knowledge distillation , we attempt to minimize DKL [ pθ ( yt | ht ) ‖qφ ( yt | ht ) ] ( 5 ) = − [ Eyt∼pθ ( yt|bt ) [ logqφ ( yt | ht ) ] +H [ pθ ( yt | ht ) ] ] ( 6 ) Eqn.6 can be explained as an Actor-Critic algorithm ( Konda & Tsitsiklis , 2000 ) that maximizes the expected rewards with an entropy term , where “ Critic ” : ( aka teacher models ) estimate the rewards of producing token yt given preceding tokens y < t and the source sentence x , i.e. , Q ( a = yt , s = ht ) = logqφ ( yt | ht ) ( 7 ) “ Actor ” : updates the policy pθ ( yt | ht ) guided by the ” Critic ” As opposed to behavior cloning , student models ask teacher models to evaluate their behaviors and then update accordingly when producing the next token . By comparing with KD that attempts to minimize DKL [ q‖p ] , our approach is actually a kind of reverse KD , which we call ” Knowledge Acquisition ” ( KA ) . 3.1 LEARNING STRATEGY . Without loss of generality , we simplify the problem to a 1-D problem where the student model p ( x ) is a single-modal Gaussian and the teacher model q ( x ) is a mixture of two Gaussians , as shown in Fig . 2 . Let ’ s take a depth-analysis of their objective functions and their derivatives w.r.t . p ( x ) to see what changes after swapping p ( x ) and q ( x ) . 3.1.1 OBJECTIVE FUNCTIONS . The entropy term in Eqn . ( 6 ) goes away in KD because q ( x ) is a fixed value and never be optimized during training . It ’ s clear that the entropy term helps to avoid over-fitting the model to human references and generalize well on unseen test data . 3.1.2 DERIVATIVES . Based on Lagrangian relaxation ( see Appendix A.1 ) , the derivatives of DKL w.r.t . p ( x ) in both orders are ∂DKL ( q ( x ) ||p ( x ) ) ∂p ( x ) = 1− q ( x ) p ( x ) ︸ ︷︷ ︸ Gq‖p ( x ) , ∂DKL ( p ( x ) ||q ( x ) ) ∂p ( x ) = log p ( x ) q ( x ) ︸ ︷︷ ︸ Gp‖q ( x ) ( 8 ) Eqn . ( 8 ) tells us that ( a ) q ( x ) > p ( x ) ⇒ Gq‖p ( x ) < 0 and Gp‖q ( x ) < 0 ( b ) q ( x ) < p ( x ) ⇒ Gq‖p ( x ) > 0 and Gp‖q ( x ) > 0 That is , the probability of x is pulled up when q ( x ) dominates p ( x ) , and pushed down otherwise . However , they take efforts at different levels ( see Appendix A.2 ) . In ( a ) , ∣∣Gq‖p ( x ) ∣∣ > ∣∣Gp‖q ( x ) ∣∣ while in ( b ) , ∣∣Gq‖p ( x ) ∣∣ < ∣∣Gp‖q ( x ) ∣∣ In other words , minimizing DKL ( q ( x ) ‖p ( x ) ) attempts to pull up the probability of x that underestimated by the student models while minimizing DKL ( p ( x ) ‖q ( x ) ) attempts to push down the the probability of x that was over-estimated by the student models . If we define Recallk′ ( p ( x ) , k ” ) = |p′s Top-k′ ∩ q′s Top-k ” | k ” ( 9 ) Precisionk ” ( p ( x ) , k′ ) = |p′s Top-k′ ∩ q′s Top-k ” | k′ ( 10 ) where , p′s Top-k′ is a set including k′ items having the highest rankings in p ( x ) and q′s Top-k ” is a set including k ” items having the highest rankings in q ( x ) . Clearly , minimizing DKL ( q ( x ) ‖p ( x ) ) is to maximize recall because its role is to increase the chance that x ’ s in q′s Top-k ” are truly picked . By contrast , minimizing DKL ( p ( x ) ‖q ( x ) ) is to maximize precision since its role is to avoid x ’ s that are over-estimated being picked by mistake . Overall , sequence modelling cares about precision because at test time , the generation is sensitive to the model ’ s predictions . The errors caused by inaccurate predictions in the top-k′ accumulate and make the generation brittle .

Summary: In this paper, the authors proposed a new method called knowledge acquisition (KA) for distilling the learned knowledge from the teacher model to the student model. Unlike the conventional KL-divergence based knowledge distillation method, the authors take the reverse version which learns the student to increase the precision. The paper gave a thorgough analysis on the proposed KA strategy and compared with other strategies like KL and JS divergence. On the sequence generation task (translation), the authors showed that the proposed KA strategy achieved better performance compared with KD based methods when distilling the knowledge from a teacher model to a student model.