Prior Knowledge Representation for Self-Attention Networks

1 INTRODUCTION . Self-attention networks ( SANs ) have attracted increasing attention in the natural language processing community . Instead of using complex recurrent or convolutional neural networks ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) , SANs first use the positional encoding mechanism Gehring et al . ( 2017 ) to encode order dependencies in the language . The learned positional embedding is then added to corresponding word embedding to obtain an input representation , based on which SANs perform ( multi-head ) and stack ( multi-layer ) self-attentive functions ( Vaswani et al. , 2017 ) in parallel to learn language representation . The SAN-based models are iteratively optimized to model language knowledge on the whole training dataset , which has achieved state-of-the-art performance in many natural language processing tasks pairs ( Barrault et al. , 2019 ; Oepen et al. , 2019 ; Weissenbacher & Gonzalez-Hernandez , 2019 ; Demner-Fushman et al. , 2019 ) . Despite the success , SANs gradually model language knowledge on the batch-level datasets and does not consider the prior knowledge on the whole dataset from the beginning of training , which may decrease its language representation capability . For example , the SAN-based neural machine translation ( NMT ) model often mistranslate into words that seem to be natural in the target language sentence , but do not reflect the original meaning of the source language sentence ( Arthur et al. , 2016 ; Wang et al. , 2017a ) . As a result , the NMT model produces fluent yet sometimes inadequate translations ( Tu et al. , 2016 ; 2017 ) . To address this issue , recent studies explored the prior knowledge which has the stringer ability to model the fluency of translation in traditional SMT ( Koehn et al. , 2003 ; Liu et al. , 2006 ; Chiang , 2007 ; Liu et al. , 2007 ) . Take the prior translation lexicon knowledge as an example , Arthur et al . ( 2016 ) directly biased or interpolated the bilingual lexicon translation distribution with the output of the softmax layer of NMT to improve the translations of infrequency words . An auxiliary classifier ( Zhao et al. , 2018 ; Wang et al. , 2018 ) were employed to integrate the SMT recommendations with NMT generations to generate the faithful translation . In addition , the phrase translation rules was as the recommendation memory to make better predictions in NMT ( Wang et al. , 2017b ; Zhao et al. , 2018 ) . Although these studies successfully improved the issue of inadequate translations in NMT , they tended to focus on exploring the prior translation lexicon knowledge by using their specific methods . In other words , these unique methods make it difficult to explore other prior knowledge in a universal way and to determine which of the prior knowledge and the unique method this improvement comes from . Meanwhile , these studies directly utilized the probability distribution of the prior knowledge and lacked the neural network ’ s ability to semantically generalize , while will further hinder the language representation ability of SANs . In this paper , we propose a simple and general representation method to introduce the prior knowledge into SANs . In particular , we package the prior knowledge related with one source sentence to a continuous space matrix , which allows SANs to utilize the prior knowledge from the beginning of training , thereby better performing language representation in a universal way compatible with neural networks . To maintain the simplicity and flexibility of the SANs , we use the prior knowledge representation in parallel and stacked ways to learn the representation of the input sentence . Furthermore , we use the proposed method to explore one prior word frequency knowledge for the monolingual data and other prior translation lexicon knowledge for the bilingual data , respectively . Empirical results on two widely used translation data sets , including WMT14 English→German and WMT17 Chinese→English , to verify the effectiveness and universality of the proposed method over a strong Transformer-based baseline . 2 SELF-ATTENTION NETWORKS . The self-attention networks ( SANs ) ( Vaswani et al. , 2017 ) is composed of a stack of N identical layers , each of which includes two sub-layers . Formally , given a input sentence with the length J , X= { x1 , x2 , · · · , xJ } , the positional encoding mechanism ( Gehring et al. , 2017 ) is used to compute a positional embedding of each word based on its position index . The positional embedding is then added to the corresponding word embedding as an combined embedding , thereby gaining a sequence of input representation H0= { v1 , v2 , · · · , vJ } . Moreover , the stacked SANs is organized as follows : Hn = LN ( SelfAttn ( Qn−1 , Kn−1 , Vn−1 ) + Hn−1 ) , Hn = LN ( FFNn ( Hn ) + Hn ) , ( 1 ) where SelfAttn ( · ) , LN ( · ) , and FFNn ( · ) are self-attention module , layer normalization ( Ba et al. , 2016 ) , and feed-forward network for the n-th identical layer , respectively . Qn−1 , Kn−1 , and Vn−1 are query , key , and value matrices that are transformed from the ( n-1 ) -th layer Hn−1 . For example , Q0 , K0 , and V0 are packed from the H0 learned by the positional encoding mechanism ( Gehring et al. , 2017 ) . In particular , SelfAttn ( · ) is applied on the { Qn−1 , Kn−1 , Vn−1 } of the n-1 layer : SelfAttn ( Qn−1 , Kn−1 , Vn−1 ) = softmax ( Qn−1Kn−1 > / √ dmodel ) Vn−1 , ( 2 ) where dmodel is the dimension size of the query and key vectors . As a result , the output of the N -th layer HN is the representation of the input sentence . Moreover , the self-attention mechanism can be further refined as multi-head self-attention to jointly attend to the information from different representation sub-spaces at different positions . 3 PRIOR KNOWLEDGE REPRESENTATION . In this section , we propose a simple and general representation method to encode the prior knowledge , which allows SANs to model the prior knowledge in a manner compatible with neural networks . Given a input sentence X= { x1 , x2 , · · · , xJ } with the length J , we represent the associated prior knowledge as a matrix M : M = m11 m21 · · · mK1 m12 m22 · · · mK2 ... ... . . . ... m1J m2J · · · mKJ , ( 3 ) where each row denotes the prior knowledge related with word xj and each element mtj is a fixed size vector . Also , M is packed into a key and value matrix pair { K , V } for the prior knowledge . The prior { K , V } and the current Q are the input to the self-attention mechanism ( see Eq . ( 2 ) ) to learn a prior knowledge representation PK for the input sentence X : PK = LN ( SelfAtt ( Q , K , V ) + H ) , ( 4 ) where Q is transformed from the current sentence representation H , for example , the n-th layer output Hn of the stacked SANs in Eq. ( 1 ) . Finally , PK is the expected prior knowledge representation ( PKR ) . Later , we will apply the proposed method to one prior word frequency knowledge for the monolingual data and other prior translation lexicon knowledge for the bilingual data , respectively , thereby enhancing the language representation . 3.1 WORD FREQUENCY BASED PRIOR KNOWLEDGE REPRESENTATION . Recent studies have shown the effectiveness of the prior word frequency information for improving the translations of content words ( Arthur et al. , 2016 ; Wang et al. , 2018 ; He et al. , 2019 ; Chen et al. , 2020 ) . Intuitively , these content words generally have more meanings and lower word frequency than function words in one sentence . To capture the prior knowledge , we use word frequency information to enhance the importance of content words in the sentence representation . Inspired by the previous studies ( Setiawan et al. , 2007 ; 2009 ; Zhang & Zhao , 2013 ; Chen et al. , 2020 ) , we use the frequency of each word w on the monolingual corpus F to distinguish content words and function words in the input sentence . To this end , let the B most frequent words in the monolingual corpus denote the function words while the remaining words are regarded as content words for the input sentence . In other words , let the prior word frequency knowledge of each word denote a binary mask : fj = { 0 , xj ∈ B 1 , xj /∈ B ( 5 ) Furthermore , the prior word frequency knowledge of the input sentence is a diagonal matrix according to Eq . ( 3 ) : MF = f1 0 · · · 0 0 f2 · · · 0 ... ... . . . ... 0 0 · · · fJ . ( 6 ) Finally , we use MF to mask the representation of function words in H , and thereby gain key and value matrices { KF , VF } to learn a prior word frequency based PKR PKF for the input sentence : PKF = LN ( SelfAtt ( Q , KF , VF ) + H ) , ( 7 ) where Q is transformed from the current sentence representation H , for example , Hn in Eq . ( 1 ) . 3.2 BILINGUAL TRANSLATION LEXICON BASED PRIOR KNOWLEDGE REPRESENTATION . We apply the proposed PKR method into the prior bilingual translation lexicons in SMT ( Brown et al. , 1993 ) . Discretely , each source word xj has multiple associated target translation options retrieved from the bilingual translation lexicon table . For simplicity , we select the top L target translation options as the prior translation knowledge of word xj are remained according to their lexicon translation probabilities and the entry corresponds to the target word . Formally , all retrieved lists for the input sentence X are constructed as a J×K×dmodel prior translation knowledge matrix MT according to Eq . ( 3 ) : MT = t11 t21 · · · tL1 t12 t22 · · · tL2 ... ... . . . ... t1J t2J · · · tLJ . ( 8 ) where each row corresponds to the retrieved list of the word xj , and the entry corresponds to the embedding of the target word . In particular , when the number of the associated target words for one source word xj is less than L , we use the embedding of a special placeholder “ / ” to fill the j-th row in MT up to L. Also , the MT is packed into a key and value matrix pair { KT , VT } . { Q , KT , and VT } are the input to Eq . ( 4 ) to learn a prior translation knowledge representation PKT for the source sentence : PKT = LN ( SelfAtt ( Q , KT , VT ) + H ) , ( 9 ) where Q is transformed from the current sentence representation H , for example , Hn in Eq. ( 1 ) . 4 SANS WITH PRIOR KNOWLEDGE REPRESENTATION Intuitively , the prior knowledge assists SANs to enhance the language representation in the SAN-based models of NLP tasks . In other words , the prior knowledge plays an auxiliary role during the processing of learning language representation . Therefore , we compute a gate scalar gn ∈ [ 0 , 1 ] to weight the expected importance of the prior knowledge representation PK at the n-th layer : gn = Sigmoid ( Ung H n + Wng PK ) , ( 10 ) where the Sigmoid is an active function , and Wng∈Rdmodel×1 and Ung∈Rdmodel×1 are model parameters . We then fuse Hn and PK through the gate scalar gn to learn the sentence representation at the nth layer : Hn = Hn + gn · PK . ( 11 ) Also , the new stacked SAN is formally denoted as the modified version of Eq . ( 1 ) : Hn = LN ( SelfAttn ( Qn−1 , Kn−1 , Vn−1 ) +Hn−1 ) , PKn = LN ( SelfAttn ( Qn , K , V ) + Hn ) , gn = Sigmoid ( Ung H n + Wng PK n ) , Hn = Hn + gn · PKn , Hn = LN ( FFNn ( Hn ) +Hn ) , ( 12 ) where H0 is the initialized sentence representation H0 as in the Section 2 . As a result , there is a more effective sentence representation HN for the input sentence . Later , HN will be fed into the Decoder of the SAN-based model to model NLP tasks , for example , machine translation task .

Summary: This submission works on the neural machine translation problem. The authors extend the previous works on leveraging language statistics or prior knowledge (SMT model or whatever) in LSTM based NMT models in self-attention based NMT models, Transformer model. The authors propose two alternatives to incorporate prior knowledge, which are the word frequency information for the monolingual data and the prior translation lexicon information for the bilingual data. These resources are integrated into the hidden representations from the self-attention computations and then the two output hidden representations are gated together for upper computations. The experiments are conducted on two typical NMT datasets: WMT14 En->De and WMT17 Zh->En, the results show that the proposed method can improve the NMT model performances.