Prior Knowledge Representation for Self-Attention Networks

1 INTRODUCTION . Self-attention networks ( SANs ) have attracted increasing attention in the natural language processing community . Instead of using complex recurrent or convolutional neural networks ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) , SANs first use the positional encoding mechanism Gehring et al . ( 2017 ) to encode order dependencies in the language . The learned positional embedding is then added to corresponding word embedding to obtain an input representation , based on which SANs perform ( multi-head ) and stack ( multi-layer ) self-attentive functions ( Vaswani et al. , 2017 ) in parallel to learn language representation . The SAN-based models are iteratively optimized to model language knowledge on the whole training dataset , which has achieved state-of-the-art performance in many natural language processing tasks pairs ( Barrault et al. , 2019 ; Oepen et al. , 2019 ; Weissenbacher & Gonzalez-Hernandez , 2019 ; Demner-Fushman et al. , 2019 ) . Despite the success , SANs gradually model language knowledge on the batch-level datasets and does not consider the prior knowledge on the whole dataset from the beginning of training , which may decrease its language representation capability . For example , the SAN-based neural machine translation ( NMT ) model often mistranslate into words that seem to be natural in the target language sentence , but do not reflect the original meaning of the source language sentence ( Arthur et al. , 2016 ; Wang et al. , 2017a ) . As a result , the NMT model produces fluent yet sometimes inadequate translations ( Tu et al. , 2016 ; 2017 ) . To address this issue , recent studies explored the prior knowledge which has the stringer ability to model the fluency of translation in traditional SMT ( Koehn et al. , 2003 ; Liu et al. , 2006 ; Chiang , 2007 ; Liu et al. , 2007 ) . Take the prior translation lexicon knowledge as an example , Arthur et al . ( 2016 ) directly biased or interpolated the bilingual lexicon translation distribution with the output of the softmax layer of NMT to improve the translations of infrequency words . An auxiliary classifier ( Zhao et al. , 2018 ; Wang et al. , 2018 ) were employed to integrate the SMT recommendations with NMT generations to generate the faithful translation . In addition , the phrase translation rules was as the recommendation memory to make better predictions in NMT ( Wang et al. , 2017b ; Zhao et al. , 2018 ) . Although these studies successfully improved the issue of inadequate translations in NMT , they tended to focus on exploring the prior translation lexicon knowledge by using their specific methods . In other words , these unique methods make it difficult to explore other prior knowledge in a universal way and to determine which of the prior knowledge and the unique method this improvement comes from . Meanwhile , these studies directly utilized the probability distribution of the prior knowledge and lacked the neural network ’ s ability to semantically generalize , while will further hinder the language representation ability of SANs . In this paper , we propose a simple and general representation method to introduce the prior knowledge into SANs . In particular , we package the prior knowledge related with one source sentence to a continuous space matrix , which allows SANs to utilize the prior knowledge from the beginning of training , thereby better performing language representation in a universal way compatible with neural networks . To maintain the simplicity and flexibility of the SANs , we use the prior knowledge representation in parallel and stacked ways to learn the representation of the input sentence . Furthermore , we use the proposed method to explore one prior word frequency knowledge for the monolingual data and other prior translation lexicon knowledge for the bilingual data , respectively . Empirical results on two widely used translation data sets , including WMT14 English→German and WMT17 Chinese→English , to verify the effectiveness and universality of the proposed method over a strong Transformer-based baseline . 2 SELF-ATTENTION NETWORKS . The self-attention networks ( SANs ) ( Vaswani et al. , 2017 ) is composed of a stack of N identical layers , each of which includes two sub-layers . Formally , given a input sentence with the length J , X= { x1 , x2 , · · · , xJ } , the positional encoding mechanism ( Gehring et al. , 2017 ) is used to compute a positional embedding of each word based on its position index . The positional embedding is then added to the corresponding word embedding as an combined embedding , thereby gaining a sequence of input representation H0= { v1 , v2 , · · · , vJ } . Moreover , the stacked SANs is organized as follows : Hn = LN ( SelfAttn ( Qn−1 , Kn−1 , Vn−1 ) + Hn−1 ) , Hn = LN ( FFNn ( Hn ) + Hn ) , ( 1 ) where SelfAttn ( · ) , LN ( · ) , and FFNn ( · ) are self-attention module , layer normalization ( Ba et al. , 2016 ) , and feed-forward network for the n-th identical layer , respectively . Qn−1 , Kn−1 , and Vn−1 are query , key , and value matrices that are transformed from the ( n-1 ) -th layer Hn−1 . For example , Q0 , K0 , and V0 are packed from the H0 learned by the positional encoding mechanism ( Gehring et al. , 2017 ) . In particular , SelfAttn ( · ) is applied on the { Qn−1 , Kn−1 , Vn−1 } of the n-1 layer : SelfAttn ( Qn−1 , Kn−1 , Vn−1 ) = softmax ( Qn−1Kn−1 > / √ dmodel ) Vn−1 , ( 2 ) where dmodel is the dimension size of the query and key vectors . As a result , the output of the N -th layer HN is the representation of the input sentence . Moreover , the self-attention mechanism can be further refined as multi-head self-attention to jointly attend to the information from different representation sub-spaces at different positions . 3 PRIOR KNOWLEDGE REPRESENTATION . In this section , we propose a simple and general representation method to encode the prior knowledge , which allows SANs to model the prior knowledge in a manner compatible with neural networks . Given a input sentence X= { x1 , x2 , · · · , xJ } with the length J , we represent the associated prior knowledge as a matrix M : M = m11 m21 · · · mK1 m12 m22 · · · mK2 ... ... . . . ... m1J m2J · · · mKJ , ( 3 ) where each row denotes the prior knowledge related with word xj and each element mtj is a fixed size vector . Also , M is packed into a key and value matrix pair { K , V } for the prior knowledge . The prior { K , V } and the current Q are the input to the self-attention mechanism ( see Eq . ( 2 ) ) to learn a prior knowledge representation PK for the input sentence X : PK = LN ( SelfAtt ( Q , K , V ) + H ) , ( 4 ) where Q is transformed from the current sentence representation H , for example , the n-th layer output Hn of the stacked SANs in Eq. ( 1 ) . Finally , PK is the expected prior knowledge representation ( PKR ) . Later , we will apply the proposed method to one prior word frequency knowledge for the monolingual data and other prior translation lexicon knowledge for the bilingual data , respectively , thereby enhancing the language representation . 3.1 WORD FREQUENCY BASED PRIOR KNOWLEDGE REPRESENTATION . Recent studies have shown the effectiveness of the prior word frequency information for improving the translations of content words ( Arthur et al. , 2016 ; Wang et al. , 2018 ; He et al. , 2019 ; Chen et al. , 2020 ) . Intuitively , these content words generally have more meanings and lower word frequency than function words in one sentence . To capture the prior knowledge , we use word frequency information to enhance the importance of content words in the sentence representation . Inspired by the previous studies ( Setiawan et al. , 2007 ; 2009 ; Zhang & Zhao , 2013 ; Chen et al. , 2020 ) , we use the frequency of each word w on the monolingual corpus F to distinguish content words and function words in the input sentence . To this end , let the B most frequent words in the monolingual corpus denote the function words while the remaining words are regarded as content words for the input sentence . In other words , let the prior word frequency knowledge of each word denote a binary mask : fj = { 0 , xj ∈ B 1 , xj /∈ B ( 5 ) Furthermore , the prior word frequency knowledge of the input sentence is a diagonal matrix according to Eq . ( 3 ) : MF = f1 0 · · · 0 0 f2 · · · 0 ... ... . . . ... 0 0 · · · fJ . ( 6 ) Finally , we use MF to mask the representation of function words in H , and thereby gain key and value matrices { KF , VF } to learn a prior word frequency based PKR PKF for the input sentence : PKF = LN ( SelfAtt ( Q , KF , VF ) + H ) , ( 7 ) where Q is transformed from the current sentence representation H , for example , Hn in Eq . ( 1 ) . 3.2 BILINGUAL TRANSLATION LEXICON BASED PRIOR KNOWLEDGE REPRESENTATION . We apply the proposed PKR method into the prior bilingual translation lexicons in SMT ( Brown et al. , 1993 ) . Discretely , each source word xj has multiple associated target translation options retrieved from the bilingual translation lexicon table . For simplicity , we select the top L target translation options as the prior translation knowledge of word xj are remained according to their lexicon translation probabilities and the entry corresponds to the target word . Formally , all retrieved lists for the input sentence X are constructed as a J×K×dmodel prior translation knowledge matrix MT according to Eq . ( 3 ) : MT = t11 t21 · · · tL1 t12 t22 · · · tL2 ... ... . . . ... t1J t2J · · · tLJ . ( 8 ) where each row corresponds to the retrieved list of the word xj , and the entry corresponds to the embedding of the target word . In particular , when the number of the associated target words for one source word xj is less than L , we use the embedding of a special placeholder “ / ” to fill the j-th row in MT up to L. Also , the MT is packed into a key and value matrix pair { KT , VT } . { Q , KT , and VT } are the input to Eq . ( 4 ) to learn a prior translation knowledge representation PKT for the source sentence : PKT = LN ( SelfAtt ( Q , KT , VT ) + H ) , ( 9 ) where Q is transformed from the current sentence representation H , for example , Hn in Eq. ( 1 ) . 4 SANS WITH PRIOR KNOWLEDGE REPRESENTATION Intuitively , the prior knowledge assists SANs to enhance the language representation in the SAN-based models of NLP tasks . In other words , the prior knowledge plays an auxiliary role during the processing of learning language representation . Therefore , we compute a gate scalar gn ∈ [ 0 , 1 ] to weight the expected importance of the prior knowledge representation PK at the n-th layer : gn = Sigmoid ( Ung H n + Wng PK ) , ( 10 ) where the Sigmoid is an active function , and Wng∈Rdmodel×1 and Ung∈Rdmodel×1 are model parameters . We then fuse Hn and PK through the gate scalar gn to learn the sentence representation at the nth layer : Hn = Hn + gn · PK . ( 11 ) Also , the new stacked SAN is formally denoted as the modified version of Eq . ( 1 ) : Hn = LN ( SelfAttn ( Qn−1 , Kn−1 , Vn−1 ) +Hn−1 ) , PKn = LN ( SelfAttn ( Qn , K , V ) + Hn ) , gn = Sigmoid ( Ung H n + Wng PK n ) , Hn = Hn + gn · PKn , Hn = LN ( FFNn ( Hn ) +Hn ) , ( 12 ) where H0 is the initialized sentence representation H0 as in the Section 2 . As a result , there is a more effective sentence representation HN for the input sentence . Later , HN will be fed into the Decoder of the SAN-based model to model NLP tasks , for example , machine translation task .

Summary: This paper proposes a method to introduce **prior knowledge** into Transformer-based sentence encoders, here in the context of neural machine translation (NMT). More concretely, the prior knowledge is represented in the form of a matrix $\boldsymbol{M}$, where each row denotes a vector of prior knowledge associated with each word $x_i$. The prior knowledge matrix $\boldsymbol{M}$ is then represented as a (key, value) pair that can be attended by the query matrix $\boldsymbol{Q}$ (the same query matrix as used in the main NMT component) using a standard Transformer self-attention mechanism. This procedure results in a prior knowledge representation matrix $\boldsymbol{PK}$, which is then combined with the standard Transformer encoder output using a simple gating mechanism.