Contextualized Role Interaction for Neural Machine Translation

1 INTRODUCTION . Existing deep learning approaches to natural language processing and machine translation ( MT ) are usually constructed as a two-stage process . In the first stage , each discrete token in an input sentence is converted into a continuous vector via a table lookup , resulting in a sequence of token representations . This sequence is then processed by the main part of the network for solving the problem at hand . In this paper , we focus on the first stage , the token representation . The standard table lookup , or word embedding layer , embeds each token into a high-dimensional , continuous vector space independently from the other tokens in the input . It was noticed earlier , for instance by Bengio et al . ( 2003 ) and Choi et al . ( 2017 ) , that this high-dimensional token vector encodes multiple aspects of the token ’ s meaning . It is then left for the subsequent layers of a neural network to accurately extract one of these aspects based on the context . This approach has been successful in applications such as machine translation ( Bahdanau et al. , 2015 ; Gehring et al. , 2017 ; Vaswani et al. , 2017 ) , where a lot of data is available and powerful architectures , such as recurrent networks , convolutional networks and self-attention , are used for the subsequent layers . In this paper , we examine whether we can enhance token representations by augmenting networks with a novel layer that captures and resolves the latent roles of each token based on the context . This layer , which we call a “ role interaction layer ” ( RIL ) , works on top of any sequence of vectors , by first extracting the role assignment of each token based on the entire context and then transforming each token vector representation based on its assigned role ( s ) . More specifically , we introduce a small recurrent neural network that succinctly represents the context and subsequently determines the role assignment of each token using one of three strategies : dense , additive and sparse assignment . Each role has a trainable transformation matrix that rotates and scales the corresponding token ’ s vector representation . These transformed role-specific vectors are then combined by a weighted sum based on the role assignment and fed into the subsequent layers . We extensively evaluate the proposed RIL on machine translation using two language pairs ( En-De and En-Fi ) and three datasets with varying sizes ( IWSLT ’ 14 De-En–160k sentence pairs , WMT ’ 14 En-De–4.5M sentence pairs , and WMT ’ 17 En-Fi–2.3M sentence pairs ) . We test variants of the proposed approach against the baselines trained with the same setup . We observe that the proposed etet−1 et+1 ... e1 ... eT ⊗ Role Encoder g rt ẽt re si du al Figure 1 : Illustration of proposed role interaction layer . A role encoder function g ( in our work , a CNN or LSTM ) computes a contextual role representation rt which is used to modulate representation et resulting in ẽt . The additive , residual connection ( dotted line ) is optional . RIL , which explicitly models latent roles , indeed improves translation quality over a strong baseline when there is only limited amount of training data . The improvement , however , diminishes as the amount of data grows . On the large scale WMT ’ 14 En-De dataset , improvements are small compared to our carefully tuned baseline , indicating that role interaction can indeed be captured implicitly by powerful models with enough training data . However , qualitative analysis reveals that the proposed RIL does still extract meaningful roles that are highly context-dependent , giving us an insight into the internal mechanism of state-of-the-art neural machine translation systems . 2 BACKGROUND : NEURAL MACHINE TRANSLATION . Most existing neural machine translation systems , i.e. , recurrent ( Bahdanau et al. , 2015 ) , convolutional ( Gehring et al. , 2017 ) and self-attention-based systems ( Vaswani et al. , 2017 ) , are special cases of a conditional neural language models that consist of an encoder and a decoder . The encoder takes as input a sequence of source-side tokens X = ( x1 , . . . , xT ) , and first turns it into a sequence of token vectors : EX = ( e X 1 , . . . , e X T ) , ( 1 ) where eXt = f X emb ( xt ) . It then processes this sequence into a set of vector representations H to be used by the decoder . The decoder on the other hand takes as input a partial prefix of the target side tokens at each time step along with the encoded representation H : Yt = ( y1 , . . . , yt−1 ) . Similarly to the encoder , the prefix Yt is first transformed into a sequence of ( target ) token vectors : EY = ( e Y 1 , . . . , e Y t−1 ) , ( 2 ) where eYt = f Y emb ( yt ) , similarly to the encoder . Given Yt and H , the decoder computes the distribution over all possible target tokens , log p ( yt|Yt , H ) . By summing these log-probabilities over the target sentence , we get the log-probability over the entier sentence Y given a source sentence X : log p ( Y |X ) = T ′∑ t=1 log p ( yt|y < t , X ) . ( 3 ) We train this neural machine translation system by maximizing the log-likelihood over a training set consisting of source-target pairs : max ∑n n=1 ∑T ′n t=1 log p ( y n t |yn < t , Xn ) . 3 MODULATING TOKEN REPRESENTATION WITH CONTEXTUAL ROLES . We focus on token embedding in Eqs . ( 1 ) – ( 2 ) . A usual approach to embedding a discrete token into a continuous space is to use a so-called table lookup or embedding layer ( Bengio et al. , 2003 ) , which is equivalent to multiplying an embedding matrix with a one-hot vector of a token : fXemb ( xt ) =W X emb1xt , where WXemb ∈ Rh×|VX | is an embedding matrix and 1xt is a one-hot vector which is has value 1 only in dimension xt and is 0 otherwise . This effectively assigns one trainable h-dimensional continuous vector to each token , regardless of the context in which the token appears.1 1 The same applies to the target side by replacing X with Y . This high-dimensional , distributed representation encodes multiple aspects of the token ’ s underlying meaning . Choi et al . ( 2017 ) demonstrated this property by applying PCA to the continuous vectors of a set of neighbouring tokens given a target token and inspecting the neighbouring tokens lying along each principal axis . In their analysis , for instance , along the first principal axis of “ notebook ” lied “ notebooks ” , “ sketchbooks ” , “ diary ” and “ jottings ” , while “ laptop ” , “ ipaq ” and “ palmtop ” lied along the second principal axis . In other words , various aspects/meanings of a single token are entangled into a single continuous vector in this conventional approach to token embedding , and it is up to follow-up layers in the network to disentangle and select a relevant subset of these aspects . Instead , we propose to augment this embedding layer with a specialized mechanism for separately modelling different aspects of tokens based on the context . We first define a role assignment of each token as a low-dimensional vector rt ∈ Rhr , where hr h.2 The role assignment vector is computed based on the context , in our case a small recurrent neural network with long short-term memory ( LSTM ) units ( Hochreiter & Schmidhuber , 1997 ) followed by a single nonlinear layer : ( r1 , r2 , . . . , rT ) = frole ( LSTM ( e1 , e2 , . . . , eT ) ) . ( 4 ) Each dimension i ∈ { 1 , . . . , hr } of the role assignment vector corresponds to the degree to which the corresponding token is assigned to the i-th role . Each role i is associated with a distinct trainable transformation matrix Ui ∈ Rh×h . This transformation matrix rotates and scales the token representation et in its own way , and the transformed token representations are combined by ẽt = hr∑ i=1 rt , i [ Uiet ] . ( 5 ) We call this layer the role interaction layer ( RIL ) . The role assignment vector rt = [ rt,1 , . . . , rt , hr ] > indicates the role ( s ) i ∈ [ 1 , . . . , hr ] to which the token xt is assigned . Each role corresponds to a subset of aspects encoded in the token embedding , and is extracted by linearly transforming the token embedding , i.e. , Uiet . These extracted role-specific token vectors are then weighted-summed using the role assignment vectors in Eq . ( 5 ) . See Fig . 1 for a graphical illustration of the RIL . 3.1 ROLE ASSIGNMENT . The proposed RIL leaves the design of the role assignment computation in Eq . ( 4 ) open . In this paper we consider three different approaches to role assignment . Dense assignment We affine-transform the output r̂t from the LSTM layer at each time step in Eq . ( 4 ) and apply a hyperbolic tangent element-wise : rdenset = tanh ( Wr̂t + b ) . We call this a dense role assignment , as it corresponds to assigning the token to all the roles ( hence , dense ) with varying degrees of assignment . Additive assignment : Softmax assignment Instead , we can constrain the role assignment to be additive with varying strengths . We use softmax normalization ( Bridle , 1990 ) : rsoftmaxt , i = exp ( u > i r dense t ) ∑hr j=1 exp ( u > j r dense t ) > 0 . Unlike the dense assignment , the additive assignment , or softmax assignment , only allows “ positive ” assignment of a token to each role . Discrete assignment : One-hot assignment The softmax assignment above can be thought of as computing the categorical distribution over the hr roles . In other words , rsoftmaxt , i is the probability of the token being assigned to the role i : p ( ronehott , i = 1|X ) = rsoftmaxt , i . ( 6 ) 2 We omit the superscript , X or Y , whenever it is unnecessary . From this distribution , we can sample a single role during training , or choose the most likely role during test , to which the corresponding token is assigned . One difficulty arising from this discrete assignment is that we can not use backpropagation due to the non-differentiability . In the experiments later , we use Gumbel-Softmax relaxation ( Maddison et al. , 2017 ; Jang et al. , 2017 ) to address this issue during training . Role Assignment in the Target Side On the source side , the role assignment vector for each token can be computed using the entire context , that is , considering both the previous and future tokens . It is , however , not possible to do so on the target side , due to the autoregressive nature of conditional language modelling in Eq . ( 3 ) . The role assignment of target tokens must be done only based on the previous tokens . To accommodate this restriction on the target side and the lack thereof on the source side , we use the unidirection LSTM on the target side and the bidirectional LSTM on the source side : rXt = f X role ( Bi-LSTM ( e X 1 , e X 2 , . . . , e X T ) t ) r Y t = f Y role ( LSTM ( e Y 1 , e Y 2 , . . . , e Y t−1 ) t ) .

Summary: This paper proposes “a role interaction layer” (briefly, RIL) that consists of context-dependent (latent) role assignments and role-specific transformations: Given an RIL layer, different dimensions of an embedding vector are “interacted” based on Eqn. (5), Eqn. (7), etc. The authors work on IWSLT De->En and WMT En->De, En->Fi to verify their proposed algorithm with case study included.