Discrete Structural Planning for Generating Diverse Translations

1 INTRODUCTION . When humans speak , a planning phase exists when we want to say in a particular structure or style . Linguists have found particular speech errors or behaviors that indicate that speakers are sometimes planning ahead although they may not notice it ( Redford , 2015 ) . This planning phase is considered important both at the sentence and discourse levels . For example , we may decide to use the “ In order to ... , he ... , because ... ” structure before telling a long story . Therefore , it is meaningful to explore the benefits for a neural language generation model to have a such a planning phase . In this work , we are interested in letting the model plan the sentence structure before producing words . If this can be done , then we may be able to select a structure beforehand , or obtain multiple sentences with diverse structures . Such features are useful for language generation tasks such as machine translation , as users may desire to know how a sentence can be translated with different structures . Therefore , we focus on machine translation task when developing our models . In contrast to human , a neural machine translation ( NMT ) model does not have a planning phase when it is asked to generate a sentence . One can argue that the planning is performed in the hidden layers . However , as a source sentence can have multiple valid translations with different structures . When we use the cross-entropy loss to train the model , it has to assign probabilities to all possible translations with different structures . Therefore , the structural information remains probabilistic but not deterministic in the neural network . In other words , the model is unaware of the “ big picture ” of the output sentence before translation . Consequently , we can not control the sentence structure beforehand . To obtain diverse sentences , conventional NMT models allow one to sample translations from theN - best list produced by beam search , but the sampled translation results usually share similar syntactic structures . One naive approach to explicitly control the output structure is to prefix translations with their structural ( e.g. , part-of-speech ) tags . In this case , the NMT model has to predict the tags before generating words according to these tags . Thus , we can obtain structural diverse translations by using different tags . This approach is referred to as tag planning . In our experiments , we found this approach results in significant degradation in translation quality because errors in the planning phase directly affect the correctness of the translation . To overcome this problem , we propose learning a discrete structural representation ( i.e. , discrete codes ) that capture structural variations without negatively affecting the translation quality . Fig . 1 shows our proposed two-phase approach for training the NMT model . In the first phase , we parse target sentences to obtain their structural tags . In this work , we use simplified POS tags to represent the coarse structure of a sentence . Next , we design a coding model that encodes information about the structural variations into discrete codes . In the second phase , we prefix each target sentence in the training data with the structural codes produced by the coding model . As shown in Fig . 1 , we use a special token “ 〈eoc〉 ” to denote the end of the codes . Finally , we train an NMT model on the enhanced dataset ; the model architecture does not require any modification . To maintain translation performance , our coding model eliminates partial information from the structural tags that is unambiguous when the source sentence is given . For example , for the Spanish sentence shown in Fig . 1 , it is unambiguous that the translation will contain a noun , a pronoun , and a verb . Such obvious information does not require planning , and the translation quality may be impacted seriously if it is predicted wrongly . In this example , the structural choice between “ he is hired by our company ” or “ our company hired him ” is ambiguous . In this work , the coding model is designed to capture only the ambiguous part of the sentence structure when the source sentence is known . We call this approach discrete planning , as the planning component is actually the artificially created discrete codes . Please note that the planning component can also be a continuous latent vector , which requires a discriminator to train the model in order that the latent captures disentangled structural information . However , such a discriminator is difficult to implement . A detailed discussion is given in related work . Our approach of learning discrete structural representation has the advantage of simplicity , and no modification to the NMT model is required . The contributions of this work can be summarized as follows : 1 . We propose a novel approach to plan the sentence structure with a discrete structural representation . By carefully learning the discrete codes to capture pure structural variation , the translation quality remains intact . 2 . We are able to obtain drastically different translations by manipulating the codes . To qualitatively evaluate the diversity , we propose a structural diversity metric . The evaluation results show that our approach is capable of producing sentences with highly diverse structures . 2 LEARNING STRUCTURAL CODES . This section discusses the motivation and methodology for learning discrete structural codes . First , we describe a naive approach that extracts the global sentence structure from part-of-speech ( POS ) tags . Then , we describe a coding model that learns discrete structural codes and the advantages of doing so . 2.1 EXTRACTING GLOBAL SENTENCE STRUCTURES . To plan the structures of translations , we need to obtain the structural representation of a sentence . However , it is extremely difficult for a neural network to automatically realize and extract the sentence structure . Therefore , we consider utilizing the results of POS tagging or syntactic parsing . However , most parsing results including POS tags contain detailed structural information , which is not desirable . For example , “ destroy ” and “ break down ” are interchangeable , but they have different POS tags . Such minor differences do not affect the global structure of a sentence . Therefore , we need a structural representation that captures the sentence structure at a coarse level . In this work , we extract such a global sentence structure through a simple two-step process that simplifies the POS tags : 1 . Remove all tags other than “ N ” , “ V ” , “ PRP ” , “ , ” and “ . ” . Note that all tags begin with “ N ” ( e.g . NNS ) are mapped to “ N ” , and tags begin with “ V ” ( e.g . VBD ) are mapped to “ V ” . 2 . Remove duplicated consecutive tags . The following list gives an example of the process : Input : He found a fox behind the wall . POS Tags : PRP VBD DT NN IN DT NN . Step 1 : PRP V N N . Step 2 : PRP V N . It should be noted that other parse results can also be potentially useful to represent the global structure ; this is left for future works to explore . In our experiments , we found simplified POS tags to be effective and convenient for implementation . 2.2 CODE LEARNING . Although we can directly condition the word prediction using the structural tags , we found that errors during structural prediction significantly degraded the translation performance . To mitigate their impact , we propose learning a set of discrete codes that capture only the ambiguous structural choice when the source sentence is given . Code Encoder Given the sentence pair ( X , Y ) , let the structural tags of the target sentence be S1 , ... , ST , the encoder computes the logits of N softmax vectors : h̄t = LSTM ( E ( St ) , h̄t+1 ; φe ) , ( 1 ) [ C̃1 , ... , C̃N ] = softplus ( f ( h̄1 ; φh ) ) , ( 2 ) where the tag sequence S1 , ... , ST is firstly encoded using a backward LSTM ( Hochreiter & Schmidhuber , 1997 ) . E ( · ) denotes an embedding function , and f ( · ) denotes a linear transformation . Then , we compute a set of continuous vectors C̃1 , ... , C̃N , which are latterly discretized into approximated one-hot vectorsC1 , ... , CN using Gumbel-Softmax reprarameterization trick ( Jang et al. , 2016 ; Mad- dison et al. , 2016 ) as : Ci = g ( S , ; φ ) i ( 3 ) = softmaxτ ( log C̃i + gumbel ( ) ) . ( 4 ) Here , τ is the temperature of the softmax . The computation of Gumbel-softmax relies on the Gumbel noise , which is defined as gumbel ( ) = − log ( − log ( ) ) . As both C̃i and are vectors in Eq . 4 , the log ( · ) here is an element-wise operation . The random variable is sampled from a uniform distribution U ( 0 , 1 ) . In Eq . 3 , g ( S , ; φ ) represents the encoder function , which is used to produce the codes once its parameter φ is trained . Decoder In the decoder , we combine the information from X and C to initialize a decoder LSTM that sequentially predicts the structural tags S1 , ... , ST : st = LSTM ( E ( Xt ) , st+1 ; θx ) , ( 5 ) h0 = f ( [ C1 , ... , CN ] ; θh ) + s1 , ( 6 ) ht = LSTM ( E ( St−1 ) , ht−1 ; θd ) , ( 7 ) where [ C1 , ... , CN ] is a concatenation of N Gumbel-softmax code vectors . Note that only ht is computed with a forward LSTM . Finally , we predict the probability of emitting each tag St with P ( St|S < t , X , C ) = softmax ( fout ( ht ; θout ) ) . ( 8 ) Loss Function The architecture of the code learning model is depicted in Fig . 2 , which can be seen as a sequence auto-encoder with an extra context input X to the decoder . The loss function for optimizing the entire model has the following form given one training pair : L ( φ , θ ) = E ∼U ( 0,1 ) [ − logP ( S|X , g ( S , ; φ ) ; θ ) ] , ( 9 ) = E ∼U ( 0,1 ) [ T∑ t=1 − logP ( St|S < t , X , g ( S , ; φ ) ; θ ) ] . ( 10 ) The codes are limited to have only a few bits of information capacity in our experiments . In order to minimize the loss function , the codes produced by the encoder g ( · ) has to capture the structural information contained in S that can not be inferred by X . 2.3 NMT WITH STRUCTURAL PLANNING . Once the coding model is trained , the Gumbel-softmax code vectors C1 , ... , CN shall be very close to one-hot vectors . We can extract N discrete codes by applying argmax on the softmax vectors , and prefix the target sentences in the training data with the codes , resulting in ( X , C ; Y ) pairs . We connect the codes and target sentences with a “ 〈eoc〉 ” token . To evaluate the performance of structural planning with discrete codes , we train a regular NMT model with the modified dataset . The codes are removed from the translation results after decoding . Similarly , we can evaluate the tag planning model using ( X , S ; Y ) pairs .

Summary: The authors propose modeling structural diversity of translations by conditioning the generation on both the source sentence and a latent encoding of the overall structure (captured by simplified part-of-speech tags). Specifically, they first train a conditional autoencoder to learn a latent code optimized towards reconstructing the tag sequence. They then prefix the inferred latent code to the target sentence before generation. A diversity metric which measures pairwise BLEU scores between beam items is also proposed. Experiments show that the latent codes lead to greater structural diversity as well as marginally improved translation results when combined with beam search.