Discrete Structural Planning for Generating Diverse Translations

1 INTRODUCTION . When humans speak , a planning phase exists when we want to say in a particular structure or style . Linguists have found particular speech errors or behaviors that indicate that speakers are sometimes planning ahead although they may not notice it ( Redford , 2015 ) . This planning phase is considered important both at the sentence and discourse levels . For example , we may decide to use the “ In order to ... , he ... , because ... ” structure before telling a long story . Therefore , it is meaningful to explore the benefits for a neural language generation model to have a such a planning phase . In this work , we are interested in letting the model plan the sentence structure before producing words . If this can be done , then we may be able to select a structure beforehand , or obtain multiple sentences with diverse structures . Such features are useful for language generation tasks such as machine translation , as users may desire to know how a sentence can be translated with different structures . Therefore , we focus on machine translation task when developing our models . In contrast to human , a neural machine translation ( NMT ) model does not have a planning phase when it is asked to generate a sentence . One can argue that the planning is performed in the hidden layers . However , as a source sentence can have multiple valid translations with different structures . When we use the cross-entropy loss to train the model , it has to assign probabilities to all possible translations with different structures . Therefore , the structural information remains probabilistic but not deterministic in the neural network . In other words , the model is unaware of the “ big picture ” of the output sentence before translation . Consequently , we can not control the sentence structure beforehand . To obtain diverse sentences , conventional NMT models allow one to sample translations from theN - best list produced by beam search , but the sampled translation results usually share similar syntactic structures . One naive approach to explicitly control the output structure is to prefix translations with their structural ( e.g. , part-of-speech ) tags . In this case , the NMT model has to predict the tags before generating words according to these tags . Thus , we can obtain structural diverse translations by using different tags . This approach is referred to as tag planning . In our experiments , we found this approach results in significant degradation in translation quality because errors in the planning phase directly affect the correctness of the translation . To overcome this problem , we propose learning a discrete structural representation ( i.e. , discrete codes ) that capture structural variations without negatively affecting the translation quality . Fig . 1 shows our proposed two-phase approach for training the NMT model . In the first phase , we parse target sentences to obtain their structural tags . In this work , we use simplified POS tags to represent the coarse structure of a sentence . Next , we design a coding model that encodes information about the structural variations into discrete codes . In the second phase , we prefix each target sentence in the training data with the structural codes produced by the coding model . As shown in Fig . 1 , we use a special token “ 〈eoc〉 ” to denote the end of the codes . Finally , we train an NMT model on the enhanced dataset ; the model architecture does not require any modification . To maintain translation performance , our coding model eliminates partial information from the structural tags that is unambiguous when the source sentence is given . For example , for the Spanish sentence shown in Fig . 1 , it is unambiguous that the translation will contain a noun , a pronoun , and a verb . Such obvious information does not require planning , and the translation quality may be impacted seriously if it is predicted wrongly . In this example , the structural choice between “ he is hired by our company ” or “ our company hired him ” is ambiguous . In this work , the coding model is designed to capture only the ambiguous part of the sentence structure when the source sentence is known . We call this approach discrete planning , as the planning component is actually the artificially created discrete codes . Please note that the planning component can also be a continuous latent vector , which requires a discriminator to train the model in order that the latent captures disentangled structural information . However , such a discriminator is difficult to implement . A detailed discussion is given in related work . Our approach of learning discrete structural representation has the advantage of simplicity , and no modification to the NMT model is required . The contributions of this work can be summarized as follows : 1 . We propose a novel approach to plan the sentence structure with a discrete structural representation . By carefully learning the discrete codes to capture pure structural variation , the translation quality remains intact . 2 . We are able to obtain drastically different translations by manipulating the codes . To qualitatively evaluate the diversity , we propose a structural diversity metric . The evaluation results show that our approach is capable of producing sentences with highly diverse structures . 2 LEARNING STRUCTURAL CODES . This section discusses the motivation and methodology for learning discrete structural codes . First , we describe a naive approach that extracts the global sentence structure from part-of-speech ( POS ) tags . Then , we describe a coding model that learns discrete structural codes and the advantages of doing so . 2.1 EXTRACTING GLOBAL SENTENCE STRUCTURES . To plan the structures of translations , we need to obtain the structural representation of a sentence . However , it is extremely difficult for a neural network to automatically realize and extract the sentence structure . Therefore , we consider utilizing the results of POS tagging or syntactic parsing . However , most parsing results including POS tags contain detailed structural information , which is not desirable . For example , “ destroy ” and “ break down ” are interchangeable , but they have different POS tags . Such minor differences do not affect the global structure of a sentence . Therefore , we need a structural representation that captures the sentence structure at a coarse level . In this work , we extract such a global sentence structure through a simple two-step process that simplifies the POS tags : 1 . Remove all tags other than “ N ” , “ V ” , “ PRP ” , “ , ” and “ . ” . Note that all tags begin with “ N ” ( e.g . NNS ) are mapped to “ N ” , and tags begin with “ V ” ( e.g . VBD ) are mapped to “ V ” . 2 . Remove duplicated consecutive tags . The following list gives an example of the process : Input : He found a fox behind the wall . POS Tags : PRP VBD DT NN IN DT NN . Step 1 : PRP V N N . Step 2 : PRP V N . It should be noted that other parse results can also be potentially useful to represent the global structure ; this is left for future works to explore . In our experiments , we found simplified POS tags to be effective and convenient for implementation . 2.2 CODE LEARNING . Although we can directly condition the word prediction using the structural tags , we found that errors during structural prediction significantly degraded the translation performance . To mitigate their impact , we propose learning a set of discrete codes that capture only the ambiguous structural choice when the source sentence is given . Code Encoder Given the sentence pair ( X , Y ) , let the structural tags of the target sentence be S1 , ... , ST , the encoder computes the logits of N softmax vectors : h̄t = LSTM ( E ( St ) , h̄t+1 ; φe ) , ( 1 ) [ C̃1 , ... , C̃N ] = softplus ( f ( h̄1 ; φh ) ) , ( 2 ) where the tag sequence S1 , ... , ST is firstly encoded using a backward LSTM ( Hochreiter & Schmidhuber , 1997 ) . E ( · ) denotes an embedding function , and f ( · ) denotes a linear transformation . Then , we compute a set of continuous vectors C̃1 , ... , C̃N , which are latterly discretized into approximated one-hot vectorsC1 , ... , CN using Gumbel-Softmax reprarameterization trick ( Jang et al. , 2016 ; Mad- dison et al. , 2016 ) as : Ci = g ( S , ; φ ) i ( 3 ) = softmaxτ ( log C̃i + gumbel ( ) ) . ( 4 ) Here , τ is the temperature of the softmax . The computation of Gumbel-softmax relies on the Gumbel noise , which is defined as gumbel ( ) = − log ( − log ( ) ) . As both C̃i and are vectors in Eq . 4 , the log ( · ) here is an element-wise operation . The random variable is sampled from a uniform distribution U ( 0 , 1 ) . In Eq . 3 , g ( S , ; φ ) represents the encoder function , which is used to produce the codes once its parameter φ is trained . Decoder In the decoder , we combine the information from X and C to initialize a decoder LSTM that sequentially predicts the structural tags S1 , ... , ST : st = LSTM ( E ( Xt ) , st+1 ; θx ) , ( 5 ) h0 = f ( [ C1 , ... , CN ] ; θh ) + s1 , ( 6 ) ht = LSTM ( E ( St−1 ) , ht−1 ; θd ) , ( 7 ) where [ C1 , ... , CN ] is a concatenation of N Gumbel-softmax code vectors . Note that only ht is computed with a forward LSTM . Finally , we predict the probability of emitting each tag St with P ( St|S < t , X , C ) = softmax ( fout ( ht ; θout ) ) . ( 8 ) Loss Function The architecture of the code learning model is depicted in Fig . 2 , which can be seen as a sequence auto-encoder with an extra context input X to the decoder . The loss function for optimizing the entire model has the following form given one training pair : L ( φ , θ ) = E ∼U ( 0,1 ) [ − logP ( S|X , g ( S , ; φ ) ; θ ) ] , ( 9 ) = E ∼U ( 0,1 ) [ T∑ t=1 − logP ( St|S < t , X , g ( S , ; φ ) ; θ ) ] . ( 10 ) The codes are limited to have only a few bits of information capacity in our experiments . In order to minimize the loss function , the codes produced by the encoder g ( · ) has to capture the structural information contained in S that can not be inferred by X . 2.3 NMT WITH STRUCTURAL PLANNING . Once the coding model is trained , the Gumbel-softmax code vectors C1 , ... , CN shall be very close to one-hot vectors . We can extract N discrete codes by applying argmax on the softmax vectors , and prefix the target sentences in the training data with the codes , resulting in ( X , C ; Y ) pairs . We connect the codes and target sentences with a “ 〈eoc〉 ” token . To evaluate the performance of structural planning with discrete codes , we train a regular NMT model with the modified dataset . The codes are removed from the translation results after decoding . Similarly , we can evaluate the tag planning model using ( X , S ; Y ) pairs .

Summary: The authors consider the problem of generating diverse translations from a neural machine translation model. This is a very interesting problem and indeed, even the best models lack meaningful diversity when generating with beam-search. The method proposed by the authors relies on prefixing the generation with discrete latent codes. While a good general approach, it is not new (exactly the same general approach that was used in the "Discrete Autoencoders for Sequence Models" [1] paper, https://arxiv.org/abs/1801.09797, for generating diverse translations, which is not cited directly but a follow-up work is cited, though without mentioning that a previous work has tackled the same problem). Also, the authors rely on additional supervised data (namely POS tags) which has no clear motivation and seems to cause a number of problems -- why not use a purely unsupervised approach when it has already been demonstrated on the same problem? Additionally, the authors compare to a weak translation baseline on small data-sets, making it impossible to judge whether the results would hold on a larger data-set. So the following ablations and comparison to baselines are missing: