Amortized Context Vector Inference for Sequence-to-Sequence Networks

1 INTRODUCTION . Sequence-to-sequence ( seq2seq ) or encoder-decoder models ( Sutskever et al. , 2014 ) constitute a novel solution to inferring relations between sequences of different lengths . They are broadly used for addressing tasks including machine translation ( MT ) ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) , abstractive document summarization ( ADS ) , descriptive caption generation ( DCG ) ( Xu et al. , 2016 ) , and question answering ( QA ) ( Sukhbaatar et al. , 2015 ) , to name just a few . Seq2seq models comprise two distinct RNN models : an encoder RNN , and a decoder RNN . Their main principle of operation is based on the idea of learning to infer an intermediate context vector representation , c , which is “ shared ” among the two RNN modules of the model , i.e. , the encoder and the decoder . Specifically , the encoder converts the source sequence to a context vector ( e.g. , the final state of the encoder RNN ) , while the decoder is presented with the inferred context vector to produce the target sequence . Despite these merits , though , baseline seq2seq models can not learn temporal dynamics over long horizons . This is due to the fact that a single context vector c is capable of encoding rather limited temporal information . This major limitation has been addressed via the development of neural attention ( NA ) mechanisms ( Bahdanau et al. , 2015 ) . NA has been a major breakthrough in Deep Learning for Natural Language Processing , as it enables the decoder modules of seq2seq models to adaptively focus on temporally-varying subsets of the source sequence . This capacity , in turn , enables flexibly capturing long temporal dynamics in a computationally efficient manner . Among the large collection of recently devised NA variants , the vast majority build upon the concept of Soft Attention ( SA ) ( Xu et al. , 2016 ) . Under this rationale , at each sequence generation ( decoding ) step , NA-obtained context vectors essentially constitute deterministic representations of the dynamics between the source sequence and the decodings obtained thus far . However , recent work in the field of amortized variational inference ( AVI ) ( Jimenez Rezende & Mohamed , 2015 ; Kingma & Welling , 2013 ) has shown that it is often useful to treat representations generated by deep networks as latent random variables . Indeed , it is now well-understood that , under such an inferential setup , the trained deep learning models become more effective in inferring representations that offer stronger generalizaton capacity , instead of getting trapped to representations of poor generalizaton quality . Then , model training reduces to inferring posterior distributions over the introduced latent variables . This can be performed by resorting to variational inference ( Attias , 2000 ) , where the sought variational posteriors are parameterized via appropriate deep networks . Motivated from these research advances , in this paper we consider a novel formulation of SA . Specifically , we propose an NA mechanism formulation where the generated context vectors are considered random latent variables with finite mixture model posteriors , over which AVI is performed . We dub our approach amortized context vector inference ( ACVI ) . To exhibit the efficacy of ACVI , we implement it into : ( i ) Pointer-Generator Networks ( See et al. , 2017 ) , which constitute a state-of-the-art approach for addressing ADS tasks ; ( ii ) baseline seq2seq models with additive SA , applied to the task of VC ; and ( iii ) baseline seq2seq models with multiplicative SA , applied to MT . The remainder of this paper is organized as follows : In Section 2 , we briefly present the seq2seq model variants in the context of which we implement our method and exhibit its efficacy . In Section 3 , we introduce the proposed approach , and elaborate on its training and inference algorithms . In Section 4 , we perform an extensive experimental evaluation of our approach using benchmark ADS , MT , and VC datasets . Finally , in the concluding Section , we summarize the contribution of this work . 2 METHODOLOGICAL BACKGROUND . 2.1 ABSTRACTIVE DOCUMENT SUMMARIZATION . ADS consists in not only copying from an original document , but also learning to generate new sentences or novel words during the summarization process . The introduction of seq2seq models has rendered ADS both feasible and effective ( Rush et al. , 2015 ; Zeng et al. , 2017 ) . Dealing with out-ofvocabulary ( OOV ) words was one of the main difficulties that early ADS models were confronted with . Word and/or phrase repetition was a second issue . The pointer-generator model presented in ( See et al. , 2017 ) constitutes one of the most comprehensive efforts towards ameliorating these issues . In a nutshell , this model comprises one bidirectional LSTM ( Hochreiter & Schmidhuber , 1997 ) ( BiLSTM ) encoder , and a unidirectional LSTM decoder , which incorporates an SA mechanism ( Bahdanau et al. , 2015 ) . The word embedding of each token , xi , i ∈ { 1 , . . . , N } , in the source sequence ( document ) is presented to the encoder BiLSTM ; this obtains a representation ( encoding ) hi = [ −→ h i ; ←− h i ] , where −→ h i is the corresponding forward LSTM state , and ←− h i is the corresponding backward LSTM state . Then , at each generation step , t , the decoder LSTM gets as input the ( word embedding of the ) previous token in the target sequence . During training , this is the previous word in the available reference summary ; during inference , this is the previous generated word . On this basis , the decoder updates its internal state , st , which is then presented to the postulated SA network . Specifically , the attention distribution , at , is given by : eit = v T tanh ( Whhi +Wsst + battn ) ( 1 ) at = softmax ( et ) , et = [ e i t ] i ( 2 ) where the W· are trainable weight matrices , battn is a trainable bias vector , and v is a trainable parameter vector of the same size as battn . Then , the model updates the maintained context vector , ct , by taking an weighted average of all the source token encodings ; in that average , the used weights are the inferred attention probabilities . We obtain : ct = ∑ i aithi ( 3 ) Eventually , the predictive distribution over the next generated word yields : P vocabt = softmax ( V ′tanh ( V [ st ; ct ] + b ) + b ′ ) ( 4 ) where V and V ′ are trainable weight matrices , while b and b′ are trainable bias vectors . In parallel , the network also computes an additional probability , pgent , which expresses whether the next output should be generated by sampling from the predictive distribution , P vocabt , or the model should simply copy one of the already available source sequence tokens . This mechanism allows for the model to cope with OOV words ; it is defined via a simple sigmoid layer of the form : pgent = σ ( w T c ct +w T s st +w T x xt + bptr ) ( 5 ) where xt is the decoder input , while the w· and bptr are trainable parameter vectors . The probability of copying the ith source sequence token is considered equal to the corresponding attention probability , ait . Eventually , the obtained probability that the next output word will be β ( found either in the vocabulary or among the source sequence tokens ) yields : Pt ( β ) = p gen t P vocab t ( β ) + ( 1− p gen t ) ∑ i : βi=β ait ( 6 ) Finally , a coverage mechanism may also be employed ( Tu et al. , 2016 ) , as a means of penalizing words that have already received attention in the past , to prevent repetition . Specifically , the coverage vector , kt , is defined as : kt = [ k i t ] N i=1 = t−1∑ τ=0 aτ ( 7 ) Using the so-obtained coverage vector , expression ( 1 ) is modified as follows : eit = v T tanh ( Whhi +Wsst +wkk i t + battn ) ( 8 ) where wk is a trainable parameter vector of size similar to v. Model training is performed via minimization of the categorical cross-entropy , augmented with a coverage term of the form : λ ∑ i ∑ t min ( ait , c i t ) ( 9 ) Here , λ controls the influence of the coverage term ; in the remainder of this work , we set λ = 1 . 2.2 VIDEO CAPTIONING . Seq2seq models with attention have been successfully applied to several datasets of multimodal nature . Video captioning constitutes a popular such application . In this work , we consider a simple seq2seq model with additive SA that comprises a BiLSTM encoder , an LSTM decoder , and an output distribution of the form ( 4 ) . The used encoder is presented with visual features obtained from a pretrained convolutional neural network ( CNN ) . Using a pretrained CNN as our employed visual feature extractor ensures that all the evaluated attention models are presented with identical feature descriptors of the available raw data . Hence , it facilitates fairness in the comparative evaluation of our proposed attention mechanism . We elaborate on the specific model configuration in Section 4.2 . 2.3 MACHINE TRANSLATION . Machine translation constitutes one of the first sequential data modeling applications where seq2seq models were shown to obtain state-of-the-art performance . In this work , we perform MT by means of a baseline seq2seq model comprising a BiLSTM encoder , an LSTM decoder , a predictive distribution over the next generated word which is given by ( 4 ) , and a multiplicative SA mechanism . The latter is described by ( Luong et al. , 2015 ) : eit = hiWst ( 10 ) in conjunction with Eq . ( 2 ) ; therein , W is a trainable weights matrix . Our consideration here of multiplicative SA both serves the purpose of implementing and evaluating our approach under diverse SA variants , and is congruent with the best reported results in the related literature . 3 PROPOSED APPROACH . We begin by introducing the core assumption that the computed context vectors , ct , constitute latent random variables . Further , we assume that , at each decoding step , t , the corresponding context vector , ct , is drawn from a distribution associated with one of the available source sequence encodings , { hi } Ni=1 . The selection of the source sequence encoding to associate with is determined from the output sequence via the decoder state , st , as we explain next . Let us introduce the set of binary latent indicator variables , { zit } Ni=1 , zit ∈ { 0 , 1 } , with zit = 1 denoting that the context vector ct is drawn from the ith density , that is the density associated with the ith source encoding , hi , and zit = 0 otherwise . Then , we postulate the following hierarchical model : ct|zit = 1 ; D ∼ p ( θ ( hi ) ) ( 11 ) zit = 1|D ∼ πit ( ait ) ( 12 ) where D comprises the set of both the source and target training sequences , θ denotes the parameters set of the context vector conditional density , and πit denotes the probability of drawing from the ith conditional at time t. Notably , we assume that the component assignment probabilities , πit , are functions of the attention probabilities , ait . Thus , the selection of the mixture component density that we draw the context vector from at decoding time t is directly determined from the value of the current decoder state , st , via the corresponding attention probabilities . A higher affinity of the current decoder state st with the ith encoding , hi , at time t , results in higher probability that the context vector be drawn from the corresponding conditional density . Having defined the hierarchical model ( 11 ) - ( 12 ) , it is important that we examine the resulting expression of the posterior density p ( ct ; D ) . By marginalizing over ( 11 ) and ( 12 ) , we obtain : p ( ct ; D ) = N∑ i=1 πit ( a i t ) p ( θ ( hi ) ) ( 13 ) In other words , we obtain a finite mixture model posterior over the context vectors , with mixture conditional densities associated with the available source sequence encodings , and mixture weights that are functions of the corresponding attention vectors , and are therefore determined by the target sequences . In addition , it is interesting to compare this expression to the definition of context vectors under the conventional SA scheme . From ( 3 ) , we observe that conventional SA is merely a special case of our proposed model , obtained by introducing two assumptions : ( i ) that the postulated mixture component assignment probabilities are identity functions of the associated attention probabilities , i.e . p ( zt ; D ) = Cat ( zt ∣∣πt ) , zt = [ zit ] Ni=1 , πt = [ πit ( ait ) ] Ni=1 s.t . πit ( a i t ) , p ( z i t = 1 ; D ) = ait = softmax ( et ) ; ( 14 ) and ( ii ) that the conditional densities of the context vectors have all their mass concentrated on hi , that is they collapse onto the single point , hi : p ( ct|zit = 1 ; D ) = δ ( hi ) ( 15 ) Indeed , by combining ( 13 ) - ( 15 ) , we yield : p ( ct ; D ) = N∑ i=1 aitδ ( hi ) ( 16 ) whence we obtain ( 3 ) with probability 1 . Thus , our approach replaces the simplistic conditional density expression ( 15 ) with a more appropriate family p ( θ ( hi ) ) , as in ( 13 ) . Based on the literature of AVI , e.g . ( Jimenez Rezende & Mohamed , 2015 ; Kingma & Welling , 2013 ; Sønderby et al. , 2016 ) , we posit that such a stochastic latent variable consideration may result in significant advantages for the postulated seq2seq model . Specifically , our trained model becomes more agile in searching for effective context representations , as opposed to getting trapped to poor local solutions . In the following , we examine conditional densities of Gaussian form . Adopting the inferential rationale of AVI , we consider that these conditional Gaussians are parameterized via the postulated BiLSTM encoder . Specifically , we assume : p ( ct|zit = 1 ; D ) = N ( ct|hi , diag ( σ2 ( hi ) ) ) ( 17 ) where logσ2 ( h ) = ReLU ( h ) ( 18 ) ReLU ( · ) is a trainable ReLU layer of size dim ( h ) , and the encodings , hi , are obtained from a BiLSTM encoder , similar to conventional models . Hence : p ( ct ; D ) = N∑ i=1 aitN ( ct|hi , diag ( σ2 ( hi ) ) ) ( 19 ) Thus , we have arrived at an approximate ( variational ) posterior expression for the context vectors , ct . In our variational treatment , both the component-conditional means , hi , and their variances , σ2 ( hi ) , are obtained from ( amortizing ) neural networks presented with the source sequences . On the other hand , though , the assignment probabilities , πit , in the variational posterior are taken as the attention probabilities , ait . Thus , they are determined by the target sequences , which are generated from the decoder of the model . Hence , our treatment represents a valid approximate posterior formulation , overall conditioned on both the source and target sequences . This concludes the formulation of ACVI . Relation to Recent Work . From the above exhibition , it becomes apparent that our approach generalizes the concept of neural attention by introducing stochasticity in the computation of context vectors . As we have already discussed , the ultimate goal of this construction is to allow for inferring representations of better generalization capacity , by leveraging Bayesian inference arguments . We emphasize that this is in stark contrast to recent efforts toward generalizing neural attention by deriving more complex attention distributions . For instance , ( Kim et al. , 2017 ) have recently introduced structured attention . In that work , the model infers complex posterior probabilities over the assignment latent variables , as opposed to using a simplistic gating function . Specifically , instead of considering independent assignments , they postulate the first-order Markov dynamics assumption : p ( { zt } Tt=1 ; D ) = p ( z1 ; D ) T−1∏ t=1 p ( zt+1|zt ; D ) ( 20 ) Thus , ( Kim et al. , 2017 ) compute posterior distributions over the attention assignments , while ACVI provides a method for obtaining improved representations through the inferred context vectors . Note also that Eq . ( 20 ) gives rise to the need of executing much more computationally complex algorithms to perform attention distribution inference , e.g . the forward-backward algorithm ( Rabiner , 1989 ) . In contrast , our method imposes computational costs comparable to conventional SA . Similar is the innovation in the variational attention method , recently presented ( Deng et al. , 2018 ) . In essence , its key conceptual difference from structured attention is the consideration of full independence between the attention assignments { zt } Tt=1 . Among the several alternatives considered in ( Deng et al. , 2018 ) to obtain stochastic gradient estimators of low variance , it was found that an approach using REINFORCE ( Williams , 1992 ) along with a specialized baseline was effective . Another noteworthy recent work , closer related to ACVI , is the variational encoder-decoder ( VED ) method presented in ( Bahuleyan et al. , 2018 ) . Among the several alternative formulations considered in that paper , the one that clearly outperformed the baselines in terms of the obtained accuracy ( BLEU scores ) combined seq2seq models with SA with an extra variational autoencoder ( VAE ) module . This way , apart from the context vector , which is computed under the standard SA scheme , an additional latent vector ξ is essentially inferred . The imposed prior over it is a standard N ( 0 , I ) , while the inferred posterior is a diagonal Gaussian parameterised by a BiLSTM network presented with the input sequence ; the final BiLSTM state vector is presented to dense layers that output the posterior means and variances of the latent vectors ξ . Both the context vector , c , as well as the latent vectors , ξ , are fed to the final softmax layer of the model that yields the generated output symbols . We shall provide comparisons to all these related approaches in the experimental section of our paper . Training Algorithm . To perform training of a seq2seq model equipped with the ACVI mechanism , we resort to maximization of the resulting evidence lower-bound ( ELBO ) expression . To this end , we need first to introduce some prior assumption over the context latent variables , ct. To serve the purpose of simplicity , and also offer a valid way to effect model regularization , we consider : p ( ct ) = N ( ct|0 , I ) ( 21 ) On the grounds of these assumptions , it is easy to show that the resulting ELBO expression becomes : L = ∑ t { Ep ( ct ; D ) [ −Jt ] −KL [ p ( ct ; D ) ||p ( ct ) ] } ( 22 ) In this expression , Ep ( ct ; D ) [ −Jt ] is the posterior expectation of the model log-likelihood , which is an integral part of the ELBO definition . In the following , we approximate all the entailed ELBO terms by drawing MC samples from the context vector posterior . In this work , we are dealing with a one-out-of-many predictive selection ; hence , the model likelihood is a simple Categorical . As such , Ep ( ct ; D ) [ −Jt ] essentially reduces to the negative categorical cross-entropy of the model , averaged over multiple MC samples of the context vectors , drawn from ( 19 ) . Besides , to ensure that the resulting MC estimators will be of low variance , we adopt the reparameterization trick . To this end , we rely on the posterior expressions ( 17 ) and ( 14 ) ; we express the drawn MC samples as follows : c ( k ) t = N∑ i=1 z ( k ) ti c ( k ) ti ( 23 ) In this expression , the c ( k ) ti are samples from the conditional Gaussians ( 17 ) , which employ the standard reparameterization trick rationale , as applied to Gaussian variables : c ( k ) ti = hi + σ ( hi ) ◦ ( k ) ti , ∼ N ( 0 , I ) ( 24 ) On the other hand , the z ( k ) ti are samples from the Categorical distribution ( 14 ) . To allow for performing backpropagation through these samples , while ensuring that the obtained gradients will be of low variance , we may draw z ( k ) ti by making use of the Gumbel-Softmax relaxation ( Jang et al. , 2017 ) . We have empirically found it suffices that we employ the Gumbel-Softmax trick for the last 10 % of the model training iterations1 ; previously , we merely adopt the following heuristic , without any statistically significant performance deviation : We use a simple weighted average of the samples c ( k ) ti , with the weights being the attention probabilities , ait : c ( k ) t ← N∑ i=1 aitc ( k ) ti ( 25 ) This way , we alleviate the computational costs of employing the Gumbel-Softmax relaxation , which dominates the costs of sampling from the mixture posterior ( 19 ) . Having obtained a reparameterization of the model ELBO that guarantees low variance estimators , we proceed to its maximization by resorting to a modern , off-the-shelf , stochastic gradient optimizer . Specifically , we adopt simple stochastic gradient descent ( SGD ) for the MT tasks , and Adam with its default settings ( Kingma & Ba , 2015 ) for the rest . Inference Algorithm . To perform target decoding by means of a seq2seq model that employs the ACVI mechanism , we resort to Beam search ( Russel & Norvig ) . In our experiments , Beam width is set to five for the ADS and VC tasks ( Sections 4.1 and 4.2 ) , and to ten for the MT tasks ( Section 4.3 ) .

Summary: This paper tries to introduce stochasticity over context representation to enable the neural networks to search for effective representation so as to avoid getting trapped into local optima. The basic idea is to formulate the posterior probability of context representation via a mixture gaussian model. The standard attention model SA can be thought of as a special case of the proposed method. The proposed method outperforms a number of baselines on ADS, video captioning, and machine translation datasets.