Amortized Context Vector Inference for Sequence-to-Sequence Networks

1 INTRODUCTION . Sequence-to-sequence ( seq2seq ) or encoder-decoder models ( Sutskever et al. , 2014 ) constitute a novel solution to inferring relations between sequences of different lengths . They are broadly used for addressing tasks including machine translation ( MT ) ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) , abstractive document summarization ( ADS ) , descriptive caption generation ( DCG ) ( Xu et al. , 2016 ) , and question answering ( QA ) ( Sukhbaatar et al. , 2015 ) , to name just a few . Seq2seq models comprise two distinct RNN models : an encoder RNN , and a decoder RNN . Their main principle of operation is based on the idea of learning to infer an intermediate context vector representation , c , which is “ shared ” among the two RNN modules of the model , i.e. , the encoder and the decoder . Specifically , the encoder converts the source sequence to a context vector ( e.g. , the final state of the encoder RNN ) , while the decoder is presented with the inferred context vector to produce the target sequence . Despite these merits , though , baseline seq2seq models can not learn temporal dynamics over long horizons . This is due to the fact that a single context vector c is capable of encoding rather limited temporal information . This major limitation has been addressed via the development of neural attention ( NA ) mechanisms ( Bahdanau et al. , 2015 ) . NA has been a major breakthrough in Deep Learning for Natural Language Processing , as it enables the decoder modules of seq2seq models to adaptively focus on temporally-varying subsets of the source sequence . This capacity , in turn , enables flexibly capturing long temporal dynamics in a computationally efficient manner . Among the large collection of recently devised NA variants , the vast majority build upon the concept of Soft Attention ( SA ) ( Xu et al. , 2016 ) . Under this rationale , at each sequence generation ( decoding ) step , NA-obtained context vectors essentially constitute deterministic representations of the dynamics between the source sequence and the decodings obtained thus far . However , recent work in the field of amortized variational inference ( AVI ) ( Jimenez Rezende & Mohamed , 2015 ; Kingma & Welling , 2013 ) has shown that it is often useful to treat representations generated by deep networks as latent random variables . Indeed , it is now well-understood that , under such an inferential setup , the trained deep learning models become more effective in inferring representations that offer stronger generalizaton capacity , instead of getting trapped to representations of poor generalizaton quality . Then , model training reduces to inferring posterior distributions over the introduced latent variables . This can be performed by resorting to variational inference ( Attias , 2000 ) , where the sought variational posteriors are parameterized via appropriate deep networks . Motivated from these research advances , in this paper we consider a novel formulation of SA . Specifically , we propose an NA mechanism formulation where the generated context vectors are considered random latent variables with finite mixture model posteriors , over which AVI is performed . We dub our approach amortized context vector inference ( ACVI ) . To exhibit the efficacy of ACVI , we implement it into : ( i ) Pointer-Generator Networks ( See et al. , 2017 ) , which constitute a state-of-the-art approach for addressing ADS tasks ; ( ii ) baseline seq2seq models with additive SA , applied to the task of VC ; and ( iii ) baseline seq2seq models with multiplicative SA , applied to MT . The remainder of this paper is organized as follows : In Section 2 , we briefly present the seq2seq model variants in the context of which we implement our method and exhibit its efficacy . In Section 3 , we introduce the proposed approach , and elaborate on its training and inference algorithms . In Section 4 , we perform an extensive experimental evaluation of our approach using benchmark ADS , MT , and VC datasets . Finally , in the concluding Section , we summarize the contribution of this work . 2 METHODOLOGICAL BACKGROUND . 2.1 ABSTRACTIVE DOCUMENT SUMMARIZATION . ADS consists in not only copying from an original document , but also learning to generate new sentences or novel words during the summarization process . The introduction of seq2seq models has rendered ADS both feasible and effective ( Rush et al. , 2015 ; Zeng et al. , 2017 ) . Dealing with out-ofvocabulary ( OOV ) words was one of the main difficulties that early ADS models were confronted with . Word and/or phrase repetition was a second issue . The pointer-generator model presented in ( See et al. , 2017 ) constitutes one of the most comprehensive efforts towards ameliorating these issues . In a nutshell , this model comprises one bidirectional LSTM ( Hochreiter & Schmidhuber , 1997 ) ( BiLSTM ) encoder , and a unidirectional LSTM decoder , which incorporates an SA mechanism ( Bahdanau et al. , 2015 ) . The word embedding of each token , xi , i ∈ { 1 , . . . , N } , in the source sequence ( document ) is presented to the encoder BiLSTM ; this obtains a representation ( encoding ) hi = [ −→ h i ; ←− h i ] , where −→ h i is the corresponding forward LSTM state , and ←− h i is the corresponding backward LSTM state . Then , at each generation step , t , the decoder LSTM gets as input the ( word embedding of the ) previous token in the target sequence . During training , this is the previous word in the available reference summary ; during inference , this is the previous generated word . On this basis , the decoder updates its internal state , st , which is then presented to the postulated SA network . Specifically , the attention distribution , at , is given by : eit = v T tanh ( Whhi +Wsst + battn ) ( 1 ) at = softmax ( et ) , et = [ e i t ] i ( 2 ) where the W· are trainable weight matrices , battn is a trainable bias vector , and v is a trainable parameter vector of the same size as battn . Then , the model updates the maintained context vector , ct , by taking an weighted average of all the source token encodings ; in that average , the used weights are the inferred attention probabilities . We obtain : ct = ∑ i aithi ( 3 ) Eventually , the predictive distribution over the next generated word yields : P vocabt = softmax ( V ′tanh ( V [ st ; ct ] + b ) + b ′ ) ( 4 ) where V and V ′ are trainable weight matrices , while b and b′ are trainable bias vectors . In parallel , the network also computes an additional probability , pgent , which expresses whether the next output should be generated by sampling from the predictive distribution , P vocabt , or the model should simply copy one of the already available source sequence tokens . This mechanism allows for the model to cope with OOV words ; it is defined via a simple sigmoid layer of the form : pgent = σ ( w T c ct +w T s st +w T x xt + bptr ) ( 5 ) where xt is the decoder input , while the w· and bptr are trainable parameter vectors . The probability of copying the ith source sequence token is considered equal to the corresponding attention probability , ait . Eventually , the obtained probability that the next output word will be β ( found either in the vocabulary or among the source sequence tokens ) yields : Pt ( β ) = p gen t P vocab t ( β ) + ( 1− p gen t ) ∑ i : βi=β ait ( 6 ) Finally , a coverage mechanism may also be employed ( Tu et al. , 2016 ) , as a means of penalizing words that have already received attention in the past , to prevent repetition . Specifically , the coverage vector , kt , is defined as : kt = [ k i t ] N i=1 = t−1∑ τ=0 aτ ( 7 ) Using the so-obtained coverage vector , expression ( 1 ) is modified as follows : eit = v T tanh ( Whhi +Wsst +wkk i t + battn ) ( 8 ) where wk is a trainable parameter vector of size similar to v. Model training is performed via minimization of the categorical cross-entropy , augmented with a coverage term of the form : λ ∑ i ∑ t min ( ait , c i t ) ( 9 ) Here , λ controls the influence of the coverage term ; in the remainder of this work , we set λ = 1 . 2.2 VIDEO CAPTIONING . Seq2seq models with attention have been successfully applied to several datasets of multimodal nature . Video captioning constitutes a popular such application . In this work , we consider a simple seq2seq model with additive SA that comprises a BiLSTM encoder , an LSTM decoder , and an output distribution of the form ( 4 ) . The used encoder is presented with visual features obtained from a pretrained convolutional neural network ( CNN ) . Using a pretrained CNN as our employed visual feature extractor ensures that all the evaluated attention models are presented with identical feature descriptors of the available raw data . Hence , it facilitates fairness in the comparative evaluation of our proposed attention mechanism . We elaborate on the specific model configuration in Section 4.2 . 2.3 MACHINE TRANSLATION . Machine translation constitutes one of the first sequential data modeling applications where seq2seq models were shown to obtain state-of-the-art performance . In this work , we perform MT by means of a baseline seq2seq model comprising a BiLSTM encoder , an LSTM decoder , a predictive distribution over the next generated word which is given by ( 4 ) , and a multiplicative SA mechanism . The latter is described by ( Luong et al. , 2015 ) : eit = hiWst ( 10 ) in conjunction with Eq . ( 2 ) ; therein , W is a trainable weights matrix . Our consideration here of multiplicative SA both serves the purpose of implementing and evaluating our approach under diverse SA variants , and is congruent with the best reported results in the related literature . 3 PROPOSED APPROACH . We begin by introducing the core assumption that the computed context vectors , ct , constitute latent random variables . Further , we assume that , at each decoding step , t , the corresponding context vector , ct , is drawn from a distribution associated with one of the available source sequence encodings , { hi } Ni=1 . The selection of the source sequence encoding to associate with is determined from the output sequence via the decoder state , st , as we explain next . Let us introduce the set of binary latent indicator variables , { zit } Ni=1 , zit ∈ { 0 , 1 } , with zit = 1 denoting that the context vector ct is drawn from the ith density , that is the density associated with the ith source encoding , hi , and zit = 0 otherwise . Then , we postulate the following hierarchical model : ct|zit = 1 ; D ∼ p ( θ ( hi ) ) ( 11 ) zit = 1|D ∼ πit ( ait ) ( 12 ) where D comprises the set of both the source and target training sequences , θ denotes the parameters set of the context vector conditional density , and πit denotes the probability of drawing from the ith conditional at time t. Notably , we assume that the component assignment probabilities , πit , are functions of the attention probabilities , ait . Thus , the selection of the mixture component density that we draw the context vector from at decoding time t is directly determined from the value of the current decoder state , st , via the corresponding attention probabilities . A higher affinity of the current decoder state st with the ith encoding , hi , at time t , results in higher probability that the context vector be drawn from the corresponding conditional density . Having defined the hierarchical model ( 11 ) - ( 12 ) , it is important that we examine the resulting expression of the posterior density p ( ct ; D ) . By marginalizing over ( 11 ) and ( 12 ) , we obtain : p ( ct ; D ) = N∑ i=1 πit ( a i t ) p ( θ ( hi ) ) ( 13 ) In other words , we obtain a finite mixture model posterior over the context vectors , with mixture conditional densities associated with the available source sequence encodings , and mixture weights that are functions of the corresponding attention vectors , and are therefore determined by the target sequences . In addition , it is interesting to compare this expression to the definition of context vectors under the conventional SA scheme . From ( 3 ) , we observe that conventional SA is merely a special case of our proposed model , obtained by introducing two assumptions : ( i ) that the postulated mixture component assignment probabilities are identity functions of the associated attention probabilities , i.e . p ( zt ; D ) = Cat ( zt ∣∣πt ) , zt = [ zit ] Ni=1 , πt = [ πit ( ait ) ] Ni=1 s.t . πit ( a i t ) , p ( z i t = 1 ; D ) = ait = softmax ( et ) ; ( 14 ) and ( ii ) that the conditional densities of the context vectors have all their mass concentrated on hi , that is they collapse onto the single point , hi : p ( ct|zit = 1 ; D ) = δ ( hi ) ( 15 ) Indeed , by combining ( 13 ) - ( 15 ) , we yield : p ( ct ; D ) = N∑ i=1 aitδ ( hi ) ( 16 ) whence we obtain ( 3 ) with probability 1 . Thus , our approach replaces the simplistic conditional density expression ( 15 ) with a more appropriate family p ( θ ( hi ) ) , as in ( 13 ) . Based on the literature of AVI , e.g . ( Jimenez Rezende & Mohamed , 2015 ; Kingma & Welling , 2013 ; Sønderby et al. , 2016 ) , we posit that such a stochastic latent variable consideration may result in significant advantages for the postulated seq2seq model . Specifically , our trained model becomes more agile in searching for effective context representations , as opposed to getting trapped to poor local solutions . In the following , we examine conditional densities of Gaussian form . Adopting the inferential rationale of AVI , we consider that these conditional Gaussians are parameterized via the postulated BiLSTM encoder . Specifically , we assume : p ( ct|zit = 1 ; D ) = N ( ct|hi , diag ( σ2 ( hi ) ) ) ( 17 ) where logσ2 ( h ) = ReLU ( h ) ( 18 ) ReLU ( · ) is a trainable ReLU layer of size dim ( h ) , and the encodings , hi , are obtained from a BiLSTM encoder , similar to conventional models . Hence : p ( ct ; D ) = N∑ i=1 aitN ( ct|hi , diag ( σ2 ( hi ) ) ) ( 19 ) Thus , we have arrived at an approximate ( variational ) posterior expression for the context vectors , ct . In our variational treatment , both the component-conditional means , hi , and their variances , σ2 ( hi ) , are obtained from ( amortizing ) neural networks presented with the source sequences . On the other hand , though , the assignment probabilities , πit , in the variational posterior are taken as the attention probabilities , ait . Thus , they are determined by the target sequences , which are generated from the decoder of the model . Hence , our treatment represents a valid approximate posterior formulation , overall conditioned on both the source and target sequences . This concludes the formulation of ACVI . Relation to Recent Work . From the above exhibition , it becomes apparent that our approach generalizes the concept of neural attention by introducing stochasticity in the computation of context vectors . As we have already discussed , the ultimate goal of this construction is to allow for inferring representations of better generalization capacity , by leveraging Bayesian inference arguments . We emphasize that this is in stark contrast to recent efforts toward generalizing neural attention by deriving more complex attention distributions . For instance , ( Kim et al. , 2017 ) have recently introduced structured attention . In that work , the model infers complex posterior probabilities over the assignment latent variables , as opposed to using a simplistic gating function . Specifically , instead of considering independent assignments , they postulate the first-order Markov dynamics assumption : p ( { zt } Tt=1 ; D ) = p ( z1 ; D ) T−1∏ t=1 p ( zt+1|zt ; D ) ( 20 ) Thus , ( Kim et al. , 2017 ) compute posterior distributions over the attention assignments , while ACVI provides a method for obtaining improved representations through the inferred context vectors . Note also that Eq . ( 20 ) gives rise to the need of executing much more computationally complex algorithms to perform attention distribution inference , e.g . the forward-backward algorithm ( Rabiner , 1989 ) . In contrast , our method imposes computational costs comparable to conventional SA . Similar is the innovation in the variational attention method , recently presented ( Deng et al. , 2018 ) . In essence , its key conceptual difference from structured attention is the consideration of full independence between the attention assignments { zt } Tt=1 . Among the several alternatives considered in ( Deng et al. , 2018 ) to obtain stochastic gradient estimators of low variance , it was found that an approach using REINFORCE ( Williams , 1992 ) along with a specialized baseline was effective . Another noteworthy recent work , closer related to ACVI , is the variational encoder-decoder ( VED ) method presented in ( Bahuleyan et al. , 2018 ) . Among the several alternative formulations considered in that paper , the one that clearly outperformed the baselines in terms of the obtained accuracy ( BLEU scores ) combined seq2seq models with SA with an extra variational autoencoder ( VAE ) module . This way , apart from the context vector , which is computed under the standard SA scheme , an additional latent vector ξ is essentially inferred . The imposed prior over it is a standard N ( 0 , I ) , while the inferred posterior is a diagonal Gaussian parameterised by a BiLSTM network presented with the input sequence ; the final BiLSTM state vector is presented to dense layers that output the posterior means and variances of the latent vectors ξ . Both the context vector , c , as well as the latent vectors , ξ , are fed to the final softmax layer of the model that yields the generated output symbols . We shall provide comparisons to all these related approaches in the experimental section of our paper . Training Algorithm . To perform training of a seq2seq model equipped with the ACVI mechanism , we resort to maximization of the resulting evidence lower-bound ( ELBO ) expression . To this end , we need first to introduce some prior assumption over the context latent variables , ct. To serve the purpose of simplicity , and also offer a valid way to effect model regularization , we consider : p ( ct ) = N ( ct|0 , I ) ( 21 ) On the grounds of these assumptions , it is easy to show that the resulting ELBO expression becomes : L = ∑ t { Ep ( ct ; D ) [ −Jt ] −KL [ p ( ct ; D ) ||p ( ct ) ] } ( 22 ) In this expression , Ep ( ct ; D ) [ −Jt ] is the posterior expectation of the model log-likelihood , which is an integral part of the ELBO definition . In the following , we approximate all the entailed ELBO terms by drawing MC samples from the context vector posterior . In this work , we are dealing with a one-out-of-many predictive selection ; hence , the model likelihood is a simple Categorical . As such , Ep ( ct ; D ) [ −Jt ] essentially reduces to the negative categorical cross-entropy of the model , averaged over multiple MC samples of the context vectors , drawn from ( 19 ) . Besides , to ensure that the resulting MC estimators will be of low variance , we adopt the reparameterization trick . To this end , we rely on the posterior expressions ( 17 ) and ( 14 ) ; we express the drawn MC samples as follows : c ( k ) t = N∑ i=1 z ( k ) ti c ( k ) ti ( 23 ) In this expression , the c ( k ) ti are samples from the conditional Gaussians ( 17 ) , which employ the standard reparameterization trick rationale , as applied to Gaussian variables : c ( k ) ti = hi + σ ( hi ) ◦ ( k ) ti , ∼ N ( 0 , I ) ( 24 ) On the other hand , the z ( k ) ti are samples from the Categorical distribution ( 14 ) . To allow for performing backpropagation through these samples , while ensuring that the obtained gradients will be of low variance , we may draw z ( k ) ti by making use of the Gumbel-Softmax relaxation ( Jang et al. , 2017 ) . We have empirically found it suffices that we employ the Gumbel-Softmax trick for the last 10 % of the model training iterations1 ; previously , we merely adopt the following heuristic , without any statistically significant performance deviation : We use a simple weighted average of the samples c ( k ) ti , with the weights being the attention probabilities , ait : c ( k ) t ← N∑ i=1 aitc ( k ) ti ( 25 ) This way , we alleviate the computational costs of employing the Gumbel-Softmax relaxation , which dominates the costs of sampling from the mixture posterior ( 19 ) . Having obtained a reparameterization of the model ELBO that guarantees low variance estimators , we proceed to its maximization by resorting to a modern , off-the-shelf , stochastic gradient optimizer . Specifically , we adopt simple stochastic gradient descent ( SGD ) for the MT tasks , and Adam with its default settings ( Kingma & Ba , 2015 ) for the rest . Inference Algorithm . To perform target decoding by means of a seq2seq model that employs the ACVI mechanism , we resort to Beam search ( Russel & Norvig ) . In our experiments , Beam width is set to five for the ADS and VC tasks ( Sections 4.1 and 4.2 ) , and to ten for the MT tasks ( Section 4.3 ) .

Summary: This paper presents a new way to construct the context vector in seq2seq networks. Specifically, the context vector in the proposed model is treated as a latent variable with a normal prior. The posterior is "defined" as a mixture of the latent presentations of the input sequence and the mixing weight is the attention. To do inference, maximising VAE-like "ELBO" is used. The proposed model is reported to achieve better performance in the seq2seq tasks including document summarisation and video caption.