Year: 2018

Abstract:
Neural attention (NA) has become a key component of sequence-to-sequence models that yield state-of-the-art performance in as hard tasks as abstractive document summarization (ADS), machine translation (MT), and video captioning (VC). NA mechanisms perform inference of context vectors; these constitute weighted sums of deterministic input sequence encodings, adaptively sourced over long temporal horizons. Inspired from recent work in the field of amortized variational inference (AVI), in this work we consider treating the context vectors generated by softattention (SA) models as latent variables, with approximate finite mixture model posteriors inferred via AVI. We posit that this formulation may yield stronger generalization capacity, in line with the outcomes of existing applications of AVI to deep networks. To illustrate our method, we implement it and experimentally evaluate it considering challenging ADS, VC, and MT benchmarks. This way, we exhibit its improved effectiveness over state-of-the-art alternatives.

heading: 1 INTRODUCTION
text: Sequence-to-sequence (seq2seq) or encoder-decoder models (Sutskever et al., 2014) constitute a novel solution to inferring relations between sequences of different lengths. They are broadly used for addressing tasks including machine translation (MT) (Bahdanau et al., 2015; Luong et al., 2015), abstractive document summarization (ADS), descriptive caption generation (DCG) (Xu et al., 2016), and question answering (QA) (Sukhbaatar et al., 2015), to name just a few. Seq2seq models comprise two distinct RNN models: an encoder RNN, and a decoder RNN. Their main principle of operation is based on the idea of learning to infer an intermediate context vector representation, c, which is “shared” among the two RNN modules of the model, i.e., the encoder and the decoder. Specifically, the encoder converts the source sequence to a context vector (e.g., the final state of the encoder RNN), while the decoder is presented with the inferred context vector to produce the target sequence.
Despite these merits, though, baseline seq2seq models cannot learn temporal dynamics over long horizons. This is due to the fact that a single context vector c is capable of encoding rather limited temporal information. This major limitation has been addressed via the development of neural attention (NA) mechanisms (Bahdanau et al., 2015). NA has been a major breakthrough in Deep Learning for Natural Language Processing, as it enables the decoder modules of seq2seq models to adaptively focus on temporally-varying subsets of the source sequence. This capacity, in turn, enables flexibly capturing long temporal dynamics in a computationally efficient manner.
Among the large collection of recently devised NA variants, the vast majority build upon the concept of Soft Attention (SA) (Xu et al., 2016). Under this rationale, at each sequence generation (decoding) step, NA-obtained context vectors essentially constitute deterministic representations of the dynamics between the source sequence and the decodings obtained thus far. However, recent work in the field of amortized variational inference (AVI) (Jimenez Rezende & Mohamed, 2015; Kingma & Welling, 2013) has shown that it is often useful to treat representations generated by deep networks as latent random variables. Indeed, it is now well-understood that, under such an inferential setup, the trained deep learning models become more effective in inferring representations that offer stronger generalizaton capacity, instead of getting trapped to representations of poor generalizaton quality. Then, model training reduces to inferring posterior distributions over the introduced latent variables. This can be performed by resorting to variational inference (Attias, 2000), where the sought variational posteriors are parameterized via appropriate deep networks.
Motivated from these research advances, in this paper we consider a novel formulation of SA. Specifically, we propose an NA mechanism formulation where the generated context vectors are considered random latent variables with finite mixture model posteriors, over which AVI is performed. We dub our approach amortized context vector inference (ACVI). To exhibit the efficacy of ACVI, we implement it into: (i) Pointer-Generator Networks (See et al., 2017), which constitute a state-of-the-art approach for addressing ADS tasks; (ii) baseline seq2seq models with additive SA, applied to the task of VC; and (iii) baseline seq2seq models with multiplicative SA, applied to MT.
The remainder of this paper is organized as follows: In Section 2, we briefly present the seq2seq model variants in the context of which we implement our method and exhibit its efficacy. In Section 3, we introduce the proposed approach, and elaborate on its training and inference algorithms. In Section 4, we perform an extensive experimental evaluation of our approach using benchmark ADS, MT, and VC datasets. Finally, in the concluding Section, we summarize the contribution of this work.

heading: 2 METHODOLOGICAL BACKGROUND

heading: 2.1 ABSTRACTIVE DOCUMENT SUMMARIZATION
text: ADS consists in not only copying from an original document, but also learning to generate new sentences or novel words during the summarization process. The introduction of seq2seq models has rendered ADS both feasible and effective (Rush et al., 2015; Zeng et al., 2017). Dealing with out-ofvocabulary (OOV) words was one of the main difficulties that early ADS models were confronted with. Word and/or phrase repetition was a second issue. The pointer-generator model presented in (See et al., 2017) constitutes one of the most comprehensive efforts towards ameliorating these issues.
In a nutshell, this model comprises one bidirectional LSTM (Hochreiter & Schmidhuber, 1997) (BiLSTM) encoder, and a unidirectional LSTM decoder, which incorporates an SA mechanism (Bahdanau et al., 2015). The word embedding of each token, xi, i ∈ {1, . . . , N}, in the source sequence (document) is presented to the encoder BiLSTM; this obtains a representation (encoding) hi = [ −→ h i; ←− h i], where −→ h i is the corresponding forward LSTM state, and ←− h i is the corresponding backward LSTM state. Then, at each generation step, t, the decoder LSTM gets as input the (word embedding of the) previous token in the target sequence. During training, this is the previous word in the available reference summary; during inference, this is the previous generated word. On this basis, the decoder updates its internal state, st, which is then presented to the postulated SA network. Specifically, the attention distribution, at, is given by:
eit = v T tanh(Whhi +Wsst + battn) (1)
at = softmax(et), et = [e i t]i (2)
where the W· are trainable weight matrices, battn is a trainable bias vector, and v is a trainable parameter vector of the same size as battn. Then, the model updates the maintained context vector, ct, by taking an weighted average of all the source token encodings; in that average, the used weights are the inferred attention probabilities. We obtain:
ct = ∑ i aithi (3)
Eventually, the predictive distribution over the next generated word yields:
P vocabt = softmax(V ′tanh(V [st; ct] + b) + b ′ ) (4)
where V and V ′ are trainable weight matrices, while b and b′ are trainable bias vectors.
In parallel, the network also computes an additional probability, pgent , which expresses whether the next output should be generated by sampling from the predictive distribution, P vocabt , or the model should simply copy one of the already available source sequence tokens. This mechanism allows for the model to cope with OOV words; it is defined via a simple sigmoid layer of the form:
pgent = σ(w T c ct +w T s st +w T x xt + bptr) (5)
where xt is the decoder input, while the w· and bptr are trainable parameter vectors. The probability of copying the ith source sequence token is considered equal to the corresponding attention probability,
ait. Eventually, the obtained probability that the next output word will be β (found either in the vocabulary or among the source sequence tokens) yields:
Pt(β) = p gen t P vocab t (β) + (1− p gen t ) ∑ i:βi=β ait (6)
Finally, a coverage mechanism may also be employed (Tu et al., 2016), as a means of penalizing words that have already received attention in the past, to prevent repetition. Specifically, the coverage vector, kt, is defined as:
kt = [k i t] N i=1 = t−1∑ τ=0 aτ (7)
Using the so-obtained coverage vector, expression (1) is modified as follows:
eit = v T tanh(Whhi +Wsst +wkk i t + battn) (8)
where wk is a trainable parameter vector of size similar to v. Model training is performed via minimization of the categorical cross-entropy, augmented with a coverage term of the form:
λ ∑ i ∑ t min(ait, c i t) (9)
Here, λ controls the influence of the coverage term; in the remainder of this work, we set λ = 1.

heading: 2.2 VIDEO CAPTIONING
text: Seq2seq models with attention have been successfully applied to several datasets of multimodal nature. Video captioning constitutes a popular such application. In this work, we consider a simple seq2seq model with additive SA that comprises a BiLSTM encoder, an LSTM decoder, and an output distribution of the form (4). The used encoder is presented with visual features obtained from a pretrained convolutional neural network (CNN). Using a pretrained CNN as our employed visual feature extractor ensures that all the evaluated attention models are presented with identical feature descriptors of the available raw data. Hence, it facilitates fairness in the comparative evaluation of our proposed attention mechanism. We elaborate on the specific model configuration in Section 4.2.

heading: 2.3 MACHINE TRANSLATION
text: Machine translation constitutes one of the first sequential data modeling applications where seq2seq models were shown to obtain state-of-the-art performance. In this work, we perform MT by means of a baseline seq2seq model comprising a BiLSTM encoder, an LSTM decoder, a predictive distribution over the next generated word which is given by (4), and a multiplicative SA mechanism. The latter is described by (Luong et al., 2015): eit = hiWst (10) in conjunction with Eq. (2); therein, W is a trainable weights matrix. Our consideration here of multiplicative SA both serves the purpose of implementing and evaluating our approach under diverse SA variants, and is congruent with the best reported results in the related literature.

heading: 3 PROPOSED APPROACH
text: We begin by introducing the core assumption that the computed context vectors, ct, constitute latent random variables. Further, we assume that, at each decoding step, t, the corresponding context vector, ct, is drawn from a distribution associated with one of the available source sequence encodings, {hi}Ni=1. The selection of the source sequence encoding to associate with is determined from the output sequence via the decoder state, st, as we explain next.
Let us introduce the set of binary latent indicator variables, {zit}Ni=1, zit ∈ {0, 1}, with zit = 1 denoting that the context vector ct is drawn from the ith density, that is the density associated with the ith source encoding, hi, and zit = 0 otherwise. Then, we postulate the following hierarchical model:
ct|zit = 1;D ∼ p(θ(hi)) (11)
zit = 1|D ∼ πit(ait) (12) where D comprises the set of both the source and target training sequences, θ denotes the parameters set of the context vector conditional density, and πit denotes the probability of drawing from the ith conditional at time t. Notably, we assume that the component assignment probabilities, πit, are functions of the attention probabilities, ait. Thus, the selection of the mixture component density that we draw the context vector from at decoding time t is directly determined from the value of the current decoder state, st, via the corresponding attention probabilities. A higher affinity of the current decoder state st with the ith encoding, hi, at time t, results in higher probability that the context vector be drawn from the corresponding conditional density.
Having defined the hierarchical model (11)-(12), it is important that we examine the resulting expression of the posterior density p(ct;D). By marginalizing over (11) and (12), we obtain:
p(ct;D) = N∑ i=1 πit(a i t) p(θ(hi)) (13)
In other words, we obtain a finite mixture model posterior over the context vectors, with mixture conditional densities associated with the available source sequence encodings, and mixture weights that are functions of the corresponding attention vectors, and are therefore determined by the target sequences.
In addition, it is interesting to compare this expression to the definition of context vectors under the conventional SA scheme. From (3), we observe that conventional SA is merely a special case of our proposed model, obtained by introducing two assumptions: (i) that the postulated mixture component assignment probabilities are identity functions of the associated attention probabilities, i.e.
p(zt;D) = Cat(zt ∣∣πt), zt = [zit]Ni=1,πt = [πit(ait)]Ni=1
s.t. πit(a i t) , p(z i t = 1;D) = ait = softmax(et);
(14)
and (ii) that the conditional densities of the context vectors have all their mass concentrated on hi, that is they collapse onto the single point, hi:
p(ct|zit = 1;D) = δ(hi) (15)
Indeed, by combining (13) - (15), we yield:
p(ct;D) = N∑ i=1 aitδ(hi) (16)
whence we obtain (3) with probability 1.
Thus, our approach replaces the simplistic conditional density expression (15) with a more appropriate family p(θ(hi)), as in (13). Based on the literature of AVI, e.g. (Jimenez Rezende & Mohamed, 2015; Kingma & Welling, 2013; Sønderby et al., 2016), we posit that such a stochastic latent variable consideration may result in significant advantages for the postulated seq2seq model. Specifically, our trained model becomes more agile in searching for effective context representations, as opposed to getting trapped to poor local solutions.
In the following, we examine conditional densities of Gaussian form. Adopting the inferential rationale of AVI, we consider that these conditional Gaussians are parameterized via the postulated BiLSTM encoder. Specifically, we assume:
p(ct|zit = 1;D) = N ( ct|hi,diag(σ2(hi)) ) (17)
where logσ2(h) = ReLU(h) (18)
ReLU(·) is a trainable ReLU layer of size dim(h), and the encodings, hi, are obtained from a BiLSTM encoder, similar to conventional models. Hence:
p(ct;D) = N∑ i=1 aitN ( ct|hi,diag(σ2(hi)) ) (19)
Thus, we have arrived at an approximate (variational) posterior expression for the context vectors, ct. In our variational treatment, both the component-conditional means, hi, and their variances, σ2(hi), are obtained from (amortizing) neural networks presented with the source sequences. On the other hand, though, the assignment probabilities, πit, in the variational posterior are taken as the attention probabilities, ait. Thus, they are determined by the target sequences, which are generated from the decoder of the model. Hence, our treatment represents a valid approximate posterior formulation, overall conditioned on both the source and target sequences.
This concludes the formulation of ACVI.
Relation to Recent Work. From the above exhibition, it becomes apparent that our approach generalizes the concept of neural attention by introducing stochasticity in the computation of context vectors. As we have already discussed, the ultimate goal of this construction is to allow for inferring representations of better generalization capacity, by leveraging Bayesian inference arguments.
We emphasize that this is in stark contrast to recent efforts toward generalizing neural attention by deriving more complex attention distributions. For instance, (Kim et al., 2017) have recently introduced structured attention. In that work, the model infers complex posterior probabilities over the assignment latent variables, as opposed to using a simplistic gating function. Specifically, instead of considering independent assignments, they postulate the first-order Markov dynamics assumption:
p({zt}Tt=1;D) = p(z1;D) T−1∏ t=1 p(zt+1|zt;D) (20)
Thus, (Kim et al., 2017) compute posterior distributions over the attention assignments, while ACVI provides a method for obtaining improved representations through the inferred context vectors. Note also that Eq. (20) gives rise to the need of executing much more computationally complex algorithms to perform attention distribution inference, e.g. the forward-backward algorithm (Rabiner, 1989). In contrast, our method imposes computational costs comparable to conventional SA.
Similar is the innovation in the variational attention method, recently presented (Deng et al., 2018). In essence, its key conceptual difference from structured attention is the consideration of full independence between the attention assignments {zt}Tt=1. Among the several alternatives considered in (Deng et al., 2018) to obtain stochastic gradient estimators of low variance, it was found that an approach using REINFORCE (Williams, 1992) along with a specialized baseline was effective.
Another noteworthy recent work, closer related to ACVI, is the variational encoder-decoder (VED) method presented in (Bahuleyan et al., 2018). Among the several alternative formulations considered in that paper, the one that clearly outperformed the baselines in terms of the obtained accuracy (BLEU scores) combined seq2seq models with SA with an extra variational autoencoder (VAE) module. This way, apart from the context vector, which is computed under the standard SA scheme, an additional latent vector ξ is essentially inferred. The imposed prior over it is a standard N (0, I), while the inferred posterior is a diagonal Gaussian parameterised by a BiLSTM network presented with the input sequence; the final BiLSTM state vector is presented to dense layers that output the posterior means and variances of the latent vectors ξ. Both the context vector, c, as well as the latent vectors, ξ, are fed to the final softmax layer of the model that yields the generated output symbols.
We shall provide comparisons to all these related approaches in the experimental section of our paper.
Training Algorithm. To perform training of a seq2seq model equipped with the ACVI mechanism, we resort to maximization of the resulting evidence lower-bound (ELBO) expression. To this end, we need first to introduce some prior assumption over the context latent variables, ct. To serve the purpose of simplicity, and also offer a valid way to effect model regularization, we consider:
p(ct) = N ( ct|0, I) (21)
On the grounds of these assumptions, it is easy to show that the resulting ELBO expression becomes: L = ∑ t { Ep(ct;D)[−Jt] −KL[p(ct;D)||p(ct)] } (22)
In this expression, Ep(ct;D)[−Jt] is the posterior expectation of the model log-likelihood, which is an integral part of the ELBO definition. In the following, we approximate all the entailed ELBO
terms by drawing MC samples from the context vector posterior. In this work, we are dealing with a one-out-of-many predictive selection; hence, the model likelihood is a simple Categorical. As such, Ep(ct;D)[−Jt] essentially reduces to the negative categorical cross-entropy of the model, averaged over multiple MC samples of the context vectors, drawn from (19). Besides, to ensure that the resulting MC estimators will be of low variance, we adopt the reparameterization trick. To this end, we rely on the posterior expressions (17) and (14); we express the drawn MC samples as follows:
c (k) t = N∑ i=1 z (k) ti c (k) ti (23)
In this expression, the c(k)ti are samples from the conditional Gaussians (17), which employ the standard reparameterization trick rationale, as applied to Gaussian variables:
c (k) ti = hi + σ(hi) ◦ (k) ti , ∼ N (0, I) (24)
On the other hand, the z(k)ti are samples from the Categorical distribution (14). To allow for performing backpropagation through these samples, while ensuring that the obtained gradients will be of low variance, we may draw z(k)ti by making use of the Gumbel-Softmax relaxation (Jang et al., 2017). We have empirically found it suffices that we employ the Gumbel-Softmax trick for the last 10% of the model training iterations1; previously, we merely adopt the following heuristic, without any statistically significant performance deviation: We use a simple weighted average of the samples c(k)ti , with the weights being the attention probabilities, ait:
c (k) t ← N∑ i=1 aitc (k) ti (25)
This way, we alleviate the computational costs of employing the Gumbel-Softmax relaxation, which dominates the costs of sampling from the mixture posterior (19).
Having obtained a reparameterization of the model ELBO that guarantees low variance estimators, we proceed to its maximization by resorting to a modern, off-the-shelf, stochastic gradient optimizer. Specifically, we adopt simple stochastic gradient descent (SGD) for the MT tasks, and Adam with its default settings (Kingma & Ba, 2015) for the rest.
Inference Algorithm. To perform target decoding by means of a seq2seq model that employs the ACVI mechanism, we resort to Beam search (Russel & Norvig). In our experiments, Beam width is set to five for the ADS and VC tasks (Sections 4.1 and 4.2), and to ten for the MT tasks (Section 4.3).

heading: 4 EXPERIMENTAL EVALUATION2

heading: 4.1 ABSTRACTIVE DOCUMENT SUMMARIZATION
text: Our experiments are based on the non-anonymized CNN/Daily Mail dataset, similar to the experiments of (See et al., 2017). To obtain some comparative results, we use pointer-generator networks as our
1We use a Gumbel-Softmax temperature hyperparameter value of 0.5. 2We have developed our source codes in Python, using the TensorFlow library (Abadi et al., 2015).
evaluation platform (See et al., 2017); therein, we employ our ACVI mechanism, the standard SA mechanism used in (See et al., 2017), VED (Bahuleyan et al., 2018), variational attention (Deng et al., 2018), as well as structured attention using the first-order Markov assumption (20) (Kim et al., 2017). The observations presented to the encoder modules constitute 128-dimensional word embeddings of the original 50K-dimensional one-hot-vectors of the source tokens. Similarly, the observations presented to the decoder modules are 128-dimensional word embeddings pertaining to the summary tokens (reference tokens during training; generated tokens during inference). Both these embeddings are trained, as part of the overall training procedure of the evaluated models. To allow for faster training convergence, we split training into five phases, as suggested in (See et al., 2017). Following the suggestions in (See et al., 2017), we evaluate all approaches with LSTMs that comprise 256-dimensional states and do not employ Dropout. We have tested VED with various selections of the dimensionality of the autoencoder latent vectors, ξ; we report results with 128-dimensional latent vectors, which yielded the best performance in our experiments3.
Finally, for completeness sake, we also evaluate the Transformer network (Vaswani et al., 2017), which is a popular alternative to seq2seq models with SA, based on the notion of self-attention. Following Fevry (2018), Transformer is evaluated with 256-dimensional word embeddings, 4 encoding and decoding layers of 256 units each, 4 heads, and a Dropout rate of 0.2.
We use ROUGE4 (Lin, 2004) and METEOR5 (Denkowski & Lavie, 2014) as our performance metrics. METEOR is evaluated both in exact match mode (rewarding only exact matches between words) and full mode (additionally rewarding matching stems, synonyms and paraphrases). In all our experiments, we restrict the used vocabulary to the 50K most common words in the considered dataset, similar to (See et al., 2017). Note that this is significantly smaller than typical in the literature (Nallapati et al., 2016). Our quantitative evaluation is provided in Table 1. Some indicative examples of generated summaries can be found in Appendix A (Tables 7-10).
As we observe, utilization of ACVI outperforms all the alternatives by a large margin. It is also interesting that the Transformer network yields the lowest performance among the considered alternatives; the obtained results are actually very poor. This is commensurate with the results reported by other researchers, e.g. Fevry (2018).
Finally, it is interesting to examine whether ACVI increases the propensity of a trained model towards generating novel words, that is words that are not found in the source document, as well as the capacity to adopt OOV words. The related results are provided in Table 2. We observe that ACVI increases the number of generated novel words by 3 times compared to the best performing alternative, that is VED (Bahuleyan et al., 2018). In a similar vein, ACVI appears to help the model better cope with OOV words.
3This selection is also commensurate with the ξ dimensionality reported in Bahuleyan et al. (2018). 4pypi.python.org/pypi/pyrouge/. 5www.cs.cmu.edu/~alavie/METEOR.

heading: 4.2 VIDEO CAPTIONING
text: Our evaluation of the proposed approach in the context of a VC application is based on the Youtube2Text video corpus (Yao et al., 2015). We split the available dataset into a training set comprising the first 1,200 video clips, a validation set composed of 100 clips, and a test set comprising the last 600 clips in the dataset. To reduce the entailed memory requirements, we process only the first 240 frames of each video. To obtain some initial video frame descriptors, we employ a pretrained GoogLeNet CNN (Szegedy et al., 2015) (implementation provided in Caffe (Jia et al., 2014)). Specifically, we use the features extracted at the pool5/7x7_s1 layer of this pretrained model. We select 24 equally-spaced frames out of the first 240 from each video, and feed them into the prescribed CNN to obtain a 1024 dimensional frame-wise feature vector. These are the visual inputs presented to the trained models. All employed LSTMs entail 1000-dimensional states. These are mapped to 100-dimensional features via the matricesWh andWs in Eq. (1). The autoencoder latent variables, ξ, of VED are also selected to be 100-dimensional vectors. The decoders are presented with 256-dimensional word embeddings, obtained in a fashion similar to our ADS experiments. In all cases, we use Dropout with a rate of 0.5.
We yield some comparative results by evaluating seq2seq models configured as described in Section 2.2; we use ACVI, structured attention in the form (20), VED, variational attention, or the conventional SA mechanism. Our quantitative evaluation is performed on the grounds of the ROUGE-L and CIDEr (Vedantam et al., 2015) scores, on both the validation set and the test set. The obtained results are depicted in Table 3; they show that our method outperforms the alternatives by an important margin. It is also characteristic that Structured Attention yields essentially identical results with Variational Attention. Thus, the first-order Markovian assumption does not offer practical benefits when generating short sequences like the ones involved in VC. Finally, we provide some indicative examples of the generated results in Appendix B (Figs. 1-8).

heading: 4.3 MACHINE TRANSLATION
text: Our experiments make use of publicly available corpora, namely WMT’16 English-to-Romanian (En→Ro) and Romanian-to-English (Ro→En), as well as IWSLT’15 English-to-Vietnamese (En→Vi) and Vietnamese-to-English (Vi→En). We benchmark the evaluated models against word-based vocabularies, and present our results in terms of the BLEU score (Papineni et al., 2002).
Following the related literature, we utilize byte pair encoding (BPE) (Sennrich et al., 2016) in the case of the (En, Ro) pair. This allows for seamlessly handling rare words, by breaking a given vocabulary into a fixed-size vocabulary of variable-length character sequences (subwords). Subword vocabularies are shared among the languages of a source/destination pair. This way, we promote frequent subword units, thus improving the coverage of the available dictionary words.
We obtain some comparative performance results by evaluating seq2seq models using ACVI, conventional SA, structured attention, as well as both the variational alternatives (Bahuleyan et al., 2018; Deng et al., 2018) discussed in Section 3. The trained architecture is homogeneous across all our comparisons. Specifically, both the encoders and the decoders of the evaluated models are presented with 256-dimensional trainable word embeddings. We utilize 2-layer BiLSTM encoders, and 2-layer LSTM decoders; all comprise 256-dimensional hidden states on each layer, similar to the summarization task, and employ a Dropout rate of 0.2. For VED, we employ 100-dimensional latent variables ξ, following Bahuleyan et al. (2018). Finally, we also provide the performance of the
Transformer network. The trained Transformer network comprises 4 heads, 4 encoder/decoder layers of 256 units, and a Dropout rate of 0.1.
Our results in Table 4 show inferior performance for our variational inference-based competitors. We observe that VED is competitive to ACVI in two of the four development sets, but fails to generalize as well across test sets. Some indicative examples of generated outputs from the considered variational alternatives are provided in Appendix C, Tables 11-16.

heading: 4.4 FURTHER INVESTIGATION: DOMAIN ADAPTATION
text: Finally, we wish to examine the capability of ACVI to generalize across domains. We have already elaborated on our expectation that modeling the context vectors as latent random variables should yield improved generalization performance. We attribute to this fact the improved accuracy ACVI obtained in our experimental evaluations. However, if this is the case, one would probably expect the method to also generalize better across different domains.
To investigate this aspect, we use the trained ADS models described in Section 4.1 to generate summaries for the documents of the DUC2004 dataset 6. This is an English dataset comprising 500 documents. Each document contains 4 model summaries written by experts. In Table 5, we show how our method performs in this setting, and how it compares to the alternative variational methods considered in Section 4.1. We observe that ACVI yields a clear improvement over the alternatives, while all variational methods perform significantly better than baseline SA. These findings seem to support our theoretical intuitions.

heading: 5 CONCLUSIONS
text: In this work, we cast the problem of context vector computation for seq2seq-type models employing SA into amortized variational inference. We made this possible by considering that the sought context vectors are latent variables following a Gaussian mixture posterior; therein, the mixture component densities depend on the source sequence encodings, while the mixture weights depend on the target sequence attention probabilities. We exhibited the merits of our approach on seq2seq architectures addressing ADS, VC, and MT tasks; we used benchmark datasets in all cases.
We underline that our approach induces only negligible computational overheads compared to conventional SA. Specifically, the only extra trainable parameters that our approach postulates stem from Eq. (17); these are of extremely limited size compared to the overall model size, and correspond to merely few extra feedforward computations at inference time. Besides, our sampling strategy does not induce significant computational costs, since we adopt the reparameterization (25) for the most part of the model training algorithm. In the future, we aim to consider how ACVI can cope with power-law distributions (Chatzis & Demiris, 2012; Chatzis & Kosmopoulos, 2015); such a capacity is of importance to real-world natural language generation.

heading: APPENDIX A
text: We provide some further details on the experimental setup of Section 4.1. To begin with, the used dataset comprises 287,226 training pairs of documents and reference summaries, 13,368 validation pairs, and 11,490 test pairs. In this dataset, the average article length is 781 tokens; the average summary length is 3.75 sentences, with the average summary being 56 tokens long.
To allow for faster training convergence, we split it into five phases, following See et al. (2017). On each phase, we employ a different number of maximum encoding steps for the evaluated models (i.e., the size of the inferred attention vectors), as well as for the maximum allowed number of decoding steps. We provide the related details in Table 6. During these phases, we train the employed models with the coverage mechanism being disabled; that is, we set wk = 0. We enable this mechanism only after these five training phases conclude. Specifically, we perform a final 3K iterations of model training, during which we train the wk weights along with the rest of the model parameters.
In Tables 7-10, we provide some indicative examples of produced summaries. We also show what the initial document has been, as well as the available reference summary used for quantitative performance evaluation. In all cases, we annotate OOV words in italics, we highlight novel words in purple, we show contextual understanding in bold, while article fragments also included in the generated summary are highlighted in green.

heading: Article

heading: Article

heading: Article

heading: Article

heading: APPENDIX B
text: The considered Video Captioning task utilizes a dataset that comprises 1,970 video clips, each associated with multiple natural language descriptions. This results in a total of approximately 80,000 video / description pairs; the used vocabulary comprises approximately 16,000 unique words. The constituent topics cover a wide range of domains, including sports, animals and music. We preprocess the available descriptions only using the wordpunct tokenizer from the NLTK toolbox7.
Moving on, we provide some characteristic examples of generated video descriptions. In the captions of the figures that follow, we annotate minor deviations with blue color, and use red color to indicate major mistakes which imply wrong perception of the scene.
Reference Description: someone is cutting a piece of meat
Reference Description: a hamster is eating
7http:/s/www.nltk.org/index.html.
Reference Description: someone is pouring something into a bowl
Reference Description: a woman is riding a horse
Reference Description: a car is driving down the road
Reference Description: a boy is playing the guitar
Reference Description: a girl is riding a bicycle

heading: Source sentence

heading: Reference Translation

heading: Generated Translation - Baseline

heading: Generated Translation - Structured Attention

heading: Generated Translation - VED

heading: Generated Translation - Variational Attention

heading: Generated Translation - ACVI

heading: APPENDIX C
text: Let us first provide some details on the datasets used in the context of our MT experiments. The WMT’16 task comprises of data from combining the Europarl v7, News Commentary v10 and Common Crawl corpora. For the (En, Ro) pair, this amounts to∼400K parallel sentences. The shared vocabulary sizes (obtained from BPE) total ∼31.7K words. We use newsdev2016 as our development set (∼1.9K sentences), and newstest2016 as our test set (∼1.9K sentences) for the (En, Ro) pair. On the other hand, the IWSLT’15 task boasts a dataset with ∼133K training sentence pairs from translated TED talks, provided by the IWSLT 2015 Evaluation Campaign (Cettolo et al., 2015). Following the same preprocessing steps as in (Luong et al., 2015), we use TED tst2012 (∼1.5K sentences) as our validation set for hyperparameter tuning, and TED tst2013 (∼1.3K sentences) as our test set. The Vietnamese and English vocabulary sizes are ∼7.7K and ∼17.2K, respectively. We prefer default settings for the hyperparameters of the trained seq2seq models, as used in the code8. These hyper-parameters remain unchanged for the VED and Variational Attention implementations as well. We have migrated the code9 of the former, provided from the authors, to ensure identical data processing. For the latter, we use their codebase10 directly.
In conclusion, we provide some characteristic examples of generated translations for all examined models. In the Tables that follow, we annotate minor and major deviations from the reference translation with blue and red respectively. Synonyms are highlighted with green. We also indicate missing tokens by adding the [missing] identifier mid-sentence, i.e. verbs, articles, adjectives, etc.
8https://github.com/harvardnlp/struct-attn. 9https://github.com/HareeshBahuleyan/tf-var-attention
10https://github.com/harvardnlp/var-attn

heading: Generated Translation - ACVI

heading: Generated Translation - Variational Attention

heading: Generated Translation - VED

heading: Generated Translation - Structured Attention

heading: Generated Translation - Baseline

heading: Reference Translation

heading: Họ thậm chí sẽ sử dụng những công cụ như Trojan Scuinst để lây nhiễm vào máy tính của bạn , và từ đó họ có thể có được mọi thông tin bạn trao đổi , có được mọi cuộc hội thoại qua mạng của

heading: Source sentence

heading: Generated Translation - ACVI

heading: Generated Translation - Variational Attention

heading: Generated Translation - VED

heading: Generated Translation - Structured Attention

heading: Generated Translation - Baseline

heading: Reference Translation

heading: Source sentence

heading: Generated Translation - ACVI

heading: Generated Translation - Variational Attention

heading: Generated Translation - Structured Attention

heading: Generated Translation - Baseline

heading: Reference Translation

heading: Source sentence

heading: Generated Translation - ACVI

heading: Generated Translation - Variational Attention

heading: Generated Translation - Structured Attention

heading: Generated Translation - Baseline

heading: Reference Translation

heading: Source sentence

heading: Generated Translation - ACVI

heading: Generated Translation - Variational Attention

heading: Generated Translation - Structured Attention

heading: Generated Translation - Baseline

heading: Reference Translation

heading: Source sentence

Review summaries:
This paper tries to introduce stochasticity over context representation to enable the neural networks to search for effective representation so as to avoid getting trapped into local optima. The basic idea is to formulate the posterior probability of context representation via a mixture gaussian model. The standard attention model SA can be thought of as a special case of the proposed method. The proposed method outperforms a number of baselines on ADS, video captioning, and machine translation datasets.

This paper presents a new way to construct the context vector in seq2seq networks. Specifically, the context vector in the proposed model is treated as a latent variable with a normal prior. The posterior is "defined" as a mixture of the latent presentations of the input sequence and the mixing weight is the attention. To do inference, maximising VAE-like "ELBO" is used. The proposed model is reported to achieve better performance in the seq2seq tasks including document summarisation and video caption.