Learning to Drop Out: An Adversarial Approach to Training Sequence VAEs
Authors: Đorđe Miladinović
Kumar Shridhar
Kushal Jain
Max B. Paulus
Joachim M. Buhmann
Carl Allen
Venue: NeurIPS 2022 Accept (2022)
Keywords: VAE
dropout
posterior collapse

Abstract: In principle, applying variational autoencoders (VAEs) to sequential data offers a method for controlled sequence generation, manipulation, and structured representation learning. However, training sequence VAEs is challenging: autoregressive decoders can often explain the data without utilizing the latent space, known as posterior collapse. To mitigate this, state-of-the-art models `weaken' the `powerful decoder' by applying uniformly random dropout to the decoder input.
We show theoretically that this removes pointwise mutual information provided by the decoder input, which is compensated for by utilizing the latent space. We then propose an adversarial training strategy to achieve information-based stochastic dropout. Compared to uniform dropout on standard text benchmark datasets, our targeted approach increases both sequence modeling performance and the information captured in the latent space.

Learning to Drop Out: An Adversarial Approach to Training Sequence VAEs
Dorde Miladinovié
Kumar Shridhar
Kushal Jain 9
Max B. Paulus
Joachim M. Buhmann
Mrinmaya Sachan
Carl Allen
GGKKai
*ETH Zürich
University of California, San Diego
Abstract
In principle, applying variational autoencoders (VAEs) to sequential data offers a method for controlled sequence generation, manipulation, and structured represen- tation learning. However, training sequence VAžs is challenging: autoregressive decoders can often explain the data without utilizing the latent space, known posterior collapse. To mitigate this, state-of-the-an models 'weaken' the 'powerful' decoder by applying uniformly random dropout the decoder input. We show the- oretically that this removes pointwise mutual information provided by the decoder input, which is compensated for by utilizing the latent space. We then propose an adversarial training strategy to achieve information-basee stochastic dropout. Compared uniform dropout on standard text benchmark datasets, our targeted approach increases both sequence modeling performance and the information captured in the latent space.
Introduction
Training autoregressive models via maximum likelihood estimation (MLE) a common strategy for representing sequential data. Such autoregressive models obtain state-of-the-art in modeling text [Brown et al. 2020 Chowdhery al. 2022 Zhang et al. 2022 speech [Oord al. 2018 Conneau et al. 2021] and video sequences Babaeizadeh et al. 2018 J. In this simple and intuitive approach, the joint distribution of sequence factorized into a product of conditional distributions, with each sequence element conditioned on its history. However, in their basic form, autoregressive models do not necessarily learn latent variables that encode informative content, as often desired. such, supplementing autoregressive models with latent variables is promising way to enhance control in sequence generation, and enable structured representation learning.
A recent approach to latent variable modeling of sequences is to integrate autoregressive components into the variational autoencoder (VAE) framework (Kingma and Welling 2014 Rezende al. 2014). Sequence VAEs offer theoretically principled solution, but have not been widely adopted. Arguably. this is largely due to the phenomenon of posterior collapse [Bowman al. 2016 Chen et al. 2017 [hen Van Den Oord et al. 2017 Chen et al. 2020). Posterior collapse describes when a VAE's posterior probability over latent variables 'collapses' to the latent prior, rendering the latent space completely
uninformative, or independent, of the data. Recent years have seen an abundance of research into
posterior collapse [Bowman al.. 2015 Chen et al., 2017]] Lucas et al.) 2019 Dai et al. 2020 Fang et al. 2021 Pang et al. 2021], identifying the 'power" of autoregressive decoding
main cause. In particular, autoregressive decoders are shown obtain satisfactory sequence modeling performance without utilizing the latent space (Bowman et al.] [0066] Chen et al. 2017]. This suggests that for sequence VAES to progress, improved techniques are needed to understand and alleviate posterior
"Equal contribution; correspondence at: (eluma@@ethz.om
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Encoder (ag al)
Decode
C2
C,
VAE
are sequenze
awesooee
Advereary
dut
Gripoct Protatakils
Figure 1: Adversarial training of sequence VAEs. Our proposed model comprises an encoder, an autoregressive sequence decoder (together an autoregressive VAE), and an adversary Ag. During training, the encoder learns a representation of full input sequence z; the same sequence serves as both input (together with the hidden representation) and target to train the decoder. The adversary learns to stochastically drop out sequence elements that the decoder requires most, masking (0/1) each sequence element v; with probability d;. See Section 4.2 for details.
collapse, and thereby learn informative latent structure. In this work, we propose theoretically principled way to mitigate posterior collapse when training autoregressive sequence VAEs.
The intuition that autoregressive VAE decoders too 'powerful' has led the notion of 'weakening' them, e.g. by regularization. However, weakening an autoregressive model arbitrarily might of course harm its performance, leading trade off between informativeness of the latent space and the quality of sequence modeling (or density estimation performance). State-of-the-an VAEs apply variants of dropout (Srivastava al. 2014 the sequence input into the decoder: either individual dimensions of sequence elements Kim et al. 2018), or to mask entire sequence elements [lyyer] al. 2015 Bowman et al. (0166) In either case, dropout generally applied uniformly random, meaning that each sequence element has an equal probability of being dropped. From the intuition that different words carry different information about the next word, this work investigates whether a non-uniform dropout policy can achieve a better trade-off between capturing latent information and sequence modeling performance. Specifically, by stochastically dropping each sequence element according to its importance to the autoregressive decoder, our method takes a targeted approach to dropout relative to uniform sampling, maintaining sequence modeling performance while achieving the benefits of dropout on the latent space. To apply non-uniform, data-dependent dropout scheme, our approach introduces an 'adversary' that learns which elements to drop. The proposed framework is represented in Figure 1 theoretically developed in Section 4.1 and implemented in Section 4.2
The contributions of this paper can be summarized follows: (i) We propose an adversarial approach to learning a stochastic dropout policy that mitigates posterior collapse in sequence VAEs. (ii) We show theoretically that dropping out sequence elements deducts mutual information learned by the model, between one sequence element and the next, and that our approach maximises this quantity, leading constrained minimax ELBO objective that explains an adversarial approach. (iii) We evaluate the proposed scheme on standard text benchmarks, showing that our approach renders an informative latent space, without trading-off but rather improving sentence modeling. (iv) We examine properties of the adversarial network and find that the adversary typically selects words that carry sentence semantics that fit our theoretical analysis.
already
2 Background
Here, we formally describe the use of autoregressive decoding within VAEs for sequential data and the problem of posterior collapse that often arises (refer Appendix for some clarification around the term posterior collapse).

---

For notation: X denotes a random variable, (boldface) z denotes a sequence that is realization of X, (subscript) æ; denotes the i-th sequence element of æ and <<i collectively denotes all sequence clements prior æt. z denotes global latent vector and (superscript) (+)) denotes the i-th sample from a dataset.
The task is approximate the true probability density function p. (a) of random sequence X (x) using model pe(x) with parameters 6. Given set of observations (z(1)) Ji=1 O can be estimated by maximum likelihood estimation (MLE):
MMLE = argmnax
Σ log po (i)
(I)
If the data has latent generative factors 2, modelled as pe (x) (x)==ff( (o(2222))o Eq is intractable and one can instead maximize the evidence lower bound (ELBO), which for any observation æ is:
log p(x) 2 log (ax) 2 ELBOO,,(()) & S (0(()ar)) log log - (2((2/2)) log
(=(m))) Po(=)
where, (2(x)) approximates the posterior, parameterized by ø. and (2()) is a prior distribution
over latent variables. In this work, we consider the framework of variational autoencoders (VAEs)
Kingma and Welling 2014 Rezende et al. 2014], where the encoder (x(2)))) and decoder po (x/22)
parameterised by neural networks and (=) is standard Gaussian, denoted po(2). Further,
focus specifically on VAFs where the decoder po (x]l²) is autoregressive [Bowman et al. 2016 Chen
al. 2020
He et al. 2019], meaning log po
E; log (2; mr 2), and the ELBO becomes:
ELBOA (x)
(0(2(2))
Σ log pe (a: (<.,))
(2(22)) log (a(a))
po(=)
KL
Posterior collapse: Autoregressive models improve density estimation [Oord et al. 2016) by
explicitly modeling statistical dependencies between sequence elements. However, when used as a VAF decoder po(ar) (ar z). autoregressive models are often found ignore i.e. p(ar p())) and the posterior is said to 'collapse' to the prior, i.e. (2|1z) Pp(o)), rendering = de facto independent of a [Bowman al., 2016 He al. 2019 In terms of Eq this implies ( | 2) Pe (az (= From this, posterior collapse can be considered due to the information about æ; provided by æ< being such that each æ; is (approximately) conditionally independent of z given wei. in other words z is essentially redundant. Based on this, we interpret the notion that autoregressive models are too powerful for use as VAE decoder to mean that they allow too much information flow.
3 Related Work
Posterior collapse
The phenomenon of posterior collapse has been observed in the context of text
[Bowman et al. (0166] Yang et al. [0177) images [Chen et al.) a 2017 Razavi al. 2018 Miladinovié et al. 2021], videos [Babaeizadeh et al. 2018 Miladinović et al. 2019a), speech [Chorowski et al., 2019) and graphs (Kipf et al.) 2018). In broad terms, previous solutions can be divided into two complementary categories: (i) naltent-vrriable-oriee
methods that focus mainly on relaxing the KL. penalty; and (ii) decoder-orientedd methods that regularise or 'weaken' the autoregressive decoder. Typically, both types required in order to learn informative latent space [Bowman et al. 2016
Chen et al. 2017 Kim et al. 2018).
solutions Bowman et al. 2016 suggest KL annealing, technique that introduces the KL term gradually into training according to a predefined schedule. Free bits (Kingma et al. 2016) prevent penalizing of the KL term if its magnitude is below a predefined threshold. Van Den Oord al. 2017] use a discrete VAE, avoiding posterior collapse by design. Fu al. 2019 apply cyclical form of KI annealing. Razavi et al. (2088) constrain the variational family of the posterior distribution (encoder), preventing from closely approximating the prior and so holding the
Latent-variable-orieee
KL term away from zero. Lagging inference networks [He et al. 2019 'aggressively' optimize the
3

---

encoder before each decoder update. A similar procedure is followed by Semi-Amortized VAEs Kim al., 2018). Generative skip models [Dieng et al. 2019 introduce skip connections to create more explicit link between the latent variables and the likelihood function. [Sinha and Dieng (202T propose a consistency regularizer by minimizing the KI. divergence between the posterior approximations of an observation and random transformation of it.
Decoder-oriented solutions Bowman et al. [2016] apply word dropout [lyyer et al.
2015 to
uniformly drop words during autoregressive decoding. Other methods [Kim el al. 2018
He et al.
2019] apply parameter dropout (Srivastava et al. 2014] to word embeddings. Chen et al. [2017].
Semeniuta et al. [2017], Yang et al. [2017] constrain the receptive field of the decoder, limiting the window of autoregression, however, this is not readily applicable to RNN architectures given their unbounded receptive field. Our proposed adversarial method falls into this category as an extension of word dropout, which is then subsumed as a special case of adversarial word dropout (Section 4.2)
Previous work has recognized the benefit of adapting the dropout rate across different architectural components, though in entirely different contexts [Kingma al., 2015 Gal et al. 2017 Achille and Soatto| 2018). These works introduce different approaches to regularizing the magnitude of weights or activations (groups of weights). Elements that are deemed irrelevant during training are then dropped. For instance, variational dropout [Kingma et al. 2015) has been used to prune weights deep neural networks [Molchanov et al. 2017]. A key difference to our approach is that instead of regularization term, we employ an adversary that selects elements for dropout during training based on their information content not yet learned by the model.
Non-uniform dropout
Exposure bias The method by which the autoregressive decoder is trained, known as leacher forcing, bears connection to exposure bias, which refers to the gap between training and inference [Ranzato et al. 2015]. Namely, in language generation, trained model generates sequences at test time without access to the ground truth history that was accessible during training. Since the model was not trained to continue its predictions, this can lead to error accumulation [Ranzato et al., 20155] The most common approach to tackle exposure bias is to switch between conditioning on ground truth and model predictions, with the latter being preferred towards the end of training (Daumé et al. 2009 Bengio et al. [015]] Ranzato et al.) 2015). In preliminary studies, we implemented scheduled sampling [Bengio al. 2015]] but did not find it to outperform even uniform dropout.
4 Adversarial Word Dropout (AWD)
4.1 Theoretical Basis
As precursor to adversarial approach, first derive the effect of stochastic dropout the ELBO under the autoregressive assumption (Equation 3).
Our interpretation of posterior collapse at the end of Section 2 suggests that to 'weaken the decoder' one should restrict the available information, such that is no longer redundant and must capture some of the restricted information. Word dropout (Bowman et al. 2016 can be seen to do this by stochastically masking the previous word x;-1 with probability d,, which substitutes p=<x,/<<i with pe( xz,
Word dropout:
(2 2) in Equation 3 i.c.:
WD (x) = (0(2) ELBO
(1 d:) log (2 2) + d, log (o ( 22)1)))1111) KL
(4)
0,
= / (0(2))))
Σ (a:
Σ logog pola,le, po(m)lmee-11)))
KL
(5)
PMI (e(-1.)))
Equation 5 shows the effect (in expectation) of applying word dropout when maximising the ELBO with an autoregressive decoder in 3 which 'KL'' denotes the last term). In effect, weighted sum introduced with weights given by dropout probabilities d, and components that define conditional point-wise mutual information (PMI), where classic point-wise mutual information between two variables is defined as (MI x22 p( (222) Each PMI term in Eq captures the information p(my)
that one sequence element .; has regarding the next F; over and above any information from
carlier elements Ezi-1 and the latent state Since the propout-adapter ELBO (Eq 5 is maximised

---

w.r.t. O and é, the weighted PMI term is minimised, which is only achievable by increasing the information learned by the model: extracted from I <zj- or captured in z. Interestingly this shows that word dropout quite literally follows the earlier intuition and 'weakens" the decoder by restricting information. Thus, word dropout leads to a looser variational objective, i.e. ELBO ELBO
WD (x) I
0.0 (z). Rainforth et al. (2018) has argued that tighter bounds are not necessarily better. Hence, our analysis provides justification for using word dropout.
The minimax objective: Under uniform dropout, all d; are equal over sequence, and using dropout to 'push information into z' is applied evenly. However, some sequence elements may hold more information than others, e.g. in language, given syntactic rules and a summary of the previous text, some words may be highly predictable without specifically knowing which word preceded them, whereas others may depend heavily on their predecessor despite the other information. This suggests that information content may be non-uniform and so applying dropout non- con-uniformly may be more appropriate. We, therefore, propose targeted dropout of elements according to their 'incremental' information content defined by the PMI terms, i.e. that not yet learned by the model. Further, since such incremental information content not explicitly computed and will vary over training as the model learns, we train an adaptive dropout schedule to continually maximise the information dropped, and so minimize Eq with respect to d,. Since Eq maximised with respect all other parameters, this leads to minimax objective.
We introduce the two 'players' in the proposed minimax setting: VAE,,0, trained to maximize the ELBO, and the adversary Ag trained to do the opposite by determining each probability d, of dropping out the previous sequence element -; preventing from helping to predict its successor Tú. The full objective for an observed sequence is given by
max min Expp, (ax) oC (a) + R. (x]]
(6)
0.0
where (2r) is a regularization term explained below (Section 4.3) and
C = (+)())
Σ
log Po (mask/ ( ( xc
(7)
((x) is a modified version of the ELBO in Equation 5 with the additional parameters vi of the adversary Ag. The adversary manifests via the masking operator ma conditioned on the entire input sequence æ. Since unconstrained minimisation of Eq 7 w.r.t v would cause all elements be dropped out, a constraint K is introduced such that exactly K elements of are dropped during autoregressive decoding. In practice, K is sequence degphdee,,,,et,, treated as hyperparameter that controls the level of adversarial regularization. The adversary plays no role test time.
4.2 Implementation
A high-level overview of our framework is shown in Figure [ An encoder maps the input sequence
into a hidden representation to produce the parameters of go (2(xr) from which is sampled using the 'reparametrization trick' [Kingma and Welling 2014 Rezende et al. 2014]. We implement
the decoder using unidirectional LSTM [Hochreiter and Schmidhuber] for consistency with the RNN-VAE network of Bowman et al. (2016). because LSTMs are the most popular backbone for sequence VAEs and to facilitate comparison with prior work. Inspired by the recent work of (Dieng ct al. 2019). we choose Double-LSTM, recurrent unit that elucidates the skip-connections to promote higher latent information content flow (see Appendix for details).
As described in Section 4.T the main innovation of our proposed method is the adversarial training procedure: instead of dropping out decoder inputs uniformly at random, introduce trainable adversary to drop words most important for the VAE to reconstruct the original sequence. Below we describe implementation details of the adversary. specifying the relevant components in Figure T A more comprehensive figure specific to the implemented architecture is provided in Appendix B Note that the theoretical framework in Section 4. permits many alternative approaches for generating per word dropout probabilities d, such as attention mechanisms mentioned above. We leave further refinement of the adversarial architecture future work.
Producing dropout scores determine which elements to drop out of sequence, the adversary Ay samples score s; E IR for each sequence element x,. Sequence elements with the K smallest
5

---

PPL
-ELOO
KL
MI
PPL
-ELBO 1
KL 1
MI
Existing sequence
VA
Yahoo
Yelp
CNN Yang al. 2017
63.90
10.0
41.1
7.6
Lagging Hc ct al
2019
328.4 (0.2)
5.7 (0.7)
2.90
357.2(0.1)
3.8 (0.2)
2.4
SA Kim el al. 2018
60.40
7.19
Skip
Dieng
al. 2019
60.90
330.3
15.05
7.47
FBP
el al. 2019
59.51
330.3
15.02
Our sequence VAE
unregularized
60.30
328.8 (0.2)
4.2 (0.2)
3.14
40.1
356.4
(0.2)
2.3 (0.2)
1.0
word dropout [0.4]
59.55
329.5 (0.4)
14.4 (0.4)
13.6
38.5
354.2
(0.3)
5.9 (0.4)
4.9
AWD (ours) [0.3]
59.05
328.4 (0.3)
14.4 (0.4)
13.6
38.2
354.2 (0.3)
6.5 (0.4)
5.8
Table 1: Results of text modeling the Yahoo and Yelp datasets. Standard deviations provided
in the brackets. Squared bracket contains the dropout rate DR. PPL. perplexity; ELBO evidence lower bound; -L-in Eq (7) MI mutual information / (X; 2) from Section
scores dropped. The scores sampled from distributions (s; conditioned on the input sequence æ, modelled as a series of Gaussians with mean (f)) and variance (a) parameterised by the outputs of unidirectional LSTM. Scores generated using the reparameterization trick, reducing gradient variance (Kingma and Welling 2014]. Thus, Pe (8: (z) is described as
8: ~ . Nn,, 0z; v))
where,
[n4, 0a] = Linear, (LSTM,,, (2))
The stochastic generation of scores gives the adversary an 'exploratory' capability during training, e.g. preventing it becoming 'stuck' a of sequence elements with high information content that the model cannot learn, meaning that their PMI terms (in Eq remain large and the adversary repeatedly selects them minimise the ELBO.
Top-K word selection Based on the sampled scores s= (= the subset of K words with the smallest scores are masked during decoding. To estimate gradients of the objective with respect to the parameters vb, we use stochastic softmax trick from Paulus et al. [2020a]: during training, the stochastic subset selected the forward pass is relaxed to admit a (biased) reparameterization gradient in the backward pass. In experiments,
straight-through variant (Bengio et al.)
2013] Paulus et al. (0200b of the trick for our method to resemble word dropout, i.c. to produce discrete values 1]. The relaxation is only used in the backward pass compute the gradient estimator.
Gradient reversal The final component of the adversarial network is the gradient reversal layer [Ganin et al. [2166]. In the forward pass, the layer performs no transformation to the input; in the backward pass, the gradients are negated. Gradient reversal offers computationally simple method to ensure that the parameters y of Ag are updated such that ELBO is minimized. If I is an identity matrix, the pssuudoffunctio of gradient reversal can described as
0f(z) -I Ōx
((x) = x
(forward pass)
(backward pass)
4.3 Optimization challenges
Since our adversarial dropout network is fully differentiable, it is readily optimized by gradient methods, such as backpropagation. However, the magnitudes of the word dropout scores s sampled from distributions parameterized by an LSTM are unconstrained. Controlling the magnitude of the scores was found be important for the adversary to maintain exploratory capability (diseussed above). We therefore add a KL-divergence between the distribution of each score p(s,(z.) and standard Gaussian pa (a(s;) =N(s;; i 0, 1) to 'regularize' scores in Eq (6). subject to a scalar À 0:
(x) = A Σ KL(Po(s (1))(s)))))
(8)

---

Adversarial vs. random word dropout Standard word dropout [[yyer et al. 2015 Bowman et al. 2016] can be seen as a special case of adversarial word dropout. By setting À sufficietlly high, the regularization term in 8q can be made to dominate such that (s; po():) and all scores are effectively sampled from standard Gaussian (o;)) and words are dropped approximately uniformly. On the other hand, for small values of X, Au will learn which words VA relies upon most to accurately decode the sequence and target those for drop out. As a result, whilst adding another hyperparameter to calibrate, À offers simple way to moderate the difference between adversarial word dropout and standard uniform word dropout, which we consider in our experiments.
5
Experiments
Here we present the results of our experiments that: (i) demonstrate that sequence VAE trained with adversarial word dropout (AWD) outperforms other sequence VAEs; it achieves improved sentence modeling performance and/or improved informativeness of the latent space; (ii) examine the contributions and behaviour of the adversarial network's components and hyperparameters, and (iii) qualitatively study the trained adversary and VAE.
Datasets We conducted experiments on 4 different datasets: Yahoo questions and answers Yang al., 2017]. Yelp reviews (Yang al.) 2017], Penn Tree Bank (PTB) Marcus al. [1933 and
downsampled Stanford Natural Language Inference (SNLI) corpus [Bowman et al. 2015 Li
al.
2019). Yahoo, Yelp and PTB datasets
used in many previous works Yang al. 2017 Kim
et al, 2018 He et al. 2019 Fu et al. [2019] Dieng et al. 2019] hence were used to benchmark our proposed method against comparable related works and also against standard word dropout. Yahoo, Yelp and PTB contain sentences with average lengths of 78, 96 and 22 words respectively, while SNLI sentences qualitative studies. Yahoo, Yelp and SNLI datasets contain 100K sentences in the training set, 10K in the validation set, and 10K in the test set, while PTB is much smaller with total of 42K sentences.
much shorter with an average length of 9 words and more suitable for
Experimental setup We (re- (re))implement the standard VAE, VAE with standard uniform word dropout and VAE with our ad adversarial dropout method. On each dataset, we performed the same grid search over both learning rate (from {0.0001, 0.001, 0.0, 1)) and dropout rate R (from }} for both the word dropout baseline and our method. This gives 16 different hyper-
{0.2, 0.3, 0.4, 0.5}) parameter configurations for each method on each dataset. For training, we also use an exponential learning decay of 0.96 in [Li and Arora 2019), increased the hidden state sze of the decoder LSTM from 1024 to 2048 (except on SNLI), applied Polyak averaging (Polyak and Juditsky 1992 with coefficient of 0.9995 and used KL annealing [Bowman et al. 2016].
We apply early stopping based on validation ELBO and repeat each experiment for five different random seeds to report standard deviations. All experiments are performed on 12GB Nvidia TitanXP GPU with an average run time of 4 hours for Yelp and Yahoo and hour for SNLI.
Negetive ELBO
KL (1)
MI (12
35
50
25
20
15
10
Dropout
Figure 2: Adversarial uniform word dropout (SNLI), with ablation of the various dropout rates.
7

---

Metries Overall model performance is assessed based on two main aspects: (i) sentence modeling measured with respect to ELBO and perplexity (PPL). ELBO and PPL quantify the ability of the trained decoder to recognise or generate natural language, reflecting the quality of density estimation; (ii) latent space informativeness - measured with respect to the KI. term Eq 7 and mutual information T (X; Z) between observed and latent variables. T(X; 2) is computed using the procedure of Hoffman and Johnson| [2016], as implemented by Dieng et al. [2019]. As in comparable works (Yang et al., 2017 Kim et al. 2018 He et al. 2019 Dieng et al. 2019 Li et al. 2019 hyperparameters are set based on a balanced assessment of these metries.
Adversary hyperparameters To apply the global dropout rate R to the number K of elements to be dropped out we use K round(R R x T)) where round computes the closest integer. The additional hyperparameter X in Eq 8 was set globally for all datasets to À = 1. This was based on an initial exploratory analysis on the Yahoo dataset, where we grid searched in {0.001, 0.01, 0.1, 1, 5, 10) for various combinations of dropout and learning rates to find that À = 1 consistently achieved best validation performance. Please refer to the detailed analysis of the appendix.
5.1 Quantitative analysis
[ (i) the VAE trained with no dropout; (ii) the VAE trained with uniform word dropout [owmmnn et al. 2016] (dropout rate R-0.4 found to be best); (iii) a VAE trained with adversarial word dropout (Rt 0.3 found best); and (iv) previously reported results for comparable models (Section 3p. Our adversarial dropout method trains models that achieve better sentence modeling (lower BLBO, PPL) with an equally informative latent space (Yahoo, PTB) more informative latent space (higher KL, MI) while maintaining sentence modeling performance (Yclp) and improves both metrics SNLI (Appendix Table Thus, it allows users more effectively trade-off sentence modelling and informativenes of the latent space than standard word dropout. The gains are modest in size, but larger SNLI and PTB, and comparable those improvements reported in previous word, e.g., (He et al. 2019] (Appendix 5)) Our method also compares favourably to previous models, consistently achieving an improved balance between sentence modeling performance (e.g. PPL) and an informative latent space (e.g. MI). We note also that the vanilla VAE does not obtain lower perplexity than either VAE with word dropout, as might be anticipated if dropout were an arbitrary 'regularisation' method that may improve the latent space but at a cost to sequence modeling performance. This supports our theoretical analysis indicating that word dropout is not a typical 'regularizer' (Section 4.1), rather that it disrupts the flow of available information during training of the autoregressive decoder. foreing it to compensate by storing information in the latent space. Figure 2 compares adversarial and uniform word dropout, varying the dropout rate on the SNLI dataset. For any given dropout rate, adversarial dropout learns more informative latent space (with higher KL and MI) metrics with comparable to lower negative ELBO.
The role of X The hyperparameter À can be seen to determine the sxploration-exploiaaaiii trade-off of the adversary. As shown in Figure for small values of X, the magnitudes and standard deviations of word dropout scores grow large, causing the distribution of dropped words become concentrated. For large values of X, scores become very small with low variance and the adversary converges towards uniform word dropout. The right part Figure depicts why x is a good choice.
Score Norm
Score STD
Gradient Norm
0.8
0.6
0.2
0.001 0.01 0.1
1
10
0.001 0.01
0.1
10
0.001 0.01
0.1
1 20
Figure 3: The role of X. On Yahoo dataset, we computed various statistics from the data collected across iterations during one training run; (left and mid) Mean & norm and standard deviation of dropout score vectors (s); (right) Gradient norm of the adversary signifies the magnitude of the parameter updates and hence the quality of learning.
8

---

<S08> the
person giving the demonstration man <eos>
<SOS> dog participating race while wearing the number 6 <cos>
<SOS> three women are ta talking about their matching wigs and silver leggings <eos>
<808> there any
truth to the rumor that the host of the unk on unk greg _unk is dead
unk has been
down and there no new information on the web about the future of the
site or the unk greg unk what going on how could unk not comment on the past us
election nov. 7th nor the iranian nuclear program something must be wrong where is this guy
internet chat rooms have noted that unk was injured in fly fishing accident earlier
this
month details are unknown can anyone clear this up <eos>
<SOS>
is
okay
mind
Lunk
with
my goldfish yes
but
be
careful
you
may be
overwhelmed
when
you
connect to
such
clearly superior intellect <eos>>
Table 2: Analysis of the Adversary. For selected SNLI (upper) and Yahoo (lower) sentences, word dropout scores are from a trained adversary and normalized per sentence. Darker colouring indicates a higher dropout probability. Boxed words are those selected to be dropped. <sos> start of the sequence token; <eos> end of the sequence token; unk unknown token.
Qualitative analysis We obtain further insight into what the adversary learns by analyzing the word dropout scores for different sentences. Table that the adversary applies lower dropout probabilities to less informative words such 'unknown' tokens that replace all out-of-dictionay words, and so offer little information about the next word. Depending on the data semanties, the adversary selects different types of words: for SNLI, verbs tend to be picked that explain the activity, e.g. working and participating; for Yahoo, words are identified that carry question semanties, e.g. what, how, if. and when. Figure 4 (a) shows a quantitative analysis of dropout saliency map across different part-of-speech (POS) tags. Verb (verb), interjections (intj), and nouns (noun) have higher saliency scores (higher chances of being dropped) compared punctuation (punc), determiners (det), and the start (sos) and end tokens (cos) which are relatively easier to predict given previous words. We also show that adversarial training learns useful generative model with meaningful latent space by interpolating between sentences (Table 33. Computing BERT F1 score Zhang et al. (022) between the interpolated sentences with the source and target sentence shows the increasing trend toward the target sentence for each interpolation (as each interpolated sentence is getting away from the source and closer the target) and decreasing for the source sentence (Figure (b)).
70
60
50
90.5
30
90.0
20
89.5
890
055
wer in6, nounprosnuu dec con)
DUX ado ounc 505
Sontl
Sem12
SertE3
-OS
Incorpolated sentonces
(a) Quantitative analysis of dropout saliency map across different part-of-speech (POS) tags. Dotted horizontal line represents the average saliency across all words.
(h) BERT Fj score computed between the interpolated sentences with the source and target sentence. Sent represents the interpolated sentences (set three in our analysis).
Figure 4: Quantitative analysis of the interpolations and saliency map presented in the paper.

---

6 Conclusions
In this work, address the phenomenon of posterior collapse that occurs in autoregressive variational autoencoders (VAEs) used for sequential data. To mitigate posterior collapse, we propose a novel adversarial approach that learns to drop words based on their information content, leading to an improvement in both the learned latent structure and sequence modeling performance. We theoretically derive the effect of word dropout to show that, during training, it removes incremental information that each word provides about the next above that available from earlier words or the latent variable z. The only way for the model compensate and minimise the incremental information lost is to learn more information in the lateni space. We believe this finding provides novel and interesting insight as a means of manipulating information within hierarchical latent variable model since dropout is used to 'push" specific information into the latent variable. For future work, it may be interesting extend our method
wansformer-basee architectures (Vassaani et al. [201]] that can exacerbate posterior collapse when used for decoding and self-supervised learning, where imput (words in NLP, image patches computer vision) are often masked randomly (Devlin al.) 2019 He et al.
2022]. but learnt policy might improve performance or convergence.
Sequence VAES a promising framework that, beyond purely autoregressive models [Brown et al. 2020]. hold the prospect of controlled sequence generation. Improvements to such general methods may inevitably be used for good or ill, from the generation of targeted fake news on the one hand to the possibility of personalised human-computer interactions in, say, the medical domain on the other (e.g. for modeling sleep [Miladinović al., 2019b Nowak et al. (02T)). )) In future work, we plan to explore alternative implementation options, particularly of the adversary, and extend adversarial dropout other domains, such images, speech, or dynamical systems (Bauer al. 2017).
I 'm sure what all the hype about. 've been here a few times and just ok. nothing special. I would n't t go out of my way to come here.
've been here a few times and always been good. the food is good, but the service not so great. 've been here a few times and always been good. ''ve had the chicken and waffles and the service was good
Great place to go for quick bite to eat. the food is great and the service great. . i have been here a few times and have never been disappointed.
Great food, great service, and great service. i ''ve been here a few times and have never been disappointed
Table 3: Sentence interpolation (Yelp dataset). Representations of two sentences (top, bottom) are obtained by feeding them through an adversarially trained VAE encoder. Three linearly interpolated representations are passed the VAE decoder and sentences generated by greedy sampling (middle).
7
Acknowledgments
We thank Taylor Berg-Kirkpatrick for his thoughtful insights and valuable feodback. Carl is gratefully supported by an ETH AI Centre Postdoctoral Fellowship. responsible AI grant by the Haslerstiftung; Swiss National Science Foundation (project 201009), and an ETH Grant (ETH-19 21-1).
10

---

References
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEFE Transactions on Pattern Analysis and Machine Intelligence, 40 (12):2897-299)) 2902,, 2018. (Cited on
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Representations, 2018. (Cited on
Stefan Bauer, Nico S Gorbach, Djordje Miladinovic, and Joachim M Buhmann. Efficient and flexible inference for stochastic systems. Advances in Neural Information Processing Systems, 30, 2017. (Cited on IO)
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in Neural Information Processing Systems, 2015. (Cited on 4
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013. (Cited on
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing, 2015. (Cited on 17 17
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. In Computational Natural Language Learning, 2016. (Cited on
I6
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020. (Cited on [0
Guanyi Chen, Ruizhe Li, Xiao Li, Xiao Li, and Chenghua Lin. Improving variational autoencoder for text modelling with timestep-wise regularisation. ArXiv, abs/201 2020. (Cited on
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. International Conference on Learning Representations, 2017. (Cited on
Jan Chorowski, Ron J Weiss, Samy Bengio, and Aäron van den Oord. Unsupervised speech representation learning using wavenet autoencoders. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(12)) 2019. (Cited o3
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022. (Cited on [
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised Cross-Lingual Representation Learning for Speech Recognition. In Interspeech, 2021. (Cited [
Bin Dai, Ziyu Wang, and David Wipf. The usual suspects? Reassessing blame for VAE posterior collapse. In International Conference Machine Learning, 2020. (Cited on [
Hal Daumé, John Langford, and Daniel Marcu. Search-based structured prediction. Machine Learning, 2573)2297-325, 2009, (Cited 4n
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training trainiiiing of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. (Cited on 10)

---

Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. Avoiding latent variable collapse with generative skip models. In Interationall Conference on Artificial Intelligence and Statistics, 2019. (Cited on 5 8
16
Xianghong Fang, Haoli Bai, Jian Li, Zenglin Xu, Michael Lyu, and Irwin King. Discrete Auto- regressive Variational Attention Models for Text Modeling. In International Joint Conference on Neural Networks, 2021. (Cited on T
Hao Fu, Chumyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: simple approach mitigating KI. vanishing. North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019. (Cited on 3
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in Neural Information
Processing Systems, 2017. (Cited on
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Dovain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096-200) 2016. (Cited on 6 16)
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatricck Lagging Inference Networks and Posterior Collapse in Variational Autoencoders. In International Conference on Learning Representations, 2019. (Cited on
17)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022. (Cited on 10
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017. (Cited on [7]
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation. (Cited on I6
Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, 2016.
(Cited on (
Mohit lyyer, Varun Manjunatha, Jordan Bead-Graber, and Hal Daumé III. Deep unordered composi- tion rivals syntactic methods for text classification. In Association for Computational Linguistics and International Joint Conference on Natural Language Processing. 2015. (Cited on 2
Yoon Kim, Sam Wiseman, Andrew Miller, David Sontag, and Alexander Rush. Semi- amni-imortized variational autoencoders. In International Conference on Machine Leurning, 2018. (Cited on 2
7
8
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representutions, 2014. (Cited on 3 5 6)
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. (Cited on 4
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, 2016. (Cited on 3
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, volume 80 of Proceedings of Machine Learning Research, pages
PMLR, 2018. (Cited on 3
12

---

Bohan Li, Junxian He, Graham Neubig, Taylor Berg- Kirkpatrick, and Yiming Yang. A surprisingly effective fix for deep latent variable modeling of text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP), Hong Kong, China, November 2019. Association for Computational Linguistics. (Cited on 6
17
Zhiyuan Li and Sanjeev Arora, An exponential learning rate schedule for deep learning. arXiv preprint arXiv:1910.07454 2019. (Cited on
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Don't blame the elbo!
linear vae perspective on posterior collapse. In Advances in Neural Information Processing Systems, 2019. (Cited on !
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):11-330, 1993. URL
https://aclanthologgy org/ J93- J93-2004 (Cited on
7 17)
Dorde Miladinović, Muhammad Waleed Gondal, Bernhard Schölkopf, Joachim M Buhmann, and Stefan Bauer. Disentangled state space representations. arXiv preprint arXiv:1906.03225, 2019a. (Cited on ))
Dorde Miladinović, Christine Muheim, Stefan Bauer, Andrea Spinnler, Daniela Noain, Mojtaba Bandarabadi, Benjamin Gallusser, Gabriel Krummenacher, Christian Baumann, Antoine Adaman- tidis, et al. Spindle: End-to-end learning from eeg/emg to extrapolate animal sleep scoring across experimental settings, labs and species. PLoS computational biology, 15(4):e100666) 2019b. (Cited on I0)
Dorde Miladinović, Aleksandar Stanié, Stefan Bauer, Jürgen Schmidhuber, and Joachim M. Buh- mann. Spatial dependency networks: Neural layers for improved generative image modeling. In International Conference on Learning Representations, 2021. URL httpps://openrevieu net/
Forun/1d=14c4K9WBM (Cited on
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, 2017. (Cited on 4
Nora Nowak, Thomas Gaisl, Djordje Miladinovic, Ricards Marcinkevics, Martin Osswald, Stefan Bauer, Joachim Buhmann, Renato Zenobi, Pablo Sinues, Steven A Brown, et al. Rapid and reversible control of human metabolism by individual sleep states. Cell Reports, 37(4): 109903, 2021. (Cited on 10)
1794)00
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International Conference on Machine Learning, 2018. (Cited on
Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukeuoglu. Conditional image generation with pixelenn decoders. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS 16, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819, (Cited on
Bo Pang, Erik Nijkamp, Tian Han, and Ying Nian Wu. Generative text modeling through short inference. In Conference of the European Chapter of the Association for Computational Linguistics, 2021. (Cited on
Max Paulus, Dami Choi, Daniel Tarlow, Andreas Krause, and Chris Maddison. Gradient estimation with stochastic softmax tricks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5691-5704. Curran Associates, Inc., 2020a. (Cited on 6
Max B Paulus, Chris J Maddison, and Andreas Krause. Rao-blackwellizingg the ttraight-through gumbel-softmax gradient estimator. arXiv preprint arXiv:2010.04883 2020b. (Cited on 6
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838- 835-55, 1992. (Cited on 7
13

---

Tom Rainforth, Adam Kosiorek, Tuan Anh Le, Chris Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. Tighter variational bounds are not necessarily better. In International Conference on Machine Learning, pages 4277-4285. PMLR, 2018. (Cited 5n
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.066 06732, 2015. (Cited on 44)
Ali Razavi, Aaron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with vaesa-vees. In International Conference on Learning Representations, 2018. (Cited on
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages
PMLR, 2014. (Cited on
3
Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. A hybrid convolutional variational autoencoder for text generation. In Conference on Empirical Methods in Natural Language Processing, 2017. (Cited on
Samarth Sinha and Adji Bousso Dieng. Consistency regularization for variational auto-encoders, In Advances in Neural Information Processing Systems, 2021. (Cited on
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958 2014. (Cited on 2
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, 2017. (Cited on [3
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. (Cited on [Op
Zichao Yang. Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatricke Improved variational autoencoders for text modeling using dilated convolutions. In International Conference on Machine Learning, 2017. (Cited on
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00 01066, 2022. (Cited on T
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. 8th International Conference on Leurning Representations, ICLR 2020, Addis Abubu, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. (Cited on
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] (Section 4.2)
(b) Did you describe the limitations of your work? [Yes] (Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] (Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] (All Sections)
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] (Section 4.1
(b) Did you include complete proofs of all theoretical results? [Yes] (Section 4.1
3. If you experiments...
14

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] (Supplementary)
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] (Section
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] (Table [
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] (Section 5
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] (Section S
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "eKTkwjaYeuH", "reviewer": "Reviewer_WbUQ", "summary": "The authors tackle the problem known as \"posterior collapse\" which arises while applying variational autoencoders (VAEs) to sequential data in an autoregressive manner. The authors extend a previous method known as \"word dropout\" which uniformly drops words during the autoregressive decoding process. The authors suggest to formulate the problem as a minimax optimization task. They present an adversary, that predicts the most important words (in term of information gain) and mask them. To regularize this training scheme, the authors suggest to mask only the top K words, where K is an hyperparameter of the method. Another regularization which adds exploratory factor to the process is that the adversary estimates a Gaussian distribution parameters as scores for the masking process, then the scores are sampled from this distribution. Another regularization added to the method is that the estimated adversary Gaussian distribution score for each of the words  is also being regularized using KL-divergence term to the normal Gaussian distribution.", "questions": "- The paper suggests the section 4.1 permits alternative approaches for generating per word dropout probabilities. It would have been helpful to discuss approaches that the authors tried and did not work for them.", "limitations": "I believe there is no potential negative societal impact to this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is written in a clear and easy to follow manner, high clarity.\n- The approach is novel and original, although extended a previous uniform sampling method.\n- The work provides some new insights to the \"posterior collapse\" in terms of theoretical analysis.\n- The work present experimentation study which support the claims and proposed method in this paper\n\n**Weaknesses**\n- The significance of the work is not completely clear to me, I am not confident that the improvements over the baseline are significant.\n- The ablation study is missing a more in-depth analysis of the parameter \\lambda. It is not clear why the authors chose \\lambda=1 as their choice. \n- In Table 1, we do not see improvement in KL of MI over uniform sampling on the Yahoo dataset, the paper does not discuss it. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IYb8gan3epl", "reviewer": "Reviewer_Lidz", "summary": "This paper aims to address the posterior collapse issue that occurred during training sequence VAEs. To this end, the authors first provide theoretical support for the effectiveness of word-level Dropout regularization to mitigate this problem. Then, they propose a new non-uniform, data-dependent Dropout scheme by introducing the adversarial training strategy with proper regularization. Finally, the empirical effectiveness of the proposed method is demonstrated with three different datasets by comparing it with other sequence VAEs.\n\n", "questions": "Please carefully read the above cons and address the related questions and suggestions. ", "limitations": "The authors have adequately addressed their work's limitations and potential negative social impact at the end of the conclusions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### **Pros.**\n\n- **Clarity**. Overall, the writing is clear and easy to follow. In addition, the organization of the main draft is well-established.\n- **Well motivated problem and intuitive solution**. Resolving the posterior collapse issue is an important and interesting problem to solve. Also, the provided theoretical support and the proposed method are intuitive.\n\n### **Cons.**\n\n- **Insufficient empirical demonstration**. Although the clear advantage of the provided theoretical support and intuitive method, there are clear limitations in the current empirical results. \n  1. *Fairness with the hyper-parameter search*: as presented in lines 271-274, the authors conduct an extensive hyper-parameter search for the proposed method. However, such a hyper-parameter search is not conducted for the baselines, e.g., word Dropout with a fixed-rate 0.4. As the optimal hyper-parameter largely varies on each dataset, the authors should conduct the same hyper-parameter search procedure to baselines for a fair comparison. \n  2. *Limited improvement*: even though the adversarial training scheme might increase the overall computations during the training, the improvement is relatively marginal compared to the vanilla Word Dropout. \n  3. *Absence of the results with SNLI*: the authors mention that they have conducted experiments on three different datasets (lines 237-239). However, the results on only two datasets are presented in Table 1. Although they present some qualitative results on SNLI, quantitative results (like Table 1) should be provided to demonstrate its effectiveness in various scenarios. \n  4. *Quantitative support in addition to qualitative results in Tables 2 and 3*: although the provided qualitative results are intuitive and helpful to demonstrate the benefit of addressing the posterior collapse issue with the proposed method, they also have some concerns for the generalization to other samples. Therefore, if the authors can provide additional quantitative results, such as alignment with the saliency map for Table 2 or the similarity between original samples and interpolated ones, then it would significantly enhance the merit of the proposed method.\n\n### **Other Comments.**\n\n- Minor comments\n    - Mis-allocation; line 185 and line 190.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-IAry_ocigp", "reviewer": "Reviewer_KBue", "summary": "The paper aims to investigate the problem of posterior collapse that variational autoencoders often suffer from. To tackle this issue, the authors propose an adversarial training strategy. To be specific, an adversary is employed to learn the probabilities of the decoder input dropout. The method is first demonstrated theoretically to remove pointwise mutual information and then evidenced by positive experimental results on Yahoo, Yelp, and SNLI against several baselines with double LSTMs as the decoder. A quantitative analysis follows to demonstrate the method further (e.g., the trade-off between exploration and exploitation).", "questions": "1). Is double lstm a part of the contributions? This misleads the point of this work.\n\n2). Since the idea itself is fairly simple, are these notations and formulas necessary for its presentation?\n\n3). The ablation study regarding the double lstm is necessary. In other words, experiments against various types of the decoder are required.\n\n4). (line 152) The problems described lack the support of previous works.\n\n5). (line 185) A wrong reference to the appendix.\n\n6). (figure 2) What does the y-axis correspond to?\n\n7). Tables 2 & 3 can be placed in the appendix to make some space for more experiments and studies.\n", "limitations": "Limitations are mentioned in Section 6, while it seems to be not sufficient.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:1). The authors touch on the issue shared by most autoregressive decoders, that is, posterior collapse; thus, works in this direction, I believe, will benefit the whole community due to the popularity of the autoregressive decoders. 2). The method is well-motivated by the problem of posterior collapse and well-defined theoretically. The idea is fairly simple for an easy understanding and application. 3). The introduced method is successfully examined to outperform baselines by thoughtful experiments on several benchmarks. A quantitative analysis is conducted, showing specific examples of the adversary.\n\nWeaknesses:1). Technically speaking, the contribution of this work is incremental, and its technique deep is shallow. The proposed probabilistic word dropout is not that impressive or novel. To me, it sounds like a probabilistic teacher forcing. The massive notations and formulas appear to be not that necessary for me to understand the idea. 2). The improvements on three tasks over the previous works and self-implemented baselines are marginal. Further analysis beyond the main experiments is not sufficient. 3). The backbone is constrained to be the double LSTM, while popular transformers is not involved. Although this seems to work for different encoders, pre-trained language models are not covered as well. The application of this method to more extensive model structures remains a potential concern.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept