Introduction
Training autoregressive models via maximum likelihood
estimation (MLE) is a common strategy for representing sequential data. Such autoregressive models obtain state-of-the-art in modeling text (Brown et al., 2020 ; Chowdhery et al., 2022 ; Zhang et al., 2022 ) , speech (Oord et al., 2018 ; Conneau et al., 2021 ) , and video sequences (Babaeizadeh et al., 2018 ) .
In this simple and intuitive approach, the joint distribution of a sequence is factorized into a product of conditional distributions, with each sequence element conditioned on its history.
However, in their basic form, autoregressive models do not necessarily learn latent variables that encode informative content, as often desired. As such, supplementing autoregressive models with latent variables is a promising way to enhance control in sequence generation, and enable structured
representation learning.

A recent approach to latent variable modeling of sequences is to integrate autoregressive components into the variational autoencoder (VAE) framework (Kingma and Welling, 2014 ; Rezende et al., 2014 ) . Sequence VAEs offer a theoretically principled solution, but have not been widely adopted. Arguably, this is largely due to the phenomenon of posterior collapse (Bowman et al., 2016 ; Chen et al., 2017 ; Van Den Oord et al., 2017 ; Chen et al., 2020 ) . Posterior collapse describes when a VAE’s posterior probability over latent variables ‘collapses’ to the latent prior, rendering the latent space completely uninformative, or independent , of the data.
Recent years have seen an abundance of research into posterior collapse (Bowman et al., 2015 ; Chen et al., 2017 ; Lucas et al., 2019 ; Dai et al., 2020 ; Fang et al., 2021 ; Pang et al., 2021 ) , identifying the ‘power’ of autoregressive decoding as a main cause.
In particular, autoregressive decoders are shown to obtain satisfactory sequence modeling performance without utilizing the latent space (Bowman et al., 2016 ; Chen et al., 2017 ) .
This suggests that for sequence VAEs to progress, improved techniques are needed to understand and alleviate posterior collapse, and thereby learn informative latent structure.
In this work, we propose a theoretically principled way to mitigate posterior collapse when training autoregressive sequence VAEs.

The intuition that autoregressive VAE decoders are too ‘powerful’ has led to the notion of ‘weakening’ them, e.g. by regularization. However, weakening an autoregressive model arbitrarily might of course harm its performance, leading to a trade off between informativeness of the latent space and the quality of sequence modeling (or density estimation performance).
State-of-the-art VAEs apply variants of dropout (Srivastava et al., 2014 ) to the sequence input into the decoder: either to individual dimensions of sequence elements (Kim et al., 2018 ) , or to mask entire sequence elements (Iyyer et al., 2015 ; Bowman et al., 2016 ) .
In either case, dropout is generally applied uniformly at random, meaning that each sequence element has an equal probability of being dropped.
From the intuition that different words carry different information about the next word, this work investigates whether a non-uniform dropout policy can achieve a better trade-off between capturing latent information and sequence modeling performance.
Specifically, by stochastically dropping each sequence element according to its importance to the autoregressive decoder, our method takes a targeted approach to dropout relative to uniform sampling, maintaining sequence modeling performance while achieving the benefits of dropout on the latent space.
To apply a non-uniform, data-dependent dropout scheme, our approach introduces an ‘adversary’ that learns which elements to drop .
The proposed framework is represented in Figure 1 , theoretically developed in Section 4.1 and implemented in Section 4.2 .

The contributions of this paper can be summarized as follows: (i) We propose an adversarial approach to learning a stochastic dropout policy that mitigates posterior collapse in sequence VAEs. (ii) We show theoretically that dropping out sequence elements deducts mutual information not already learned by the model , between one sequence element and the next, and that our approach maximises this quantity, leading to a constrained minimax ELBO objective that explains an adversarial approach. (iii) We evaluate the proposed scheme on standard text benchmarks, showing that our approach renders an informative latent space, without trading-off but rather improving sentence modeling. (iv) We examine properties of the adversarial network and find that the adversary typically selects words that carry sentence semantics that fit our theoretical analysis.