RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Background
Here, we formally describe the use of autoregressive decoding within VAEs for sequential data and the problem of posterior collapse that often arises (refer to Appendix D for some clarification around the term posterior collapse ).

For notation: \bm{X} denotes a random variable,
(boldface) \bm{x} denotes a sequence that is a realization of \bm{X} ,
(subscript) \bm{x}_{i} denotes the i -th sequence element of \bm{x} and \bm{x}_{<i} collectively denotes all sequence elements prior \bm{x}_{i} . z denotes a global latent vector and (superscript) \bm{x}^{(i)} denotes the i -th sample from a dataset.

The task is to approximate the true probability density function p_{*}(\bm{x}) of a random sequence \bm{X}\sim p_{*}(\bm{x}) using a model p_{\theta}(\bm{x}) with parameters \theta . Given a set of observations \{\bm{x}^{(i)}\}^{N}_{i=1} , \theta can be estimated by maximum likelihood estimation (MLE):

\theta_{\emph{MLE}}=\operatorname*{argmax}_{\theta}\left[\frac{1}{N}\sum_{i=1}^{N}\log p_{\theta}(\bm{x}^{(i)})\right] (1)

If the data has latent generative factors z , modelled as p_{\theta}(\bm{x})\!=\!\int_{z}p_{\theta}(\bm{x}|z)p_{\theta}(z) , Eq 1 is intractable and one can instead maximize the evidence lower bound (ELBO), which for any observation \bm{x} is:

\log p(\bm{x})\,\geq\log p_{\theta}(\bm{x})\,\geqELBO_{\theta,\phi}(\bm{x})\,\triangleq\!\int_{z}q_{\phi}(z|\bm{x})\log p_{\theta}(\bm{x}|z)-\int_{z}q_{\phi}(z|\bm{x})\log\tfrac{q_{\phi}(z|\bm{x})}{p_{\theta}(z)} (2)

where, q_{\phi}(z|\bm{x}) approximates the posterior, parameterized by {\phi} , and p_{\theta}(z) is a prior distribution over latent variables.
In this work, we consider the framework of variational autoencoders (VAEs) (Kingma and Welling, 2014 ; Rezende et al., 2014 ) , where the encoder q_{\phi}(z|\bm{x}) and decoder p_{\theta}(\bm{x}|z) are parameterised by neural networks and p_{\theta}(z) is a standard Gaussian, denoted p_{0}(z) .
Further, we focus specifically on VAEs where the decoder p_{\theta}(\bm{x}|z) is autoregressive (Bowman et al., 2016 ; Chen et al., 2020 ; He et al., 2019 ) , meaning \log p_{\theta}(\bm{x}|z)\!=\!\sum_{i}\log p_{\theta}({x}_{i}|\bm{x}_{<i},z) , and the ELBO becomes:

ELBO^{{\textbf{AR}}}_{\theta,\phi}(\bm{x}) =\int_{z}q_{\phi}(z|\bm{x})\sum_{i}\log p_{\theta}(\bm{x}_{i}|\bm{x}_{<i},z)-\underbrace{\int_{z}q_{\phi}(z|\bm{x})\log\tfrac{q_{\phi}(z|\bm{x})}{p_{0}(z)}}_{KL} (3)
Posterior collapse:
Autoregressive models improve density estimation (Oord et al., 2016 ) by explicitly modeling statistical dependencies between sequence elements.
However, when used as a VAE decoder p_{\theta}(\bm{x}|z) , autoregressive models are often found to ignore z , i.e. p(\bm{x}|z)\!\approx\!p(\bm{x}) , and the posterior is said to ‘collapse’ to the prior, i.e. q_{\theta}(z|\bm{x})\!\approx\!p_{0}(z) , rendering z de facto independent of \bm{x} (Bowman et al., 2016 ; He et al., 2019 ) .
In terms of Eq 3 , this implies p_{\theta}(\bm{x}_{i}|\bm{x}_{<i},z)\!\approx\!p_{\theta}(\bm{x}_{i}|\bm{x}_{<i}) .
From this, posterior collapse can be considered due to the information about \bm{x}_{i} provided by \bm{x}_{<i} being such that each \bm{x}_{i} is (approximately) conditionally independent of z given \bm{x}_{<i} , in other words z is essentially redundant . Based on this, we interpret the notion that autoregressive models are too powerful for use as a VAE decoder to mean that they allow too much information flow .
Posterior collapse
The phenomenon of posterior collapse has been observed in the context of text (Bowman et al., 2016 ; Yang et al., 2017 ) , images (Chen et al., 2017 ; Razavi et al., 2018 ; Miladinović et al., 2021 ) , videos (Babaeizadeh et al., 2018 ; Miladinović et al., 2019a ) , speech (Chorowski et al., 2019 ) and graphs (Kipf et al., 2018 ) .
In broad terms, previous solutions can be divided into two complementary categories: (i) latent-variable-oriented methods that focus mainly on relaxing the KL penalty; and (ii) decoder-oriented methods that regularise or ‘weaken’ the autoregressive decoder.
Typically, both types are required in order to learn an informative latent space (Bowman et al., 2016 ; Chen et al., 2017 ; Kim et al., 2018 ) .

Latent-variable-oriented solutions Bowman et al. ( 2016 ) suggest KL annealing , a technique that introduces the KL term gradually into training according to a predefined schedule. Free bits (Kingma et al., 2016 ) prevent penalizing of the KL term if its magnitude is below a predefined threshold. Van Den Oord et al. ( 2017 ) use a discrete VAE, avoiding posterior collapse by design. Fu et al. ( 2019 ) apply a cyclical form of KL annealing. Razavi et al. ( 2018 ) constrain the variational family of the posterior distribution (encoder), preventing it from closely approximating the prior and so holding the KL term away from zero. Lagging inference networks (He et al., 2019 ) ‘aggressively’ optimize the encoder before each decoder update. A similar procedure is followed by Semi-Amortized VAEs (Kim et al., 2018 ) . Generative skip models (Dieng et al., 2019 ) introduce skip connections to create a more explicit link between the latent variables and the likelihood function. Sinha and Dieng ( 2021 ) propose a consistency regularizer by minimizing the KL divergence between the posterior approximations of an observation and a random transformation of it.

Decoder-oriented solutions Bowman et al. ( 2016 ) apply word dropout (Iyyer et al., 2015 ) to uniformly drop words during autoregressive decoding.
Other methods (Kim et al., 2018 ; He et al., 2019 ) apply parameter dropout (Srivastava et al., 2014 ) to word embeddings. Chen et al. ( 2017 ); Semeniuta et al. ( 2017 ); Yang et al. ( 2017 ) constrain the receptive field of the decoder, limiting the window of autoregression, however, this is not readily applicable to RNN architectures given their unbounded receptive field.
Our proposed adversarial method falls into this category as an extension of word dropout, which is then subsumed as a special case of adversarial word dropout (Section 4.2 ).
Non-uniform dropout
Previous work has recognized the benefit of adapting the dropout rate across different architectural components, though in entirely different contexts (Kingma et al., 2015 ; Gal et al., 2017 ; Achille and Soatto, 2018 ) .
These works introduce different approaches to regularizing the magnitude of weights or activations (groups of weights). Elements that are deemed irrelevant during training are then dropped. For instance, variational dropout (Kingma et al., 2015 ) has been used to prune weights in deep neural networks (Molchanov et al., 2017 ) . A key difference to our approach is that instead of a regularization term, we employ an adversary that selects elements for dropout during training based on their information content not yet learned by the model.
Exposure bias
The method by which the autoregressive decoder is trained, known as teacher forcing , bears a connection to exposure bias , which refers to the gap between training and inference (Ranzato et al., 2015 ) .
Namely, in language generation, a trained model generates sequences at test time without access to the ground truth history that was accessible during training. Since the model was not trained to continue its predictions, this can lead to error accumulation (Ranzato et al., 2015 ) .
The most common approach to tackle exposure bias is to switch between conditioning on ground truth and model predictions, with the latter being preferred towards the end of training (Daumé et al., 2009 ; Bengio et al., 2015 ; Ranzato et al., 2015 ) .
In preliminary studies, we implemented scheduled sampling (Bengio et al., 2015 ) but did not find it to outperform even uniform dropout.
Theoretical Basis
As a precursor to our adversarial approach, we first derive the effect of stochastic dropout on the ELBO under the autoregressive assumption (Equation 3 ).

Word dropout: Our interpretation of posterior collapse at the end of Section 2 suggests that to ‘weaken the decoder’ one should restrict the available information , such that z is no longer redundant and must capture some of the restricted information. Word dropout (Bowman et al., 2016 ) can be seen to do this by stochastically masking the previous word \bm{x}_{i\scalebox{0.7}[1.0]{$-$}1} with probability d_{i} , which substitutes p_{\theta}(\bm{x}_{i}|\bm{x}_{{<i}},z) with p_{\theta}(\bm{x}_{i}|\bm{x}_{{<i\scalebox{0.7}[1.0]{$-$}1}},z) in Equation 3 , i.e.:

ELBO^{{\textbf{WD}}}_{\theta,\phi}(\bm{x}) =\int_{z}q_{\phi}(z|\bm{x})\Big{\{}\sum_{i}(1\,\scalebox{0.7}[1.0]{$-$}\,d_{i})\log p_{\theta}(\bm{x}_{i}|\bm{x}_{{<i}},z)+d_{i}\log p_{\theta}(\bm{x}_{i}|\bm{x}_{{<i\scalebox{0.7}[1.0]{$-$}1}},z)\Big{\}}-KL (4) =\int_{z}q_{\phi}(z|\bm{x})\Big{\{}\sum_{i}\log p_{\theta}(\bm{x}_{i}|\bm{x}_{\tiny{<i}},z)-\sum_{i}d_{i}\ \!\!\underbrace{\log\tfrac{p_{\theta}(\bm{x}_{i}|\bm{x}_{\tiny{i\scalebox{0.7}[1.0]{$-$}1}},\bm{x}_{\tiny{<i\scalebox{0.7}[1.0]{$-$}1}},z)}{p_{\theta}(\bm{x}_{i}|\bm{x}_{\tiny<i\scalebox{0.7}[1.0]{$-$}1},z)}}_{PMI(\bm{x}_{i},\bm{x}_{i-1}|\bm{x}_{<i-1},z)}\Big{\}}-KL (5)

Equation 5 shows the effect (in expectation) of applying word dropout when maximising the ELBO with an autoregressive decoder in Eq 3 (of which ‘KL’ denotes the last term). In effect, a weighted sum is introduced with weights given by dropout probabilities d_{i} and components that define conditional point-wise mutual information (PMI), where classic point-wise mutual information between two variables is defined as \text{PMI}(\bm{x}_{1},\bm{x}_{2})\!=\!\tfrac{p(\bm{x}_{1}|\bm{x}_{2})}{p(\bm{x}_{1})} . Each PMI term in Eq 5 captures the information that one sequence element \bm{x}_{i\scalebox{0.7}[1.0]{$-$}1} has regarding the next \bm{x}_{i} , over and above any information from earlier elements \bm{x}_{<i\scalebox{0.7}[1.0]{$-$}1} and the latent state z . Since the dropout-adapted ELBO (Eq 5 ) is maximised w.r.t. \theta and \phi , the weighted PMI term is minimised, which is only achievable by increasing the information learned by the model : extracted from \bm{x}_{<i\scalebox{0.7}[1.0]{$-$}1} or captured in z .
Interestingly this shows that word dropout quite literally follows the earlier intuition and ‘weakens’ the decoder by restricting information . Thus, word dropout leads to a looser variational objective, i.e. \text{ELBO}^{{\textbf{WD}}}_{\theta,\phi}(\bm{x})\leq\text{ELBO}^{{\textbf{AR}}}_{\theta,\phi}(\bm{x}) . Rainforth et al. ( 2018 ) has argued that tighter bounds are not necessarily better. Hence, our analysis provides a justification for using word dropout.
The minimax objective:
Under uniform dropout, all d_{i} are equal over a sequence, and using dropout to ‘push information into z ’ is applied evenly. However, some sequence elements may hold more information than others, e.g. in language, given syntactic rules and a summary of the previous text, some words may be highly predictable without specifically knowing which word preceded them, whereas others may depend heavily on their predecessor despite the other information. This suggests that information content may be non-uniform and so applying dropout non-uniformly may be more appropriate. We, therefore, propose targeted dropout of elements according to their ‘incremental’ information content defined by the PMI terms, i.e. that not yet learned by the model. Further, since such incremental information content is not explicitly computed and will vary over training as the model learns, we train an adaptive dropout schedule to continually maximise the information dropped, and so minimize Eq 5 , with respect to d_{i} . Since Eq 5 is maximised with respect to all other parameters, this leads to a minimax objective.

We introduce the two ‘players’ in the proposed minimax setting: \text{VAE}_{\phi,\theta} , trained to maximize the ELBO, and the adversary \text{A}_{\psi} trained to do the opposite by determining each probability d_{i} of dropping out the previous sequence element \bm{x}_{i\scalebox{0.7}[1.0]{$-$}1} , preventing it from helping to predict its successor \bm{x}_{i} . The full objective for an observed sequence is given by

\max_{\phi,\theta}\;\min_{\psi}\;\mathbb{E}_{\bm{x}\sim p_{*}(\bm{x})}\left[\mathcal{L}_{\psi,\phi,\theta}(\bm{x})+\mathcal{R}_{\psi}(\bm{x})\right] (6)

where \mathcal{R}_{\psi}(\bm{x}) is a regularization term explained below (Section 4.3 ) and

\mathcal{L}_{\psi,\phi,\theta}=\mathbb{E} {}_{q_{\phi}(z|\bm{x})}\left[\sum_{i}\log p_{\theta}(\bm{x}_{i}|mask_{K,\psi}(\bm{x}_{i\scalebox{0.7}[1.0]{$-$}1};\bm{x}),\bm{x}_{<i\scalebox{0.7}[1.0]{$-$}1},z)\right]- KL(q_{\phi}(z|\bm{x})||p_{0}(z)) (7)

\mathcal{L}_{\psi,\phi,\theta}(\bm{x}) is a modified version of the ELBO in Equation 5 with the additional parameters \psi of the adversary \text{A}_{\psi} . The adversary manifests via the masking operator mask {}_{K,\psi}(\cdot\,;\bm{x}) , conditioned on the entire input sequence \bm{x} . Since unconstrained minimisation of Eq 7 w.r.t \psi would cause all elements to be dropped out, a constraint K is introduced such that exactly K elements of \bm{x} are dropped during autoregressive decoding. In practice, K is sequence length-dependent, treated as a hyperparameter that controls the level of adversarial regularization.
The adversary plays no role at test time.
Implementation
A high-level overview of our framework is shown in Figure 1 .
An encoder maps the input sequence into a hidden representation to produce the parameters of q_{\phi}(z|x) , from which z is sampled using the ‘reparametrization trick’ (Kingma and Welling, 2014 ; Rezende et al., 2014 ) . We implement the decoder using a unidirectional LSTM ( Hochreiter and Schmidhuber, ) for consistency with the RNN-VAE network of Bowman et al. ( 2016 ) , because LSTMs are the most popular backbone for sequence VAEs and to facilitate comparison with prior work. Inspired by the recent work of (Dieng et al., 2019 ) , we choose a Double-LSTM recurrent unit
that elucidates the skip-connections to promote higher latent information content flow (see Appendix A for details).

As described in Section 4.1 , the main innovation of our proposed method is the adversarial training procedure: instead of dropping out decoder inputs uniformly at random, we introduce a trainable adversary to drop words most important for the VAE to reconstruct the original sequence.
Below we describe implementation details of the adversary, specifying the relevant components in Figure 1 . A more comprehensive figure specific to the implemented architecture is provided in Appendix B . Note that the theoretical framework in Section 4.1 permits many alternative approaches for generating per word dropout probabilities d_{i} , such as attention mechanisms mentioned above. We leave further refinement of the adversarial architecture to future work.
Producing dropout scores
To determine which elements to drop out of a sequence, the adversary \text{A}_{\psi} samples a score s_{i}\!\in\!\mathbb{R} for each sequence element \bm{x}_{i} . Sequence elements with the K smallest scores are dropped.
The scores are sampled from distributions p_{\psi}(s_{i}|\bm{x}) conditioned on the input sequence \bm{x} , modelled as a series of Gaussians with mean ( \mu ) and variance ( \sigma ) parameterised by the outputs of a unidirectional LSTM.
Scores are generated using the reparameterization trick, reducing gradient variance (Kingma and Welling, 2014 ) .
Thus, p_{\psi}(s_{i}|\bm{x}) is described as

 s_{i}|\bm{x}\sim\mathcal{N}(\mu_{i},\sigma_{i};\bm{x},\psi)\quad\qquad\qquadwhere,\quad[\mu_{i},\sigma_{i}]=Linear_{\psi}(LSTM_{i,\psi}(\bm{x}))

The stochastic generation of scores gives the adversary an ‘exploratory’ capability during training, e.g. preventing it becoming ‘stuck’ on a set of sequence elements with high information content that the model cannot learn, meaning that their PMI terms (in Eq 5 ) remain large and the adversary repeatedly selects them to minimise the ELBO.
Top-K word selection
Based on the sampled scores \bm{s}\!=\!\{s_{i}\}_{i=1}^{T} , the subset of K words with the smallest scores are masked during decoding. To estimate gradients of the objective with respect to the parameters \psi , we use a stochastic softmax trick from Paulus et al. ( 2020a ) : during training, the stochastic subset selected in the forward pass is relaxed to admit a (biased) reparameterization gradient in the backward pass.
In our experiments, we use a straight-through variant (Bengio et al., 2013 ; Paulus et al., 2020b ) of the trick for our method to resemble word dropout, i.e. to produce discrete values \in\!\{0,1\} . The relaxation is only used in the backward pass to compute the gradient estimator.
Gradient reversal
The final component of the adversarial network is the gradient reversal layer (Ganin et al., 2016 ) .
In the forward pass, the layer performs no transformation to the input; in the backward pass, the gradients are negated.
Gradient reversal offers a computationally simple method to ensure that the parameters \psi of \text{A}_{\psi} are updated such that ELBO is minimized. If \bm{I} is an identity matrix, the ‘pseudo-function’ of gradient reversal can be described as

 f(\bm{x})=\bm{x}\qquad(forward pass)\qquad\qquad\qquad\frac{\partial f(\bm{x})}{\partial\bm{x}}=-\bm{I}\qquad(backward pass)
Optimization challenges
Since our adversarial dropout network is fully differentiable, it is readily optimized by gradient methods, such as backpropagation. However, the magnitudes of the word dropout scores \bm{s} sampled from distributions parameterized by an LSTM are unconstrained.
Controlling the magnitude of the scores was found to be important for the adversary to maintain its exploratory capability (discussed above). We therefore add a KL-divergence between the distribution of each score p(s_{i}|\bm{x}_{i}) and a standard Gaussian p_{0}(s_{i})\!=\!\mathcal{N}(s_{i};0,1) to ‘regularize’ scores in Eq ( 6 ), subject to a scalar \lambda\!>\!0 :

\mathcal{R}_{\psi}(\bm{x})=\lambda\sum_{i} KL(p_{\psi}(s_{i}|\bm{x})\,||\,p_{0}(s_{i})) (8)
Adversarial vs. random word dropout
Standard word dropout (Iyyer et al., 2015 ; Bowman et al., 2016 ) can be seen as a special case of adversarial word dropout.
By setting \lambda sufficiently high, the regularization term in Eq 8 can be made to dominate such that p_{\psi}(s_{i}|\bm{x})\approx p_{0}(s_{i}) and all scores s_{i} are effectively sampled from a standard Gaussian p_{0}(\bm{s}_{i}) and words are dropped approximately uniformly.
On the other hand, for small values of \lambda , \text{A}_{\psi} will learn which words \text{VAE}_{\phi,\theta} relies upon most to accurately decode the sequence and target those for drop out.
As a result, whilst adding another hyperparameter to calibrate, \lambda offers a simple way to moderate the difference between adversarial word dropout and standard uniform word dropout,
which we consider in our experiments.
Experiments
Here we present the results of our experiments that: (i) demonstrate that a sequence VAE trained with adversarial word dropout (AWD) outperforms other sequence VAEs; it achieves improved sentence modeling performance and/or improved informativeness of the latent space; (ii) examine the contributions and behaviour of the adversarial network’s components and hyperparameters, and (iii) qualitatively study the trained adversary and VAE.
Datasets
We conducted experiments on 4 different datasets: Yahoo questions and answers (Yang et al., 2017 ) , Yelp reviews (Yang et al., 2017 ) , Penn Tree Bank (PTB) Marcus et al. ( 1993 ) and downsampled Stanford Natural Language
Inference ( SNLI ) corpus (Bowman et al., 2015 ; Li et al., 2019 ) .
Yahoo, Yelp and PTB datasets were used in many previous works (Yang et al., 2017 ; Kim et al., 2018 ; He et al., 2019 ; Fu et al., 2019 ; Dieng et al., 2019 ) hence were used to benchmark our proposed method against comparable related works and also against standard word dropout. Yahoo, Yelp and PTB contain sentences with average lengths of 78, 96 and 22 words respectively, while SNLI sentences are much shorter with an average length of 9 words and are more suitable for qualitative studies. Yahoo, Yelp and SNLI datasets contain 100K sentences in the training set, 10K in the validation set, and
10K in the test set, while PTB is much smaller with a total of 42K sentences.
Experimental setup
We (re-)implement the standard VAE, a VAE with standard uniform word dropout and a VAE with our adversarial dropout method. On each dataset, we performed the same grid search over both learning rate (from \{0.0001,0.001,0.1,1\} ) and dropout rate R (from \{0.2,0.3,0.4,0.5\} ) for both the word dropout baseline and our method. This gives 16 different hyper-parameter configurations for each method on each dataset. For training, we also use an exponential learning decay of 0.96 as in (Li and Arora, 2019 ) , increased the hidden state sze of the decoder LSTM from 1024 to 2048 (except on SNLI), applied Polyak averaging (Polyak and Juditsky, 1992 ) with a coefficient of 0.9995 and used KL annealing (Bowman et al., 2016 ) .

We apply early stopping based on validation ELBO and repeat each experiment for five different random seeds to report standard deviations. All experiments are performed on a 12GB Nvidia TitanXP GPU with an average run time of 4 hours for Yelp and Yahoo and 1 hour for SNLI.
Metrics
Overall model performance is assessed based on two main aspects: (i) sentence modeling – measured with respect to ELBO and perplexity (PPL). ELBO and PPL quantify the ability of the trained decoder to recognise or generate natural language, reflecting the quality of density estimation; (ii) latent space informativeness – measured with respect to the KL term in Eq ( 7 ) and mutual information I(\bm{X};Z) between observed and latent variables. I(\bm{X};Z) is computed using the procedure of Hoffman and Johnson ( 2016 ) , as implemented by Dieng et al. ( 2019 ) .
As in comparable works (Yang et al., 2017 ; Kim et al., 2018 ; He et al., 2019 ; Dieng et al., 2019 ; Li et al., 2019 ) , hyperparameters are set based on a balanced assessment of these metrics.
Adversary hyperparameters
To apply the global dropout rate R to the number K of elements to be dropped out we use K=\emph{round}(R\times T) where round computes the closest integer. The additional hyperparameter \lambda in Eq 8 was set globally for all datasets to \lambda=1 . This was based on an initial exploratory analysis on the Yahoo dataset, where we grid searched \lambda in \{0.001,0.01,0.1,1,5,10\} for various combinations of dropout and learning rates to find that \lambda=1 consistently achieved best validation performance. Please refer to the detailed analysis of \lambda in the appendix.