RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Quantitative analysis
Table 1 compares (i) the VAE trained with no dropout; (ii) the VAE trained with uniform word dropout (Bowman et al., 2016 ) (dropout rate R\!=\!0.4 found to be best); (iii) a VAE trained with adversarial word dropout ( R\!=\!0.3 found best); and (iv) previously reported results for comparable models (Section 3 ). Our adversarial dropout method trains models that achieve better sentence modeling (lower ELBO, PPL) with an equally informative latent space (Yahoo, PTB) or more informative latent space (higher KL, MI) while maintaining sentence modeling performance (Yelp) and improves both metrics on SNLI (Appendix Table 5 ) . Thus, it allows users to more effectively trade-off sentence modelling and informativenes of the latent space than standard word dropout. The gains are modest in size, but larger on SNLI and PTB, and comparable to those improvements reported in previous word, e.g., (He et al., 2019 ) (Appendix Table 4 , 5 ).
Our method also compares favourably to previous models, consistently achieving an improved balance between sentence modeling performance (e.g. PPL) and an informative latent space (e.g.  MI).
We note also that the vanilla VAE does not obtain a lower perplexity than either VAE with word dropout, as might be anticipated if dropout were an arbitrary ‘regularisation’ method that may improve the latent space but at a cost to sequence modeling performance. This supports our theoretical analysis indicating that word dropout is not a typical ‘regularizer’ (Section 4.1 ), rather that it disrupts the flow of available information during training of the autoregressive decoder, forcing it to compensate by storing information in the latent space.
Figure 2 compares adversarial and uniform word dropout, varying the dropout rate on the SNLI dataset. For any given dropout rate, adversarial dropout learns a more informative latent space (with higher KL and MI) metrics with comparable to lower negative ELBO.
The role of \lambda
The hyperparameter \lambda can be seen to determine the exploration-exploitation trade-off of the adversary. As shown in Figure 3 , for small values of \lambda , the magnitudes and standard deviations of word dropout scores grow large, causing the distribution of dropped words to become concentrated.
For large values of \lambda , scores become very small with low variance and the adversary converges towards uniform word dropout.
The right part of Figure 3 depicts why \lambda\!=\!1 is a good choice.
Qualitative analysis
We obtain further insight into what the adversary learns by analyzing the word dropout scores for different sentences. Table 2 shows that the adversary applies lower dropout probabilities to less informative words such as ‘unknown’ tokens that replace all out-of-dictionary words, and so offer little information about the next word. Depending on the data semantics, the adversary selects different types of words: for SNLI, verbs tend to be picked that explain the activity, e.g. working and participating ; for Yahoo, words are identified that carry question semantics, e.g. what , how , if , and when . Figure 4 (a) shows a quantitative analysis of dropout saliency map across different part-of-speech (POS) tags. Verb (verb), interjections (intj), and nouns (noun) have higher saliency scores (higher chances of being dropped) compared to punctuation (punc), determiners (det), and the start (sos) and end tokens (eos) which are relatively easier to predict given previous words. We also show that adversarial training learns a useful generative model with meaningful latent space by interpolating between sentences (Table 3 ). Computing BERT F_{1} score Zhang et al. ( 2020 ) between the interpolated sentences with the source and target sentence shows the increasing trend toward the target sentence for each interpolation (as each interpolated sentence is getting away from the source and closer to the target) and decreasing for the source sentence (Figure 4 (b)).
Conclusions
In this work, we address the phenomenon of posterior collapse that occurs in autoregressive variational autoencoders (VAEs) used for sequential data.
To mitigate posterior collapse, we propose a novel adversarial approach that learns to drop words based on their information content , leading to an improvement in both the learned latent structure and sequence modeling performance.
We theoretically derive the effect of word dropout to show that, during training, it removes incremental information that each word provides about the next above that available from earlier words or the latent variable z . The only way for the model to compensate and minimise the incremental information lost is to learn more information in the latent space . We believe this finding provides novel and interesting insight as a means of manipulating information within a hierarchical latent variable model since dropout is used to ‘push’ specific information into the latent variable. For future work, it may be interesting to extend our method to transformer-based architectures (Vaswani et al., 2017 ) that can exacerbate posterior collapse when used for decoding and self-supervised learning, where input (words in NLP, image patches in computer vision) are often masked randomly (Devlin et al., 2019 ; He et al., 2022 ) , but a learnt policy might improve performance or convergence.

Sequence VAEs are a promising framework that, beyond purely autoregressive models (Brown et al., 2020 ) , hold the prospect of controlled sequence generation. Improvements to such general methods may inevitably be used for good or ill, from the generation of targeted fake news on the one hand to the possibility of personalised human-computer interactions in, say, the medical domain on the other (e.g. for modeling sleep (Miladinović et al., 2019b ; Nowak et al., 2021 ) ).
In future work, we plan to explore alternative implementation options, particularly of the adversary, and extend adversarial dropout to other domains, such as images, speech, or dynamical systems (Bauer et al., 2017 ) .
Acknowledgments
We thank Taylor Berg-Kirkpatrick for his thoughtful insights and valuable feedback.
Carl is gratefully supported by an ETH AI Centre Postdoctoral Fellowship, a responsible AI grant by the Haslerstiftung; Swiss National Science Foundation (project # 201009), and an ETH Grant (ETH-19 21-1).
Checklist
For all authors…

Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (Section 4.2 )

Did you describe the limitations of your work? [Yes] (Section 6 )

Did you discuss any potential negative societal impacts of your work? [Yes] (Section 6 )

Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] (All Sections)

If you are including theoretical results…

Did you state the full set of assumptions of all theoretical results? [Yes] (Section 4.1 )

Did you include complete proofs of all theoretical results? [Yes] (Section 4.1 )

If you ran experiments…

Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] (Supplementary)

Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] (Section 5 )

Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] (Table 1 )

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] (Section 5 )

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…

If your work uses existing assets, did you cite the creators? [Yes] (Section 5 )

Did you mention the license of the assets? [N/A]

Did you include any new assets either in the supplemental material or as a URL? [N/A]

Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A]

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]

If you used crowdsourcing or conducted research with human subjects…

Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Figure 1 : Adversarial training of sequence VAEs. Our proposed model comprises an encoder , an autoregressive sequence decoder (together an autoregressive VAE), and an adversary A ψ .
During training, the encoder learns a representation of a full input sequence \bm{x} ; the same sequence serves as both input (together with the hidden representation) and target to train the decoder. The adversary learns to stochastically drop out sequence elements that the decoder requires most, masking (0/1) each sequence element \bm{x}_{i} with probability \bm{d_{i}} . See Section 4.2 for details.; Adversarial training of sequence VAEs. Our proposed model comprises an encoder , an autoregressive sequence decoder (together an autoregressive VAE), and an adversary A ψ .
During training, the encoder learns a representation of a full input sequence \bm{x} ; the same sequence serves as both input (together with the hidden representation) and target to train the decoder. The adversary learns to stochastically drop out sequence elements that the decoder requires most, masking (0/1) each sequence element \bm{x}_{i} with probability \bm{d_{i}} . See Section 4.2 for details.
Figure 2 : Adversarial vs uniform word dropout (SNLI), with ablation of the various dropout rates.; Adversarial vs uniform word dropout (SNLI), with ablation of the various dropout rates.
Figure 3 : The role of \lambda . On Yahoo dataset, we computed various statistics from the data collected across iterations during one training run; (left and mid) Mean \ell_{1} -norm and standard deviation of dropout score vectors ( \bm{s} ); (right) Gradient norm of the adversary – signifies the magnitude of the parameter updates and hence the quality of learning.; The role of \lambda . On Yahoo dataset, we computed various statistics from the data collected across iterations during one training run; (left and mid) Mean \ell_{1} -norm and standard deviation of dropout score vectors ( \bm{s} ); (right) Gradient norm of the adversary – signifies the magnitude of the parameter updates and hence the quality of learning.
(a) Quantitative analysis of dropout saliency map across different part-of-speech (POS) tags. Dotted horizontal line represents the average saliency across all words.; (b) BERT F_{1} score computed between the interpolated sentences with the source and target sentence. Sent represents the interpolated sentences (set to three in our analysis).; Quantitative analysis of the interpolations and saliency map presented in the paper.
Figure 5 : Double-LSTM.; Double-LSTM.
Figure 6 : Architectural details of our proposed method: Our variant of RNN-VAE [Bowman et al., 2016 ] consists of an LSTM-based sequence encoder (depicted in blue) and a Double-LSTM-based autoregressive sequence decoder (depicted in green) . During MLE training, the ground truth sequence history is teacher-forced to the autoregressive decoder which reduces the utilization of z . To regularize autoregressive decoding, the adversary \text{A}_{\psi} (depicted in orange) obstracts VAE by learning to drop out words (sequence elements) that VAE requires. Concretely, \text{A}_{\psi} selectively masks different words at decoder’s input, effectively performing input-conditioned word dropout. In the first stage, \text{A}_{\psi} stochastically produces unnormalized word dropout scores for each word in a sequence, using another LSTM that architecturally mirrors the one in the encoder. In the second stage, \text{A}_{\psi} selects a subset of K ’dropouts’ using the differentiable top- K word selection module. Finally, to ensure that \text{A}_{\psi} optimizes an objective that is an inverted version of the VAE objective, gradient reversal [Ganin et al., 2016 ] is applied to negate the gradients in the backward pass.; Architectural details of our proposed method: Our variant of RNN-VAE [Bowman et al., 2016 ] consists of an LSTM-based sequence encoder (depicted in blue) and a Double-LSTM-based autoregressive sequence decoder (depicted in green) . During MLE training, the ground truth sequence history is teacher-forced to the autoregressive decoder which reduces the utilization of z . To regularize autoregressive decoding, the adversary \text{A}_{\psi} (depicted in orange) obstracts VAE by learning to drop out words (sequence elements) that VAE requires. Concretely, \text{A}_{\psi} selectively masks different words at decoder’s input, effectively performing input-conditioned word dropout. In the first stage, \text{A}_{\psi} stochastically produces unnormalized word dropout scores for each word in a sequence, using another LSTM that architecturally mirrors the one in the encoder. In the second stage, \text{A}_{\psi} selects a subset of K ’dropouts’ using the differentiable top- K word selection module. Finally, to ensure that \text{A}_{\psi} optimizes an objective that is an inverted version of the VAE objective, gradient reversal [Ganin et al., 2016 ] is applied to negate the gradients in the backward pass.
Results of text modeling on the Yahoo and Yelp datasets. Standard deviations are provided in the brackets.
Squared bracket contains the dropout rate DR .
PPL – perplexity; ELBO – evidence lower bound; KL - in Eq ( 7 ); MI – mutual information I(\bm{X};Z) from Section 2 .
Analysis of the Adversary. For selected SNLI (upper) and Yahoo (lower) sentences, word dropout scores are from a trained adversary and normalized per sentence. Darker colouring indicates a higher dropout probability. Boxed words are those selected to be dropped. <sos> = start of the sequence token; <eos> = end of the sequence token; _unk = unknown token.
Sentence interpolation (Yelp dataset). Representations of two sentences (top, bottom) are obtained by feeding them through an adversarially trained VAE encoder. Three linearly interpolated representations are passed to the VAE decoder and sentences generated by greedy sampling (middle) .
Results of text modeling on the PTB dataset Marcus et al. [ 1993 ] . Standard deviations are provided in the brackets.
Squared bracket contains the dropout rate DR .
ELBO – evidence lower bound; KL - KL divergence; MI – mutual information.
Results of text modeling on the SNLI dataset . Standard deviations are provided in the brackets.
Squared bracket contains the dropout rate DR .
ELBO – evidence lower bound; KL - KL divergence; MI – mutual information.
Sentence interpolation on the Yelp dataset.
Unconditional sentence generation based on the Yahoo dataset.
Neighborhood exploration based on the Yahoo dataset. The original sentence taken from the Yahoo dataset (on top) was used to infer the parameters of the posterior. We then sampled from the posterior and decoded the sentences multiple times.