Contrastive Deterministic Autoencoders For Language Modeling
Authors: Amur Ghose
Pascal Poupart
Venue: EMNLP 2023 Findings (2023)
Keywords: Autoencoders
Contrastive
Transformers
TL;DR: Contrastive learning for Transformer/LSTM-autoencoders can be improved when they are deterministic.

Abstract: Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. Training VAEs, especially for text, often runs into the issue of posterior collapse, resulting in loss of representation quality. Deterministic autoencoders avoid this issue, and have been explored particularly well for images. It is however unclear how to best modify a deterministic model designed for images into a successful one for text. We show that with suitable adaptations, we can significantly improve on batch-normed VAEs (BN-VAEs), a strong benchmark for language modeling with VAEs, by replacing them with analogous deterministic models. We employ techniques from contrastive learning to control the entropy of the aggregate posterior of these models to make it Gaussian. The resulting models skip reparametrization steps in VAE modeling and avoid posterior collapse, while outperforming a broad range of VAE models on text generation and downstream tasks from representations. These improvements are shown to be consistent across both LSTM and Transformer-based VAE architectures. Appropriate comparisons to BERT/GPT-2 based results are also included. We also qualitatively examine the latent space through interpolation to supplement the quantitative aspects of the model.

Contrastive Deterministic Autoencoders For Language Modeling
Amur Ghose and Pascal Poupart
David R. Cheriton School of Computer Science
University of Waterioo, Waterioo, Canada Vector Institute, Toronto, Canada agghose@uwaterlloo.ca poopartt@wwatrlloo.co
Abstract
between P(z x1) and the isotropic Gaussian dis- tribution, VAEs can perform both generative and denoising (reconstructive) tasks. Minimization the KL loss allows the VAE to create an isotropic Gaussian distribution to sample and decode from. However, note that the isotropic Gaussian is not required and in NLP researchers have considered training the latent distribution (Fang et al., 2019) and learning structured, discrete representations (Zhao al., 2018b).
Variational autoencoders (VAEs) are popular family of generative models with wide appli- cability. Training VAEs, especially for text, often runs into the issue of posterior collapse, resulting in loss of representation quality, De- terministic autoencoders avoid this issue, and have been explored particularly well for im- ages. is however unclear how to best modify a deterministic model designed for images into a successful one for text. We show that with suitable adaptations, we can significantly im- prove batch-normed VAEs (BN-VAFEs), strong benchmark for language modeling with VAEs, by replacing them with analogous de- terministic models. We employ techniques from contrastive learning to control the entropy of the aggregate posterior of these models to make it Gaussian. The resulting models skip reparametrization steps in VAE modeling and avoid posterior collapse, while outperforming a broad range of VAE models on text gener- ation and downstream tasks from representa- tions. These improvements are shown to be consistent across both LSTM and Transformer- based VAE architectures. Appropriate compar- isons to BERT/GPT-2 based results also included. We also qualitatively examine the la- tent space through interpolation supplement the quantitative aspects of the model.
One of the most pressing problems in practical VAE training arises when the encoder's distribu- tion collapses to the standard Gaussian for every instance, that is, P(z| becomes = /V Vi. This problem is commonly termed as posterior collapse (Bowman et al., 2016; Razavi et al., 2019; Lucas et al., 2019; He et al.. 2019) and lies at the heart of modern issues with VAEs. Many fixes have been proposed towards this problem, ranging from setting lower bound on the KL term (6-VAE) (Razavi et al., 2019), aggressively training the en- coder and analyzing the 'lag" between the encoder and the decoder (Agg-VAE) (He et al., 2019), force meaningful usage of the code z; through skip con- nections (Skip-VAE) (Dieng et al., 2019). These issues worsen when the VAE employs autore- gressive structure, such for text videos (Fang et al., 2019; Zhao et al., 2018b; Dai et al., 2020; Long et al., 2019). Thus, mitigating posterior col- lapse in VAE architcctures is likely to have out- sized benefits in NLP.
1 Introduction
The variational autoencoder (Kingma and Welling, 2013)- henceforth referred to simply as the VAE - is a classical neural model that utilizes a paired encoder-decodes structure. For every data instance Xi. the encoder network in the VAE is responsi- ble for creating compressed code distribution P(z] (x) parametrically. The decoder network, in turn, uses this x(z] form an approximation of the real input, &; through an intermediate sam- pling step. By minimizing a reconstruction loss between x; and &,, along with a KL-ddivereence
Independent of the VAE model, there exists the idea of aligning the aggregate posterior. The aggre- gate posterior consists of the latent space distribu- tion, i.e. the distribution over z formed by evaluat- ing f P P(x;) P(z((>)) over all x;, where the distri- bution over x; is usually computed by an empirical average over the training set. In these methods. the individual distributions, P(z|(x; may even be Dirac distributions, i.c. the mapping between z; and x; is purely deterministic. We can still find

---

in aggregate, distribution over z that is close an isotropic Gaussian. Methods in this vein utilize Wasserstein-based optimal transport (Tol- stikhin et al., 2017), maximum mean discrepancy (Kolouri et al., 2018), etc. for this purpose. These methods cannot truly be termed VAEs, as they are often deterministic, but they work similarly, Due to their deterministic nature and differing loss func- tions, posterior collapse does usually oceur. In VAEs, the quantity be maximized is the log like- lihood log P(x;). This proves intractable, and an equivalent optimization is done via the ELBO (Ev- idence Lower BOund) objective. While VAEs can be evaluated by log likelihood, another metric is to evaluate their reconstruction error, as well as the quality of samples generated when, in place of P(z). an isotropic Gaussian N (O,, is substituted and the Gaussian samples fed through the decoder network. This notion of sample-basce evaluation can also be done for deterministic autoencoders, which do not allow an ELBO evaluation. This al- lows us to compare deterministic autoencoders to variational options fairly to determine superiority.
Instance-wise Gaussian Noise
VAE PROPAGATION
VAE LATENT
Original Encoder (t)
Hidden layers
CEAE PROPAGATION
x
CONTRASTIVE SEMI-S SEMI-SUEERVISED LOGS
CEAE LATENT
Hidden Layers
Parameter averaged Encoder E' d'=mi E- (1-m)*E
Figure 1: CEAE (Contrastive Entropic Autoencoder) architecture replacing instance-specific noise with con- trastive loss. Constant noise (not shown) is annealed over training to make the latent ZCEAE deterministic.
1.1 Our Contributions
We seek to draw parallels to findings in image datasets that indicate deterministic autoencoding models outperform variational ones in terms of sample quality (Ghosh et al., 2019). Due to their relative freedom from posterior collapse, and the aforementioned outsized impact of posterior col- lapse in language modeling, adapting these deter- ministic architectures to NLP may improve autoen- coders. We find that replacing previous highly performing architecture - the BN-VAE (Zhu et al., 2020) with similar deterministic variant improves the performance in language modeling tasks. We engage in information theoretic analysis of the constant variance VAE, demonstrating that this case allows mutual information maximization. To aid convergence, we add an entropic regularization through contrastive learning (He et al., 2020). To our knowledge, this linkage of contrastive learn- ing to entropic maximization is novel, though en- tropic regularization has previously been utilized for deterministic autoencoders (Ghose et al., 2020). We evaluate using perplexity-basee benchmarks in both forward and reverse directions (Zhao al..
observe improvements not only over the previous BN-VAE, but over broad array of VAE architec- tures. We term architecture CEAE (Contrastive Entropic Autoencoder) figure To account for the increasing relevance of large language models, we also test against appropriate archi- tectures that utilize BERT and GPT-2 as parts of an overall VAE architecture.
Definitions and Prior Work
2.1 Variational Autoencoder
We present here first the classical set up of the VAE. We denote D, & as the decoder, encoder halves of the VAE. Generally, D, E are comparable in size, shape, architecture, etc. We propagate every data instance xạ through subnetworks E
#4 Eμ (xi), o3 E {x²
(1)
These parameters, in turn, define Gaussian distribution, from which zj is drawn.
2018 Sa) and test the accuracy of our formulation us- ing relatively large autoencoders built using trans- former encoder-decoder pairs. all cases, we
2; ~ NN4,,
(2)

---

The decoder is then used to produce (approxi- mate) reconstruction & of x;:
2.3 VAE Optimization for Language Modeling The failure mode in VAE optimization (when mod- eling text, images or general datasets) manifests itself by posterior collapse, where (0, T) takes the place of every latent code's distribution. Au- toregressive VAEs suffer the worst in this matter, which impacts NLP disproportionately as non- autoregressive models are generally not usable. It has been suggested that a primary cause the col- lapse is due to training dynamics (He et al., 2019). In this view, the inference network (encoder) is in terms of training initially at very poor perfor- mance, which causes the generator network (de- coder) to neglect the latent codes generated by the encoder. This in turn leads to uninformative codes (the posterior collapse phenomenon). Alternative fixes include looking at weighing of the KL.-term. Such methods include the 3-VAE which adds a weight of 3 to the KL term (Higgins et al., 2016), and methods which do not allow the KL to go to zero (Razavi et al., 2019). Architecture-wise, skip- connections reduce posterior collapse (Dieng al., 2019), does reducing the complexity of the de- coder network. During the main training loop, the loss can be amortized (Kim et al., 2018), annealed, or applied in a cyclic manner (Fu et al., 2019), all of which reduce the phenomenon. Finally, the op- timizer itself may be changed, with SGD being somewhat preferable to ADAM (Srivastava and Sutton, 2017) for the purposes of avoiding poste- rior collapse. Orthogonal to all these fixes, may of course simply use deterministic autoencoders.
&; - (zi)
The loss function for VAE optimization is:
2 b118; x-1 (L=N((); I))
(4)
where the || sign between the two normal distribu- tions denotes the Kl.-divergence. Note also that the first term (squared loss) is commonly used in pa- pers that deseribe VAEs, but it should be replaced by cross entropy in classification tasks (including most NLP tasks). Furthermore, the loss function may not be the same for all variants of the VAE. For instance, the V-V (Higgins et al., 2016) per- forms the optimization based on
5 || x112 3KL (.. D)) (5)
3 turns into a tunable hyperparameter, and is generally not 1. Such tuning may improve perfor- mance in particular avoiding posterior collapse.
2.2 Deterministic Autoencoders
Utilizing the same notation the VAE, if we in- stead consider replacing the sampling step
2i Nr,, of)
(6)
with Mig the resulting flow is completely deterministic. In this case, training minimizes:
2.4 BN-VAE Architecture Strong Baseline
! 1| - x1| R(zi)
Let Paj. Oi denote the j-th index of the posterior parameters of the i-th instance of a latent code of dimension K. It may be shown that the expectation of the KL divergence obeys the relation (Zhu al., 2020):
where R can be any regularizer on with the sim- plest form being least-squares regularizer, i.e. R(z) = | forming Regularized autoen- coder (RAE) (Ghosh et al., 2019). To generate samples from the RAE, a Gaussian, or Gaussian mixture model, is to the distribution of after training. With a suitable regularizer R, such as the MMD regularizer forming the Wasserstein autoen- coder (WAE) (Tolstikhin et al., 2017), guarantees can be made on the latent space distribution Pzz) that do not require this post hoc fitting and allow drawing samples from (0, I) directly- this case is our focus of interest. For image datasets such as MNIST, CIFAR-10, 10, and CelebA, deterministic au- tuencoders have a notable advantage (Ghosh et al., 2019) and we would like to carry this over to text.
K
1
E[KL] 2 Eluij
Σ
Elog]]
Ellog of]
(8)
where the expectation is taken over the samples i.c., over i. We directly use the resulting inequality:
E
E
Eloog - -1 0 e" 1. We know that for any random variable that
Because Elo

---

has defined first and second moment, B[X2] V Var(X) (E We enforee batch norm con- straint on jij. that fixes the expectation and/or the variance, thereby setting the lower bound on the ex- pectation of KI.. Batch normalization here simply refers to the near-ubiquitous practice of minibatch- level normalization, i.e., adding layer of the form:
H(p + 21p) - H(
Z is independent of μs. This differs from general VAEs, where the variance is instance dependent and ;; 14 related, relating Z, ps. Note that here we refer to the instance-inder i and not the dimension index j as we discuss instance-level de- pendence. This yields the final mutual information expression as:
(Nx):) = Xi μ
(9)
σ
where μe, represent the mean and standard de- viation computed over the minibatch of x;- This batch norm layer is usually accompanied by an affine function, i.e.,& function of the form f(x) Ax + b (Ioffe and Szegedy, 2015). We will ex- plicitly make a distinction between the two parts. Over experiments against other architectures such as Skip-VAE and others (all of which are purported designs to circumvent the posterior collapse issue) BN-VAE results in superior performance in both language modeling (NLL metrics) and the down- stream usage of learnt representations (Zhu al., 2020). Our lesson from this success is set baseline with a VAE which batch normalizes the means of latent representations.
H(z) - HZ)
Now, (z) has a fixed value of Elz] and also a fixed variance, since it is the direct sum of two ran- dom variables Z and both of which, by hypoth- esis, have fixed means and variances. Under this condition of fixed mean and variance, it is known that the entropy H is maximized iff is distributed as a Gaussian (Thomas and Cover, 1999). There- fore, a more informative constant variance batch normed VAE induces a more Gaussian represen- tation on z. Since μ = z - Z. and z approaches a Gaussian, while Z itself one, the desired latent mean μ also approaches a Gaussian as mutual in- formation rises. Note that our analysis holds for any e 0. This means even very low values of c- slowly annealed to approximating the determin- istic autoencoder case will work as long as the mutual information required is high. To create an aggregate posterior which is the isotropic Gaussian, we assume a = 0, b = 1, i.e.
3 Information Theory of Constant Variance Batch Normed Autoencoders
Let the generation process of the latent code be:
Zij N (pijs ofy
With the batch norm constraint that
[[m}] 0, Varlu4] [p4] =1
When μ becomes a Gaussian with the above two constraints, our job of creating an appropri- ate deterministic autoencoder with the right ag- gregate posterior is done. We have already dis- cussed the interaction of mutual information with that process, but now, observe that becoming Gaussian can also be done by controlling the en- tropy H, which is maximized iff μr is Gaussian, which also implies 2; is Gaussian. The process accelerates when minimizing a regularizer the form of ( O.
E[usy] = Var[uy)] = bb20 0
Consider intermediate case between the VAE (where each oy can be distinct) and the determin- istic autoencoder (each o, = 0). Set every oj = c. e > 0 constant. Then, indexwise and vectorwise.
zij = M4j c, #4 Z
where Z is a random vector from (O eI). Consider the mutual information between z, μ. This denotes the amount of useful information sent through the encoding and maximizing it avoids posterior collapse (Zhao et al., 2017). It cquals
3.1 Entropy and Contrastive Regularization
We require an effective entropic estimator for (u4) make Gaussian. A first step may be repulsion-based estimators such as the Kozachenko-Lconene estimator : (Delattre and Fournier, 2017) for a sample consisting of
((z) ((((u)
Where H is the entropy function. Now, z (n + 2),. Therefore, the required entropy is of

---

X2....... XNII 1 E Rd drawn from an unknown distribution P. assuming N 1, With R² minjz: || - X,l12. Y, = (( By the vol- ume of the unit ball in IR", ? the MascheMas constant z 0.577, an estimate of the entropy of the distribution is:
loss, this method is well understood empirically in terms of training and implementation. Further,
can be shown that this loss approximates the entropic regularizer for Gaussian distributions in high dimensions. The full proof and derivation appears in the appendix. We use the following loss function Cent, with pi, M4 being respectively gen- erated by Eg, Egr, where 0' = mll + (1 m)0 is time averaged version of the main model Eg.
N+1 log Y, + log Ba +
H(P) ~ N+1
Σ
exp(
The estimator relies primarily on the leading sum over Y,, which computes "repulsions" between la- tent representations which are too close. Only this sum (with weight A) needs to be computed at all for gradient based learning, as the other terms are constants. In practice, this estimator has been used only sporadically for image autoencoders (Ghose et al., 2020) and rarely in general for neural net- works, and direct implementations of the method for language autoencoders leads to convergence failure.
Cent -x X log
Σ Z expl (1))
2EEX
The details of how to choose m, d and their justifications also appear in the appendix. The over- all training loss is formed by adding Lent to the reconstruction loss Cree = =
xil1?
4 Experimental Details and Methodology
4.1 Dataset Choices
We present results primarily using the Yahoo and Yelp corpora (Yang et al., 2017) for language mod- eling tasks using LSTM encoder-decoder architec- ture autoencoders. This maintains consistency with the BN-VAE (Zhu et al., 2020) in terms of compar- ing performance for these tasks. We additionally use a small-scale transformer, with its structure based on the Transformer-Xi model (Dai et al., 2019) specifically for Penn Tree Bank(PTB) dataset (Mareus et al., 1993), for results this dataset (in appendix). Unlike transformer-XL, which is decoder-only model, we employ an encoder- decoder transformer, but keep hyperparameter and training recommendations aligned with the original source code for the PTB Transformer-XL. details. We name our architecture CEAE Contrastive En- tropic Autoencoder.
We turn to the contrastive learning literature to look for a solution, which has recently emerged as strong baseline in representation learning for both unsupervised (Chen et al., 2020; Zhang et al., 2022) and supervised (Khosla et al., 2020; Zhang et al., 2022) contexts. In unsupervised contrastive learning, it is desired to learn two alternative rep- resentations Z₁ Z of some instance X; E xK, so that Zi, Z are close (e.g. by the inner product) and Z, arising from X, E X.j + has a low inner product. This is done by minimizing:
exp( 2,+)) E Ex exp(( (Z;, Zz))
- E log X.Et
7/' arises from noisy or augmented version of X5, such crop (if X is an image). One suitable method is momentum contrast (MoCo) (He al., 2020), where Z: is generated by a model with pa- rameters 8, and a model with A', a time average of 0. generates Z,+_ Therefore, the encoding method learns to be insensitive to these noises and learns encodings Z, z,+ that are more or less invariant to such. Simultancously, the denominator discour- ages proximity between codes Zi, Zj arising from different instances X,, X, a repulsive regulariza- tion controlling the entropy which sketches our derivation. Unlike directly controlling the en- tropy with a Kozachenko-Leennenke repulsion
4.2 Metries
Generally, variational autoencoders are compared on the following metrics:
Negative log logllikeldhood of the data (usually estimated via ELBO/importance sampling)
Mutual information between x;, 2,, capturing latent code quality(Alemi et al., 2016)
KL. divergence (averaged) between each latent code's distribution and the isotropic Gaussian, i.e. (W (O)

---

Other metrics such as active units (AU) (Burda al., 2015) may be used, which capture the num- ber of latent dimensions which are actually being utilized in the autoencoder. None of the above mentioned metrics (except the AU) can be used to compare VAEs with deterministic autoencoders. We hence use different suite of metrics based on forward and reverse perplexities (Zhao et al., 201 Sa; Gagnon-Marchand et al., 2019):
Von-mises fischer (vMF) VAB (Xu and Dur- rett, 2018)
Semi-amortized (SA) VAE (Kim et al., 2018)
Aggressive- VAE (He et al., 2019)
These benchmarks correspond to the ones (Zhu et al., 2020), from which we also find the required reconstruction metrics and BN-VAE vari- ants. We add a vanilla VAE and LSTM language model for baselines. For the PTB dataset, we com- pare the deterministic autoencoder to a VAE setup optimized analogously to the much larger BERT- GPT-2 VAE in (Li et al., 2020). Only the standard VAE considered, with KL annealing kept for con- sistency. Standard deviations, implementation and architectural details, hyperparameters etc. are in the appendix.
Reconstruction error, measured by negative log likelihood of reconder-ecconstruueed input.
Forward perplexity, where we generate and decode 1000 samples from (N I). The per- plexity of this sample evaluated by a critic model, optionally after training on the corre- sponding train segment of the corpus.
Reverse perplexity, in which 10, 000 samples are generated from the autoencoder just as above, but now used for training different model, then tested the test segment of the corresponding corpus.
4.4 Large Language Model Comparison
We also add comparison to OPTIMUS architec- ture of VAEs (Li et al., 2020) with BERT as the encoder, GPT-2 as the decoder, and pre-training on the Wikipedia dataset (details in appendix). Fur- ther, the embeddings learnt by our method were compared to BERT zero-shot embeddings per sentence (this result appears in the appendix).
We chose two different critic models to reflect two different ends of the model power spectrum: simple LSTM model for language modeling, and GPT-2 (Radford et al., 2019) (standard configura- tion, 117M parameters). The reverse-perplexity task was performed only with the LSTM critic, as it was found that training GPT-2 on the (relatively low quality compared to real text) samples hurt downstream performance on the uncorrupted test corpus. We add human evaluation of generated samples as a sanity check.
Results
In terms of text modeling, results appear in Ta- bles and 2. In general, our method outperforms the competitors and if not is close the top per- former. should noted that realistically, the LSTM critic's performance relative to GPT-2 is due to the fact the LSTM trained on the rele- vant corpus while GPT-2 is tested zero-shot. Even though GPT-2 is stronger model, the LSTM has more domain knowledge, causing their perplexities to be close. This implies that GPT-2 evaluates the samples based general knowledge of the English language and plausibility as English sentences (as is tested zero-shot) while the LSTT evaluates it with emphasis on the domain knowledge (which is the sole train data). Having both perplexities thus evaluates differently, and we perform well on both. To evaluate the quality of the latent space for downstream tasks, extract the latent representa- tions in shortened Yelp dataset following (Shen et al., 2017), along with the labels for small frac- tion of the dataset. These labels reflect the nature of the shortened review (positive or negative). We
4.3 Comparisons and Benchmarking
We include a full suite comparisons for the Ya- hoo and Yelp corpora, with the following architec- tures targeting the posterior collapse problem:
Skip-VAE: VAE: latent-decoder skip connection (Dieng et al., 2019)
Annealing the KL loss (Bowman et al., 2015)
3-VAE (KL weight 3) (Higgins et al., 2016)
Cyclic annealing of KL loss (Fu al., 2019)
Free bits in the KL loss (Kingma et al., 2016)
8- VAE (minimum KL 8) (Razavi et al., 2019)

---

Yahoo
Yelp
Model
Rec
GPT2-F
Rec
GPT2-F
L-F
L-R
LSTM-LMM
328.0
136.7
171.5
202.3
351.1
125.2
95.1
132.0
VAE
328.6
118.2
141.7
175.8
357.9
100.1
87.8
115.5
3-VAE (0.4)
322.4
126.4
136.3
173.9
354.0
97.4
91.8
113.6
cyclic
328.5
125.4
131.8
182.6
357.5
100.6
87.8
112.8
VAE
326.2
130.3
138.4
175.9
355.7
95.2
91.7
109.6
SA-VAE
322.0
124.6
139.2
177.5
353.1
96.6
86.3
119.4
Agg-Vis VAE
321.0
118.9
145.7
178.6
352.1
97.5
85.0
121.7
FB (4)
326.9
129.5
134.4
172.7
355.2
92.1
82.6
109.8
FB (5)
324.9
128.2
131.5
170.2
354,9
98.8
87.2
109.0
8-VAE (0.1)
327.5
122.0
130.7
171.8
356.6
95.5
84.2
109.4
MF-VVA (13)
325.4
117.6
134.8
168.2
355.5
105.2
85.3
115.0
BN-VAE (0.6)
320.5
110.6
128.6
163.4
350.5
92.2
78.2
112.7
(0.7)
318.6
109.0
124.5
168.9
346.8
90.5
80.5
109.2
CEAE 316.7
107.6
119.8
163.1
342.0
88.1
78.0*
110.2
Table Language modeling. Reconstruction log loss (Rec), Forward perplexity with GPT-2 (GPT2-F), Forward perplexity with LSTM (L-F), Reverse perplexity with LSTM (L-R) in that order Yahoo, Yelp. Reverse perplexity with GPT-2 was not meaningful as the fine-tuning is without effect due to pre-existing large corpus in the pretraining phase. Statistical analysis standard deviations and confidence intervals etc. appears in appendix. * indicates statistically insignificant best method.
Yahoo
Yelp
Small-scale transtormer results
Model
Rec
G2-F
L-F
L-R
Rec
G2-F
L-F
L-R
VAE
304.6
121.4
130.5
328.2
77.8
95.2
8-VAE (0.4)
305.2
120.2
330.2
72.4
71.9
102.8
eyclic
304.6
85.6
91.8
103.5
332.7
75.6
70.4
97.5
FB (7)
303.2
79.6
94.9
105.1
329.4
74.6
68.5
98.1
BN-VAE (0.7)
301.7
76.9
102.0
112.9
328.5
65.7
70.2
96.8
CEAE
300.2
72.5
89.2
101.6
326.1
69.7
70.9
92.1
OPTIMUS (BERT encoder, GPT-2 decoder) results
Model
Rec
G2-F
L-F
L-R
Rec
G2-F
L-F
L-R
OPTIMUS
282.8
43.8
80.5
96.1
334.3
47.0
61.8
83.7
BN-VAE (0.7)
285.8
45.5
85.4
99.2
330.5
45.8
60.4
82.5
CRAE 278.7
42.4
79.4
91.8
328.7
45.0*
57.2
80.6
Table 2: Performance of transformer autoencoders on Yahoo and Yelp, evaluated using the same metrics.
then apply simple shallow neural classifier to get the labels (review valence). Note that the task of representation learning always has access the same number of unlabeled sentences, only the num- ber labels is varied. Our method proves superior especially as more labels become available with all results in Table 3. For human evaluation, we gathered five individuals (graduate students ma- chine learning engineers) who were provided 200 choice-based questions and asked pick the most coherently generated choice among permuted op- tions. Each choice corresponded a method on corpus. We compare to the BN-VAE and vanilla VAE and find a significant advantage in terms of being chosen across both LSTM and transformer architectures, as shown in Tables 4 and 5. A follow-up with GPT-4 appears in the ap- pendix. Overall, across broad variety of tasks, we improve on the BN-VE -NE architecture.
#label
100
500
1k
2k
10k
AE
78.6
87.1
91.0
92.4
94.7
VAE
56.5
86.7
91.4
91.1
95.1
6-VAE
57.6
73.4
90.8
91.0
94.5
Agg- VAB
72.3
82.4
90.2
94.2
cyclic
65.4
88.5
93.7
FB (9)
87.2
90.6
95.2
AE+FB (6)
82.4
91.0
93.1
95.3
BN-VAE (0.6)
92.4
93.0
95.3
CEAE
85.6
92.7
93.5
94.6
95.5
Table 3: Accuracy Yelp downstream task perfor- mance, with small MI.P trained on labeled samples of fixed number. indicates statistical insignificance.
sample from
Candidate Yahoo Yelp
CEAE
BN-VAE
VAE
46.2
36.2
17.6
30.4%
13.5
Table 4: Frequency of choice among generated samples among LSTM models.

---

Candidate
CEAF
BN-VAE
VAE
that often, decoder-only models are used for text generation, whereas encoder-only models are used for representation learning, using both simultane- ously creates a VAE-like architecture. This anal- ogy has been used to train massive VAEs for NLP e.g. OPTIMUS (Li et al., 2020) that employ pre- trained encoders. We view our work as an indi- cation that deterministic autoencoding, unlike tra- ditional VAEs, design better autoencoders for text. Issues of VAE training, namely posterior col- lapse, worsen with increasing model power (Yang et al., 2017; Semeniuta et al., 2017; Razavi et al., 2019). Powerful VAE design requires tackling this and deterministic models may offer the solution. We also consider our work to be of significance to the field in its successful usage of contrastive learn- ing for NLP, which for text often suffers from less clear augmentations (Rethmeier and Augenstein, 2021) relative to images. We focus on BatchNorm, however, natural language tasks, increasing em- phasis is being laid on layer normalization aka Lay- erNorm (Ba et al., 2016; Xu et al., 2019) which forms a key part of Transformers. We discuss the Layer norm case further in the appendix.
Yahoo
49.8
30.5
19.7
Yelp
43.6
33.6 %
22.8
Table 5: Frequency of choice among generated samples among transformer models.
6 Qualitative and Quantitative Analysis of Interpolations
Autoencoders allow smooth linear interpolation be- tween two sentences in the latent space in a manner that should allow both syntactic (sentence struc- ture) and semantic meaning. This is captured using the subset of the short Yelp review dataset, which consists of small single sentence reviews of estab- lishments with either 0 or 1 for a negative or posi- tive review, and used in the classification task in Table We perform the following sanity checks:
That interpolating between a positive and a negative review yields a neutral review.
That interpolating between two reviews of the same nature poositive-positive or negative- negative) always yields reviews of the same nature, but of differing content or sentence structure, reflecting the source sentences.
8 Limitations
Our methodology focuses autoencoders which may include transformer architectures, however, by necessity, autoencoders involve an encoder-decoder pairing which may be absent in some architectures which may be encoder-only (for word embeddings) or decoder-only (for language generation). In these cases, our approach is not scalable and will require some rethinking. Further, text generation is field in a state of flux. Although we have tested with large models in the form of GPT-2, it is possible our results do not scale to as-of yet unreleased but extremely potent models, such as GPT-4.
That these interpolations have numerical scores (from the classifier of Table 3) that match the decoded content.
Results demonstrating these qualitative charac- teristics are summarized in Table 6. Moving be- tween reviews of different kinds changes the score as expected one which is ambivalent i.e. around 0.5, which reflects in the text as well such as in the example on row 3 of the positive and negative inter- polation. Between reviews of the same nature (clus- tered around 0 or 1) interpolation causes changes in sentence structure and content in the case of two negative reviews, the interpolation closer to the sen- tence "are you kidding me?" begins with "are you", and the interpolation involving two positive reviews also associates common sentence structures.
Ethics Statement
Text generation may include harmful content, un- desirable generation and algorithmic bias. do not, however, view our work as being particularly prone to these failure modes any more than other papers in this domain, and we believe no particu- larly strong ethical statement is warranted.
7 Conclusions and Future Work
Attention based transformer models (Vaswani et al., 2017) have achieved great success in all areas of
NLP (Devlin al., 2018: Radford et al., 2018, 2019). Transformer models retain an autoencoder- like parallel with encoder and decoder modules. Though they are dissimilar to VAEs in the sense
References
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. 2016. Deep variational information bottleneck, arXiv preprint 1rXiv:1612.00417

---

Interpolation between two negative reviews
Type
Text
Score
Source
was really disappointed.
0.22
Interpolation
was really disappointed with the beans. are you kidding with the salt? are you kidding me ?
0,20
Interpolation
0.15
Source
0.17
Interpolation between two positive reviews
Type
Text
Score
Source
their ranch sauce is the best i ever had.
0.91
Interpolation
their pizza is the best in madison.
0.88
Interpolation
also love the tasty steak
0.82
Source
i also love the crunchy tacos and chicken enchiladas Interpolation between a positive and negative review
0.80
Type
Text
Score
Source
the pollo especial is excellent but my favorite is the fiesta platter.
0.86
Interpolation
the coffee is solid, and good first impression.
0.78
Interpolation
don't 't get their burger and there is zero energy.
0,44
Source
don't get how they use better ingredients and it tastes terrible.
0.29
Table 6: Interpolation between different reviews using LSTM models, with weights (0.8, and (0.2, 0.8) respectively for the first/second interpolations relative to source sentences. There is change of estimated review score calculated by the classifier used to compute the accuracy in Table 3 matching the text. Sentences tagged with "Source" are original sentences retrieved as-is from the corpus.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hin- ton. 2016. Layer normalization. arXiv preprint arXiv:1607.064500
Shuyang Dai, Zhe Gan, Yu Cheng, Chenyang Tao, Lawrence Carin, and Jingjing Liu. 2020. Apo-vae: Text generation in hyperbolic space. arXiv preprint arXiv:2005.00m
Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. 2005. Clustering on the unit hypersphere using mises-fisher dis- tributions. Journal of Machine Learning Research, 6(9).
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Car- bonell, Quoe V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language mod- cls beyond a fixed-length context. arXiv preprint
Etienne Barnard, 2010. Maximum leave-one-out like- lihood for kernel density estimation. Pattern Recog- nition Association of South Africa and Mechatron- ies
Sylvain Delattre and Nicolas Fournier. 2017. On the kozachenko-leennnkko entropy estimator. Journal of Statistical Planning and Inference, 185:69-93.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, An- drew M Dai, Rafal Jozefowicz, and Samy Bengio. 2015. Generating sentences from continuous space. arXiv preprint 1.1.66499
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training deep bidirectional transformers for language understand- ing, arXiv preprint arXiv: 181.0080.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, An- drew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from continuous space. 20th SIGNLL Conference on Computational Natu- ral Language Learning, CoNLL 2016, pages 10-21. Association for Computational Linguistics (ACL).
Adji Dieng, Yoon Kim, Alexander M Rush, and David Blei. 2019. Avoiding latent variable col- lapse with generative skip models. In The 22nd In- lernational Conference Artificial Intelligence and Statistics, pages 2397-2403 PMLR.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2015. Importance weighted autoencoders. arXiv preprint arXiv: 1509.0051.
Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. 2019. Implicit deep latent vari- able models for text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)) pages 3946-3956.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020. Improved baselines with momentum con- trastive learning. arXiv preprint XXiv:2003.004999

---

Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao. Asli Celikyilmaz, and Lawrence Carin. 2019. Cycli- cal annealing schedule: A simple approach to mitigat- ing kl vamishing. arXiv preprint arXiv: 1119999015555.
Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde. 2018. weeeddwwsserstein autoen- coder: An embarrassingly simple generative model. arXiv preprint 100.v:1804.01444
Jules Toggon-Marchaan Hamed Sadeghi. Md Akmal Haidar, and Mehdi Rezagholizadeh. 2019. Salsa- text: self attentive latent space based adversarial text generation. In Canadian Conference on Artificial Intelligence, pages 119 131. Springer.
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng. Xiujun
Li, Yizhe Zhang, and Jianfeng Gao. 2020. Optimus: Organizing sentences via vre-trained modeling of latent space. arXiv preprint arXiv:2004.04092
Amur Ghose, Abdullah Rashwan, and Pascal Poupart. 2020. Batch norm with entropic regularization turns deterministic autoencoders into generative models. In Conference on Uncertainty in Artificial Intelli- gence, pages 1079-1088. PMLR.
Teng Long, Yanshuai Cao, and Jackie Chi Kit Che- ung. 2019. On posterior collapse and encoder fea- ture dispersion in sequence vaes. arXiv preprint arXiv:1911.03976
James Lucas, George Tucker, Roger B Grosse, and Mo- hammad Norouzi. 2019. Don't blame the elbo! lin- ear vae perspective on posterior collapse. Advances in Neural Information Processing Systems, 32:9408- 9418.
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. 2019. From variational to deterministic autoencoders. arXiv prepriat arxiv: 1903. 10003.1240 12436.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg- Kirkpatrick. 2019. Lagging inference networks and posterior collapse in variational autoen- coders. arXiv preprint 1xXiv:1901.00
Mitchell Marcus, Beatrice Santorini, and Mary Ann Mareinkiewicz, 1993. Building a large annotated corpus of english: The penn treebank.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for un- supervised visual representation learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):9.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess. Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2016. beta-vae: Learning basic visual concepts with constrained variational framework.
Ali Razavi, Aäron den Oord, Ben Poole, and Oriol Vinyals. 2019. Preventing posterior collapse with delta-vaes. arXiv preprint arXiv:1901 1901.03416.
Sergey loffe and Christian Szegedy. 2015. Batch malization: Accelerating deep network training by re- ducing internal covariate shift. In International con- ference machine learning, pages 448 456. PMLR.
Nils Rethmeier and Isabelle Augenstein. 2021. A primer contrastive pretraining in language processing: Methods, lessons learned and perspectives. arXiv preprint urXiv:2102.1 12982.
S Sathiya Keerthi and Chih-Jen Lin. 2003. Asymptotic behaviors of support vector machines with gaussian kernel. Neural computation, 1567).1667-168.
Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. 2017. A hybrid convolutional variational autoencoder for text generation. arXiv preprint 172iv:170..022300
Prannay Khosla, Piotr Teterwak. Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Su- pervised contrastive learning. Advances in Neural Information Processing Systems, 136116611.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non- parallel text by cross moss-alignment. arXiv preprint arXiv:1705.09665:
Yoon Kim, Sam Wiseman, Andrew Miller, David Son- tag, and Alexander Rush. 2018. Semi-amortized vari- ational autoencoders. In International Conference on Machine Learning, pages 2678-2687. PMIR.
Akash Srivastava and Charles Sutton. 2017. Autoen- coding variational inference for topic models. arXiv preprint arXiv: (103::::77300485
Diederik P Kingma and Max Welling. 2013. Auto- encoding variational bayes. arXiv preprint arXiv:1312.6114
Joy A Thomas and Thomas M Cover. 1999, Elements of information theory. John Wiley & Sons.
Durk P Kingma, Tim Salimans, Rafal Jozefowiez, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Im- proved variational inference with inverse autoregres- sive flow, Advances in neural information proceetsiop processing systems, 29:4743-4751
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. 2017. Wasserstein auto- encoders. arXiv preprint arXiv:111..0155..

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 5998-6008.
Roman Vershynin. 2018. High-dimensional probubility: An introduction with applications in data science, volume 47. Cambridge university press.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference on Machine Learning, pages 9929-9939, PMI.R.
Jiacheng Xu and Greg Durrett. 2018. Spherical latent spaces for stable variational autoencoders. arXiv prepriat arXiv: 1808. 10805.
Jingjing Xu, Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. 2019. Understanding and improving layer normalization. arXiv preprint arXiv:1911.07013.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg- Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated con- volutions. International conference on machine learning, pages 3881-3890 PMLR.
Rui Zhang, Yangfeng Ji, Yue Zhang, and Rebecca J Passonneau. 2022. Contrastive data and learning for natural language processing. the 2022 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies: Tutorial Abstracts, pages 49.
Proceedings of
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. 2018a. Adversarially regularized autoencoders. In International conference on ma- chine learning, pages 5902-5911, PMLR.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. 2017. Infovae: Information maximizing variational autoen- coders. arXiv preprint arXiv:1706.02262
Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2018b. Unsupervised discrete sentence representa- tion learning for interpretable neural dialog genera- tion. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume Long Papers), pages 1098-1107.
Qile Zhu, Jianlin Su, Wei Bi, Xiaojiang Liu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. 2020. A batch normalized inference network keeps the kl vanishing away. arXiv preprint MXXv:2004.12558

---

Appendix: contents
Lent taking over since the need to raise mutual information arises from the high values of ct. we utilize an epoch dependent regularization on H (14) minimizing the contrastive loss, unifying notation from X; to 24
In order, we go over:
The training process and hyperparameters and how to choose them
Proof of how MoCo begins to approximate the entropy regularizer
exp( 1,,1) x,ER Σ Eit expl (pp((Nis 11))
Lent - x Σ log
Additional results penn tree bank
with A. - (1 11-11/ A/ and Hs. H4 being respec- tively generated by Eg, Egr, where 0' = m0 + (1 m)0 is a time averaged version of the main model Eg. m 1 and we m 0.999 make 0' update slowly in keeping with standard implemen- tations (He al., 2020). The overall training loss is formed by adding Lent to the reconstruction loss Free = } || x:112 Due to Lent in the second and e in the first stage of training, there is no need to fit a post hoc distribution (Ghosh et al., 2019) and the resultant posterior is Gaussian. Two-stage training is necessary as the contrastive loss only be- gins to approximate the entropy under assumptions reached later in the training process.
GPT-4 validation of human evaluators
BERT validation of embeddings
Layer norm discussion
Confidence intervals and standard deviations of results
A Details of Training Process and Hyperparameters
A.1 Training Flow
From the discussion section 3.1, is clear that we will achieve our goal of an aggregate Gaussian- distributed, deterministic μ4 by either driving up its entropy or requiring high mutual information. Let the 0; The overall maximum mutual information equals:
A.2 Architectural Details
For the LSTM-based VAE for Yahoo and Yelp, architectural details were tuned to match the BN- VAE, with latent dimension of 512, and single layer LSTMs of size 1024 for both decoder and encoder, with corresponding feeding patterns of the latent code following exactly the BN-VAE implementa- tion. Minibatches of size 32 with SGD and gradient clipping were used for training, along with anneal- ing on either the KL or the entropy loss following original BN-VAE implementation
(z)-H(C
By applying the Gaussian entropy formula which states that for variance V, the entropy of a Gaussian random variable is 1/ log(e we obtain that the mutual information per latent dimension equals
For training critic models in the forward and re- verse perplexity tasks, a simple recurrent LSTM with two hidden layers of size 200 was used and trained also by SGD, While the LSTM was trained the text corpus before evaluation on the sam- ple for forward perplexity, GPT-2 was tested zero- shot. Due to the structure of GPT-2, the sample is also evaluated for coherence over all generated sentences, even though each sentence is generated individually. However, we expect differences be- tween our models to respect only the individual quality of generated sentences, all of them have this independent generation framework. For the Penn Tree Bank models using transformer model based Transformer-XL, we built on source code hhtne//ggihuuu.omm/vllleeeaabbae
1 log (T e)) log log(2ie((c2)
1
2
log(1 + 1/22)
The upper bound is reached, and the Gaussian distribution formed only when c is high, and the expression is bounded above by low value which is reached to lower the reconstruction error. We set for epoch t c co/t, / < fmaax C. = Ovt > tmar tmar is the point where deterministic train- ing begins, i.e. e 0. Depending on the dataset, the value of co is can be chosen with justifica- tions and our process of choosing it appears in appendix A.4. As e goes to zero, the final epochs are trained deterministically. with the entropic loss

---

released from the author's website. The decoder- only model was modified (keeping hyperparame- ters and latent sizes) to an encoder-decoder model. Architectural details were kept as-is, with 6 lay- ers for both encoding and decoding and using a sentence level representation based on the start of sentence token.
ularizer's weight to 10-3. This hyperpa- rameter was based on trying all combinations of {2, 3, 6, 7 x 10(-3, i.e among choices. All entropic hyperparameters with 10-3 "order obtained comparable results to the ones reported in the pa- per, and with 10 still outperformed BN-VAE. In general. all architectural details, parameters, hy- perparameters that do not differ between BN-VAE and our method follow exactly for LSTMs. For transformers, the same KL weight, r, K, m was kept as for LSTMs. However the other parame- ters were shifted to match the implementation a small scale transformer for Penn Tree Bank as in Transformer-XL with the exception of chang- ing the decoder-only model to model using both transformer encoder and decoder layers to more closely match the autoencoder framework.
A.3 Randomness and Seed Dependence, Hardware, Number of Runs
We do not consider our method to depend on randomness i.c. the seed meaningfully. For the BN-VAE architectures we ran with both the pre- provided fixed seed 783435 for the BN-VAE repos- itory and also without setting any seed averaged over 10 runs. The best result among the two was re- ported (except the reconstruction loss for which we directly use the previous figures reported we could not actually replicate those numbers in re-running them, however, we got within 0.1 so we consider them correct in the original BN-VAE paper). For and the average over 10 runs was reported directly. Results were reported with a Tesla V100 32 GB, using Pytorch 1.6, on Ubuntu 18.04.
Choice of co and associated parameters We recall from the main text that at epoch zero the initial channel capacity (maximum mutual infor- mation) of the autoencoder with P latent units is
our methods, fixed seed
P 2 log(1 1//
For perfect decoding of even the train set consist- ing of sentences with lengths n....., R,M, we have to consider all valid targets. For a sentence, this is any contiguous subsentence. For a length ni equals 2-(m- So we compute the total valid subsegments of the train set as
A.4 Hyperparameter Optimization and Notes on Transformer Training
general, we keep all hyperparameter and archi- tectural details in line with previous implementa- tions (linked in footnotes). As reported in the paper, the number of cached minibatches is r = 3, and ev- the global statistics
ery K 5 minibatches of the batchnorm. Further we try 0.99, 0.999 for the momentum update step and find better re- sults with 0.99 which leads the figures reported in the paper, however m = 0.999 also outperforms BN-VAE. The SGD trained with a gradient clip of 5.0 and initial learning rate of 0.5 (BN-VAE) and 0.1 (ours) decayed by factor of 2 (i.e. multi- plied with 0.5) at most 5 times with decay crite- rion based non-improvement validation for 5 epochs (same as BN- BAE).E). However, in our ex- periments, we find that the gradient clip is the only sensitive hyperparameter for both architectures.
S
By the channel capacity theorem, for perfect reconstruction we would need
(1 + 1/22)
However, this assumes perfect function approx- imator part of the decoder and encoder. As such, choose co support + 1/22) 5S.
A.5 OPTIMUS training
We followed the training procedure from the orig- inal paper (Li et al., 2020), taking the lowest À model unless there ues and a better reconstruction was available. Note that we also examined experimental results using different values. The following figures should be taken in context with respect to Table 2. For Yahoo,
The BN-VAE parameters that fix batch norm statistics (7) are set according to the original imple- mentation's 's best performances, at 0.6, 0.7, while for case the results reported in the paper cor- respond setting the contrastive entropic reg-
tie in the perplexity val-
>htp://zihangdai, hht.aaamnaai.iiituuiioooomohh/ii

---

both X 0.5, 1 do obtain better reconstruction ac- curacy (275.9, 270.8) respectively for OPTIMUS. However, this comes at the cost of significantly worse metrics for G2-F, L-F, and L-R (written as triplets): : (45.2, 84.7, 97.5) for À = 0.5 (i.e. sta- tistically comparable to BN-VAE and sigmificantly worse than CEAE) and (47.6, 88.2. 100.2) at X = 1 which is worse than the other two models. For Yelp, only À 1 is superior with reconstruc- tion of 325.8. However, this comes with G2-F, L-F, L-R triplet of (49.2, 63.1, 84.8) which again significantly worse than our CEAE result.
distinct encoders respectively both of which re- ceive X, or z; (abusing notation), one of which has parameters 0, and the other has parameters #' - mf + (1 - mm), updated with m ~ 1. When near convergence, we may assume that 14: R 14,", since 0' ~ 8. Let us now assume for the moment that || = constant. The above sum for the second case then becomes:
Σ log exp(c2) X,CX
Σx,ex exp( 14))
Since is constant, the log be taken out, and Σx,xx is just expectation over the dataset. We are left with minimizing
For adapting BN-VAE and CEAE, we note that the original training process used KL threshold- ing. This was kept as-ís. To create smoother train- ing, we train GPT-2 for epoch as fine-tune step as per OPTIMUS before beginning the main train- ing loop for all of our models.
E log Σ
exp(1p4 1())
X,CX
The term within the log, in expectation, is a ker- nel function (a valid kernel for probability density estimation and thus an estimator of the density). Specifically, it is the (unscaled) vMF (Von Mises Fisher) kernel (Banerjee et al., 2005), which uses the cosine distance on the hypersphere. We then have that
B Relation of MoCo to Entropy Approximation
Our analysis here follows previous theoretical anal- yses of contrastive learning (Wang and Isola, 2020). We also repeatedly use the property that high di- mensional isotropic Gaussians are clustered around a scaled hypersphere, that is, their L2 norm is tightly concentrated. For an exposition on these matters, we refer the reader to (Vershynin, 2018). We use the properties of isotropic Gaussians in high dimensions without further reference. We re- call that we use a MoCo loss of the form (with the understanding that μ4 arises from X; through an encoder Eg and correspondingly 14 is from an encoder of parameters 0':
Σ exp((us,1s)) = C(4 X5eX
Where C'd,, X()) is scaling function for the ker- nel density dependent on the number of elements and the dimension, and P(4,) is the kernel estimate of the probability. The expectation of the log then asymptotically converges to the entropy. Now let us examine the cases of
Σ log exp( (((miiiit))) X.EX'
Hg is not of constant norm
Ex,ex exp((1is
The sum does not include the positive term
Generally, this loss includes the positive pair in the denominator, i.e.
The convergence not asymptotic
Varying norm : In this case, we turn
Σ log exp((ussHtt) X,EX
exp(c?) Σ log Σx,ex exp( mj))
Ex,xx expexx (ppp p3))
X.cX
Note that in the general case there is also tem- perature hyperparameter 7 which scales the inner products, i.e. (pt, (y becomes (man))) and on. We ignore this for the sake of exposition and set т = 1, our methods will carry over to that case as well. Recall that ps. p.) for us arise from two
into
Σ log exp(1lux13
Σx,ex exp( (Mi, 1z))
Xiex
Which can be turned into minimizing:

---

(e Mg)
E llg
exp(
B + E log
exp(ies: Rg))
T
XyEN
Xyeæ
However, we recognize that the sum (after E log) of random variable which is the exponential of a Gaussian when (Ai; Pj are isotropic Gaussians. This is because (15,/5) ) is the scaled projection of all μj on fixed random vector μ4: This is a 1-dimensional gaussian, making the sum sum of log normal random variables. While this can be approximately expressed via methods such the Fenton- Wilkinson moment matching method, this far less clean than the case of low т.
But by hypothesis, all b are batch-normed with fixed statistics (0 mean 1 variance). We conclude that the term introduced is constant, and cannot influence optimization.
More pressingly, we are no longer on the hy- persphere, and cannot use the vMF kernel without checking works. However, note that the vMF kemnel, i.e.
Instead, consider T << 1. We have that
1 2 1uy 11²)
exp((ui, py))a exp(( Mg) 2 1:11
(pis
E log
Σ exp(
where denotes proportionality, and this pro- portionality holds when ||]]| ||l and constant. We recognize the right hand side as the Gaussian kernel (Keerthi and Lin, 2003), which is always applicable. simply remains to ask if the cor- rection factor exp(- 1| 2 the two kernels is strongly concentrated. By by- pothesis, we are near convergence, i.e. f4 is ap- proximately distributed in a Gaussian fashion. We know that high dimensional Gaussians are closely approximated by the uniform distribution the hypersphere that is, || strongly concen- trated around a constant (1) and thus, since the function question is Lipschitz over the domain, exp(- 1 2 b1l |مl | 2) also concentrates in mea- sure.
T
XycX'
is approximately equal to
(pis Mq)
arg max j
)l?) between
We recall the definition of the Kozachenko Leonenko estimator for sample consisting of X1,X2,.... . E Rd drawn from an un- known distribution P, assuming R; = minjæi ||x; - Xý1l2, Y N(R)).. Ba the volume of the unit ball in Rd, the Euler- mascheroni constant 0.577, an estimate of the entropy of the distribution is:
> 1, With
N-1 log Y, + log Ba +
Σ H(P) R N+1 1=1
Non-inclusion of the positive term If we in-
stead have
We do not need to calculate Ba and as they are constants per instance. Rather, we observe that
Elog
Σ
exp((16,11))
log Y; d(log N log R:)
Then this is leave-one-out evaluator of the kernel (Barnard, 2010). It is well known that in this case, the sum (upto some scaling, and with different bandwidth than the original sum) again approximates the entropy, but with an error equal to (in expectation) the generalization error. Hence, these two analyses do not differ asymptotically. Non asymptotic convergence: For this case, bring back the temperature term and assume it be set "correctly". Asymptotic results guar- antee convergence for all temperatures, but in the nonasymptotic domain, this case is only analyzable in the low temperature limit, because under the low temperature scenario we have:
Changing notation from X; to μı for the purpose of umifying our derivations, note that R, is attained at the lowest value of ||us pyl12 i.e. at the highest value of (pi Py) if ||pl ||,,ll are constants.
However, since we do not care about scaling constant shifts (recall that optimizing function f is equivalent to working with Af + for any X > 0 and any constant c). we can express log R; as
1 1 x 2 log R. log R
2
2
arg min log I|m Mjll

---

Expanding the norm squared, we get
is the most consistent and high quality semantically, grammatically, and linguistically ?". This yielded the following results. Note that GPT-4 actually yielded more strong results towards our model than the human evaluators.
1 2
arg min log (og(11 ||a/112 2(p4 pg))
We can use the fact || || concentrate to approximate the above
Candidate
CEAE
BN-VAE
VAE
Yahoo
55.7 %
28.9 %
15.4 %
Yelp
67.3 %
24.5%
8.2 %
1 2
arg min
1og(2 - 2/p Mj))
Table 7: Frequency of choice among generated samples among LSTM models by GPT-4.
Take the 2 in common, and note that this comes out of the log log(ab) log log b. But do not care about such constants as optimizing f is equal to working with linear transform of f upto the learning rate, leaving us with
Candidate
CEAE
BN-VAE
VAE
Yahoo
59.4 %
25.1 %
15.5 %
Yelp
52.0 %
30.4 %
17.6 %
Table 8: Frequency of choice among generated samples among transformer models by GPT-4.
1 log(1 (1)) 2
arg min
Finally, note that pt, Hi are high dimensional isotropic Gaussians of dimension Thus, (Mi, Mz) is a zero mean univariate Gaussian variance 3. i.. with high probability, we have that:
E BERT validation
We randomly selected 1000 sentences from Yelp and Yahoo corpora. This yields total of 1000 999 1 pairs of sentences, each of which yields an inner product similarity. The corresponding similarities were computed from BERT (Devlin et al., 2018). We can then compare the quality of embeddings from BN-VAF, VAE, CEAE with respect to BERT over three choices of correlation metrics: Pearson (linear), Spearman (rank-based) and Kendall (pairwise). The results are follows.
(4is M4) <<1
Allowing us to replace the above term with (ap- plying the identity that for x << 1, log(1++))) x))
arg min 1 2 ( ((m 14)))
Metrie CEAE
BN.VAE
VAE
Swapping argmin with argmax and positive with minus finally get
Pearson 0.46
0.35
0.33
Spearman
0.76
0.70
0.68
Kendall 0.82
0.75
0.70
arg max
2 ((it)l))
Table 9: Yahoo dataset, comparison of correlation met- rics w.r.t BERT.
Which is (upto scaling by ?) the desired non- asymptotic approximation.
The Layer norm case
C Extra Results - Penn Tree bank
Under layer normalization (Ba et al., 2016), the normalization is not done over minibatch but over layer. A consequence of this is the fact that the resulting latents form hyperspherical space, i.e. a|| where constant. Now, we know that
Here, we add results on the penn tree bank dataset in table 11. Only a subset of all models that do not use fixes specific to the LSTM architecture were considered, since we had to adapt the tricks to transformers. Our base model is Transformer-XL.
D GPT-4 validation
arg max H (2)
We asked GPT-4 validate the choices generated by our models, using the following prompt : "You are human asked to choose between more realistic sentences. Among the following sentences, which
is the uniform distribution over the hyperspheri- cal shell of radius c. It is also well known that in high dimensions, the multivariate isotropic Gaus- sian has almost all of its support (Vershynin, 2018)

---

Metric CEAR
BN-VAE
VAE
Pearson
0.58
0.48
Spearman
0.82
0.75
Kendall 0.86
0.75
0.79
Table 10: Yelp dataset, comparison of correlation met- rics w.t. BERT.
concentrated around the hyperspherical shell of Vd, where is the dimensionality. Hence, under the entropic regularizer, Layernorm approximates hyperspherical distribution which in turn approx- imates a Gaussian.
Although this direction is promising, we were only able to get our method (CEAB) operational under these circumstances, while BN-VAE did not converge. In the absence of full evaluation, the Layernorm case is of now inconclusive.

---

G
Statistical Analysis
Here we present results (in tables 12 and 13) on the standard deviation and confidence intervals on Yahoo and Yelp for selected models that perform well in terms of point estimates and rank at or near the top. These are the V-V BN-VAE and our method (CEAE). Bolding indicates passing a 1-test with p 0.01.

---

Model
Rec
GPT2-F
L-F
L-R
VAE
42.8
62.7
138.5
211.7
(-VAF (0.4)
43.2
64.2
140.5
225.8
eyclic
70.2
141.8
230.2
FB (7)
41.7
56.5
130.2
215.6
6-VAE (0.1)
42.0
58.9
132.9
208.3
BN-VAE (0.7)
41.5
135.6
202.8
CEAE
39.2
54.0
127.7
195.4
Table II: Language modeling results on Penn Tree Bank using 6-6 encoder-decoder transformer model with hyperparameters based on the Transformer- XL setup for Penn Tree Bank as released by the authors.
Model
Rec
GPT2-F
L-F 129.5-1318 (0.8)
L-R
D-V (0.1)
327.2-328.0KC (00.3)
120.8-124.0 (I.I)
169.5-174.2
BN-VAE (0.7) CEAE
317.7-319.5(0.6)
108.6- 10.0 (0.5)
122.9-126.7 (1.2)
166.2-170.1 (1.3)
316.0-317.5 (0.4)
107.2-108.4 (0.4)
118.5-122.3 (0.3)
162.2-164.7 (0.8)
Table 12: 5 to 95 percentile confidence intervals and standard deviations in brackets on Yahoo.
Model
Rec
GPT2-F
L-F
L-R
à- d-VAE (0.I)
354.3-358. 7(1.)
93.2-97.0(1.3
83.5-85.0(0.5)
106.7-113.5
BN-VAE (0.7) CEAE
345.2-347.6 (0.8)
89.0-91.2 (0.7)
79.4-81.2 (0.6)
107.2*-112.4 (1.7)
340.9-343.5 (1.2)
87.2-88.6 (0.5)
76.8-79.7 (1.0)
108.4-112.9 (1.6)
Table 13: 5 95 percentile confidence intervals and standard deviations in brackets Yelp. indicates statistically insignificant best result.


Reviews:
[{"review_id": "kTFxH6gcY2", "reviewer": "Reviewer_kNe8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for the review. T5 (along with some other models, e.g. BART) does have encoder-decoder structure but no latent space structure like a VAE, which means it is not directly comparable (this is noted in the initial parts of the OPTIMUS paper). It can be made comparable via introduction of a suitable latent space, just as we fuse BERT, GPT-2 and a latent space to create the OPTIMUS-VAE. We were unable to replicate the steps done for OPTIMUS on the T5 model to convert it to a VAE with satisfactory convergence, however, and we suspect a different approach is required. (The metrics of T5 without a latent space cannot be directly compared.)\n\nThat said, we view BERT / GPT-2 as language models of comparable “power” to T5 in terms of encoding and decoding respectively, and we believe that it should suffice. Further, note that decoder-only models (unlike encoder-decoder models) are increasingly more prominent, and as such, pairing encoders and decoders is likely to be the way moving forward to convert large models to large VAEs, instead of inserting a latent space in an already existing encoder-decoder pair.\n\n\nWe are keenly aware that the advent of large models will change the NLP landscape along with pre-training. But the comparison with BERT-GPT-2 models is meant to alleviate this concern and show VAE models can scale up reasonably well in the presence of large and pre-trained models. Indeed, note that the scaling procedure simply joins any two encoders and decoders of arbitrary size.\n\nAs to the superiorities of a structured latent space over conventional LM, there are multiple positives. For instance, interpolation with a known latent space can be done better exploiting the geometry (i.e., geodesic interpolation) over linear interpolation. This interpolation can then give rise to superior generations from the interpolated representation. This empirically manifests not just in better generation, but better downstream classification accuracy, as we show in our experiments. Text generation is also easier as there is a known, closed-form source distribution to sample from.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method consistently outperforms other VAE-based methods in two different datasets and works with different architectures (LSTM & Transformer). The Gaussian latent space, resulting from the autoencoder training, allows linear interpolation between sentences, which could be useful for future text generation or interpretation research.", "reasons_to_reject": "It's unclear how the proposed method is superior to the conventional language modeling objective. Especially, a T5-style model training with its standard objective should be included as a baseline since it also uses an encoder-decoder architecture. Last, as language modeling is shifting to large scale pretraining, the method's scaling capability remains a question.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a framework that adapts deterministic autoencoders for language modeling. This method employs contrastive learning, instead of the conventional KL loss, to make the posterior distribution Gaussian and hence avoid the posterior collapse.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pONt8Co23f", "reviewer": "Reviewer_1cF2", "summary": "", "questions": "- **Question A**: The paper mentions that BERT+GPT2 was selected due to the lack of an encoder-decoder structure that mirrors an autoencoder. However, models like BART and T5, which are based on encoder-decoders, have demonstrated state-of-the-art performance across various tasks. Why weren't experiments conducted using these models?\n  \n- **Question B**: The term \"outsized benefits in NLP\" mentioned in line 66 is somewhat vague. Could the author provide a more detailed explanation or context regarding this phrase?\n  \n- **Question C**: In line 270, the paper states, \"Under this condition of fixed mean and variance, it is known that the entropy H is maximized iff z is distributed as a Gaussian.\" Could the author elaborate on the underlying rationale for this statement?\n  \n- **Question D**: Typically, layer normalization is favored over batch normalization for text data. While this is acknowledged in Appendix F, the paper also mentions that the conclusion was not definitive without a \"full evaluation.\" Could the author clarify what constitutes a \"full evaluation\" in this context?\n  \n- **Question E**: In Table 6, phrases like 'with the beans' and 'with the salt' appear during interpolation, even though they don't seem to have a direct correlation with the source sentence. Could this phenomenon be indicative of posterior collapse?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThanks for your review ! First, we will address the reasons to reject, in sequence.\n\n1. We believe our paper, though somewhat more ML-oriented than purely NLP, has ample relevance towards NLP, and we have submitted specifically to the Machine learning for NLP track for this reason.\n\n2. We would appreciate knowing a specific experiment that you feel is lacking - this comment is a bit vague to work with. We do perform posterior collapse experiments as every test on latent representation quality, interpolation, etc. measures the degree of collapse.\n\n3. Again, the advantages for posterior collapse generally arise in representation quality and the reconstruction error - both of which do provide advantages for our model.\n\n4. We have used GPT-2 in conjunction with BERT, which is a reasonably large model (we are aware that there have been massive advancements in this area recently, and are trying to update accordingly)\n\n5. BN-VAE is relevant due to its utilization of batch norm with VAEs for NLP, and its (at the time) strong results with smaller architectures.\n\nThen, regarding the questions : \n\nA. Please see our response to reviewer kNe8. The fine-tuning of OPTIMUS does not carry over to T5 cleanly in a way that allows convergence.\n\nB. Certainly. Text relies more heavily on autoregressive models than images. As such, since posterior collapse afflicts autoregressive models more, fixing it is more likely to impact text performance than image performance.\n\nC. It is a standard result in information theory. The result is one of the Gibbs distributions, i.e. maximum entropy distributions. Basically, with some moments fixed, the max entropy distribution is known. With fixed mean and variance, it is Gaussian. Under absolute deviation constraint, it is the Laplacian, and so on.\n\nD. We could not get the BN-VAE models to converge under the LayerNorm scenario, so we could not know if they truly perform worse, or if we did not give them a fair chance.\n \nE. We do not see why that would follow. This dataset is about reviews of restaurants in yelp, and it is reasonable to have food-related items appear in the interpolation without necessarily implying that posterior collapse has occurred.\n\nWe will also take your suggestion for changing the placement of the human eval section into account.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **Comprehensive Review of Existing Studies**: The paper provides a well-organized overview of existing studies addressing the posterior collapse issue.\n- **Innovative Approach**: The build-up to address posterior collapse is distinct from existing methods, offering a fresh perspective on the problem.\n- **Effective Model Structure**: The author introduces a model structure that adeptly integrates Contrastive Learning and Batch Normalization, showcasing its potential in addressing the challenges of VAE-based language models.\n- **Reproducibility**: Essential details for reproducing the experiments and results are meticulously organized in the Appendix, promoting transparency and facilitating further research.\n- **Latent Space Exploration**: The paper's examination of the latent space through interpolation provides deeper insights into the model's capabilities and behavior.", "reasons_to_reject": "- **Not Tailored for NLP Audience**: The paper, while technically sound in some areas, does not seem to be specifically tailored for an NLP audience, which might limit its impact and relevance in the community.\n- **Insufficient Experiments on Posterior Collapse**: While the author emphasizes the mitigation of posterior collapse in existing VAE-based language models, there seems to be a lack of comprehensive experiments and analysis to support this claim.\n- **Unclear Advantages Over Existing Models**: The paper does not clearly delineate the advantages of the proposed model compared to existing solutions for posterior collapse, leaving readers uncertain about its unique contributions.\n- **Questionable Large Language Model Comparison**: The experiments conducted for comparing large language models appear to be inadequate. Labeling BERT as a \"Large Language Model\" seems outdated given the recent advancements in the field.\n- **Over-reliance on BN-VAE**: A significant portion of the paper is based on BN-VAE (Batch Normalization VAE). However, the rationale for its specific use and its significance in the context of the paper's objectives is not adequately explained.", "questions_to_authors": "- **Question A**: The paper mentions that BERT+GPT2 was selected due to the lack of an encoder-decoder structure that mirrors an autoencoder. However, models like BART and T5, which are based on encoder-decoders, have demonstrated state-of-the-art performance across various tasks. Why weren't experiments conducted using these models?\n  \n- **Question B**: The term \"outsized benefits in NLP\" mentioned in line 66 is somewhat vague. Could the author provide a more detailed explanation or context regarding this phrase?\n  \n- **Question C**: In line 270, the paper states, \"Under this condition of fixed mean and variance, it is known that the entropy H is maximized iff z is distributed as a Gaussian.\" Could the author elaborate on the underlying rationale for this statement?\n  \n- **Question D**: Typically, layer normalization is favored over batch normalization for text data. While this is acknowledged in Appendix F, the paper also mentions that the conclusion was not definitive without a \"full evaluation.\" Could the author clarify what constitutes a \"full evaluation\" in this context?\n  \n- **Question E**: In Table 6, phrases like 'with the beans' and 'with the salt' appear during interpolation, even though they don't seem to have a direct correlation with the source sentence. Could this phenomenon be indicative of posterior collapse?", "paper_topic_and_main_contributions": "The paper delves into the challenge of posterior collapse in variational autoencoders (VAEs) when applied to language modeling.\n\n**Main Contributions**:\n1. **Introduction of the Contrastive Entropic Autoencoder (CEAE)**: A structure that leverages contrastive learning to adjust entropy, aiming to mitigate the posterior collapse problem inherent in VAE-based language models.\n2. **Highlighting the limitations of existing VAE-based models**: The paper emphasizes that while many VAE models suffer from posterior collapse, the methods proposed to address this often negate the benefits of VAEs.\n3. **Proposing a solution that retains VAE benefits**: The CEAE is designed to counteract posterior collapse without sacrificing the advantages offered by VAEs.\n4. **Utilization of contrastive learning**: The CEAE employs contrastive learning specifically for Batch Normalization-based VAEs.\n5. **Comparative analysis**: The paper showcases the superiority of the proposed method by comparing it with various VAE-based language models and state-of-the-art models like GPT-2 and BERT.\n6. **Examination of the latent space**: The authors explore the latent space through interpolation, providing deeper insights into the model's capabilities.\n7. **Emphasis on the importance of addressing posterior collapse**: The paper underscores the criticality of mitigating posterior collapse in VAE architectures, especially in the context of NLP.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- **Section Organization**: The experimental design for human evaluation, mentioned around line 480, would be more appropriately placed in the \"4. Experimental Details and Methodology\" section rather than in \"5. Results\". This reorganization could enhance the flow of the paper and improve readability for the audience."}}, {"review_id": "P0LHXHkuGq", "reviewer": "Reviewer_1NbG", "summary": "", "questions": "(1)  In line 933-934 is written that the chosen $\\lambda$ parameter [of the Optimus model by Li et al. (2020)] corresponds to models with the lowest perplexity. Why did you choose this criteria? Why not to choose $\\lambda$ values for which the MI in Optimus is large? Note that the reconstruction error for larger $\\lambda$ values reported by Li et al. (2020) are sometimes better than those obtained with CEAE.\n\n(2) When training CEAE for comparison against Optimus, was CEAE pretrained on English Wikipedia as Optimus? The experimental section about the comparison with Optimus should probably be extended. And CEAE models should probably be compared with Optimus models with other $\\lambda$ values\n\n(3) Can you please explain in more detail whether the scale of the variance (i.e. $c$) in CEAE is annealed to zero? \n\n*Link to References*\n\nLi et al. (2020): http://aclanthology.lst.uni-saarland.de/2020.emnlp-main.378/", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe thank you for the review. Our reply follows. \n\nFirst, in the reasons to reject :\n\n1.  Our paper is indeed somewhat written with a general ML audience in mind - hence the choice of the track “Machine learning for NLP”. We will certainly add NLP-specific works for VAEs in the related work, and you are correct that our primary inspiration was from the image-based works and trying to extend them further. We did not have enough space for a dedicated related works section, but we do have prior work section, which is of the same purpose. We agree that perhaps too much space went into the generic AE literature over the specific case of AEs for NLP, however, we did not wish to confuse those without background in autoencoders, which have recently become less popular.\n\n2. This section generically sets up the AE. No, the reconstruction loss here is not directly used in the NLP loss (Reconstruction) in the experiments, for which indeed the likelihood is used. We apologize for this - this is meant as a general overview of AE, which generally utilizes the square loss for most circumstances outside NLP. We should have clarified that this carries over to likelihood in the sense that under Gaussian likelihood, the square loss can be interpreted as a log likelihood term.\n\n3. Correct - the large advances in foundational models does pose problems to the relevance of VAE and older models in general. Our work tries to upgrade the VAE to the modern world by seeing if the techniques hold up with relatively larger encoder-decoder pairs like BERT. We agree that more work needs to be done to see if these techniques scale up to large, GPT-scale models, but we did not possess the compute for such.\n\n4. Thank you for the references. Although, we note that we had previously seen Zhao 2018 and cited it (InfoVAE). We will integrate these.\n\n5. Right, we only chose the Gaussian as training in this case for a larger model, in the form of OPTIMUS had been attempted before and thus had some foundation.\n\nComing to the questions : \n\nA. Agreed, this is arbitrary, and was chosen as perplexity is a popular metric of model quality. If you prefer we can take the best $\\lambda$ for the reconstruction instead.\n\nB. Yes. In the OPTIMUS part of the results, only the batch norm and the way the loss is crafted changes and nothing else is, for reasons of consistency.\n\nC. As stated in A.1, $c$ is set to zero after a threshold of epochs, and not approaching zero.\n\nWe will take your presentation and grammar-based issues into account, specifically re : variables and instantiations, and inconsistent capitalizations. And yes, line 251 is somewhat confusing in its indexing. We will make $i,j$ consistent.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper represents a first successful attempt to employ deterministic autoencoders for language modelling.", "reasons_to_reject": "(1) The paper needs rewriting. It doesn't seem to be written for an NLP audience:\n\n- When reading the paper one gets the impression that the authors have not considered the large literature on the posterior collapse problem of VAE **specific to language modelling**.  For example, posterior collapse in VAE models for language modelling (LM) was identified already in the work of Bowman et al. (2016), which to my knowledge was the very first attempt to use VAE with text data. This work is however not cited by the authors, but several more recent works on VAE for images are cited instead. Unfortunately, the lack of a related work section exacerbates this feeling. For example, the paper relies on contrastive regularization but does not cite prior work on contrastive learning and NLP (see e.g. the review in Zhang et al. 2022).\n\n- Some of the notation used by the authors seems to have been taken directly from a VAE paper on images. Look for example at Eq. 4. Is the reconstruction loss used by the authors truly the mean squared error between word embeddings? or is it instead the log likelihood functions of a categorical distribution over some vocabulary (i.e. cross-entropy)? If it is indeed a mean squared error, then I don't understand the scales of the reconstruction values in Tables 1 and 2.\n\n- It's not so clear the relevance of this work when compared with the current state of affairs in NLP. I believe a paper on sentence representation nowadays should try to connect with the current paradigm of foundational models and their most pressing problems. For example, how would large pretrained language models benefit from inferring continuous representations of sentences? The authors make little to no reference to foundational models. Their only motivation seems to be to bypass the posterior collapse problem.\n\n- Somewhat more general, the paper relies on information theoretic concepts to infer its representations. Yet no connection is made with all previous literature on VAE, the ELBO and the mutual information between data and representation (see e.g. Zhao et al. 2018 in general or Zhao T. et al. 2018 and Fang et al. 2019 which are specific to LM). It'd be nice to relate their approach to previous work, since both ultimately rely on maximizing the mutual information between data and representation\n\n- Finally, the presentation of VAE focuses only on the cases where the prior is an isotropic Gaussian. But the prior needs not be an isotropic Gaussian. Indeed researchers in NLP have considered training the prior distribution (Fang et al. 2019) or learning structured, discrete representations (e.g. Zhao T. et al. 2018). \n\n(2) See also Question 1 below.\n\n*Links to references*\n\nBowman et al. (2016): https://aclanthology.org/K16-1002.pdf\n\nZhang et al. (2022): https://aclanthology.org/2022.naacl-tutorials.6.pdf\n\nLi et al. (2020): http://aclanthology.lst.uni-saarland.de/2020.emnlp-main.378/\n\nZhao et al. (2018): https://arxiv.org/pdf/1706.02262.pdf\n\nZhao T. et al. (2018): https://aclanthology.org/P18-1101.pdf\n\nFang et al. (2019): https://aclanthology.org/D19-1407.pdf", "questions_to_authors": "(1)  In line 933-934 is written that the chosen $\\lambda$ parameter [of the Optimus model by Li et al. (2020)] corresponds to models with the lowest perplexity. Why did you choose this criteria? Why not to choose $\\lambda$ values for which the MI in Optimus is large? Note that the reconstruction error for larger $\\lambda$ values reported by Li et al. (2020) are sometimes better than those obtained with CEAE.\n\n(2) When training CEAE for comparison against Optimus, was CEAE pretrained on English Wikipedia as Optimus? The experimental section about the comparison with Optimus should probably be extended. And CEAE models should probably be compared with Optimus models with other $\\lambda$ values\n\n(3) Can you please explain in more detail whether the scale of the variance (i.e. $c$) in CEAE is annealed to zero? \n\n*Link to References*\n\nLi et al. (2020): http://aclanthology.lst.uni-saarland.de/2020.emnlp-main.378/", "paper_topic_and_main_contributions": "This paper proposes a deterministic autoencoder for language modelling as an alternative to variational autoencoders (VAE) of text data, which are known to exhibit the so-called posterior collapse problem. Their methodology builds on the work of Ghose et al. (2020), but regularizes the model's objective function with a contrastive learning term. The latter is shown to approximate an entropic regularizer for Gaussian distributions in the limit of high dimensions. \n\nThe authors trained their model on three public datasets (PTB, Yelp and Yahoo) and showed that their models outperform different VAE models on both text generation and downstream tasks. The authors additionally showed their results to be consistent across different sequence-processing architectures (i.e. LSTMs vs Transformers), and even considered pretrained language models as the encoder-decoder pair. See however our questions below on the comparison with the work of Li et al. (2020).\n\n*Links to References:*\n\nGhose et al. 2020: http://proceedings.mlr.press/v124/ghose20a/ghose20a.pdf\n\nLi et al. 2020: http://aclanthology.lst.uni-saarland.de/2020.emnlp-main.378/", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Bowman et al. (2016): https://aclanthology.org/K16-1002.pdf"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is difficult to read because of the lack of consistency and formality in notation:\n\n- it seems the authors use capital and lower case letters to denote both random variables and their values interchangeably. See e.g. Eq. 1 ($\\mu$ as a realization) vs equation below line 251 ($\\mu$ as a random variable) vs that below line 255 ($Z$ as a random variable). Then again in the Equations below lines 306, 327 and 350.\n\n- it seems the authors use the index \"i\" to denote instances of the dataset and \"j\" to denote the components of the representations. But then it's not clear what the difference is, notation-wise, between the expectations in Eqs 8 and e.g. the expectation below line 251. Sometimes these indices are even used together, as in line 227. \n\n[EDIT: I removed the comment regarding including comparison against Sanchez et al. 2023 (https://aclanthology.org/2023.acl-long.263.pdf), for the latter was officially published after the EMNLP deadline]"}}]

Author rebuttal:

Meta-review: This paper proposes a deterministic autoencoder for NLP, as an alternative to VAE. The key idea is to introduce a contrastive learning loss to approximate entropy regularization. The paper also presents experiments on a variety of model architectures and downstream tasks. 

Overall, reviewers have divided view on soundness while show excitement in this paper's approach. 
Reviewers like the novel technical method, and the strong empirical performance on various architectures. However, reviewers also point out missing discussion with vast past literature on VAE for text, use of contrastive loss in NLP, analysis of mode/posterior collapse issues in VAE, isotropic Gaussian prior versus other priors (e.g. mixture-of-Gaussian). Reviewers also have concerns about the paper's reference to "large language models" since BERT, GPT-2 are no longer considered large nowadays.
Decision: Accept-Findings