Isotropic Contextual Representations through Variational Regularization
Authors: Cornelia Ferner
Stefan Wegenkittl
Venue: ICLR 2022 Submitted (2022)

Abstract: Contextual language representations achieve state-of-the-art performance across various natural language processing tasks. However, these representations have been shown to suffer from the degeneration problem, i.e. they occupy a narrow cone in the latent space. This problem can be addressed by enforcing isotropy in the latent space. In analogy to variational autoencoders, we suggest applying a token-level variational loss to a Transformer architecture and introduce the prior distribution's standard deviation as model parameter to optimize isotropy. The encoder-decoder architecture allows for learning interpretable embeddings that can be decoded into text again. Extracted features at sentence-level achieve competitive results on benchmark classification tasks.

Under review as a conference paper at ICLR 2022
ISOTROPIC CONTEXTUAL REPRESENTATIONS THROUGH VARIATIONAL REGULARIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Representations from common pretrained language models have been shown to suffer from the degeneration problem, i.e. they occupy a narrow cone in latent space. This problem can be addressed by enforcing isotropy in latent space. In analogy to variational autoencoders, we suggest applying token-level variational loss to a Transformer architecture and optimize the standard deviation of the prior distribution in the loss function as model parameter to increase isotropy. The resulting latent space is complete and interpretable: any given point is valid embedding and can be decoded into text again. This allows for text manipulations directly in latent space. Surprisingly, features extracted at sentence-level show competitive results on benchmark classification tasks.
INTRODUCTION
Self-supervised, attention-based language models are prone to overfitting memorizing input data, especially when trained on smaller datasets. Nevertheless, models such as BERT (Devlin et al.) 2019). based on an encoder-only architecture and trained on very large datasets, exhibit state-of-the- art performance on common natural language processing tasks. The resulting token representations of these models are distributed and highly contextual, but the latent space does not exhibit additional structural properties such as isotropy and resulting completeness (Ethayarajhi 2019).
We develop an autoencoder architecture that avoids overfitting by increasing latent space isotropy. At the same time, enforcing complete latent space leads decoder that is able to interpret (i.e. decode) any given point in latent space. While the primary goal are tasks that require paraphrase generation, the resulting encoders surprisingly also perform on par in transfer tasks with smaller training setups.
Previous work has shown that isotropic latent representation, i.c. distributed across latent space instead of populating a narrow cone (Ethayarajh) 2019), can improve the performance of language models transfer tasks. [Gao et al. (2019) improve the performance of Transformer- bbased model with increased isotropy by directly optimizing the cosine distance between the latent representa- tions. Waan al. (2020) suggest control the singular value distribution of the output represen- tation matrix and Li et al. (2020a) transform the original representation distribution into a Gaussian distribution through normalizing flows to increase the isotropy of the underlying models.
Similar to Gao et al. (2019), we propose to apply regularizing constraint during training to enforce an isotropic and, even more, complete latent space. A common regularization network is the so- called Variational Autoencoder (VAE) (Kingma & Welling 2014). VAE networks consist of an encoder that maps given input data not to a point latent space but a distribution. A VAE's decoder is required to successfully reconstruct the original input from samples of the latent distribution. Thus, the VAE is optimized to reconstruct the given input sequence whilst enforcing the latent distributions match a given prior distribution.
We propose a Variational Auto-Transformer (VAT), VAE based on a Transformer architecture, where the variational loss is computed at token level. We show that by enforcing Gaussian distri- bution as latent prior, the latent token-level representations become more isotropic in comparison to models such as BERT. We introduce the prior distribution's 's standard deviation model parameter to optimize isotropy and balance the language generation variety against the network's 's reconstruc- tion ability. Besides isotropy, the completeness of the latent space is demonstrated. The pretrained

---

Under review as a conference paper at ICLR 2022
decoder map back every point in latent space textual representation, be it actual encoded or synthetic, sampled latent points. This allows for text generation through "variational sampling" and other manipulations such as interpolation directly in latent space. While the token-level representa- tions can be used to generate or manipulate text, we also show that sentence-level representations, e.g. obtained through averaging, suitable for sentence classification tasks.
2 BACKGROUND
An autoencoder is neural network architecture consisting of an encoder enc, that maps any input IR a point enc(#) in latent space, and decoder dee, such that Rz dec(enc(æ)) (Good- fellow et al. 2016). The chosen network architecture defines the families E and D of the encoder and decoder. During training of the autoencoder, the network parameters are optimized to find the optimal (enc*, dec") pair that minimizes given loss function L:
(ené" ddec))
arg min (C) (enc
For the standard autoencoder with continuous inputs a E Rd, the reconstruction loss C = LEEC is:
CREC = | dec(enc(x))]l
(2)
Over-complete autoencoder architectures with many degrees of freedom are prone to particular form of overfitting: The encoder maps each data point to an isolated point in the latent space such that the decoder memorizes the lossless reconstruction of each of these codes. This highly discrete latent space lacks completeness and limited use for advanced NI.P applications.
2.1 VARIATIONAL AUTOENCODER
The variational autoencoder (VAE) (Kingma & Welling 2014) can be thought of as generative au- toencoder. A VAE encodes an imput data point as a distribution over the latent space by adding regularization term the loss function. The regularization term LREG assesses the Kulback- Leibler divergence between the latent distribution and a standard Gaussian based on their means μ and covariances Σ:
LREG DKI (N N(,
I and E are either estimated using standard point estimators from the encoder outputs, or computed by learned functions such that g(x) and Σ = l(x)) with G and h E H. where G and H are families of network architectures. G and H can contribute to desired properties representation by implicitly implementing dimensionality reduction or feature disentangling.
latent
Decoding from the latent representation requires sampling from ~ N(,, E). order to enable backpropagation despite this sampling operation, Kingma and Welling (Kingma & Welling 2014) introduced the reparameterization trick: Instead of sampling from the latent distribution, a random sample N(0, /) from standard Gaussian is drawn and then transformed by the computed mean and standard deviation z = - + μ., where L is the Cholesky factor of Σ.
The VAE's total loss is the sum of the regularization and reconstruction term: (enc", dec")=
arg min (ene,dec)EEx.D
(CREC + LREG)
arg min (x
- dec())) - DEL 2N,,
(4)
3 VARIATIONAL
Initially proposed for machine translation, the Transformer (Vaswani et al. 2017) can be used as language model and framed autoencoder. Extended with token-level variational loss, the VAT
2

---

Under review as a conference paper at ICLR 2022
Oulput Piokabiiilies
Softmax
Lincar
Tolon peeniieee
Encoder- Decoder
Bioerder Loyers
Attantion
reed Torward
MultiHead
Multi Head Attention
Attention
Input Positional Embeddiings
Output Positional Embaddings
Inpute
Outputs (shifted right)
Figure 1: The proposed Variational Auto-Transton Transformer architecture: Mean and covariance for the variational loss (blue, dashed lines) are computed through two independent linear layers. The en- coder and decoder (grey, solid lines) architecture is the same in the original Transformer.
maps input tokens to context-aware, distributed latent representations. Figure I illustrates the ar- chitecture the VAT. The set of encoders E and decoders is defined by the Transformer's net- work architecture based self-attention modules. The encoder maps the embedded input sequence X ( .2,,..,,
E TR''xd to a latent representation where T is the sequence length and d is the model dimension. To do so, the model predicts mu, and o, from æ, and samples 2 N (Jte ot). Z is then used as attention source in the decoder to predict the next word based on the already produced output. As X and Z depend on T,, the objective to minimize
(=1, 22 2T).
RR'xd,
Ex - dec(2z))l2 DEL [g(z,(X),,)))))))))))())) p(ze))
(5)
T
where p(zz) NN I) and {(x][X) Nu,, Ex). More precisely, we let E, diag (exp on) where the vector o, = (o,))==... is predicted, so that computing the actual value of Σ involves the exponential non-linear activation function.
The output of the encoder is passed to two linear layers in parallel. The linear layers have the same number of input and output nodes and learn a transformation to predict μ4, and ar for each token. The latent representation used as input in the encoder-decoder attention layer of the Transformer's decoder.
3.1 SCALING THE REGULARIZATION
During our experiments, we experienced overfitting on the training data for the reconstruction loss. The learning curves of corresponding experiment are illustrated Figure Weighting the reg- ularization loss according to logistic annealing function as suggested by Bowman et al. (2016) or by a scaling factor 3 similar to the scaling of the beta- VAE (Higgins al. 2017) but with 1, only had little effect.
Thus, we propose to scale the covariance matrix of the target distribution of LREg instead. This is motivated by the observation that standard Gaussians in d-dimensional latent spaces (e.g. d > 128) will overlap considerably if their mean values are at the same time regularized to zero. As a consequence, LREG cannot be minimized by the model without increasing LREC in undesirable way. Scaling the standard deviation to a too small value, though, might result in peaky distributions, having regulatory effect and resulting in trivial reconstruction objective.

---

Under review as a conference paper at ICLR 2022
Regularization Loss, s=1
Regularization Loss, s 0.4
10-2
10-7
10"6
10-2
10-10
10-10
100000
200000
300000
100000
200000
300000
Reconstruction Loss,
Reconstruction Loss, 5 0.4
4.00
4.00
2.00
0.00
0.00
100000
200000
300000
0
100000
200000
300000
train
test
train
test
(a) Standard Gaussian NO, I)
(h) Gaussian N (0, (0.4)
Figure 2: Weighted regularization (top) and reconstruction loss (bottom) for train and test data over several epochs for two different Gaussians applied as prior distribution for the regularization term.
We adapt the computation of the regularization loss to incorporate scaling factor for the standard deviation as hyperparameter. The closed form of the VAE loss function with a prior standard deviation op = s E (0, 1] and prior Up = 0 for specific token's 's layer outputs o, and l becomes
of. uh 2s²
- DKL [N(H,, ((aggd)
= Σ log(o 1=1
log(ss) +
(6)
This results in codes 2, o e 4 being distributed according Gaussian with zero mean and predefined standard deviation, e' ~ NO, diag(s)). The optimal balance between LREc and CEGG as expressed by the value of s (0, 1] is related to the representations' isotropy and will be determined experimentally.
4 EXPERIMENTS
The aim of the experiments to balance the two loss terms. LREC should be low to maintain the reconstruction ability of the network, while too small scaling value s implies that the VAT essentially behaves as regular AF. At the same time, we want to optimize the isotropy of the latent representations. order to choose the best setting, we observe both, the resulting loss values and the properties of the latent space in terms of similarity between representations.
Our VAT model architecture is smaller than the original Transformer architecture, as it tended for machine translation. The VAT consists of N = encoder and decoder layers, respec- tively, with H attention heads each and model dimensionality d 128. A logistic annealing function (Bowman et al. 2016) is applied to weight LREG- The VAT is trained on the train split of the WMTT9 de-en dataset (Barrault al.) 2019) using English sentences only. WMT19 contains data from the news commentary, Wiki titles, Europarl, ParaCrawl and Common Crawl corpora. The data tokenized using subword tokenization with a target vocabulary of 2 tokens. The full details on model and training parameters are listed in the appendix.
in-
4.1 OPTIMAL SCALING PARAMETER
The isotropy of the learned representations is assessed as a function of the scaling factor & E (0, 1]. Ettayarajh (2019) introduced the notion of isotropy of the latent space the mean cosine similarity between vector representations of random tokens. In an isotropic latent space, representations of randomly sampled tokens have low cosine similarity and do cluster in specific direction.
Refer to Odaibo 2019) for the derivation of the closed form.
4

---

Under review as a conference paper at ICLR 2022
Casine Similarity
155
2.0
0.5
0.2
0.0
1.0
0.5
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Scaling Factor
Scaling Factor
Similartty becaeen Random Words (Isctrapy)
Total loss
Average
Similarly
Average Self-Simllrrit
Rlguller stalion loss
(a) Cosine similarities the latent representations VAT the end of the training process de- noting the isotropy of the latent space and the con- textuality of token representations.
(b) Reconstruction loss, weighted regularization loss total loss of the VAT at the end training process.
Figure Optimizing the scaling factor s with respect to isotropy and loss values.
In addition to isotropy, Ethayarajh (2019) introduces the notion of self-similarity and intra-sentence similarity. high self-similarity, i.c. mean cosine similarity between representations of the same to- ken in different sentences, indicates that neighboring representations capture similar concepts which contributes the smoothness of the latent space. Intra- iitra-eentence similarity measures the mean cosine similarity of tokens occurring in the same sentence to the mean sentence representation, thus being related to contextuality.
BERT exhibits low isotropy (mean cosine similarity >> 0.4) in its last layers (Ethayarajh, 2019).
phenomenon known as representation degeneration (Gao et al., 2019). In contrast to BERT with high contextuality with respect to the low isotropy. expect more complete latent space for the VAT and thus high isotropy. The results Figure 3$ are computed for the STS 12 datasets Agirre al. (2012). The average cosine similarity between randomly sampled words is low for smaller values, suggesting an isotropic latent space with representations distributed across space. The latent space most isotropic
0.4.
The average self-similarity of tokens greater than 0.6 suggests a mapping of similar concepts into the same region, indicating smoothness in the latent space. The intra-sentence similarity for 0.5 2 s 2 0.1 is lower than the self-similarity, but still above the similarity of random words. This indicates that contextual information is captured in the representations. Interestingly, for 2 2 0.6, the token representations seem to "collapse" into an identical representation for all positions in sequence resulting an intra-sentence similarity of 1. Among the words with most context-specife representations are mainly stopwords as they have lowest self- seff-smillartt.
we report the final loss values after training the models for epochs with different scaling factors E (0, 1]. Scaling values 1 2 2 0.6 result in near zero regularization losses and high reconstruction error. The original input cannot be reproduces any more. This observation corresponds to the findings from Figure where the representations within sentence are identical. Starting from 0.5, the reconstruction loss drops and with s = 0.4, the reconstruction loss falls below the regularization loss. Values smaller than 0.4 further decrease the positive effects of the variational model, as the achievable variety generated text is reduced. This phenomenon is referred to as posterior collapse (Lucas al. 2019).
In Figure 3b
Combining the findings from evaluating the loss and latent space properties in Figure 3 we select 0.4 scaling factor for further evaluating the VAT. This value corresponds good reconstruc- tion ability and isotropic representations which impacts both the generation of text as well as the performance on classification tasks. Also, the learning curves corresponding to s 0.4 exhibit no overfitting any more (see Figure 2b).
Different from Ethayarajh 2019), Figure 3a depict the plain intra-sentence and self similarity, i.c. not adjusted for isotropy.

---

Under review as a conference paper at ICLR 2022
In this class, we will introduce several fundamental concepts.
In this class, we will introduce several fundamental concepts.
In this class, we will introduce several fundamental principles. In this area, we will introduce several fundamental principles. In this process, this class, will introduce several fundamental principles.
will introduce several educational concepts.
I want to talk to you. want to report.
In this progress, we will introduce private C fundamental concepts. In this class, we have received several fundamental concepts. In this class, will introduce several public projects. April 2013, we will introduce several fundamental issues. this class, we will find several fundamental principles. In one class, she will introduce several fundamental concepts.
said:
She didn't want to say.
She didn't want to be with him.
He silent for long moment.
He was silent for long moment. He was my father. It was my face.
Table I: Variational Sampling. Example sentence (in bold) was used as input to the VAT, with the decoded mean representation and samples from the latent distribution (in italics).
It was my turn.
Table 3: Interpolation. First and last sentence (in bold) were given as input to the VAT. Intermedi- ate sentences were obtained by token-wise linear interpolation be- tween the two latent representa- tions padded to the same length.
Generative models have shown great promise in modeling complex distributions.
advantages, results, achievements, quality, answers participation, moments, experience, passion, joy, benefits, in- terests
Table 2: Variational Sampling. Example sentence (in bold) that was used as input to the VAT, with only the underlined token being sampled. Samples from the latent distribution in italics.
5 VARIATIONAL LANGUAGE MODEL
During training. the VAT encoder produces variants of the input tokens by drawing from the latent distribution. The VAT decoder has to be able to reproduce the original sequence given variational input. At test time, both approaches are possible: Decoding with the standard deviation set to 0, i.e. 2, = pr: leads to deterministic representation and is used for classification tasks. For sequence generation, decoding & o E' + μ4; allows to generate variants of the original input sequence, process we refer to as variational sampling. Variational language modeling thus denotes the various ways of manipulations and computations in the latent space that possible with the VAT's latent distributions. As examples, we discuss anecdotal results from variational sampling and interpolation.
Tables [ and 2 illustrate the generating capabilities of the VAT with variational sampling. In the second Tine in Table [ the VAT is able to reconstruct the original sequence (first line), which refers to decoding with z f4t, i.e. without sampling. The following lines show exemplary variants of the input sentence obtained by randomizing the latent representations, displaying 12 variational samples. It interesting that, while mostly maintaining the original sentence structure and original context, variants are found for almost all tokens. This gives an idea of the structure of the latent space and the contextuality of the representations. The approach is comparable to paraphrase generation, which is often obtained through transsaannttioon (Sennrich et al., 2016), (Wieting & Gimpel| (018)) or by directly trained on supervised data in the form of paraphrase sentence pairs (Gupta et al., 2018).
Table illustrates variational sampling for a single token, where the input sequence (in bold) is reconstructed according to z, lr except for the underlined token for which sampling is allowed. This setting is similar to a nearest neighbor search over a fixed corpus. The second line lists the obtained samples. By masking the underlined token, this approach is similar to gap filling (Donahuel et al. 2020). (Wu et al., 2019). Variational sampling, both for tokens and sequences, can be useful for generating paraphrases in tasks such as dialog generation, question answering etc.
We also experimented with latent space interpolation similar to (Bowman et al. 2016). (Liu & Liu 2019) (Li et al. 2020b). We linearly interpolate the latent representations of two sentences (padded

---

Under review as a conference paper at ICLR 2022
STS12
STS13
STS14
STS15
STS16
STSb
SICK-R
Average
Published in (Reimers & Gurevych, 2019]
BERT-star
20.16 38.78
30.01
20.09
36.88
38,08
16.50
42.63
29.19
BERT mmean
57.98
57.98
63.15
61.06
46.35
58.40
54.81
Our results
MiniBERT-starr
-2.19
0.87
1.27
0.81
-3,46
-2.41
0.61
-0.64
MiniBERT-mean
-1.14
-3.83
1.63
0.21
7.25
-5.3
-1.73
-0,42
MiniBERT-sun
-0.50
-1.54
0.60
-0.72
0.11
1.21
1.67
0..12
VAT-start 17.45
12.10
14.74
20.09
31.14
30.70
37.15
23.34
VAT-mean
45.60
45,43
52.07
55.45
58.13
50.83
47.08
50.66
VAT
45.60
45.43
52.07
55.45
58.13
51.56
48.88
51.02
Table 4: Spearman rank correlation between the cosine similarity of sentence representations and gold label similarity for different semantic textual similarity (STS) tasks without fine-tuning the underlying models the target data.
to the same length) with three intermediate steps. Decoding the intermediate representations results in the sentences illustrated in for two examples. Observing a smooth interpolation trajectory as in the examples is not always the case. It is possible that after decoding, the intermediate steps are mapped to the same sentence as seen for one sentence in the second example. This is especially true with increasing size of training data: The more training data, the more contextualized representations for the same word type that are mapped close to each other in latent space will exist.
SENTENCE REPRESENTATIONS
For the construction of sentence representations, compare two pooling operations (average, sum). Additionally, the start token ("CLS") representation can be used as embedding for the whole sen- tence. but this approach has been shown be inferior mean pooling semantic similarity tasks for BERT (Reimers & Gurevych 2019). We denote the different approaches as VAT-mean, VAT- sum and VAT-start, respectively. We compare our model the original BERT model (base) and, for a fair comparison in terms of model size, to a smaller variant with similar network dimensions than the VAT. BERT has d 768. The smaller BERT mode (Ture et al. 2019). which we refer to MiniBERT, has 128 and L = 4, the same size as the VAT. Both BERT and MiniBERT were trained on a much larger collection of datasets (Wikipedia, BookCorpus, CommonCrawl) than the VAT.
12 encoder layers and the model and representation dimension is
The sentence representations are tested using the SentEval (Conccuu & Kicla 2018) toolki The toolkit evaluates static sentence representations on two different classes of tasks: semantic similarity and sentence classification tasks. Table 4 lists the correlation values of the different sentence repre- sentation methods on the semantic textual similarity (STS) tasks. For each pair of semantically sim- ilar sentences, the spearman correlation rank between the cosine of their latent representations and human-labeled gold standard (between and 5) is computed (no fine-tuning or transfer learning). The correlation of VAT-based sentence representations is on par with BERT-based representations which is surprising given the smaller model architecture and capacity the VAT. For both VAT and BERT, start token based representations show less correlation than those obtained by average or sum pooling. Representations obtained from the MiniBERT model do not show any correlation at all, indicating that the BERT architecture requires larger model size.
The classification tasks included in the SentEval toolkit comprise binary (MR, CR, SST2) and fine- grained (SST5) sentiment polarity classification tasks, paraphrase detection (MRPC), natural lan- guage inference (SICK-E) and question-type classification (TREC) tasks. As paraphrase detection and natural language inference require the comparison of two sentences, the pair of input sentences concatenated and separated by a special separation token to form a single input vector. For the classification tasks, we compare the performance of the VAT to that of MiniBERT in two ways: in feature-based approach and in a fine-tuning approach.

---

Under review as a conference paper at ICLR 2022
MR
CR
SST2
SSTS
TREC
MRPC
SICK-E
caature-baseld Approach
MiniBERT-stan -start
50.8/50.1
63.9/63.8
51.2/49.9
27.7/24.6
23.7/18.2
67.8/66.5
56.4/56.7
MiniBERT
50.9/50.1
61.6/60.9
53.0/50.3
24.4/25.1
22.9/18.2
65.5/65.9
55.6/55.9
MiiiBRT -sum
51.9/49.6
62.2/59.7
51.4/51.5
25.8/23.0
23.0/20.8
64.8/57.6
48.4/49.7
VAT-start
60,5/59.6
63.8/63.8
61.6/63.3
29.4/29.3
35.1/40.4
67.5/66.5
56.4/56.7
VAT-mean
61.0/60.2
63.9/63.8
61.4/60.7
30.4/31.0
44.1/48.6
67.5/66.6
59.2/58.5
VAT-sum
61.2/60.5
66.8/65.7
61.0/62.6
31.5/32.2
46.8/54.8
69.6/68.5
67.8/66.6
Fine-tuning Approach
MiniBERT-stari
72.8/72.1
64.4/61.9
80,6/83.2
36.7/36.1
32.1/38.8
68.9/66.5
56.4/56.7
MiniBERT
76.0/75.6
73.4/65.6
81.0/83.3
38.0/37.9
72.4/73.2
71.8/6866
58.2/58.
MiniBERT-sua
73.8/70.7
75.4/67.6
70.0/81.6
35.8/32.8
70.8/69.6
68. 68.9/63.0
57.6/44.3
VAT-start 74.5/74.6
76.3/72.5
82.7/82.6
37.8/38.6
82.7/87.2
70.0/69.9
61.6/61.3
VAT-mean 73.8/74.1
75.0/72.5
81.7/82.4
39.9/40.0
1/85.2
68.9/63.7
64.2/61.8
VAT-sum
72.9/73.2
78.6/74.5
82.6/83.3
40.7/39.9
84.7/88.2
69.7/68.3
62.0/60.8
Table 5: Dev/test accuracies on the SentEval transfer classification tasks. A single dense layer with nonlinear activation and Adam optimizer used as classification layer. Bold values indicate the best test result for each task in the feature-base based and fine-tuning approach.
Both presented approaches are different from how state-of-ther -frtte-rrt models are trained for transfer tasks. Given the smaller model capacity, do not expect on-par performance, but rather want to better understand the kind of information stored in the representations and the potential that comes with additional classification layers on top of the model. The comparison to MiniBERT serves as baseline.
For the feature-based approach, the sentence representations are extracted from the pretrained mod- els and then passed to single classification layer trained with an Adam optimizer (Kingma & Ba 2013). Given the results in Table 5 (top), the sum of the individual token representations is best suited standalone feature for sentence classification. As VAT-sum outperforms VAT tasks, the dimensions of the latent representations seem to be well disentangled. With the excep- tion of the SST2 task, the start representation yields lowest accuracy values suggesting that the first representations does not capture the full context of a sentence. MiniBERT, which is designed for fine-tuning rather than feature extraction, mostly does not reach the performance of VAT variants. The difference between MiniBERT and VAT is especially large for the TREC task.
all
For assessing the performance with fine-tuning approach, extend the underlying models by single non-linear dense classification layer and optimize with Adam. For the VAT, we only reuse the encoder part to access the latent representations Z. For each task, we trained for five epochs Note that the MiniBERT results can differ from the results in the original publication (Ture et al. 2019). as we do not further tune the model or the training process.
In (bottoo,), we see performance leap for all of the models compared to their feature-based results. While VAT-sum was performing best for the feature-based approach, the method to produce sentence-level representations is not that decisive for the fine-tuning approach. The differences between sum, mean and start are not as great as for the semantic similarity tasks any more, neither for MiniBERT nor for VAT. It is interesting that MiniBERT is in lead for sentiment classification tasks, whereas VAT shows outstanding performance on the TREC (topic classification) task.
7 RELATED WORK
The architecture of variational autoencoders has been optimized for various natural language pro- cessing tasks by implementing various kinds of networks for encoder and decoder. The variational loss is applied not contain a decoder any more, such that the latent representations are not interpretable.
sentence-level latent representation in all these models. Some of the networks do
Several models were suggested to solve common document text classification tasks: Gururangan jet al.] (2019) use MLPs as encoder and decoder to learn latent vectors from bag-of-word inputs that used for document classification. Mahabadi et al. (2021) compress BERT-baasd sentence
8

---

Under review as a conference paper at ICLR 2022
representations through a Gaussian prior distribution for transfer classification tasks. Deudon (2018) implements a siamese network architecture based on sentence representations from a VAE with Bi- LSTMs for semantic similarity classification.
Zhao et al. (0118) use LSTM-based VAE for dialog generation. Miao et al. (2016) rely on MLPs in VAE for both document modeling and answer selection. Wang & Wan (2019) introduce Transformer- based VAE for story completion, where the missing plot is conditioned on the latent representation z capturing the distribution of the story plot. [Shu al. (2020) propose a Transformer- based non-autoregressive model specifically for machine translation that incorporates a predictor network for the length of the target sentence.
Various network architectures have been proposed for the more general task of language modeling. Bowman et al. (016)) apply the variational loss on the last hidden state LSTM sequence-to- sequence model. In iVAE (Fang et al. 2019), an MLP produces sample-bar -ased distributions from an LSTM's hidden representation concatenated to random noise. Yang et al. (2017) apply an LSTM as encoder and CNN as decoder in language model. Liu & Liu (2019) add ferd-forward layer to Transformer model to map the encoder outputs to a mean and variance vector which are then upsampled and passed to the decoder followed by an LSTM layer. OPTIMUS (Li et al.) 2020b) extends this idea to large-scale language models. The encoder weights are initialized with pre-trained BERT weights, the decoder with those from pre-trained GPT-2 model. The latent representation of the start token is used as sentence-level representation.
In contrast these networks, the VAT's variational loss is applied at token-level which allows for direct manipulations in the latent space. Token-level reguralisation has been previously applied to RNNs for language modeling (Li al. 2020c). While language modeling is the training objective of the VAT, we also investigate the performance of the trained model on text classification tasks.
DISCUSSION AND CONCLUSION
The goal of the proposed VAT model is to obtain isotropic representations mapped to smooth and complete latent space. The hyperparameter and scaling factor 8 is directly optimized to meet these criteria. The effect can be seen especially for text generation, where the VAT is able to produce coherent sequences through manipulations directly in latent space. However, optimizing REDG comes at the cost of LrEc- Perfect reconstruction of the original input not possible at all times any more.
Reducing the representations to single sentence-level vector preserves contextual information. Semantic similarity tasks demonstrate that the smaller model capacity of the VAT is sufficient to capture semantic information that is par with larger BERT architectures. However, models explicitly trained to improve the performance on semantic similarity tasks are out of reach.
The VAT able produce more robust standalone sentence-level features when compared with MiniBERT for the feature-based classification approach. VAT outperforms the similar-sized MiniB- ERT also when being fine-tuned on sentence classification tasks, but is far from the state-of-the-art performance known from BERT-sized models with more sophisticated training procedures. Inter- estingly. VAT shows its peak performance for the topic classification task TREC. In our setup. the VAT captures topic-related information even better than sentiment.
Tasks involving the comparison of two sentences (MRPC, SICK-E), represented by a single con- catenated vector, cannot be successfully solved by either VAT or MiniBERT. Some model variants (MiniBERT included) only learn to predict the most frequent class. The reason could be the smaller model sizes that are incapable of capturing the information of two concatenated sentences, espe- cially the VAT was never trained on such a setting. Instead a concatenated vector, the comparison task could be solved using siamese network architecture, similar to Deudon Optimmizng the regularization loss at token-level could even allow the training a multilingual model or a translation model, given sufficient training data.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
All data used for training and evaluation of the model are publicly available. The WMTI9 dataset obtained through the tensorflow library https:: http://wwwwttt.oott/tt/t wwwsorflsoooooooo wow.//r//woooooooooooooooooo all other datasets are available through the Sen- The source code for instantiating and training our model as well as for decoding and generating variational samples made available as supplementary material. Hyperparameters and training settings are described in Section 4 and - in more detail in Appendix A
datasets http://aaaalloottttsstttaaaaaaaa
tEval toolkit htthttsss/gggiiii
com/
REFERENCES
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Konzalez-Agirre. SemEval-2012 Task 6: Pi- lot Semantic Textual Similarity. In *SEM 2012: The First Joint Conference Lexical and Computational Semantics - Volume 2: Proceedings of the Sixth International Workshop on Se- mantic Evaluation (SemEval 2012), pp. 385-393, Montréal, Canada, June 2012. Association for Computational Linguistics.
Loic Barrault, Ondiej Bojar, Marta Costa-Jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, et al. Findings of the 2019 Conference on Machine Translation (WMT19).. In Proceedings of the Fourth Conference on Machine Translation ( (Volume 2: Shared Task Papers, Day 1). pp. 1-61, 2019.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating Sentences from a Continuous Space. In Proceedings of The 20th SIGNLL Confer- ence on Computational Natural Language Learning, pp. 10-21, Berlin, Germany, August 2016. Association for Computational Linguistics.
Alexis Conneau and Douwe Kiela. SentEval: An Evaluation Toolkit for Universal Sentence Repre- sentations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Associa- tion (ELRA).
Michel Deudon. Learning Semantic Similarity in Continuous Space. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- BesacBianchh, and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems, volume 31. Curran Associates, Inc., 2018.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Chris Donahue, Mina Lee, and Percy Liang. Enabling Language Models to Fill in the Blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- Online, July 2020. Association for Computational Linguistics.
Kawin Ethayarajh. How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNIP), pp. 55-65, Hong Kong, China, November 2019. Association for Computational Linguistics.
Le Fang, Chumyuan Li, Jianfeng Gao. Wen Dong, and Changyou Chen. Implicit Deep Latent Vari- able Models for Text Generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing TEMNN.P-IJCNLL pp. 3946-3956, Hong Kong, China, November 2019. Association for Computational Linguisties.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation Degeneration Prob- lem in Training Natural Language Generation Models. In International Conference on Learning Representations, 2019.
10

---

Under review as a conference paper at ICLR 2022
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. A deep generative framework for paraphrase generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. Variational Pretraining for Semi- supervised Text Classification. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguiitics, pp. 3880-5894, Florence, Italy, July 2019. Association for Computa- tional Linguistics.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, Toulon, France, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations, Banff, AB, Canada, 2014.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the Carbon Emissions of Machine Learning. CoRR, arXiv:1910.09700, 2019.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the Sentence Embeddings from Pre-trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9119-9130, Association for Computational Linguistics, November 2020a.
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. Op- timus: Organizing Sentences via Pre-trained Modeling of a Latent Space. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 4678- 4699, Online, November 2020b. Association for Computational Linguistics.
Ruizhe Li, Xiao Li, Guanyi Chen, and Chenghua Lin. Improving Variational Autoencoder for Text Modelling with Timestep-Wiss Regularisation. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 2381-2397, Barcelona, Spain (Online), December 2020c. International Committee on Computational Linguistics.
Danyang Liu and Gongshen Liu. A Transformer- Based Variational Autoencoder for Sentence Gen- eration. In 2019 International Joint Conference on Neural Networks (IJCNN) pp. 1-7, 2019.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Understanding Posterior Collapse in Generative Latent Variable Models. In International Conference on Learning Repre- sentations, DeepGenStruct Workshop, New Orleans, LA, USA, 2019.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. Variational Information Bot- tleneck for Effective Low-Resource Fine-Tuning. In International Conference on Learning Rep- resentations, 2021.
Yishu Miao, Lei Yu, and Phil Blunsom. Neural Variational Inference for Text Processing. In Pro- ceedings of the 33rd International Conference on Machine Learning Volume 48, ICML' pp. 1727-1736. JMLR.org, 2016.
Stephen Odaibo. Tutorial: Deriving the Standard Variational Autoencoder (VAE) Loss Function. 2019.
CoRR,
Nils Reimers and Iryna Gurevych. Sentence-BERT Sentence Embeddings using Siamese BERT- Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP- IJCNLP), pp. 3982-3992, Hong Kong, China, November 2019. Association for Computational Linguisties.
11

---

Under review as a conference paper at ICLR 2022
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics, pp. 86-96, Berlin, Germany, August 2016. Association for Computational Linguistics.
Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. Latent-variable Non- autoregressive Neural Machine Translation with Deterministic Inference using a Delta Posterior. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8846-8852 2020.
Tulia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. CoRR
2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Infor- mation Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017.
Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, and Quanquan Gu. Im- proving Neural Language Generation with Spectrum Control. In International Conference on Learning Representations, 2020.
Tianming Wang and Xiaojun Wan. T-CVAE: Transformer-Base Conditioned Variational Autoen- coder for Story Completion. In IJCAI, pp. 5233-5239, 2019.
John Wieting and Kevin Gimpel. ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties, pp. 451 1622 Melbourne, Australia, July 2018. 8. Association for Computational Linguistics.
Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. Mask and Infill: Applying Masked Language Model for Sentiment Transfer. In Proceedings of the Twenty-Eighe Interna- fional Joint Conference on Artificial Intelligence, IJCAI-19, pp. 5271-5277. International Joint Conferences on Artificial Intelligence Organization, 7 2019.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor
Improved Variational Autoencoders for Text Modeling Using Dilated Convolutions. In Proceedings of the 34th Inter- national Conference on Machine Learning Volume 70, ICMI'11, pp. 3881-3890. JMLR.org, 2017.
Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. Unsupervised Discrete Sentence Represen- tation Learning for Interpretable Neural Dialog Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1098- 1107, Melbourne, Australia, July 2018. Association for Computational Linguistics.
A APPENDIX
MODEL DETAILS
The implemented VAT consists of N = 4 encoder and decoder layers, respectively, with H = 8 attention heads each. The encoder and decoder architecture including the positional embedding is the same in the original Transformer (Vaswani et al.) 2017) and can be seen in Figure [ The model dimensionality = 128, the dimension in the feed forward layers is 512. Dropout during training is set to 0.1. The Noam optimizer Vaswani et al. (2017) operates with 50, 000 warmup steps. A logistic annealing function Bowman et al. (2016) with 50, 000 warmup steps and an initial value of 0.00025 is applied to weight the regularization term of the loss function.
The VAT is trained on the train split of the WMT19 de-en dataset Barrault et al. (2019) using English sentences only. The data is tokenized using subword tokenization with a target vocabulary of 2 tokens. Batch size is set to 128. Decoded sentences that are presented as results are obtained through beam search with beam size
12

---

Under review as a conference paper at ICLR 2022
ENERGY CONSUMPTION
Experiments were conducted using private infrastructure, which has an estimated carbon efficiency of 0. kgCOzeq/kWh. A single epoch takes 20 hours of computation on RTX 2080 Ti (TDP of 250W) GPU. For the training of a VAT model for epochs, total emissions are estimated to be 0.99 kgCOzeq according to the ML impact calculator Lacoste al. 2019) available
ht ht hetps://mlco2/ htt.://ccoo...mmm wwt.oo...ittu........ 10/ impact #compute
13


Reviews:
[{"review_id": "h1yYZBVSOYC", "reviewer": "Reviewer_j72L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- Clear description of the proposed model.\n- Extensive experiments show the usefulness of various design choices (e.g., scaling factor) of the proposed method.\n\n**Weaknesses**\n\n- For the evaluation of isotropy, currently there seems to be only those of VAT (with different hyperparameters). Some comparisons to other models (esp: MiniBERT_*) would better support the effectiveness of VAT in isotropy. This way the readers would have an idea of how the improvements of VAT relates to the improvements in classification tasks.\n- The evaluation on interpretability can be deeper. Section 6 has some examples, but additional elaboration would be appreciated. \n- There are already a lot of evaluations, but various virtues of the models can be connected by further evaluations. E.g., how do the isotropy and the performance on the transfer classification tasks relate?\n\n**Comments and questions**\n\n- Page 1 in the abstract \"prior distribution\". What does the prior refer to?\n- Page 2, equation 2: is the norm L1? (Or L2?)\n- Section 5.1: Did you normalize the self-similarity and intra-sentence similarity like the description of Ethayarajh (2019)?\n- Also in section 5.1: Is the anisotropy the average cosine similarity?\n- Section 6 briefly mentions the interpretability of the language model by presenting some samples from the latent distributions. In what way are the examples special?\n- Adam optimizer: Please cite the paper proposing this algorithm.\n- Several other papers use variational methods in language modeling. Worth mentioning in the related work section:\n\nYang, Zichao, et al. \"Improved variational autoencoders for text modeling using dilated convolutions.\" *International conference on machine learning*. PMLR, 2017.\n\nLi, Ruizhe, et al. \"Improving variational autoencoder for text modelling with timestep-wise regularisation.\" *COLING*, 2020.\n\nFang, Le, et al. “Implicit Deep Latent Variable Models for Text Generation.” *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, Association for Computational Linguistics, 2019, pp. 3946–56.", "summary_of_the_paper": "Provide a brief summary of the paper and its contributions.\n\nThis paper proposes Variational Auto-Transformer to encourage isotropy in the latent representation space. The resulting encoder-decoder architecture allows interpretable embeddings. On various tasks, VAT shows better performances than other language models.\n\nContributions:\n\n- A novel architecture based on Transformer with a token-level variational loss.\n- Extensive evaluations on the effectiveness of Transformer.", "summary_of_the_review": "Proposes a novel method and clearly elaborates it. Extensive experiments verifies the effectiveness, as well as the design choices.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6aLoWI8o73R", "reviewer": "Reviewer_pcez", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nIsotropy of word embedding space has been a topic of strong interest to the NLP community in recent years, and it is reasonable to adopt VAE as the underlying method.\n\n**Weaknesses**\n\nUnfortunately, almost all of the experiments have some flaws that are difficult to ignore, and the manuscript might not show the superiority of the proposed method.\n\n1. Regarding variational sampling: Various methods have been proposed that sample sentences with similar meanings to input sentences, such as back-translation [1,2], text infilling [3], and VAE [4]. Since there is no comparison between these existing methods and the proposed method in the manuscript, we must say that the superiority of the proposed method is unknown. Additionally, only one example is given for the output of the proposed method, and we cannot shake the concern that this is champion data. The persuasiveness of the paper will be improved if the proposed method is appropriately compared to existing methods through descriptions and experiments.\n2. Regarding interpolation: There is no mathematical or procedural description of the computation and no citations. It must be said that the manuscript lacks self-containedness and reproducibility. The reader would benefit from a clear description of the algorithm. In addition, as the authors stated, if it does not work well in general, then it is somewhat inadequate to convey the merits of the proposed method.\n3. Regarding sentence representation: There are two problems with the comparison with MiniBERT.\n    - First, as mentioned by the authors, the training corpus is different for the proposed and baseline models. Two variables are simultaneously changed, the domain of the data and the model, and this cannot be said a control experiment. Furthermore, the community has developed the learning tool for BERT [5], and comparative experiments can be performed easily.\n    - Second, the NLP community rarely uses BERT-mini in pure research or practical applications. Therefore, even if the proposed method outperforms BERT-mini in controlled experiments, it is hardly validated that the proposed method is a valuable model. The empirical persuasiveness of this study would be significantly improved by increasing the parameters of the proposed model and comparing it with commonly used models such as BERT-base and BERT-large.\n\n---\n\n- [1] Sennrich et al, Improving Neural Machine Translation Models with Monolingual Data (ACL 2016)\n- [2] Wieting and Gimpel, ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (ACL 2018)\n- [3] Donahue et al, Enabling Language Models to Fill in the Blanks (ACL 2020)\n- [4] Gupta et al, A Deep Generative Framework for Paraphrase Generation (AAAI 2018)\n- [5] Wolf et al, Transformers: State-of-the-Art Natural Language Processing (EMNLP 2020, System Demonstrations)\n", "summary_of_the_paper": "To obtain an isotropic word embedding space, the authors proposed a new transformer-based autoencoder in which each word is represented as a normal distribution, and a variational loss is used to make it closer to an isotropic normal distribution. By carefully adjusting the weights of the variational loss (magnitude of variance), the learning converged, and an isotropic latent representation was obtained. They also tried to show some of the advantages of the proposed approach:\n1. The proposed model can variationally sample sentences with slightly different meanings from the input sentences.\n2. In some cases, the proposed model can generate a semantic \"interpolation\" between two given sentences.\n3. Solving sentence-level tasks using word representations obtained by the proposed model outperforms MiniBERT.", "summary_of_the_review": "I have no objection to the importance of the theme or the validity of the idea. However, almost all of the experiments did not show the superiority of the proposed method, and it would be difficult to accept it to ICLR, a leading conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3gdpaLhzI8h", "reviewer": "Reviewer_ZoL1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of the paper:\n1. The discussion on selecting proper $sigma$ and how it affects the training, provides a good hint on selection of the hyper-parameters.\n2. The variational sampling and interpolation is interesting.\n\nThe weakness of the paper:\n1. Some part of the paper is not clearly written, e.g. Table 3, for miniBERT, I guess they use a relative number ?\n2. The major flaw is the experiments. Although the paper mentioned it is not going to provide on-par performance with the SOTA BERT, it is promissing that since the small-sized model can achieve better performance, potentially also applies to larger models. This is a handwaving claim and cannot justify the much worse performance (Table 4) compared with those smaller models such as DistilBERT or TinyBERT [2] (also 4 layers). This makes it very hard to judge the contribution of this paper, as it does not obtain even a close-to-SOTA (e.g. for MRPC, 70 in this paper vs 85 in tinyBERT) performance. If comparable results could be obtained, e.g. both on 12-layer fully trained transformers, then I can believe the variational autoencoder layer would no hurt the performance.\n\nMinors:\n1. Bottom of page 2, before equation (4), where the reparameterized sample $z$ does not equal to $\\Sigma \\epsilon + \\mu$, but rather $L^T \\epsilon + \\mu$, where $L^T L = \\Sigma$.\n2. The isotropy talked in this paper is referring to the latent code $z$, rather than the hidden representations $h$ which is most common in literatures (e.g.  Ethayarajh, 2019)\n3. The idea is very similar to the variational information bottleneck, which is also used in language models e.g. [1], as a regularizer in model training/fine-tuning.\n\n[1] Mahabadi, R. K., Belinkov, Y., & Henderson, J. (2021). Variational Information Bottleneck for Effective Low-Resource Fine-Tuning. arXiv preprint arXiv:2106.05469.\n\n[2] Jiao, Xiaoqi, et al. \"TinyBERT: Distilling BERT for Natural Language Understanding.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings. 2020.", "summary_of_the_paper": "This paper proposes to add a variational loss for each token, in the transformer architecture, to increase isotropy of deep language models. To achieve stable training, the paper proposed to adjust $\\sigma$ of the prior Gaussian distribution, and obtain a smoother training curve. The paper evaluated the proposed method from different aspects, including variational sampling, interpolation, semantic textual similarity and semantic classification tasks.", "summary_of_the_review": "The idea is similar to variational information bottleneck. The main concern is the experiments, especially table 4, that is too far behind other state-of-the-art models that have a similar size. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0y-u4qTQru", "reviewer": "Reviewer_Fv5U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using a variational autoencoder to address the degeneracy of the representations of BERT-like models is sensible, intuitive, and appears to be novel. However, the paper contains significant weaknesses:\n\n- In the experiments, it is not clear that MiniBERT is the fairest baseline.\n    - Although the encoder of VAT may have the same number of parameters as the MiniBERT model, VAT also has a decoder during training. Therefore it can be argued that a more appropriate baseline would be a BERT model which has the same total number of parameters as the entire VAT model (decoder included).\n- Overall, the writing lacks clarity.\n    - Section 2, paragraph 3\n        - It is stated that the \"highly discrete latent space lacks completeness and is of limited use for advanced NLP applications\"? Are the authors able to provide justification for this? One could argue that BERT (+ variants) are \"over-complete\" autoencoders as described, and they have revolutionized NLP.\n    - Section 2.1, paragraph 3\n        - At the end of this paragraph, should it not be the Cholesky root of $\\Sigma$ which multiplies $\\epsilon$?\n    - Equations (2) and (4)\n        - The L2 loss notation (with the double vertical lines) can be misleading. What if $x$ is discrete? It would be clearer to say something along the lines of $x$ is distributed according to some distribution $p_{X}(\\cdot)$ and then $L_{REC} = -\\log p(dec(enc(x)))$ or $L_{REC} = -\\log p(dec(z))$.\n    - Section 3\n        - The paper would flow better with Section 3 appearing after Section 4 instead of before. As it currently stands, it is difficult for the reader to infer the differences between your work and the various prior methods referred to. This would be easier if your method had already been described.\n    - Section 4\n        - The authors don't ever explicitly define how the model is constructed / what the objective actually is. Based on Equation (4) and the ensuing description in Section 4, the reader is left to assume that the objective must be $\\sum_{t} \\mathbb{E}_{q(z_{1:T}|x_{1:T})}[\\log p(x_{t}|x_{1:t-1},z_{1:T})] - D_{KL}[q(z_{t}|x_{1:T}) || p(z_{t})]$ but this is never actually stated, nor is a graphical model provided.\n    - Section 4.1, paragraph 1\n        - The KL annealing suggested by Bowman et al. (2016) isn't intended to combat overfitting, but rather the so-called 'KL collapse' phenomenon. This is where the latent variable gets ignored because the model $p(x|z)$ is able to reconstruct $x$ while ignoring $z$, and so is able to set $q(z|x) = p(z)$ for all $x$ such that $KL[q(z|x)||p(x)] \\rightarrow 0$. The curves in Figure 2 only seem to start at iteration 50,000 but as shown, they don't indicate that the model is suffering from KL collapse. If this is this what happens in the experiments, the explanation for using KL annealing should be amended.\n    - Section 5.1, paragraph 5 \n        - As mentioned above, posterior collapse is when the regularization loss is low and reconstruction loss is low, not when regularization loss is low and reconstruction loss is high. It is not clear that the authors have understood this phenomenon.", "summary_of_the_paper": "This paper presents a variational autoencoder model for learning representations of tokens in a sequence of text. The authors build upon prior work which argues that the point-estimate representations learned by Transformer-based language models are degenerate, i.e. they only populate a narrow subspace. The authors address this problem by learning Gaussian-distributed representations (instead of point-estimates) which are regularized using a KL divergence loss. Empirical results on classification tasks are competitive with point-estimate baselines.", "summary_of_the_review": "The core idea presented in this paper is strong, but the lack of an appropriate baseline and the significant rewriting required make it unsuitable for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a variational auto transformer model (VAT), a VAE based on the transformer (encoder-decoder) architecture designed to provide isotropic representations by adding a token-level loss for isotropy. All the reviewers agree that this is a novel architecture with a valid and interesting goal behind it.

Reviewers varied somewhat on their impressions of the paper, but none were strongly positive on accepting it. I think the strongest and most aligned concerns were from reviewers ZoL1 and pcez. They both feel that the experiments do not convincingly demonstrate what is required. It would be good to better establish the success of variational sampling and the usefulness of isotropic representations. I would think that even a page of examples in the appendix, contrasting sampling by various methods, would add a lot of information to what is presented here. It would be even better to have experiments showing the relation between improved isotropy and improved task performance (suggested by j72L). Both reviewers are concerned about the small model and weak results and whether these results would extend to larger models that people actually use. While on the one hand, controlled comparisons are valuable, it is also true that people in NLP routinely like to see results on models of a reasonably competitive size. In practice, for 2019-2021, it seems that people regard having models of BERT-base size as the "reasonable" small size that they will accept and for which there is reasonably good performance and lots of available empirical results. Transformers directly trained with very few layers do not perform that well. Reviewer pcez is also concerned about the change of the data set in the MiniBERT comparison, which seems valid, and reviewer 5v5U is concerned about what's fair in terms of parameter counts.

This paper needs further work with larger and more careful experimental comparisons to meet the needed level of experimental rigor to be convincing. The authors were not able to iterate sufficiently quickly to achieve this during the ICLR reviewing period, so it seems best that the paper be rejected for now, and the authors look to subsequently submit a more developed version of this work.
Decision: Reject