Year: 2021

Abstract:
Representations from common pretrained language models have been shown to suffer from the degeneration problem, i.e. they occupy a narrow cone in latent space. This problem can be addressed by enforcing isotropy in latent space. In analogy to variational autoencoders, we suggest applying a token-level variational loss to a Transformer architecture and optimize the standard deviation of the prior distribution in the loss function as model parameter to increase isotropy. The resulting latent space is complete and interpretable: any given point is a valid embedding and can be decoded into text again. This allows for text manipulations directly in latent space. Surprisingly, features extracted at sentence-level show competitive results on benchmark classification tasks.

heading: 1 INTRODUCTION
text: Self-supervised, attention-based language models are prone to overfitting or memorizing input data, especially when trained on smaller datasets. Nevertheless, models such as BERT (Devlin et al., 2019), based on an encoder-only architecture and trained on very large datasets, exhibit state-of-theart performance on common natural language processing tasks. The resulting token representations of these models are distributed and highly contextual, but the latent space does not exhibit additional structural properties such as isotropy and resulting completeness (Ethayarajh, 2019).
We develop an autoencoder architecture that avoids overfitting by increasing latent space isotropy. At the same time, enforcing a complete latent space leads to a decoder that is able to interpret (i.e. decode) any given point in latent space. While the primary goal are tasks that require paraphrase generation, the resulting encoders surprisingly also perform on par in transfer tasks with smaller training setups.
Previous work has shown that isotropic latent representation, i.e. distributed across latent space instead of populating a narrow cone (Ethayarajh, 2019), can improve the performance of language models on transfer tasks. Gao et al. (2019) improve the performance of a Transformer-based model with increased isotropy by directly optimizing the cosine distance between the latent representations. Wang et al. (2020) suggest to control the singular value distribution of the output representation matrix and Li et al. (2020a) transform the original representation distribution into a Gaussian distribution through normalizing flows to increase the isotropy of the underlying models.
Similar to Gao et al. (2019), we propose to apply a regularizing constraint during training to enforce an isotropic and, even more, complete latent space. A common regularization network is the socalled Variational Autoencoder (VAE) (Kingma & Welling, 2014). VAE networks consist of an encoder that maps given input data not to a point in latent space but a distribution. A VAE’s decoder is required to successfully reconstruct the original input from samples of the latent distribution. Thus, the VAE is optimized to reconstruct the given input sequence whilst enforcing the latent distributions to match a given prior distribution.
We propose a Variational Auto-Transformer (VAT), a VAE based on a Transformer architecture, where the variational loss is computed at token level. We show that by enforcing a Gaussian distribution as latent prior, the latent token-level representations become more isotropic in comparison to models such as BERT. We introduce the prior distribution’s standard deviation as model parameter to optimize isotropy and balance the language generation variety against the network’s reconstruction ability. Besides isotropy, the completeness of the latent space is demonstrated. The pretrained
decoder can map back every point in latent space to a textual representation, be it actual encoded or synthetic, sampled latent points. This allows for text generation through ”variational sampling” and other manipulations such as interpolation directly in latent space. While the token-level representations can be used to generate or manipulate text, we also show that sentence-level representations, e.g. obtained through averaging, are suitable for sentence classification tasks.

heading: 2 BACKGROUND
text: An autoencoder is a neural network architecture consisting of an encoder enc, that maps any input x ∈ Rd to a point z = enc(x) in latent space, and a decoder dec, such that x ≈ dec(enc(x)) (Goodfellow et al., 2016). The chosen network architecture defines the families E and D of the encoder and decoder. During training of the autoencoder, the network parameters are optimized to find the optimal (enc∗, dec∗) pair that minimizes a given loss function L:
(enc∗, dec∗) = argmin (enc,dec)∈E×D (L) (1)
For the standard autoencoder with continuous inputs x ∈ Rd, the reconstruction loss L = LREC is:
LREC = ∥x− dec(enc(x))∥2 (2)
Over-complete autoencoder architectures with many degrees of freedom are prone to a particular form of overfitting: The encoder maps each data point x to an isolated point z in the latent space such that the decoder memorizes the lossless reconstruction of each of these codes. This highly discrete latent space lacks completeness and is of limited use for advanced NLP applications.

heading: 2.1 VARIATIONAL AUTOENCODER
text: The variational autoencoder (VAE) (Kingma & Welling, 2014) can be thought of as generative autoencoder. A VAE encodes an input data point as a distribution over the latent space by adding a regularization term to the loss function. The regularization term LREG assesses the KulbackLeibler divergence between the latent distribution and a standard Gaussian based on their means µ and covariances Σ:
LREG = −DKL [N(µ,Σ), N(0, I)] (3)
µ and Σ are either estimated using standard point estimators from the encoder outputs, or computed by learned functions such that µ = g(x) and Σ = h(x) with g ∈ G and h ∈ H , where G and H are families of network architectures. G and H can contribute to the desired properties of the latent representation by implicitly implementing dimensionality reduction or feature disentangling.
Decoding from the latent representation requires sampling from z ∼ N(µ,Σ). In order to enable backpropagation despite this sampling operation, Kingma and Welling (Kingma & Welling, 2014) introduced the reparameterization trick: Instead of sampling from the latent distribution, a random sample ϵ ∼ N(0, I) from a standard Gaussian is drawn and then transformed by the computed mean and standard deviation z = LT ϵ+ µ, where LT is the Cholesky factor of Σ.
The VAE’s total loss is the sum of the regularization and reconstruction term:
(enc∗, dec∗) = argmin (enc,dec)∈E×D
( LREC + LREG ) = argmin
(enc,dec)∈E×D
( ∥x− dec(z)∥2 −DKL [N(µ,Σ), N(0, I)] ) (4)

heading: 3 VARIATIONAL AUTO-TRANSFORMER
text: Initially proposed for machine translation, the Transformer (Vaswani et al., 2017) can be used as language model and framed as autoencoder. Extended with a token-level variational loss, the VAT
maps input tokens to context-aware, distributed latent representations. Figure 1 illustrates the architecture of the VAT. The set of encoders E and decoders D is defined by the Transformer’s network architecture based on self-attention modules. The encoder maps the embedded input sequence X = {x1,x2, . . . ,xT }, X ∈ RT×d to a latent representation Z = {z1, z2, . . . ,zT }, Z ∈ RT×d, where T is the sequence length and d is the model dimension. To do so, the model predicts mut and σt from xt and samples zt ∼ N(µt,σt). Z is then used as attention source in the decoder to predict the next word based on the already produced output. As X and Z depend on T , the objective is to minimize
L = 1 T T∑ t=1 ( Eq(Z|X)∥xt − dec(zt)∥2 −DKL [q(zt|X), p(zt)] ) (5)
where p(zt) ∼ N(0, I) and q(zt|X) ∼ N(µt,Σt). More precisely, we let Σt = diag (expσt) where the vector σt = (σj)j=1...d is predicted, so that computing the actual value of Σ involves the exponential as a non-linear activation function.
The output of the encoder is passed to two linear layers in parallel. The linear layers have the same number of input and output nodes and learn a transformation to predict µt and σt for each token. The latent representation Z is used as input in the encoder-decoder attention layer of the Transformer’s decoder.

heading: 3.1 SCALING THE REGULARIZATION LOSS
text: During our experiments, we experienced overfitting on the training data for the reconstruction loss. The learning curves of a corresponding experiment are illustrated in Figure 2a. Weighting the regularization loss according to a logistic annealing function as suggested by Bowman et al. (2016) or by a scaling factor β similar to the scaling of the beta-VAE (Higgins et al., 2017), but with β < 1, only had little effect.
Thus, we propose to scale the covariance matrix of the target distribution of LREG instead. This is motivated by the observation that standard Gaussians in d-dimensional latent spaces (e.g. d ≥ 128) will overlap considerably if their mean values are - at the same time - regularized to zero. As a consequence, LREG cannot be minimized by the model without increasing LREC in an undesirable way. Scaling the standard deviation to a too small value, though, might result in peaky distributions, having no regulatory effect and resulting in a trivial reconstruction objective.
We adapt the computation of the regularization loss to incorporate a scaling factor s for the standard deviation as hyperparameter. The closed form of the VAE loss function1 with a prior standard deviation σp = s ∈ (0, 1] and prior µp = 0 for a specific token’s layer outputs σt and µt becomes
−DKL [N(µt,σt), N(0, diag(s))] = d∑
i=1
1 2 log(σ2t,i)−
σ2t,i + µ 2 t,i
2s2 − log(s) + 1 2 (6)
This results in codes zt = σt · ϵ′t + µ being distributed according to a Gaussian with zero mean and a predefined standard deviation, ϵ′ ∼ N(0, diag(s)). The optimal balance between LREC and LREG as expressed by the value of s ∈ (0, 1] is related to the representations’ isotropy and will be determined experimentally.

heading: 4 EXPERIMENTS
text: The aim of the experiments is to balance the two loss terms. LREC should be low to maintain the reconstruction ability of the network, while a too small scaling value s implies that the VAT essentially behaves as regular AE. At the same time, we want to optimize the isotropy of the latent representations. In order to choose the best setting, we observe both, the resulting loss values and the properties of the latent space in terms of similarity between representations.
Our VAT model architecture is smaller than the original Transformer architecture, as it is not intended for machine translation. The VAT consists of N = 4 encoder and decoder layers, respectively, with H = 8 attention heads each and a model dimensionality of d = 128. A logistic annealing function (Bowman et al., 2016) is applied to weight LREG. The VAT is trained on the train split of the WMT19 de-en dataset (Barrault et al., 2019) using English sentences only. WMT19 contains data from the news commentary, Wiki titles, Europarl, ParaCrawl and Common Crawl corpora. The data is tokenized using subword tokenization with a target vocabulary of 215 tokens. The full details on model and training parameters are listed in the appendix.

heading: 4.1 OPTIMAL SCALING PARAMETER
text: The isotropy of the learned representations z is assessed as a function of the scaling factor s ∈ (0, 1]. Ethayarajh (2019) introduced the notion of isotropy of the latent space as the mean cosine similarity between vector representations of random tokens. In an isotropic latent space, representations of randomly sampled tokens have low cosine similarity and do not cluster in a specific direction.
1Refer to Odaibo (2019) for the derivation of the closed form.
In addition to isotropy, Ethayarajh (2019) introduces the notion of self-similarity and intra-sentence similarity. A high self-similarity, i.e. mean cosine similarity between representations of the same token in different sentences, indicates that neighboring representations capture similar concepts which contributes to the smoothness of the latent space. Intra-sentence similarity measures the mean cosine similarity of tokens occurring in the same sentence to the mean sentence representation, thus being related to contextuality.
BERT exhibits low isotropy (mean cosine similarity > 0.4) in its last layers (Ethayarajh, 2019), a phenomenon known as representation degeneration (Gao et al., 2019). In contrast to BERT with high contextuality with respect to the low isotropy, we expect a more complete latent space for the VAT and thus high isotropy. The results in Figure 3a2 are computed for the STS12 datasets Agirre et al. (2012). The average cosine similarity between randomly sampled words is low for smaller s values, suggesting an isotropic latent space with representations distributed across space. The latent space is most isotropic at s = 0.4.
The average self-similarity of tokens greater than 0.6 suggests a mapping of similar concepts into the same region, indicating smoothness in the latent space. The intra-sentence similarity for 0.5 ≥ s ≥ 0.1 is lower than the self-similarity, but still above the similarity of random words. This indicates that contextual information is captured in the representations. Interestingly, for 1 ≥ s ≥ 0.6, the token representations seem to ”collapse” into an identical representation for all positions in a sequence resulting in an intra-sentence similarity of 1. Among the words with most context-specific representations are mainly stopwords as they have lowest self-similarity.
In Figure 3b, we report the final loss values after training the models for 3 epochs with different scaling factors s ∈ (0, 1]. Scaling values 1 ≥ s ≥ 0.6 result in near zero regularization losses and high reconstruction error. The original input cannot be reproduces any more. This observation corresponds to the findings from Figure 3a, where the representations within a sentence are identical. Starting from s = 0.5, the reconstruction loss drops and with s = 0.4, the reconstruction loss falls below the regularization loss. Values smaller than s = 0.4 further decrease the positive effects of the variational model, as the achievable variety of generated text is reduced. This phenomenon is referred to as posterior collapse (Lucas et al., 2019).
Combining the findings from evaluating the loss and latent space properties in Figure 3, we select s = 0.4 as scaling factor for further evaluating the VAT. This value corresponds to good reconstruction ability and isotropic representations which impacts both the generation of text as well as the performance on classification tasks. Also, the learning curves corresponding to s = 0.4 exhibit no overfitting any more (see Figure 2b).
2Different from Ethayarajh (2019), Figure 3a depict the plain intra-sentence and self similarity, i.e. not adjusted for isotropy.
In this class, we will introduce several fundamental concepts. In this class, we will introduce several fundamental concepts.
In this class, we will introduce several fundamental principles. In this area, we will introduce several fundamental principles. In this process, we will introduce several educational concepts. In this class, I will introduce several fundamental principles. In this progress, we will introduce private fundamental concepts. In this class, we have received several fundamental concepts. In this class, we will introduce several public projects. In April 2013, we will introduce several fundamental issues. In this class, we will find several fundamental principles. In one class, she will introduce several fundamental concepts.
Table 1: Variational Sampling. Example sentence (in bold) was used as input to the VAT, with the decoded mean representation and samples from the latent distribution (in italics).
Generative models have shown great promise in modeling complex distributions. advantages, results, achievements, quality, answers participation, moments, experience, passion, joy, benefits, interests
Table 2: Variational Sampling. Example sentence (in bold) that was used as input to the VAT, with only the underlined token being sampled. Samples from the latent distribution in italics.
I want to talk to you. I want to report. I said: She didn’t want to say. She didn’t want to be with him. He was silent for a long moment. He was silent for a long moment. He was my father. It was my face. It was my turn.
Table 3: Interpolation. First and last sentence (in bold) were given as input to the VAT. Intermediate sentences were obtained by token-wise linear interpolation between the two latent representations padded to the same length.

heading: 5 VARIATIONAL LANGUAGE MODEL
text: During training, the VAT encoder produces variants of the input tokens by drawing from the latent distribution. The VAT decoder has to be able to reproduce the original sequence given variational input. At test time, both approaches are possible: Decoding with the standard deviation set to 0, i.e. zt = µt, leads to a deterministic representation and is used for classification tasks. For sequence generation, decoding zt = σt · ϵ′ + µt allows to generate variants of the original input sequence, a process we refer to as variational sampling. Variational language modeling thus denotes the various ways of manipulations and computations in the latent space that are possible with the VAT’s latent distributions. As examples, we discuss anecdotal results from variational sampling and interpolation.
Tables 1 and 2 illustrate the generating capabilities of the VAT with variational sampling. In the second line in Table 1, the VAT is able to reconstruct the original sequence (first line), which refers to decoding with zt = µt, i.e. without sampling. The following lines show exemplary variants of the input sentence obtained by randomizing the latent representations, displaying 12 variational samples. It is interesting that, while mostly maintaining the original sentence structure and original context, variants are found for almost all tokens. This gives an idea of the structure of the latent space and the contextuality of the representations. The approach is comparable to paraphrase generation, which is often obtained through back-translation (Sennrich et al., 2016), (Wieting & Gimpel, 2018) or by directly trained on supervised data in the form of paraphrase sentence pairs (Gupta et al., 2018).
Table 2 illustrates variational sampling for a single token, where the input sequence (in bold) is reconstructed according to zt = µt except for the underlined token for which sampling is allowed. This setting is similar to a nearest neighbor search over a fixed corpus. The second line lists the obtained samples. By masking the underlined token, this approach is similar to gap filling (Donahue et al., 2020), (Wu et al., 2019). Variational sampling, both for tokens and sequences, can be useful for generating paraphrases in tasks such as dialog generation, question answering etc.
We also experimented with latent space interpolation similar to (Bowman et al., 2016), (Liu & Liu, 2019), (Li et al., 2020b). We linearly interpolate the latent representations of two sentences (padded
to the same length) with three intermediate steps. Decoding the intermediate representations results in the sentences illustrated in Table 3 for two examples. Observing a smooth interpolation trajectory as in the examples is not always the case. It is possible that after decoding, the intermediate steps are mapped to the same sentence as seen for one sentence in the second example. This is especially true with increasing size of training data: The more training data, the more contextualized representations for the same word type that are mapped close to each other in latent space will exist.

heading: 6 SENTENCE REPRESENTATIONS
text: For the construction of sentence representations, we compare two pooling operations (average, sum). Additionally, the start token (”CLS”) representation can be used as embedding for the whole sentence, but this approach has been shown to be inferior to mean pooling in semantic similarity tasks for BERT (Reimers & Gurevych, 2019). We denote the different approaches as VAT-mean, VATsum and VAT-start, respectively. We compare our model to the original BERT model (base) and, for a fair comparison in terms of model size, to a smaller variant with similar network dimensions than the VAT. BERT has L = 12 encoder layers and the model and representation dimension is d = 768. The smaller BERT model3 (Turc et al., 2019), which we refer to as MiniBERT, has d = 128 and L = 4, the same size as the VAT. Both BERT and MiniBERT were trained on a much larger collection of datasets (Wikipedia, BookCorpus, CommonCrawl) than the VAT.
The sentence representations are tested using the SentEval (Conneau & Kiela, 2018) toolkit4. The toolkit evaluates static sentence representations on two different classes of tasks: semantic similarity and sentence classification tasks. Table 4 lists the correlation values of the different sentence representation methods on the semantic textual similarity (STS) tasks. For each pair of semantically similar sentences, the spearman correlation rank between the cosine of their latent representations and a human-labeled gold standard (between 0 and 5) is computed (no fine-tuning or transfer learning). The correlation of VAT-based sentence representations is on par with BERT-based representations which is surprising given the smaller model architecture and capacity of the VAT. For both VAT and BERT, start token based representations show less correlation than those obtained by average or sum pooling. Representations obtained from the MiniBERT model do not show any correlation at all, indicating that the BERT architecture requires a larger model size.
The classification tasks included in the SentEval toolkit comprise binary (MR, CR, SST2) and finegrained (SST5) sentiment or polarity classification tasks, paraphrase detection (MRPC), natural language inference (SICK-E) and question-type classification (TREC) tasks. As paraphrase detection and natural language inference require the comparison of two sentences, the pair of input sentences is concatenated and separated by a special separation token to form a single input vector. For the classification tasks, we compare the performance of the VAT to that of MiniBERT in two ways: in a feature-based approach and in a fine-tuning approach.
3https://github.com/google-research/bert 4https://github.com/facebookresearch/SentEval
Both presented approaches are different from how state-of-the-art models are trained for transfer tasks. Given the smaller model capacity, we do not expect on-par performance, but rather want to better understand the kind of information stored in the representations and the potential that comes with additional classification layers on top of the model. The comparison to MiniBERT serves as baseline.
For the feature-based approach, the sentence representations are extracted from the pretrained models and then passed to a single classification layer trained with an Adam optimizer (Kingma & Ba, 2015). Given the results in Table 5 (top), the sum of the individual token representations is best suited as standalone feature for sentence classification. As VAT-sum outperforms VAT-mean on all tasks, the dimensions of the latent representations seem to be well disentangled. With the exception of the SST2 task, the start representation yields lowest accuracy values suggesting that the first representations does not capture the full context of a sentence. MiniBERT, which is designed for fine-tuning rather than feature extraction, mostly does not reach the performance of VAT variants. The difference between MiniBERT and VAT is especially large for the TREC task.
For assessing the performance with a fine-tuning approach, we extend the underlying models by a single non-linear dense classification layer and optimize with Adam. For the VAT, we only reuse the encoder part to access the latent representations Z. For each task, we trained for five epochs Note that the MiniBERT results can differ from the results in the original publication (Turc et al., 2019), as we do not further tune the model or the training process.
In Table 5 (bottom), we see a performance leap for all of the models compared to their feature-based results. While VAT-sum was performing best for the feature-based approach, the method to produce sentence-level representations is not that decisive for the fine-tuning approach. The differences between sum, mean and start are not as great as for the semantic similarity tasks any more, neither for MiniBERT nor for VAT. It is interesting that MiniBERT is in lead for sentiment classification tasks, whereas VAT shows outstanding performance on the TREC (topic classification) task.

heading: 7 RELATED WORK
text: The architecture of variational autoencoders has been optimized for various natural language processing tasks by implementing various kinds of networks for encoder and decoder. The variational loss is applied to a sentence-level latent representation in all these models. Some of the networks do not contain a decoder any more, such that the latent representations are not interpretable.
Several models were suggested to solve common document or text classification tasks: Gururangan et al. (2019) use MLPs as encoder and decoder to learn latent vectors from bag-of-words inputs that are used for document classification. Mahabadi et al. (2021) compress BERT-based sentence
representations through a Gaussian prior distribution for transfer classification tasks. Deudon (2018) implements a siamese network architecture based on sentence representations from a VAE with BiLSTMs for semantic similarity classification.
Zhao et al. (2018) use a LSTM-based VAE for dialog generation. Miao et al. (2016) rely on MLPs in a VAE for both document modeling and answer selection. Wang & Wan (2019) introduce a Transformer-based VAE for story completion, where the missing plot is conditioned on the latent representation z capturing the distribution of the story plot. Shu et al. (2020) propose a Transformerbased non-autoregressive model specifically for machine translation that incorporates a predictor network for the length of the target sentence.
Various network architectures have been proposed for the more general task of language modeling. Bowman et al. (2016) apply the variational loss on the last hidden state of a LSTM sequence-tosequence model. In iVAE (Fang et al., 2019), an MLP produces sample-based distributions from an LSTM’s hidden representation concatenated to random noise. Yang et al. (2017) apply an LSTM as encoder and a CNN as decoder in a language model. Liu & Liu (2019) add a feed-forward layer to a Transformer model to map the encoder outputs to a mean and variance vector which are then upsampled and passed to the decoder followed by an LSTM layer. OPTIMUS (Li et al., 2020b) extends this idea to large-scale language models. The encoder weights are initialized with pre-trained BERT weights, the decoder with those from a pre-trained GPT-2 model. The latent representation of the start token is used as sentence-level representation.
In contrast to these networks, the VAT’s variational loss is applied at token-level which allows for direct manipulations in the latent space. Token-level reguralisation has been previously applied to RNNs for language modeling (Li et al., 2020c). While language modeling is the training objective of the VAT, we also investigate the performance of the trained model on text classification tasks.

heading: 8 DISCUSSION AND CONCLUSION
text: The goal of the proposed VAT model is to obtain isotropic representations mapped to a smooth and complete latent space. The hyperparameter and scaling factor s is directly optimized to meet these criteria. The effect can be seen especially for text generation, where the VAT is able to produce coherent sequences through manipulations directly in latent space. However, optimizing LREG comes at the cost of LREC . Perfect reconstruction of the original input is not possible at all times any more.
Reducing the representations to a single sentence-level vector preserves contextual information. Semantic similarity tasks demonstrate that the smaller model capacity of the VAT is sufficient to capture semantic information that is on a par with larger BERT architectures. However, models explicitly trained to improve the performance on semantic similarity tasks are out of reach.
The VAT is able to produce more robust standalone sentence-level features when compared with MiniBERT for the feature-based classification approach. VAT outperforms the similar-sized MiniBERT also when being fine-tuned on sentence classification tasks, but is far from the state-of-the-art performance known from BERT-sized models with more sophisticated training procedures. Interestingly, VAT shows its peak performance for the topic classification task TREC. In our setup, the VAT captures topic-related information even better than sentiment.
Tasks involving the comparison of two sentences (MRPC, SICK-E), represented by a single concatenated vector, cannot be successfully solved by either VAT or MiniBERT. Some model variants (MiniBERT included) only learn to predict the most frequent class. The reason could be the smaller model sizes that are incapable of capturing the information of two concatenated sentences, as especially the VAT was never trained on such a setting. Instead of a concatenated vector, the comparison task could be solved using a siamese network architecture, similar to Deudon (2018).Optimizing the regularization loss at token-level could even allow the training of a multilingual model or a translation model, given sufficient training data.
REPRODUCIBILITY STATEMENT
All data used for training and evaluation of the model are publicly available. The WMT19 dataset was obtained through the tensorflow library at https://www.tensorflow.org/ datasets/catalog/wmt19_translate, all other datasets are available through the SentEval toolkit at https://github.com/facebookresearch/SentEval. The source code for instantiating and training our model as well as for decoding and generating variational samples is made available as supplementary material. Hyperparameters and training settings are described in Section 4 and – in more detail – in Appendix A.

heading: A APPENDIX
text: MODEL DETAILS
The implemented VAT consists of N = 4 encoder and decoder layers, respectively, with H = 8 attention heads each. The encoder and decoder architecture including the positional embedding is the same as in the original Transformer (Vaswani et al., 2017) and can be seen in Figure 1. The model dimensionality d = 128, the dimension in the feed forward layers is 512. Dropout during training is set to 0.1. The Noam optimizer Vaswani et al. (2017) operates with 50, 000 warmup steps. A logistic annealing function Bowman et al. (2016) with 50, 000 warmup steps and an initial value of 0.00025 is applied to weight the regularization term of the loss function.
The VAT is trained on the train split of the WMT19 de-en dataset Barrault et al. (2019) using English sentences only. The data is tokenized using subword tokenization with a target vocabulary of 215 tokens. Batch size is set to 128. Decoded sentences that are presented as results are obtained through beam search with beam size 5.
ENERGY CONSUMPTION
Experiments were conducted using a private infrastructure, which has an estimated carbon efficiency of 0.197 kgCO2eq/kWh. A single epoch takes 20 hours of computation on a RTX 2080 Ti (TDP of 250W) GPU. For the training of a VAT model for 3 epochs, total emissions are estimated to be 0.99 kgCO2eq according to the ML impact calculator Lacoste et al. (2019) available at:
https://mlco2.github.io/impact#compute.

Review summaries:
To obtain an isotropic word embedding space, the authors proposed a new transformer-based autoencoder in which each word is represented as a normal distribution, and a variational loss is used to make it closer to an isotropic normal distribution. By carefully adjusting the weights of the variational loss (magnitude of variance), the learning converged, and an isotropic latent representation was obtained. They also tried to show some of the advantages of the proposed approach: 1. The proposed model can variationally sample sentences with slightly different meanings from the input sentences. 2. In some cases, the proposed model can generate a semantic "interpolation" between two given sentences. 3. Solving sentence-level tasks using word representations obtained by the proposed model outperforms MiniBERT.

Provide a brief summary of the paper and its contributions.  This paper proposes Variational Auto-Transformer to encourage isotropy in the latent representation space. The resulting encoder-decoder architecture allows interpretable embeddings. On various tasks, VAT shows better performances than other language models.  Contributions:  - A novel architecture based on Transformer with a token-level variational loss. - Extensive evaluations on the effectiveness of Transformer.

This paper proposes to add a variational loss for each token, in the transformer architecture, to increase isotropy of deep language models. To achieve stable training, the paper proposed to adjust $\sigma$ of the prior Gaussian distribution, and obtain a smoother training curve. The paper evaluated the proposed method from different aspects, including variational sampling, interpolation, semantic textual similarity and semantic classification tasks.

This paper presents a variational autoencoder model for learning representations of tokens in a sequence of text. The authors build upon prior work which argues that the point-estimate representations learned by Transformer-based language models are degenerate, i.e. they only populate a narrow subspace. The authors address this problem by learning Gaussian-distributed representations (instead of point-estimates) which are regularized using a KL divergence loss. Empirical results on classification tasks are competitive with point-estimate baselines.