Title: COUPLE-VAE: MITIGATING THE ENCODER-DECODER INCOMPATIBILITY IN VARIATIONAL TEXT MODELING WITH COUPLED DETERMINISTIC NETWORKS

Abstract: The variational autoencoder (VAE) combines latent variable models and amortized variational inference. Despite its theoretical attractiveness, the optimization of VAE for text modeling suffers from the posterior collapse problem, where the decoder ignores the latent codes, and the posterior becomes nearly identical to the prior. We demonstrate that the VAE training dynamics face the challenge of encoder-decoder incompatibility, in which the encoder receives scarce backpropagated gradients from the decoder, and little encoded information is passed to the decoder. We propose a model-agnostic approach, named Couple-VAE, to mitigate this issue. Specifically, we couple the VAE model with a deterministic network with the same structure, which is optimized with the reconstruction loss without any regularization (e.g., the KL divergence). To enrich the backpropagated gradients for the encoder, we share the encoder between the deterministic network and the stochastic network. To encourage nontrivial decoding signals, we propose a coupling loss that pushes the stochastic decoding signals to the deterministic ones. We conduct extensive experiments on the Penn Treebank, Yelp, and Yahoo. We apply the proposed method to various variational text modeling models with different regularization terms, posterior families, decoder architectures, and optimization strategies and observe consistently improved text modeling results in terms of probability estimation and the richness of the encoded text. 1

1. INTRODUCTION:

The variational autoencoder (VAE) (Kingma & Welling, 2014 ) is a generative model that combines neural latent variables and amortized variational inference, which is capable of both estimating and sampling from the data distribution. It infers a posterior distribution for each instance with a shared inference network and optimizes the evidence lower bound (ELBO) instead of the intractable marginal log-likelihood. Given its potential to learn representations from massive text data, there has been much interest in using VAE for text modeling (Xu & Durrett, 2018; He et al., 2019) .

Prior work has observed that the optimization of VAE suffers from the posterior collapse problem, i.e., the posterior becomes nearly identical to the prior and the decoder degenerate into a standard language model (Bowman et al., 2016; Zhao et al., 2017) . A widely mentioned explanation is that a strong decoder makes the collapsed posterior a good local optimum of ELBO, and existing solutions include weakened decoders (Yang et al., 2017; Semeniuta et al., 2017) , modified regularization terms (Higgins et al., 2017; Wang & Wang, 2019) , alternative posterior families (Rezende & Mohamed, 2015; Davidson et al., 2018) , richer prior distributions (Tomczak & Welling, 2018) , improved optimization strategies (He et al., 2019) , and narrowed amortization gaps (Kim et al., 2018) .

In this paper, we provide a new perspective on the posterior collapse problem. In Section 3, by tracking the gradient norms of multiple model components w.r.t. the encoded text, we demonstrate the encoder-decoder incompatibility issue in both the forward pass and the backpropagation of VAE training, which leads to a poorly expressive encoder and an over-expressive decoder. To mitigate this issue, we propose a model-agnostic approach in Section 4, named Couple-VAE. We couple the VAE model with a deterministic network with the same structure, which is optimized purely with the reconstruction loss without any regularization. To propagate richer gradients into the encoder, we share the encoder between the coupled networks. To learn nontrivial decoding signals, we propose a coupling loss that pushes the stochastic decoding signals to the deterministic decoding signals. Since our approach does not make any assumption on the regularization term, the posterior family, the decoder architecture, or the optimization strategy, we apply our method to various VAE-based models for text modeling. Experimental results on the Penn Treebank, Yelp, and Yahoo show that the proposed method consistently improves the performance of various VAE-based models in terms of probability estimation and the richness of the encoded information.

2. BACKGROUND:



2.1. VARIATIONAL INFERENCE FOR TEXT MODELING:

VAE adopts a two-step view of the generative process of text data. It first samples the latent code z from the prior distribution P(z) and then samples the text x from the generator P (x|z; θ). By marginalizing out the latent variable, the likelihood of each datapoint is derived as

P (x; θ) = E z∼P(z) [P (x|z; θ)]

(1) As maximizing the log-likelihood with exact marginalization is usually intractable, VAE uses a variational family of posterior distributions Q(z|x; φ) and derives the evidence lower bound (ELBO)

EQUATION

For training, as shown in Figure 1 (a), the encoded text is transformed into posterior via a posterior network. A low-dimensional latent code is sampled from the posterior and then transformed into the decoding signal h with an MLP. Finally, the decoder infers the input with the decoding signal. The VAE objective can be generally interpreted as a reconstruction loss L rec plus a regularization loss L reg , whose concrete forms can be modified, i.e., L = L rec + L reg (3) However, the optimization of the VAE objective is challenging. We usually observe a very small L reg and a L rec similar to a standard language model, i.e., the well-known posterior collapse problem.

2.2. DETERMINISTIC AUTOENCODERS:

Our new perspective on the posterior collapse problem is based on the comparison between VAE and the deterministic autoencoder (DAE, or simply, autoencoders) (Rumelhart et al., 1986; Ballard, 1987) . Figure 1 (b) shows a graphical overview of DAE for text modeling, which is composed of a text encoder, an optional MLP, and a text decoder. The objective for DAE is the reconstruction loss, which is empirically usually much lower than that of VAE after convergence.

3. ENCODER-DECODER INCOMPATIBILITY IN VAE FOR TEXT MODELING:

To understand the posterior collapse problem, we take a deeper look into the training dynamics of VAE. In this part, we investigate the following questions. How much backpropagated gradient does the encoder receive from reconstruction? How much does it receive from regularization? How much information does the decoder receive from the encoded text?

3.1. TRACKING TRAINING DYNAMICS:

To answer the first question, we study the gradient norm of the reconstruction loss w.r.t. the encoded text, i.e., ∂L rec /∂e 2 , which according to the chain rule is the amount of backpropagated gradient received by the encoder parameters. From Figure 2 (a), we observe that for DAE it constantly increases, while for VAE it increases marginally in the early stage and then decreases continuously. It shows that the reconstruction loss actively optimizes the DAE encoder throughout the training phase, while the VAE encoder lacks backpropagated gradients after the early stage of training.

We seek the answer to the second question by studying the gradient norm of the regularization loss w.r.t. the encoded text, i.e., ∂L reg /∂e 2 . The parameters of the posterior network consist of both weights and bias, and a larger gradient norm shows that the model relies more on the weights than the bias to optimize L reg . Figure 2 (b) shows a constant decrease of the gradient norm in VAE from the 2.5K step until convergence, which shows that the posterior collapse is aggravated as the KL weight increases (KL annealing is applied from step 1K to 41K).

For the third question, we compute the normalized gradient norm of the decoding signal w.r.t. the encoded text, i.e., ∂h/∂e F / h 2 . As this term shows how relatively the decoding signal changes with the perturbation of the encoded text, it reflects the amount of information passed from the encoder to the decoder. Figure 2(c) shows that for DAE it constantly increases. For VAE, it at first increases even faster than DAE, then slows down, and finally decreases until convergence, indicating that the VAE decoder to some extent ignores the encoder in the late stage of training.

3.2. ENCODER-DECODER INCOMPATIBILITY:

The above observations indicate an incompatibility between the encoder and the decoder in VAE training. We argue that this incompatibility is caused by the regularization on the posterior, which does not allow much encoded information to pass through and thus forces the decoder to exploit its expressive power. Thus, the decoder is usually over-expressive after convergence. The failure in using the encoded information, in turn, results in the scare backpropagated gradients from the reconstruction, resulting in a poorly expressive encoder. The poorly expressive encoder then encourages the posterior network to rely more on the bias rather than the weights, as discussed in Section 3.1.

4. COUPLING VARIATIONAL TEXT MODELING WITH DETERMINISTIC NETWORKS:

To mitigate the encoder-decoder incompatibility issue, we propose to couple the VAE model with a deterministic network, displayed in which is mapped to the coupled decoding signal h c by MLP c . The coupled decoder Decoder c then infers the text with h c . Whenever sampling is applied in the stochastic network, we use the predicted mean vector for the deterministic network, e.g., for the Gaussian posterior, we simply discard the predicted standard deviation vector and use the predicted mean vector for later computation. Please find details for other posterior families in Appendix B. Similar to DAE, the coupled deterministic network is optimized solely by the coupled reconstruction loss L c rec , which is the same autoregressive cross-entropy loss as L rec . We share the encoder between the stochastic network and the deterministic network, which enriches the reconstruction gradients backpropagated to the encoder (compared with VAE) by leveraging the coupled reconstruction loss. To encourage nontrivial decoding signals, we propose a coupling loss L couple that pushes the stochastic decoding signals to the deterministic ones. Formally, the objective of our approach is given as

EQUATION

where λ c is a hyperparameter, L c rec is the coupled reconstruction loss, and the coupling loss L couple is essentially a distance metric between h and h c . Since the decoding signals are usually not distributed in a Euclidean space, we adopt the Rational Quadratic kernel at multiple scales, i.e.,

L couple = - s (1 + h -Detach(h c ) 2 s • C ) -1 (5)

where C is a hyper-parameter, s ∈ {0.1, 0.2, 0.5, 1, 2, 5, 10}, and Detach(h c ) prevents gradients to be propagated into h c since we would like h c to guide h but not the opposite. In Section 5.5, we show how Couple-VAE has improved training dynamics compared with VAE.

One would resort to the universal approximation theorem (Hornik et al., 1989) and question the necessity to share the module structures between the deterministic network and the stochastic network. Indeed, implementing the deterministic network as an MLP is theoretically adequate. However, we argue that every structure has its favored geometry of the learned manifold, on which the decoding signals are distributed. For example, the latent space learned by planar normalizing flows (Rezende & Mohamed, 2015) , which has compression and expansion, and vMF-VAE (Xu & Durrett, 2018; Davidson et al., 2018) , which is supported on a (d -1)-dimensional sphere in R d , may significantly influence the learned manifold of decoding signals.

5. EXPERIMENTS:



5.1. EXPERIMENTAL SETUP:

We conduct the experiments on three commonly used datasets for text modeling, i.e., the Penn Treebank (PTB) (Marcus et al., 1993) , Yelp (Xu et al., 2016) (He et al., 2019) . We also show the result of GRU-LM (Cho et al., 2014) and SA-VAE (Kim et al., 2018) . We do not apply our method to SA-VAE since it does not follow the amortized variational inference framework. We use the setups and open-source implementations of SA-VAE and Lagging-VAE. Please find more details in Appendix C.2 

5.2. LANGUAGE MODELING RESULTS:

We report negative log-likelihood (NLL), KL divergence, and perplexity as the metrics for language modeling. NLL is estimated with importance sampling, KL is approximated by its Monte Carlo estimate, and perplexity is computed based on NLL. Please find the metric details in Appendix D.

Table 1 shows the language modeling results. We find that for all models, our proposed approach achieves smaller negative log-likelihood and lower perplexity, which shows the effectiveness of our approach to improve the probability estimation capability of various VAE-based models. Larger KL divergence is also observed, showing that our approach helps address the posterior collapse problem.

5.3. MUTUAL INFORMATION AND RECONSTRUCTION:

Language modeling results only evaluate the probability estimation ability of VAE. We are also interested in how rich the latent space is. We report the mutual information (MI) between the text x and the latent code z under Q(z|x), which is approximated with Monte Carlo estimation. Better reconstruction from the encoded text is another way to show the richness of the latent space. For each text x, we sample ten latent codes from Q(z|x) and decode them with greedy search. We report the BLEU-1 and BLEU-2 scores between the reconstruction and the input. Please find the metric details in Appendix E. In Table 2 , we observe that our approach improves MI on all datasets, showing that our approach helps learn a richer latent space. BLEU-1 and BLEU-2 are consistently improved on Yelp and Yahoo, but not on PTB. Given that text samples in PTB are significantly shorter than those in Yelp and Yahoo, we conjecture that it is easier for the decoder to reconstruct on PTB by exploiting its autoregressive expressiveness, even without less rich latent codes.

5.4. ANALYSIS OF THE COUPLING WEIGHT:

We investigate the model performance with different coupling weights, shown in Table 3 . With larger coupling weight, the model achieves higher KL divergence, MI, and reconstruction metrics. It shows that by pushing the stochastic decoding signals closer to the deterministic decoding signals, we get more complex posterior distribution and latent codes that contains richer text information. Note that the best NLL does not guarantee the highest other metrics, which justifies the necessity to evaluate VAE models with multiple metrics. 

5.5. ANALYSIS OF GRADIENT NORMS:

We study the three gradient norms defined in Section 3. Table 4 displays the gradient norms the models reported in the main experiments. Notably, ∂L c rec /∂e 2 in Couple-VAE is even larger than ∂L rec /∂e 2 in DAE. It has two indications. First, the encoder indeed encodes rich information of the text. Second, compared with DAE, Couple-VAE better generalizes to the test sets, which we conjecture is due to the regularization on the posterior. Couple-VAE also has a larger ∂L reg /∂e 2 compared with VAE, which based on the argument in Section 3 indicates that Couple-VAE relies on the weights (which learn a better aggregated posterior) than the bias (which leads to a collapsed posterior) of the posterior network. We also observe larger ∂h/∂e F / h 2 in Couple-VAE, which indicates that the decoder in Couple-VAE uses more encoded information than in VAE.

To show how Couple-VAE ameliorates the training dynamics, we also track the gradient norms of Couple-VAE (with λ c = 10.0 for a clearer comparison), plotted along with VAE and DAE in Figure 2 . The curve for Couple-VAE in Figure 2 (a) stands for ∂(L rec + L c rec )/∂e 2 . Please find the plots for more datasets in Appendix F. We observe that Couple-VAE receives constantly increasing backpropagated gradients from the reconstruction. In contrast to VAE, the ∂L reg /∂e 2 in Couple-VAE does not decrease significantly as the KL weight increases. The decrease of ∂h/∂e F / h 2 , which VAE suffers from, is not observed in Couple-VAE.

5.6. DIVERSITY AND SAMPLES FROM THE PRIOR DISTRIBUTION:

Given the generative nature of VAE, we evaluate the diversity and the quality of samples from the prior distribution. For diversity, we sample 3200 texts from the prior and report the Dist-1 and Dist-2 metrics (Li et al., 2016) , which are the ratios of distinct unigrams and bigrams over all generated unigrams and bigrams. For quality, we provide the first three texts sampled from each model. Table 5 displays the diversity metrics and the sampled texts on Yelp, and results on other datasets are in Appendix G. Dist-1 and Dist-2 show that texts sampled from Couple-VAE are more diverse than those from VAE. Qualitatively, we observe that the long texts generated from VAE have more redundancies compared with Couple-VAE. Given that both models have the same latent dimension, the indication is that Couple-VAE is using the latent codes more efficiently. Given limited space, we put the interpolation for qualitative analysis in Appendix H.

6 RELATION TO RELATED WORK Bowman et al. (2016) first identify the posterior collapse problem of VAE for text modeling and propose KL annealing and word drop to alleviate the problem. Zhao et al. (2017) propose an auxiliary bag-of-words (BoW) loss to mitigate this issue. Later work that works on the posterior collapse problem mainly focuses on using less powerful decoders (Yang et al., 2017; Semeniuta et al., Table 5 : Diversity and the first three samples from each model on Yelp. Dist-1 and Dist-2 stand for the ratios of distinct unigrams and bigrams over all generated ones. Redundancies are shown in red.

VAE:

Dist-1 = 0.0062 Dist-2 = 0.0248 1. the food is good , but the food is good . i had the chicken fried steak with a side of mashed potatoes , and it was a good choice . the fries were good , but the fries were good . i had the chicken breast with a side 2. ok , so i was excited to check out this place for a while . i was in the area , and i was n't sure what to expect . i was a little disappointed with the food , but i was n't sure what to expect . i was 3. we went to the biltmore fashion park . we were seated right away , but we were seated right away . we were seated right away , but we were seated right away . we were seated right away and we were seated right away . the staff was very Couple-VAE (λc = 10.0) Dist-1 = 0.0115 Dist-2 = 0.0593 1. i 'm a fan of the " asian " restaurants in the valley , and i 'm not sure what to expect , but i 'm not sure what the fuss is about . the meat is fresh and delicious . i 'm not a fan of the " skinny 2. i 'm not a fan of the fox restaurants in phoenix , but i have to say that the service is always a great experience . the atmosphere is a little dated and there is a great view of the mountains . 3. i have been here twice , and the food was good , but the service was good , but the food was good . i had a great time , but the service was great . the food was a bit pricey , but the service was a bit slow 2017), modifying the regularization objective (Higgins et al., 2017; Bahuleyan et al., 2019; Wang & Wang, 2019) , seeking alternative posterior families (Rezende & Mohamed, 2015; Xu & Durrett, 2018; Davidson et al., 2018; Xiao et al., 2018) , finding richer prior distributions (Tomczak & Welling, 2018) , improving optimization strategies (He et al., 2019; Fu et al., 2019) , incorporating skip connections into the posterior network (Dieng et al., 2019) , adopting hierarchical or autoregressive posterior distributions (Park et al., 2018; Du et al., 2018) , and narrowing the amortization gap (Hjelm et al., 2016; Kim et al., 2018; Marino et al., 2018) .

A model to be noted is β-VAE (Higgins et al., 2017) , in which the improvement of reconstruction and the sacrifice of regularization are modeled as a trade-off via a hyperparameter. Different from their work, our approach can be viewed as multi-task learning, where the coupled reconstruction task helps learn richer encoded representations and the distance minimization task helps learn nontrivial decoding signals. Since the auxiliary tasks are locally applied to the encoded texts and the decoding signals, they do not necessarily require a sacrifice of other components of the overall objective. He et al. (2019) demonstrate the incompatibility between the variational posterior and the true model posterior, i.e., the variational posterior sometimes lags behind the true model posterior. We investigate in another direction and provide a perspective that focuses on the encoder-decoder incompatibility, which comes from the stochasticity and over-regularization. In our experiments, we show that mitigating the encoder-decoder incompatibility can further improve the results in He et al. (2019) . Ghosh et al. (2019) propose to remove the stochasticity in VAE by directly injecting noises into a deterministic autoencoder, which empirically generates less "blurry" images. Different from their work, we still strictly follow the probability (density) estimation nature of VAE, and the deterministic network in our approach serves as an auxiliary to enrich the encoded representations and the decoding signals.

7. CONCLUSIONS:

In this paper, we provide the encode-decoder incompatibility as a new perspective on the posterior collapse problem of VAE for text modeling. By tracking and comparing the gradient norms of multiple components in DAE and VAE, we demonstrate that the incompatibility exists in both the forward pass and the backpropagation during VAE training. We propose a model-agnostic approach termed Couple-VAE, which mitigates the encoder-decoder incompatibility by enriching the reconstruction gradients for the encoder and encouraging nontrivial decoding signals. Experimental results show the effectiveness of our approach to improve various VAE-based models for text modeling in terms of probability estimation and the richness of the learned latent space. The training dynamics demonstrate that our approach mitigates the encoder-decoder incompatibility compared with VAE.

C DETAILS OF THE EXPERIMENTAL SETUP:

The dimension of latent vectors is 32. The dimension of word embeddings is 200. The encoder and the decoder are one-layer GRUs with the hidden state size of 128 for PTB and 256 for Yelp and Yahoo. For optimization, we use Adam (Kingma & Ba, 2015) with a learning rate of 10 -3 and β 1 = 0.9, β 1 = 0.999. The decoding signal is viewed as the first word embedding and also concatenated to the word embedding in each decoding step. After 30K steps, the learning rate is decayed by half each 2K steps. Dropout (Srivastava et al., 2014) 

D ESTIMATION OF LANGUAGE MODELING METRICS:

For language modeling, we report negative log-likelihood (NLL), KL divergence, and perplexity. To get more reliable results, we make the estimation of each metric explicit. For each test sample x, NLL is estimated by importance sampling, and KL is approximated by its Monte Carlo estimate:

EQUATION

where z (i) ∼ Q(z|x) are sampled latent codes and all notations follow Eq. ( 2). We report the averaged NLL and KL on all test samples. Perplexity is computed based on the estimated NLL. For validation, the number of samples is N = 10; for evaluation, the number of samples is N = 100.

E ESTIMATION OF MUTUAL INFORMATION AND RECONSTRUCTION:

We report the mutual information (MI) between the text x and the latent code z under Q(z|x) to investigate how much useful information is encoded. The MI component of each test sample x is approximated by Monte Carlo estimation:

MI x = E z∼Q(z|x) [log q(z|x) q(z) ] ≈ 1 N N i=1

(log q(z (i) |x) -log q(z (i) ))

where the aggregated posterior density q(z (i) ) is approximated with its Monte Carlo estimate:

EQUATION

where x (j) are sampled from the test set. For convenience, most previous work uses the texts within each batch as the sampled x (j) 's (which are supposed to be sampled from the entire test set). However, this convention results in a biased estimation since the q(z (i) |x (i) ) is computed when j = i, i.e., the text itself is always sampled when computing its MI component. We remedy it by skipping the term when j = i. The overall MI = E x [MI x ] is then estimated by averaging MI x over all test samples. We set the numbers of samples as N = 100 and M = 512.

For reconstruction, we sample ten latent codes from the posterior of each text input and decode them with greedy search. We compute BLEU-1 and BLEU-2 between the reconstruction and the input with the Moses script. Text A (PTB): now those routes are n't expected to begin until jan they are n't expected to be completed both sides are expected to be delivered at their contract the new york stock exchange is scheduled to resume today both sides are expected to be delivered at least the new york stock exchange is scheduled to resume both sides have been able to produce up with the current level it is n't clear that it will be sold through its own account it also has been used for comment it is n't a major source of credit it also has been working for the first time it also has a major chunk of its assets it also has a new drug for two years it also has a major pharmaceutical company it also has a $ N million defense initiative Text B (PTB): it also has a unk facility in california

H INTERPOLATION:

A property of VAE is to match the interpolation in the latent space with the smooth transition in the text space (Bowman et al., 2016) . In 

:

We will make our code public.

We will make the code for each baseline and its coupled version public.

Figures and Tables:

Figure 1: General frameworks of VAE and DAE for text modeling.

Figure 2: Training dynamics of DAE, VAE, and the proposed Couple-VAE (λ c = 10.0) on the Yelp test set. Please find the analysis in Section 3 and Section 5.5. Best viewed in color.

Figure 3: A graphical overview of Couple-VAE. The upper path is deterministic and optimized by the coupled reconstruction loss L c rec , and the lower path is the VAE model optimized by the reconstruction loss L rec , the regularization loss L reg , and the coupling loss L couple .

Language modeling results. NLL is estimated with importance sampling. PPL is based on the estimated NLL. KL and MI are approximated by their Monte Carlo estimates. Couplestands for "with the coupled deterministic network". The better results between a model and the version with the coupled deterministic network are shown in bold. *The exact NLL is reported. † Using or modifying open-source code which does not follow our setup and evaluation. ‡ Previously reported.

Table content: <table><tr><td/><td>PTB</td><td>Yelp</td><td>Yahoo</td><td/></tr><tr><td/><td>NLL (KL)</td><td>PPL NLL (KL)</td><td>PPL NLL (KL)</td><td>PPL</td></tr><tr><td>GRU-LM*</td><td>105.8 (-)</td><td>125.3 196.3 (-)</td><td>57.3 347.9 (-)</td><td>78.0</td></tr><tr><td>VAE</td><td>103.6 (8.6)</td><td>112.9 193.7 (7.2)</td><td>54.3 344.5 (12.4)</td><td>74.7</td></tr><tr><td>Couple-VAE</td><td>103.1 (9.5)</td><td>110.5 191.2 (8.0)</td><td>51.6 342.4 (12.8)</td><td>72.8</td></tr><tr><td>β(0.8)-VAE</td><td colspan="3">103.8 (11.0) 113.9 193.8 (10.2) 54.5 344.9 (16.1)</td><td>75.1</td></tr><tr><td>Couple-β(0.8)-VAE</td><td colspan="3">103.3 (12.1) 111.5 191.5 (12.2) 51.9 342.8 (17.0)</td><td>73.2</td></tr><tr><td>β(1.2)-VAE</td><td>103.7 (7.8)</td><td>113.3 193.7 (6.0)</td><td>54.3 345.3 (10.5)</td><td>75.5</td></tr><tr><td>Couple-β(1.2)-VAE</td><td>102.9 (8.6)</td><td>109.6 191.2 (6.9)</td><td>51.6 342.3 (11.3)</td><td>72.7</td></tr><tr><td>vMF-VAE</td><td>103.6 (2.0)</td><td>113.2 195.4 (0.0)</td><td>56.3 344.5 (2.5)</td><td>74.7</td></tr><tr><td>Couple-vMF-VAE</td><td>103.0 (3.0)</td><td>110.1 191.2 (2.8)</td><td>51.6 342.2 (4.0)</td><td>72.5</td></tr><tr><td>CNN-VAE</td><td colspan="3">118.5 (29.6) 222.6 194.2 (12.8) 54.8 344.3 (19.7)</td><td>74.5</td></tr><tr><td>Couple-CNN-VAE</td><td colspan="3">118.2 (30.2) 219.7 193.9 (13.7) 54.6 343.3 (22.4)</td><td>73.6</td></tr><tr><td>WAE</td><td colspan="3">103.7 (11.0) 113.3 193.7 (10.7) 54.3 344.7 (16.6)</td><td>74.9</td></tr><tr><td>Couple-WAE</td><td colspan="3">103.2 (12.5) 110.9 191.3 (12.5) 51.7 343.3 (18.2)</td><td>73.6</td></tr><tr><td>VAE-NF</td><td>103.3 (5.5)</td><td>111.3 193.9 (5.3)</td><td>54.5 344.3 (8.1)</td><td>74.5</td></tr><tr><td>Couple-VAE-NF</td><td>102.6 (5.7)</td><td>108.1 191.8 (5.6)</td><td>52.2 342.6 (8.8)</td><td>73.0</td></tr><tr><td>WAE-NF</td><td>103.4 (6.7)</td><td>111.9 194.1 (7.0)</td><td>54.7 344.3 (10.6)</td><td>74.5</td></tr><tr><td>Couple-WAE-NF</td><td>102.7 (7.4)</td><td>108.4 192.1 (7.4)</td><td>52.5 342.7 (11.0)</td><td>73.1</td></tr><tr><td>SA-VAE  †</td><td>100.7 (7.7)</td><td>98.7 183.5 (3.8)</td><td colspan="2">44.0 327.5 (7.2)  ‡ 60.4  ‡</td></tr><tr><td>Lagging-VAE  †</td><td>98.8 (6.0)</td><td>90.7 182.5 (1.2)</td><td>43.1 326.7 (6.0)</td><td>59.7</td></tr><tr><td colspan="2">Couple-Lagging-VAE  † 98.7 (11.0)</td><td>90.4 182.3 (3.8)</td><td>42.9 326.2 (7.4)</td><td>59.3</td></tr></table>

Mutual information (MI) and reconstruction metrics (i.e., BLEU-1 and BLEU-2). MI is approximated by its Monte Carlo estimate. Other notations follow Table1.

Table content: <table><tr><td/><td/><td>PTB</td><td/><td>Yelp</td><td/><td>Yahoo</td></tr><tr><td/><td>MI</td><td>BLEU-1/2</td><td>MI</td><td>BLEU-1/2</td><td>MI</td><td>BLEU-1/2</td></tr><tr><td>VAE</td><td>10.48</td><td>23.2 / 4.4</td><td>8.28</td><td colspan="2">28.7 / 5.3 15.43</td><td>21.2 / 3.6</td></tr><tr><td>Couple-VAE</td><td>11.99</td><td>23.4 / 4.5</td><td>9.65</td><td colspan="2">30.4 / 5.8 16.44</td><td>23.1 / 4.1</td></tr><tr><td>β(0.8)-VAE</td><td>15.43</td><td colspan="2">24.5 / 4.9 13.52</td><td colspan="2">30.6 / 6.0 24.16</td><td>24.0 / 4.3</td></tr><tr><td>Couple-β(0.8)-VAE</td><td>18.13</td><td colspan="2">24.3 / 4.8 17.69</td><td colspan="2">32.6 / 6.6 28.03</td><td>26.4 / 4.9</td></tr><tr><td>β(1.2)-VAE</td><td>9.16</td><td>22.8 / 4.3</td><td>6.60</td><td colspan="2">28.0 / 5.0 11.83</td><td>18.2 / 2.9</td></tr><tr><td>Couple-β(1.2)-VAE</td><td>10.28</td><td>22.9 / 4.2</td><td>7.90</td><td colspan="2">29.8 / 5.6 13.51</td><td>22.4 / 3.8</td></tr><tr><td>vMF-VAE</td><td>1.74</td><td>15.2 / 2.0</td><td>0.03</td><td>22.4 / 2.8</td><td>2.06</td><td>8.5 / 1.1</td></tr><tr><td>Couple-vMF-VAE</td><td>2.37</td><td>16.1 / 2.3</td><td>2.60</td><td>25.1 / 4.0</td><td>3.37</td><td>10.3 / 1.4</td></tr><tr><td>CNN-VAE</td><td>78.49</td><td colspan="2">32.0 / 7.8 17.26</td><td colspan="2">32.9 / 7.1 30.18</td><td>24.9 / 5.3</td></tr><tr><td>Couple-CNN-VAE</td><td>80.54</td><td colspan="2">31.8 / 7.7 19.15</td><td colspan="2">33.4 / 7.3 37.62</td><td>26.9 / 5.9</td></tr><tr><td>WAE</td><td>15.09</td><td colspan="2">24.8 / 5.1 15.08</td><td colspan="2">30.7 / 6.1 24.73</td><td>24.2 / 4.5</td></tr><tr><td>Couple-WAE</td><td>18.51</td><td colspan="2">24.7 / 5.1 18.56</td><td colspan="2">32.5 / 6.6 30.08</td><td>27.7 / 5.3</td></tr><tr><td>VAE-NF</td><td>5.63</td><td>19.2 / 3.3</td><td>5.64</td><td>25.6 / 4.5</td><td>8.02</td><td>13.7 / 2.1</td></tr><tr><td>Couple-VAE-NF</td><td>5.86</td><td>19.4 / 3.3</td><td>6.06</td><td>26.3 / 4.6</td><td>9.14</td><td>15.3 / 2.5</td></tr><tr><td>WAE-NF</td><td>7.18</td><td>19.7 / 3.5</td><td>7.95</td><td colspan="2">26.0 / 4.6 11.43</td><td>13.8 / 2.2</td></tr><tr><td>Couple-WAE-NF</td><td>8.10</td><td>20.7 / 3.7</td><td>8.53</td><td colspan="2">27.2 / 5.0 12.56</td><td>14.9 / 2.5</td></tr><tr><td>Lagging-VAE  †</td><td>2.90</td><td>-</td><td>0.96</td><td>-</td><td>3.04</td><td>-</td></tr><tr><td>Couple-Lagging-VAE  †</td><td>3.29</td><td>-</td><td>2.36</td><td>-</td><td>3.06</td><td>-</td></tr></table>

The effect of the coupling weight λ c . *Reported in the Table1 and 2.

Table content: <table><tr><td/><td/><td>PTB</td><td/><td/><td>Yelp</td><td/></tr><tr><td/><td>NLL (KL)</td><td>PPL</td><td>MI</td><td>BLEU-1/2 NLL (KL)</td><td>PPL</td><td>MI</td><td>BLEU-1/2</td></tr><tr><td>VAE</td><td>103.6 (8.6)</td><td colspan="2">112.9 10.48</td><td>23.2 / 4.4 193.7 (7.2)</td><td>54.3</td><td>8.28</td><td>28.7 / 5.3</td></tr><tr><td colspan="2">λc = 0.1* 103.1 (9.5)</td><td colspan="2">110.5 11.99</td><td>23.4 / 4.5 191.2 (8.0)</td><td>51.6</td><td>9.65</td><td>30.4 / 5.8</td></tr><tr><td>λc = 1.0</td><td colspan="3">103.3 (10.7) 111.4 14.32</td><td>24.0 / 4.8 191.1 (8.1)</td><td>51.5</td><td>9.92</td><td>30.5 / 5.8</td></tr><tr><td>λc = 2.0</td><td colspan="3">103.2 (11.8) 111.1 16.58</td><td colspan="3">24.2 / 5.0 191.7 (10.5) 52.1 14.13</td><td>31.9 / 6.2</td></tr><tr><td>λc = 5.0</td><td colspan="3">103.7 (16.1) 113.2 32.78</td><td colspan="3">26.5 / 5.8 191.5 (12.8) 51.9 19.77</td><td>32.8 / 6.5</td></tr><tr><td colspan="4">λc = 10.0 104.7 (21.8) 118.5 44.93</td><td colspan="3">29.0 / 7.0 191.8 (17.3) 52.2 27.08</td><td>34.7 / 7.2</td></tr></table>

Gradient norms of the reconstruction loss, the coupled reconstruction loss, the regularization loss, and the decoding signal w.r.t. the encoded text on each test set.

Table content: <table><tr><td/><td/><td>∂Lrec</td><td>∂L c rec</td><td>∂(Lrec + L c rec )</td><td/><td>∂Lreg</td><td>∂h ∂e F</td></tr><tr><td/><td/><td>∂e 2</td><td>∂e 2</td><td>∂e</td><td>2</td><td>∂e 2</td><td>h 2</td></tr><tr><td/><td>DAE</td><td>1719.8</td><td>-</td><td>-</td><td/><td>-</td><td>3.14</td></tr><tr><td>PTB</td><td>VAE</td><td>112.5</td><td>-</td><td>-</td><td/><td>19.4</td><td>2.05</td></tr><tr><td/><td>Couple-VAE (λc = 0.1)</td><td>148.5</td><td>2109.6</td><td>2320.2</td><td/><td>27.7</td><td>2.12</td></tr><tr><td/><td>DAE</td><td>2443.6</td><td>-</td><td>-</td><td/><td>-</td><td>2.55</td></tr><tr><td>Yelp</td><td>VAE</td><td>59.7</td><td>-</td><td>-</td><td/><td>18.8</td><td>1.62</td></tr><tr><td/><td>Couple-VAE (λc = 0.1)</td><td>84.8</td><td>3640.8</td><td>3764.7</td><td/><td>25.0</td><td>2.25</td></tr><tr><td/><td>DAE</td><td>4104.6</td><td>-</td><td>-</td><td/><td>-</td><td>3.39</td></tr><tr><td>Yahoo</td><td>VAE</td><td>257.9</td><td>-</td><td>-</td><td/><td>52.8</td><td>2.92</td></tr><tr><td/><td>Couple-VAE (λc = 0.1)</td><td>335.3</td><td>5105.0</td><td>5615.0</td><td/><td>65.0</td><td>3.91</td></tr></table>

rate is 0.2. KL-annealing(Bowman et al., 2016) is applied from step 2K to 42K (on Yelp, it is applied from step 1K to 41K for VAE, Couple-VAE, β-VAE, and Couple-β-VAE; otherwise, the KL divergence becomes very large in the early stage of training). For each 1K steps, we estimate the NLL for validation.For normalizing flows, we use planar flows(Rezende & Mohamed, 2015) with three contiguous transformations. For WAE and WAE-NF, we use Maximum Mean Discrepancy (MMD)(Gretton et al., 2012) as the regularization term. Following Wang & Wang (2019), an additional KL regularization term with the weight β = 0.8 (also with KL-annealing) is added to WAE and WAE-NF since MMD does not guarantee the convergence of the KL divergence.

Table content: <table/>

Texts generated from the interpolations of two latent codes.

Table content: <table><tr><td>VAE</td><td>Couple-VAE (λc = 10.0)</td></tr></table>

Table 7, we show the interpolation of VAE and Couple-VAE (with the coupling weight λ c = 10.0) on PTB. It shows that compared with VAE, Couple-VAE has smoother transitions of subjects (both sides → it) and verbs (are expected → have been → has been → has), indicating that the information about subjects and verbs is more smoothly encoded in the latent space of Couple-VAE.

Table content: <table/>

Peer review (iclr2020):
Idea description: In order to solve the encoder-decoder incompatibility, which means little information are backpropogated from the decoder to the encoder, the authors proposed to combine a VAE and AE by weight sharing and hidden feature matching. The idea is very intuitive and the experiments on the perplexity and KL indeed proved this intuition: adding AE term could help us to reconstruct a sentence easier and harm the KL divergence. To further prove the effectiveness of this method, the author could either do more experiments on image or do more text generation tasks, e.g. conditional generation, dialogue generation