Improving Variational Autoencoders with Density Gap-based Regularization
Authors: Jianfei Zhang
Jun Bai
Chenghua Lin
Yanmeng Wang
Wenge Rong
Venue: NeurIPS 2022 Accept (2022)
Keywords: Variational Autoencoders
Posterior Collapse
Hole Problem

Abstract: Variational autoencoders (VAEs) are one of the most powerful unsupervised learning frameworks in NLP for latent representation learning and latent-directed generation. The classic optimization goal of VAEs is to maximize the Evidence Lower Bound (ELBo), which consists of a conditional likelihood for generation and a negative Kullback-Leibler (KL) divergence for regularization. In practice, optimizing ELBo often leads the posterior distribution of all samples converging to the same degenerated local optimum, namely posterior collapse or KL vanishing. There are effective ways proposed to prevent posterior collapse in VAEs, but we observe that they in essence make trade-offs between posterior collapse and the hole problem, i.e., the mismatch between the aggregated posterior distribution and the prior distribution. To this end, we introduce new training objectives to tackle both problems through a novel regularization based on the probabilistic density gap between the aggregated posterior distribution and the prior distribution. Through experiments on language modeling, latent space visualization, and interpolation, we show that our proposed method can solve both problems effectively and thus outperforms the existing methods in latent-directed generation. To the best of our knowledge, we are the first to jointly solve the hole problem and posterior collapse.

Improving Variational Autoencoders with Density Gap-based Regularization
Jianfei Zhang!?
Jun Bai
Chenghua Lin Yanmeng Wang*
Wenge Rong" 1.2
State Key Laboratory of Software Development Environment, Beihang University, China SSchoo of Computer Science and Engineering, Beihang University, China
Department of Computer Science, University of Sheffield, United Kingdom
*Ping An Technology, China
{zhangjf bai_jun, www.r).ng)).........o..... @buaa. edu. c. L.in@sheffield ac. uk, wangyamneng22900pnnaan com. cn
Abstract
Variational autoencoders (VAEs) are one of the most powerful unsupervised learn- ing frameworks in NLP for latent representation learning and latent-directed gener- ation. The classic optimization goal of VAEs is to maximize the Evidence Lower Bound (ELBo), which consists of conditional likelihood for generation and a negative Kullback-Leibler (KL.) divergence for regularization. mizing ELBo often leads the posterior distribution of all samples converging the same degenerated local optimum, namely posterior collapse or KL vanishing. There are effective ways proposed to prevent posterior collapse in VAEs, but we observe that they essence make trade-offs between posterior collapse and the hole problem, i.e., the mismatch between the aggregated posterior distribution and the prior distribution. To this end, we introduce new training objectives to tackle both problems through novel regularization based on the probabilistic density gap between the aggregated posterior distribution and the prior distribution. Through experiments language modeling, latent space visualization, and interpolation, we show that our proposed method can solve both problems effectively and thus outperforms the existing methods in latent-directed generation. To the best of knowledge, we are the first to jointly solve the hole problem and posterior collapse.
practice, opti-
Introduction
As one of the most powerful likelihood-basse generative models, variational autoencoders (VAEs) [21, 32] are designed for probabilistic modeling directed by continuous latent variables, which successfully applied many NLP tasks, e.g., dialogue generation [45, 14], machine translation [34, 12], recommendation [10], and data augmentation [43, 39]. One of the major advantages of VAEs is the flexible latent representation space, which enables easy manipulation of high-level semantics on corresponding representations, e.g., guided sentence generation with interpretable vector operators.
Despite the attractive theoretical strengths, VAFS are observed to suffer from a well-known problem named posterior collapse or KI. vanishing [21, 24], an optimum state of VAEs when the posterior distribution contains little information about the corresponding datapoint, which is particularly obvious when strong auto-regressive decoders are implemented [46, 4].
Another challenge for VAEs the hole problem, the state when the aggregated (approximate) posterior fails to fit the prior distribution, and thus the inference from the prior distribution becomes no longer suitable to describe the global data distribution [33], which can lead to poor generation quality in VAEs [1, 25].
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

BN-VAE(L
B-VAE10.4
FB-VAE(36)
asgregstidd proterior datapcinns Yahoo test-sst Le. (x-2
0J5
605
655
-25
35
-15
54
-3 -35 59
-50 -25
59
-56
-25 OF 50
A-VAE(0.4)
VAE(36)
DG-VAE
the posterice centers of datapcints Yahoo f (Forttt,,
Figure 1: The visualization of the aggregated posterior distributions (the first line) and the posterior centers distributions (the second line) for models on the Yahoo test-set. The vamilla VAEs suffer from posterior collapse, i.e.. the posterior centers collapse to the same position. Mcanwhile, BN-VAEs [46], 3-VAEs [18] and FB-VA VAES [20] can solve posterior collapse effectively at the cost of bringing the hole problem, i.e., mismatch between the aggregated posterior and the prior. Our proposed DG-VAE intends to solve both problems through a novel regularization based on the density gap. Illustrations for more datasets, more models, and more dimensions, are shown in Appendix G.
In this work, we perform systematic experiments on VAEs for text generation to study posterior collapse and the hole problem existing methods. We demonstrate that VAEs with specific network structures [9, 45] or modified training strategies [6. 13] have limited effect on solving posterior collapse, while VAEs with hard restrictions [8, 41, 46] or weakened KL regularization [18, 20] on the posterior distribution can solve posterior collapse effectively at the expense of the hole problem, as illustrated in Figure 1.
On that basis, we hypothesize that these two problems stem from the conflict between the KL regularization in ELBo and the function definition of the prior distribution. As such, we propose novel regularization to substitute the KL regularization in ELBo for VAEs, which is based on the density gap between the aggregated posterior distribution and the prior distribution. We provide theoretical proof that our method in essence maximizes the ELBo as well as the mutual information between the input and the latent variable.
In terms of Gaussian distribution-based VAES, we further propose the corresponding marginal regularization on each dimension respectively, and we prove in essence maximizes the ELBo as well as the sum of mutual information between the input and the latent variable on all dimensions.
To validate methods in practice, take experiments language modeling, latent-guided generation and latent space visualization. We demonstrate that our methods form latent spaces that are both active and consistent with the prior, and thus generate smoother sentences from latent interpolation. The code and data are available at https::/ /ttps:://gthubb com/ ffma/hhagjjfff wnlpefffnnpp------ VAEs.
Background and Related Work
2.1 VAEs and ELBo
VAEs are proposed to perform efficient inference and learning in directed probabilistic models [21], where the random generation process consists of two steps: (1) sample latent value from the prior Pe (z); (2) generate a datapoint x from the conditional distribution pe (x(x))) As the true posterior pe (z) x) is intractable, a recognition model q (z/x) is introduced to approximate the true posterior [21].
Specifically, VAES represent an observation x as latent distribution (z(x). from which latent variables are sampled to direct the reconstruction of x. As the optimization goal VAFs, the Evidence Lower Bound (EI.o) is composed of Kullback-Leible (KI.) divergence for regularization on the posterior and a log likelihood for reconstruction conditioned on posterior. The ELBo in fact forms a lower bound on the marginal likelihood given prior latent variables [21], illustrated in Eq. 1.
L LELBO (0, Φb x) (x [log Po (x/2)) De (gos (z (z/x ())
(1)
s (o))po(x) log po (

---

2.2 Posterior Collapse
encourages the approximate posterior distribution of every single datapoint to be close to the the prior [21]. This intends to ensure the prior distribution can depict the latent variable distribution over the data distribution, but it can also lead to posterior collapse when DxL. Dx (2)(2())]pp (z)) has much stronger force on Iø (z)2 x) than gg (log) pe (x|z)] does, which leads that l ( x) = po (z(z)Vz. In such condition, the sampled latent variable, z ~ ( (Z)z)) contains much more noise than useful information about æ [27], and thus the decoder pe x (2) becomes insensitive to z [46].
Intuitively, the KL divergence in ELBo (i.e., (
Early works to solve posterior collapse attribute it the difficulty in optimizing ELBo, so their methods mainly focus on the training strategies [6, 13, 23]. Some works put emphasis on the semantic learning of the latent variable through specific model structures [9. 45, 29, 17J. Instead of Gaussian distribution, IMF-VAE [8, 41] adopts the von Mises-Fisher distribution for latent variables, which restricts the posterior latent to a hyperspherical space and forms constant KI. divergence. BN- VAE [46] restricts the posterior distribution through a batch normalization layer with fixed scale 7, so as to guarantee positive lower bound of the KL divergence. 3- B-V [18] directly changes the weight (denoted 3) of the KL term inside ELBo, while free-bits [20] changes the KL term inside ELBo to a hinge loss term.
In contrast, we hypothesize posterior collapse is due to the conffict between the KL regularization in ELBo and the function definition of the prior distribution, and tackle it through replacing the KL regularization in ELBo with a novel regularization on the aggregated posterior distribution.
2.3
Hole Problem
The aggregated (approximate) posterior qs (z) refers to the expectation of the approximate posterior distribution on the data distribution, as defined in Eq. 2, where the distribution of observation is represented by the discrete distribution of datapoints the dataset, i.e., X ) 90 (n( = N
in practice.
1 N Σ g (z)
(2) Ex (x((((z)x))
2
Formally, the hole problem refers to the phenomenon that the aggregated posterior distribution as (z) fails to the prior distribution (e(z). Inferences located in the holes (i.e., areas with mismatch between density in g (z) and (z)) are observed generate images that are obscure and corrupted [1], or sentences with incorrect syntax and abnormal semantics [25].
The hole problem of VAEs for image generation is observed in several studies, commonly ascribed to the limited expressivity of the prior distribution and tackled by increasing the flexibility of the prior distribution via hierarchical priors [22]. auto-regressive models [16], mixture of encoders [37], normalizing flows [40], resampled priors [3], and energy-based models [1]. In contrast, we observe that the vanilla VAEs (with standard prior distributions) for text generation have no hole problem, but it arises when existing methods are applied to solve posterior collapse. Therefore, our work is targeted at solving posterior collapse and avoiding the hole problem at the same time, for VAEs with standard prior distributions.
2.4 Regularization on Aggregated Posterior
As the approximate posterior distribution is introduced to approximate the true posterior, i.e., (z|x) s pe(z)x), the aggregated posterior should be close to the prior as a result, i.c., (z(z) (z).). From this point of view, several works are proposed replace KL regulariza- tion (on the posterior distribution of each datapoint separately) in ELBo with regularization on the aggregated posterior distribution, which can be summarized Eq. 3, where D is the divergence (or discrepancy) between two distributions.
Cp(0, Φ; x) = (lo pe (x z)] D(go (z) (z)
Among them, Adversarial Auto-Encoder (AAE) [28] adopts the Generative Adversarial Network (GAN) [15] framework to regularize the aggregated posterior distribution through Sansen-Shanno
3

---

divergence DJs: Wasserstein Auto-Encoder (WAE) [36, 2] regularizes the aggregated posterior distribution through minimizing Maximum Mean Discrepancy (MMD); Implicit VAE with Mutual Information regularization (iVAEMr) regularizes the aggregated posterior through a dual form of KL. divergence D on the basis of Implicit VAE (iVAE) [11]. These methods have the same weakness that their approximations of the divergence between two continuous distributions are depicted by merely sampling sets from the distributions, which involves noise from random sampling and can
hardly be zero, even for the same distributions.
In contrast, our method approximates the divergence between two continuous distributions in the perspective of their mismatch in PDFs, which we quantify through the density gap that can be zero if and only they are the same, which we describe in section 3.
We validate our method against the aforementioned methods through experiments on a synthetic dataset, the details and results of which are presented in Appendix A.
3 Methodology
Density Gap-based Discrepancy One of the straight manifestations of holes in latent space the mismatch of probabilistic density between (2) and po (z). We quantify this mismatch at a specific position in the latent space through DG z), which we refer to as Density Gap. Here we only consider = E (z)ga(=) > 0}. We assume (=) and (z(z) are differentiable and (z) > O.
DG Φ; 2) = log L0 (=)
1 Σn (0(2(xn)) N
log
(4)
po
(2
can be inferred that the expectation of DG ( Φ; 2) on qo (z) equals to the KL divergence between qo (z) and pe (z). illustrated in Eq. 5, which is strict divergence, i.c. (x (DG Ø)
O iff (z) (z)...
(z)2),
Ex~gg(2) (2) (DG(0, = Ex flog
( 2
(5)
po
So, we can approximate and optimize D (gs (z) (2)) (z)) via Monte Carlo, as illustrated in Eq. 6, where 2, idd (z) denotes the th random sample from the aggregated posterior distribution.
1 s
(eL(go(z)(po(z))) ~ DG
(6)
should be noted that D (go [ the posterior distribution of all datapoints as whole, instead of averaging (xz(qo(z/x)]lpo] (z)) across all datapoints as ELBo does.
(z)) approximated by this is an overall divergence, as it consid-
On that basis, we can implement LD Φ; x) with D DKL. which is equivalent to replacing the KI term in ELBo with D DxI (z) l| (2)) approximated by Eq. 6. According to the decom- position (illustrated in Eq. 7) of the KL term in ELBo given by Hoffman et al. [19], maximizing DD (0, Φi xn)) on the whole dataset,
= { 2o (n) a equivalent to maximizing ELBo as well as (n.) 2], the mutual information of and n in their joint distribution qs (n, 2). illustrated in Eq. 8.
DRI (ko(go(2/2n))lpl D (po(z) + - (0,
2 ( z)
where II (n, Eg 2 flog
go (n)(n)
where (n, z) = (z) N g (z 2
Althooug have (z) 0, Pim when the latent variable follows von Mises- (yMF) distribution, we do not need to consider such points in regularization.
Posterior collapse (or KL vanishing) can be solved effectively by maximizing this mutual information term it is lower bound of the vanished KL divergence term in ELBo according to Eq. 7.

---

N
DDu (0, 0; xn)]
[e (0goeel..llopoo( plog - D
N
==
(8)
N
LELBO (0,00 xx,,]] + tea(nx))
Optimization on Mini-Batch Theoretically attractive as it is, maximizing CDKL (0, Φ; Fn)) approx- imated by (G Φ; 2,) ) undesirable for training VAEs on large datasets, because the probabilistic density of q4 (z) at needs computation across the whole dataset and is changing in every training step. practice, training deep networks such VAEs commonly adopts mini-batch gradient descent, where only small subset of the dataset is used for calculating gradients and updating parameters in each iteration step.
So, a practicable way to aggregate the posterior of datapoints inside a mini-batch { 96 ( (n) B stated in Eq. 9, where 2m,m is the m sample from the posterior of datapoint En- Here, stratified sampling is used to ensure a steady Monte Carlo approximation." and the reparameterization trick [21] is applied ensure differentiable output.
(0.B(2)
Σ>E |B
(0(2)an)
DG(0, Φ 2;z) log
log
(=
po(z)
(9)
B
M
~
Σ m==1
DG(0, Φ, B; 2n,m), where 2n,m
idd
9Φ (z|
B
M
n=1
Through this approximation, we can implement LD (0, Φ, B;xx) that regularizes (z) towards (z) for mini-batch B, stated in Eq. 10.
B
B
CDKL (0, 0, B;xa)l
Σ
( Logpo (Z,|2)]] D (xx(go.oe<))loll
B
[
(10)
B
(C,
+ Lyg (n.))
It should be noticed that the mutual information term (m.x) z] is different in Eq. 8 (on the whole from
dataset) and Eq. 10 (on a mini- mini-batch), because the range of discrete variable
N in Eq. 8, but 1 to [B] in Eq. 10. Consequently, [ [n, has an upper bound of H = log N in Eq. 8, but H (n) = log B in Eq. 10. In other words, maximizing (n.x) ( [n, z] in Eq. 8 intends to distinguish z of xnn from that of N - other datapoints, while it is limited to B - other datapoints in Eq. 10.
Marginal Regularization for More Mutual Information As described above. approximating and optimizing DxL. (go.n(2 (zo (()o))) (2) is practicable but has limited effect. Empirically, Gaussian distribution-hasee VAES trained by Eq.. 10 still have limited active units, which means the encoded latent variable z still collapses to the prior on most dimensions, where it provides little information.
To activate on all dimensions, respectively, i.e., regularize the marginal distribution of qo. B B()) on each dimension, as illustrated in Eq. 11, where z; E R is the 4th component of z Rim and the corresponding probability density functions are of the marginal distributions on the ith dimension; and žn,m, is the 7th component of
propose regularize (z. towards pe (z) on each dimension
"In other words, we sample S ge (z) for each datapoint
|B] x A samples from gg.0(2) through sampling M samples from
5

---

the mh sample from the posterior of datapoint Fn-
(5. po(2i)
[3]
DGmrg(0,0g (0, log
loog
Po(2s)
Dim
DKL DKL.mrg(9o.B(2)lol
Σ
DKL Bkilgo.b(zi)llo(0m
(11)
Dim
8
Σ
DGmrg (0, B; Bizmma))
M
In Gaussian distribution-base VAEs, the marginal distributions of on different dimensions are independent, i.e., Us (z|2x, II Dim as (z, and po po(z) = Di po(z;), so their KL divergence can be decomposed as Drz (90 (zloo))lpo(z) Σ; Dim (xz (z)( Thus, we can infer the decomposition of D L.mmr (() (z))lep (z)) through Eq. 12.
Dim
DKt.mggg
Dx (2).o
Dim
- [n,
(12)
B
[B]
Dim
1
(
Σ (n,x))))
B
So, maximizing DD (0, Φ. B; xn) derived from Eq. 11 is equivalent to maximizing ELBo as well as the mutual information of n and z, for each dimension i respectively, stated in Eq. 13. In other words, it intends to distinguish z; of xi from that of [ - 1 other datapoints for each dimension i respectively. We refer our models based this Density Gap-based regularization DG-VAEs.
DDDLL (0, x1) =
| llogro D.L. (<)ll()))o
B
n=1
B
Dim
(0,
Σ (n.,),n 2i]]
(13)
Aggregation Size for Ablation As discussed above, the size of mini-batch [B] sets an upper bound of the mutual information term (n.()))) [n, z] (or (n,) [n, zz). To validate this impact, we further extend DG-VABs through dividing the mini-batch into non-overlapping subsets B
U bj. s.t. by nb, = 0 iff i + j. calculating and optimizing the KI. divergence over each subset, i.e.,
-. b. =1 LDKLL... (0, Φb, by; ..))], where C denotes the number of subsets and [b: is the size of those subsets, which we refer to as the aggregation size and denote as |b] for simplification.
can be inferred that DG-VAE with |b| is equivalent to the vanilla VAE trained by ELBo, except that it approximates the KL term through Monte Carlo instead of integration.
Extension to von Mises-Fisher Distribution-based VAEs Besides the commonly used Gaussian distribution- based VAEs, we also consider von Mises-Fisher (vMF) distribution-basee VAEs. As the decomposition (z) = IIPim (z,))) is not established for latent variables following vMF distributions (i.e., z ~ uM F(J,, k)),, marginal regularization for vMF-VAES may be not interpretable, we only implement Eq. 10 in VMF-VAEs. We refer to those extensions DG-vMF-VAN

---

Table 1: Statistics of sentences in the datasets
Dataset
Train
Valid
Test
Vocab size
Length (avg 1 std)
Yelp
100,000
10,000
10,000
19997
98.01 + 48.86
Yahoo
100,000
10,000
10,000
20001
80.76 + 46.21
Short-Yel Yelp
100,000
10,000
10,000
8411
10.96 3.60
SNLI
100,000
10,000
10,000
9990
11.73 + 4.33
4 Experiments
4.1 Experimental Setup
Datasets We consider four public available datasets commonly used for VAE-based language modeling tasks in our experiments: Yelp [42], Yahoo [42, 44], downsampled version of Yelp [35] (we denote this as Short-Yelp) and downsampled version of SNLI [5, 23J. The statistics of these datasets are illustrated in Table 1. It can be viewed that Yelp and Yahoo contain long sentences while Short- Delp and SNLI contain short sentences.
Baselines We consider wide range of VAEs for solving posterior collapse in text generation, where the hyperparameters are set according to Zhu et al. [46]:
VAFs with modified training strategies (i.e., KI. annealing): VAE with linear KI. annealing in the first 10 epochs (default) [6]; VAE with linear KI. annealing for 10 epochs at the start of every 20 epochs (cyclic-VAr) [13];
VAES with specific model structures: VAE with additional Bag-of- -ordd loss (bow- VAE) [45], and VAE with skip connection from the latent variable the vocabulary classifier for generation (skip-VAE) [9];
VAEs with hard restrictions the posterior distribution: 8-VAE with the committed rate 8 = 0.15 [31]; BN-V with the scale of BN layer 7 E (0.6, 0.7, 0.9, 1.2, 1,, 1.8) [46]; {13, 25, 50, 100, 200} [8, 41];
vMF-VAEs with the distribution's 's concentration
VAEs with weakened KL regularization: FB-VAEs (fire--bits with the target KL AKL {4, 9, 16, 25, 36, 49} [20]; 3-VAEs with the weight of the KL term in ELBo B E {0.0, 0.1, 0.2, 0.4, 0.8) [18].
Configurations We completely follow Zhu al. [46] in the models' backbone structures, data pre-processing, and training procedure, which we describe in detail in Appendix B.
4.2 Language Modeling
We evaluate the performance of our methods and the baselines on language modeling, where the following metrics are reported: the prior log likelihood priorLL(O) and the posterior log likelihood LL () for generation quality; the KL term in ELBo KLΦ)), the mutual information M M(Φ) of z and and the number of active units (U( [7] for posterior collapse; and the number of consistent units (UØΦ) (we propose) for the hole problem. The corresponding expressions and explanations are presented Appendix C.
We illustrate part of the results on Yahoo in Table 2 and all results on all datasets in Appendix D. can be observed that: (1) models with modified training strategies specific model structures can alleviate the problem of posterior collapse but has limited effect according to MI( and AU (Φ);; (2) models with hard restrictions
weakened KL regularization on the posterior solve posterior collapse better through harder restrictions or further weakening according to the increase of K L()). (I and AU (Φ). but the decrease of (+)) indicates that their posterior latent spaces tend to be increasingly inconsistent with that of the prior; (3) in contrast, our proposed DG-VAE has similar performance to the vanilla VAE when [b] 1, and with the increase of [b], it can solve posterior collapse effectively and avoid the hole problem at the same time.4
'There's 's little difference between DG- VAE (b) = 32) and DG-VAE (default): DG- AG-VAE (b) = 32) ignores data batch B if B < 32 while DG-VAN VAE (default) accepts it through adapting its batch size.
7

---

Table 2: Results of Language Modeling Yahoo dataset. We bold up M MI > 9.0, AU( 30,
()) 2 30, the highest priorLL(6 and postLL(0, Φ) for the same methods.
Models
prior OL
postLL(O, +)
KL(Φ)
MI(Φ)
(
(U(
VAE (default)
-330.7
-33..7
32
cyclic- VAE
-329.8
-328,9
1.1
1.0
31
bow VAE
-330.5
-330.5
0.0
0.0
32
skip-VAE
-330.1
-325.2
5.0
4.3
31
8-VAF(0.155
-330.5
-330.6
4.8
0.0
BN- (AB(0.6)
-327.6
321.1
6.6
5.9
32
32
BN- BN-VAB(1.2)
-330.9
310.11
26.2
32
BN-VAE(1.5
-34.55
-308.6
51.3
9.2
32
FB-VAE(4)
-329.8
-328.4
3.9
1.8
32
32
FB-VAE(16)
335.77
-320.8
16.1
8.5
32
FB-VAE(49)
-34..6
-296.1
50.0
9.2
32
8- VAE(0.4)
-330.8
-324.8
7.0
6.7
31
3-VAE(0.2)
-338.6
-30.3
30.1
9.2
22
25
3-VAB(0.1)
-369.9
-289.6
83.7
9.2
32
DG-VAE (b)
-330.7
-330.7
0.0
0.0
32
DG-VAE (6)
-330.4
-318.3
14.3
9.1
32
VG- (6) 32)
-355.4
-294.1
65.2
9.1
32
32
DG-VAE (default)
-358.0
-290.8
70.8
9.1
32
32
It also be viewed that with the increase of MI(Ø) KL()), postLL(O, Φ) tends to increase, while prior (0) tends to decrease, as the decoder 0 becomes more dependent on the encoder Φ. We further plot the curves of (rior..T.()) and L. <) for models with different hyperparameters in Figure 2, where we can observe that DG-VAFs make better trade-offs than BN-VAEs and BA-AFF do on short datasets and perform competitively to BN-VAEs and FB-VAEs on long datasets.
Shart-Wel
SNL
Yahoo
Vel
-32
-32
20111/0 51
-390
-02
-50
-72
priotL.0)
prion/L10)
Figure 2: The curves prior LL) and postLL(O, Φ) in Gaussian distribution basseb VAEs.
We also compare the performance of DG-vMF-VAE with VAE- under different settings of k. As they have the same (()), while (U( and (U()) are inappropriate to report for vMF distributions, only plot their curves of prior (0) and Lo Φ) in Figure 3. It can be observed that DG-vMF-V. VAEs outperform MF-VAES in most cases.
Shart-Wetp
Ye
-34
-38
-33
332
-333
prior1C05)
(hir1L080
Figure 3: The curves of Lr)) and LL,, in vMF distribution-baee VABs.

---

4.3 Visualization of the Posterior
To further investigate the posterior distribution in latent space of those models, we visualize the aggre- gated posterior distributions and the posterior centers distributions on the 2 most active dimensions, ie., the two dimensions with the highest Varx-xx (e)()))) [z]], as depicted in Figure 4.
Figure 4: The visualization of the aggregated posterior distributions (red-in-black) and the posterior centers distributions (blue-in-white) for BN-VAEs, FB-VAES 3-VAEs, and DG-VAEs on the Yahoo test-set. Illustrations for more datasets, more models, and more dimensions, are shown in Appendix G.
Here, we can observe that BN-VAEs, FB-VAEs and 3-VAEs can better solve posterior collapse with harder restrictions further weakening, but meanwhile they are faced with different kinds of mismatch between the aggregated posterior distribution and the prior distribution, i.c., the hole problem. contrast, with increase of aggregation size (b], DG- VAE can better solve posterior collapse and avoid the hole problem in the meantime.
4.4 Interpolation Study
One of the main advantages of VAEs over normal language models (e.g., GPT-2 [30]) is that VAEs embed datapoints into continuous latent space and thus enable latent-guided generation. We evaluate this ability through interpolation, where the models encode two sentences x, and X;, as their posterior centers, i.e., 2e (x)x..) and 2/ E (z/xx [z). and decode the variables between them, i.c..
2 = Za (1 - >) + 26 X, E (0.0, (.10.0.1... 0.1..... 1.0). The interpolated sentences are wished to be evaluate through the average Rouge-L F1-score [26], as stated in Eq. 14, where Fics denotes the Fi-score of Longest Common Subsequence (LCS). We plot the curves of Rouge-L Fl-score and X for models Yahoo dataset in Figure 5 and those curves on other datasets in Appendix E.
semantically smooth and meaningful, which
RougeL (Xa: Xb, XA) = 2 (Xico XA) + Fice(x5, xx))
(14)
E-VAES
0004 (SCA4))
Rouges.r
11.21
0.22
141/6 2011
0228
032
Figure 5: The curves of Rouge-L. F1-score and for models' interpolation performance on Yahoo.
As shown in Figure the average Fi-score of LCS tends to be lower in the middle than the ends, which indicates that generated sentences tend to not be smooth in the middle, which corresponds to the phenomenon of generation performs the worst as it suffers from posterior collapse, and only generates the same plain sentence;
to holes observed in previous work [25] The vanilla VAE
'ee only consider greedy search for generation this work.
Forr further illustration this phenomenon, provide study in Appendix F.

---

meanwhile, DG-VAE outperforms BN- VAEs, VAES and B- 3-VAEs in the quality of interpolation on the Yahoo dataset as can solve posterior collapse and avoid the hole problem the same time.
In summary, the existing methods for solving posterior collapse in VAEs either have limited effect effectively solve posterior collapse the cost of bringing the hole problem. In contrast, our proposed DG-VAE can effectively solve posterior collapse and avoid the hole problem at the same time, which is demonstrated by the posterior centers spread in latent space and the aggregated posterior distribution consistent with the prior distribution. Furthermore, our proposed DG-VAE outperforms the existing methods in the quality of ditent-guided generation due these improvements in latent space.
5 Discussion
Conclusion In this work, we perform systematic experiments to demonstrate posterior collapse and the hole problem in existing continuous VAEs for text generation. To solve both problems at the same time, we propose a density gap-based regularization on the aggregated posterior distribution to replace the KL regularization in ELBo, and prove it in essence maximizes the ELBo as well as the mutual information between the latent and the input. Experiments on real-world datasets prove the effectiveness of our method in solving both problems and its improvement in latent-guided generation.
Limitation & Future work Both the theory and the ablation study show that the effectiveness of our proposed method depends on the aggregation size [6], which is still limited by the batch size during training. Therefore a promising future direction is to find a solution to break this limit, such like the memory bank mechanism in contrastive learning [38].
Acknowledgments and Disclosure of Funding
This work was supported in part by the State Key Laboratory of the Software Development Environ- ment of China under Grant SKL.SDE-202ZXX11
References
[1] Jyoti Aneja, Alexander G. Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. In Proceedings of the 2021 Annual Conference on Neural Information Processing Systems, pages 480-493, 2021.
[2] Hareesh Bahuleyan, Lili Mou, Hao Zhou, and Olga Vechtomova. Stochastic Wasserstein autoencoder for probabilistic sentence generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4068-4076, 2019.
[3] Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, pages 66-75, 2019.
[4] Sam Bond- -aylor, Adam Leach, Yang Long, and Chris G. Willcocks. Deep generative modelling: comparative review of VAEs, GANs, normalizing flows, energy based and autoregressive models. CoRR, abs/2103.04922. 2021.
[5] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, 2015.
[6] Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. Generating sentences from continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, 2016.
[7] Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In Proceedings of the 4th International Conference Learning Representations, 2016.
10

---

[8] Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomezak. Hyper- spherical variational auto-encoders. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, pages 856-865, 2018.
[9] Adji Dieng, Yoon Kim, Alexander M. Rush, and David M. Blei. Avoiding latent variable collapse with generative skip models. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, pages 2397-2405, 2019.
[10] Yue Ding, Yuxiang Shi, Bo Chen, Chenghua Lin, Hongtao Lu, Jie Li, Ruiming Tang, and Dong Wang. -etemdeterministicc and contrastive variational graph autoencoder for recommendation. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management, pages 282-391. 2021.
[11] Le Fang. Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen, Implicit deep latent variable models for text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3944 3954, 2019.
[12] Qingkai Fang and Yang Feng. Neural machine translation with phrase-level universal visual representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 5687-5698, 2022.
[13] Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach to mitigating KL vanishing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 240-204200 2500,50,200 2019.
[14] Tingchen Fu, Xueliang Zhao, Chongyang Tao, Ji-Rong Wen, and Rui Yan. There are thousand hamlets in a thousand people's eyes: Enhancing knowledge-grone grouuded dialogue with personal memory. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3901-3913 2022.
[15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of 2014 Annual Conference on Neural Information Processing Systems, pages 2672-2680, 2014.
[16] Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taïga, Francesco Visin, David Vázquez, and Aaron C. Courville. PixeIVAE: A latent variable model for natural images. In Proceedings of the 5th International Conference on Learning Representations, 2017.
[17] Serhii Havrylov and Ivan Titov. Preventing posterior collapse with levenshtein variational autoencoder. CoRR, abs/2004. 14758, 2020.
[18] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning basic visual concepts with constrained variational framework. In Proceedings of the 5th International Conference Learning Representations, 2017.
[19] Matthew D Hoffman and Matthew J Johnson. ELBO surgery: Yet another way to carve up the variational evidence lower bound. In Proceedings of 2016 NIPS Workshop on Advances in Approximate Bayesian Inference, 2016.
[20] Diederik P. Kingma, Tim Salimans, Rafal Józefowicz. Xi Chen, Ilya Sutskever, and Max Welling. Improving variational autoencoders with inverse autoregressive flow. In Proceedings of the 2016 Annual Conference on Neural Information Processing Systems, pages 1736-4744, 2016.
[21] Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In Proceedings of the 2nd International Conference on Learning Representations, 2014.
[22] Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. Learn- ing hierarchical priors in VABs. In Proceedings of the 2019 Annual Conference on Neural Information Processing Systems, pages 2866-2875, 2019.

---

[23] Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, and Yiming Yang. A sur- prisingly effective fix for deep latent variable modeling of text. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3601-3612, 2019.
[24] Ruizhe Li, Xiao Li, Guanyi Chen, and Chenghua Lin. Improving variational autoencoder for text modelling with timestep-wise regularisation. In Proceedings of the 28th International Conference Computational Linguistics, pages 2381-2397, 2020.
[25] Ruizhe Li, Xutan Peng, and Chenghua Lin. On the latent holes of VAEs for text generation. CoRR, abs/2110.03318,, 2021.
[26] Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Proceedings of 2004 Text Summarization Branches Out, 2004.
[27] Micha Livne, Kevin Swersky, and David J. Fleet. SentenceMIM: A latent variable language model. CoRR, abs/2003.02645, 2020.
[28] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and lan J. Goodfellow. Adversarial autoencoders. CoRR, abs/1511.05565 2015.
[29] Alban Petit and Caio Corro. Preventing posterior collapse in variational autoencoders for text generation via decoder regularization. CoRR, 0bs/21100.1999 2021.
[30] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[31] Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta- VAEE. In Proceedings of the 7th International Conference on Learning Representa- tions, 2019.
[32] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastie backpropagation and approximate inference in deep generative models. In Proceedings of the 31th International Conference on Machine Learning, pages 1278-1286, 2014.
[33] Danilo Jimenez Rezende and Fabio Viola. Taming VAEs. CoRR, abs/1810.00597, 2018.
[34] Harshil Shah and David Barber. Generative neural machine translation. In Proceedings of the 2018 Annual Conference on Neural Information Processing Systems, pages 1353-1362, 2018.
[35] Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Style transfer from non- parallel text by cross-alignment. In Proceedings of the 2017 Annual Conference Neural Information Processing Systems, pages 6830-6841, 2017.
[36] Ilya O. Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein auto-encoders. In Proceedings of the 6th International Conference on Learning Representations, 2018.
[37] Jakub M. Tomczak and Max Welling. VAE with a vampprior. In Proceedings of the 1 International Conference on Artificial Intelligence and Statistics, pages 1214-1223, 2018.
[38] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non parameettt instance discrimination. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-37423 2018.
[39] Congying Xia, Caiming Xiong, and Philip S. Yu. Pseudo siamese network for few-shot intent generation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2005-2009, 2021.
[40] Haowen Xu, Wenxiao Chen, Jinlin Lai, Zhihan Li, Youjian Zhao, and Dan Pei. On the necessity and effectiveness of learning the prior of variational auto-encoder. CoRR, abs/1908 13452, 2019.
12

---

[41] Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 1500-4513, 2018.
[42] Zichao Yang. Zhiting Hu, Ruslan Salakhutdinov. and Taylor Berg-Kirkpatrick. Improved variational autoencoders for text modeling using dilated convolutions. Proceedings of the 34th International Conference on Machine Learning, pages 3881-3890, 2017.
[43] Kang Min Yoo, Youhyun Shin, and Sang-goo Lee. Data augmentation for spoken language understanding via joint variational generation. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pages 7402-7409, 2019.
[44] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proceedings of the 2015 Annual Conference on Neural Information Processing Systems, pages 649-654 657, 2015.
[45] Tiancheng Zhao, Ran Zhao, and Maxine Eskénazi. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Compitational Linguistics, pages 654 664, 2017.
[46] Qile Zhu, Wei Bi, Xiaojiang Liu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. A batch normalized inference network keeps the KI. vanishing away. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2636-2649, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope? [Yes] We propose a novel regularization for VAEs solve both posterior collapse and the hole problem.
(b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We state the full set of assumptions in the first paragraph of Section 3. (b) Did you include complete proofs of all theoretical results? [Yes] We provide complete
proofs for all theoretical results in Section 3.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We include the code, scripts for downloading data and instructions for reproduction in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See Appendix B.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B.
4. If you are using existing assets (e.g., code, data, models) or curating/releasnng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "e6gW1OGY6-", "reviewer": "Reviewer_rX6Q", "summary": "This paper proposes a unified solution to two problems which can occur when training VAEs - the hole problem (where the aggregated variational distribution fails to fit the prior) and posterior collapse (where the variational distribution becomes the same for every data point and is therefore uninformative).\n\nThe proposed solution is a modification to the VAE's training objective, where the per-data-point KL divergence term is replaced by the KL divergence from the aggregated posterior to the prior. This aggregate KL divergence term is expressed as the sum of KL divergences over the dimensions of the latent variable.\n\nThe authors compare their method to various baselines, all of which were designed to solve the posterior collapse issue. The proposed method appears to address the posterior collapse problem, while suffering the hole problem to a lesser extent than the baselines.", "questions": "- Is your method the first which intends to jointly solve the hole problem and posterior collapse?\n    - If so, this would be a valuable statement to add.\n\nOther questions included in the strengths and weaknesses section.", "limitations": "- The authors do not discuss the limitations of their method.\n    - Arguably, the most important limitation of the proposed method is that the training objective is no longer a lower bound on the true log likelihood of the data. Does this mean that the model is no longer suitable for tasks such as density estimation, out-of-distribution detection, etc. which the vanilla VAE is otherwise useful for?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This is a conceptually simple method which appears to be effective at solving both the hole problem and posterior collapse when training VAEs.\n- The authors do a good job at explaining the hole problem and posterior collapse, as well as the apparent trade-off between the two.\n    - Figure 1 is particularly clear for this.\n- The empirical results appear to outperform the baselines, in terms of having all of the latent dimensions being both 'active' and 'consistent' (as defined in Appendix C), as well as the latent representations and observations having a high amount of mutual information.\n    - The latent space visualisations appear to indicate that the proposed method clearly outperforms the baselines at solving both the hole problem and posterior collapse.\n    - In addition, the proposed method appears to consistently outperform baselines at generating interpolations between sentences.\n- Although this may not be the most significant advance in generative modelling by itself, it does seem to be a piece of work that the community could easily build on in order to make incremental advances in the field.\n\n\nWeaknesses:\n- Although the authors explain the hole problem and posterior collapse well, the explanation of their actual method difficult is to follow. Some examples include:\n    - Throughout, the use of $\\mathbf{x}$ and $\\mathbf{y}$ to refer to instances of the same variable is confusing. Why not just use $\\mathbf{x}$ throughout as is commonly done in the literature, avoiding the introduction of unnecessary notation?\n    - Although it is clear why we would expect the proposed method to solve the hole problem, it is not explained why we would expect it to solve posterior collapse? Surely, as with the vanilla VAE, there is a local optimum where $q_{\\phi}(\\mathbf{z}|\\mathbf{x}) = p(\\mathbf{z}) \\forall \\mathbf{x}$? \n    - When reading the paper from beginning to end in order, it is unclear what the point is of Equation (3) and the paragraph just preceding it. I believe the clarity would be improved were this part moved to the part before Equation (8).\n    - In Figure 1, should the $p_{\\phi}$ terms not be $q_{\\phi}$?\n- The evaluation in the experiments section is not entirely convincing.\n    - There are several missing details, e.g.\n        - Are the results in Table 2 computed on the test set?\n        - How many samples are used for the evaluation?\n        - Why not report the (importance weighted) ELBO taken with a large number of samples, as is commonly done in the literature?\n    - The interpolation task used to measure the ability of the model to do latent-guided generation isn't totally convincing.\n        - The examples shown in Figures 6 and 7 are extremely short sentences, and at least qualitatively it is not clear that the DG-VAE is better than the $\\beta$-VAE.\n        - Why only show examples of the $\\beta$-VAE and not the other baselines?\n\nUPDATE: Increased score post rebuttal.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Pe6Ro9En2Co", "reviewer": "Reviewer_FTv2", "summary": "This paper addresses posterior collapse in VAE and also tries to mitigate the issue with many existing solutions for this which is the trade off with poor fit to the prior.\n\nThe authors propose a novel regularization to substitute the KL regularization in ELBo for VAEs, which is based on the density gap between the aggregated posterior and the prior. Since quantities related to aggregated posterior need for the regularizer depend on the whole dataset are expensive to compute, this paper further changes the objective to consider aggregation over minibatches only. \n\nThis regularizer maximizes the ELBO as well as the mutual information between the input and the latent variable.\n\nFor Gaussian settings, the authors present a regularizer for more aggressive mutual information between the latent variable and data by imposing a regularizer over marginal distributions over each dimension of the latent variable.\n\nThe empirical comparison is done with relevant baselines on text datasets.\n", "questions": "some notes on the presentation:\nLine 53: q(z|x) instead of y, rather x and y are confusing, its the same thing\nIndicator operator used for MI\nLine 82\nText in math is poorly formatted\n", "limitations": "Please address the runtime of the proposed approach.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is sound and the baselines are well chosen covering a range of related work.\n\nThe technical contribution hinges very heavily on the paper by Hoffman et al.'s formulation, but I believe the proposed regularizer is novel.\n\nThe interpolation study shows is interesting and the proposed approach seems to be better than the baselines on the chosen metric for interpolation. However, this metric is not directly related to mutual information between latent variables and text or the diversity one can expect from the samples of the generative model.\n\nMy main concern is that the main empirical results show that baseline/competing models are as good/better than the proposed method -- especially BN-VAE. The only metric BN-VAE does slightly worse is CU, and I am unsure how to interpret this unconventional metric.\n\nThe proposed approach also is more expensive due to the quantities involved with the aggregated posterior. A comparison of runtime with respect to other approaches would be helpful.\n\nSome related work:\nVAE with a VampPrior, Tomczak et al.\nLearning to Explain: An Information-Theoretic Perspective on Model Interpretation, Chen et al.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FBUX9rf-Ta", "reviewer": "Reviewer_xN7d", "summary": "This paper focuses on a well-known problem in variational auto-encoders: the learned latent representation often has \"gaps\" in its prior, i.e. there are latent samples with high prior PDF values that fail to generate coherent outputs.\n\nThe authors analyse the problem and propose a novel objective to fix the issue that combines the ELBO with a mutual information term. Importantly, the proposed surrogate objective is well-motivated.", "questions": "- is the proposed objective still a lower bound to the log-likelihood?\n- l. 123: I don't understand what is the point of stating that \"we only consider  z \\in ...\" => q(z) is Gaussian, so we can't have q(z) = 0 anyway, no?\n- in the end, the objective is only a combination of ELBo + mutual information? Could you explain a little bit why is this novel? Especially, what is the difference with [1]?\n\n[1] MIM: Mutual Information Machine (Micha Livne et al.)", "limitations": "nothing to report\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- important problem in deep generative modeling\n- contribution is interesting and well-motivated\n- good experimental section\n\n**Weaknesses**\n\n- not sure to understand whether the proposed objective is still a lower bound to the log-likelihood\n- weird notation, e.g. the use of X and Y variables, it should be X only, I don't understand why to introduce a second RV for the same observation. Also, q(n) is the same as p(n), no? Why a proposal distribution here? It is fixed(?)\n\n**Missing citations**\n\nThe following citations are missing and they provide a broader view on how researchers tackle the problem in the NLP community:\n- SentenceMIM: A Latent Variable Language Model (Micha Livne, Kevin Swersky, David J. Fleet)\n- A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text (Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, Yiming Yang)\n- Preventing posterior collapse in variational autoencoders for text generation via decoder regularization (Alban Petit, Caio Corro)\n- Preventing Posterior Collapse with Levenshtein Variational Autoencoder (Serhii Havrylov, Ivan Titov)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept