On the Latent Holes 🧀 of VAEs for Text Generation
Authors: Ruizhe Li
Xutan Peng
Chenghua Lin
Venue: ICLR 2022 Submitted (2022)
Keywords: Latent Discontinuity
Variational Auto-Encoder
Natural Language Generation
Generative Model

Abstract: In this paper, we provide the first focused study on the discontinuities (aka. holes) in the latent space of Variational Auto-Encoders (VAEs), a phenomenon which has been shown to have a detrimental effect on model capacity. When investigating la- tent holes, existing works are exclusively centred around the encoder network and they merely explore the existence of holes. We tackle these limitations by proposing a highly efficient Tree-based Decoder-Centric (TDC) algorithm for latent hole identification, with a focal point on the text domain. In contrast to past studies, our approach pays attention to the decoder network, as a decoder has a direct impact on the model’s output quality. Furthermore, we provide, for the first time, in-depth empirical analysis of the latent hole phenomenon, investigating several important aspects such as how the holes impact VAE algorithms’ performance on text generation, and how the holes are distributed in the latent space.

Under review as a conference paper at ICLR 2022
ON THE LATENT HOLES OF VAES FOR TEXT GENERATION
A nonyymous authors
Paper under double-blind review
ABSTRACT
In this paper, we provide the first focused study on the discontinuities (aka. holes) in the latent space of Variational Auto-Encoders (VAEs), phenomenon which has been shown to have detrimental effect on model capacity. When investigating la- tent holes, existing works are exclusively centred around the encoder network and they merely explore the existence of holes. We tackle these limitations by propos- ing a highly efficient Tree-based Decoder-Centric (TDC) algorithm for latent hole identification, with focal point on the text domain. In contrast to past studies, our approach pays attention to the decoder network, as decoder has direct im- pact on the model's output quality. Furthermore, we provide, for the first time, in-depth empirical analysis of the latent hole phenomenon, investigating several important aspects such as how the holes impact VAE algorithms' performance on text generation, and how the holes are distributed in the latent space.
INTRODUCTION
Variational Auto-Encoders (VAEs) are powerful unsupervised models for learning low-dimensional manifolds (aka. a latent space) from non-trivial high-dimensional data (Kingma & Welling [2014] Rezende al., 2014). They have found successes a number of downstream tasks across different
application domains such as text classification (Xu et al. (017). transfer learning (Higgins et al.)
20176), image synthesis (Huang et al. 2018 Razavi et al. 2219). language generation (Bowman
et al. 2016 He et al.] 2019), and music composition (Roberts et al. 2018).
Various effort has been made to improve the capacity of VAEs, where the majority of the extensions are focused on increasing the flexibility of the prior and approximating posterior. For instance, Davidson et al. (2018) introduced the von Mises-Fisher (vMF) distribution to replace the standard Gaussian distribution; Kalatzis et al. (2020) assumed Riemannian structure over the latent space by adopting the Riemannian Brownian motion prior. A few recent studies attempted to investigate the problem more fundamentally, and revealed that there exist discontinuous regions (we refer to them as "latent holes" following past literature) in the latent space, which have a detrimental effect on model capacity. Falorsi et al.; (2018) approached the problem from a theoretical perspective of manifold mismatch and showed that this undesirable phenomenon is due to the latent space's ss topological incapability of accurately capturing the properties of dataset. Xu et al. (2020) examined the obstacles that prevent sequential VAEs from performing well in unsupervised controllable text generation, and empirically discovered that manipulating the latent variables for semantic variations in text often leads to latent variables to reside in some latent holes. As result, the decoding network fails to properly decode or generalise when the sampled latent variables land in those areas.
Although the works on investigating latent holes are still relatively sparse, they have opened up new opportunities for improving VAE models, where one can design mechanisms directly engineered for mitigating the hole issue. However, it should be noted that existing works (Falorsi et al.) 2018 Xu et al.) 2020) exclusively target at the encoder network when investigating holes in the latent space, and they merely explored its existence without providing further in-depth analysis of the phenomenon. has also been revealed that the hole issue is more severe on text compared to the image domain, due to the discreteness of text data (Xu et al. 2020).
In this paper, we tackle the aforementioned issues by proposing a novel tree-based decoder centric (TDC) algorithm for latent hole identification, with focus on the text domain. In contrast existing

---

Under review as a conference paper at ICLR 2022
works which are encoder-centric our approach is centric to the decoder network, decoder has direct impact on the model's 's performance, c.g., for text generation. Our TDC algorithm is also highly efficient for latent hole searching when compared to existing approaches, owing to the dimension reduction and Breadth-First Search strategies. Another important technical contribution is that we theoretically unify the two prior indicators for latent hole identification, and evidence that the one of Falorsi et al. (2018) is more accurate, which forms the basis of our algorithm detailed in $ 3
In terms of analysing the latent hole phenomenon, we provide, for the first time, an in-depth em- pirical analysis that examines three important aspects: (i) how the holes impact VAE models' per- formance on text generation; (ii) whether the holes are really vacant. i.., useful information is not captured by the holes at all; and (iii) how the holes are distributed in the latent space. To validate our theory and to demonstrate the generalisability of our proposed TDC algorithm, we pre-train five strong and representative VAE models for producing sentences, including the state-of-the-arr model. Comprehensive experiments on the text task involving four large-scale public datasets show that the output quality is strongly correlated with the density of latent holes; that from the perspective of the decoder. the Latent Vacancy Hypothesis proposed by Xu et al. (2020) does not hold empirically; and that holes are ubiquitous and densely distributed in the latent space. Our code will be made publicly available upon the acceptance of this paper.
PRELIMINARIES
2.1
VARIATIONAL AUTOENCODER
VAE is generative model which defines a joint distribution over the observations x and the la- tent variable ž, i.e., p(x, ø) = p(x[2))(()) Given a dataset X = | with N i.i.d. datapoints, need to optimise the marginal likelihood p(X) # Σ, J p(xx ()) over the entire train- ing set. However, this marginal likelihood is intractable. A common solution is to maximise the Evidence Lower BOund (ELBO) via variational inference for every observation x:
C( = (x)(( (logPo (x 2)) KK. (Zx()))) Ip(ž).
(1)
where g (2|x) is variational posterior to approximate the true posterior p(ž[x). The variational posterior l xx) (aka. encoder) and the conditional distribution (x(i) (aka. decoder) are set up using two neural networks parameterised by 6 and 0, respectively. Normally, the first term in Eq. ( is the expected data reconstruction loss showing how well the model can reconstruct data given a latent variable. The second term is the KL-divergence of the approximate variational posterior from the prior, i.e., a regularisation forcing the learned posterior to be as close to the prior as possible.
2.2 EXISTING LATENT HOLE INDICATORS
To our knowledge, there are only two prior works which directly determine whether latent region is continuous or not. One work formalises latent holes based on the relative distance of pairwise points taken from the latent space and the sample space (Falorsi et al., 2018). Concretely speaking, given a pair of vectors 2; and Zi4. which are closely located on a latent path, and their corresponding samples x': and x' 1 in the sample space, a latent hole indicator is computed as
Jurr() (t) = D sampleeX' ;, x (211)/1///// / Diteett (2, Zt+1).
where D sample and D latent respectively denote the metrics measuring the sample and latent spaces (NB: D Daatent is arbitrary metric, e.g., the Euclidean distance and Riemannian distance). Falorsi (et al. (2018) focused on the image domain and utilised Euclidean distance for both spaces. Based on the concept of Lipschitz continuity, Falorsi al. 2018) then proposed to measure the continuity of a latent region as follows: under the premise that 2,+1 does not land on a hole, 2, is recognised as belonging to a hole if the corresponding (1.p(1) is a large outlier
Another line of work (Xu et al. 2020) signals latent holes based on the so-called aggregated poste- rior, with focus on sequential VAES for language modelling. This approach interpolates series of vectors on a latent path at a small interval, and then scores the i-th latent vector 2, as
3 (i) := EM NL, /
(3)
Unless otherwise stated, outliers are detected by comparing the subject data point with a fixed bound, which is pre-determined based on a percentile of all data points.
2

---

Under review as a conference paper at ICLR 2022
where (() is the sample of the posterior distribution of the t-th out of the total M training samples, e.g., when studying holes on the encoder side, this distribution can be computed using qs (z x) in Eq. (Xu et al. 2020). ( serves as the reference when calculating the Negative Log-Likelihood (NLL). After all the interpolated vectors on the latent path are traversed, similar the first method,
vectors with large outlier indicators (JAGG) are identified as in latent holes.
We note that these two indicators actually stem from different intuitions. For JrIP, there an under- lying assumption that mapping between the sample and latent spaces should have good stability in terms of relative distance change in order guarantee good continuity in the latent space. In contrast, JAG is based on the belief that small perturbations on the non-hole regions should not lead to large offsets on the absolute dissimilarity between posterior samples z and the sample 2, and hence the calculation is performed only in the latent space and only around one single latent posi- tion. While seemly distinct, we show that (in $ 3.2) both indicators actually have tight underlying connections and can be unified in a shared mathematical framework. Moreover, the first indicator (JLP) is proofed to be more comprehensive than the second (TAGG) and thus can reduce false nega- tives when identifying holes in the latent space. This forms the basis of our algorithm in $ 3 which is the first attempt to identify VAE decoder's latent holes for language generation.
METHODOLOGY
In this section, we describe our tree-based decoder-centric (TDC) algorithm for latent hole identifica- tion. which consists of three main components. We first introduce our heuristic- based Breadth- First Search (BFS) algorithm for highly efficient latent space searching ($ [.1I.. We then theoretically proof, for the first time, that existing holes indicators can be unified under the same framework and that JLIP is more suitable choice for identifying latent holes ($ 3.2 Finally, we extend FLR
the text domain by incorporating the Wasserstein distance for the sample space ($ 3.3)
3.1 TREE-BASED DECODER- CENERICC LATENT HOLE IDENTIFICATION
As discussed earlier, existing works for investi- gating latent holes of VAFs all exclusively fo- the encoder network (e.g.. Falorsi et al. (2018); Xu et al. (2020), and they cannot be trivially applied to the decoders (which play ul- timately important roles on generation tasks) due to metric incompatibility. especially for VAEs in the text domain (see detailed discus- sion in 3.3). Another drawback of existing indicators that they have very limited effi- ciency. Theoretically, their time complexity for traversing a d-dimensional latent space with I interpolations per path is O(I)) at the optimal efficiency, which is computationally prohibitive as typically and / are larger than 30 and 50 for VAES in practice. Each path is parallel to one axis of the traversed latent space" Empiri- cally, we observe that even finding handful of latent holes has been shown to be difficult for existing methods (Falorsi et al., 2018 Xu et al. 2020). Therefore, we tackle both challenges by proposing a highly efficient algorithm for decoder-centric latent hole identification. The pipeline of our TDC algorithm is described in Algorithm| [ and we give detailed discussion as follows. For the visualisation of TDC's 's work- ing process in practice, please see Fig.
PCAIZ...))
0.08
0.06
0.02
0.00
-1.24
-1.22
-1.20
-1.34
-1.18
-1.38
-1.16
-1.42
-1.40
Figure 1: A cubic fence C' in the latent space of Vanilla-VAE trained on the Wiki dataset, with d, set to facilitate visualisation (cf. $ 4 C, whose 12 edges are illustrated by dashed lines, surrounds the dimensionally reduced expectation of three encoded training samples. Solid lines within the cube indicate the traversed paths.
For example, in 3-dimensional latent space with 4 interpolations per path, as each point is the intersection of 3 paths, 4 = 64 points in total determined. The space is then equally divided into 64 cubes.
3

---

Under review as a conference paper at ICLR 2022
Algorithm 1 TDC for latent hole identification
Input: Trained VAE model w/ d-dimensional latent space; original training set X; reduced dimension d.; the desired number of detected vectors in latent holes Nhole
Output: Zhole
1: Z + 0; V Vrain f- 0: Di Dirain +- 0
2: Vx X. Z &- Z [2]
II i is the encoded I/ [B(-)) 3 is the expectation I1 a (-) is the standard deviation // Dimension reduced from to dr 6: C E a randomly-picked closed cube which contains d, vectors of Z', w/ edges
3: Vx X, Verain f- V Vrain U (E(x))
Vx X, Drain Daran U {a(x))
+ PZA(Z)
parallel to d, dimensional axes
Zholo <- 0; Z hub 0: II
while [Zaole| S Nhole do
9:
if Z' bub
0 then
10:
Z' hub + fa random point in C} end if
Restart BFS
11:
12:
II unvisited line segments: passing through vectors in Z/ hub A parallel
the d, dimenssons A w/ endpoints on C
// Depth increases by one
13:
Z' hub 6
14:
for each path (cf. $222 in 11 do
15:
Sample 2; on path at an interval 0.01 min( (urai))
16:
Vi. 2, f- INVESSE.APP))
17:
Vi, decode 2, to compute 3(i) w/ Vuain and Derain if ((1) is an outlier then
1/ Cf. Eq.
in
2.2
18:
19:
Zhole + Zhole U { Z' hub Z' Z'hob U { end if
20:
21:
end for end while
22:
Dimensionality Reduction. One problem for the current indicators is their limited searching ca- pacity (as evidenced by their time complexity O(I²)) over the target space. Concretely speaking, both indicators rely on signalling latent holes through I-dimensional traversal, but a latent space normally has dozens of dimensions to guarantee modelling capacity. To alleviate this issue, after feeding all training samples in X to the forward pass of trained VAE and storing the encoded la- tent variables in Z (Step 2), we perform dimension reduction using Principal Component Analysis (PCA) (Jolliffe 1987) and conduct search in the resulting d, dddimensionnal space instead of the original d-dimensional space (Step 5). We further save the mathematical expectation and standard deviation of each training sample in V train (Step 3) and Draain (Step 4), respectively. In addition, instead of traversing unconstrained paths like past studies, we only visit latent vectors through paths parallel the d, dimensions (see Step 12 and the next paragraph). Such a setup is based on the intuition that these top principal components contain more information about the latent space, and thus they are more likely be useful when capturing latent holes.
Initialising Infrastructures for Search. To further boost efficiency, we propose to conduct a search on tree-based structure within pre-established cubic fence. To be more concrete, at Step 6 we first locate cube C' which surrounds d, encoded training samples from Z' (i.e., Z after dimension reduction). These d, posterior vectors serve as references when analysing the distribution of latent hole (cf. $ 4.22. We restrict the edges of the dd-dimensional C' to be parallel to the d, latent dimensional axes and treat as the range of our search. Next, we regard each sampled latent vector after dimension reduction ž node, and in order expand the search regions rapidly, to visit these nodes following a BFS-based procedure (Skienal 2008). Therefore, our algorithm maintains set Z' Zubb keep track of all untraversed hub nodes, where the root (aka. the first hub node) is randomly initialised C' (Step 10). For each hub node, we define d, orthogonal paths, each of which is line segment that passes through the hub node and is parallel one dimension. At Step 12, we log paths having not been previously processed in set 11.
need
Identifying Latent Holes. Following the principle of BFS, the TDC algorithm processes all nodes at the same depth (i.e.. all nodes on the paths in II) before moving to the next depth. On each path, following Falorsi al. 2018) and Xu et 22020). at Step 15 we sequentially sample series
"To avoid cherry-picking and parameter deredundancy, we select d, the number of contained x' E Z'.
4

---

Under review as a conference paper at ICLR 2022
of z To ensure the sampling is fine-grained, we set the interpolation interval at the 0.01 times minimum standard deviation of all elements in Duaan (see Step 4). After that, we utilise the inverse transformation of PCA (Developers) 2011) to reconstruct ž to the original d-dimensional latent space at Step 16 and generate output samples through the decoder at Step 17. One core question raised is how choose the indicator J between the existing ones which seem quite distinct (cf. $ (.2). We eventually select the scheme of Falorsi et al. (2018) (i.e., JLIP in Eq. (2) and further adopt the Wassertein distance as the metric for the sample space. Detailed justifications are provided in $ 3.2 and $ $3.3 respectively. After all paths in II are investigated, our algorithm pushes the tree search to its next depth by reloading the emptied Z' hub with newly identified latent variables in the holes (Step 19). The motivation for treating them new hub nodes comes from our observation that holes tend to gather as clusters. In case that no hub node is added, which suggests the end of current BFS, TDC will bootstrap another tree by randomly picking a new root. The algorithm halts when more than Noole holes are identified.
In practice, we find that our tree-search strategy with dimension reduction not only boosts the ef- ficiency from an algorithmic perspective, but is also highly parallelisable by nature and thus can reduce computational time. In theory, the time complexity of TDC can be reduced to OI,,"), where d, can be as small as 3 (ef. $ $ and L, is typically less than 2, thanks to the parallelism of our al- gorithm. In experiments, when the device is equipped with a Nvidia GTX Titan-X GPU and a Intel 19-9900K CPU, in most cases TDC (with d, 8) can return more than 200 holes in less than 5 min- utes, whereas the methods of Falorsi et (2018) and Xu et al. (2020) often need at least 30 minutes to find hole in the same setup our TDC.
3.2 PICKING INDICATOR FOR TDC
Obviously, the indicator used by TDC (Step 17 in Algorithm 1 plays crucial role as it directly affects the effectiveness of identifying latent holes. By analysing the two existing indicators in $ [.2] we demonstrate that (I) although developed under different intuitions, they can actually be unified within a common framework; (2) although both indicators have been tested successfully in validating the presence of latent holes, the indicator of Falorsi al. (2018) (JLIP) is more accurate as it has better completeness and is thus more suitable to our algorithm. To begin with, we prove the following lemma:
Lemma NLL(x, P), the NLL of a data point x under a multivariate normal distribution with independent dimensions P can be numerically linked with De, the so-called Generalized Squared Interpoint Distance (Gnanadesikan & Kettenring 1972), as
1 D 11) + 8(k)) s.t. P P=N(M,KP))
NLI.(X, P)=
(4)
where μ denotes the mean, Kp denotes the covariance matrix, DG is the so-called Generalized Squared Interpoint Distance (Gnanadesikan & Kettenring 1972), and 5(-) is single value function.
Proof. See Appendix A
Based on this lemma, we find that the right hand of Eq. (3 is numerically equivalent to directly calculating NLL(2;, ()) for posterior Z(o,, yielding
M
()) = Σ
1
_ Do (2,,
DZ.2., n (0) + 8 8(K2
| /M s.t. z = =N(1(0) KZ()))
(5)
Note that as ((() deterministic, 5(Kz settles as a constant term. By integrating Eq. 2 w.l.o.g.
can theoretically prove that if a latent position is signalled to be discontinuous by the indicator of Xu et al. (2020), will be identified using that Falorsi et al. (2018).
Proof. See Appendix B
Apart from theoretical proof, empirically we also observe cases showing JuP has better complete- ness than J JAGG- We present one toy example in Appendix C To conclude, FLI should be adopted to reduce the false-negative rate of TDC.
"Our implementation parallelises the computation process at two hierarchies: different paths at the same BFS depth and different z' the same path.
5

---

Under review as a conference paper at ICLR 2022
3.3 PICKING SAMPLE SPACE METRIC
We find that it is impossible to directly apply the indicator of Falorsi et al. (2018) (OLie) for VAEs for NLP tasks: the Euclidean distance is used as D sample in the original study which is on vision VAEs, but it cannot be used to measure the distance between sentence One straightforward solution is to directly follow Xu et al. (2020) who select NLL, a long-standing and popular metric in past VAE studies on NLP tasks (Bowman et al. 2016 Fu al., 2019: Zhu et al. 2020). However, it does not make a valid metric for the decoder of VAEs for language generation. To be more concrete, while on the encoder side NI.L can be calculated d (7)x) in Eq. ( (Xu et al. 2020) and is thus normal and thus has a metric- numerical equivalent NLL (cf. the prooted lemma in $(2.2, on the decoder side the posterior distribution of output sentence is generally computed by a Logsoftmas layer in practice and is thus no longer normal. Instead, coupling the logso ftmax layer with NLL yiedd cross-entropy (Contributors 2019), as
O( DKL (PII))
where P and Q are two probability distributions and (P) is the entropy of P. It is obvious that 5PP.C Q) does not qualify as a statistical metric, because it does not not satisfy symmetry nor Triangle In- equality. A workaround which adopts the symmetric cross entropy (Wang et al. 2019) and replaces KL-divergence with the positive squared root of its smoothed version, JS JS-divergence, can somehow alleviate the issues (Osán et al., 2018). Nonetheless, the resulting formula may dramatically lose its measurement capacity when there is no overlap between P and Q (Lin 1991) (which is common when testing VAE for language generation) and is thus unsuitable neither.
Finally, we refer to the Wasserstein distance of finite first moment as our final candidate:
inf Dwi(vp, = TEP(P~Ur.Qu)) Q~vo)) B(P.o)-rIIP.(Il.
where P(P ~ Up, Q ~ v2) is set of all joint distributions of (P.Q) with marginals up and vo respectively. Dw1 has been adopted in large body of recent VAE studies, such as Chewi et al. (2021); Tolstikhin et al. (2018); Wu et al. (2019). Moreover, to further enhance efficiency, following Patrini et al. (2020), we select the lightspeed Sinkhorn algorithm (Cuturi 2013) to compute Dw1-
4 EMPIRICAL STUDIES
In this section, we describe our experiment for validating the effectiveness of the proposed TDC algorithm. We first describe our setup, followed by three empirical studies investigating the impact of latent holes on text generation, the vacancy of holes, and how the holes are distributed.
4.1 EXPERIMENTAL SETUP
Models. To demonstrate the generalisability of our proposed TDC algorithm, we pretrain five strong and representative VAE models for language generation, including the state-of-the-art iVAE model: Vanilla-VAE (Bowman et al. (2116). which uses LSTM and KL annealing for mitigating the posterior collapse issue; 33- V-E (Higgins et al. 201 which utilises an adjustable 3 to balance the reconstruction loss and the KL term; Cyc- VAE (uu et al. 2019), which employs cyclical annealing for the KI. term; iVA (Fang et al. 2019). which replaces the Gaussian-based posteriors with the sample-based distributions; BN-VAE (Zhu et al. 2020), which leverages the batch normalisation for the variational posterior's parameters.
Datasets. We consider four large-scale datasets, three of which have been commonly used in pre- vious studies for testing VAEs on the language generation task: Yelp15 (Yang et al. (017), Ya- hoo (Zhang et al., 22055] Yang et al. 2017). and a downsampled version of SNLI (Bowman et al., 2015 Li et al.] 2019). We additionally constructed dataset (called Wiki) by downloading the latest English Wikipedia articles and then randomly sampling 1 % sentences from the whole set. The size of Wiki is 10 times larger than other datasets and it contains more training samples which can cover more areas of the latent space during training VAEs. For Yahoo, Yelp15 and SNLI, their training and validation sets are all 100K and 10K, respectively. For Wiki, the training and validation sets are 1. 1.3M and 141K, respectively.
'In principle, by simply adopting metrics such Fuclidean distance, TDC can also be applied on VAFs for image generation. We will explore this direction in the future.

---

Under review as a conference paper at ICLR 2022
Hyper-parameter Settings. We adopt the official code of each tested models and apply the same pretraining syper-parameters to all models. To be concrete, the encoders and decoders of all models constructed using one-layer LSTM with 1024 hidden units and 512D word embeddings. The dimension of the latent space is 32. KI. annealing (Bowman et al. 2016) is applied to all models, and the scalar weight of the KI. term linearly increases from 0 to 1 during the first 10 epochs. Dropout layers with a probability 0.5 are installed on the encoder's s both input-to-hidden and hidden- to-output layers, All baselines are trained with Adam optimiser with an initial learning rate of 8c-4. Parameters of all models are initialised using uniform distribution U (-000,, 0.01) except for word embeddings with O 0.1). The gradients are clipped at 5.0. During training, we set patience at 5 epochs, and adopt early stopping based on Perplexity (PPL) with standard validation splits. For 3-VAE and BN-, the corresponding /3 and y are set at 0.4 and 0.7, respectively.
Configurations of the TDC Algorithm. As discussed earlier, the dimensions of the original latent space 32. When performing dimension reduction, we experiment with d, = {3, 4, 8) for all setups. Empirically, we observe that results for different d, setting show very similar trends. We report the results based on d, = 8 in the main body and provide the results for other settings in Appendices E, F and G When computing our hole indicator (Eq. (2) we follow Falorsi et al. (2018) and adopt the Euclidean distance for Datent (NB: for sample space (D (Dammle) we adopt the Wasserstein distance as discussed in $ (.3). Following [Hoallin et al. (1986). at Step 18 of TDC we adopt the popular Inter- tuter-Qurttile Range measure that defines large outliers as data points falling above Q3 + 1.5 (Q3 Q1). where Q1 and Q3 respectively denote the lower and upper quartile. In all runs, we set Nhole = 200, i.c., the program halts when more than 200 holes are identified and we store the first 200 holes in Zuole for evaluation. For stochastic analysis, we run TDC 50 times for each setup, yielding 50 x 200 = 10K latent holes per setup. Recalling that there are 5 models and 4 datasets, we totally have 20 setups.
4.2 RESULTS AND ANALYSIS
Impact of Latent Holes on Text Generation. In this experiment, we investigate how latent holes impact VAE models' performance on text generation. To our knowledge, this is the first such study as prior works (Falorsi et al.] 2018: Xu et al. (020) merely explored the existence of holes and their schemes are incapable to discover a sufficient amount of holes for quantitative analysis due to algorithm inefficiency (cf. $ 3...
Our analysis is established on the correlation between models' performance on text generation and the density of latent holes As discussed in $ 4. we identify 10K holes for each setup using our TDC algorithm, based on which 10K sentences were decoded. We then calculate the average PPI. of those 10K sentences using a pre-trained GPT model (Radford et al., 2018) following the practice of Dathathri et al. (2020). As for the density estimation of latent holes, we utilise the average number of paths traversed before the number of identified holes reaches the algorithm halting threshold Nhole = 200. Intuitively, the fewer paths visited, the denser the holes are distributed, and vice versa.
Fig. 2 shows the average PPL versus the num- ber of paths traversed (when reaching 200 iden- tified holes) for each setup. It can be observed that there is strong negative correlation be- tween the average PPL. (lower the better) and the number of visited paths, where the corpus-
2500
6.. Wiki: r - 1.0. -0.99
Vanila-VAE
Cyo-VAE
B-VAE
2000
BN-VAE IVAE
Average perplexlly
wise Spearman's 's correlation coefficient тa is
1600
consistently below or equal -0.70. It can also be observed that the Person's 's correlation coef- ficient rp is below -0.72 for all datasets, show- ing a certain degree of linearity for the correla- tion. summary, the above observations verify the intuition that denser latent hole distribution leads to higher average PPL, and hence worse performance of VAEs for text generation.
SNLI: 1-- -.7. 0.72
1000
Yahoo: r.- 0.9.6
0,77
Yelp15: 6= -10,
90
100
110
120
130
140
150
of paths Iraversed TDC halls
Figure 2: Average PPL and # of paths traversed until > Nhole holes are identified. Correlation co- efficients r, and rp are marked corpus-wisely.
Corpus-wiselyy we notice that models trained on the Wiki dataset, i.e., our largest training dataset, do not seem to yield improvement for hole reduction when comparing to the much
7

---

Under review as a conference paper at ICLR 2022
ePCAl
E Viue
Van
Cys-WEE
BN-WAD
80
90
of traversed path
30
50
50
15
10
10
26
20
6rm)
20w5
3.2
4..5
3 3.2 4645
# of identified holes
Figure 3: Visualisation of the latent space of the Vanilla-VAE (trained on the Wiki dataset). Please Ap- pendix for other setups.
Figure 4: Distribution of quantity of identified holes per
latent path for models trained on the Wiki dataset when d 8. Results for other datasets are in Appendix G
Table 1: Average PPL (divided by 1K) of sentences decoded via vectors of HOLE, NORM, and RAND in all setups. indicates the PPL of a model via NORM significantly lower than via HOLE
(p .05); t indicates the PPL of a model via RAND significantly larger than via HOLE (p .005).
Velp15
SNLT
HOLE
NORM
RAND
HoLE
NORM
RAND
HOLE
NORM
RAND
HOLE
NORM
RAND
Vanilla-VAF
0.428
0.386
18.241
0.872
0.831
18.736
1.569
1.5291
41.247
2.443
2.357
5.377*
Cyc
0.3391
18.293
0.741
0.704
18.576
1.1291
41.026
1.856
5.354
B.VAE
0.362
0.356
18.349
0.756
0.710'
19.027
1.133
1.0687
40.781*
1.640
1.587'
5.338*
BN-VAB
0.348
0.303*
18.234
0.561
0.527
20.343
0,995
0.9471
40.774
1.095
1.041"
5,3471
IVAEs
0.298
0.294
18.211
0.541
0.519
18.356
0.975
0.911
40.692
1.090
1.039"
5.320
smaller datasets such as Yelp15. Furthermore, sentences decoded by models trained on Wiki have lower quality than those decoded by the corresponding models trained on Yelp15 and Yahoo. One plausible explanation is that the complexity (e.g.. topic coverage) of datasets plays a more important role than the corpus size when training VAES for language generation. For instance, while SNLI contains the same number of sentences as Yelp15 and Yahoo, models trained on SNL.I are substan- tially inferior to the models trained on the other two datasets in terms of average PPI. Manually examining the datasets reveals that the topics covered topics in Yelp and Yahoo datasets are less diverse than that of SNLI and Wiki, e.g., SNLI was constructed based on Flickr30k (Young et al., 2014), which includes captions for real-world images across wide range of categories.
Probing the Vacancy of Latent Holes. The previous experiment empirically shows that latent holes indeed have detrimental effect VAEs's generation performance. A recent study (Xu) ct al. 2020) proposed the so-called Latent Vacancy Hypothesis, assuming holes are vacant with no meaningful information encoded. This motivates Specifically, we conduct analysis by comparing the sentences decoded by latent vectors from an untrained decoder and by the hole vectors from a VAE decoder trained following the setup in $ 1 For completeness, we also show the sentence decoded by normal (not in hole) vectors from trained VAE. To summarise, we consider three different types of vectors. (1) Hole vectors (HOLE), those being investigated in our previous experiments. (2) Normal vectors (NORM), sampled from the continuous regions near a hole, i.e., Zi+1 a normal vector if Z, is identified be in a hole in JL.P. (3) Vectors from the latent space of an untrained VAE (RAND). For controlled analysis, we randomly initialised VAE model and pick latent vectors whose coordinates are the same as those of HOLE vectors. As this VAE is untrained, its latent vectors should carry zero information by nature.
further probe the vacancy of latent holes.
We compute the PPL of the sentences generated by the vectors of each of the above categories. As expected, results in Tab. D show that the sentences decoded via HOLE vectors are significantly inferior to those via NORM vectors in almost all setups tested (two-tailed t-test with Bonferroni correction (Dror et al. 2018); p < .05). It can also be observed that sentences decoded via HOLE vectors are a lot better than the random output generated via the RAND vectors (p < .005). This observation suggests that the Latent Vacancy Hypothesis proposed by Xu et al. (0020) does not
8

---

Under review as a conference paper at ICLR 2022
hold empirically, i.e., the regions containing HOLE vectors are not vacant, which do capture some information from the training corpus.
Finally, we qualitatively analyse some sentence examples generated by different types of vectors, as shown in Tab.2] First, we observe that although topologically adjacent in the latent space, HOLE and NORM vectors are decoded into completely irrelevant sentences semantically, indicating that holes, due to severely harming the smoothness of latent continuity, do have a detrimental effect on model's generation quality. Second, it can be observed that the output sentences generated via RAND vectors are neither syntactically correct, nor making any sense semantically. In contrast, although sentences decoded via HOLE vectors tend to have problematic word matching and contain content which is against common sense, at least they still follow basic grammars in most cases, which once again verifies that HOLF vectors contain some useful information. Based on this finding, one implication of the future work is to introduce a novel regularisation term in the objective function and utilise the detected latent holes to regularise the latent space. In addition, TDC is a plugin for other existing VAE models. During training, TDC can be regarded
data augmentation approach to treat the detected latent holes as negative samples under contrastive learning framework.
The Distribution of Latent Holes. Finally, we explore how the latent holes are distributed in the latent space. While a prior study (de Haan & Falorsi, 2018) proposed a theoretical hypoth- esis that latent holes should be densely distributed, has never been investigated empirically.
We visualise one of TDC in Fig. [ As described in 3.I C' is the mini- mum cube which can surround the 3 en- coded training samples on a local latent region and thus spans quite narrowly (with a side length being around 0.1, while the width of the latent space is more than 5). However, even in this small search space, TDC still success- fully halted and identified more than 200 (defined by Nhole. cf. $ 3.I la- tent holes, showing that the distance between these holes is tiny and their distribution is very dense. Moreover, all these latent holes are detected by traversing only 85 paths, meaning that more than 2 latent holes exist on each path, on average. Similar finding can be obtained in Fig. 4 (we further inves- tigate the fine- grained quantity distribu- tion of identified holes per latent path in Appendix GG. In Fig. 3 holes look ubiquitous in the entire latent space, and again we can see that in the 50 ex-
Table 2:
Examples of sentences decoded via vectors of HOLE, NORM, and RAND. More examples of different setups are given in Appendix [
Vanilla-VAE. VAE x SNL/
HOLE the bridge was an old gentleman. NORM married couple is resting. RAND waling speedo ever vehicle birhhous supports tahoe va-
cant commute
HOLE a crowd smiles at people..
NORM old man plays with his dogs. RAND inspect rioting shivering entrance back-to-back seeker wheeling
TVALI x Yahoo
HOLE 't's JUNK to do it or you just put home sick the back.
NORM i 'm thinking of buying the _UNK on the internet from pennsylvania.
RAND
drin ;-Irb- parker vastly san ripped fountain tais com- pared gratuit
HOLE this is not place of all or more specifically my life. NORM is that what you want do when your _UNK exceeds ? RAND selves t-mobile sad nondescript up-sell dominos con- cern newly
plored regions (the spaces which have been surrounded by C' of each run of TDC), the identified latent holes are very close to each other and even form clusters.
CONCLUSION
In this paper, we provide a focused study on the discontinuities (aka. holes) in the latent space of VAEs, a phenomenon which has been shown to have detrimental effect on model capacity. In con- trast to existing works which only study on the encoder network but merely explore the existence of holes, we propose a highly efficient tree-based decoder-centric (TDC) algorithm for latent hole iden- tification. Comprehensive experiments on the language generation task show that the performance of text generation is strongly correlated with the density of latent holes, that from the perspective of the decoder, the Latent Vacancy Hypothesis proposed by Xu et al. (2020) does not hold empirically; and that holes are ubiquitous and densely distributed in the latent space.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, 2015.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowiez, and Samy Bengio. Generating sentences from continuous space. In 20th SIGNLL Conference on Computational Natural Language Learning. CoNLL 2016, pp. 10-21, Association for Computational Linguistics (ACL), 2016.
Sinho Chewi, Julien Clancy, Thibaut Le Gouic, Philippe Rigollet, George Stepaniants, and Austin J. Stromme. Fast and smooth interpolation on wasserstein space. In International Conference on Artificial Intelligence and Statistics, pp. 3061-3069. PMLR, PM.....R....
2021. Torch Contributors. Cross Entropy Loss of Torch. https//yyyooboooggoooossbeeeeeee mm.CrossEntrroyyoos....m 2019.
Marco Cuturi. Sinkhorn distances: lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292-2300 2013.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to con- trolled text generation. In International Conference on Learning Representations, 2020. URL
Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyperspher- ical variational auto-encoders. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp. 856-865. Association For Uncertainty in Artificial Intelligence (AUAI), 2018
Pim de Haan and Luca Falorsi. Topological constraints on homeomorphic auto-encoding. arXiv, 2018.
Scikitlearn Developers. Inverse transformation of PCA.
@ww.aualeeeaccloooooooo...eetaen Cnn.ioiissiiioo....nnnnneeeeeee wra.snnnssstttttsfrrrmr 2011.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1392, 2018.
Luca Falorsi, Pim de Haan, Tim R Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S Cohen. Explorations in homeomorphic variational auto-encoding. In TADGM Workshop @ ICML, 2018
Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. Implicit deep latent variable models for text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3946-3956, 2019.
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach to mitigating KL vanishing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 240-250, 2019.
R. Gnanadesikan and J. R. Kettenring. Robust estimates, residuals, and outlier detection with mul- tiresponse data. Biometrics, pp. 81-124, 1972.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatriee Lagging inference net- works and posterior collapse in variational autoencoders. In International Conference on Learning Representations, 2019. URL http:://geeenevvvweeee/ommmmmccn
10

---

Under review as a conference paper at ICLR 2022
1. Higgins, Loic Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerehner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017a. URL https:
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning. In International Conference Machine Learning, pp. 1490- PMLR, 2017b.
David C Hoaglin, Boris Iglewicz, and John W Tukey, Performance of some resistant rules for outlier labeling. Journal of the American Statistical Association, (1(6))::991-99) 1986.
Huaïbo Huang, Ran He, Zhenan Sun, and Tieniu Tan. Introvae: Introspective variational autoen-
coders for photographic image synthesis. Advances in Neural Information Processing Systems, 31, 2018.
Ian T Jolliffe. Principal Component Elsevier, 1987.
Dimitris Kalatzis, David Eklund, Georgios Arvanitidis, and Søren Hauberg. Variational autoen- coders with riemannian brownian motion priors. In International Conference Muchine Learn- ing, pp. 5053-5066. PMLR, 2020.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. stat, 1050:1, 2014.
Bohan Li, Junxian He, Graham Neubig, Taylor BerggKirkpatrrich and Yiming Yang. A surprisingly effective fix for deep latent variable modeling of text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3603-3614, 2019.
J. Lin. Divergence measures based on the shannon entropy. IEEE Transactions Information theory, 37(1):145-151, 1991.
Tristán M Osán, Diego G Bussandri, and Pedro W Lamberti. Monoparametrie family of metrics derived from classical jensen-shannon plications, 495:336-344 -44, 2018.
shannon divergence. Physica A: Statistical Mechanics and its Ap-
Giorgio Patrini, Rianne den Berg, Patrick Forre, Marcello Carioni, Samarth Bhargav, Max Welling, Tim Genewein, and Frank Nielsen. Sinkhorn autoencoders. In Uncertainty in Artifi- cial Intelligence, pp. 733-743, PMLR, 2020.
Simon JD Prince. Computer vision: models, learning, and inference. Cambridge University Press, 2012.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, openai.com, 2018.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in neural information processing systems, pp. 14866-14876, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286 PMLR, 2014.
Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Fck. hierarchical latent vector model for learning long-term structure in music. In International conference on machine learning, pp. 4364-4373, PMLR, 2018
Steven S. Skiena. The Algorithm Design Manual. Springer, 2008.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. In International Conference on Learning Representations, 2018. URL https://
11

---

Under review as a conference paper at ICLR 2022
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 322-330 2019.
Jiqing Wu, Zhiwu Huang, Dinesh Acharya, Wen Li, Janine Thoma, Danda Pani Paudel, and Luc Van Gool. Sliced wasserstein generative models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3713-3722, 2019.
Peng Xu, Jackie Chi Kit Cheung, and Yanshuai Cao. On variational learning of controllable rep- resentations for text without supervision. In International Conference Machine Learning, pp- 10534-10543. PMLR, 2020.
Weidi Xu, Haoze Sun, Chao Deng, and Ying Tan, Variational autoencoder for semi-supervised text classification. In Thirty-First AAAI Conference Artificial Intelligence, 2017.
Zichao Yang. Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. Improved varia- tional autoencoders for text modeling using dilated convolutions. In International conference on machine learning. pp. 3881-3890. PMLR, 2017.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28:649-657, 2015.
Qile Zhu, Wei Bi, Xiaojiang Liu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. batch normalized inference network keeps the KL vanishing away. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 2636-2649, 2020.
12

---

Under review as a conference paper at ICLR 2022
A PROVING THE CONNECTION BETWEEN NLL AND DISTANCE METRICS
The probability density of P at observation x can be computed as Prince 2012)
exp - (x) -)) ((()))
Kr|²)
(8)
Therefore, the NLL of x under P becomes
NLL(x, P) 2 [[x - K) 11) + Krl) + log )d
(9)
Additionally, by defining function 6(-) as
8(-) = 2 [log(1 1) (og())d
(10)
we can see that By substituting Eq.
1 2
NLL(X, P)
[x
T p (x -w] + 8(Kp).
(11)
As Da between x and y is written as
Dx( = (x k 11),
(12)
12 into Eq.
11 we have
NLL(x, P) 2
Da p)) 5(Kp).
(13)
B PROVING THE UPPER BOUND OF AGG
For latent position ū; if it is classified as continuous with continuous neighbour že41 (i.e..
based on Jir( + 1) and the outlier criterion as discussed in $ 2.2). we know that the indicator Di + 1) is not large outlier and thus is bounded (considering the original formalisation of I. Lipschitz continuity). To start with, considering the proofed lemma, we can further specify D Dspace Eq. ( with DNLL that numerically equal NL.., yielding
DNLL(X's x'; Datent (Z;, Zi+1) < ALIP s.t. DNLL == D ()) + 5(K)), (14)
2
10). Note that Daateut (Z,, 2;+1) is now constant term because the positions of z; and Zi+1 are determinate. Sim- ilarly, as neighbour Zi+1 continuous as given, we have JAGG + 1) bounded and thus there exists threshold AAGG, such that
where ALI is pre-defined threshold (e.g., Falorsi
(2018) set
M Σ
M
JAGG + +1==
D (041,1(0) + 8(221)) / Σ
DNLL (2,+1, H10)/M
<\AGG Intent
2,,2,+11 (2,
<
AAGG-
(15)
where there must exist larger upper bound (i.e., the threshold AsGa) and smaller one (i.e., AAGG ALpeDiatent (2,, 1))) Note that both of dLT and Diatent (Zi, 2,+1) 1) are constant terms mentioned above.
By definition, the Triangle Inequality always holds for established metrics such as Dg. Therefore, taking Zi41 an anchor point we can show that
M
1 DD (Z,, Z.+1) + De De(2411 + 5(K2m)) /M
Eq. 5
2
D D (Z.41, (1 (1) +
D NIL (Z,, Z
(16)
13

---

Under review as a conference paper at ICLR 2022
Sample space
Outlier bound
7(1)
Z (
Z(3)
7(4)
x
3
21
Z2 23 7
Z5
Latent space
Figure 5: A toy example where 24 is in latent hole but may be falsely ignored by JAGG.
Index of interpolated vectors
Figure 6: 2LIP of traversed vectors on one latent path of Vanilla-VAN trained on the Yahoo dataset.
Further incorporating Eq. (T6) with Eq. (14) and Eq. 15p finally yields
JAGG (1) < AAGG ALIP latent (z., 2i+1) + ALIP latent (Z,, 1/+1)/M
AAGG ALAPD (Zi, Zi+ 1) + XI.PP (Zi, Zi41) AAGG,
(17)
which suggests fixed upper bound for 3 AG) Therefore, is continuous under the criterion of Xu et al. (2020). This demonstrates that V latent positions,, if they are not identified as holes under the criterion of Xu et al. (2020), they will not be identified in holes under the criterion of Falorsi et al. (208
C FALSE NEGATIVE OF J Aggregation
As illustrated by Fig. Z. is in discontinuous latent region its corresponding x', greatly departs from the samples of other latent vectors on the same path. However, when x'4 and {x'1, 1, x'2. x's. x's} are roughly symmetric the posteriors (~ normal distributions with same standard deviation) of M = 4 test samples, JAGG (4) is not a large outlier and the hole may thus be ignored. However, this hole can be identified using the other indicator as JT makes a large outlier in this scenario.
D GATHERING LATENT HOLES
Fig. exhibits observation where multiple outlier JrMp identified after visiting just 100 latent vectors on a path. Such example confirms the motivation of the TDC algorithm, i.e., latent holes often gather in small regions and the principal components tend to pass through them.
E PATHS TRAVERSED AND DEPTHS REACHED TILL TDC HALTS
As shown in Tab. for all cubes with different dimension in all datasets, iVAA needs to search much more paths and depths than other models to reach the halt condition, and performs best. On the contrary, the overall worst-performing model, Vanilla- -AE, covers the fewest paths and depths. In addition, when d, increases, we find that the quantity of traversed path gradually increases but the quantity of reached depths decreases, indicating that the distribution of holes is denser in lower- dimensional cube. By comparing results across different datasets, the distribution of holes is denser in Wiki dataset for VAEs, which agrees with finding in Fig.
14

---

Under review as a conference paper at ICLR 2022
Table 3: Average quantities of traversed paths and reached depths in each C of 3D, 4D and 8D until 200 latent holes are identified.
Datasets
VAE
3-VE SIN-VAEiVAL
path
99.9
110.0
120.8
132.4
141.9
3D
depth
8.0
9.3
14.9
15.8
16.4
path
10I.0
I12.5
128.3
135.4
142.1
Yelp
4D
depth
4.7
8.5
13.5
19.2
22.7
path
109.4
119.3
138.4
142.2
149.7
8D
depth
3.1
3.7
5.0
5.6
7.4
path
99.3
110.0
120.8
132.4
141.9
3D
depth
7.3
11.3
13.7
14.6
15.0
path
102.9
113.8
135.5
136.1
140.8
Yahoo
4D
depth
5.2
5.4
16.6
19.9
21.1
path
113.7
116.1
144.9
145.4
148.0
8D
depth
3.4
3.5
8.9
6.6
11.4
path
99.7
88.3
88.2
120.5
131.4
3D
depth
38.6
14.4
9.4
10.5
17.7
path 99.6
90.7
89.6
121.1
132.5
SNLI
4D
depth
11.2
10.5
4.8
8.9
13.5
path
99.8
92.0
99.7
133.2
139.1
8D
depth 4.2
3.7
3.1
6.5
14.7
path
85.5
118.7
125.0
131.5
134.2
3D
depth
4.9
11.9
13.5
14.3
16.4
path
95.3
119.3
127.3
139.4
140.4
Wiki 4D
depth
3.7
6.4
7.4
9.9
15.8
path
99.5
121.4
129.4
141.8
148.2
8D
depth
2.8
3.4
4.8
5.8
6.4
F IMPACT OF LATENT HOLES WHEN d, E {3, 4)
1800
Vari VAV
Cyc-VA
1900
Cyc-ARz
Vori VIE
Wiki
PANE
Avarage parplxxty
1400
DN-VAC
Avarage perploxty
BIN-VAC
SNU
1.000
1000
1000
400
Yahoo
Yelp
Yelp15
400
paths (b) d, 4
d,
Figure 7: Average PPL. and the number of paths traversed until TDC halts for all setups
15

---

Under review as a conference paper at ICLR 2022
G QUANTITY DISTRIBUTION OF IDENTIHIED HOLES
25
of Inverrid path
25
identified holes
(a) Wiki dataset when d, 3
PIAE
of of serraal path
20
ent of identified holes
(b) Wiki dataset when d,- =
C/HAE
#f travered path
ikeiifeed
(c) Yclp15 dataset when d,
Cyc-We
pivie
of travened path
30
20
of identified holes
(d) Yahoo dataset when d, 8
of trovensed path
Sonieiid boks
(c) SNLI dataset when d. =
Figure 8: Quantity distribution of identified holes per discontinuous latent path for models trained on the different datasets.
16

---

Under review as a conference paper at ICLR 2022
H LATENT SPACE VISUALISATION
(a) Cye- VAE (trained on the Yelp15 dataset)
(b) 8-VAE (trained on the Yahoo dataset)
Figure 9: Visualisation of the latent space of different baselines.
c eRAZe..
Muve
(a) RAE (trained on the SNLI datasct)
(b) iVAEsu (trained on the Wiki datasct)
Figure 10: Visualisation of the latent space of different baselines.
Fig. and 10 show that holes are ubiquitously distributed in the entire latent space for different baselines.
17

---

Under review as a conference paper at ICLR 2022
I MORE DECODED SEQUENCES
Table 4: Examples of sentences decoded via vectors of HOLE, NORM, and RAND from Yelp15 and
Wiki datasets.
BN-VAE x Velp15
HOLE
free vip and ate well some of the sushi options around to UNK you in the guest !! there 's more wine that an awesome hot chocolate cake then fair grade
NORM
so tend to get some good red salsa when go to the restaurant always get the turkey wings, cornbread, risotto the fries are very good as well !
RAND
told 18th maintenance crappy awsome devoured confit mosh sorely expiration cinna- compassion refused abroad perfectly cant hokkaido
HOLE
the dude working back was great if your perfectly .UNK then try it there a safe bet with light fluffy slices and some new soul. if you 're regular, this is really a good place to with your family its vegetarian dishes no more like shredded beef". what do you want there lot of onions on the side, but the noodles are a bit excelent styrofoam thighs extra scots roadside poof cart massaman meters miracles boneless cannon oxymoron spoiled maui retain 12.50 dating
NORM
RAND
3-VAE X Wiki
HOLE
from the UNK that considered religious adventures were evolutionary lived of definition.
NORM
the first section of the -UNK"" " in the late 14th century, relief efforts were accom- plished.
RAND
eviction abbe cultural biannual highfield aqua 27.7 ieyasu slowed gretchen fb raping charadriformesfamily cleaner municipal
HOLE NORM
fully investing by means in kyiv and enough budget genetic compliance egypt. that they had a girl to set up the system seems to be UNK
RAND
f3 albrecht rendell dubstep elland sinhalese pediments namely anxieties amrita nootka worked brownish tatars luxury analogues europe/africa
18


Reviews:
[{"review_id": "wUQQfwgNxmd", "reviewer": "Reviewer_gj1W", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is well-written and easy to follow.\n2. The paper proposes an efficient tree-based search algorithm for latent hole identification, and it is easy to parallelize. In experiments, the proposed TDC algorithm takes less time to find a hole compared with previous methods.\n3. Detailed theoretical proof of the underlying connections of two latent hole indicators. This paper proves that one of two indicators can be more comprehensive than the other when identifying holes in the latent spaces, so this forms the basis of their algorithm.\n4. Comprehensive experiments on the language generation task show that the performance of text generation is strongly correlated with the density of latent holes, that from the perspective of the decoder, the Latent Vacancy Hypothesis proposed by previous work does not hold empirically; and that holes are ubiquitous and densely distributed in the latent space. \n\nCons:\n1. The proposed TDC algorithm cannot obtain the density of latent holes precisely. In this paper, the density estimation of latent holes is represented as the average number of paths traversed before the number of identiﬁed holes reaches the algorithm halting threshold. So, the traversal algorithm means a lot. A different traversal algorithm will result in different density estimations.\n2. To promote efficiency, this paper performs dimension reduction using PCA and conducts a search in the resulting lower-dimensional space. However, the density loss during the dimension reduction process is not taken into account.\n", "summary_of_the_paper": "This paper focuses on the discontinuities (aka. holes) in the latent space of VAE. Unlike previous work which concentrates on the encoder side, this paper pays attention to the decoder network who plays an important role in generation tasks. This paper analyzes two existing latent hole indicators and proves that they can actually be unified within a common framework by detailed theoretical proof. The author proposes a heuristic-based BFS algorithm for highly efﬁcient latent space searching. And comprehensive experiments on the language generation task show how the latent holes harm the performance of VAEs and how the holes are distributed. Experiments also show that the latent vacancy hypothesis proposed by previous work does not hold empirically.", "summary_of_the_review": "I think this paper has comprehensive theoretical analysis and experimental results, so I recommend to accept it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "uJvYXXtSfTT", "reviewer": "Reviewer_PV7H", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The theoretical result is interesting.\n* The novel algorithm is simple and effective at finding holes in the latent space.\n* Much of the empirical study is convincing and well done. It is particularly helpful in measuring the proneness of different VAE models to latent holes.\n\nWeaknesses:\n* It is not clear how the theoretical result is important to the rest of the paper. It doesn't seem that the result necessarily extends to the new algorithm that was proposed, or at least it is not stated explicitly. There are also no empirical results for the more powerful algorithm, in which the theoretical result could serve as a performance bound. Some clarifications would help here.\n* There is no empirical analysis of TDC in comparison to existing algorithms. I understand that existing algorithms are too slow for realistic scenarios, but they could still be employed in toy scenarios; this is particularly important because TDC is motivated as the first method to pay attention to the decoder network. Can you answer the following questions: 1) Does TDC find different holes than previous algorithms? E.g, are these holes deeper in the sense that they decode into worse text? Does TDC find more holes? How does the decoder-based criterion compare to encoder-based criteria?\n* It is not clear to me that comparing PPL across different datasets is valid, which is a long discussion point in the empirical results. It is not clear that the absolute PPL numbers relate to the ability of VAEs to learn good latent spaces in that embedding space. To check that, the authors could assess the correlation with PPL of a plain language model. If the correlation is high, the problem probably has nothing to do with the latent space.\n* The Latent Vacancy Hypothesis is claimed to be disproved through the following experiment. The authors measure the quality of a) text from non-holes in a trained VAE vs b) text generated from holes in a trained VAE vs c) text generated from an untrained VAE. The authors find c) < b) < a) (Table 1), and conclude that the holes DO hold information because c) < b), refuting the hypothesis. However, this is not necessarily true: Due to the power of the decoder language model, trained VAEs can generate reasonably good text even if the latent representation is completely uninformative (Bowman et al., 2016). That is, b) might be better than c) not because the hole holds information, but because the decoder language model was trained in b) but not in c). In order to actually verify that the hole holds no information useful to the decoder, it should be compared to the scenario where the latent representation actually holds no information, or where the decoder's access to the latent variable is restricted. This might be achieved by manually activating the forget-gate of the LSTM decoder to 1 (assuming the memory is initialized with the latent representation).\n\nFurther questions:\n* You state that the algorithm by Falorsi et al. (2018) takes at least 30 minutes to find a hole. But this algorithm is not really defined for text, since there is no straight forward way to pick two adjacent x_i and x_{i + 1} in the discrete text space. Could you elaborate how you did this?\n\nReference:\n(Bowman et al., 2016): Generating Sentences from a Continuous Space", "summary_of_the_paper": "The paper is concerned with discontinuities in the latent space of Variational Autoencoders, specifically with finding those so called holes algorithmically. There are three main contributions:\n* A theoretical analysis of two existing algorithms for finding holes. The result shows that one is strictly more powerful than the other.\n* A novel algorithm, TDC, for finding holes that is substantially faster than the existing ones, primarily due to dimensionality reduction techniques. Different from previous algorithms, the hole criterion takes into account the ability of the decoder to generate from a point in latent space.\n* An empirical analysis of the density of latent holes in different types of VAEs and on varying datasets. Among other things, the paper claims to have found empirical evidence for the Latent Vacancy Hypothesis, which states that holes don't carry any information. To this end, they compare the quality of text generated from a hole to the quality of text generated from an untrained VAE.", "summary_of_the_review": "The paper provides a useful method and conducts a worthwhile empirical study. I explicitly encourage the authors to keep working on this paper. In the current state, however, the paper lacks support for several of the central claims made in the paper. Most importantly: Is the decoder-centric hole-criterion really useful? Are the results in Table 1 not explained by the power of the decoder language model?\nIf these reservations are addressed, I am willing to significantly raise my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hG8cMUHeJkJ", "reviewer": "Reviewer_6sDa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My major concerns are: (a) the presentation of the algorithm, especially some math basics, (b) scalability and sensitivity (c) the experiment setting. \n\n## Basic Notations and Clarity\n\nI get a rough idea that algorithm 1 is trying to map the latent space to its principle components then tree-search the principle subspaces. If this is what algorithm 1 is doing then it indeed makes good sense. However, algorithm 1 and related equations are quite hard to parse and many of the basics are not clear, specifically: \n\n- Algorithm 1\n    - Line 3, isn't x a sequence of categorical random variables (if x is a sentence)? If yes how can it have expectation? An expectation always comes with a probability distribution to integrate over, in this case, what is the underlying probability? The decoder?\n    - Similarly in line 4, how is the standard deviation calculated? What is the random variable and what is the probability distribution?\n    - Line 5, are you assuming a Gaussian latent? Is Z the gaussian mean?\n- Equation 3\n    - What does it mean by NLL(z_i, Z^t)? The NLL is usually interpreted as the negative log probability of a random variable. In this case, what is the random variable (z_i or Z) and what is its distribution?\n    - Is z_i the gaussian mean of the i-th sentence? What is Z^t? by \"the sample of the posterior distribution of the t-th out of the total M training samples\", which posterior does this paper refer to? the true posterior p(z | x) or the variational posterior q(z | x)? What does the index t mean? The t-th sentence? Why compare the posterior of the i-th (z_i) with the t-th sentence (Z^t)?\n\n## Scalability of the Algorithm\n\nThe dimension of the latent space used in this paper is only 32. I am aware that this follows previous literature on simple settings, yet realistic VAEs have hundreds of dimensions (like 768 in [1]). Since the complexity of algorithm 1 increases exponentially with the projected latent dimension, I am wondering how it will scale in more realistic scenarios?\n\n## Global properties, Stability and Sensitivity of the Algorithm\n\nMy understanding of Algorithm 1 is a heuristic local search algorithm that largely depends on the randomly chosen C.  My concerns are:\n\n- Since it is a local search algorithm, to what extent does it reveal the global properties of the latent holes, like how these holes distribute over the full space?\n- How sensitive it is to random initialization? What will happen if run it multiple times with different random seeds?\n\n## Experiments\n\n- Why the numbers for normal vectors in Table 1 are so large (at least 500+ in the paper)? [2] reports 320+ PPL on the Yahoo dataset. If these vectors are normal, why it is significantly larger (I understand PPL is on an exponential scale, yet this is still abnormal) than normal? Can you report your test PPL (should be estimated by importance sampling)?\n- Is it possible that you also report the average PPL of the interpolation of two latent codes as a sanity check? By this I mean\n    - for all sentence pairs (x_i, x_j) in the dev set, use the encoder to predict their Gaussian mean vectors (z_i, z_j)\n    - then linearly interpolate between z_i and z_j,\n    - then use the decoder to decode sentences from these interpolations\n    - then estimate their PPL with importance sampling.\n- It is well-observed that the interpolation of two latent codes may not decode solid sentences, which is largely conjectured as a consequence of holes. So can you add these PPLs to Table 1 for better comparison?\n\n## References\n\n[1]  Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, Jianfeng Gao. OPTIMUS: Organizing Sentences via Pre-trained Modeling of a Latent Space. EMNLP 2020\n\n[2]  Junxian He, Daniel Spokoyny, Graham Neubig, Taylor Berg-Kirkpatrick. Lagging Inference Networks and Posterior Collapse in Variational Autoencoders. ICLR 2019 ", "summary_of_the_paper": "This paper studies the holes within the latent space of text VAEs. The major contribution is a hole detection algorithm, which firstly projects the latent representations to a principle subspace, then performs tree-based BFS to detect holes. However, this paper suffers from its unclear presentation of the algorithm (with many mistakes), concerns regarding algorithm scalability and sensitivity, and experiment settings. ", "summary_of_the_review": "This paper discuss an interesting problem, the holes, in the representation space of text VAEs. The is indeed a well-observed problem yet not thoroughly discussed. I think the authors are generally in a good direction towards good solutions. My major concerns are the clarity of the algorithm, the scalability and sensitivity, and the experiment settings. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies discontinuities (i.e., holes) in the latent space of text VAE. Analysis of previous hole detection methods are conducted, and a new efficient hole detection algorithm is proposed. It is an interesting work, but the paper in its current form has a few weaknesses/flaws regarding the proposed algorithm, experiment designs and the resulting conclusions. Reviewers have made various constructive suggestions, which the authors acknowledged.
Decision: Reject