On the Latent Holes 🧀 of VAEs for Text Generation

1 INTRODUCTION . Variational Auto-Encoders ( VAEs ) are powerful unsupervised models for learning low-dimensional manifolds ( aka . a latent space ) from non-trivial high-dimensional data ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) . They have found successes in a number of downstream tasks across different application domains such as text classification ( Xu et al. , 2017 ) , transfer learning ( Higgins et al. , 2017b ) , image synthesis ( Huang et al. , 2018 ; Razavi et al. , 2019 ) , language generation ( Bowman et al. , 2016 ; He et al. , 2019 ) , and music composition ( Roberts et al. , 2018 ) . Various effort has been made to improve the capacity of VAEs , where the majority of the extensions are focused on increasing the flexibility of the prior and approximating posterior . For instance , Davidson et al . ( 2018 ) introduced the von Mises-Fisher ( vMF ) distribution to replace the standard Gaussian distribution ; Kalatzis et al . ( 2020 ) assumed a Riemannian structure over the latent space by adopting the Riemannian Brownian motion prior . A few recent studies attempted to investigate the problem more fundamentally , and revealed that there exist discontinuous regions ( we refer to them as “ latent holes ” following past literature ) in the latent space , which have a detrimental effect on model capacity . Falorsi et al . ( 2018 ) approached the problem from a theoretical perspective of manifold mismatch and showed that this undesirable phenomenon is due to the latent space ’ s topological incapability of accurately capturing the properties of a dataset . Xu et al . ( 2020 ) examined the obstacles that prevent sequential VAEs from performing well in unsupervised controllable text generation , and empirically discovered that manipulating the latent variables for semantic variations in text often leads to latent variables to reside in some latent holes . As a result , the decoding network fails to properly decode or generalise when the sampled latent variables land in those areas . Although the works on investigating latent holes are still relatively sparse , they have opened up new opportunities for improving VAE models , where one can design mechanisms directly engineered for mitigating the hole issue . However , it should be noted that existing works ( Falorsi et al. , 2018 ; Xu et al. , 2020 ) exclusively target at the encoder network when investigating holes in the latent space , and they merely explored its existence without providing further in-depth analysis of the phenomenon . It has also been revealed that the hole issue is more severe on text compared to the image domain , due to the discreteness of text data ( Xu et al. , 2020 ) . In this paper , we tackle the aforementioned issues by proposing a novel tree-based decoder-centric ( TDC ) algorithm for latent hole identification , with a focus on the text domain . In contrast to existing works which are encoder-centric , our approach is centric to the decoder network , as a decoder has a direct impact on the model ’ s performance , e.g. , for text generation . Our TDC algorithm is also highly efficient for latent hole searching when compared to existing approaches , owing to the dimension reduction and Breadth-First Search strategies . Another important technical contribution is that we theoretically unify the two prior indicators for latent hole identification , and evidence that the one of Falorsi et al . ( 2018 ) is more accurate , which forms the basis of our algorithm detailed in § 3 . In terms of analysing the latent hole phenomenon , we provide , for the first time , an in-depth empirical analysis that examines three important aspects : ( i ) how the holes impact VAE models ’ performance on text generation ; ( ii ) whether the holes are really vacant , i.e. , useful information is not captured by the holes at all ; and ( iii ) how the holes are distributed in the latent space . To validate our theory and to demonstrate the generalisability of our proposed TDC algorithm , we pre-train five strong and representative VAE models for producing sentences , including the state-of-the-art model . Comprehensive experiments on the text task involving four large-scale public datasets show that the output quality is strongly correlated with the density of latent holes ; that from the perspective of the decoder , the Latent Vacancy Hypothesis proposed by Xu et al . ( 2020 ) does not hold empirically ; and that holes are ubiquitous and densely distributed in the latent space . Our code will be made publicly available upon the acceptance of this paper . 2 PRELIMINARIES . 2.1 VARIATIONAL AUTOENCODER . A VAE is a generative model which defines a joint distribution over the observations x and the latent variable z̃ , i.e. , p ( x , z̃ ) = p ( x|z̃ ) p ( z̃ ) . Given a dataset X = { xi } Ni=1 with N i.i.d . datapoints , we need to optimise the marginal likelihood p ( X ) = 1N ∑N i ∫ p ( xi|z̃ ) p ( z̃ ) dz̃ over the entire training set . However , this marginal likelihood is intractable . A common solution is to maximise the Evidence Lower BOund ( ELBO ) via variational inference for every observation x : L ( θ , φ ; x ) = Eqφ ( z̃|x ) ( log pθ ( x|z̃ ) ) −DKL ( qφ ( z̃|x ) ‖p ( z̃ ) ) , ( 1 ) where qφ ( z̃|x ) is a variational posterior to approximate the true posterior p ( z̃|x ) . The variational posterior qφ ( z̃|x ) ( aka . encoder ) and the conditional distribution pθ ( x|z̃ ) ( aka . decoder ) are set up using two neural networks parameterised by φ and θ , respectively . Normally , the first term in Eq . ( 1 ) is the expected data reconstruction loss showing how well the model can reconstruct data given a latent variable . The second term is the KL-divergence of the approximate variational posterior from the prior , i.e. , a regularisation forcing the learned posterior to be as close to the prior as possible . 2.2 EXISTING LATENT HOLE INDICATORS . To our knowledge , there are only two prior works which directly determine whether a latent region is continuous or not . One work formalises latent holes based on the relative distance of pairwise points taken from the latent space and the sample space ( Falorsi et al. , 2018 ) . Concretely speaking , given a pair of vectors z̃i and z̃i+1 which are closely located on a latent path , and their corresponding samples x′i and x′i+1 in the sample space , a latent hole indicator is computed as ILIP ( i ) : = Dsample ( x ′ i , x ′ i+1 ) /Dlatent ( z̃i , z̃i+1 ) , ( 2 ) where Dsample and Dlatent respectively denote the metrics measuring the sample and latent spaces ( NB : Dlatent is an arbitrary metric , e.g. , the Euclidean distance and Riemannian distance ) . Falorsi et al . ( 2018 ) focused on the image domain and utilised Euclidean distance for both spaces . Based on the concept of Lipschitz continuity , Falorsi et al . ( 2018 ) then proposed to measure the continuity of a latent region as follows : under the premise that z̃i+1 does not land on a hole , z̃i is recognised as belonging to a hole if the corresponding ILIP ( i ) is a large outlier1 . Another line of work ( Xu et al. , 2020 ) signals latent holes based on the so-called aggregated posterior , with a focus on sequential VAEs for language modelling . This approach interpolates a series of vectors on a latent path at a small interval , and then scores the i-th latent vector z̃i as IAGG ( i ) : = ∑M t=1NLL ( z̃i , Z ( t ) ) /M , ( 3 ) 1Unless otherwise stated , outliers are detected by comparing the subject data point with a fixed bound , which is pre-determined based on a percentile of all data points . where Z ( t ) is the sample of the posterior distribution of the t-th out of the total M training samples , e.g. , when studying holes on the encoder side , this distribution can be computed using qφ ( z̃|x ) in Eq . ( 1 ) ( Xu et al. , 2020 ) . Z ( t ) serves as the reference when calculating the Negative Log-Likelihood ( NLL ) . After all the interpolated vectors on the latent path are traversed , similar to the first method , vectors with large outlier indicators ( IAGG ) are identified as in latent holes . We note that these two indicators actually stem from different intuitions . For ILIP , there is an underlying assumption that a mapping between the sample and latent spaces should have good stability in terms of relative distance change in order to guarantee good continuity in the latent space . In contrast , IAGG is based on the belief that small perturbations on the non-hole regions should not lead to large offsets on the absolute dissimilarity between posterior samples Z ( · ) and the sample z̃i , and hence the calculation is performed only in the latent space and only around one single latent position . While seemly distinct , we show that ( in § 3.2 ) both indicators actually have tight underlying connections and can be unified in a shared mathematical framework . Moreover , the first indicator ( ILIP ) is proofed to be more comprehensive than the second ( IAGG ) and thus can reduce false negatives when identifying holes in the latent space . This forms the basis of our algorithm in § 3 , which is the first attempt to identify a VAE decoder ’ s latent holes for language generation . 3 METHODOLOGY . In this section , we describe our tree-based decoder-centric ( TDC ) algorithm for latent hole identification , which consists of three main components . We first introduce our heuristic-based Breadth-First Search ( BFS ) algorithm for highly efficient latent space searching ( § 3.1 ) . We then theoretically proof , for the first time , that two existing holes indicators can be unified under the same framework and that ILIP is a more suitable choice for identifying latent holes ( § 3.2 ) . Finally , we extend ILIP to the text domain by incorporating the Wasserstein distance for the sample space ( § 3.3 ) . 3.1 TREE-BASED DECODER-CENTRIC LATENT HOLE IDENTIFICATION . As discussed earlier , existing works for investigating latent holes of VAEs all exclusively focus on the encoder network ( e.g. , Falorsi et al . ( 2018 ) ; Xu et al . ( 2020 ) ) , and they can not be trivially applied to the decoders ( which play ultimately important roles on generation tasks ) due to metric incompatibility , especially for VAEs in the text domain ( see detailed discussion in § 3.3 ) . Another drawback of existing indicators is that they have very limited efficiency . Theoretically , their time complexity for traversing a d-dimensional latent space with I interpolations per path is O ( Id ) at the optimal efficiency , which is computationally prohibitive as typically d and I are larger than 30 and 50 for VAEs in practice . Each path is parallel to one axis of the traversed latent space2 . Empirically , we observe that even finding a handful of latent holes has been shown to be difficult for existing methods ( Falorsi et al. , 2018 ; Xu et al. , 2020 ) . Therefore , we tackle both challenges by proposing a highly efficient algorithm for decoder-centric latent hole identification . The pipeline of our TDC algorithm is described in Algorithm 1 and we give a detailed discussion as follows . For the visualisation of TDC ’ s working process in practice , please see Fig . 1 . 2For example , in a 3-dimensional latent space with 4 interpolations per path , as each point is the intersection of 3 paths , 43 = 64 points in total are determined . The space is then equally divided into 64 cubes . Algorithm 1 TDC for latent hole identification Input : Trained VAE model w/ a d-dimensional latent space ; original training set X ; reduced dimension dr ; the desired number of detected vectors in latent holes Nhole Output : Zhole 1 : Z← ∅ ; Vtrain ← ∅ ; Dtrain ← ∅ 2 : ∀x ∈ X , Z← Z ∪ { z̃ } // z̃ is the encoded x 3 : ∀x ∈ X , Vtrain ← Vtrain ∪ { E ( x ) } // E ( · ) is the expectation 4 : ∀x ∈ X , Dtrain ← Dtrain ∪ { σ ( x ) } // σ ( · ) is the standard deviation 5 : Z′ ← PCA ( Z ) // Dimension reduced from d to dr 6 : C ← a randomly-picked closed cube which contains dr vectors of Z′ , w/ edges parallel to dr dimensional axes 7 : Zhole ← ∅ ; Z′hub ← ∅ ; Π← ∅ 8 : while |Zhole| ≤ Nhole do 9 : if Z′hub == ∅ then 10 : Z′hub ← { a random point in C } // Restart BFS 11 : end if 12 : Π← unvisited line segments : passing through vectors in Z′hub ∧ parallel to one of the dr dimensions ∧ w/ endpoints on C // Depth increases by one 13 : Z′hub ← ∅ 14 : for each path ( cf . § 2.2 ) in Π do 15 : Sample z̃′i on path at an interval of 0.01 ∗min ( Dtrain ) 16 : ∀i , z̃i ← INVERSE PCA ( z̃′i ) 17 : ∀i , decode z̃i to compute I ( i ) w/ Vtrain and Dtrain // Cf . Eq . ( 2 ) in § 2.2 18 : if I ( i ) is an outlier then 19 : Zhole ← Zhole ∪ { z̃i } ; Z′hub ← Z′hub ∪ { z̃′i } 20 : end if 21 : end for 22 : end while Dimensionality Reduction . One problem for the current indicators is their limited searching capacity ( as evidenced by their time complexity O ( Id ) ) over the target space . Concretely speaking , both indicators rely on signalling latent holes through 1-dimensional traversal , but a latent space normally has dozens of dimensions to guarantee modelling capacity . To alleviate this issue , after feeding all training samples in X to the forward pass of a trained VAE and storing the encoded latent variables in Z ( Step 2 ) , we perform dimension reduction using Principal Component Analysis ( PCA ) ( Jolliffe , 1987 ) and conduct a search in the resulting dr-dimensional space instead of the original d-dimensional space ( Step 5 ) . We further save the mathematical expectation and standard deviation of each training sample in Vtrain ( Step 3 ) and Dtrain ( Step 4 ) , respectively . In addition , instead of traversing unconstrained paths like past studies , we only visit latent vectors through paths parallel to the dr dimensions ( see Step 12 and the next paragraph ) . Such a setup is based on the intuition that these top principal components contain more information about the latent space , and thus they are more likely to be useful when capturing latent holes . Initialising Infrastructures for Search . To further boost efficiency , we propose to conduct a search on a tree-based structure within a pre-established cubic fence . To be more concrete , at Step 6 we first locate a cube C which surrounds dr encoded training samples from Z′ ( i.e. , Z after dimension reduction ) . These dr posterior vectors serve as references when analysing the distribution of latent holes3 ( cf . § 4.2 ) . We restrict the edges of the dr-dimensional C to be parallel to the dr latent dimensional axes and treat C as the range of our search . Next , we regard each sampled latent vector after dimension reduction z̃′i as a node , and in order to expand the search regions rapidly , we need to visit these nodes following a BFS-based procedure ( Skiena , 2008 ) . Therefore , our algorithm maintains a set Z′hub to keep track of all untraversed hub nodes , where the root ( aka . the first hub node ) is randomly initialised in C ( Step 10 ) . For each hub node , we define dr orthogonal paths , each of which is a line segment that passes through the hub node and is parallel to one dimension . At Step 12 , we log paths having not been previously processed in a set Π. Identifying Latent Holes . Following the principle of BFS , the TDC algorithm processes all nodes at the same depth ( i.e. , all nodes on the paths in Π ) before moving to the next depth . On each path , following Falorsi et al . ( 2018 ) and Xu et al . ( 2020 ) , at Step 15 we sequentially sample a series 3To avoid cherry-picking and parameter deredundancy , we select dr as the number of contained z̃′ ∈ Z′ . of z̃′i . To ensure the sampling is fine-grained , we set the interpolation interval at the 0.01 times minimum standard deviation of all elements in Dtrain ( see Step 4 ) . After that , we utilise the inverse transformation of PCA ( Developers , 2011 ) to reconstruct z̃′i to the original d-dimensional latent space at Step 16 and generate output samples through the decoder at Step 17 . One core question raised is how to choose the indicator I between the two existing ones which seem quite distinct ( cf . § 2.2 ) . We eventually select the scheme of Falorsi et al . ( 2018 ) ( i.e. , ILIP in Eq . ( 2 ) ) and further adopt the Wassertein distance as the metric for the sample space . Detailed justifications are provided in § 3.2 and § 3.3 , respectively . After all paths in Π are investigated , our algorithm pushes the tree search to its next depth by reloading the emptied Z′hub with newly identified latent variables in the holes ( Step 19 ) . The motivation for treating them as new hub nodes comes from our observation that holes tend to gather as clusters . In case that no hub node is added , which suggests the end of current BFS , TDC will bootstrap another tree by randomly picking a new root . The algorithm halts when more than Nhole holes are identified . In practice , we find that our tree-search strategy with dimension reduction not only boosts the efficiency from an algorithmic perspective , but is also highly parallelisable by nature4 and thus can reduce computational time . In theory , the time complexity of TDC can be reduced toO ( Irdr ) , where dr can be as small as 3 ( cf . § 4 ) and Ir is typically less than 2 , thanks to the parallelism of our algorithm . In experiments , when the device is equipped with a Nvidia GTX Titan-X GPU and a Intel i9-9900K CPU , in most cases TDC ( with dr at 8 ) can return more than 200 holes in less than 5 minutes , whereas the methods of Falorsi et al . ( 2018 ) and Xu et al . ( 2020 ) often need at least 30 minutes to find a hole in the same setup as our TDC . 3.2 PICKING INDICATOR FOR TDC Obviously , the indicator used by TDC ( Step 17 in Algorithm 1 ) plays a crucial role as it directly affects the effectiveness of identifying latent holes . By analysing the two existing indicators in § 2.2 , we demonstrate that ( 1 ) although developed under different intuitions , they can actually be unified within a common framework ; ( 2 ) although both indicators have been tested successfully in validating the presence of latent holes , the indicator of Falorsi et al . ( 2018 ) ( ILIP ) is more accurate as it has better completeness and is thus more suitable to our algorithm . To begin with , we prove the following lemma : Lemma 1 NLL ( x , P ) , the NLL of a data point x under a multivariate normal distribution with independent dimensions P can be numerically linked with DG , the so-called Generalized Squared Interpoint Distance ( Gnanadesikan & Kettenring , 1972 ) , as NLL ( x , P ) ≡ 1 2 DG ( x , µ ) + δ ( KP ) s.t . P = N ( µ , KP ) , ( 4 ) where µ denotes the mean , KP denotes the covariance matrix , DG is the so-called Generalized Squared Interpoint Distance ( Gnanadesikan & Kettenring , 1972 ) , and δ ( · ) is a single value function . Proof . See Appendix A . Based on this lemma , we find that the right hand of Eq . ( 3 ) is numerically equivalent to directly calculating NLL ( z̃i , Z ( t ) ) for posterior Z ( t ) , yielding IAGG ( i ) ≡ M∑ t=1 [ 1 2 DG ( z̃i , µ ( t ) ) + δ ( KZ ( t ) ) ] /M s.t . Z ( t ) = N ( µ ( t ) , KZ ( t ) ) . ( 5 ) Note that as Z ( t ) is deterministic , δ ( KZ ( t ) ) settles as a constant term . By integrating Eq . ( 2 ) , w.l.o.g. , we can theoretically prove that if a latent position is signalled to be discontinuous by the indicator of Xu et al . ( 2020 ) , it will be identified using that of Falorsi et al . ( 2018 ) . Proof . See Appendix B . Apart from theoretical proof , empirically we also observe cases showing ILIP has better completeness than IAGG . We present one toy example in Appendix C. To conclude , ILIP should be adopted to reduce the false-negative rate of TDC . 4Our implementation parallelises the computation process at two hierarchies : different paths at the same BFS depth and different z̃′ on the same path .

Summary: The paper is concerned with discontinuities in the latent space of Variational Autoencoders, specifically with finding those so called holes algorithmically. There are three main contributions: * A theoretical analysis of two existing algorithms for finding holes. The result shows that one is strictly more powerful than the other. * A novel algorithm, TDC, for finding holes that is substantially faster than the existing ones, primarily due to dimensionality reduction techniques. Different from previous algorithms, the hole criterion takes into account the ability of the decoder to generate from a point in latent space. * An empirical analysis of the density of latent holes in different types of VAEs and on varying datasets. Among other things, the paper claims to have found empirical evidence for the Latent Vacancy Hypothesis, which states that holes don't carry any information. To this end, they compare the quality of text generated from a hole to the quality of text generated from an untrained VAE.