On the Latent Holes 🧀 of VAEs for Text Generation

1 INTRODUCTION . Variational Auto-Encoders ( VAEs ) are powerful unsupervised models for learning low-dimensional manifolds ( aka . a latent space ) from non-trivial high-dimensional data ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) . They have found successes in a number of downstream tasks across different application domains such as text classification ( Xu et al. , 2017 ) , transfer learning ( Higgins et al. , 2017b ) , image synthesis ( Huang et al. , 2018 ; Razavi et al. , 2019 ) , language generation ( Bowman et al. , 2016 ; He et al. , 2019 ) , and music composition ( Roberts et al. , 2018 ) . Various effort has been made to improve the capacity of VAEs , where the majority of the extensions are focused on increasing the flexibility of the prior and approximating posterior . For instance , Davidson et al . ( 2018 ) introduced the von Mises-Fisher ( vMF ) distribution to replace the standard Gaussian distribution ; Kalatzis et al . ( 2020 ) assumed a Riemannian structure over the latent space by adopting the Riemannian Brownian motion prior . A few recent studies attempted to investigate the problem more fundamentally , and revealed that there exist discontinuous regions ( we refer to them as “ latent holes ” following past literature ) in the latent space , which have a detrimental effect on model capacity . Falorsi et al . ( 2018 ) approached the problem from a theoretical perspective of manifold mismatch and showed that this undesirable phenomenon is due to the latent space ’ s topological incapability of accurately capturing the properties of a dataset . Xu et al . ( 2020 ) examined the obstacles that prevent sequential VAEs from performing well in unsupervised controllable text generation , and empirically discovered that manipulating the latent variables for semantic variations in text often leads to latent variables to reside in some latent holes . As a result , the decoding network fails to properly decode or generalise when the sampled latent variables land in those areas . Although the works on investigating latent holes are still relatively sparse , they have opened up new opportunities for improving VAE models , where one can design mechanisms directly engineered for mitigating the hole issue . However , it should be noted that existing works ( Falorsi et al. , 2018 ; Xu et al. , 2020 ) exclusively target at the encoder network when investigating holes in the latent space , and they merely explored its existence without providing further in-depth analysis of the phenomenon . It has also been revealed that the hole issue is more severe on text compared to the image domain , due to the discreteness of text data ( Xu et al. , 2020 ) . In this paper , we tackle the aforementioned issues by proposing a novel tree-based decoder-centric ( TDC ) algorithm for latent hole identification , with a focus on the text domain . In contrast to existing works which are encoder-centric , our approach is centric to the decoder network , as a decoder has a direct impact on the model ’ s performance , e.g. , for text generation . Our TDC algorithm is also highly efficient for latent hole searching when compared to existing approaches , owing to the dimension reduction and Breadth-First Search strategies . Another important technical contribution is that we theoretically unify the two prior indicators for latent hole identification , and evidence that the one of Falorsi et al . ( 2018 ) is more accurate , which forms the basis of our algorithm detailed in § 3 . In terms of analysing the latent hole phenomenon , we provide , for the first time , an in-depth empirical analysis that examines three important aspects : ( i ) how the holes impact VAE models ’ performance on text generation ; ( ii ) whether the holes are really vacant , i.e. , useful information is not captured by the holes at all ; and ( iii ) how the holes are distributed in the latent space . To validate our theory and to demonstrate the generalisability of our proposed TDC algorithm , we pre-train five strong and representative VAE models for producing sentences , including the state-of-the-art model . Comprehensive experiments on the text task involving four large-scale public datasets show that the output quality is strongly correlated with the density of latent holes ; that from the perspective of the decoder , the Latent Vacancy Hypothesis proposed by Xu et al . ( 2020 ) does not hold empirically ; and that holes are ubiquitous and densely distributed in the latent space . Our code will be made publicly available upon the acceptance of this paper . 2 PRELIMINARIES . 2.1 VARIATIONAL AUTOENCODER . A VAE is a generative model which defines a joint distribution over the observations x and the latent variable z̃ , i.e. , p ( x , z̃ ) = p ( x|z̃ ) p ( z̃ ) . Given a dataset X = { xi } Ni=1 with N i.i.d . datapoints , we need to optimise the marginal likelihood p ( X ) = 1N ∑N i ∫ p ( xi|z̃ ) p ( z̃ ) dz̃ over the entire training set . However , this marginal likelihood is intractable . A common solution is to maximise the Evidence Lower BOund ( ELBO ) via variational inference for every observation x : L ( θ , φ ; x ) = Eqφ ( z̃|x ) ( log pθ ( x|z̃ ) ) −DKL ( qφ ( z̃|x ) ‖p ( z̃ ) ) , ( 1 ) where qφ ( z̃|x ) is a variational posterior to approximate the true posterior p ( z̃|x ) . The variational posterior qφ ( z̃|x ) ( aka . encoder ) and the conditional distribution pθ ( x|z̃ ) ( aka . decoder ) are set up using two neural networks parameterised by φ and θ , respectively . Normally , the first term in Eq . ( 1 ) is the expected data reconstruction loss showing how well the model can reconstruct data given a latent variable . The second term is the KL-divergence of the approximate variational posterior from the prior , i.e. , a regularisation forcing the learned posterior to be as close to the prior as possible . 2.2 EXISTING LATENT HOLE INDICATORS . To our knowledge , there are only two prior works which directly determine whether a latent region is continuous or not . One work formalises latent holes based on the relative distance of pairwise points taken from the latent space and the sample space ( Falorsi et al. , 2018 ) . Concretely speaking , given a pair of vectors z̃i and z̃i+1 which are closely located on a latent path , and their corresponding samples x′i and x′i+1 in the sample space , a latent hole indicator is computed as ILIP ( i ) : = Dsample ( x ′ i , x ′ i+1 ) /Dlatent ( z̃i , z̃i+1 ) , ( 2 ) where Dsample and Dlatent respectively denote the metrics measuring the sample and latent spaces ( NB : Dlatent is an arbitrary metric , e.g. , the Euclidean distance and Riemannian distance ) . Falorsi et al . ( 2018 ) focused on the image domain and utilised Euclidean distance for both spaces . Based on the concept of Lipschitz continuity , Falorsi et al . ( 2018 ) then proposed to measure the continuity of a latent region as follows : under the premise that z̃i+1 does not land on a hole , z̃i is recognised as belonging to a hole if the corresponding ILIP ( i ) is a large outlier1 . Another line of work ( Xu et al. , 2020 ) signals latent holes based on the so-called aggregated posterior , with a focus on sequential VAEs for language modelling . This approach interpolates a series of vectors on a latent path at a small interval , and then scores the i-th latent vector z̃i as IAGG ( i ) : = ∑M t=1NLL ( z̃i , Z ( t ) ) /M , ( 3 ) 1Unless otherwise stated , outliers are detected by comparing the subject data point with a fixed bound , which is pre-determined based on a percentile of all data points . where Z ( t ) is the sample of the posterior distribution of the t-th out of the total M training samples , e.g. , when studying holes on the encoder side , this distribution can be computed using qφ ( z̃|x ) in Eq . ( 1 ) ( Xu et al. , 2020 ) . Z ( t ) serves as the reference when calculating the Negative Log-Likelihood ( NLL ) . After all the interpolated vectors on the latent path are traversed , similar to the first method , vectors with large outlier indicators ( IAGG ) are identified as in latent holes . We note that these two indicators actually stem from different intuitions . For ILIP , there is an underlying assumption that a mapping between the sample and latent spaces should have good stability in terms of relative distance change in order to guarantee good continuity in the latent space . In contrast , IAGG is based on the belief that small perturbations on the non-hole regions should not lead to large offsets on the absolute dissimilarity between posterior samples Z ( · ) and the sample z̃i , and hence the calculation is performed only in the latent space and only around one single latent position . While seemly distinct , we show that ( in § 3.2 ) both indicators actually have tight underlying connections and can be unified in a shared mathematical framework . Moreover , the first indicator ( ILIP ) is proofed to be more comprehensive than the second ( IAGG ) and thus can reduce false negatives when identifying holes in the latent space . This forms the basis of our algorithm in § 3 , which is the first attempt to identify a VAE decoder ’ s latent holes for language generation . 3 METHODOLOGY . In this section , we describe our tree-based decoder-centric ( TDC ) algorithm for latent hole identification , which consists of three main components . We first introduce our heuristic-based Breadth-First Search ( BFS ) algorithm for highly efficient latent space searching ( § 3.1 ) . We then theoretically proof , for the first time , that two existing holes indicators can be unified under the same framework and that ILIP is a more suitable choice for identifying latent holes ( § 3.2 ) . Finally , we extend ILIP to the text domain by incorporating the Wasserstein distance for the sample space ( § 3.3 ) . 3.1 TREE-BASED DECODER-CENTRIC LATENT HOLE IDENTIFICATION . As discussed earlier , existing works for investigating latent holes of VAEs all exclusively focus on the encoder network ( e.g. , Falorsi et al . ( 2018 ) ; Xu et al . ( 2020 ) ) , and they can not be trivially applied to the decoders ( which play ultimately important roles on generation tasks ) due to metric incompatibility , especially for VAEs in the text domain ( see detailed discussion in § 3.3 ) . Another drawback of existing indicators is that they have very limited efficiency . Theoretically , their time complexity for traversing a d-dimensional latent space with I interpolations per path is O ( Id ) at the optimal efficiency , which is computationally prohibitive as typically d and I are larger than 30 and 50 for VAEs in practice . Each path is parallel to one axis of the traversed latent space2 . Empirically , we observe that even finding a handful of latent holes has been shown to be difficult for existing methods ( Falorsi et al. , 2018 ; Xu et al. , 2020 ) . Therefore , we tackle both challenges by proposing a highly efficient algorithm for decoder-centric latent hole identification . The pipeline of our TDC algorithm is described in Algorithm 1 and we give a detailed discussion as follows . For the visualisation of TDC ’ s working process in practice , please see Fig . 1 . 2For example , in a 3-dimensional latent space with 4 interpolations per path , as each point is the intersection of 3 paths , 43 = 64 points in total are determined . The space is then equally divided into 64 cubes . Algorithm 1 TDC for latent hole identification Input : Trained VAE model w/ a d-dimensional latent space ; original training set X ; reduced dimension dr ; the desired number of detected vectors in latent holes Nhole Output : Zhole 1 : Z← ∅ ; Vtrain ← ∅ ; Dtrain ← ∅ 2 : ∀x ∈ X , Z← Z ∪ { z̃ } // z̃ is the encoded x 3 : ∀x ∈ X , Vtrain ← Vtrain ∪ { E ( x ) } // E ( · ) is the expectation 4 : ∀x ∈ X , Dtrain ← Dtrain ∪ { σ ( x ) } // σ ( · ) is the standard deviation 5 : Z′ ← PCA ( Z ) // Dimension reduced from d to dr 6 : C ← a randomly-picked closed cube which contains dr vectors of Z′ , w/ edges parallel to dr dimensional axes 7 : Zhole ← ∅ ; Z′hub ← ∅ ; Π← ∅ 8 : while |Zhole| ≤ Nhole do 9 : if Z′hub == ∅ then 10 : Z′hub ← { a random point in C } // Restart BFS 11 : end if 12 : Π← unvisited line segments : passing through vectors in Z′hub ∧ parallel to one of the dr dimensions ∧ w/ endpoints on C // Depth increases by one 13 : Z′hub ← ∅ 14 : for each path ( cf . § 2.2 ) in Π do 15 : Sample z̃′i on path at an interval of 0.01 ∗min ( Dtrain ) 16 : ∀i , z̃i ← INVERSE PCA ( z̃′i ) 17 : ∀i , decode z̃i to compute I ( i ) w/ Vtrain and Dtrain // Cf . Eq . ( 2 ) in § 2.2 18 : if I ( i ) is an outlier then 19 : Zhole ← Zhole ∪ { z̃i } ; Z′hub ← Z′hub ∪ { z̃′i } 20 : end if 21 : end for 22 : end while Dimensionality Reduction . One problem for the current indicators is their limited searching capacity ( as evidenced by their time complexity O ( Id ) ) over the target space . Concretely speaking , both indicators rely on signalling latent holes through 1-dimensional traversal , but a latent space normally has dozens of dimensions to guarantee modelling capacity . To alleviate this issue , after feeding all training samples in X to the forward pass of a trained VAE and storing the encoded latent variables in Z ( Step 2 ) , we perform dimension reduction using Principal Component Analysis ( PCA ) ( Jolliffe , 1987 ) and conduct a search in the resulting dr-dimensional space instead of the original d-dimensional space ( Step 5 ) . We further save the mathematical expectation and standard deviation of each training sample in Vtrain ( Step 3 ) and Dtrain ( Step 4 ) , respectively . In addition , instead of traversing unconstrained paths like past studies , we only visit latent vectors through paths parallel to the dr dimensions ( see Step 12 and the next paragraph ) . Such a setup is based on the intuition that these top principal components contain more information about the latent space , and thus they are more likely to be useful when capturing latent holes . Initialising Infrastructures for Search . To further boost efficiency , we propose to conduct a search on a tree-based structure within a pre-established cubic fence . To be more concrete , at Step 6 we first locate a cube C which surrounds dr encoded training samples from Z′ ( i.e. , Z after dimension reduction ) . These dr posterior vectors serve as references when analysing the distribution of latent holes3 ( cf . § 4.2 ) . We restrict the edges of the dr-dimensional C to be parallel to the dr latent dimensional axes and treat C as the range of our search . Next , we regard each sampled latent vector after dimension reduction z̃′i as a node , and in order to expand the search regions rapidly , we need to visit these nodes following a BFS-based procedure ( Skiena , 2008 ) . Therefore , our algorithm maintains a set Z′hub to keep track of all untraversed hub nodes , where the root ( aka . the first hub node ) is randomly initialised in C ( Step 10 ) . For each hub node , we define dr orthogonal paths , each of which is a line segment that passes through the hub node and is parallel to one dimension . At Step 12 , we log paths having not been previously processed in a set Π. Identifying Latent Holes . Following the principle of BFS , the TDC algorithm processes all nodes at the same depth ( i.e. , all nodes on the paths in Π ) before moving to the next depth . On each path , following Falorsi et al . ( 2018 ) and Xu et al . ( 2020 ) , at Step 15 we sequentially sample a series 3To avoid cherry-picking and parameter deredundancy , we select dr as the number of contained z̃′ ∈ Z′ . of z̃′i . To ensure the sampling is fine-grained , we set the interpolation interval at the 0.01 times minimum standard deviation of all elements in Dtrain ( see Step 4 ) . After that , we utilise the inverse transformation of PCA ( Developers , 2011 ) to reconstruct z̃′i to the original d-dimensional latent space at Step 16 and generate output samples through the decoder at Step 17 . One core question raised is how to choose the indicator I between the two existing ones which seem quite distinct ( cf . § 2.2 ) . We eventually select the scheme of Falorsi et al . ( 2018 ) ( i.e. , ILIP in Eq . ( 2 ) ) and further adopt the Wassertein distance as the metric for the sample space . Detailed justifications are provided in § 3.2 and § 3.3 , respectively . After all paths in Π are investigated , our algorithm pushes the tree search to its next depth by reloading the emptied Z′hub with newly identified latent variables in the holes ( Step 19 ) . The motivation for treating them as new hub nodes comes from our observation that holes tend to gather as clusters . In case that no hub node is added , which suggests the end of current BFS , TDC will bootstrap another tree by randomly picking a new root . The algorithm halts when more than Nhole holes are identified . In practice , we find that our tree-search strategy with dimension reduction not only boosts the efficiency from an algorithmic perspective , but is also highly parallelisable by nature4 and thus can reduce computational time . In theory , the time complexity of TDC can be reduced toO ( Irdr ) , where dr can be as small as 3 ( cf . § 4 ) and Ir is typically less than 2 , thanks to the parallelism of our algorithm . In experiments , when the device is equipped with a Nvidia GTX Titan-X GPU and a Intel i9-9900K CPU , in most cases TDC ( with dr at 8 ) can return more than 200 holes in less than 5 minutes , whereas the methods of Falorsi et al . ( 2018 ) and Xu et al . ( 2020 ) often need at least 30 minutes to find a hole in the same setup as our TDC . 3.2 PICKING INDICATOR FOR TDC Obviously , the indicator used by TDC ( Step 17 in Algorithm 1 ) plays a crucial role as it directly affects the effectiveness of identifying latent holes . By analysing the two existing indicators in § 2.2 , we demonstrate that ( 1 ) although developed under different intuitions , they can actually be unified within a common framework ; ( 2 ) although both indicators have been tested successfully in validating the presence of latent holes , the indicator of Falorsi et al . ( 2018 ) ( ILIP ) is more accurate as it has better completeness and is thus more suitable to our algorithm . To begin with , we prove the following lemma : Lemma 1 NLL ( x , P ) , the NLL of a data point x under a multivariate normal distribution with independent dimensions P can be numerically linked with DG , the so-called Generalized Squared Interpoint Distance ( Gnanadesikan & Kettenring , 1972 ) , as NLL ( x , P ) ≡ 1 2 DG ( x , µ ) + δ ( KP ) s.t . P = N ( µ , KP ) , ( 4 ) where µ denotes the mean , KP denotes the covariance matrix , DG is the so-called Generalized Squared Interpoint Distance ( Gnanadesikan & Kettenring , 1972 ) , and δ ( · ) is a single value function . Proof . See Appendix A . Based on this lemma , we find that the right hand of Eq . ( 3 ) is numerically equivalent to directly calculating NLL ( z̃i , Z ( t ) ) for posterior Z ( t ) , yielding IAGG ( i ) ≡ M∑ t=1 [ 1 2 DG ( z̃i , µ ( t ) ) + δ ( KZ ( t ) ) ] /M s.t . Z ( t ) = N ( µ ( t ) , KZ ( t ) ) . ( 5 ) Note that as Z ( t ) is deterministic , δ ( KZ ( t ) ) settles as a constant term . By integrating Eq . ( 2 ) , w.l.o.g. , we can theoretically prove that if a latent position is signalled to be discontinuous by the indicator of Xu et al . ( 2020 ) , it will be identified using that of Falorsi et al . ( 2018 ) . Proof . See Appendix B . Apart from theoretical proof , empirically we also observe cases showing ILIP has better completeness than IAGG . We present one toy example in Appendix C. To conclude , ILIP should be adopted to reduce the false-negative rate of TDC . 4Our implementation parallelises the computation process at two hierarchies : different paths at the same BFS depth and different z̃′ on the same path .

Summary: This paper studies the holes within the latent space of text VAEs. The major contribution is a hole detection algorithm, which firstly projects the latent representations to a principle subspace, then performs tree-based BFS to detect holes. However, this paper suffers from its unclear presentation of the algorithm (with many mistakes), concerns regarding algorithm scalability and sensitivity, and experiment settings.