Identifiable Deep Generative Models via Sparse Decoding
Authors: Gemma Elyse Moran
Dhanya Sridhar
Yixin Wang
David Blei
Venue: Accepted by TMLR ()

Abstract: We develop the sparse VAE for unsupervised representation learning on high-dimensional data.  The sparse VAE learns a set of latent factors (representations) which summarize the associations in the observed data features. The underlying model is sparse in that each observed feature (i.e. each  dimension of the data) depends on a small subset of the latent factors.  As examples, in ratings data each movie is only described by a few genres; in text data each word is only applicable to a few topics; in genomics, each gene is active in only a few biological processes.  We prove such sparse deep generative models are identifiable: with infinite data, the true model parameters can be learned. (In contrast, most deep generative models are not identifiable.)  We empirically study the sparse VAE with both simulated and real data.  We find that it recovers meaningful latent factors and has smaller heldout reconstruction error than related methods.


Published in Transactions on Machine Learning Research (09/2022)
Identifiable Deep Generative Models via Sparse Decoding
Gemma E. Moran
gm291 Bac.oaarooaaaaac.ooooooooo edu
Columbia University
Dhanya Sridhar
Mila Quebec Al Institute and Université de Montréal
Yixin Wang University of Michigan
David M. Blei Columbia Universily
Reviewed on OpenReview: https: openreview. net/ forum? id= d=vd00nGWE
Abstract
We develop the sparse VAE for unsupervised representation learning on high-dimensional data. The sparse VAE learns a set of latent. factors (representations) which summarize the associations in the observed data features. The underlying model is sparse in that each observed feature (i.e. each dimension of the data) depends on small subset of the latent factors. As examples, in ratings data each movie is only described by few genres; in text data each word is only applicable to a few topics; in genomics, each gene is active in only a few biological processes. We prove such sparse deep generative models are identifiable: with infinite data, the true model parameters can be learned. (In contrast, most deep generative models are not identifiable.) We empirically study the sparse VAE with both simulated and real data. We find that it recovers meaningful latent factors and has smaller than related methods.
heldout reconstruction
1 Introduction
In many domains, high-dimensional data exhibits variability that can be summarized by low-dimensional latent representations, or factors. These factors can useful a variety of tasks including prediction, transfer learning, and domain adaptation (Bengio et al., 2013).
To learn such factors, many rescarchers fit deep generative models (DGMs) (Kingma and Welling, 2014; Rezende et al., 2014). A DGM models each data point with latent low-dimensional representation (its factors), which is passed through neural network to generate the observed features. Given a dataset, DGM can be fit with a variational autoencoder (VAE), a method that provides both the fitted parameters to the neural network and the approximate posterior factors for each data point.
In this paper, we make two related contributions to the study of deep generative models.
First, we develop the sparse DGM. This model is DGM where each observed feature only depends a subset of the factors that is, the mapping from factors to features is sparse. This notion of sparsity often reflects the type of underlying patterns that we anticipate finding in real-world data. In genomics, each gene is associated with a few biological processes; in text, each term is applicable to few underlying topics; in movie ratings, each movie is associated with a few genres.
In detail, the model implements sparsity through a per-feature masking variable. When generating a data point, this mask is applied to the latent representation before producing each feature. In practice, we learn the per-ueeture mask with the help of the Spike-and-Slab Lasso prior (Ročková and George, 2018),

---

Published in Transactions on Machine Learning Research (09/2022)
4
4.2
A
x,5
ALL
(a) Sparse DGM
(b) DGM
Figure 1: In a DGM, feature Tij depends on all factors, Zik- A sparse DGM is displayed where features 222 depend only on 211; 24 depends on and z;2; and Ius depend only on 2/2. The features are passed through the same noural network fa.
sparsity-inducing prior which enjoys good properties in more classical settings. We fit this sparse DGM and SSL prior with amortized variational inference (Gershman and Goodman, 2014; Kingma and Welling, 2014). We call this procedure the sparse VAE.
Second, we study identifiability in sparse DGMs. A model identifiable each setting of its parameters produces unique distribution of the observed data. With an unidentifiable DGM, even with infinite data from the model, we cannot distinguish the true factors. Identifiable factor models tasks such as domain adaptation (Locatello et al., 2020), transfer learning (Dittadi et al., 2021), and fair classification (Locatello et al., 2019a). Most DGMs are not identifiable (Locatello et al., 2019b).
important to
Specifically, we prove that a sparse DGM is identifiable if each latent factor has at least two "anchor features." An anchor feature a dimension of the data which depends on only one latent factor. For example, in text data, an anchor feature is word that is applicable to only one underlying topic (Arora et al., 2013). Note that the anchor features do not need to be known in advance for our results to hold; only need to assume they exist. Further, we do not need to assume that the true factors are drawn independently; in many applications, such an independence assumption is unrealistic (Träuble et al., 2021).
The sparse VAE and the anchor assumption are related in that the SSL prior will likely yield the kind of sparse mapping that satisfies the anchor assumption. What this means is that the data comes from an identifiable DGM, i.e., one that satisfies the anchor assumption, then the SSL prior will encourage the sparse VAF to more quickly find good solution. In this sense, this paper is in the spirit of research in Bayesian statistics, which es- tablishes theoretical conditions on identifiability and then produces priors that favor identifiable distributions.
Empirically, we compare the sparse VAE to existing algorithms for fitting DGMs: the VAE (Kingma and Welling, 2014), 3-VAE (Higgins et al., 2017), Variational Sparse Coding (VSC, Tonolini et al., 2020), and OI-VAE (Ainsworth et al., 2018). We find that (i) on synthelic data, the sparse VAE recovers ground truth factors, including when the true factors are correlated; (ii) on text and movie-ratings data, the sparse VAE achieves better heldout prodictive performance; (iii) on semi-synthetic data, the sparse VAE has better heldout predictive performance on test data that distributed differently to training data; (iv) the sparse VAF finds interpretable structure in text, movie-ratings and genomics data.
In summary, the contributions of our paper are as follows:
We develop a sparse DGM which has Spike-and-Slab Lasso priors (Ročková and George, 2018) and develop an algorithm for fitting this model, the sparse VAE.
We prove that sparse DGMs are identified under an anchor feature assumption.
We study the sparse VAE empirically. It oulperforms existing methods on synthetic, semi-synthetic, and real datasets of text, ratings, and genomics.
Related Work. The sparse DGM provides contribution to sparse methods in linear and nonlinear representation learning. The sparse DGM has sparsity-inducing prior the factureto-feeture mapping
2

---

Published in Transactions on Machine Learning Research (09/2022)
(i.e., the decoder). Similar priors have also been applied in linear factor analysis to induce sparsity in the factor-to-feature mapping (i.e,, the loadings matrix); see, for example Bernardo et al. (2003); Bhattacharya and Dunson (2011); Carvalho et al. (2008); Knowles and Ghahramani (2011); Ročková and George (2016).
Beyond linearity, Barello et al. (2018) considers a hybrid Ainer/nooniieat VAE which uses a sparse linear decoder and a neural network encoder. In nonlinear representation learning, Tonolini et al. (2020) imposes sparsity-inducing priors directly on the latent factors. Instead, the sparse DGM of this paper involves a sparsity-inducing prior the factor-to-feature mapping. Rhodes and Lee (2021) indirectly induces sparsity in the cector-to-fetture mapping of their DGM via L1 regularization of the mapping's Jacobian.
The sparse VAE closely relates to the OI-VAE (Ainsworth et al., 2018), a method for modeling grouped features where the factor-to-group mapping is sparse. However, there are several differences. First, the OL-VAE uses a Group Lasso prior while the sparse VAE has a Spike-and George, 2018). The SSL prior mitigates common issues with the Lasso prior, including under-regullarizaton of small coefficients and over-regularization of large coefficients (Ghosh el al., 2016), and a sub-optimal posterior contraction rate (Castillo et al., 2015). Second, we study the identifiaiility of sparse DGMs, property that Ainsworth et al. (2018) does not investigate.
Lasso prior (SSL, Ročková and
This paper also contributes to the literature that uses the anchor feature assumption for identifiability. In linear models, the anchor feature assumption has led to identifiability results for topic models (Arora et al., 2013), non-negative matrix factorization (Donoho and Stodden, 2003), and linear factor analysis (Bing et al., 2020). The key idea is that the anchor assumption removes the rotational invariance of the latent factors. This idea also arises in identifiable factor analysis (Rohe and Zeng, 2020) and independent. component analysis, where rotational invariance is removed by assuming that the components are non-Gaussian (Eriksson and Koivumen, 2004).
Finally, this paper is related to methods in identifiable nonlinear representation learning. To achieve identifi- ability, most methods require weak supervision: Khemakhem et al. (2020); Mita et al. (2021); Zhou and Wei (2020) require auxiliary data; Locatello et al. (2020) requires paired samples; Kügelgen on data augmentation; and Hälvä al. (2021): Ahuja et al. (2022); Lachapelle et al. (2022) leverage known temporal or spatial dependencies among the samples. In contrast, the sparse DGM does not need auxiliary information weak supervision for identifiability; instead, the anchor feature assumption is sufficient. Relat- edly, Horan et al. (2021) achieves identifiable representations under a "local isometry" assumption, meaning a small change in a factor corresponds to a small change in the features. In contrast to this paper, however, Horan et al. (2021) requires the factors to be independent; we do not need such an independence assumption.
al. (2021) relies
2 The Sparse Variational Autoencoder
The observed data is a vector of G features I; E RC for (1..... N} data points. The goal is to estimate a low-dimensional set of factors z E TRK where K G.
In a standard DGM, the vector 2, is a latent variable that is fed through a neural network to reconstruct the distribution of I,, The sparse DGM introduces an additional parameter w, E RK j (l...., ......G G}, a sparse per-feature vector that selects which of the K latent factors are used to produce the jth feature of x, Further, the sparse DGM models the prior covariance between factors with the positive definite matrix Σ, E R The sparse DGM is:
Wijk Spiko-and-Slab Lasso d1, a, b), =, K (0, Ez), i= 1....,N N mij [[[[o(w @ 2z)); of), j j= i.....,
where So R + R is a neural network with weights 0, of is the per-feature noise variance and @ denotes dlement-wise multiplication. The Spike-and-Slat Lasso (Ročková and George, 2018) is a sparsity- inducing prior which we will describe below. (Any sparsity-inducing prior on Wis would be appropriate for a sparse DGM; we choose SSL because enjoys theoretical guarantees and fast computation in more classical settings.)
3

---

Published in Transactions on Machine Learning Research (09/2022)
Algorithm 1: The sparse VAE
input: data X, hyperparameters Ao, A1 b, E; output: factor distributions (=(z(z), selector matrix W, parameters 0 while not converged do for j=l..... G; k == =, l,..., K do
Update
1 7e vo (Wzx) 1/ (1(())
E (ysk/ mel
1+
for k = 1, ,....,K do Update
Σ j=
Tak Wirk Mel +
b + G-2
Update 0, o, W with SGD according to Eq. 8;
In the data generating distribution in Eq. 1, the parameter wik controls whether the distribution of zú depends on factor k. If wykk # O, then zit contributes to Fij, while if wyk = 0, 2ik cannot contribute to Tij- If wy is sparse, then Zej depends on only small set of factors. Such a sparse factor-to-feature relationship is shown in Figure 1.
Note that zúj depends on the same set of factors for every sample i. This dependency only makes sense when each Zij has consistent meaning across samples. For example, in genomics data Ziy corresponds the same gene for all samples. (This dependency is not reasonable for image data, where pixel has consistent meaning across samples.)
Spdle-and-Slab Lasso. The parameter wyy has a SSL prior (Ročková and George, 2018), which is defined as hierarchical model,
7 ~ Beta(a, b),
(2)
7/k ~ Bernoull((w))
(3)
Wyk (W)))(W)))) +1 (o
In this prior, (w) d exp(-A.lwl) is the Laplace density and do > A1. The variable wyn is drawn a 2
priori from either a Laplacian "spike" parameterized by Ao, and consequentially negligible, or a Laplacian "slab" parameterized by \1, and can be large.
Further, the variable 7ik a binary indicator variable that determines whether Wik is negligible. The Beta-Bernoulli prior T/k allows for uncertainty in determining which factors contribute to each feature.
Finally, the parameter E [0 controls the proportion of features that depend on factor k. By allowing to vary, the sparse DGM allows each factor to contribute to different numbers of features. In movie-ralings data, for example, a factor corresponding to the action nationnadveeture genre may be associated with more movies than a more esoteric genre.
Notice the prior on 77 helps "zero out" extraneous factor dimensions and consequently estimate the number of factors K. If the hyperparameters are set to x 1/K and b the Beta-Bernoull prior corresponds to the finite Indian Buffet Process prior (Griffiths and Ghahramani, 2005).
2.1 Inference
In inference, we are given a datasct a and want to calculate the posterior p(0, W, I, m, 2 (an)) We will use approximate maximum posteriori (MAP) estimation for W, 9 and n, and amortized variational inference for 21:N. The full procedure for approximating this posterior is called sparse VAE.
4

---

Published in Transactions on Machine Learning Research (09/2022)
(a) Sparse VAE
[c] True
24
X-
6
X-m 64 (int))
Xy 6.3
30
20
Teature
20
10
-10
-2
-2. 0.0 255
-2.
5.
-25
0.0
25
50
10
2 (estimated)
(b) VAE
(d) Estimated
-2.2 2
5
Xa-1.8-2
X--47
xo-122
Tealbrre
10
-10
(estimated)
Factors
Figure 2: (a-b) The sparse VAE estimates factors which recover the true generative process; the VAE does not. The observed data is plotted against the estimated factors. The true factor-feature relationship is the red line; the best fit coefficients for the estimated factors in the grey boxes. (c) The true W matrix. (d) The sparse VAE estimate of W. (VAE has no W matrix).
The exact MAP objective is
N
N
Σ log (2; W.n) = Σ log
fmoant W) W)p(z))des + log
/ |(W]T))(TTnb)())d)
(5)
where r= ( j,k= denotes the binary indicators. We lower bound Eq. 5 with variational approximations of the posterior of 11: and the exact posterior the latent r, which comes from the SSL prior. We approximale the posterior of the factors po(z; (a.) by the variational family:
(,<z) Nx (xx)) (xx)),
(6)
where (-(-) and o2(() are neural networks with weights o, in standard VAE (Kingma and Welling, 2014; Rezende et al., 2014).
These approximations yield our final objective function:
L W W,n) Σ (ee W)] (xx
(7)
Er,,,,
(8)
To optimize C(B,G, Φ. W, .7), we alternate between an expectation step and a maximization stop. the E-step, we calculate the complete conditional posterior of T. In the M-step, we take gradient steps in model parameters and variational parameters with the help of reparameterization gradients (Kingma and Welling, 2014; Rezende al., 2014). The sparse VAE procedure is in Algorithm 1.
The sparse VAE has higher computational burden than VAE. A VAE typically requires one forward pass through the neural network at each gradient step. In contrast, the sparse VAE requires G forward passes. This additional computation is because for each of the G features, the sparse VAE uses different decoder input wj @ 2,, for j = 1....,G. G. (The VAE and sparse VAE optimize the same number neural network parameters, however).
2.2 Synthetic example
For intuition, consider a synthetic dataset. We generate = 1000 samples from a factor model, each with G features and K latent factors. We take the true factors be corrclated and generate them as 2, N(0, C) where the true factor covariance C has diagonal entries Cut and off-diagonal entries C = 0.6 for k + K.
5

---

Published in Transactions on Machine Learning Research (09/2022)
Given the factors, the data is generated as:
a ~ NG(f(zi), (z) with 0² = 0.3.
(9)
The fecture mapping f R RRG is sparse:
f(zi) = (2:1 22,1 32 42 52,2, 6in(2 72 2:2).
This mapping corresponds the W matrix in Figure 2c.
We compare the sparse VAE with a VAF. For both, we initialize with overestimate of the true number of factors (Ku 5). The sparse VAE finds factors that reflect the true generative model (Figure 2a). Moreover, the sparse VAE correctly finds the true number of factors by setting three columns of W to zero (Figure 2d). In contrast, the VAE recovers the second factor fairly well, but doesn't 't recover the first factor (Figure 2b).
We analyze this synthotic data further in $ 4, as well as text, movic ratings and genomics data.
3 Identifiability of Sparse DGMs
For an identifiable model, given infinite data, is possible to learn the true parameters. In both linear and nonlinear factor models, the factors are usually unidentifiable without additional constraints (Yalcin and Amomiya, 2001; Locatello al., 2019b). This lack of identifiability means that the true factors 2, can have the same likelihood as another solution ž, Even with infinite data, we cannot distinguish between these representations based on the likelihood alone. Further inductive biases model constraints necessary to narrow down the set of solutions.
In this section, we demonstrate that sparsity is a useful inductive bias for model identifiability. Consider the sparse deep generative model:
2, ~ (K, 1= Baj ~ (((oo( O 03), j=
(10)
where C' is the true covariance matrix of the factors. Informally, if the masking variable W is sparse in a particular way, then we can identify the latent. factors, up to coordinate-wise transformation. (Note: we only make an assumption about the sparse support of W, not its distribution; the SSL prior in $ tool to expore the space of sparse W in the process of fitting the model.)
Specifically, we prove that for any solutions z and 2 with equal likelihood, have:
Zik gx(())) i= =
(11)
for all factors k: I,..., K (up to permutation of k), where gk + R are invertible and differentiable functions.
This definition of identifiability is weaker than the canonical definition, for which we would need the two solutions z;, ž, to be exactly equal. We use this weaker notion of identifiability because our goal is to isolate the dimensions of z; which drive the observed response, and not necessarily find their exact value-i.e i.. we want to avoid mixing the dimensions of 2,. For example, we want to be able to distinguish between solutions and Pz, for arbitrary rotation matrices P.
Starting point. This paper builds body of work in identifiable linear unsupervised models (Arora et al., 2013; Bing et al., 2020; Donoho and Stodden, 2003). These works assume the existence of anchor features, features which depend on only one factor. The existence of anchor features helps with idcntifiability because they leave a detectable imprint in the covariance of the observed data.
We extend these results to the nonlinear setting. The key insight is that two anchor features have the same nonlincar dependence on factor, then we can apply results from the linear setting.
We first formally state the anchor feature assumption.

---

Published in Transactions on Machine Learning Research (09/2022)
A1. (Anchor feature assumption) For every factor, 2 there are at least two features Z. which depend only on that factor. Moreover, the two features have the same mapping Jj from the factors z.: that is, for all i = N:
E0 1 zi] = fz(w),
E Blayy = (zsk).
(12)
We refer to such features "anchor features."
The connection between the anchor feature assumption and the masking variable W is the following: Tf feature is an anchor for factor k, then wisk + 0 and wys' =0 for all k + k'.
Roadmap. We prove the identifiability of the sparse DGM in two parts. First, we prove that the latent factors are identifiable up to coordinate-wise transformation when the anchor features are known (Theorem 1). Second, we prove that the anchor features can be detected from the observed covariance matrix (Theorem 2). Together, Theorem and Theorem 2 give the identifiability of the sparse DGM.
Known anchor features. The first result, Theorem proves that the sparse DGM factors are identifiable if given the anchor features. If feature j is anchor feature for factor k, set w 1 and Wyk 0 for all V' k.
Theorem 1. Suppose we have infinile dula drown from the model in Eq. 10 and A1 holds. Assume are given the rows of W corresponding to the anchor features. Suppose we have two solutions with equal likelihood: {) and {0,2) 2), with
P W) = 1g(2=/2 W).
(13)
Then, the factors ž and 2 are equal up to coordinate wise transformations. For Is : :R - R and N,
xx) =
9K (2:))).
(14)
The proof Theorem is in Appendix A.1. The proof idea that if the anchor features are known, then we are given noisy transformations of the coordinates of z. Knowing these noisy transforms allows us identify up to coordinato-wist transform.
Unknown anchor features. The second result proves that the anchor features can be determined from approximation of the covariance matrix of the data X.
This result requires additional assumptions. The next assumption concerns the weights of the neural network so- This assumption needs some additional notation. After applying the chain rule to the first. layer, we rewrite the derivative of mapping fo follows:
@[fo(w, O 2:)ss
@mo(usg))
H Wik:
(15)
dzik
duijd.
where (11 D dk J. are the weights of the first neural network layer and Dj is the dimension of the first layer,
Uijd = Σk H Tak is the first layer before the activation function, and my Dr - RG is the rest of
the neural network which takes as input the first layer u.y- = (u Let Bijk = Σd Ousya
@ma(u,g:)) H (1)
A2. Suppose is an anchor feature for factor k. For another feature I, we assume that
N
N
K
2
Jwee || Bijae Bare
(16)
i=1 #=1
with equality when is also anchor feature for k and inequality otherwise.
This assumption ensures that the covariance between two anchor features is larger than the covariance between anchor feature and non-anchor feature. This consequence then allows us to pinpoint the

---

Published in Transactions on Machine Learning Research (09/2022)
anchor features in the covariance matrix of the data. In Appendix A.3, we show this assumption holds for neural network with ReLU activations and independent weights.
Finally, we also require an assumption regarding the covariance of the factors: this is the same as assumption (iii) of Bing et al. (2020), which studies linear models.
A3. Denote the true covariance matrix of the factors as Cov(z,) = C. We assume min [CRk, CRe } > |Ckarl for all k, k' =I....,K K.
Assumption A3 requires that the variance of each factor be greater than its covariance with any other factor. To gain some intuition, consider a datasel of news articles where the latent factors are topics, sports and politics. Assumption A3 would be violated if every article about sports also discussed politics (and vice versa). In this case, the anchor word for sports will be equally as correlated with an anchor word for politics as it is with another anchor word for sports. That is, there is no discernible difference in the data that helps us distinguish the sports and politics factors.
Theorem 2. Assume the model Eq. 10 with A1-3 holds. Then, the set of anchor features can be determined uniquely from N Eiz Cov(æ;) as N + 2o (given additional regularity conditions detailed in Appendir A.2).
The proof of Theorem 2 is in Appendix A.2.
Proof idea: We adapt the proof technique of Bing el al. (2020). The proof idea of that paper is that for any two features z and ø.y that are anchors for the same factor, their covariance Cov(x.,, x.y) will be greater than their covariance with any other non-anchor fcature (under some conditions on the loadings matrix, analogous to A2). Then, the anchor features can be pinpointed from the observed covariance matrix. In the nonlinear case, we can apply similar strategy: even if the mapping is nonlinear, if the mapping is the same for the anchor features, we can pinpoint the two anchors in the covariance matrix.
Connection to the sparse VAE algorithm. Theorem 2 proves that the existence of anchor features implies a particular structure in the covariance of the data. Consequently, an appropriate estimation proce- dure can then pinpoint the anchor features. In the sparse VAE, we do not directly consider the covariance matrix as it would involve laborious hyperparameter tuning to determine which covariance entries are "close enough' to be anchor features. Instead, estimate W with an SST prior, motivated by the sparsity in the decoder exhibited by anchor features. That is, it is the sparse decoding for the anchor features that is needed for Theorem 2, not the SSL prior; the SSL prior is a modeling choice that will likely yield the kind of sparse mapping that satisfies the anchor assumption. The SSL prior is not the only sparsity-induccng prior that we could have used; however, we found it to work well empirically. We compare the SSL prior to the horseshoe prior (Carvalho et al., 2009) in Appendix C.4.1.
Consistency. An important implication of identifiability is consistency: if we learn the optimal parameters of the sparse VAE, then we will recover the true factors in the limit of infinite data. Specifically, if (i) the variational family (0(2(zz) contains the true posterior p=(x|z) and (fi) the ELBO is maximized with respect to the parameters 0, & and W, then in the limit of infinite data, the sparse VAE will learn the true factors up to coordinate-wiee transform and permutation.
4 Experiments
We empirically study the sparse VAE using a mix of synthetic, semi-synthetic and real data!. We consider the following questions: 1) How well does the sparse VAE recover ground truth factors when (i) the factors are uncorrelated and (ii) the factors are correlated? 2) How does the heldout predictive performance of the sparse VAE compare to related methods? 3) How does the sparse VAE perform compared to related methods when the correlation between factors is different in the training and test data? 4) Docs the sparse VAE find meaningful structure in data?
We compare the sparse VAE to non-negative matrix factorization (NMF) and algorithms for DGMs: the VAE (Kingma and Welling, 2014); 3-VAE (Higgins et al., 2017); VSC (Tonolini et al., 2020); and OL-VAE
The sparse VAE implementation may be found httpp://ggit
8

---

Published in Transactions on Machine Learning Research (09/2022)
0
0-0.2
2-0.4
0.6
p-0.3
Method
2.0-
2.0
2.0-
20-
2.0
1.5
1.5
1.5
15-
1.5
0.75
0.75
0.75
0.75
S SpanseVAE
MSE
1.0-
1.0
10-
1.0
0.50
0.50
0.50
0.50
0.50
VAE
0.5
03-
0.25
0.25
0.25
0.25
0.25
OIVAE
0.0
00-
0.0
0.00
0.00
0:00
0.00
VSC
(a) Heldout mean squared error (lower is better)
(b) Ground truth factor recovery (higher is better)
Figure 3: Synthetic data. (a) Sparse VAE has better heldout predictive performance than the VAE over range of factor correlation levels. (b) Sparse VAE recovers the true factors better than the VAE. (3-VAE performed similarly to VAE). Scores are shown for 25 datasots per correlation setting.
(Ainsworth et al., 2018). None of the underlying DGMs have identifiablity guarantees. We find that: 1) on synthetic data, the sparse VAE achioves the best recovery of ground truth factors both when the factors are correlated and uncorrelated; 2) the sparse VAE achieves the best predictive performance on both synthetic and real datasets; 3) the sparse VAE achieves the best predictive performance generated from factors that different correlation to those in the training data; and (4) the sparse VAE finds interpretable factors in both text and movie-ratings data. On a single-cell genomics dataset, the sparse VAF learns factors which predict cell type with high accuracy.
heldout data that is
Datascts. We consider the following datascts; further details about the data are given in Appendix C.I.
Synthetic data. We consider again the syuthetic dataset generated as Eq. We vary correlation between the true factors from p 0, 0.2, 0.4, 0.6, 0.8.
PeerRead (Kang et al., 2018). Dataset of word counts for paper abstracts (N 10, 000, G 500).
MovieLens (Harper and Konstan, 2015). Dataset of binary user- movie ratings (N 100, 000, 300).
Semi-synthetic PeerRead. A semi-synthotic version of the PocrRead dataset in which the correlations between dala-generating factors differ across the test and training data, inducing different training and test distributions.
Zeisel (Zeisel al., 2015). Datasel of RNA molecule counts in mouse cortex cells (N 3005, G 558).
Implementation details. Empirically; we found that setting the prior covariance of the factors to Σ. IK worked well, even when the true generative factors were correlated (Figure 3a). Further implementation details for all methods are in Appendix C.
Recovering the ground truth factors. How well does the sparse VAE recover the factors when the ground truth factors are known? We assess this question with simulated data. We use the DCT disentanglement score to measure the fidelity between estimated factors and true factors (Eastwood and Williams, 2018). The DCI score is an average mcasure of how relevant each estimated factor is for the truc factors. This score also penalizes estimated factors that equally informative for multiple true factors.
We create synthetic datasets with the model given in Eq. 9 and evaluate the DCL metric between estimated and true factors as the true factors are increasingly correlated. As the correlation increases, we expect a standard VAE to conflate the factors while the sparse VAE recovers the two true underlying factors.
We see this phenomenon in Figure 3b; the sparse VAE has higher DCI scores than the VAE in all settings. The sparse VAE is robust to factor correlations up p = 0.4, with decreasing performance as p is further increased. Here, VSC performs worse than both the sparse VAE and VAE (the MSE scores for VSC in were too large for visualization in Figure 3a). The poor performance of VSC is likely due to the true generative factors in Eq. 9 not being sparse; only the mapping between the factors and features is sparse. Note that we do not compare with NMF in this example; as NMF is a linear method, it cannot reconstruct the nonlinear data generation process using only two latent dimensions.
Heldout reconstruction. We compare the negative likeliiod on heldout data achieved by the sparse VAE and related methods. All results are averaged over five splits of the data, with standard deviation in
9

---

Published in Transactions on Machine Learning Research (09/2022)
Table 1: On movie-ratings and text data, the sparse VAE achieves the lowest heldout negative log- log-likelhhood (NLL). For the B-VAE, we show the result with best performing B.
(a) MovieLens
(b) PeerRead
Method
NLL
Recall@5
NDCG@10
Method
NLL
Sparse VAE
170.9 (2.1)
0.98 (0.002)
0.98 (0.003)
Sparse VAE
245.0 (2.0)
VAE
175.9 (2.4)
0.97 (0.001)
0.96 (0.001)
VAE
252.6 (1.4)
3-VAE (8= (8=2) 2)
178.2 (2.4)
0.95 (0.002)
0.93 (0.002)
8-VAR (8=2) = 2)
254.5 (3.0)
OI-VAE
212.1 (2.6)
0.51 (0.004)
0.50 (0.004)
OI-VAE
260.6 (1.2)
VSC
192.2 (2.3)
0.79 (0.008)
0.77 (0.009)
VSC
252.9 (2.0)
NMF
198.6 (2.5)
0.90 (0.004)
0.85 (0.004)
NMF
267.2 (1.0)
Table 2: On the semi-syuthetic PeerRead data, the sparse VAE achieves the lowest heldout negative log- likelihood. We create three settings where the difference between training and test data distributions range from high (hardest) to low (easiest).
Difference between train and test Low
High
Medium
Method
Negative ligeliidl
Sparse VAE
52.4 (0.4)
49.2 (0.3)
48.6 (0.1)
VAE
54.6 (0.5)
52.3 (0.2)
50.8 (0.2)
3-VAE (3= =
54.7 (0.3)
52.1 (0.2)
51 (0.4)
OI-VAE
65.7 (0.3)
64.4 (0.3)
64.6 (0.2)
VSC
58.7 (0.6)
56.1 (0.3)
55.4 (0.2)
NMF
60.1 (0.02)
58.0 (0.08)
57.3 (0.6)
parentheses. Figure 3a shows that the sparse VAE performs better than the compared methods on synthetic data. On MovieLens and PeerRead datasets, Tables la and 1b show that the SparseVAE achieves the lowest heldout negative log- ligeliidd among the compared methods. For the MovieLens dataset, Table la addi- tionally shows that the sparse VAE has the highest heldout Recallå5 and normalized discounted cumulative gain (NDCG@10), which compare the predicted rank of heldout items to their true rank (Liang et al., 2018).
Different training and test distributions. How does the sparse VAE perform when the factors that generate data are correlated differently across training and test distributions? This particular type of shift in distribution alfects many real world settings. For example, we may estimate document representations from scientific papers where articles about machine learning also often discuss genomics, and want to analyze new collections of papers where articles about machine learning rarely
use the representations involve genomics. We hypothesize that because the sparse VAE associates each latent factor with only few features (e.g., words) even when the factors are correlated, it will reconstruct differently distributed test data better than the related methods.
We assess this question using the semi-synthetic PeerRead dataset, where the train and test data were generated by factors with different correlations. Table 2 summarizes the results from three settings where the difference between training and test data distributions range from high (hardest) to low (casicst). We report the average negative log-likelihood across five semi-simulated datasets. The sparse VAE performs the best, highlighting its ability to estimate models which generalize better to test data where the factors have a different distribution.
Interpretability. We now examine whether the sparse VAE finds interpretable structure in the data. For each factor in the MovieLens dataset, we consider the four movies with the largest selector variable Øik values (Table 3a). The sparse VAE finds clear patterns: for example, the top movics in first factor are all science fiction movies; the second factor contains animated children's movies; the third factor contains three Star Wars movies and an Indiana Jones movie, all blockbuster science fiction movies from the 1980s.
10

---

Published in Transactions on Machine Learning Research (09/2022)
Table 3: On movie-ratings and text data, the sparse VAE finds meaningful topics via the matrix W.
(a) MovieLens
(b) PeerRead
Topic
Movics
Topic
Words
The Fifth Flement; Alien; Gattaca; Aliens
A
task; pollcy; planning: heuristic; decision
B
A Bug's Life; Monsters, Inc.; Toy Story 3 & 2
B
information; feature; complex; sparse; probability
C
Star Wars V, IV & IV; Indiana Joncs 1
C
given; uetwork; method; bayesian; neural
Table 4: On the Zeisel data, the sparse VAE has the best heldout cell type prediction accuracy, followed closely by a VAF.
Methad
Sparse VAE
VAE (8 = 1)
NMF
VSC
OIVAE
Accuracy
0.95 (0.003)
0.94 (0.016)
0.91 (0.007) 0.89 (0.062)
0.80 (0.019)
We perform the same analysis for the PeerRead dataset (Table 3b). The sparse VAE finds some themes of computer science papers: the first factor contains words related to reinforcement learning; the second factor contains words about information theory; the third factor involves words about neural networks. We note that NMF also finds interpretable topics in both MovieLens and PeerRead (Appendix C.5.1); however, NMF has worse heldout predictive performance than the sparse VAE (Tables la and 1b). The sparse VAE retains the interprctability of NMF while incorporating flexible function estimation.
Finally, we consider the single-cell genomics dataset of Zeisel et al. (2015). We examine how well the factors learned by each method predict the cell type in a multinomial regression (note the cell type is not used when lcarning the factors). The sparse VAE is the best performing mothod (Table 4). Although the VAE is competitive this setting, we see that. the sparse VAE does better than methods such as OI-VAE and VSC that were designed to produce interpretable results.
5 Discussion
We developed the sparse DGM, a model with SSL priors that induce sparsity in the mapping between factors and features. Under an anchor feature assumption, we proved that the sparse DGM model has identifiable factors. To fit the sparse DGM, we develop the sparse VAE algorithm. On real and synthetic data, show the sparse VAE performs well. (i) It has good heldout predictive performance. (ii) It generalizes to out of distribution data. (iii) It is interpretable.
There are a few limitations of this work. First, the sparse DGM is designed for tabular data, where each feature has a consistent meaning across samples. Image data does not have this property as a specific pixel has no consistent meaning across samples. Second, the sparse VAE algorithm is more computationally intensive than a standard VAE. This expense is because at each gradient step, it roquires a forward pass through the network for each of the G features. Finally, we did not provide theoretical estimation guarantees for the sparse VAF algorithm. Similarly to other algorithms for fitting DGMs, estimation guarantees for the sparse VAE are difficult to obtain due to the nonconvexity of the objective function. The majority of works which consider identifiability in DGMs also do not provide estimation guarantees for their algorithms, including Locatello et al. (2020); Khemakhem et. al. (2020). We leave such theoretical study of the sparse VAE algorithm to future work.
Acknowledgments
Funding to support this research was provided for by the Eric and Wendy Schmidt Center, the Canada-CIFAR AI Chairs program, National Science Foundation Grant 17-1-2131, ONR N00014-15-1-220 the Simons Foundation, the Sloan Foundation, and Open Philanthropy.
NSF IIS 2127869, ONR N00014-
11

---

Published in Transactions on Machine Learning Research (09/2022)
References
Ahuja, K., Hartford, J., and Bengio, Y. (2022). Properties from mechanisms: an equivariance perspective on identifiable representation learning. In International Conference on Learning Representations.
Ainsworth, S. K., Foti, N. J., Lee, A. K., and Fox, E. B. (2018). oi-VAE: Output interpretable VABs for nonlincar group factor analysis. In International Conference on Machine Learning.
Arora, S., Ge, R., Halpern, Y., Mimno, D., Moitra, A., Sontag, D., Wu, Y., and Zhu, M. (2013). A practical algorithm for topic modeling with provable guarantees. In International Conference Machine Learning.
Barello, G., Charles, A. S., and Pillow, J. W. (2018). Sparse-coding variational auto-encoders. bioRrin.
Bengio, Y., Courville, A., and Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798 1828.
Bernardo, J. M., Bayarri, M. J., Berger, J. O., Dawid, A. P., Heckerman, D., Smith, A. F. M., and West, M. (2003). Bayesian factor regression models in the "large P. small n" paradigm. Bayesion Statistics, 7:733-744
Bhattacharya, A. and Dunson, D. B. (2011). Sparse Bayesian infinite factor models. Biometrika, pages 291-306.
Bing, X., Bunca, F., Ning, Y., and Wegkamp, M. (2020). Adaptive estimation in structured factor models with applications to overlapping clustering. Annals of Statistics, 48(4):2055-208 2081.
Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993-1022 1022.
Bolstad, B. (2018). preprocessCore: A collection of pre-processing functions. R. package version 1.44.0.
Bolstad, B. M., Irizarry, R. A., Astrand, M., and Spoed, T. P. (2003). A comparison of normalization methods for high density oligonucleotide array data based variance and bias. Bioinformaliis,, 19(2):185 -99..
Carvalho, C. M., Chang, J., Lucas, E., Nevins, J. R., Wang, Q., and West, M. (2008). High-dimensional sparse factor modeling: applications in gene expression genomics. Journal of the American Statistical Association,
1456.
Carvalho, C. M., Polson, N. G., and Scott, J. G. (2009). Handling sparsity via the horseshoe. In Artificial Intelligence and Statistics.
Castillo, I., Schmidt-Hieber, J., and Van der Vaart, A. (2015). Bayesian linear regression with sparse priors. The Annals of Statistics, 13(5) 2018.
Dittadi, A., Träuble, F., Locatello, F., Wuthrich, M., Agrawal, V., Winther, O., Bauer, S., and Schölkopf, B. (2021). On the transfer of disentangled reprosentations in realistic settings. In International Conference on Learning Represenlations.
Donoho, D. L. and Stodden, V. (2003). When does non-negative matrix factorization give correct decom- position into parts? In Neural Information Processing Systems.
Eastwood, C. and Williams, C. K. (2018). A framework for the quantitative evaluation of discntangled representations. In International Conference on Learning Representations.
Eriksson, J. and Koivunen, V. (2004). Identifiability, separability, and uniqueness of linear ICA models. IEEE Signal Processing Letlers, 11()):601-60) 604.
Gershman, S. and Goodman, N. (2014). Amortized inference in probabilistic reasoning. In Proceedings of the Annual Meeting of the Cognitive Science Society.
Ghosh, P., Tang, X., Ghosh, M., and Chakrabarti, A. (2016). Asymptotic properties of Bayes risk of general class of shrinkage priors in multiple hypothesis testing under sparsity. Bayesion Analysis, 11(: 796.
12

---

Published in Transactions on Machine Learning Research (09/2022)
Ghosh, S., Yao, J., and Doshi-Velez. F. (2019). Model selection in Bayesian neural networks via horseshoe priors. Journal of Machine Learning Research, 20(182):1-46:
Griffiths, T. L. and Ghahramani, Z. (2005). Infinite latent feature models and the Indian buffet process. In Newral Information Processing Systems.
Hälvä, H., Corff, S. L., Lehéricy, L., So, J., Zhu, Y., Gassiat, E., and Hyvarinen, A. (2021). Disentangling identifiable features from noisy data with structured nonlinear ICA. In Neural Information Processing Systems.
Harper, F. M. and Konstan, A. (2015). The MovieLens Datasots: History and Context, ACM Transactions on Interactive Intelligent Systems (THS), 5(4):1 19.
Higgins, 1., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017). beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations.
Horan, D., Richardson, E., and Weiss, Y. (2021). When is unsupervised disentanglement possible? In Neural Information Processing Systems.
Kang, D., Ammar, W., Dalvi, B., van Zuylen, M., Kohlmeier, S., Hovy, E., and Schwartz, R. (2018). A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. arXiv preprint arXiv: 1xXxix1804400000 09645
Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A. (2020). Variational autoencoders and nonlinear ICA: A unifying framework. In Artificial Intelligence and Statistics.
Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. International Conference on Learning Representations.
Kingma, D. P. and Welling, M. (2014). Auto-encoding variational Bayes. In International Conference on Learning Representations.
Knowles, D. and Ghahramani, 7. (2011). Nonparametric Bayesian sparse factor models with application gene expression modeling. Annals of Applied Statistics, 5(2B):1534 1552.
Küggeggen, J. V., Sharma, Y., Gresele, L., Brendel, W., Schölkopf, B., Besserve, M., and Locatello, F, (2021). Self-supervised learning with data augmentations provably isolates content from style. In Newral Information Processing Systems.
Lachapelle, S., Rodriguez, P., Sharma, Y., Everett, K. E., Priol, R. L.,, Lacoste, A., and Lacoste-Julien, (2022). Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In Conference on Causal Learning and Reasoning.
Liang, D., Krishnan, R. G., Hoffman, M. D., and Jebara, T. (2018). Variational autoencoders for collaborative filtering. In World Wide Web Conference.
Locatello, F., Abbati, G., Rainforth, T., Bauer, S., Schölkopf, B., and Bachem, O. (2019a). On the fairness of discntangled representations. Neural Information Processing Systems.
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. (2019b). Challeng- ing common assumptions in the unsupervised learning of discntangled representations. In International Conference on Machine Learning.
Locatello, F., Poole, B., Rätsch, G., Schölkopf, B., Bachem, O., and Tschannen, M. (2020). Weakly- supervised disentanglement without compromises. In International Conference on Machine Learning.
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., and Yosef, N. (2018). Deep generative modeling for single-cell transcriptomies. Nature Methods, 15(12):1053-10 -1588
Mita, G., Filippone, M., and Michiardi, P. (2021). An identifiable double VAE for disentangled representa- tions. In International Conference Machine Learning.
13

---

Published in Transactions on Machine Learning Research (09/2022)
Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate infer- ence in deep generative models. In International Conference on Machine Learning.
Rhodes, T. and Lee, D. (2021). Local disentanglement in variational auto-encoders using Jacobian 11 regu- larization. In Neural Information. Processing Systems.
Ročková, V. and George, F. I. (2016). Fast Bayesian factor analysis via automatic rotations to sparsity. Journal of the Americon Statistical Association, 111(516):1608 1622.
Ročková, V. and George, E. I. (2018). The spike-and-slab lasso. Journal of the American Statistionl Associ- ation, (13(541):331 444.
Rohe, K. and Zeng, M. (2020). Vintage factor analysis with varimax performs statistical inference. arXiv preprint ar arXiv:2044.05887
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statisticnl Sociely: Series (Methodologicaa)) 58(1):267 288.
Tonolini, F., Jensen, B. S., and Murray-Smith, R. (2020). Variational sparse coding. In Uncertainty in Artificial Intelligence.
Träuble, F., Creager, E., Kilbertus, N., Locatello, F., Dittadi, A., Goyal, A., Schölkopf, B., and Bauer, S. (2021). On disentangled representations learned from correlated data. In International Conference Machine Learning.
Yalcin, and Amemiya, Y. (2001). Nonlinear factor analysis a statistical method. Statistical Science, pages 275-294.
Zeisel, A., MunozzManchado, A. B., Codcluppi, S., Lönnerberg, P., Manno, G. L., Juréus, A., Marques, S., Munguba, H., He, L., Betsholtz, C., Rolny, C., Castelo-Branco, G., Hjerling-Leffler, J., and Linnarsson, S. (2015). Cell types in the mouse cortex and hippocampus revealed by single-cell RNA-seq. Science, 34266626)
Zhou, D. and Wei, X.-X. (2020). Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-vae. Neural Information Processing Systems.
14

---

Published in Transactions on Machine Learning Research (09/2022)
A Proofs
A.1 Proof of Theorem 1
We consider the case of known anchor leatures, where we have at least. two anchor features per factor. We consider the case where the likelihood is Gaussian. We have solutions, (0, 2,02) and (0, 2, G²), with equal likelihood. We show that 2 must be coordinato-wise transform of Σ.
We have
N
G
Jij
f
m
Tig J0, (10g の 22);
(17)
where fay() (+) denotes fo(-), the jth output of fo.
We prove that, given the anchor feature for factor k, zu is identifiabl. As all factors are assumed to have anchor features, this holds for all k E f1....,Ky K}. Suppose j is an anchor feature for k. That is, W/t = 0 O). With slight abuse of notation, we then write
for all + Then, Wj O 2 = (O....,0, 0, 00,..,,,,,,2,0000 0,
w 24) foz Zi))
0
Now, for Eq. 17 to hold for any value of { N.G we must have, for all j = 1, I....,G: G:
Dzz
fa
(w))
Fzz
fūz (
(18)
fog Ōi)
fny (in
f (wyx Tak)
f (w
Σ
2
0
0.
77
03
dj
63
1=1
(19)
For Eq. 19 to hold for all Jaj, we must have the coefficients equal to zero:
σ² =3
(20)
03
and
fg, (w)
fo (2(a
frg (w) Sog (1))
(21)
03
03
If f FR + R is invertible, then we have
3
ū f= ( (1))
(22)
That is, zu is identifiable up to coordinate-wise transformation.. If fo, is not invertible, then there exists bijective functions g + R. IR such that 0gcZ = g( and the equality is satisfied
J (wk) Jū (b(g(())Zz))) (as fo, is not invertible, Tz Mg(()zZ()))
(23)
J Zxx)
(24)
--1 As have Ex. Wijk g( Esk), 2sk is identifiable up to coordinate-wise transformation.
15

---

Published in Transactions on Machine Learning Research (09/2022)
A.2 Proof of Theorem 2
To prove the result, we:
Approximate the covariance d Σ Cov(æ;) using a first order Taylor approximation.
Show that an anchor feature for factor always has larger covariance with another anchor feature for i than other non-anchor features. Specifically: for any ls E (I,...,K) and j = anchor for k, we have
E N Cov(æaj, za) CR E Bij for all ( anchor for
i=l
1= Cov(zij, xa) < CR E Bigk for all I o anchor for k.
3. Apply Theorem of Bing et al. (2020) to prove that the anchor features can be identified from the covariance the data.
Regularity condition. Let ((zi) = E B(ax./2.))
We assume: for all z, in neighborhood of 2, B(z,],
0p(z;)
((z;)) = N
2 +
2; -,:2((1 2n) + 01((
(25)
N
0z:
where (1) denotes random variable converging to 0 in probability.
Step 1. Here is the marginal covariance of the Sparse VAE:
Var(æ;) = E [Var(æ: xx]] + Var(E [2s|2:])
(26)
= Σ Var(u(z;))
(27)
Take the first order Taylor expansion juzz)) = 2, +
0u(x,)
dz: 2; (z: - 2,).
Then
0u(2,)
@u(zz) Var(z;)
Var(u(z:)) =
(28)
dzi
dzi =2,=2;
Now, (1z;)) (Se, (w, x))5 Then, for any two features, and have
Sf,, (wj 3 2;)
0fe, (w) O 2i)
Cov(I.y,
CR
(29)
0zik
0zw
R
=1=21
where (ar(2;)=C::
We re-write fa, (-) to expose the first layer of the neural network (before the activation function is applied):
Step 2.
fo, (un, ④ 2,) mo, (usz.)
(30)
where u.y { with
K
Mijd
de Wik
(31)
R=1
where (H ( D..K
Jd,k=1 are the weights of the first neural network layer with dimension Dj.
16

---

Published in Transactions on Machine Learning Research (09/2022)
Then,
K
D1
amo, (an,(ar)))) Duijd
Omo (ual)
Cov(Tig, xaa)
Wikh Wik!
Σ
H
H de
ORe
(32)
Duai
k==16 ===1
K
WikWikk Bij
Buxe CRK'
(33)
where Bigk Ede Hih
Omee (1495.)
Oussd
Suppose j is an anchor feature for k. Then the absolute covariance of feature j and any other feature is:
[Cov(rg sa)l
Wike Baw Cwvll
(34)
K
S WkeWgs Bisk
Σ
Wike Back
by A3.
(35)
k=1 by A2.
S CRaWir Big
(36)
Step 3.
Step 2, we proved the equivalent of Lemma of Bing et al. (2020), adapted for the Sparse VAE. This allows us to apply Theorem of Bing al. (2020), which proves that the anchor Teatures can be determined uniquely from the approximate covariance matrix, N. Ei Cov(æ;;, as N 00.
A.3 Discussion of identifiability assumptions
In this section, we examine the suitability of assumption A2. We do so by showing A2 holds for three-layer neural network with ReLU activation functions, independently distributed Gaussian weights and no bias terms. Specifically:
Dig m,, z,] (3) ReLU(( HeL o z()))
(37)
where H()) E P>* are the weights for layer 1 with D hidden units, H Rx are the weights for layer 2, and H denotes the jth row of the weights for layer 3, (3) RRXD
We have
Op(zz))
Σ d==1d==1
Σ
D
1 3 H.g Hadd W
way [ ( @ 2)) > 0)
Σ
① > 0 (38)
0zik
where I(-) is the indicator function.
Assune {wis) jık= Tik1 are independent and symmetric around zero. Assume all weights are independent and
(m)) s.dd distributed as: dg.ds ~ N(O, for all layers 1,2,3.
Taking the first order Taylor approximation, for any two features, j and I, we have
D ΣΣΣΣ
D
Cov(aig, R
Ck
Σ (H j.da
((2)
H wH)) (3) H(2) H())
1, dg de wee)
(39)
4=
K
x
dy
ReLU(H (w) 2.) >
Σ
H
di W 24 >
(40)
K
x H ReLU(H w. o 2 > 0/
Win Zix
(41)
di
17

---

Published in Transactions on Machine Learning Research (09/2022)
As the neural network weights independent Gaussians, we keep only the terms where dj d' and d2 d22
K
K
Cov(rij, 2
Σ
J.d2
H
1,d2
(3)
Σ
dg
Cuu
H (1) dy dy W.x
(42)
d==1
d1
1= M=1
I
ReLU
( ( 3 2:)) 0 I
H di k
Wikk 2ssk >
(43)
d2
1
H 2
ReLU(H w o 2.) =2>00
H
di. Wikk 3in > 0
(1)
(44)
If j and are both anchor features for factor k we have:
Cov(zig, 2a) Σ
H
j,dz
(3)
)2
Σ
(H Jk (H () Pu3e
(45)
d==1
d1=1
K
0/
x T
H (2)
ReI.U(H 2 2
> 0
Σ
H
(1)
(46)
j.dz as H(3) -H and Wys = wts by dcfinition of an anchor feature.
Meanwhile, if and are not anchor features for the same factor, we have
Cov(sig, vx) Σ H3 H Adz
(47)
d==1
where
Adz
(2(2) (2)
dy
)
ΣΣ
CRu H11)
"
de u
H (e 2,)) > 0]
(48)
d1
k=1
x T
K
d
0
[
dz
Re o 222 > o]
H (1)
d WiA Z
0)
(49)
As the weights are independent, we have that J,d. ( Ad, are independent. For large D, we then have
Σ
H H.d Hd
3 H 33) Ad.
+
(50)
Hence, for two anchor features and I and non-anchor feature m, have Cov(sig, > Cov(rijj Jim.)
18

---

Published in Transactions on Machine Learning Research (09/2022)
B Inference
B.1 Derivation of ELBO for identity prior covariance
In the main body of the paper, we set the prior covariance to Σ; I as it performed well empirically and was computationally efficient. For this setting, the objective is derived as
N
log (fDulesize W)p(z))dz, log (/e(IIyy(TTbbnmaaa
logPo(2, W. n)
Σ
(51)
N
p(zi) 2s (2)))))) 1)
( ((W(T)p(TT))el
Σ
log Bgo(znlar)) po( W)
+ log Briw."
W, n)
(52)
Σ
W)] Der((0(((/(s]]lo((]]]] Briw
W (oFD(((D))(TTm)>>>
C(1, W Wn
(53)
The KL divergence between the variational posterior (s(=((x;)) and the prior 2, (0, is:
K [ log(o)
Σ
- oz(2))]
(54)
km
The final term of the ELBO in Eq. 53 is:
ΣΣ
(w M1
(0) (0)
[urgel
Bl7gk 1 jk 7
(1) (0)
log m/
k=1Ø=1
(55)
G
Elaga With mh [01 +
(1)
log(1 mk),
(56)
where
T) 1/ (Wix)
E 173h Wypks mk]
(57)
7h 1h (w)z) + (1 1 (o
X (wik. = AIT Waik 7m] + Ao - B(TsK (u5ke mel).
(58)
As described Algorithm 1, we alternate between updating B[778 1u m and me, and taking gradient steps for 0,0 and W.
B.2 Derivation ELBO for general prior covariance
In this section, we dotail the sparse VAE ELBO for general Σ.. We do not study this algorithm in this paper but include the derivation here for completeness.
The change in the sparse VAE ELBO for general Σ, is the KL divergence term:
K 1 + (0g(0)(2;))) tr (r (iigooa() - m0(2)) "2' mo(2)) log Σ
D
1=1
(59)
19

---

Published in Transactions on Machine Learning Research (09/2022)
C Details of empirical studies
C.1 Dataset details and preprocessing
PecrRead (Kang et al., 2018). We discard any word tokens that appear in fewer than about 0.1% of the abstracts and in more than 90% of the abstracts. From the remaining word tokens, we consider the G = 500 most used tokens as features. The observations are counts of each feature across ~ 11,11000000 000 abstracts.
Semi-synthetic PeerRead The training and testing data are distributed differently because we vary the correlations among the latent factors that generate the features across the two datasets. This data generated as follows: (i) we applied Latent Dirichlet Allocation (LDA, Blei et al., 2003) to the PeerRead dataset using K 20 components to obtain factors 0 E RRNK and topics (loadings) 8 E RKxG (ii) We created train set factors, ftr, by dropping the last K columns of 0 and replacing them with columns calculated as: logit 0. = logit 0,2 + (O, o²) for each of the first 2 latent dimension k. We the test set factors as Pte
MovicLens (Harper and Konstan, 2015). We consider the MovicLens 25M dataset. Following (Liang al., 2018), code ratings four higher one and the remaining ratings zero. We retain users who have rated more than 20 movies. We keep the top G = 300 most rated movies. Finally, we randomly subsample N 100, 000 users.
Zeisel (Zeisel et al., 2015). We first processed the data following (Zeisel et al., 2015). Next, we normalized the gene counts using quantile normalization (Bolstad et al., 2003; Bolstad, 2018). Finally, following (Lopez et al., 2018), we retained the top G 558 genes with the greatest variability over cells.
C.2 Sparse VAE settings
For all experiments, the Sparse VAE takes the prior hyperparameters to be a 1, = G, where G is the number of observed features.
For experiments with Gaussian loss, the prior on the error variance is:
of Inverse- (rrease-Gammm////2.v///
(60)
We set 3. The hyperparameter & is set to a data-dependent value. Specifically, we first calculate the sample variance of each feature, E.j. Then, we such that the 5% quantile of the sample variances the 90% quantile of the overestimate of the true noise variance. The smaller sample variances are assumed to correspond to mostly noise and not signal - these variances are used to calibrate the prior on the noise.
prior. The idea is: the sample variance is an
C.3 Experimental settings
For the neural networks, we use multilayer perceptrons with the same number of nodes in each layer. These settings are detailed in Table 6.
For stochastic optimization, we use automatic differentiation in PyTorch, with optimization using Adam (Kingma and Ba, 2015) with default settings (betal=0.9, beta2-0.999)
For LDA, we used Python's sklearn package with default settings. For NMF, we used sklearn with alpha=1.0.
The dataset-specific experimental settings are in Table 6.
C.4 Additional experiments
C.4.1 Synthetic data
We consider additional experiments to answer the questions:
20

---

Published in Transactions on Machine Learning Research (09/2022)
p
0.6
p
1.5
1.5
1.5
15-
1.5
MSB
1.0
1.0
1.0
10-
1.0
0.8
0.50
0.50
0.75
0.50-
0.75
Method
SparsovAE
0.6-
0:25
Sparsevas StaB
0.0
0.4
0.00
(a) Heldout mean squared error (lower better)
(b) Ground truth factor recovery (higher is better)
0.8
M Method
SparsevAE
0.4
Sparevva
(c) F-score of W support recovery (higher is better)
Figure 4: Synthetic data. The Sparse VAE with less regularization on W (SparseVAE-Slab) performs slightly worse than Sparse VAE with more regularization on W (SparsoVAE) in terms (a) MSE; (b) DCT Disentanglement Score; and (c) F-score of the estimated support W.
1. How does the sparse VAE perform when the regularization parameter in the sparsity-inducing prior is decreased?
2. Does the Spike-and-Slab Lasso prior perform similarly to an alternate sparsity-inducing prior, the Horseshoe prior (Carvalho et al., 2009)?
Experiment We find that the sparse VAE with less regularization (Ao = è = 0.001) performs slightly worse than the sparse VAE with Ag 10, A1 = 1 terms of MSE (Figure 4a) and DCI (Figure 4b). The sparse VAE with less regularization still performs well, however this is because the W learns also tends to be somewhat sparse (Figure 4c). Note that when An Å1 the Spike-and-Slab Lasso prior is equivalent to Lasso prior (Tibshirani, 1996).
Experiment 2. We find that the Spike-and-Slab Lasso and the horseshoe perform similarly for lower values of the factor correlation P, but that the Spike-and-Slab has lower MSF and higher DCT than the horseshoe for higher values p.
To implement the horseshoe prior, we follow Ghosh et al. (2019) and parameterize the half- Calf-Cauchy distribu- tion C+ (0,b)) as:
a a~ C,,06 e a² ~ Inv-Gamma(1/2, 1//,, À ~ Inv-Gamma(1/22
(61)
Then, we assign W a horseshoe prior:
Aglobal Inv- Nan-Gamma(1/22 1//2, yladhat)
(62)
Atoeal
Iny-Gamma /2, 1/1 1/
(63)
Tyk Atocal Inv-Gamma/ (1/2, 1/ \/oca))
(64)
Uk Agglobal Inv Inv-Gamma (1/2, 1/Aglobat),
(65)
W Tyk: Uk ~ ((0, (f,,
(66)
where Inv-Gamma(a.) b) the unverso-aammn distribution with density p(s) x s-a-l bb/s) for O. We set the hyperparameters to bytobat blocat 1.0.
21

---

Published in Transactions on Machine Learning Research (09/2022)
p.0.2
p.0.8
0.4
2.0-
2.0
2.0-
2.0-
2.0
1.00
1.00
1.00
1.00
1.00
15
1.5
1.5
15.
1.5
0.75-
0.75
0.75
0.75
0.75
Method
MSB
1.0
10-
1.0
parssvAE
0.25
0.25
0.00
0.00
0.30
0.00
0.00
(a) Heldout mean squared error (lower is better)
(b) Ground truth factor recovery (higher is better)
Figure 5: Synthotic data. The Sparse VAE with the Spike-and-Slab- prior performs slightly better
than Sparse VAE with horseshoe prior W in lerms of (a) MSE; (b) DCT Disentanglement Score.
With this horseshoe prior on W, we again optimize W using coordinate ascent and then use closed-form updates for the SiverseGaammr parameters:
jk 0.51l (1)
(1-1) jk 11
(20k
+ +1/
(67)
G
"
( 2/(G+3)
/2 + 1/2 (4-1) jk
(0)2
global
(68)
K,G
1 2/(GK +
Σ
1/T + 1 //occal (t)2
(69)
AJ=1
(1)
globol
= =2/(K +3)
Σ
1/vb
(0)2
+ 1 l/bglobel
(70)
22

---

Published in Transactions on Machine Learning Research (09/2022)
G--00
G-1000
170
Method
160
320
SparsevAE
5000
20000
100000
5000
20000
100000
samples
(a) Negative heldout log likelihood (lower is better)
G-300
G-1000
0.9
0.9
Methad
Recall
0.7
SparsevAE
0.6
0.6
5000
20000
100000
5000
20000
100000
samples
(b) Heldout recall (higher is better)
Figure 6: MovieLens. The Sparse VAE generally performs as well as or better than a VAE on (a) negative heldout log- log-likeldhod and (b) heldout recall when varying the sample size and number of features (movics). Results averaged over splits of the data.
C.5 MovieLens: varying sample size and number of features
In this section, we consider the performance of the Sparse VAE when we vary the sample size and number of features (movies) in the MovieLens dataset. The settings are the same noted in Table 6, except for the latent space dimension which is set to K 50.
Generally, the Sparse VAE performs as well as or better than a VAE on negative heldout log- ligeliieldoood and heldout recall (Figure 6). For the largest setting (N 100, 000, 1, 000), the Sparse VAE about 100 times slower than VAF.
C.5.1 MovieLens and PeerRead: comparisons with NMF
We include additional comparisons of the Sparse VAE with non-negative matrix factorization (NMF) on the MovieLens and PeerRead datasets.
On the MovieLens data, examples of topics found by NMF include those related to science fiction; animated childrens' movies; and Star Wars (although Toy Story is included in the Star Wars topic). This performance is similar to the Sparse VAE.
On the PeerRead data, examples of topics found by NMF include those related to reinforcement learning; language modeling: and neural network models.
In summary, NMF finds interpretable topics, similarly the Sparse VAE. However, the Sparse VAE has better heldout predictive performance than NMF. This provides evidence for the Sparse VAE retaining the interpretability of linear methods while improving model flexibility.
C.6 Compute
GPU: NVIDIA TITAN Xp graphics card (24GB).
CPU: Intel E4-2620 v4 processor (64GB).
23

---

Published in Transactions on Machine Learning Research (09/2022)
Table On MovieLens (left) and PeerReads (right), NMF finds meaningful topies.
Topic
Movies
Topic
Words
A
Alien; Aliens; Terminator 2; Terminator; Blade Runner
A
agent; action; policy; state; game
B
Shrek & 2; Monsters, Inc.; Finding Nemo; The Incredibles B
word; representation;; vector; embeddinng;; semantic
C
Star Wars IV,
VI; Toy Story
model; inference; neural; latent; structure
Table 6: Settings for each experiment.
Synthetic data
MovieLens
Settings
Value
Settings
Value
# hidden layers
3
hidden layers
# layer dimension
50
# layer dimension
300
Latent space dimension
5
Latent space dimension
30
Learning rate
0.01
Learning rate
0.0001
Epochs
200
Epochs
100
Batch size
100
Batch size
100
Loss function
Gaussian
Loss function
Softmax
Sparse VAE
= Ao = 10
Sparse VAE
= 1, Ao = 10
8-VAE
[2, 4, 6, 8, 16]
B-VAE
[2, 4, 6, 8, 16)
VSC
= 0.01
VSC
0.01
OI-VAE
A = 1,p =5
OI-VAE
A == 1, CPU, 1 hour
Runtime per split
CPU, 2 mins
Runtime per split
PearRead
Semi-syuthelic PeerRead
Settings
Value
Settings
Value
# hidden layers
3
# hidden layers
# layer dimension
100
# layer dimension
50
Latent space dimension
20
Latent space dimension
20
Learning rale
0.01
Learning rate
0.01
Epochs
40
Epochs
50
Batch size
128
Batch size
128
Loss function
Softmax
Loss function
Softmax
Sparse VAE
Åι 0.001, Ao 55
Sparse VAE
Åι = 0.01, Ao =
3-VAE
[2, 4, 6, 8, 16]
3-VAE
[2, 4, 6, 8, 16]
VSC
0.01
VSC
0.01
OL-VAE
A = 1,p
OI-VAE
= 1, =
Runtime per split
GPU, 20 mins
Runtime per split
GPU, 30 mins
Zeisel
Settings
Value
# hidden layers
3
# layer dimension
100
Latent space dimension
15
Learning rate
0.01
Epochs
Batch size
512
Loss function
Gaussian
Sparse VAE
Aι =1 λo 10
VSC
a 0.01
OL-VAE
A 1,p= =
Runtime per split
CPU, 15 mins
24


Reviews:
[{"review_id": "7t1LDHOt9B", "reviewer": "Reviewer_TQnQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is well written and contributes to the DGM literature with a new modeling approach, theoretical analysis, and experiments designed to illustrate the proposed approach's main benefits.\n\nWeakness:\n* While the approach has some advantages over DGMs (e.g. identifiability) it has more limited applicability (can only be applied to tabular data, i.e. where the number of observed features is fixed and where each observed feature has the same interpretation across data points).\n* Compared to the generic VAE, the proposed approach is also more computationally expensive.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method called Sparse VAE where the mapping between latent factors and observed features is enforced to be sparse in the sense that each observed feature depends only on a small number of latent factors. This is achieved by computing each observed feature independently using the same decoder but with a different subset of latent factors as input (obtained via a mask on the full set of latent factors). The mask varies across output features but is the same across data points, which the authors motivate using a genomics application where each observed feature always corresponds to the same gene. The authors further motivate the approach through an identifiability theoretical analysis, highlighting that compared to other DGMs the proposed approach is identifiable. Finally the authors demonstrate the benefits of the approach on a number of synthetic and real tasks.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Given that identifiability has been analyzed in the context of NMF, I would consider either including NMF in Figure 3 (identifiability) or describing why one cannot use NMF in that context. In general, given the tabular data assumption, I think that the stronger the discussion (and empirical comparison) with alternative methods tailored for tabular data like movie reviews, the stronger the paper.\n\nOverall though, the paper is well written with all the strengths and weaknesses discussed clearly. The paper describes how the identifiability analysis motivates the modeling choices (e.g. highlighting that the SSL prior is likely to yield the sparse mapping that satisfies the anchor assumption required for identifiability) and there is at least one (synthetic) experiment to illustrate each claim e.g. identifiability, interpretability, and improved held out performance on real data. Therefore my recommendations above are not critical for securing my recommendation for acceptance.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "37fjeNXqvT", "reviewer": "Reviewer_1jnL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Clarity: I found the paper clear, well written and easy to follow\n* Novelty: The idea to leverage the anchor assumption to obtain identifiability is neat and this work presents a compelling instantiation of this idea.\n\nWeaknesses\n* Evaluation: I think the biggest concern that I have is that method is solely evaluated on datasets that are relatively small (the largest number of features is ~500 and the largest number of training samples is ~100K). In order to push the limits of the method, its worthwhile to extend the evaluation to larger datasets. \n   * The MovieLens dataset comes in a variety of different sizes, including the MovieLens20M which has in the tens of thousands of features. Given that this method is restricted to tabular data, I do think it is important for the work to evaluate on such larger training datasets. \n   * It would be interesting to study the effect of increasing training/feature size. Specifically, this would enable practitioners to assess the utility of this method in different datasize/feature regimes.\n   * Another reason to consider increasing the dimensionality of features is that tabular data often exhibits long-tails which can inhibit learning (e.g. https://arxiv.org/abs/1901.05534, https://arxiv.org/abs/1710.06085). This regime presents an interesting opportunity to assess the degree to which model restrictions that guarantee identifiability help/hurt learning on tabular data. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new family of deep generative models that is identifiable by virtue of the anchor assumption (which states that there are at least two features which depend solely on that factor). Intuitively, this assumption effectively means that the observation serves as a noisy proxy for the latent variable itself and is what is used to guarantee identification of the model. The model is instatiated via a spike and slab\nprior which is used to define a mask over which latent factors are used in the conditional distribution over the observations given the latent variables.\n\nThe model is evaluated on synthetic data, where it is found to recover the underlying sparsity of the generating process. In addition it is evaluated on several tabular datasets. On MovieLens (Recsys) it is found to improve upon recall and NDCG. On PeerRead, it is found to improve upon likelihood. On a data of mouse cortex cells, the latent representation learned by the model is found to improve predictive accuracy.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "* Running an evaluation on larger datasets \n* Evaluate how the model performs as a function of feature size (and number of training examples)\n* What happens in scenarios where there is model misspecification? i.e synthetic data where the anchor assumptions is violated? ", "broader_impact_concerns": "To the best of my knowledge, there are no broader impact concerns I anticipate with this manuscript.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YHRttDBccW", "reviewer": "Reviewer_KcCj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well written and easy to understand.\n- Identifiability in DGMs is a very useful but elusive feature and this work provides a way to achieve it.\n- The empirical performance of the proposed sparse VAE looks very promising.\n\nWeaknesses:\n\n- It is unclear how realistic some of the theoretical assumptions are.\n- The connection between the theory and the presented sparse VAE model could be discussed in more detail.\n- Some of the design choices are not fully motivated; ablation studies could help.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new type of deep generative models (DGMs), called sparse DGMs, and prove analytically that under certain assumptions these models are identifiable. They then instantiate this model class with an amortized inference version, called sparse VAE, and show empirically that this model outperforms existing VAE models across a range of tasks.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall, this is a really nice paper and I think it will be a nice contribution to this journal. However, I think there are a few improvements that could be made.\n\nCritical:\n\n- It is unclear to me how strong assumption A1 is, especially that the two anchor features have the exact same mapping $f_j$. Does this mean that the features essentially have to be copies of each other? How realistic is this assumption in the real data?\n- As mentioned by the authors, their theory does not apply directly to the proposed sparse VAE, due to its non-convexity. It would be useful to discuss this discrepancy a bit more and give an intuition for how \"close\" the sparse VAE gets to the identifiable regime or what kind of adjustments in the model make it more or less identifiable.\n- Relatedly, some of the design decisions are not motivated very clearly. For instance, how would the SSL prior work if the slab would not be a Laplace distribution? How would it compare to, e.g., a horseshoe prior? Some ablation studies would be useful to motivate why this particular model design is the best one.\n\nMinor:\n\n- Since the proposed sparse VAE is computationally more expensive than some of the baselines, it would be useful to report runtimes in the experiments to get a better intuition for how much additional compute one has to invest for the improvement in performance.", "broader_impact_concerns": "There are no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces sparse VAE, a deep generative model applicable when each observed dimension is dependent on a potentially different small subset of latent variables. This sparse dependence is modelled by applying a different learned sparse mask to the latent vector for each observed dimension. The authors prove that the resulting model is identifiable under certain assumptions and show that it performs well in practice. The paper is well written and makes progress in a difficult area. The authors have addressed most of the reviewer concerns with their rebuttal.
Decision: Accept (poster)