Identifying through Flows for Recovering Latent Representations
Authors: Shen Li
Bryan Hooi
Gim Hee Lee
Venue:  (2020)
Keywords: Representation learning
identifiable generative models
nonlinear-ICA

Abstract: Identifiability, or recovery of the true latent representations from which the observed data originates, is de facto a fundamental goal of representation learning. Yet, most deep generative models do not address the question of identifiability, and thus fail to deliver on the promise of the recovery of the true latent sources that generate the observations. Recent work proposed identifiable generative modelling using variational autoencoders (iVAE) with a theory of identifiability. Due to the intractablity of KL divergence between variational approximate posterior and the true posterior, however, iVAE has to maximize the evidence lower bound (ELBO) of the marginal likelihood, leading to suboptimal solutions in both theory and practice. In contrast, we propose an identifiable framework for estimating latent representations using a flow-based model (iFlow). Our approach directly maximizes the marginal likelihood, allowing for theoretical guarantees on identifiability, thereby dispensing with variational approximations. We derive its optimization objective in analytical form, making it possible to train iFlow in an end-to-end manner. Simulations on synthetic data validate the correctness and effectiveness of our proposed method and demonstrate its practical advantages over other existing methods.

Published as a conference paper at ICLR 2020
IDENTIFYING THROUGH FLOWS FOR RECOVERING LATENT REPRESENTATIONS
Shen Li
Institute of Data Science & NUS Graduate School for Integrative Sciences and Engineering National University of Singapore shen. 1 1i@u. nus.edu
Bryan Hooi & Gim Hee Lee
Department of Computer Science, National University of Singapore [bhooi, gimhee. lee comp. nus. edu.
ABSTRACT
Identifiability, or recovery of the true latent representations from which the ob- served data originates, is de facto fundamental goal of representation learning. Yet, most deep generative models do not address the question of identifiability, and thus fail to deliver on the promise of the recovery of the true latent sources that generate the observations. Recent work proposed identifiable generative mod- elling using variational autoencoders (iVAE) with a theory of identifiability. Due the intractablity of KL divergence between variational approximate posterior and the true posterior, however, iVAE has to maximize the evidence lower bound (ELBO) of the marginal likelihood, leading suboptimal solutions in both the- ory and practice. In contrast, we propose an identifiable framework for estimating latent representations using flow-based model (iFlow). Our approach directly maximizes the marginal likelihood, allowing for theoretical guarantees on iden- tifiability, thereby dispensing with variational approximations. We derive its op- timization objective in analytical form, making it possible to train iFlow in an end-to-end manner. Simulations on synthetic data validate the correctness and ef- fectiveness of our proposed method and demonstrate its practical advantages over other existing methods.
INTRODUCTION
A fundamental question in representation learning relates to identifiability: under which condition it possible to recover the true latent representations that generate the observed data? Most exist- ing likelihood-based approaches for deep generative modelling, such as Variational Autoencoders (VAE) (Kingma & Welling, 2013) and flow-based models (Kobyzev et al., (019), focus on per- forming latent-variable inference and efficient data synthesis, but do address the question of identifiability, i.c. recovering the true latent representations.
The question of identifiability is closely related the goal of learning disentangled representations (Bengio et al., 2013). While there is no canonical definition for this term, we adopt the one where individual latent units are sensitive to changes in single generative factors while being relatively invariant to nuisance factors (Bengio et al. 2013). A good representation for buman faces, for ex- ample, should encompass different latent factors that separately encode different attributes including gender, hair color, facial expression, etc. By aiming to recover the true latent representation, identifi- able models also allow for principled disentanglement; this suggests that rather than being entangled in disentanglement learning in completely unsupervised manner, go step further towards iden- tifiability, since existing literature on disentangled representation learning, such as 3 3-VAE (Higgins et al. 2017), 3-TCVAE (Chen al, 2018). DIP-VAE (Kumar al. 2017) and FactorVAE (Kim & Mnih 2018), are neither general endeavors to achieve identifiabiliy;; nor do they provide theoretical guarantees recovering the true latent sources.

---

Published as a conference paper at ICLR 2020
Recently, KKeemahhem al. (2019) introduced theory of identifiability for deep generative models, based upon which the authors proposed an identifiable variant of VAEs called iVAE, to learn the distribution over latent variables in an identifiable manner. However, the downside of learning such an identifiable model within the VAE framework lies in the intractability of KI. divergence between the approximate posterior and the true posterior. Consequently, in both theory and practice, iVAE inevitably leads to suboptimal solution, which renders the learned model far less identifiable.
In this paper, aiming at avoiding such pitfall, we propose to learn identifiable generative model through flows (short for normalizing flows (Tabak et al. 2010)] Rezende & Mohamed 2013)). A nor- malizing flow is a transformation of a simple probability distribution (e.g. standard normal) into a more complex probability distribution by composition of series of invertible and differentiable mappings (Kobyzev et al. 2019). Hence, they can be exploited to effectively model complex prob- ability distributions. In contrast to VAEs relying on variational approximations, Row-based models allow for latent-variable inference and likelihood evaluation in an exact and efficient manner, making themselves perfect choice for achieving identifiability.
To this end, unifying identifiablity with flows, we propose iFlow, framework for deep latent- variable models which allows for recovery of the true latent representations from which the observed model makes it possible to directly maximize
data originates. We demonstrate that our the conditional marginal likelihood and thus achieves identifiability in a rigorous manner. We pro- vide theoretical guarantees on the recovery of the true latent representations, and show experiments on synthetic data to validate the theoretical and practical advantages of our proposed formulation over prior approaches.
2 BACKGROUND
An enduring demand in statistical machine learning is to develop probabilistic models that explain the generative process that produce observations. Oftentimes, this entails estimation of density that can be arbitrarily complex. As one of the promising tools, Normalizing Flows are a family of generative models that fit such a density of exquisite complexity by pushing an initial density (base distribution) through series of transformations. Formally, let x X C R" be an observed random variable, and z E 2 C IR" a latent variable obeying a base distribution (z). A normalizing flow f is diffeomorphism (i.e an invertible differentiable transformation with differentiable inverse) between two topologically equivalent spaces X' and Z such that x f(z). Under these conditions, the density of well-defined and can be obtained by using the change of variable formula:
Oh Ox
of Oz
px (x(x) = (h(()) del
(z) del
where h is the inverse of f. To approximate an arbitrarily complex nonlinear invertible bijection, one can compose a series of such functions, since the composition of invertible functions is also invert- ible, and its Jacobian determinant is the product of the individual functions' Jacobian determinants. Denote & as the diffcomorphism'' 's learnable parameters. Optimization can proceed as follows by maximizing log- likelihokh for the density estimation model:
arg max Ex log (1( x)) + log det Oh(x: Φ) |
ix
3 RELATED WORK
Nonlinear independent component analysis (ICA) is one of the biggest problems remaining unresolved unsupervised learning. Given the observations alone, it aims to recover the inverse mixing function as well their corresponding independent sources. In contrast with the linear case, research on nonlinear ICA is hampered by the fact that without auxiliary variables, re- covering the independent latents is impossible (Hyvärinen & Pajunen| 1999). Similar impossibility result can be found in (Locaeelo et al. (2018). Fortunately, by exploiting additional temporal struc- ture on the sources, recent work (Hyvarinen & Morioka 2016) Hyvarinen et al. (018)) established the first identifiability results for deep latent-variable models. These approaches, however, do not explicitly learn the data distribution; nor are they capable of generating "fake" data.
Nonlinear ICA
2

---

Published as a conference paper at ICLR 2020
Khemakhem et al. (2019) bridged this gap by establishing principled connection between VAEs and an identifiable model for nonlinear ICA. Their method with an identifiable VAE (known as iVAE) approximates the true joint distribution over observed and latent variables under mild condi- tions. Nevertheless, due to the intractablity of KI. divergence hetween variational approximate pos- terior and the true posterior, iVAE maximizes the evidence lower bound on the data log-likelihood, which in both theory and practice acts as detriment to the achievement of identifiability.
We instead propose identifying through flows (normalizing flow), which maximizes the likelihood in a straightforward way, providing theoretical guarantees and practical advantages for identifiability.
Normalizing Flows Normalizing Flows are family of generative approaches that fits data distribution by learning a bijection from observations to latent codes, and vice versa. Compared with VAEs which learn posterior approximation to the true posterior. normalizing flows directly deal with marginal likelihood with exact inference while maintaining efficient sampling. Formally, a normalizing flow is a transform of a tractable probability distribution into a complex distribution by compositing a sequence of invertible and differentiable mappings. In practice, the challenge lies in designing a normalizing flow that satisfies the following conditions: (1) it should be bijective and thus invertible; (2) it is efficient to compute its inverse and its Jacobian determinant while ensuring sufficient capabilities.
The framework of normalizing flows first defined in (Tabak et al. 2010) and (Tabak & Turner 2013) and then explored for density estimation in (Rippel & Adams, 2013). Rezende & Mohamed (2015) applied normalizing flows to variational inference by introducing planar and radial flows. Since then, there had been abundant literature towards expanding this family. Kingma & Dhariwal (2018) parameterizes linear flows with the LU factorization and "1 1" convolutions for the sake of efficient determinant calculation and invertibility of convolution operations. Despite their limits in expressive capabilities, linear flows act essential building blocks of affine coupling flows as in (Dinh et al. 2014, 2016). Kingma et al. (2016) applied autoregressive models as a form of normalizing flows, which exhibit strong expressiveness in modelling statistical dependencies among variables. However, the forwarding operation of autoregressive models is inherently sequential, which makes it inefficient for training. Splines have also been used as building blocks of normalizing flows: Müller et al. (2018) suggested modelling a linear and quadratic spline as the integral of a univariate monotonic function for flow construction. Durkan et al. (2019a) proposed a natural extension to the framework of neural importance sampling and also suggested modelling coupling layer as a monotonic autional-qudrian spine (Durkan et al.] 2019b), which can be implemented either with a coupling architecture RQ-NSF(C) with autoregressive architecture RQ-NSF(AR).
The expressive capabilities of normalizing flows and their theoretical guarantee of invertibility make them natural choice for recovering the true mixing mapping from sources to observations, and thus identifiability can be rigorously achieved. In our work, we show that by aligning normalizing flows with an existing identifiability theory, it is desirable to learn identifiable latent-variable model with theoretical guarantees of identifiability.
IDENTIFIABLE FLOW
In this section, we first introduce the identifiable latent-variable family and the theory of identifi- ability (Khemakhem al. 2019) that makes it possible to recover the joint distribution between observations and latent variables. Then we derive our model, iFlow, and its optimization objective which admits principled disentanglement with theoretical guarantees of identifiability.
4.1 IDENTIFIABLE LATENT-VARIE VARIABLE FAMILY
The primary assumption leading to identifiability is conditionally factorized prior distribution over the latent variables, Pe (z|u), where u is an auxiliary variable, which can be the time index in a time series, categorical label, an additionally observed variable (Khemakhem al. 2019).
Formally, let x E X C R" and u E U C IR' be two observed random variables, and z E Z S R" latent variable that is the source of x. This implies that there can be an arbitrarily complex nonlinear mapping Z + X. Assuming that is a bijection, it is desirable to recover its inverse by approximating using family of invertible mappings h& parameterized by Φb The statistical
3

---

Published as a conference paper at ICLR 2020
dependencies among these random variables are defined by Bayesian net: - from which the following conditional generative model can be derived:
p 2 θ) p(x|z; p) u/u; A)));
where p(x 2;; (b) def Pe (x and p(z) u: T, A) assumed be factorized exponential fam-
ily distribution conditioned upon Note that this density assumption is valid in most cases. since the exponential families have universal approximation capabilities (Sriperumbudur al.) 2017). Specifically, the probability density function is given by
Qi(z,) (u) exp
&
(T.A(Z(U)
II p: (z: u)
II
Σ [t.(s))ds
(4)
where Qi the base measure, (u) is the normalizing constant, T.jss 's are the components of the sufficient statistic and (u) the natural parameters, critically depending on u. Note that k indicates the maximum order of statistics under consideration.
4.2 IDENTIFIABILITY THEORY
The objective of identifiability is to learn model that is subject to:
for each quadruplet (9, O', pe) per (x) px( =p:: pa. 2),
(5)
where @ and O' two different choices of model parameters that imply the same marginal density (Khemakhem et al. 2019). One possible avenue towards this objective is to introduce the definition of identifiability up equivalence class:
Definition 4.1. (Identifiability up equivalence elass) Let be an equivalence relation on θ, A model defined by p(x, z; θ) = p(x z; ()p(z; θ) is said to be identifiable up to ~ if
pe (x) per (x)
θ ~~ O',
(6)
where such equivalence relation in the identifiable latent-variable family is defined follows:
Proposition 4.1. (Φb T, X) and (ø', TV, T²,") are the same equivalence class if and only if there exist A and such that V X,
Txh((x)) = AT (har))()))) -
where
T(z) = (Q. À(u) = (Z, Zn (u), (1,1 (n))
Tn,n(zn))
(8)
One can easily verify that ~ is an equivalence relation by showing its reflexivity, symmetry and tran- sitivity. Then, the identifiability the latent-variable family is given by Theorem 4.1 (Khemahhem et al. 2019).
Theorem 4.1. Let 2 = 21 x 2n and suppose the following holds: (i) The set fx E (x
0} has measure zero, where Ve is the characteristic function of the density pei (ii) The sufficient statisties Tig in (2) differentiable almost everywhere and aT., /0z of almost surely for E 2, and for all {1,..., n) and k1. (iii) There exist (nk + 1) distinct priors uº u such that the matrix
(u,1 (u+) (u.)
(u A1.1 (u9)
==
(nn) - (u9) (ukkun An))
of size nk nk is invertible. Then, the parameters (Φ) T, 2) are --identifiable.
4

---

Published as a conference paper at ICLR 2020
4.3 OPTIMIZATION OBJECTIVE OF IFLOW
We propose identifying through flows (iFlow) for recovering latent representations. Our proposed model falls into the identifiable latent-variable family with O, that is, pe(-) 5(-). where is point mass, i.e. Dirac measure. Note that assumption (i) in Theorem [4. holds true for iFlow.
In stark contrast to iVAE which resorts to variational approximations and maximizes the evidence lower bound, iFlow directly maximizes the marginal likelihood conditioned on u:
The
max θ) pz (2 0) det
(10)
0x
where pz(-|u) is modeled by a factorized exponential family distribution. Therefore, the log marginal likelihood is obtained:
(x: = Σ (Lg (1(2)) log Zz(u) Tx(2;) > + log det
The 0x
(11)
where 2; is the ith component of the source z = ho(x); and and À are both n-by-k matrices.
Here, h& is normalizing flow of any kind. For the sake of simplicity, we set Qi(2,) = 1 for all i's and consider maximum order of sufficient statistics of 2 up to 2, that is, t = 2. Hence, and X given
21
E1
m
22
&2
T22
T(z)
and A(u)
(12)
Zn
En
Tn
Therefore, the optimization objective is to minimize
(()) (x. Σ log (1(
log
trace ((z))(u)²) -log log
det
Th
(13)
where pD denotes the empirical distribution, and the first term (13) is given by
log Zi(u) log
JRm
[II
exp
trace (T(e)((uTT) dz
1=1
log
exp Σ E12 + 17/24 dz
R
1=1
= log
exp (Ei + n22)dz
(14)
i=1
72
log
exp
5
4g;
i=1
Σ log
1/
45,
In practice, A(u) can be parameterized by multi-layer perceptron with learnable parameters 0, where No m - 2R Here, is the dimension of the space in which u's lies. Note that & should be strictly negative in order for the exponential family's probability density function to be finite. Negative softplus nonlinearity can be exploited force this constraint. Therefore, optimization proceeds by minimizing the following closed-form objective:
ni 4E,
trace (T()))(())T) logg det
Th Ox
min L()) E(x.u) upp 8
log
E
(15)
where O ().
5

---

Published as a conference paper at ICLR 2020
4.4 IDENTIHIABILITY OF IFLOW
The identifiability of our proposed model, iFlow, is characterized by Theorem 4.2
Theorem 4.2. Minimizing Le with respect to θ, in the limit of infinite data, learns a model that is --identifiable.
Proof. Minimizing Le with respect to & equivalent to maximizing the log conditional like- lihood, lxg (x @). Given infinite amount of data, maximizing log | θ) will give
us the true marginal likelihood conditioned on u, that is, px (x]u; θ) px (x u: θ*), where Px é = arg maxe log px (x|u: () and O* is the true parameter. According to Theorem 4.1 we obtain that and @* are of the same equivalence class defined by r. Thus, according to Definition 4.1 the joint distribution parameterized by θ is identifiable up
Consequently, Theorem 4.2 guarantees strong identifiability of our proposed generative model,
iFlow. Note that unlike Theorem 3 in (Khemakhem et al. 2019), Theorem makes assumption that the family of approximate posterior distributions contains the true posterior. And we show in experiments that this assumption unlikely to hold true empirically.
SIMULATIONS
To evaluate our method, we run simulations on a synthetic dataset. This section will elaborate on the details of the generated dataset, implementation, evaluation metric and fair comparison with existing methods.
5.1 DATASET
We generate a synthetic dataset where the sources are non-stationary Gaussian time-series, as de- scribed in (Khemakhem al. 2019): the sources divided into M segments of samples each. The auxiliary variable u is set to be the segment index. For each segment, the conditional prior distribution is chosen from the exponential family (4), where k Q,(2;) 1, and T., (2;) 2?, (=2(=)) = and the true λ., 's are randomly and independently generated across the segments and the components such that their variances obey uniform distribution on [0.5, 3]. The sources to recover mixed by invertible multi-layer perceptron (MLP) whose weight matrices ensured to be full rank.
5.2 IMPLEMENTATION DETAILS
The mapping Ag that outputs the natural parameters of the conditional factorized exponential family distribution is parameterized by multi-layer perceptron with the activation of the last layer being the softplus nonlinearity. Additionally, a negative activation is taken on the second-order natural parameters in order ensure its finiteness. The bijection h& is modeled by RQ-NSF(AR) (Durkan et al. 2019b) with the flow length of 10 and the bin 8, which gives rise to sufficient flexibility and expressiveness. For each training iteration, we use mini-batch of size 64, and an Adam optimizer with learning rate chosen in {0.01, 0.001} to optimize the learning objective (15).
5.3 EVALUATION METRIC
As a standard measure used in ICA, the mean correlation coefficient (MCC) between the original sources and the corresponding predicted latents chosen be the evaluation metric. high MCC indicates the strong corrllation between the identified latents recovered and the true sources. In experiments, we found that such a metric can be sensitive to the synthetic data generated by different random seeds. We argue that unless one specifies the overall generating procedure including random seeds particular any comparison remains debatable. This is crucially important since most of the existing works failed to do so. Therefore, run each simulation of different methods through seed to seed 100 and report averaged MCCs with standard deviations, which makes the comparison fair and meaningful.

---

Published as a conference paper at ICLR 2020
original sources
observaiion
low (MCC: 0.911 MAL (MOC) 0.73)
original sources
oiservations
IMCC: 0.90) IVAL (NCC) 0.41
(b)
originall sources
observations
0.66)
original sources
0.85) (NCC)
(d)
Figure 1: Visualization of 2-D cases (best viewed in color).
5.4 COMPARISON AND RESULTS
We compare model, iFlow, with iVAE. These two models are trained on the same synthetic dataset aforementioned, with = 40, = 1000, n = d = 5. For visualization, we also apply an- other setting with M 40, 1000, = d = 2. To evaluate iVAE's identifying performance, use the original implementation that is officially released]wai with exactly the same settings as described in (Khemakhem al. 2019) (cf. Appendix (A.2).
First, we demonstrate a visualization of identifiablity of these two models in a 2-D case (n = d 2) illustrated in Figure in which plot the original sources (latent), observations and the identified sources recovered by iFlow and iVAE, respectively. Segments are marked with different colors. Clearly, iFlow outperforms iVAE in identifying the original sources while preserving the original geometry of source manifold. It is evident that the latents recovered by iFlow bears much higher resemblance the true latent sources than those by iVAE in the presence some trivial indeterminacies of scaling, global sign and permutation of the original sources, which are inevitable even in some cases linear ICA. This exhibits consistency with the definition of identifiability up to equivalence class that allows for existence of affine transformation between sufficient statisties, as described in Proposition 4.1 As shown in Figure ((a) I(c) and I(d) iVAE achieves inferior identifying performance in the sense that its estimated latents tend to retain the manifold of the observations. Notably, we also find that despite the relatively high MCC performance of iVAE in Figure [(d] iFlow is much more likely recover the true geometric manifold in which the latent sources lie. In Figure [(b] iVAE's recovered latents collapses in face of a highly nonlinearly mixing case, while iFlow still works well in identifying the true sources. Note that these are rare
occurrences. More visualization examples can be found in Appendix A.3
https//ggthuu.ccm//ikee/////t

---

Published as a conference paper at ICLR 2020
0.9
0635) Asturni 10.686/.0 222
2261.00
0.0
IPlONeS, 8004
0.8
0.5
0.4
(a)
(b)
Figure 2: Comparison of identifying performance (MCC) and the energy value (likelihood loga- rithm) versus seed number, respectively.
corr: 0.8129
cOr:: 0.9543
COT:: 0.9318
COrT: 0.8801
COI:: 0.9460
iFlow
7
0.6887
0.8506
IVAE
Figure 3: Comparison of identifying performance (correlation coefficient) in each single dimension of the latent space, respectively. The dashed cyan line represents the source signal.
Second, regarding quantitative results as shown in Figure 2(a) our model, iFlow, consistently out- performs iVAE in MCC by considerable margin across different random seeds under consideration while experiencing less uncertainty (standard deviation as indicated in the brackets). Moreover, Fig- 2(b) also showcases that the energy value of iFlow is much higher than that of iVAE, which serves evidence that the optimization of the evidence lower bound, as in iVAE, would lead to sub- optimal identifiability. As is borne out empirically, the gap between the evidence lower bound and the conditional marginal likelihood is inevitably far from being negligible in practice. For clearer analysis, we also report the correlation coefficients for each source-latent pair in each dimension. As shown in Figure 3, iFlow exhibits much stronger correlation than does iVAE in each single di- mension the latent space.
Finally, we investigate the impact of different choices of activation for generating natural parameters of the exponential family distribution (see Appendix A for details). Note that all of these choices valid since theoretically the natural parameters form achieves the highest identifying performance, suggesting that the range of softplus allows for greater flexibility, which makes itself good choice for natural parameter nonlinearity.
space. However, iFlow(Softllus)
8

---

Published as a conference paper at ICLR 2020
6 CONCLUSIONS
Among the most significant goals of unsupervised learning is to learn the disentangled representa- tions of observed data, or identify original latent codes that generate observations (i.e. identifia- bility). Bridging the theoretical and practical gap of rigorous identifiability, we have proposed iFlow, which directly maximizes the marginal likelihood conditioned on auxiliary variables, establishing natural framework for recovering original independent sources. In theory, our contribution provides a rigorous way to achieve identifiability and hence recovery of the joint distribution between observed and latent variables that leads to principled disentanglement, Extensive empirical studies confirm our theory and showcase its practical advantages over previous methods.
ACKNOWLEDGMENTS
We thank Xiaoyi Yin for helpful initial discussions. This work is in loving memory of Kobe Bryant (1978-2020)
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1822 2013.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint 1rXiv:1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605. 08803, 2016.
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Cubic-spline flows, arXiv preprint arXiv:1906.02145. 2019a.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. arXiv preprint arXiv:1906.00 140.0022000 2019b.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2(5);6, 2017.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems, pp. 3765-3773, 2016.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12(3):429 439, 1999.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard E Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. arXiv preprint arXiv: 2018.
llyes Khemakhem, Diederik Kingma, and Aapo Hyvärinen. Variational autoencoders and nonlin- ear ica: A unifying framework. arXiv preprint arXiv: 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint 182.0.180..05983, 2018.
Diederik P Kingma and Max Welling. urXiv:1312.00 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ix1 convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.

---

Published as a conference paper at ICLR 2020
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp.
2016.
Ivan Kobyzev, Simon Prince, and Mareus A Brubaker. Normalizing flows: Introduction and ideas. arXiv preprint arXiv: (eti::/900.095555 2019.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. arXiv preprint arXiv:/711.00888, 2017.
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. arXiv preprint arXiv:/811.12359 2018.
Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. Neural im- portance sampling. arXiv preprint arXiv:1808.085 03856. 2018.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv: 055700 2015.
Oren Rippel and Ryan Prescott Adams. High-dimensional probability estimation with deep density models. arXiv preprint 13X.:113 2013.
Bharath Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Aapo Hyvärinen, and Revant Kumar. Density estimation in infinite dimensional exponential families. The Journal of Machine Learning Research, 18(1 18:0088-88888 2017.
Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics, 66(2):145-164, 2013.
Esteban G Tabak, Eric Vanden Eijnden,, et al. Density estimation by dual ascent of the log- likelihood. Communications in Mathematical Sciences, 8(1):217-233, 2010.
10

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 ABLATION STUDY ON ACTIVATIONS FOR NATURAL PARAMETERS
Figure 4 demonstrates the comparison of MCC of iFlows implemented with different nonlinear activations for natural parameters and that of iVAE, in which relu+eps denotes the ReLU activation added by small value (e.g. le-5) and sigmoid 5 denotes the Sigmoid activation multiplied by
MCC
0.6
0.5
0.4
0.3
softplus
relu+eps
sigmoidx5
sigmoid
NAE
Activations
Figure 4: Comparison of MCC of iFlows implemented with different nonlinear activations for natu- ral parameters and that of iVAE (best viewed in color).
A.2 IMPLEMENTATION DETAILS OF IVAE
As stated in Section 5.4, evaluate iVA''s identifying performance, the original implemen- tation that is officially released with the same settings as described in (Khemakhem et al. 2019). Specifically, in terms of hyperparameters inference model, as well as the conditional prior are parameterized by MLPs, where the dimension of the hidden layers is chosen from (50, 100, 200), the activation function is a leaky RELU leaky hyperbolic tangent, and the number of layers is chosen from {3, 4, 5, 6). Here we report all averaged MCC scores of different implementations for iVAE as shown Table
iVAE, the functional parameters of the decoder and the
Table indicates that adding more layers more hidden neurons does not improve MCC. score, precluding the possibility that expressive capability Instead, we argue that the assumption (i) of Theorem in (Khemakhem et al. 2019) (i.e the family of approximate posterior distributions contains the true posterior) often fails or is hard to satisfy in practice, which is of the major reasons for the inferior performance of iVAE. Additionally, Figure 2(b) demonstrates that the energy value of iFlow is much higher than that of iVAE, which provides evidence that optimizing the evidence lower bound, in iVAE, leads to suboptimal iden- tifiability.
the culprit of iVAE inferior performance.
11

---

Published as a conference paper at ICLR 2020
Table 1: Averaged MCC scores versus different iVAE implementations
NUM HIDDENS
NUM LAYERS
AVG
50
3
0.4964(=0.0723)
50
4
(.0.82(10.0740)
50
5
0.4521
50
6
0.4276(=000686)
100
3
0.4965(10.0693)
100
4
0.4696(=000688)
100
5
0.4555(=000736)
100
6
0.4253(+0.0795)
200
3
0.4961(=0.0753)
200
4
0.4723(++0070))
200
5
0.4501 (-0.066)
200
6
0.4148(=000703)
A.3 VISUALIZATION 2D CASES
original sources
dbservations
MMCC: 0.91) WAL 0.73) (a)
original
oiseervazioss iflow (MCC: 95) MAL IMCC: (b)
onginal cources
absorvations
FlON (.89) IMAE IMCC. 0.78)
onginal sources
ODSErvaTIOnS
iFlo (MCC: (AE IMCC 0.99
(d)
Figure 5: Visualization of 2-D cases (i) (best viewed in color).
12

---

Published as a conference paper at ICLR 2020
orginna sources.
deservations
IMCE: 0.92) 0.46)
original sources
observations
0.99) WAF IMOC:
onginal Sources
abservations
0.72) 0.34)
onginal
coservations
(c)
oriinnal
original
otiginal
original
(MCC::0.6)
(h)
Figure 6: Visualization of 2-D cases (ii) (best viewed in color).
13

---

Published as a conference paper at ICLR 2020
orginal sources
deservations
IMCE: 0.95) 0.39)
original sources
oiservations
0.79 WAP IMOC:
onginal abservations (c)
0.69)
onnginal
Ooservacions
0.92)
original
original
otiginal
original
(MCC))095) IAA
(h)
Figure 7: Visualization of 2-D cases (iii) (best viewed in color).
14

---

Published as a conference paper at ICLR 2020
oriinnal sources.
deservations
(MCE: 0.95) 0.87)
original sources
observations
(MCC: 0.99) WAE IMOC:
onginal abservations (c)
0.90)
onnginal
ooservations 0.9)
oriinnal
original
otiginal
original (MCC))099))
(h)
Figure 8: Visualization of 2-D cases (iv) (best viewed in color).
15


Reviews:
[{"review_id": "SJl5FcOf9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Overview\nThe paper tackles the identifiability problem in generative modeling, i.e., recovering the true latent representations from which the observed data originates. The paper argues that identifiable variational autoencoder (iVAE) suffers from intractability issue which leads to suboptimal solutions. The paper instead proposes an identifiable normalizing flows (iFlow) method as an alternative. The proposed iFlow outperforms iVAE in experiments using synthetic data. The paper is very well motivated and well supports its claim.\n\n## Summary of the contributions\n1. The paper proposes iFlow, an identifiable normalizing flow method which allows recovery of true latent space from observed data.\n2. The paper shows iFlow outperforms iVAE in synthetic experiments.\n3. The paper provides theoretical justification on the identifiability of the proposed iFlow method.\n\n## Overall feedback\nI find the paper well written and well organized. Easy to follow the content even though I am not expert on this matter. The paper provides both theoretical justification and empirical experimental validation which show the superior performance of proposed iFlow method. So I am leaning towards accepting the paper. The theory seems correct but I did not check all the equations and proofs in depth so I am not very confident about my rating.\n\n## Suggestions\n1. Please make sure all equations are properly punctuated.\n2. The comparison with iVAE seems tricky since they use different types of architectures. Can you also report the hyper parameters used for iVAE?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklCCT9ecr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about learning an identifiable generative model, iFlow, that builds upon a recent result on nonlinear ICA. The key idea is providing side information to identify the latent representation, i.e., essentially a prior conditioned on extra information such as labels and restricting the mapping to flows for being able to compute the likelihood. As the loglikelihood of a flow model is readily available, a direct approach can be used for learning that optimizes both the prior and the observation model.\t\n\nThe paper is very clear and very easy to follow. The idea is quite clear, and the direct approach is really attractive. Unfortunately, the experimental section is quite limited and does not fully study representational aspects. There is only an illustrative simulation on synthetic data, that in a sense verifies the theory. I was also not able to see the additional insight that the identifiability theory in 4.2 provides additional to Khemakhem et al. (2019). Please clarify.\n\nMy main concern is that this model is actually a supervised model that learns a mapping from u to x via z. Hence, the theoretical ‘identifiability’ guarantee needs to be stated with some care as this depends on the choice of an arbitrary u. For example if we set x = u, will learning take place? Please comment.  \n\nOverall, I like the approach but I am uncertain about the level of novelty. For such a paper, one should expect a much more involved computational study. In this respect, I feel that the paper could be accepted but it certainly feels as if it needs more computational results as otherwise the original contribution would be too incremental for ICLR standards. I am giving a provisional reject in the hope that the authors will provide convincing arguments about their original contributions for clarification.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:

Blind review #1 summarizes it well:

This paper is about learning an identifiable generative model, iFlow, that builds upon a recent result on nonlinear ICA. The key idea is providing side information to identify the latent representation, i.e., essentially a prior conditioned on extra information such as labels and restricting the mapping to flows for being able to compute the likelihood. As the loglikelihood of a flow model is readily available, a direct approach can be used for learning that optimizes both the prior and the observation model.	

--

Discussion:

Reviewer questions were mostly about clarification, which the authors addressed during the rebuttal period.

--

Recommendation and justification:

All reviewers agree the paper is a weak accept based on degree of depth, novelty, and impact.
Decision: Accept (Poster)