Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
Authors: Peter Sorrenson
Carsten Rother
Ullrich Köthe
Venue:  (2020)
Keywords: disentanglement
nonlinear ICA
representation learning
feature discovery
theoretical justification

Abstract: A central question of representation learning asks under which conditions it is possible to reconstruct the true latent variables of an arbitrarily complex generative process. Recent breakthrough work by Khemakhem et al. (2019) on nonlinear ICA has answered this question for a broad class of conditional generative processes. We extend this important result in a direction relevant for application to real-world data. First, we generalize the theory to the case of unknown intrinsic problem dimension and prove that in some special (but not very restrictive) cases, informative latent variables will be automatically separated from noise by an estimating model. Furthermore, the recovered informative latent variables will be in one-to-one correspondence with the true latent variables of the generating process, up to a trivial component-wise transformation. Second, we introduce a modification of the RealNVP invertible neural network architecture (Dinh et al. (2016)) which is particularly suitable for this type of problem: the General Incompressible-flow Network (GIN). Experiments on artificial data and EMNIST demonstrate that theoretical predictions are indeed verified in practice. In particular, we provide a detailed set of exactly 22 informative latent variables extracted from EMNIST.

Published as a conference paper at ICLR 2020
DISENTANGLEMENT BY NONLINEAR ICA WITH GENERAL INCOMPRESSIBLE-FON FLOW NETWORKS (GIN)
Peter Sorrenson, Carsten Rother, Ullrich Köthe
Visual Learning Lab Heidelberg University
ABSTRACT
A central question of representation learning asks under which conditions it is possible to reconstruct the true latent variables of an arbitrarily complex genera- tive process. Recent breakthrough work by Khemakhem et al. (2019) nonlinear ICA has answered this question for a broad class of conditional generative pro- cesses. We extend this important result real-world data. First, we generalize the theory to the case of unknown intrinsic problem dimension and prove that in informative latent variables will be automatically separated from noise by an es- timating model. Furthermore, the recovered informative latent variables will be in one-to-one correspondence with the true latent variables of the generating pro- cess, up to a trivial component-wisg transformation. Second, we introduce mod- ification of the RealNVP invertible neural network architecture (Dinh et al. 2016) which particularly suitable for this type of problem: the General Incompressible- flow Network (GIN). Experiments on artificial data and EMNIST demonstrate that theoretical predictions are indeed verified in practice. In particular, we provide a detailed set of exactly 22 informative latent variables extracted from EMNIST.
direction relevant for application
special (but very restrictive) cases,
INTRODUCTION
Deep latent-variable models promise to unlock the key factors of variation within a dataset, opening window to interpretation and granting the power to manipulate data in an intuitive fashion. The theory of identifiability linear independent component analysis (ICA) (Comon) 1994) tells us when this is possible, if we restrict the model to linear transformation, but until recently there was no corresponding theory for the highly nonlinear models needed to manipulate complex data. This changed with the recent breakthrough work by Khemakhem et al. (2019), which showed that under relatively mild conditions, it is possible to recover the joint data and latent space distribution, up to simple transformation in the latent space. The key requirement is that the generating process is conditioned on a variable which is observed along with the data. This condition could be class label, time index of time series, any other piece of information additional to the data. They interpret their theory as a nonlinear version of ICA.
This work extends this theory in direction relevant for application real-world data. The existing theory assumes knowledge of the intrinsic problem dimension, but this is unrealistic for anything but artificially generated datasets. Here, we show that in the special case of Gaussian latent space distributions, the intrinsic problem dimension can be discovered. The important latent variables are organically separated from noise variables by the estimating model. Furthermore, the variables dis- covered correspond to the true generating latent variables, up a trivial component-wise translation and scaling. Very similar results exist for other members of the exponential family with two param- eters, such as the beta and gamma distributions.
We introduce a variant of the ReaINVP (Dinh et al. 2016 invertible neural network: the General Incompressiill-ffov Network (GIN). The flow is called incompressible in reference to fluid dynam- ics, since it preserves volumes: the Jacobian determinant is simply unity. We emphasise its generality and increased expressive power in comparison to previous volume-preservee (Dinh et al. 2014). As already noted in Khemakhem et al. (201)) flow-bbsed generative models are natural fit for the theory of nonlinear ICA, the variational autoencoders (VAEs) (Kingma &
flows, such NICE

---

Published as a conference paper at ICLR 2020
Welling 2013) used in that work. For us, major advantages of invertible architectures over VAEs the ability to specify volume preservation and directly optimize the likelihood, and freedom from the requirement to specify the dimension of the model'ss latent space. An INN always has latent space of the same dimension as the data. In addition, the forward and backward models share parameters, saving the effort of learning separate models for each direction.
In summary, our work makes the following contributions:
We extend the theory of nonlinear ICA to allow for unknown intrinsic problem dimension. Doing so, we find that this dimension can be discovered and a one-to-one correspondence between generating and estimated latent variables established.
We propose implementation invertible neural network obtained by modifying the RealNVP architecture. We call our new architecture GIN: the General Incompressible-fon Network.
We demonstrate the viability of the model on artificial data and the EMNIST dataset. We extract 22 meaningful variables from EMNIST, encoding both global and local features.
2 RELATED WORK
The basic goals of nonlinear ICA stem from the original work linear ICA. An influential for-
mulation, as well as the first identifiability results, were given in Comon 1994). These stated the conditions which allow the generating latent variables to be discovered, when the mixing function
a linear transformation. However, it was shown in Hyvärinen & Pajunen (1999) that this approach to identifiability does not extend general nonlinear functions.
The first identifiability results in nonlinear ICA came in Hyvärimen & Morioka (2016) and Hyvärrinn & Morioka (2017). applied to time series, and implemented via discriminative model and semi- supervised learning. A more general formulation, valid for other forms of data, was given in Hyvärinen et al. (2018) and the theory was extended to generative models in Khemakhem et al. (201)), where experiments were implemented by VAE.
Many authors have addressed the general problem of disentanglement, and proposed models to learn disentangled features. Prominent among these is 3-VAE (Higgins et al., 2017) and its variations (e.g- Chen et al. 2018) which augment the standard ELBO loss with tunable hyperparameters to encourage disentanglement. There also attempts to modify the GAN framework (Goodfellow) (ct al.] (2044) such as InfoGAN (Chen et al. 2016), which tries to maximize the mutual information between some dimensions of the latent space and the observed data. Many of these approaches are unsupervised. However, as pointed out and empirically demonstrated in Locatello et al. (2018), unsupervised models without conditioning in the latent space are in general unidentifiable.
Several unsupervised VAE models implement conditioning in the latent space by means of Gaussian mixtures (Johnson et al.) 2016 [Dilokthanakul et al.] 2016; Zhao et al., 2019). Our work differs mainly by (i) only considering supervised tasks, therefore being safely covered by the theory of Khemakhem et al. (211)), and (ii) enforcing presev-prssrrattion,, not possible in VAE.
The invertible neural networks in this work build upon the NICE framework (Dinh et al. 2014) and its extension in RealNVP (Dinh al., (0166). A similar network design ours is Ardizzone et al. (2019). This is conditioned INN based on RealNVP, however the conditioning information is applied as a parameter of the network. The authors find in experiments with MNIST that meaningful variables are present in their latent space, but are rotated such that they are not aligned with the axes of the space. In this work, the conditioning only present as a parameter of the latent space distributions. As result, it is covered by the theory of Khemakhem et al. (2019) and its extension here, which results in non-rotated, meaningful latent variables.
3 THEORY
3.1 EXISTING THEORY OF NONLINEAR ICA
This section adapts theoretical results from Khemakhem et al. (2019) to the context of invertible neural networks. Suppose the existence of the following three random variables: latent generating
2

---

Published as a conference paper at ICLR 2020
GENERATING LATENT SPACE DISTRIBUTION
GENERATING MODEL
ESTIMATING MODEL
p(z u)
(2)
(x: 8)
C(w, u)
GENERATING LATENT
ESTIMATED LATENT VARIABLE
OBSERVED AATA CONDITION
VARIABLE
Figure 1: Relationship between the variables and functions defined in Sec. 3.1 as well as indication of the training scheme.
variable z E R", condition u Rn and a data point x E R where n S d. If the existence of a noise variable E Rd-" The variables z and make up the generating latent space, and are not necessarily known, whereas u and x are observable and therefore always known.
d, also suppose
The distribution of z is a factorial member of the exponential family with k sufficient statistics, conditioned on u. In its most general form the distribution can be written as
(z)
(z(z II ==1 (u)
Σ (is(()))sss))
exp
where the Ti., are the sufficient statistics, the >s.j their coefficients and Z; the normalizing constant. Qi is called the base measure, which in many cases is simply 1.
The distribution of must not depend on u and its probability density function must be always finite.
The variable is the result of arbitrarily complex, invertible, deterministic transformation from the generating latent space to the data space: x f(z,,), This can alternatively be formulated as an injective, non-deterministic transformation from the swerr-dmeeniioal z-space to the higher- dimensional x-space: x fz)..
In general, an observed dataset D will consist only of instances of and u. The task of nonlinear ICA is to disentangle the data to recover the generating latent variables z, as well as the form of the function f and its inverse. We can try to achieve this with sufficiently general, invertible function approximator g. which maps from a latent variable w E Rd to the data space: x = g(w; 8). Here 0 denotes the parameters of g. Note that the dimensions of the latent space and data space the same due to the invertibility of g. We assume that w follows a conditionally independent exponential probability distribution, conditioned on the same condition as z:
Q (w)) 1=1 (
d
Pw (w(w))) = II
exp Σ (wn))
(2)
The coefficients the sufficient statistics (u) are not restricted, which means that a given vari- able of the estimated latent space is allowed to lose its dependence u. If this occurs, we will consider this variable to encode noise, playing a role equivalent to E in the generating latent space.
In addition to this specification of the generative and estimating models, some conditions are nec- essary to ensure the latent variables can be recovered. The most important of these concerns the variability of ) under However, as long the Ar randomly and independently gener- ated, and there are at least nk + 1 distinct conditions u, this condition is almost surely fulfilled. See Appendix A. for further details.
In the limit of infinite data and perfect convergence, the estimating model will give the same condi- tional likelihood to all data points as the true generating model:
(xlu) = PT x (x
If this is the case, the vector of sufficient statistics T from the generating latent space will be related to that of the estimated latent space by an affine transformation:
T(z) AT' (w) +
(4)
3

---

Published as a conference paper at ICLR 2020
where A is some constant, full-rank, nk x dk matrix and E IR* some constant vector. The relationship holds for all values of z and w. The proof of this result can be found in Appendix A
This relationship is generalization of that derived in Khemakhem et al. (011). In that version, they assume knowledge of n, the dimension of the generating latent space, and give their estimating model latent space of the same dimension. In this context the matrix A defines relationship between latent spaces the same dimension, making it square and invertible.
3.2 NONLINEAR ICA WITH A GAUSSIAN LATENT SPACE
Given the relationship in equation lar special cases. We hope in particular to induce sparsity in the matrix A, so that the estimated latent space is related the true one by as simple a transformation as possible. We show (proof in Appendix D that when both the generating and estimated latent spaces follow a Gaussian dis- tribution, the generating latent space variables are recovered up to a trivial translation and scaling. Furthermore, the dimension of the generating latent space is recovered.
we can ask whether any stronger results hold for partieu-
More precisely, each generating latent variable 2; is related to exactly one estimated latent variable Wj. for some j. 2; 0:W, b, for some constant and b; Furthermore, each estimated latent variable W is related to at most one 25. If the estimating latent space has higher dimension than the dimension of the generating latent space, some estimating latent variables are not related to any gen- crating latent variables and so must encode only noise. This is the dimension discovery mechanism, since the estimated latent space organically splits into informative and non-informative parts, with the dimension of the informative part equal to the unknown intrinsic dimension the generating latent space. Very similar results can be derived for all common continuous woo-parameter members of the exponential family, including the gamma and beta distributions. See Appendix C
3.3 VOLUME PRESERVATION VIA INCOMPRESSIBLE FLOw
In a multivariate Gaussian distribution with diagonal covariance, the standard deviations are di- rectly proportional to the principal axes of the ellipsoid defining surface of constant probability. In classical PCA, this property used assign importance to each principal component based on its standard deviation. In PCA we can think of the rotation of the data into the basis defined by the principal components as the latent space. In a deep latent variable model the transformation be- tween the latent space and the data is significantly more complex, but if this transformation preserves volumes like the rotation in PCA, it will retain the desirable correspondence between the standard deviation of a latent space variable and its importance in explaining the data. Because invertible neural networks (normalizing flows) have a tractable Jacobian determinant, they open the possibility to constrain it to unity. This is equivalent to volume preservation and is the rationale behind the General Incompressible-foon Network.
EXPERIMENTS
Experiments on artificial datasets confirm the theory for a normally distributed latent space, as well identifying potential causes of failure. Experiments the EMNIST dataset (Cohen et al.) 2017) demonstrate the ability of GIN to estimate independent and interpretable latent variables from real- world data.
4.1 MODEL DESCRIPTION
The GIN model is similar in form to RealNVP (Dinh et al. (016) and shares its flexibility, but retains the volume-preserving properties of the NICE framework (Dinh et al. 2014).
Re coupling layers split D-dimensional input & into two parts, Vid and #441:D where d < D. The output of the layer is the concatentation yi.d and ya+1:D with
Vidd = 0i:d
(5)
V:+ Fd+1:D O exp(s(x1:a)) + t(x1a)
(6)
4

---

Published as a conference paper at ICLR 2020
OBSERVED DATA (PROJECTION)
GROUND TRUTH
RECONSTRUCTION
SPECTRUM
Figure 2: Successful reconstruction by GIN of the two informative latent variables out of ten in total. The other eight correctly identified noise. The observed data is ten-dimensional (projection into two dimensions shown here). The spectrum shows the standard deviation each variable the reconstruction (in black, log scale) which quantifies its importance. Ground truth is in gray. There is a clear distinction between the two informative dimensions and the noise dimensions, showing that GIN has correctly detected two-dimensional manifold in the ten-dimensional data presented to it.
where addition, multiplication and exponentiation are applied component-wise. The logarithm of the Jacobian determinant of coupling layer is simply the sum of the scaling function s(xi.))- Volume preservation achieved by setting the final component s(za) to the negative sum of the previous components, so the total sum of s(11aa) is zero (in contrast, NICE sets s(xiaa) = 0). Hence the Jacobian determinant is unity and volume preserved. The network is free to allow the volume contribution of some dimensions of the output of any coupling layer to grow, but only by shrinking the other dimensions of the output in direct proportion. As well as enforcing strong correspondence between the importance of latent variable and its standard deviation, believe vrrume-preservatioon has a regularizing effect, since it is a very strong constraint, comparable to orthonormality in linear transformations.
4.2 OPTIMIZATION
The experiments in this section deal with labeled data, where each data point belongs one of M different classes. This class label is used as the condition associated with each data point In the estimated latent space, all data instances with the same label should belong the same Gaussian distribution. Hence we are learning a Gaussian mixture in the estimated latent space, with M mixture components. Since the distribution for each class the estimated latent space is required to be factorial, the variance of each mixture component is diagonal, and we can write of (u) for the variance in the i-th dimension.
Given a set of data, condition pairs D ((x(1,, (..))...., (x()),, [N))) and model g. parameter- ized by (where maps from the latent space to the data space) we can construct a loss from the log-likelihood. Using the change of variables formula, we have log log p(x[u) no Jacobian term since the transformation w = g (x; 0) is volume-preserving likelihood of D, minimize the negative log- li-elikeiihod of in the estimated latent space:
log p(w /w u), with To maximize the
(9. (x: 0) (u 0))²
(()) (x.))D
log(or(u:0))
(u;
4.3
ARTIFICIAL DATA
4.3.1 EXPERIMENT CONDITIONS OF THEORY FULFILLED
Samples are generated in two dimensions, conditioned on five different cluster labels, see Fig. The means of the clusters chosen independently from uniform distribution on [-5. 5] and variances from a uniform distribution on [.5, 3] This data is then concatenated with independent Gaussian noise in eight dimensions make ten-dimensional generating latent space, where only the first two variables are informative. The noise scaled by 0.01 to be small in comparison to the informative dimensions. The latent space samples are then passed through a RealNVP network
'These values are the same in an artificial data experiment in Khemakhem et al. (2119). The difference is that the data will he projected into digee--dimensional space and that we will not assume that the estimating latent space is two-dimensional, this will instcad be inferred by the model.
5

---

Published as a conference paper at ICLR 2020
with 8 fully connected coupling blocks with randomly initialized weights to produce the observed data. This acts as highly nonlinear mixing which can only be successfully treated with nonlinear methods.
GIN is used as the estimating model, with fully connected coupling blocks (full details in Appendix D Training converges quickly and stably using the Adam optimizer (Kingma & Ba 2014) with
initial learning rate 10-2 and other values set to the usual recommendations Batch size is 1,000 and the data is augmented with Gaussian noise (a 0.01) each iteration. After convergence of the loss, the learning rate is reduced by factor of 10 and trained again until convergence.
Over number of experiments made the following observations:
The model converges stably and gives importance (quantified by standard deviation) to only two variables in the estimated latent space, provided there is sufficient overlap between the mixture components in the generating latent space.
Where there is not enough overlap in the generating latent space, the model cannot rec- ognize common variables across all the different classes, and tends split one genuine dimension of variation into two or more in its estimated latent space. This appears to be a problem of finite data. We have observed that when this behaviour occurs, and if the gap between mixture components is not too large, it can be prevented by increasing the number of samples so that the space between the mixture components better filled (see Fig. 6 and 7 in Appendix E). This is consistent with the theory, where equation (4) is true asymptoti- cally. Since the latent space distributions are members of the exponential family, they have support across the entire domain of the latent space, hence gaps can never remain in the limit of infinite samples.
Choice of learning rate is important. If the initial learning rate is too low, training gets stuck in bad local optima, where single true variables are split into several latent dimensions.
4.3.2 EXPERIMENT CONDITIONS OF THEORY NOT FULFILLED
Samples are generated as in Experiment 1, but with only three mixture components. Since there are two sufficient statistics per dimension, and two dimensions of variation, according to the theory we need at least nk + 1 = 5 distinct conditions u for equation 4 to hold (see section 3.1). Therefore, we might not expect successful experiments. Nonetheless, we observe essentially the same results as for the previous experiments (see Fig. in Appendix E), with the same caveats regarding gaps between the mixture components in the generating latent space. This suggests that the conditions derived in Khemakhem al. (2019), although sufficient for disentanglement, not necessary.
4.4 EMNIST
4.4.1
EXPERIMENT
SPECTRUM ON EMNIST
SPECTRUM ON EMNIST (DETAIL)
PCA SPECTRUM ON MNIST
10
GLOBAL
30-1
LOCAL
NOISE
6'
33'1
10
10
2x30
32n4
800
10
200
600
Figure 3: Spectrum of sorted standard deviations derived from training GIN on EMNIST. On the right is the equivalent spectrum from PCA, a linear method, on MNIST. The nonlinear spectrum exhibits sharp knee not obtained by linear methods. In the nonlinear spectrum, the first 22 latent variables encode information about the shape of a digit, while the rest of the latent variables encode noise. This distinction is marked with dotted line in the left and center figures (see Sec. 4.4.2 for explanation of the choice of this cut-off). Within the first 22 variables, the first eight encode global information, such as slant and width, whereas the following 14 encode more local information. This distinction is marked in the center figure only.

---

Published as a conference paper at ICLR 2020
The data comes from the EMNIST Digits training set of 240,000 images of handwritten digits with labels (Cohen et al. 2017). EMNIST is a larger version of the well-known MNIST dataset, and also includes handwritten letters. Here use only the digits. The digit label is used as the condition u, hence we construct a Gaussian mixture in the estimated latent space with 10 mixture components. According the theory (see 3.T). these are only enough conditions to guarantee identifiability if there are only four informative latent variables in the generating latent space. We expect that the true number of informative variables is somewhat higher, so as in Experiment above, we are operating outside of the guarantees of the theory. In addition, the true generative process of human handwriting may not exactly fulfill our method's s assumptions, and we might still lack data on rare or subtle variations, despite the large size of the dataset in comparison to MNIST.
The estimating model a GIN which uses convolutional coupling blocks and fully connected cou- pling blocks to transform the data to the latent space (full details in Appendix D Optimization is with the Adam optimizer, with initial learning rate 3e-4. Batch size is 240 and the data augmented with Gaussian noise (o 0.01) at each iteration. The model is trained for 45 epochs, then for further 50 epochs with the learning rate reduced by a factor of 10.
4.4.2
RESULTS
0
0
00000
9
00000
0
0
1
1
1
1
1
6
1
1
/
!
2 2
2
2
3
2 2
2
2 2
2
2
2
2
2 2
3
2
2
2
2
2
2
4
3
3
3
3
3 3
3
3
3
3
3
3 3
3
3
3
3
3
3
3
3
(a) Variable 1: upper width (b) Variable 8: lower width
(c) Variable 3: height
(d) Variable 4: bend
Figure 4: Selection of global latent variables found by GIN. Each row is conditioned on a different digit label. The variable runs from -2 to +2 standard deviations across the columns, with all other variables held constant at their mean value. The rightmost column shows a heatmap of the image areas most affected by the variable, computed as the absolute pixel difference between -1 and +1 standard deviations. Variable controls the width of the top half of digit, whereas variable 8 controls the width of the bottom half. Width in both cases is somewhat entangled with slant, Variable 3 controls the height and variable 4 controls how bent the digit is. Full set of variables for all digits in Appendix F
000
00000
00000
0
0
00000
1
1
/
1
1
/
1
1
/
1
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2 2
2
2 2 3
3
3
3
3 3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
(a) Variable 11
(b) Variable 12
Variable 13
(d) Variable 16
Figure 5: Selection of local latent variables found by GIN. Refer to Figure 4 above for explanation of the layout. Variable 11 controls the shape of curved lines in the middle right. Digits without such lines are not affected. Variable 12 controls extension towards the upper right. Variable 13 modifies the top left of 3 and 7 only (7 not shown here) and variable 16 modifies only the lower right stroke of 2. Full set of variables for all digits in Appendix F
The model encodes information into 22 broadly interpretable latent space variables. The eight vari- ables with the highest standard deviation encode global information, such as slant, line thickness and width which affect all digits more or less equally. The remaining 14 meaningful variables encode more local information, which does not affect all digits equally. We interpret all other variables as encoding noise. This is motivated mainly by our observation of an abrupt end to any observable change in the reconstructed digits when any variable after the first 22 is changed. This can be seen clearly in Fig. [5] in Appendix F Some variables, particularly the global ones, are not entirely dis- entangled from another. Nevertheless, the results are compelling and suggest that the assumptions required by the theory are approximately met.
7

---

Published as a conference paper at ICLR 2020
We observed some global features which not usually seen in disentanglement experiments MNIST (e.g. Chen et al. 2016; Dupont 2018). These experiments usually obtain digit slant, width and line thickness as the major global independent variables. We too observe slant and line thick- ness as independent variables (variables 2 and 5), but find width to be split into two variables, one governing the width of the upper half of a digit (variable 1) and the other the lower half (variable 8). This makes sense, since these can in fact vary independently. We also observe the height of a digit (variable 3) (see Fig. 4 This does not usually appear in disentanglement experiments, possibly be- cause it is too subtle variation for those experiments, but possibly because it is not present or not discoverable in the smaller MNIST dataset but is in EMNIST.
Local features are also usually not observed in such experiments, so the variables which control these are particularly interesting. These variables modify only a region of the digit, leaving the rest of the digit untouched. In addition, digits which do not have the feature which is being modified in that region are left alone. Examples include variable 13, which changes the orientation of the top-left stroke in 2, 3 and 7, and variable 16, which modifies only the lower-right stroke of 2 (sce Fig.
The full set of local and global variables can be seen in Fig. 10 to Fig. 15 in Appendix
4.4.3 NOTE ON THE CHOICE OF CONDITIONING VARTABLE
The concept of "true generative process" rests on the assumption that conditioning variables u act as causes for the latent variables 7, such that the elements of the latter become independent given the former. In the case of handwritten digits, represents a person's decision to write a particular digit, and z determines the hand motions to produce this digit. Modelling u in terms of digit labels is plausible proxy for the unknown causal variables in the human brain, and the success the resulting model suggests that the assumptions approximately fulfilled. Identifying promising conditioning variables u in less obvious situations is an important open problem for future work.
4.4.4 POTENTIAL IMPROVEMENTS
Increasing the number of conditioning variables would bring this work closer in line with the existing theory. At the current value of ten, the theory only applies if there are at most four informative generating latent variables, which is almost certainly too low a number. One option to increase the number of conditions is to relax the labels from hard to soft, i.c. compute posterior probabilities of class membership given a data example. This could be achieved by information distillation as in Hinton al. (2015). Another possibility is to split existing labels into sub-labels, for example making a distinction between sevens with and without crossbars. This may also aid the generation of more realistic samples, since we observed no sevens with crossbars in our generated samples, even though they make up approximately 5% of the sevens in the dataset.
5 CONCLUSION AND OUTLOOK
We have expanded the theory of nonlinear ICA to cover problems with unknown intrinsic dimen- sion and demonstrated an implementation with GIN, new preservie modification of the RealNVP invertible network architecture. The variables discovered by GIN in EMNIST are inter- pretable and detailed enough to suggest that the assumptions made about the generating process are approximately true. Furthermore, our experiments with EMNIST demonstrate the viability of apply- ing models inspired by the new theory of nonlinear ICA to real-world data, even when not all of the conditions of the theory are met.
is not clear if the methods from this work will scale to larger problems. However, given the recent advances of similar flow-based generative models in density estimation larger datasets such as CelebA and ImageNet (c.g. Kingma & Dhariwal 2018 Ho et al./ 2201). it is plausible prospect. In addition, it is not clear whether the method can successfully be extended to the context of semi- supervised learning, or ultimately, unsupervised learning.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Lynton Ardizzone, Carsten Lüth, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Guided image generation with conditional invertible neural networks. arXiv:1907.02392 2019.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp- 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp- 2172-2180, 2016.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: an extension of MNIST to handwritten letters. arXiv: (rXX:::702..5373: 2017.
Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314, 1994.
Nat Dilokthanakul, Pedro AM Mediano, Marta Gamelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. arXiv preprint arXiv: 1.02648, 2016.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv: 140.85/6 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. 100.v:1605.08803, 2016.
Emilien Dupont. Learning disentangled joint continuous and discrete representations. In Advances in Neural Information Processing Systems, pp. 710-720, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. ICLR, 2(5):6, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. 15X3:01503.0231,, 2015.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improv- ing flow-based generative models with variational dequantization and architecture design. arXiv: 190..005900.07 2019.
Aapo Hyvärinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. In Advances in Neural Information Processing Systems, pp. 3765-3773, 2016.
Aapo Hyvärinen and Hiroshi Morioka. Nonlinear ICA of temporally dependent stationary sources. In Proceedings of Machine Learning Research, 2017.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12(3):429-433 1999.
Aapo Hvvärinen, Hiroaki Sasaki, and Richard E Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. arXiv:1805.0866 2018
Jörn-Henrik Jacobsen, Arnold Smeulders, and Edouard Oyallon. i-RevNet: Deep invertible net- works. arXiv:1802.07088. 2018.
Matthew Johnson, David K Duvenaud, Alex Wiltschko. Ryan Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured representations and fast infer- ence. In Advances in neural information processing systems, pp. 2946-2954, 2016.

---

Published as a conference paper at ICLR 2020
llys Khemakhem, Diederik P Kingma, and Aapo Hyvärinen. Variational autoencoders and nonlin- ear ICA: A unifying framework. arXiv: 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiiv: 14, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. UrXiv:/8/1.123355 2018.
Qingyu Zhao, Nicolas Honnorat, Ehsan Adeli, Adolf Pfefferbaum, Edith Sullivan, and Kilian M Pohl. Variational autoencoder with truncated mixture of gaussians for functional connectivity analysis. In International Conference on Information Processing Medical Imaging, pp. 867- 879. Springer, 2019.
10

---

Published as a conference paper at ICLR 2020
SUPPLEMENTARY MATERIAL
A PROOF OF IDENTIHIABILITY
This section reproduces proof from Khemakhem al. (2019) with some modifications to adapt to the context of invertible neural networks (normalizing flows).
Define the domain of fe Z 21 (Ti,(z),...... ((,,))))..... and the vector of their coefficients A(u) ()I,( ....,,n,,x))))).)..
Zn. Define the vector of sufficient statistics T(z)
Theorem Assume we observe data distributed according to the generative model defined in Sec. Further suppose the following:
3.1
(i) The sufficient statistics Tı are difserentiable almost everywhere and their derivatives dT, (2)//d/dz are nonzero almost surely for all E 2, and all S and S S k.
(ii) There exist nk distinct conditions uo...., unk such that the matrix
L == (A(ui) A(uo). A(uns) -X(uo))
(8)
of size nk nk is invertible.
Then the sufficient statistics of the generating latent space are related to those of the estimated latent spuce by the following relationship:
T(z) AT (w) (uv++ +
where A is constant, full-rank nk x dk matrix and e E Ruk a constant vector.
Proof: The conditional probabilities PT.A.f.s (x(uu and pT X (x(u) are assumed to be the same in the limit of infinite data. By expanding these expressions via the change of variables formula and taking the logarithm we find
Po PT. (z/u) + logp(e) + log |ct Jy 1 [x)] = log Pr Prr.xx//uu + log det Ja (x)
(10)
where J is the Jacobian matrix. Let uo.... Un be the conditions from iii above. We can subtract this expression for Ug from the expression for some condition u,. The Jacobian terms and the term involving will vanish, since they do not depend on
logpz(z/u: log log log pw (w]u: log (w/uu))
(11)
Since the conditional probabilities of z and are exponential family members, we can write this expression
(uo) (u:
log
Σ T Ty (Z)(( (u.()
1=1
Z((uo) log Z (u:) + (T,(
(uoll)
(12)
where the base measures Q; have cancelled out, since they do not depend on u. Defining A(u) = A(u) A(uo) and writing the above in terms of inner products, find
Σ log Zi(un) (T' (w), + Σ log (u(
7((u))
(T()) X) +
(13)
Zi(u)
for 1 S < nk. Combining the nk expressions into single matrix equation we can write this in terms of L from condition (ii) above and an analogously defined L' d
LT((2) (w))
(14)
where b; Σ, log Z( Since LT invertible, we can multiply this expression by its inverse from the left to get
7.(oo)2((u))
T(z) AT' (w) +
(15)
11

---

Published as a conference paper at ICLR 2020
where A = L and c L-Tb. It now remains to show that A has full rank.
According to a lemma from Khemakhem et al. (2019). there exist & distinct values 2 to of such that
dz, (z) ).................. di, (2/) are linearly independent in R for all 1 S S n. Define is vectors
de,
= -2) (,2))...,21,, =n from the points given by this lemma. Take the derivative of equation 15 evaluated at each of these & vectors and concatentate the resulting Jacobians as Q [Jr(z') (r.').....,. JT (z*)]- Each Jacobian has size nk x n, hence Q has size nk x nk and is invertible by the lemma and the fact that each component of TT is univariate. We can construct corresponding matrix Q' made up
of the Jacobians of (g o fe (z) evaluated at the same points and write
Q = AQ'.
(16)
Since Q is full rank, both and Q' must also be full rank.
A.1 REMARK REGARDING CONDITION III
If the coefficients of the sufficient statistics À are generated randomly and independently, then con- dition (ii)ss almost surely fulfilled. In this case, we can ignore the dependence on to consider the Xuu)) as independent random variables. Then condition states that instances of these random variables are in general position ignk which is true almost surely.
A.2 REMARK REGARDING NOISE VARIABLES THE ESTIMATED LATENT SPACE
Any variables in the estimated latent space whose distribution does not depend on the conditioning variable u is considered as encoding only noise. Such variables will be cancelled out in equation 12) and the corresponding column of L will contain only zeros. This means the corresponding column of A will also contain only zeros, so any variation in such noise variable has no effect on The reverse is also true: if column of A contains only zeros, the corresponding variable in the estimated latent space does not depend on u and must encode only noise.
B SPARSITY IN UNMIXING MATRIX: GAUSSIAN DISTRIBUTION
Suppose samples z from the generating latent space follow a conditional Gaussian distribution. Sup- pose the estimating model y faithfully reproduces the observed conditional density p(x(u) and sam- ples w from its latent space also follow a conditional Gaussian distribution. Then we can apply equation (4) to relate the generating and estimating latent spaces.
The sufficient statistics of a normal distribution with free mean and variance are a and 2 Hence the relationship between the latent spaces becomes
z22
A
w
(17)
where the squaring is applied element-wise. We can write A in block matrix form as
A (1)
A
(4)
(18)
and
((1))
(19)
(22)
Then:
Z A
A (2) ())
(20)
7 (3)
(3) W
A (1) (2)
(21)
so we can write for each dimension of
2,
(1))
w Σ
A w3 +
(22)
1
27 = Σ
A wj + Σ
A
11
13
(23)
3
12

---

Published as a conference paper at ICLR 2020
In order to compare the equations, need to square (22 To do so, will have square the second term on the right hand side, involving wy. There is no matching term in (33) so we have to set all entries of A to zero. In more detail:
Σ S
ΣE
1
13
(2)
(24)
Σ
4
(25)
The first term with w/ matches term in (3) so have to set A for all i and j. This simplifies the earlier equation:
Σ A A uy + (1)
(26)
The square of the first term on the right hand side involves terms with wy cross terms:
)
Σ 44g
-
= Σ 4 + Σ (1) Alg wywy 343
(27)
(1) so we have to set A.j A 0 for all j + j'. This means that the i-th row of ( can have at most one nonzero entry. It must also have at least one nonzero entry, since if the row were all zero, a row of A would be all zero (since A (2) = 0). but A has full rank. Since there are as many or fewer rows than columns (n S d), each row of A is linearly independent, so is not possible for one to be zero. Hence each row of (1)) has exactly one nonzero entry. Moreover, two of A have their nonzero entries in the same column. If they did, the two rows would not be linearly independent, but they must be since has full rank. Therefore we can write
2 a, + bç
(28)
where 4 and b, = That is, the generating latent variable 2; is linearly related to some latent variable of the estimating model wy. This estimated latent variable is uniquely associated with 2 and any estimated latent variables not associated with a generating latent variable (in the case >> n) encode no information about the generating latent space. So the model has decoded the original latent variables z up affine transformation and permutation as subset of variables in its estimated latent space and has encoded no information (only noise) into the remaining latent variables.
C SPARSITY IN UNMIXING MATRIX: TWO-PARAMETER EXPONENTIAL FAMILY MEMBERS
The results of Appendix B can be extended to other members of the exponential family with parameters. In the general case, writing T,1 = Ti and Tı Tl for all i, equations 22 and (2) become
Ti(2;) = Σ E (T,) + Σ 1 T2 + + (1)
(29)
Ta(zi)= = Σ
A (3) Ti (w;) + Σ A T2(33) + + (2)
(30)
which, with the definition t t= T22 o T
becomes
[2,), Σ
A (w;,) Σ 1 (2) ((T((((w,)) +
(31)
(T((w)) = Σ 1 (w,)) + Σ (((((T( + +
(32)
13

---

Published as a conference paper at ICLR 2020
We can combine these equations to get
Σ A T + Σ ( -I4 Ti(s) + Eaf((T((u)) + +ol
(33)
This equation has many summed terms on the right. Suppose that & has a convergent Taylor expan- sion in some region of its domain. We can take this expansion the term on the left and compare coefficients. Since t cannot be linear, there will be terms in the expansion of order two or higher. As in the Gaussian case, these polynomial terms create cross terms which are impossible to reconcile with those the right hand side of the equation. The only consistent solutions can be found by setting all coefficients except those of functions of wy (for some j) to zero:
((T((((())))) 4') = A ((w)) (w;) c
(34)
We can further simplify this equation by examining higher-order terms, but need to know the form of to do so. In any case, we can now write equation (29
T2,2,) = AT
((TT(()))))))) + +
(35)
showing that each generating latent variable z; is related to exactly one estimated latent variable Wy. As in the Gaussian case, we use the full rank property of A to that each estimated latent variable is associated with at most one generating latent variable, and any estimated latent variables not associated with any generating latent variables must encode only noise. The task now is to check the form of t for each wwo-aaramtter member of the exponential family, to see what further constraints we can derive from equation (34). The results are stated Table
Table I: Two-parameter exponential family members and selected properties.
Distribution
Sufficient Statistics
t(xx
Latent Space Relationship
Normal
(z, (log 2, (log 2)2)
x²
amy
Lognormal
x²
log 2; log W awly or = a/ W
Inverse Gaussian
(z, 1/2)
1/2
Gamma
(log 2, 2)
exp(x)
2; aw)j
Inverse Gamma
(log 2, 1/2)
exp(
4 = aw,
log 2 log Wij
Beta
(log log(1 - 2))
log(1 - exp(x:)
or log 2 log [ w.)
14

---

Published as a conference paper at ICLR 2020
D NETWORK ARCHITECTURE
The estimating model g is built in the reverse direction for practical purposes, so the models de- scribed here are 9 which maps from the data space the latent space. The type and number of coupling blocks for the different experiments are shown below. The affine coupling function is the concatentation the scale function and the translation function t, computed together for ficiency, in Kingmaa & Dhariwal (2018). It is implemented as either a fully connected network (MLP) or convolutional network, with the specified layer widths and ReLU activation after all but the final layers. For the convolutional coupling blocks, the splits are along the channel dimension. The scale function is passed through a clamping function ttah(s)), which limits the output the range (-2, 2), in Ardizzone et al. (2019). Two affine coupling functions
applied per block, as described in Dinh et al. (2016). Downsampling increases the number of channels by a factor and decreases the image width and height by factor of 2, done in checkerboard-like manner, as described in Jacobsen et al. (2018). The dimensions are permuted in a random but fixed way before application of each fully connected coupling block and likewise the channels for the convolutional coupling blocks. The network for the artificial data experiments has 4,480 learnable parameters and the network for the EMNIST experiments has 2,620, 192 learnable parameters.
Table 2: Network architecture for artificial data experiments
Type of block
Number
Input shape
Affine coupling function layer widths
Fully Connected Coupling
10
10
10 10
Table Network architecture for EMNIST experiments
Type of block
Number
Input shape
Affine coupling function layer widths
Downsampling
(1, 28, 28)
Convolutional Coupling
4
(4, 14, 14)
2 2
16 -
16 +
Downsampling
(4, 14, 14)
Convolutional Coupling
(16, 7, 7)
8 32 + 32 + 16
Flattening
(16, 7,
Fully Connected Coupling
784
392
392
392
784
D.1 NOTE OPTIMIZATION METHOD
In the experiments described in this paper, the mean and variance of mixture component was updated at each iteration as the mean and variance of the transformations to latent space of data points belonging to that mixture component in minibatch of data D:
(u(u'; 0) + Epa-w (9, (x: 0)) of (u': < Var (9, (x: 0)).
(36)
(37)
Hence the parameters of the mixture components would change in each batch, according to the data present. The notation (u: 0) does not indicate that j directly parameterized by and learned, instead it indicates that y is function of g which is parameterized by 0. A change in will also change the output μ., given the same mini-batch of data D. The same holds for o, 0).
We expect that specifying the means and variances as learnable parameters updated in tandem with the model weights would have worked equally well.
15

---

Published as a conference paper at ICLR 2020
E
FIGURES FROM THE ARTIFICIAL DATA EXPERIMENTS
OBSERVED DATA (PROJECTION)
GROUND TRUTH
RECONSTRUCTION
SPECTRUM
Figure 6: Experiment 1: Five mixture components and 100,000 data points. GIN successfully constructs the generating latent space and gives importance to only two its ten latent variables, reflecting the two-dimensional nature of the generating latent space.
OBSERVED DATA (PROJECTION)
GROUND TRUTH
RECONSTRUCTION
SPECTRUM
Figure 7: Same experiment as in Figure 6 with the number of data points reduced to 10,000. GIN fails to successfully estimate the ground truth latent variables, due limited data: The first variable (r-axis) is well approximated in the reconstruction, but the second variable (y-axis) is split into two the reconstruction, one capturing mainly information about the lower two clusters (shown here) and another information about the other mixture components (not shown). We also observe a less clear spectrum, where three variables are given more importance than the rest, not faithfully reflecting the two-dimensional nature of the generating latent space.
OBSERVED DATA (PROJECTION)
GROUND TRUTH
RECONSTRUCTION
SPECTRUM
Figure 8: Experiment 2: Only three mixture components (not sufficient for identifiability according the theory). Nevertheless, GIN successfully reconstructs the ground truth latent variables. This suggests that the current theory of nonlinear ICA relies on sufficient, but not necessary, conditions for identifiability.
16

---

Published as a conference paper at ICLR 2020
F
EMNIST FIGURES
5 0 7
8 8 9
0
2
3
4
5
6
7
8
9
012345678
O
2
3
6
7
8
9
0128966789
0
1
2
3
4
5
7
8 9
0133456789
0
1
2
3
4
845678
9
0123456789
O 1
O 0/23/56789
0123456789
0135466769
0/23956789
012346778
0123456789
0123456789
0123456789
0123456785
0 0/23456789
0123456789
0123456787
0123456789
0129456796
0
0123466789
0 012345678
0/23
0023456780
0/23456789
0
1
2
5 6
3
4
8
0623466789
0
1
2
5 7
3
9
0/23456789
0
1
2
3
6
4 6
8 9
0123956789
0
6
2
3
9
0
3 1 9
3
5 6
7
8
9
(a) Full Temperature
(b) Reduced Temperature (T 0.8)
Figure 9: Full and reduced temperature samples from the model trained EMNIST. Reduced tem- perature samples are made by sampling from a Gaussian distribution where the standard deviation reduced by the temperature factor. The 22 most significant variables sampled, with the others kept to their mean value. This eliminates noise from the images but preserves the full variability of digit shapes. Each has the same latent code (whitened value) but conditioned different class in each column, hence the style of the digits is consistent across rows.
17

---

Published as a conference paper at ICLR 2020
0
0
00000000
0
000000000
1
1
1
1111111
1 11111111
#
2
22222222
2
2
3
3
3
3
3
3 3
3
3
3 3333333
4
4
4
444
4
4
4
4
4
5
5
5
55
5
5
6
6
6
6
6
6
6
6
6
1
6
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7 7
7
7
7
7
7
7
7
7
7 7
8
8
8
8
8
8
8
8
8 9
8
8
8
8
8
8
8
8
8
2
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9 9
(a) Variable width of top half
(b) Variable 2: slant/angle
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0 /
1
1
1
1
1
1
/
1
1
1
1
1
1
/
2
22222
2
2
2
2 3 4 5 6
2 3 4 5 6 7
2
2
2
2
2
2
2
2
2 3 4
2 3
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
1/
4 4
4
4
4
4
4
4
555
5
5
5
5
5
5 5
5
5
5
5
5
6 6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7 8
7
7
7
7
7
7
7
7
7
7
7
8
8
8
8
8
8
8
8 9
8
8
8
8
8
8
8
8
8 9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9 9
(c) Variable height
(d) Variable through center
Figure 10: Most significant latent variables 1 to 4. Each row is conditioned on different digit label. The variable runs from -2 to +2 standard deviations across the columns, with all other variables held constant at their mean value. The rightmost column shows a heatmap of the image areas most affected the variable, computed the absolute pixel difference between -1 and +1 standard deviations.
18

---

Published as a conference paper at ICLR 2020
000000000 0
000000000
000000
0
1
1
1
1
1
1
1 1
11111 1
1
2
22
2
2 2
3
3
3
3
3
3
3
3
3
3
3 3
3
3 3
3
3 3
3
3
4
4
4
4
4
4
4
4
4
4
4 4
4 4
4
55
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
666
6
6
6
6
6
6
7
7
7
7
7
7
7
7
7
7
7777
7
7
7
7
7
8
8
8
8
8
8
8
8
8
8
8 8
8
8
8
8
8
8
8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
(a) Variable line thickness
(b) Variable 6: slant of vertical bar in 4,7 and
0
0
0
0
0
0
0
0
0000
0
0
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2 3 4
2 3 4 5 6
2
2
22222
2
2
2
2 3 4 5 6
2
2
3
3
3
3
3
3
3
3333
3
3
3
3
3 4
3
4
4
1/444
4
4
4
4
444
4
4
4
4
4/
5
5
5
5
5
5
555
5
5
5
5 6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7
7 8
7
7
7
7
7
7
7
7 8
7
7
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
0
9
9
9
9
9
9 9 9
9
9
9
9
9
9
9
9
9
(c) Variable height horizontal bar
(d) Variable width bottom half
Figure 11: Most significant latent variables 5 to 8. Each row is conditioned on different digit label. The variable runs from -2 to +2 standard deviations across the columns, with all other variables held constant at their mean value. The rightmost column shows a heatmap of the image areas most affected by the variable, computed the absolute pixel difference between -1 and +1 standard deviations.
19

---

Published as a conference paper at ICLR 2020
00000 000000
0
0
0
0
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2 3
2
3
3
3
3
3
3
3
3
3 4
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
5 5
5
5
5
5 5
5
55
5
5
5
55
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7 7
7
7
7
7
7
7
7
7
7 7
7
7
7
7
8 8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
6
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
(a) Variable 9: extension of center left feature towards the left
(b) Variable 10: openness of lower loop
0
0
0
2
0
0000000
0
0
0
0
5
1
1
1
1
1
1
1
1
1
1
1
/
2
2
2
2
2
2
2
2
2 3 4
2
2
2
2
2
2
2
2
2 3
2
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3 4
3
4 4
4
4
4
4
4
4
4
4
4
4
4
4
4
55555
5
5
5
5
5
5
5
5 5555
6 666
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6 6
7
7
7
7
7
7
7
7
7 8
7
7
7 7 7 7
7
7 7
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8 8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
(e) Variable 11: shape of center right feature
(d) Variable 12: top right corner
Figure 12: Most significant latent variables to 12. Each row conditioned different digit label. The variable runs from to +2 standard deviations across the columns, with all other variables held constant their mean value. The rightmost column shows a heatmap of the image areas most affected by the variable, computed as the absolute pixel difference between -1 and +1 standard deviations.
20

---

Published as a conference paper at ICLR 2020
0000000
000000000
000
0
000
1
1
1
1
1
1
1
1
1
1
1
2
222222
2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3 4
3
3
3
3
3
3
3
3
3 4
3
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
(a) Variable 13: orientation of stroke top left for and
(b) Variable 14: shape of top part of bottom loop
0
0
0
0
0
0
0
0
00000
100000000
1
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
22 3
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3 4
3
3
3
3
3
3
3
3
3 4 5 6
3 4 5 6 7 8 9
4
4
4
4
4
4
4
4
4 4
4
4
4
4
4
5
5
155555
5
5
5
5 5 6 7 8
5 6
5
5 5
555555
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7 8
7
7
7
7
7
7
7
7 8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
9
9
9
9 9 9
9
9
9
9
9
9
9
9
(c) Variable 15: angle centrally located horizontal features
(d) Variable bottom right stroke 2
Figure 13: Most significant latent variables 13 to 16. Each row is conditioned on a different digit label. The variable runs from -2 +2 standard deviations across the columns, with all other variables held constant at their mean value. The rightmost column shows a heatmap of the image areas most affected by the variable, computed as the absolute pixel difference between -1 and +1 standard deviations.
21

---

Published as a conference paper at ICLR 2020
000000000
0
000000000
1 1 1
1111111
11
1
1 1
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3 3
4 4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
5
5
5
5555
55
5
5
5
5
6 6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
0
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
(a) Variable top left stroke of
(b) Variable top stroke 5
0
0
00000000
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2 3
2 3
2
2
2
2
2
2
2
2
2
2 3 4
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
55
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
8
8
8
8
8
8
8
8
8 9
8
8
8
8
8
8
8
8
8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9 9
9
(c) Variable extension towards top right
(d) Variable 20: curvature vertical stroke
Figure 14: Most significant latent variables 17 to 20. Each row is conditioned on a different digit label. The variable runs from -2 to +2 standard deviations across the columns, with all other variables held constant at their mean value. The rightmost column shows a heatmap of the image areas most affected by the variable, computed the absolute pixel difference between -1 and +1 standard deviations.
22

---

Published as a conference paper at ICLR 2020
000000000
0
000000000
1
1
1
1
1
1 1
1
22222221
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3 3
4
4
4
4
4
4
4
4
0/
4 4
4
4
4
4 4
5
5
5
55
5
6
6
6
6
6
6
6
6
6
&
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
(a) Variable 21: thickness of upper loop
(b) Variable 22: extension towards top left
0
0
0
0
1000000
0
00000
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
22
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
5
5
15555
5
5
5
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6 7 8 9
7
7
7
7
7
7
7
7 8
7 8 9
7
7
7
7
7
7
7
7 8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
(c) Variable 23: no effect
(d) Variable 24: no effect
Figure 15: Most significant latent variables 21 to 24. Each row is conditioned on a different digit
label. The variable runs from -2 to +2 standard deviations across the columns, with all other variables held constant at their mean value. The rightmost column shows a heatmap of the image areas most affected by the variable, computed the absolute pixel difference between -1 and +1 standard deviations.
23


Reviews:
[{"review_id": "rylG-P22FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds upon the recent theoretical framework on nonlinear ICA, put forward in recent work Khemakhem et al. (2019) that draw a lot of attention. The latter work provides an extension of the basic nonlinear ICA that is closely related to a VAE with a conditional factorized prior, essentially introducing side information (with assumed extra observables u) to resolve non-identifiability issues. \n\nThis paper proposes an invertible architecture related to RealNVP and NICE, coined as  GIN: the General Incompressible-flow network. \n\nOn the positive side,\n\nA key feature of the proposed methodology is model selection (such as determining the model order) that is in general a hard problem even in linear latent variable models. \nThe performance is illustrated on the EMNIST dataset, as well as carefully constructed synthetic experiments. The semantic descriptions of each captured dimension, as detailed in the appendix, is particularly interesting. \nThe experimental section is quite extensive. \n\nOn the negative side, \n\nThe paper is largely based on the results of a recent technical report (Khemakhem et al. (2019)) and is not self contained, hence rather hard to digest. Even the proofs in the appendix and the intuition requires the reading of this longer technical report. \n\nIt is also not at all clear where the side information (variables u) is coming from. On EMNIST a natural candidate is the digit labels (and this turns out to be is indeed the case in the experimental section) but it is not very clear what conditions need to be satisfied. What prevents us selecting simply a subset of observations x as u? \n\nLacking an explicit motivation, the exercise of writing the canonical parameters of a multivariate Gaussian in (2) - (4) is not very informative. This needs to be better motivated.\n\nThe prior structure resembles quite closely the general hierarchical priors SVAE proposed in https://arxiv.org/abs/1603.06277. It would be also informative to discuss the links with this approach.\n\nArguably, the key contribution of the current paper is in 3.1. and 3.2, but these are rather hastily written and quite short. Overall, the balance of known results and new contributions \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlju2tx9r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends recent work by Khemakhem et al on nonlinear ICA to allow for unknown number of generative factors. This is tackling an important problem in the field of generative modeling, where one would like to extract the generative factors of a dataset that independently control its features (i.e. disentanglement).\n\nThe paper is very clearly written, does a great job at motivating the problem and presenting the recent results from Khemakhem et al, before extending them with some simple theorems and demonstrating their application on toy data + EMNIST.\nI think that this research direction is extremely promising, and obtaining a theoretical understanding of when/why disentangling could work would be particularly valuable to the field, and I would lean towards acceptance so that this work gets more attention.\n\nIt is slightly surprising however that their empirical results seem to indicate that these theoretical conditions do not seem to be necessary, which should be investigated further (and might help illustrate the dichotomy in some claims and results obtained in the disentangling literature recently).\n\n1.\tIt was unclear to me how one should/would decide what to use for $u$ or what to leave out to be factorized by the method. \n\tThis may be out of scope for this current work, but one way to answer that would be to leverage datasets with more fully labelled factorised data (e.g. dSprites [1]) and present how one should leverage these with $u$, in order to identify the original generative factors.\n2.\tRelated, using EMNIST was interesting, but given the lack of “accepted” generative factors to be recovered, it is hard to tell if the 22 variables found are “correct” or more similar to using a generative model which would entangled the data more (and hence would falsely introduce extra latent variables).\n3.\tThe toy dataset, with its random RealNVP network to produce the data, was less “mixed” than I expected, looking at Figure 2B. In its projected view, the clusters are still rather easy to identify by eye, which surprised me somehow? Could you comment a bit more on how difficult is the task, or present a VAE baseline that would fail to explain that data?\n4.\tI did not see how the threshold for selecting 22 latent variables on EMNIST was set? Was the 23rd latent variable significantly less informative? The spectrum on Figure 3A was not precise enough to assess this fairly and the single mention of “measured by the standard deviations of test data transformed to the latent space” was too vague to reverse-engineer the decision.\n5.\tIt was interesting to read about the observations of when this method should fail. It would be interesting if a dataset with explicit “gaps” would be constructed to analyze this case.\n6.\tFigure 4 might benefit from mentioning “what” each variable controls for directly in their individual caption / on top of them, instead of having to read this through in the full figure caption.\n7.\tThe current model in the end is modeling the latent state using a mixture of gaussians (although these now have a theoretical connection to the true generative factors). How much does this differ to existing generative models using VAEs with a mixture of gaussian prior [2, 3]?\n\n\nReferences:\n[1] dSprites dataset: https://github.com/deepmind/dsprites-dataset \n[2] https://arxiv.org/abs/1611.02648\n[3] https://arxiv.org/abs/1902.03717\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eAihsatr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Based on a recent work on identifying the joint distribution over observed and latent variables, the paper presents an extension of it, where the dimension of the latent space is not assumed to be known. Moreover, the authors propose a new neural network architecture based on previous propositions (RealNVP and NICE) to perform this identification.\n\nThe paper's presentation is relatively clear, although all the theoretical results are relegated to the appendix. The extension to unknown latent space dimension seems to be quite straightforward, given the recent work that this paper is based on. However, the experimental results performed on EMNIST are quite convincing and the results are interesting. \nOverall, I think this paper would be interesting to the ICLR audience.\n\nIt seems currently that there is a need to choose a dimension to be large enough (i.e., larger than the true dimension of the latent space). I would have appreciated some discussion (and some experiments) if this dimension is chosen too small. \nAlso, when the dimension is chosen large enough, the non-informative learned latent variables are determined by looking at their standard deviations. Do they always have to be small compared to the informative latent variables?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper builds on the recent theoretical work by Khemakhem et al. (2019) to propose a novel flow-based method for performing non-linear ICA. The paper is well written, includes theoretical justifications for the proposed approach and convincing experimental results. Many of the initial minor concerns raised by the reviewers were addressed during the discussion stage, and all of the reviewers agree that this paper is an important contribution to the field and hence should be accepted. Hence, I am happy to recommend the acceptance of this paper as an oral. 
Decision: Accept (Spotlight)