Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
Authors: David A. Klindt
Lukas Schott
Yash Sharma
Ivan Ustyuzhaninov
Wieland Brendel
Matthias Bethge
Dylan Paiton
Venue: ICLR 2021 Oral (2021)
Keywords: disentanglement
independent component analysis
natural scene statistics

Abstract: Disentangling the underlying generative factors from complex data has so far been limited to carefully constructed scenarios. We propose a path towards natural data by first showing that the statistics of natural data provide enough structure to enable disentanglement, both theoretically and empirically. Specifically, we provide evidence that objects in natural movies undergo transitions that are typically small in magnitude with occasional large jumps, which is characteristic of a temporally sparse distribution. To address this finding we provide a novel proof that relies on a sparse prior on temporally adjacent observations to recover the true latent variables up to permutations and sign flips, directly providing a stronger result than previous work. We show that equipping practical estimation methods with our prior often surpasses the current state-of-the-art on several established benchmark datasets without any impractical assumptions, such as knowledge of the number of changing generative factors. Furthermore, we contribute two new benchmarks, Natural Sprites and KITTI Masks, which integrate the measured natural dynamics to enable disentanglement evaluation with more realistic datasets. We leverage these benchmarks to test our theory, demonstrating improved performance. We also identify non-obvious challenges for current methods in scaling to more natural domains. Taken together our work addresses key issues in disentanglement research for moving towards more natural settings. 

Published as a conference paper at ICLR 2021
TOWARDS NONLINEAR DISENTANGLEMENT IN NATURAL DATA WITH TEMPORAL SPARSE CODING
David Klindt"
Lukas Schott"
Yash Sharma"
University of Tübingen klindd gmail.com
University of Tübingen
University of Tübingen
Nawas.cchott@hetggllbb.oggg bethgelab.org
yash.sharma @ bethgelab.org
Ivan Ustyuzhaninov
Wieland Brendel
University of Tübingen
University of Tübingen
ivan.uttyuzhaninov @ @btthgelab.org
dieland.brended @ bethgelab.org
Matthias Bethge
Dylan M Paiton
University of Tübingen
University of Tübingen
metthias.bethheGe beethissbbbbe@bbbbeeggl...o.gg
Jyllnnppiton @ bethgelab.org
ABSTRACT
Disentangling the underlying generative factors from data has so far been limited to carefully constructed scenarios. We propose a path towards natural data by first showing that the statistics of natural data provide enough structure to enable disentanglement, both theoretically and empirically. Specifically, we provide evidence that objects in natural movies undergo transitions that are typically small in magnitude with occasional large jumps, which is characteristic of temporally sparse distribution. Leveraging this finding we provide a novel proof that relies a sparse prior on temporally adjacent observations recover the true latent variables up to permutations and sign flips, providing a stronger result than previous work. We show that equipping practical estimation methods with our prior often surpasses the current -fate-of-the-art on several established benchmark datasets without any impractical assumptions, such as knowledge of the number of changing generative factors. Furthermore, we contribute two new benchmarks, Natural Sprites and KITTI Masks, which integrate the measured natural dynamics to enable disentanglement evaluation with more realistic datasets. We test our theory on these benchmarks and demonstrate improved performance. We also identify non-obvious challenges for current methods in scaling more natural domains. Taken together our work addresses key issues in disentanglement research for moving towards more natural settings.
INTRODUCTION
Natural scene understanding can be achieved by decomposing the signal into its underlying factors of variation. An intuitive approach for this problem assumes that a visual representation of the world can be constructed via a generative process that receives factors as input and produces natural signals as output (Bengio et al. (2033). This analogy is justified by the fact that our world is composed of distinct entities that can vary independently, but with regularity imposed by physics. What makes the approach appealing is that it formalizes representation learning by directly comparing representations to underlying ground- -round-truth states, as opposed to the indirect evaluation of benchmarking against heuristic downstream tasks (e.g. object recognition). However, the core issue with this approach is non-identifuabiliy which means a set of possible solutions may all appear equally valid to the model, while only one identifies the true generative factors.
Our work is motivated by the question of whether the statistics natural data will allow for the formulation of identifiable model. Our core observation that enables to make progress in
* Equal contribution. Code: https: http://////gghhbbcooooe/e///eeee www..b.w..bbb.heeeeeeeeeee ab/s 1 ow_d1 sentang) 1aattang1mment ement

---

Published as a conference paper at ICLR 2021
addressing this question is that generative factorss of natural data have sparse transitions. To estimate these generative factors, we compute statistics on measured transitions of area and position for object masks from large-scale, natural, unstructured videos. Specifically, we extracted over 300,000
object segmentation mask transitions from YouTTube- VOS (Xu et al. 2018 Yang et al. 2019) and
KITTI-MOT'S (Voigtlaender et al., (2099) Geiger et al., [012)] Milan al. 2016) (discussed in detail in
Appendix D). We fit generalized Laplace distributions to the collected data (Eq. 2). which we indicate with orange lines in Fig. [ We see empirically that all marginal distributions of temporal transitions are highly sparse and that there exist complex dependencies between natural factors (e.g. motion typically affects both position and apparent size). In this study, we focus on the sparse marginals, which we believe constitutes an important advance that sets the stage for solving further issues and eventually applying the technology to real-world problems. With this information at hand, we are able to provide a stronger proof for capturing the underlying generative factors of the data up to permutations and sign flips that is not covered by previous work (Hyvärinen and Morioka 2016 2017 Khemakhem et al. 2020a). Thus, we present the first work, to the best of our knowledge, which proposes a theoretically grounded solution that covers the statistics observed in real videos.
Our contributions are: With measurements from unstructured natural video annotations we pro- vide evidence that natural generative factors un- dergo sparse changes across time. We provide proof of identifiability that relies on the ob- served sparse innovations to identify nonlinearly mixed sources up to a permutation and sign- -lips, which we then validate with practical estima- tion methods for empirical comparisons. We leverage the natural scene information to cre- ate novel datasets where the latent transitions between frames follow natural statisties. These datasets provide a benchmark to evaluate how well models can uncover the true latent genera- tive factors in the presence of realistic dynamics. We demonstrate improved disentanglement over previous models on existing datasets and our contributed ones with quantitative metrics from both the disentanglement (Locatello et al. 2018 and the nonlinear ICA community (Hyvärinen and Morioka 2016). We show via numerous visualization techniques that the learned representations for competing models have important differences, even when quantitative metrics suggest that they are performing equally well.
U
0
-2 Á (a 0.52)
-2 y (a - 0.55)
-2 A area (a 0.44)
Figure I: Statistics of Natural Transitions. The histograms show distributions over transitions of segmented object masks from natural videos for horizontal and vertical position as well as object size. The red lines indicate fits of generalized Laplace distributions (Eq. 2 with shape value a. Data shown is for object masks extracted from YouTube videos. See Appendix [G] for 2D marginals and corresponding analysis from the KITTI self-driving car dataset.
2 RELATED WORK- - DISENTANGLEMENN AND NONLINEAR ICA
Disentangled representation learning has its roots in blind source separation (Cardoso 1989 Jutten and Herault 1991) and shares goals with fields such as inverse graphics (Kulkarni et al. 2015
Yildirim et al. 2020 Barron and Malik 2012) and developing models of invariant neural computation
(Hyvärinen and Hoyer 2000 Wiskott and Sejnowskil Sohl-D.Sh--Dikstteee et al.) (010) (see Bengio
[ct al. 2013 for review). A disentangled representation would be valuable for a wide variety of
machine learning applications, including sample efficiency for downstream tasks (Locatello et al.
2018: Gao al. (2019), fairness (Locatello al.] 2019 Creager al. 2019) and interpretability
(Bengio et al. HiggHHingins et al. 2017 Adel et al. 20188). Since there is no agreed upon definition of disentanglement in the literature, we adopt two common measurable criteria: each encoding element represents a single generative factor and ii) the values of generative factors are trivially
decodable from the encoding (Ridgeway and Mozer 2018 Eastwood and Williams) 2018).
Uncovering the underlying factors of variation has been a standing goal in independent com- ponent analysis (ICA) (Comon) 1994, Bell and Sejnowski 1995), which provides an identifiable solution for disentangling data mixed via an invertible linear generator receiving at most one Gaussian factor as input. Recent unsupervised approaches for nonlinear generators have largely been based on Variational Autoencoders (VAEs) (Kingma and Welling 2013) and have assumed that the data is independent and identically distributed (f.Id.) (Locatello et al. 20188), even though nonlinear methods that make this i...d. assumption have been proven to be non-identifiable (Hyvärinen and Pajunen|
2

---

Published as a conference paper at ICLR 2021
1999:
Locatello et al., 2018). Nonetheless, the bottom-up approach of starting with nonlinear
generator that produces well-controlled data has led to considerable achievements in understanding
nonlinear disentanglement in VAEs (Higgins et al. 2017 Burgess al.) 2018 Rolinek et al. [2119]
Chen et al. 2018), consolidating ideas from neural computation and machine learning (Khemakhem
al. 2020a). and seeking a principled definition of disentanglement (Ridgeway 2016 Higgins et al.
2018
Eastwood and Williams 2018).
Recently, Hyvärinen and colleagues (Hyvärinen and Morioka 2 2017; Hyvärinen et al. 2018 showed that a solution to identifiable nonlinear ICA can be found by assuming that generative factors are conditioned on an additional observed variable, such as past states or the time index itself. This contribution was generalized by [Kbemakhem et al.] (2020a) past the nonlinear ICA domain to any consistent parameter estimation method for deep latent-variable models, including the VAE framework. However, the theoretical assumptions underlying this branch of work do not account for the sparse transitions we observe in the statistics of natural scenes, which we discuss in further detail in appendix F.I.I Another branch of work requires some form of supervision to demonstrate disentanglement (Szabó et al., 2017 Shu et al. 2019 Locatello et al. 2020). We select two of the above approaches, that are both different in their formulation and otate-of-ther ahe- in their respective empirical settings, Hyväärnnen and Morioka (0017) and Locatello et al. (020)), for our experiments below. The motivation of our method and dataset contributions address the limitations of previous approaches and to enable unsupervised disentanglement learning in more naturalistic scenarios
The fact that physical processes bind generative factors in temporally adjacent natural video segments has been thoroughly explored for learning in neural networks (Hinton 1990 Földiák 1991 Mitchison
1991. Wiskott and Sejnowski] 2002 Denton and Birodkar 2017). We propose method that uses time information in the form of an Li-parse temporal prior, which is motivated by the natural scene measurements presented above as well as by previous work (Simoncelli and Olshausen Such prior would intuitively allow for sharp changes in some latent factors, while most other factors remain unchanged between adjacent time-points. Almost all similar methods are variants of slow feature analysis (SFA, Wiskott and Sejnowski 2002). which measure slowness in terms of the Euclidean (i.e. L2, or log Gaussian) distance between temporally adjacent encodings. Related to our approach, probabilistic interpretation of SFA has been previously proposed (Turner and Sahani 2007). as well as extensions to variational inference (Grathwohl and Wilson| 2016). Additionally, Hashimoto (2003) suggested that a sparse (Cauchy) slowness prior improves correspondence to biological complex cells over the L2 slowness prior in a two-layer model. However, to the best of our knowledge, L1 temporal prior has previously only been used in deep auto-encoder frameworks when applied to semi-supervised tasks (Mobahi et al., 2009 Zou al. 2012). and was mentioned in [adiieu and Olshausen (2012). who used an L2 prior, but claimed that an L1 prior performed similarly on their task. Similar to Hyvärinen et al. (Hyvärinen and Morioka| 2016, Hyvärinen et al. 2018), we only assume that the latent factors are temporally dependent, thus avoiding assuming knowledge of the number of factors where the two observations differ (Shu et al. 2019 Locatello et al. 2020).
2001: Olshause] 2003; Hyvärnen et al. [2003] Cadieu and D
Most of the standard datasets for disentang /emmmtennnneeeenntt eement (dSprites MMathhey et al. 2017), Cars3D (Reed jet al. 2015), SmallNORB (LeCun et al.) 2004), Shapes3D (Kim and Mnih| 2018), MPI3D (Gondal
et al. 2019g) have been compiled into a disentanglement library (Disl.bb) by Locatello et al. (201 8). However, all of the DisLib datasets are limited in that the data generating process is independent and identically distributed (i.i.d) and all generative faccorrs are assumed to be discrete. In a follow-up study, Locatello et al. 2020) proposed combining pairs of images such that only k factors change, as this matches their modeling assumptions required to prove identifiability. Here, h U(1, D 1) and D denotes the number of ground-truth factors, which are then sampled uniformly. We additionally use the measurements from Fig. | to construct datasets for evaluating disentanglement that have time transitions which directly correspond to natural dynamics.
'ss in slow feature analysis, we consider learning from videos without labels as unsupervised.
3

---

Published as a conference paper at ICLR 2021
3 THEORY
3.1
GENERATIVE MODEL
We have provided evidence to support the hypothesis that generative factors of natural videos have sparse temporal transitions (see Fig. D To model this process, we assume temporally adjacent input pairs Xc) coming from nonlinear generator that maps factors to images (2),, where generative factors are dependent over time:
p(z, 2t-1) = p(z((2-1))((z--)
Assume the observed data (x,, xx-1) comes from the following generative process, where different latent factors are assumed to be independent (ef. Appendix F22
x= gZ),, p(z(-1)
II p(=)1,)), p(z)|z(-1) II (=1
αd 21(1/a) exp - (A|22,, =-1,s1"), (2)
where À is the distribution rate, p(z;-1) ) is factorized Gaussian prior (0. I) (as in Kingma and Welling 2013) and ((1) is a factorized generalized Laplace distribution (Subbotin, 1923)
with shape parameter a, which determines the shape and especially the kurtosis of the function Intuitively, smaller a implies larger kurtosis and sparser temporal transitions of the generative factors (special cases are Gaussian,
2. and Laplacian, a < 2 to ensure that temporal transitions are sparse. The novelty our approach lies in our explicit modeling of sparse transitions that cover the statisties of natural data, which results in a stronger identifiability proof than previously achieved (see Appendix F.I. for a more detailed comparison with Hyvärinen and Morioka 2017 Khemakhem et al. 2020a).
= 1). Critically, for our proof we assume
3.2 IDENTIFIABILTTY PROOF
Theorem For a ground-truth (g", xx,x") o²)) and a learned (g. X, a) model as defined in Eq. (2). if the functions g" and g are injective and differentiable almost everywhere, X* == a² = 2 (ie. there is no model misspecification) and the distributions of pairs of images generated from the priors ~ p" (z) and z ~ p(z) generated as (g" (z)-1): g" (zi)) and (g(z,-1), respectively, are matched almost everywhere, then = 9 o G, where a is composed of permutation and sign flips.
The formal proof is provided in Appendix A.I Similar to linear ICA, but in the temporal domain, have to assume that the transitions of generative factors across time be non-Gaussian. Specifically, if the temporal changes of ground-trui factors are sparse, then the only generator consistent with the observations is the mround-truth one (up to permutation and sign flips). The main idea behind the proof is to represent g as g o h and note that if h were not permutation, then the distributions ((g" o (2 (g* (2.)) and (g" (z;_1). gzz)) would not match, due to the injectivity of g*. Whether or not these distributions are
Z1
Z
zz
Zo
U
4
the same is equivalent to whether or not the distributions of pairs (21-1.2) 24) and (h(2-1)), h(z))) are the same. For these distributions to be the same, the function / must preserve the Gaussian marginal for the first time step well as the joint distribution, implying that it must preserve both the vector lengths and distances in the latent space. As we argue in the extended proof, this can only be the case if is a composition of permutations and sign flips.
X+11
X
Figure 2: Proof Intuition. Latent repre- sentation and example generated image pairs for ground truth (blue) and entan- gled (red) model. See text below for details.
Intuition Fig. iillssrrtess, by contradiction, why the model defined in Eq. (2) is identifiable. We consider temporal pairs of latents represented by connected points. sparse transition prior encour- ages axis- alignenig,m,n, as can be seen from the Laplace transition prior in the third image of Fig. 3
Forr a stationary stochastic process, p(2(-1) represents the instantancous marginal distribation and p(z: 12))) the transition distribution. In case of an autoregressive process with non non- Gaussian innovations with finite variance, it follows from the central limit theorem that the marginal distribution converges to Gaussian in the limit of large X.
4

---

Published as a conference paper at ICLR 2021
This results in lines that are parallel with the axes in both the ground truth (left, blue, z²) and learned model (right, red, z). In this example, 4 corresponds to horizontal position, while 2f corresponds to vertical position. The learned model must satisfy two criteria: (1) the latent factors should match the sparse prior (axis-aligned) and (2) the generated image pairs should match the ground-truth image pairs. If the learned latent factors were mismatched, for example by rotation, then the image pair distributions would not be matched. In this example, the ground truth model would produce image pairs with typically vertical or horizontal transitions, while the learned model pairs result mostly diagonal transitions. Thus, the learned model cannot satisfy both criteria without aligning the latent axes with the ground axes.
3.3 SLOW VARIATIONAL. AUTOENCODER
In order to validate our proof, we must choose probabilistic latent variable model for estimating the data density. We chose to build upon the framework of VAEs because of their efficiency in estimating a variational approximation to the ground truth posterior of a deep latent variable model (Kingma and Welling 2013). We will refer to this model as Slow VAE. In Appendix B we note shortcomings of such approach and test an alternative flow-based model.
The standard VAE objective assumes i.i.d. data and a standard normal prior with diagonal covariance _N I). To extend this to sequences, assume the
on the learned latent representations same functional form for our model prior as in Eq. ① and Eq. (2). The posterior of our model is independent across time steps. Specifically,
d II ((xx ==1
g(2,, Z4 |x+, X xt- ((z,)) xx) ((z) 1 ((2/x)
where (x) and (x) are the input-dependent mean and variance of our model's posterior. We visualize this combination of priors and posteriors in Fig. 3 For a given pair of inputs (X,, X;-1). 1), the full evidence lower bound (ELBO, which we derive in Appendix (.2 can be written
C(x,, x:-1) Eg(n, 1|x,x, .) log p(x,, X 2e, z111 DEL (1(2)-1/xx11)p/zz
(4)
Ix. (q p(z 11)1
where is a regularization term for the sparsity prior, analogous to 8 in 3-VAEs (Higgins et al. 2017) (technically, Ess only an ELBO with 7 S 1). The first term on the right-hand side the log-likelihood (i.e. the negative reconstruction error, with p(x,, x;-1 1 [2,, 2:-1) parameterized by the
decoder of the VAE), the second term is the KL to a normal prior as the standard VAE and the last term is expectation of the KL between the posterior time step t and the conditional prior p(z: 1).. The expectation in the last term taken over samples from the posterior at the previous time step (z,-11 (x))) We observed empirically that taking the mean, (xxx-1),, produces good results, analogous to the log- likellidddood that is typically evaluated at a single sample from the posterior (see Blei et al. (017) for context).
single sample
p(z,-1)
q(z(-1/x->)
pizz(z,-))
giz(x.)
In practice, we need to choose a, X, and 7. For the latter two, we perform a random search for hyper- parameters, as we discuss below. For the former, any would break the general rotation symmetry by having optimum for axis-alignes representations, which theorem [ includes as a requirement for iden- tifiability. As can be seen in Figs. and I1 a z 0.5 provides the best fit to the ground-truth marginals. However, we used = 1 as a parsimonious choice for SlowVAE, since the Laplace is a well-understood distribution that allows to derive simple closed- form solution for the BLBO in Eq. which we derive
Figure 3: SlowVAE illustration. The prior and posterior for two-dimensional latent space. Left to right: Normal prior for t 1, posterior for t - 1, conditional Laplace prior for I,, and posterior for l. The blue cross in the right three plots indicates the mean of the posterior for t
in Appendix A.2
3.4 TOWARDS AN APPROXIMATE THEORY OF DISENTANGLEMENT
number of our theoretical assumptions are violated in practice: After non-convex optimization, on finite data sample, the distributions p(x,, x,-1) and p" (x,, x;-1) are probably not perfectly
5

---

Published as a conference paper at ICLR 2021
matched. In addition, the model assumptions on p(ze, Z{-1) likely do not fully match the distribution of the ground truth factors. For example, the model may be misspecified such that a # a" or À * X², or the chosen family of distributions may be incorrect altogether. In the following section we will present results on several datasets where the marginal distributions p(z+-1) ) are drawn from Uniform (not Normal) distribution, and some of them are over unordered sets (categories) or bounded periodic spaces (rotation). Also, in practice the model latent space is usually chosen to have more dimensions than the ground truth generative model. On real data, factors of variation may be dependent (Täuble et al., 2020; Yang et al., 2020). We show this is the case on YouTube- VOS and KITTI-MÓTS in Appendix Gland we provide evidence that breaking these dependencies has no clear consequence on disentanglement in Appendix F.2 A more formal treatment of dependence is done by Khemakhem et al. (2020b) who relax the independence assumption of ICA to Independently Modulated Components Analysis (IMCA) and introduce a family of conditional energy-based models that are identifiable up to simple transformations. Furthermore, the hypothesis class G of learnable functions in the VAE architecture may not contain the invertible ground truth generator g G. it exists at all (e.g. occlusions may already lead to non-invertibility). Despite these violations, we consider it strength of our method that the practical implementation still achieves improved disentanglement over previous approaches. However, we note understanding the impact of these violations as an important focus area for continued progress towards developing a practical yet theoretically supported method for disentanglement on natural scenes.
4 DATASETS WITH NATURAL TRANSITIONS
While the standard datasets compiled by DisLib are important step towards real-world applications, they still assume the data is ii.d. As described in section 2 Locatello et al. (2020) proposed uniformly sampling the number of factors to be changed, & Rnd, and changing said factors by uniformly sampling over the possible set of values. What we refer to as "UNI" is a dataset variant modeled after the described scheme (Locatello et al. 2020) (further details in Appendix D). Considering our natural data analysis presented in Figure such transitions are certainly unnatural. Given the current state of evaluation, we provide set of incrementally more natural datasets which are otherwise comparable to existing work. We propose that said datasets should be included in the standard benchmark suite to provide step towards disentanglement in natural data.
(1) Laplace Transitions (LAP) is a procedure for constructing image pairs from DisLib datasets by sampling from a sparse conditional distribution. For each ground-trute truth factor, the first value in the pair is chosen i.i.d. from the dataset and the second is chosen by weighting nearby factor values using Laplace distributed probabilities. LAP a step towards natural data that closely resembles previous extensions of DisLib datasets to the time domain, but in a way that matches the marginal distribution of natural transitions (see Appendix [D2 for more details).
(2) Natural Sprites consists of pairs of rendered sprite images with generative factors sampled from real YouTube- VOS transitions. For a given image pair, the position and scale of the sprites are set using measured values from adjacent time points in YouTube- VOS. The sprite shapes and orientations are simple, like dSprites, and are fixed for a given pair. While fixing shape follows the natural transitions of objects, it is unclear how accurately estimate object orientation from the masks, and thus we fixed the factor to avoid introducing artificial transitions. We additionally consider a version that discretized the same number of object states as dSprites, which i) allows to use the standard DisLib evaluation metrics and ii) helps isolate the effect of including natural transitions from the effect of increasing data complexity (see Appendix D.4 for more details).
(3) KITTI Masks is composed of pedestrian segmentation masks from the autonomous driving vision benchmark KITTI-MOTE thus with natural shapes and continuous natural transitions in all underlying factors. We consider adjacent frames which correspond to mean( At) 0.05s in physical time (we report the mean because of variable sampling rates in the original data); as well as frames with a larger temporal gap of mean At) 0.15s, which corresponds to samples of pairs that are at most 5 frames apart. We show in Appendix G.3 that Slow disentanglement performance increases and then plateaus
continue increase mean At).
In summary, we construct datasets with (1) imposed sparse transitions, (2) augmented with natural continuous generative factors using measurements from unstructured natural videos, well as (3) data from unstructured natural videos themselves, but provided as segmentation masks to ensure visual complexity is manageable for current methods. For the provided datasets, the object categories

---

Published as a conference paper at ICLR 2021
Model
Data
BetaVAE
FactorVAE
MIG
MCC
DCI
Modularity
SAP
PCL
dSprites (Uniform)
(0.4)
62.1 (0.9)
(7.4)
(1.5
(1.4
99.7 (0.6)
6.0 (2.7)
Ada-GVAE
dSprites (Uniform)
88.0 (2.7)
73.1 (3.9)
46.0 (4.7)
(4.6)
(1.8
6.6.)
SAO
dSprites (Uniform)
(11.)) (5.0)
(11.5)
5888 (8.9)
47.7 (8.5)
(2.8)
4.00)
PCL
dSprites (Laplace)
99.9 (0.1)
(3.1)
(3.1)
(3.9
(3.5
93.2 (0.9)
(1.6
Ada-GVAE
dSprites (Laplace)
91.4 (1.6)
(5.9
(4.9)
56.9 (4.2)
(4.0
7.2 87.6
Slow VAE
dSprites (Laplace)
100.0 (0.0)
(3.0)
(9.3)
69.8 (2.3)
(3.6)
(1.6)
8.1 (3.0)
PCL
Natural (Discrete)
82.4 (6.7)
68.3 (8.0)
(4.2) (2.8)
(3.0)
2.1(1.1) (319
Ada-GVAF
Natural (Discrete)
83.4 (1.1)
74.8 (4.4)
(2.5) (3.2)
(2.9)
(1.4) (2.5)
VAE VAE
Natural (Discrete)
(2.6
76,2 (4.8)
(5.0)
(4.1)
(8.9
88.1 (3.6)
(2.3)
Table I: Mean and standard deviation (s.d.) metric scores across 10 random seeds. PCL is a scaled-up implementation of the method described by Hyvärinen and Morioka (2017), leveraging the encoding architecture and training hyperparameters specified in appendix E| Ada-GVAE is the leading method proposed by Locatello et al.] (2020). Bold indicates statistical significance above the next highest score (independent T-test, p 0.05). Red indicates statistical significance below the next lowest score. Results for additional datasets and models are in Table and Appendix
never change across transitions reflecting natural object permanence, Finally, as (2) and (3) use factor transitions measured from natural videos, they exhibit any natural statistical structure present for those factors, such as natural dependencies (further discussion is in Appendix [22
5 EXPERIMENTS
5.1 EMPIRICAL STUDIES
We evaluate models using the DisLib implementation for the following supervised metries: Be- VAE (Higgins et al., 2017): FactorVAE (Kim and Mnih 2018); Mutual Information Gap (MIG; Chen et al., [2018;; Disentanglement, Compactness, and Informativeness (DCI Disen- tanglement; Eastwood and Williams 2018); Modularity (Ridgeway and Mozer 2018); and Separated Attribute Predictability (SAP: Kumar et al. 2018) (see Appendix Cior metric details). None of the DisLib metrics support ground-truth labels with continuous variation, which is required for evaluation on the continuous Natural Sprites and KITTI Masks datasets. To reconcile this, we measure the Mean Correlation Coefficient (MCC), standard metric in the ICA literature that is applicable to continuous variables. We report mean and standard deviation across 10 random seeds.
In order to select the conditional prior regularization and the prior rate in unsupervised manner, we perform a random search over y E [1, 16] and À E [1, 10] and compute the recently proposed unsupervised disentanglement ranking (UDR) scores (Duan et al., 2020). We notice that the optimal values are close to = 10 and À = 6 on most datasets, and thus use these values for all experiments. We leave finding optimal values for specific datasets future work, but note that it is strong advantage of our approach that it works well with the same model specification across 13 datasets (counting LAP and UNI for DisLib and optional discretization for Natural Sprites), addressing concern posed in (Locatello et al. 2018). Additional details on model selection and training can be found in Appendix E Although we train on image pairs, our model does not need paired data points at test time. For all visualizations, we pick the models with the highest average score across the Lib metrics.
To compare our model fairly against other methods that also take image pairs as inputs, we also present performance for Commutation-Contrasiii Learning from nonlinear ICA (PCL, Hyvärinen and Morioka 2017) and Ada-GVAE, the leading method in the study by (Locatello et al. 2020).
We scaled up the implementation of PCL for evaluation on our nigh-dimensional pixel inputs, and note this method does not have any hyperparameters. For Ada-GVAE, following the paper's recommendations, we select (per dataset) using the considered parameter set [1, 2, 4, 6, 8, 16), and
the reconstruction loss the unsupervised model selection criterion (Locatello al. 2020
7

---

Published as a conference paper at ICLR 2021
Ada-GVAE PCL
SlowVAE
McC -
76.0
MCC 62.7
MCC 90.5
77 27
13
66 15
Latent
18
31
69
t
16
48
53
95
Figure 4: KITTI Masks (mean At) 0.15s). (Left) MCC correlation matrix of the top 3 latents corresponding to y-position, x-position and scale. (Right) Images produced by varying the Slow latent unit that corresponds the corresponding row in the MCC matrix.
5.2 RESULTS ON DISLIB AND NEW BENCHMARKS
In Table demonstrate favorable performance compared to PCL and Ada-GVAE across all applicable metrics for discrete ground-truth variable datasets. The relative improvement on UNI is particularly surprising given the drastic mismatch between UNI and Slow SlowVAE's VAE's assumptions. In Appendix G we report results for the remaining DisLib datasets, where the observed dSprites results largely transfer. We also outperform PCL with a (flow-based) exact likelihood implementation of our slow transition prior in Appendix F.I.T In Appendix F3
we show that model with an L2 transition (a = 2) prior performs much worse, supporting our theoretical prediction.
On the KITTI Masks dataset, one source of variation in the data is the average temporal separation within pairs of images mean( At). We present two settings (mean(Al) = 0.05s, mean (Ai) = 15s) and observe comparative increase in MCC for the latter (Table 2). Namely, the increase in performance for larger time gap is more pronounced with SlowVAE than the baselines, resulting in a statistically significant MCC gain. We pro- vide details on the settings and ablate over the mean At) parameter in Appendix G.3 where we observe a positive trend between mean At) and MCC (reflecting Table 2, in Oord et al. 2018). Finally, we also verify that the transition distributions remain sparse despite the increase in this parameter (Appendix G.3). In Fig. 4 we can see that SlowVAt has learned latent dimensions which have correspondence with the estimated ground truth factors of x/y-position and scale.
Model PCL
MCC
Natural (Continuous)
$1.7 (3.0)
Ada-GVAF
Natura (Continuous)
48.4 (4.8)
Slow
Natural (Continuous)
49.1
PCL
Kitti (mean) Ar) 0.05s)
(5.1)
Ada GVAE
Kitti (mean( At) 0.05s)
62.6
Slow
Kitti (mean( Ar) 0.05x)
66.1 (4.5)
PCL
Kitti (mean( Ar) 0.15a)
(3.3)
Ada Ada-GVAE
Kitti (meaní Ai) 0.15w)
(7.6
Slow
Kitti (mean( (r) 0.15x) (5.8)
Table Continuous ground-truth variable datasets. See T for details.
Locatello et al. (2018) showed that all i..dd models performed similarly across the DisLib datasets and metrics when testing was carefully controlled. However, in Fig. 5 we observe that the different modeling assumptions result in differences in representation quality. To construct the visuals, we first compute the sorted correlation matrix between the latents (rows) and generative factors (columns), which visualize as correlation matrices. The matrices are sorted via linear sum assignment such that each ground-truth factor is non- greedrledlly associated with the latent variable with highest correlation (Hyvärinen and Morioka 2016). Below the matrices scatter plots that reveal the decodability of the assigned latent factors. In each scatter plot, the horizontal axis indicates the ground truth value, the vertical axis indicates the corresponding latent value, and the colors indicate object shape. The models displayed are those with the maximum average score across evaluated metrics.
The latent space visualizations use the known ground-truth factors to aid in understanding how each factor is encoded in way that is more informative than exclusively visualizing latent traversals or embeddings of pairs of latent units (Cheung et al., 20044] Chen et al., 2016 Szabó et al., 2017: Ma al. 2018). For example, the third row, we observe that several models have sinusoidal variation with frequencies ~ w, 2w, and 4w, which correspond to the three distinct rotational symmetries of the shapes: heart, ellipse and square. This directly impacts MCC performance (third row the MCC matrix), which measures rank correlation between the matching latent factor (an angular variable) and the ground truth, which encodes the angles with monotonically increasing indices. Furthermore, the square has four-fold rotational symmetry and repeats after 90", but it is represented in full 360" rotation in the DisLib ground truth encoding format, resulting in different ground truth labels for identical input images.
8

---

Published as a conference paper at ICLR 2021
AnnealedVAE
B-TCVAE
B-VAE
DIP-VAE-I
DIP-VAE-II
Ada-GVAE (LAP)
PCL (LAP)
SlOWVAE (LAP)
MCC 60 6
MCC
65.7
61.1
MCC 38.7
MCC-
60.8
MCC
MCC
42
12
37
92
R2
38
91
2
21
0
24
1
3
19
2
80
16
38
2
21
100
0
100
0
2
100
43
2
85
45
100
17
0
100
0
Latents
0
100
1
0
O
100
0
0
0
100
14
53
2
4
0
26
90
11
2
79
14
82
1
0
0
1
100
25
14
20
2
66
24
2
5
32
29
2
6
1
2
28
2
46
64
47
2
1
5
10
27
5
42
18
5
27 16
2
11
1
41
30
1
0
15
81
20
2
10
1
0
1
1
5
2
2
10
11
22
5
5
11
47
10
3
9
5
11 24
17
29
1
43
32
18
5
2
11
12
12
22
47
14
Scale Shape
Pos
Figure 5: DSprites Latent Representations: (Top) shows absolute MCC between generative and model factors (rows are rearranged for maximal correlation on the main diagonal). The columns correspond to generative factors (shape, scale, rotation, sly-position) and the values correspond to percent correlation. A more diagonal structure in the upper half corresponds to better one-to-one mapping between generative and latent factors. (Bottom) shows individual latent dimensions (y-axis) over the matched generative factors (x-axis). Colors encode shapes: heart/yellow, elliisseturquoise, and square/purple.
A similar observation can be made with respect to the categorical factors, which are also represented as ordinal ground truth variables. For example, the PCL correlation score (top left element in the PCL MCC matrix) is quite high, while the corresponding shape correlation score for Slow VAE is quite low. However, we consider the shape scatter plots, we clearly see that VAE separates the three shapes more distinctively than PCL, only in an order that differs from the ground truth. One solution to modify MCC to report the maximum correlation over all permutations of the ground truth assignments, although brute force methods for this would scale poorly with the number of categories. We also note that datasets where we see small performance differences among models (e.g., Cars3D) have significantly more discrete categories (e.g.. 183) than the other datasets (3 6). This could also explain why all models considered in Table and 2 perform comparably on the Natural Sprites datasets, where unlike KITTI Masks the ground truth evaluation includes categorical and angular variables. We note that properly evaluating disentanglement is an ongoing area of research (Duan) et al. 2020), with notable preliminary results in recent work (Higgins et al. 2018 [Bouchacourt et al. 2021
Tonnaer et al. 2020)
6 CONCLUSION
We provide evidence to support the hypothesis that natural scenes exhibit highly sparse marginal transition probabilities. Leveraging this finding, we contribute novel nonlinear ICA framework that is provably identifiable up to permutations and sign-flips a stronger result than has been achieved previously. With the VAE model we provide a parsimonious implementation that is inspired by a long history of learning visual representations from temporal data (Sutton) 1988 Hinton 1990 Földiák 1991). We apply this model to current metric-based disentanglement benchmarks to demonstrate that it outperforms existing approaches (Locatello et al. 2020; Hyvärinen and Moriokal 2017) on aggregate without any tuning of hyperparameters to individual datasets. We also provide novel video dataset benchmarks guide disentanglement research towards more natural domains.

---

Published as a conference paper at ICLR 2021
We observe that these datasets have complex dependencies that our theory will have to be extended to account for, although we demonstrate with empirical comparisons the efficacy of our approach. In addition to Natural Sprites and KITTI Masks, suggest that YouTube- VOS will be valuable as a large-scale dataset that is unconstrained by object type and scenario for more advanced models. Variance in such categorical factors is problematic for evaluation due the cited drawbacks of existing quantitative metrics, which should be addressed in tandem with scaling to natural data. Taken logether, our dataset and model proposals set the stage for utilizing knowledge of natural scene statistics to advance unsupervised disentangled representation learning.
In our experiments see that approximate identification measured by the different disentan- glement metrics increases despite violations of theoretical assumptions, which is in line with prior studies (Shu et al. 2019; Khemakhem et al. 2020a Locatello et al. 2020). Nevertheless, future work should address gaining better understanding of the theoretical and empirical consequences of such model misspecifications, in order make the theory of disentanglement more predictive about empirically found solutions.
10

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
The authors would like to thank Francesco Locatello for valuable discussions and providing numerical results to facilitate our experimental comparisons. Additionally, we thank Luigi Gresele, Matthias Tangemann, Roland Zimmermann. Robert Geirhos, Matthias Kümmerer, Cornelius Schröder, Charles Frye, and Sarah Master for helpful feedback in preparing the manuscript. Finally, the authors would like to thank Johannes Ballé, Jon Shlens and Eero Simoncelli for early discussions related to the ideas developed this paper.
This work was supported by the Deutsche Forschungsgemeinccaatt (DFG) in the priority program 1835 under grant BR2321/5-2 and by SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms (TP3), project number: 276693517. We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting LS and YS. DP was supported by the German Federal Ministry of Education and Research (BMBF) through the Tübingen Al Center (FKZ: 1010011S11999 A8039A). IU, WB, and MB are supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (Dol/IBC) contract number D16PC00003. The U.S. Government is authorized to reproduce and distribute reprints for Gov- ernmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed implied, of IARPA, DoI/IBC, or the U.S. Government.
The authors declare no conflicts of interests.
BROADER IMPACT
Representation learning is the heart of model building for cognition. Our specific contribution is focused on core methods for modeling natural videos and the datasets used are more simplistic than real-world examples. However, foundational research on unsupervised representation learning has potentially large impact on AI for advancing the power of self-learning systems.
The broader field of representation learning has a large number of focused research directions that span machine learning and computational neuroscience. As such, the application space for this work vast. For example, applications in unsupervised analysis of complicated and unintuitive data, such medical imaging and gene expression information, have great potential to solve fundamental problems in health sciences. A future iteration of our disentangling approach could be used to encode such complicated data into diwer-diensioonall and more understandable space that might reveal important factors variation to medical researchers. Another important and complex modeling space that could potentially be improved by this line of research is in environmental sciences and combating global climate change.
Nonetheless, acknowledge that any machine learning method can be used for nefarious purposes, which can be mitigated via effective, scientifically informed communication, outreach, and policy direction. We unconditionally denounce the of derivatives of work for weaponized or wartime applications. Additionally, due to the lack of interpretability generally found in modern deep learning approaches, it is possible for practitioners machine learning applications. Although we certainly do not solve this problem, our focus on providing identifiable solutions to representation learning likely beneficial for both interpretabilityy and fairness in machine learning.
inadvertenlly introduce harmful biases or errors in

---

Published as a conference paper at ICLR 2021
REFERENCES
Tameem Adel, Zoubin Ghahramani, and Adrian Weller. Discovering interpretable representations for both deep generative and discriminative models. In International Conference on Machine Learning, pages 50-59, 2018.
Jonathan Barron Jitendra Malik. Shape, albedo, and illumination from a single image of an unknown object. In 2012 IEEE Conference on Computer Vision and Puttern Recognition, pages 334-341 IEEE, 2012.
Anthony Bell and Terrence Sejnowski. An information information-maxiiizztion maximization approach to blind separation and blind dcconvolution. Neural Computation, 7(6): 112--1999, 1995.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 15(8):1798-11 -1888, 2013.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859 877, Apr 2017. ISSN 274X. doi: 10.1080/ 01621459.2011 20155 20t.220117.1777... (RLhhtp::// //cx doi org/ 10t.10000000012222222020 2017 12355733
Diane Bouchacourt, Mark Ibrahim, and Stéphane Deny. Addressing the topological defects of disentanglement via distributed operators, 2021.
Samuel R. Bowman, L. Vilnis, Oriol Vinyals, Andrew M. Dai, R. Józefowicz, and S. Bengio. Generating sentences from continuous space. In CoNLL, 2016.
Christopher P Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling beta-vae. arXiv preprint arXiv:1804, 03599, 2018.
Christpher P. Burgess, Loie Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint MxXiv:/901.113900 2019.
Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal scientific computing, 16(5):1190-120) 1995.
Charles F Cadieu and Bruno A Olshausen. Learning intermediate- representations of form and motion from natural movies. Neural Computation, 24(4):827-866, 2012.
J-F Cardoso. Source separation using higher order moments. In International Conference Acoustics, Speech, and Signal Processing, pages 2109-2112. IARF, 1989.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pages 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abheel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances neural information processing systems, pages 2172-2180, 2016.
Brian Cheung, Jesse A Livezey, Arjun K Bansal, and Bruno Olshausen. Discovering hidden factors of variation in deep networks. arXiv preprint arXiv: 1412.633, 2014.
Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314, 1994.
Elliot Creager, David Madras, Jörn- Henrie Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In International Conference on Machine Learning, page 1436 1436-1445, 1445, 2019.
Emily L Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In Advances in Neurul Information Processing Systems, pages 4414-4423, 2017.
Adji B. Dieng, Yoon Kim, Alexander M. Rush, and D. Blei. Avoiding latent variable collapse with generative skip models. ArXiv, abs/1807 80888807734,,,. 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. ArXiv, abs/1410.8516. 2017a.
Laurent Dinh, Jascha Sohl- Sohl-Dickstein, and S. Bengio. Density estimation using real nvp. ArXiv, 005//60500883, 2017b.
12

---

Published as a conference paper at ICLR 2021
Sunny Duan, Loic Matthey, Andre Saraiva, Nick Watters, Christopher Burgess, Alexander Lerchner, and Irina Higgins. Unsupervised model selection variational disentangled representation learning. International Conference on Learning Representations (ICLR), 2020.
Cian Eastwood Christopher KI Williams. framework for quantitative evaluation disentangled representations. In In International Conference on Learning Representations. 2018.
Peter Földiák. Learning invariance from transformation sequences. Neural Computation, 3(2):144-000, 1991.
Lijian Gao, Qirong Mao, Ming Dong, Yu Jing, and Ratna Chinnam. On learning disentangled representation for acoustic event detection. In Proceedings of the 27th ACM International Conference on Multimedia, pages 2006-2014, 2019.
Andreas Geiger, Philip Lenz, and Raquel Urtasun, Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Muhammad Walced Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer, On the transfer of inductive bias from simulation to the real world: disentanglement datasct. In Advances in Neural Information Processing Systems, pages 15714-15725, 2019.
Will Grathwohl and Aaron Wilson. Disentangling space and time in video with hicrarchical variational auto- encoders. arXiv preprint 1/2.444// 2016.
Klaus Greff, Raphael Lopez Kaufmman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick. and Alexander Lerehner. Multi-object representation learning with iterative variational inference. arXiv preprint 1rXXiv:103..0442 2019.
Luigi Gresele, Giancarlo Fissore, Adrian Javaloy, Bernhard Scholkopf. and Aapo Hyvarinen. Relative gradient optimization of the jacobian term in unsupervised deep learning. ArXiv, abss22006.15090 2020.
Wakako Hashimoto. Quadratic forms natural images. 765-788, 2003.
Network: Computation Neural Systems, 14(4):
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick, Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint 1911v:190 2019.
Olivier J Hénaff, Robbe LT Goris, and Eero Simoncelli, Perceptual straightening of natural vidcos. Nature neuroscience, 22(6))984-991, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-vae: Learning basic visual concepts with constrained variational framework. International Conference on Learning Representations (ICLR), 2(5)66, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerehner. Towards definition of disentangled representations. arXiv preprint arXiv: arXiv:1812.022 1812.0223. 2018.
Geoffrey E Hinton. Connectionist learning procedures. In Machine learning, page 208. Elsevier, 1990.
Aapo Hyvärinen and Patrik Hoyer. Emergence of phase-and shift-invariant features by decomposition of natural images into independent feature subspaces. Neural computation, 12(7):1005-1720, 2000.
Aapo Hyvärinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems, pages 3765-3773, 2016.
Aapo Hyvärinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources, In Proceedings of Machine Learning Research, 2017.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Nenworks, 12(3):429-44 439, 1999.
Aapo Hyvärinen, Jarmo Hurri, and Jaakko Väyrynen. Bubbles: unifying framework for low-level statistical properties of natural image sequences. JOSA A, 20(7):1377-1255 2003.
Aapo Hyvärinen, Hiroaki Sasaki, and Richard E Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. arXiv preprint arXiv:1805.0m 08651, 2018
Christian Jutten and Jeanny Herault. Blind separation of sources, part An adaptive algorithm hased on neuromimetic architecture. Signal Processing, 24(1):1-10, 1991.
13

---

Published as a conference paper at ICLR 2021
Ilyes Khemakhem, Diederik Kingma, and Aapo Hyvärinen. Variational autoencoders and nonlinear ica: unifying framework. International Conference on Artificial Intelligence and Statistics (AISTATS), 2020a.
Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-heem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 33,
2020h.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint wrXiv:1802.05985. 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in neural information processing systems, pages 10215-10224, 2018
Durk P Kingma, Tim Salimans, Rafal Jozefowicz. Xi Chen, Ilya Sutskever, and Max Welling, Improved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pages 4743-4751, 2016.
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. Advances neural information processing systems, pages 2539-2547, 2015.
Abhishek Kumar. Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pages II-104. IEEE, 2004.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. arXiv preprint arXiv:1811. 2018.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness disentangled representations. In Advances in Neural Information Processing Systems, pages 14611 1 14624, 2019.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. arXiv preprint Stti:22002008888 2020.
James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't blame the elbo! linear vae perspective posterior collapse. In Advances in Neural Information Processing Systems, pages 9408 9418, 2019.
Liqian Ma, Qianru Sun, Stamatios Georgoulis, Lue Van Gool, Bernt Schiele, and Mario Fritz. Disentangled person image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 99-108, 2018
Lars Maalşt, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. In Advances in neural information processing systems, pages 6551-6562, 2019.
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In Proceedings of the 36th International Conference on Machine Learning, pages 4402-4412, 2019.
Loie Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https//ggttth..oooddeepiii/////s
2017.
Stanislaw Mazur and Stanislaw Ulam. Sur transformations isométriques e'espaces vectoriels normés. Acad. Sci. Paris, 194(946-948):116, 1932.
Anton Milan, Laura Taixé, Ian Reid, Stefan Roth, and Konrad Schindler. MOT16: A benchmark for multi-object tracking. arXiv:1603.008: [cs]. March 2016.
Graeme Mitchison. Removing time variation with the anti-hebbian differential synapse. Neural Computation, 3 (3):312-320 1991.
Hossein Mobahi, Ronan Collobert, and Jason Weston. Deep learning from temporal coherence in video. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 737-744, 2009.
14

---

Published as a conference paper at ICLR 2021
Hiroshi Morioka. Time-contrastive learning (tel), 2018. URLLpttpp///githubb.cmm
Bruno A Olshausen. Learning sparse, overcomplete representations of time-varying natural images. In Proceedings 2003 International Conference on Image Processing (Cat. No. 03CH37429), volume 1, pages 1-41. IEFE, 2003.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 187.03388 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, pages 8024 8035, 2019.
E. Pedregosa, G. Varoquaux, Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, Duchesnay. Scikit-lcarn:
V. Dubourg, J. Vanderplas, A. Passos, Cournapeau, M. Brucher, M. Perrot, Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Edouard Pineau, Razakarivony, and T. Bonald. Time series source separation with slow flows. ArXiv, abs/2007.100 101 2020.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making, In Advances in neural information processing systems, pages 1252-1260, 2015.
Karl Ridgeway. A survey of inductive biases for factorial representatiio-lcarmigg arXiv preprint arXiv:161..05299, 2016.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. In Advances Neural Information Processing Systems, pages 185-194, 2018
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea directions (by accident). In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 12406-12415, 2019.
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantecs. arXiv preprint 1rXivv1910.09771 2019.
Fero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Annual review of neuroscience, 24(1): 193-1126, 2001.
Fabian Sinz, Sebastian Gerwinn, and Matthis Bethge. Characterization of the p-generalized normal distribution. Journal of Multivariate Analyxis, 10005):817-820, 2009.
Jascha Sohl-Dickstein, Ching Ming Wang, and Bruno A Olshausen. An unsupervised algorithm for learning lie group transformations. arXiv preprint arXiv:1001.100: 2010.
Peter Sorrenson, Carsten Rother, and UIrich Kothe. Disentanglement by nonlinear ica with general incompressible-flow networks (gin). ArXiv, abs/2001.04872. 2017.
Mikhail Fedorovich Subbotin. On the law of frequency of error. Mat. Sh., (1)))2996-31,, 1923.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9-44, 1988.
Attila Szahó, Qiyang Hu, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro. Challenges in disentangling independent factors of variation. arXiv preprint arXiv:1711.02244, 2017.
Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algorithms. Communica- tions on Pure and Applied Mathematics, 66(2):155-664, 2013.
Esteban G Tabak, Eric Mijddnn cations in Mathematical Sciences, 8(1):217-233 2010.
Density estimation by dual ascent of the log- lilkkkieehodd.. Communi-
Lock Tonnaer, Luis A. Pérez Rey, Vlado Menkovski, Mike Holenderski, and Jacobus W. Portegies. Quantifying and learning disentangled representations with limited supervision, 2020.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Anirudh Goyal, Francesco Locatello, Bernhard Schölkopf. and Stefan Bauer. Is independence all you need? on the generalization of representations learned from correlated data, arXiv preprint arXiv:2006.0788 07886, 2020.
15

---

Published as a conference paper at ICLR 2021
Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, and Mario Lucic. Sclf-supervised learning of video-induced visual invariances. arXiv preprint 1rXiv:19/..02783, 2019.
Richard Turner and Maneesh Sahani. A maximum-likellienn interpretation computation, 19(4):1022-1038, 2007.
feature analysis. Neural
Rishi Veerapaneni, John Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction visual model-based reinforcement learning. arXiv preprint arXiv: 12827, 2019.
Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sckar, Andreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmentation. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Nicholas Watters, Loic Matthey, Sebastian Borgeaud, Rishabh Kabra, and Alexander Lerchner. Spriteworld: A flexible, configurable reinforcement learning environment, 2019. URL https://gitthbbb com/
deepmi nd/ http///ttttwwor/////o
Marissa A. Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S. Ecker. Unmasking the inductive biases of unsupervised ohject representations for video sequences. arXiv preprint arXiv:2006.070:4 2020.
Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14(4))715-770 2002.
Markus Wulfmeier, Arunkumar Byravan, Tim Hertweck, Trina Higgins, Ankush Gupta, Tejas Kulkarni, Malcolm Reynolds, Denis Teplyashin, Roland Hafner, Thomas Lampe, and Martin Ricdmiller. Representation matters: Improving perception and exploration for robotics. arXiv preprint arXiv:201 1.0::20/1.007550 1.01758, 2020.
Ning Xu, Linjie Yang, Yuchen Fan, Dingeheng Yue, Yuchen Liang, Jianchao Yang. and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint
2018.
Linjic Yang, Yuchen Fan, and Ning Xu. Vidco instance segmentation. arXiv preprint arXiv: 2019.
Mengyue Yang. Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Wang. Causalvae: Structured
causal disentanglement variational autoencoder. arXiv preprint arXiv:2004.0860 2020.
llker Yildirim, Mario Belledonne, Winrich Freiwald, and Josh Tenenbaum. Efficient inverse graphies biological face processing. Science Advances, 6( 611))caax5979, 2020.
Will Zou, Shenghuo Zhu Kai Yu, and Andrew Ng. Deep learning invariant features simulated fixations video. In Advances in neural information processing systems, pages 3203-3211, 2012.
16

---

Published as a conference paper at ICLR 2021
APPENDIX
A Formal Methods
18
Proof of Identifiability
18
A.2 Kullback Leibler Divergence of Slow Variational Autoencoder
20
Choosing a Latent Variable Model
23
Disentanglement Metries
24
Mean Correlation Coefficient
25
C.2 DisLib Metrics
25
D Natural Datasets
27
D.1
Uniform Transitions (UNI)
27
D.2 Laplace Transitions (LAP)
28
D.3 YouTube-VOS
28
D.4 Natural Sprites
28
D.5
KITTI MOTS Pedestrian Masks (KITTI Masks)
29
Model Training and Selection
31
Extended Comparisons and Controls
32
Comparison Nonlinear ICA
32
Joint Factor Dependence Evaluation
33
Transition Prior Ablation
33
Additional Results
34
G.1
Extended Data Analysis
34
DisLih Results
37
KITII Masks At Ablation
37
G.4
Latent Space Visualizations
38
17

---

Published as a conference paper at ICLR 2021
A FORMAL METHODS
Function variable
Description
g
Generator
Prior shape
A
Prior rate
p(z)
Prior
2 ~ p(z)
Latent variables
X g(z)
Generated images
((2
Variational posterior
Table Glossary terms. We use a (i.e. 9²) when necessary to highlight that we are referring to the ground truth model.
A.1 PROOF OF IDENTIFIABILITY
To study disentanglement, we assume that the generative factors z E R.D are mapped to images
E N
(usually D & N, but see section B by nonlinear ground-truth generator g" z 1 x.
Theorem Let (g", X², a²) and (g, X, a.) respectively be ground-truth and learned generative mod- els as defined in Eq. the following conditions are satisfied:
(i) The generators 9" and are defined everywhere in the latent space. Moreover, they are injective and differentiable almost everywhere,
(ii) There is no model misspecification i.e. a = a and x= X3, so p(z) = p" (z).
(iii) Pairs of images are generated as (x--1xx)) = (z,-1,, g" (z:)) and
(xt-1,X,) (g(z)))).g(zz)),
(iv) The distributions (xi 1: x;) and X,) are the same (i.e. the corresponding densities are equal almost everywhere: p" (Xt-1, X.) = p(Xt-1,X, Xt),
then g g" o, where a is composition of permutation and sign flips.
Proof. Since g(z) can be written for some function h on the latent space.
(g" - (z),, we can assume that g g"
We first show that the function h is bijection on the latent space. It is injective, since both and g" are injective. Because of continuity of h, if it were not surjective, there would be some neighborhood Uş of that would not have a pre-image under h. This would mean that images generated by g" from Uş would have zero density under the distribution of images generated by g (i.e. p(g" (U=)) 0). This density would be non-zero under the distribution of images directly generated by the ground-true -truh generator g" (i.e. p" (U2)) * O), which contradicts the assumption that these distributions are equal. It follows that is bijective.
In the next step, we show that the distribution of latent space pairs (h(z,-1), fz())) matches the latent space prior distribution (i.e. h preserves the prior distribution in the latent space). Indeed, using the assumption that the distributions of (g" (2:-1), g" (z,)) and ((g" (2, (g" o (2,)) are the same, we can write the following equality using the change of variables formula:
det d(g")='
p" px(x-11.x)) p((''))=(xx-11111 (g")
((xx-1xxx))
d(g")-1
-pa(l")) -x (xx (xx) det
d(xx-
=((x-1,x1),
where p and Pa are densities of (z,-1, 2,) and (h(z)-1), hz,))) Since the determinants above cancel, these densities equal the pre-image of any pair of images (x,-1, x,). Because g" is defined
18

---

Published as a conference paper at ICLR 2021
everywhere in the latent space, p and Pa equal for any pair of latent space points. Applying the change of variables formula again, we obtain the following equation:
dh-1
p(z--1,21 11) - p(h (0:h-(((,)))
7(2 1),(((())))))))
det (d(2--122))
(11--((z-11)) pp (zi) h (z- det (dh (all-((t-1))
(dh- dz
)
(6)
det
dz
pp(z-11 p(zz 11-1).
Note that the probability measure is the same before and after the change of variables, since we showed that the prior distribution in the latent space must be invariant under the function h. The same condition for the marginal p( ((Zz-1) is as follows:
(2t- 1)) det dh (Z--))
p(zc-1) p(h ((04-((2-11)
dz,-1
Solving for the determinant of the Jacobian in 7 and plugging into we obtain
p(z))
plze 22-1) p(h h (Zc-))
p(h (z+))
Taking logs both sides, we arrive at the following equation:
A(I2z, z la (z,) h (2. B(1 ||1 (z: where A and B are the constants appearing in the exponentials in p(z--1) and p(z: 11-.). The logs of normalization constants cancel out.
For any Z₁ we can choose Z = making the left hand side in (9) equal to zero. This implies that | / ||h [2 for any Z,, i.c. function h-1 preserves the 2-norm. Moreover, the preservation
of the 2-norm implies that p(z(-1) 1) = p(h (z+-1) and therefore it follows from 7 that for any
dh (
det
= 1.
(10)
dz
Thus, the left hand side of can be re- re-written as |
11
1 h (2:-1(221))))) =
(11)
This means that -11 preserves the a-distances between points. Moreover, because is bijective, the Mazur-Ulam theorem (Mazur and Ulam 1932) tells us that h must be affine transform.
In the next step, to prove that must be a permutation and sign flip, let us choose an arbitrary point Z:-1 and - εe, (21,1...., 21,t + Z1,.). Using [1] and performing Taylor expansion around 2 we obtain the following:
Ea |z. 2--1l/a
/ (2.-1 +eex)
1- (2-
(12)
(hh (Oh (2)-1)
Oh (Zo-1) + +0
E
1z1-1.
dz--1.k
The higher-order terms (ss²) are zero since h is affine, therefore dividing both sides of the above equation by ex we find that
(Ohj
(2
(z(-1)
ah. [2
1
(13)
0zz-1.b
Oz1-1k 1.k
The vectors of k-th partial derivatives of components of h are columns of the Jacobian matrix dh (xx) Using the fact that the determinant of that matrix equal to and applying Hadamard's inequality, we obtain that
dot (dh (an)-(2) - 15 II
(
..... ah (z+ (2)1(211)) (2)))) d2-114
dz -15
(14)
θ21-1.
k=
19

---

Published as a conference paper at ICLR 2021
Since < 2, for any vector it holds that ||v||2 Σ
l|vlla. with equality only if at most one component of v is non-zero. This inequality implies that both (13) and (14) hold at the same time and only if
(( @ (2--)
Tn (2 (2-11)
-
(10 021-1,k
[ (0/1((41) (z--1)
Oh (7-(2))) (2--1)
a2 )
=, (15)
02--1,k
d2--1.8
meaning that only one element of these vectors of k-th partial derivatives non-zero, and it equal to I or -1. Thus, the function h is composition of a permutation and sign flips at every point. Potentially, this permutation might be input-dependent, but we argued above that affine, therefore the permutation must be the same for all points.
A.2 KULLBACK LEIBLER DIVERGENCE OF SLOW VARIATIONAL AUTOENCODER
The VAE learns a variational approximation to the true posterior by maximizing lower bound on the log-likelihood of the empirical data distribution D
Ex,-1xx>> boo-Dllogpie (0x x:) z
(16)
Ex (lt. 1o, 2,
log ((z,, 2 x,,1),
For this, we need to compute the Kullback-Leibler divergence (KL) between the posterior (Zz, Zz-1 X,, x;-1) and the prior p(z,, Zt-1). Since all of these distributions are per design factorial, we will, for simplicity, derive the KL below for scalar variables (log probaabbbbilitis will simply have to be summed to obtain the full expression). Recall that the model prior and posterior factorize like
p(zu 26-1) p(z) 2- p(2(-1) q(2,, 21 2xx 11,, x1-1) = (( x1) a(2 1(21-11x0-1).
(17)
Then, given pair of inputs (x)-1,X)). the KL can be written
Dez(g(z6 21-1 x ((2,,) 2t 1)) 1o log (((()x)) g(= 1
p(2122111 (((2-1)
= E wella 11xe log a( (((e-11x(-1) Exces reglaam 1xaxx
[log (
p(=1-1)
p(z 21-1
(ol))))H((((ee H(g(zo(x))) + Essinglan xe-1)
[H( (x)) p(
-
(18)
Where we use the fact that KL divergences decompose like (x)(X,Y)=H(XX1) (differential) cross-entropy H(X, Y) and entropy H(X). The first term of the last line in (18) is the same KI. divergence as the standard VAE, namely between a Gaussian distribution ((x x 1) with some ((x;-1) and (xx-1) and standard Normal distribution p(2-1). The solution of the KL is given by (9(2t11 (1)2((1))1)))))) = log (x;-1) + & (x(xx=1)² + oxx-1)² -) (Bishop 2006). The second term the RHS, i.c. the entropy of Gaussian is simply given by H (gg (z x))) (x.()
H(X) into
To compute the last term on the RHS, let us recall the Laplace form of the conditional prior
(=( = exp -1 - 2-11--
(19)
Thus the cross-entropy becomes
= (o
(20)
= = log
+ XF 2-11]-
2
20

---

Published as a conference paper at ICLR 2021
Now, if some random variable X (H, o²), then Y [X] follows folded normal distribution, for which the mean is defined as
2 exp
4
2Φ (
E||=||] o
(21)
2r²
where & is the cumulative distribution function of a standard normal distribution (mean zero and variance one). Thus, denoting s(x+) and (x,) ) the mean and variance of g(zx(x)). and defining u(xx,2111 p(x)) 21-1. we can rewrite further
((((2)))))
-log 2 2
A (xxx)\
+
exp
(- [_(Xxe, 21-1)) -u(xe, 21-1) (1-20
20 (x
[u(xe, (1(x,, 2(-1)
o(xx)
π
(22)
21

---

Published as a conference paper at ICLR 2021
MCC
loss
z²*
%
X"
Az Z
X
0.9
7.8
0.5
5.3
1.0
8.3
0.5
5.8
1.0
9.0
0.7
6.5
1.0
10.1
0.9
7.0
1.0
11.6
0.9
7.4
2x
(a) Slow VAE performance.
z
Az*
Az
1.0
8.8
0.6
2.8
1.0
8.9
0.7
4.2
1.0
9.0
0.7
5.0
1.0
9.1
5.6
1.0
9.2
0.7
6.1
(b) SlowFlow performance.
Figure 6: VAE failure modes. Rows respectively indicate x = 0.2, 0,4, 0.8, 1.0 from Eq. (4) The left five columns show values for 100 randomly chosen examples, while the μ and columns show values for the full training set. Columns the sets (z", z). (Az", Az), (x*, x) all have the same (arbitrary) scale factors the axes. Lines indicate trajectories from time-point t to + and color indicates the angle the trajectory vector with respect to the canonical variable axes. The f axes is scaled from -4 4, and axes scaled from encoding values from test images. The rightmost plots show a shift in the relationship between the mean correlation coefficient (MCC) (black, higher is better) and training loss (red, lower is better) as one increases k.
where individual dots represent latent
22

---

Published as a conference paper at ICLR 2021
B CHOOSING A LATENT VARIABLE MODEL
Our proposed method for disentanglement can be implemented in conjunction with different proba- bilistic latent variable models. In this section, we compare VAEs and normalizing flows as possible candidates.
Variational Autoencoders (VAEs) (Kingma and Welling 2013) are a widely used probabilistic latent variable model. Despite their simple structure and empirical success, VAEs can converge to pathological solution called posterior collapse (Lucas et al. 2019 Bowman et al., 2016 He et al.) 2019). This solution results in the encoder's variational posterior approximation matching the prior, which is typically chosen to be a multivariate standard normal (z/|x) R p(z) = (0, I). This disconnects the encoder from the decoder, making them approximately independent, i.e. p(x(z) Rs p(x). The failure mode is often observed when the decoder architecture is overly expressive, i.e. with autoregressive models, or when the likelihood p(x) is easy to estimate. Approaches that alleviate this problem rely on modifying the ELBO training objective (Bowman et al. 2016 Kingma et al. 2016) or restricting the decoder structure (Dieng al. 2019 Maalóe et al. 2019p. However, these approaches come with various drawbacks, including optimization issues (Lucas et al. 2019).
Another approach to estimate latent variables are normalizing flows which describe sequence of invertible mappings by iteratively applying the change of variables rule (Dinh et al. (0176)) Unlike VAEs, flow based latent variable models allow for direct optimization of the likelihood (Dinh et al.) 2017b). Most normalizing flow models rely a fast and reliable calculation of the determinant of the Jacobian of the outputs with respect to the inputs, which constrains the architectural design and limits the capacity of the network (Tabak et al.) 2010: Tabak and Turner [2013] Dinh al. 2017b). Thus, competitive flows require very deep architectures in practice (Kingma and Dhariwal 2018). Furthermore, flows not directly suited for scenario where the observation space is higher dimensional than the generating latent factors, dim(z) < dim(x), ; as the computation of the determinant requires a square Jacobian matrix. We tried setting dim(z) = dim(x) dim(z"), but observed instability while optimizing the objective defined below.
It is straightforward derive flow-based objective based on the assumptions in Eq. ( We consider a normalizing flow with with K blocks fx) fK o o : + z. The coupling blocks can refer to nonlinear mixing similar to Kingma and Dhariwal (2018). or in the linear case (K to an invertible de-mixi mixing matrix. This leads to the following estimation of the likelihood
05k
-1
K II
0fs
p(x = x)) P((x))
II det
det
025-111-1 1,1-1
OZ5-111
(23)
Note that (x)(x= is Gaussian and (x: is a Laplacian, similar to Eq. 2 During optimization we take the log of both sides and minimize w.r.t. the parameters of f. We refer to
this estimator as SlowFlow. Our SlowFlow model is very similar to the flow described in (Pineau al. 2020), who use Gaussian transition prior and therefore would have weaker identifiability guarantees. Next, we compare SlowFlow and Slow VAE in the context of disentanglement.
To demonstrate the posterior collapse in VAEs, we generate data points (x₂, xt-1) according to Eq. (2) with two dimensional latent space dim (z*) 2. We consider a trivial linear mixing of x W'z = =z" with
W =iag(1) k)
(24)
and x. E [0. As can be seen by looking at the and μ outputs of the encoder in Fig 64, for 0.4, the encoder for the minor axis collapses to the prior. The decoder then tries to minimize the reconstruction loss by solely covering the first principal component of the data, which is also described in Rooliek et al. (2019). Despite the collapse and decrease in MCC, the Slow VAE loss from Eq. 4 still improves during training. On the other hand, a simple linear SlowFlow model f(x) = Wx, which directly optimizes the likelihood, recovers the latents consistently as seen by the MCC measure (F)
To show the strength of the VAE model we increase the complexity of the data-distti by using mon-linear expanding decoder such that dim(x) > dim(z") In Fig. 7 we observe that increasing the input dimensionality is sufficient for Slow VAE to find the corresponding latents and achieve high MCC with low loss.
23

---

Published as a conference paper at ICLR 2021
MCC
loss
z
%
X
Z
Az
X
0.3
36.6
0.2
34.1
0.6
81.7
0.3
61.0
1.0
538.4
0.3
119.7
π 2x
Figure VAEs perform better when data dimensionality exceeds the latent dimensionality. VAFs prefer data dimensions to be greater than latent dimensions. Individual subplots are as described in Fig. 6 For all data in this experiment we used 20-dimensional latent space, dim(z") 20. Each row corresponds to the dimensionality of the x", with values of 20, 200, and 2000. The first two dimensions of zz are plotted as well as the two dimensions of z with the highest corresponding mean correlation coefficient (MCC). The x' and data are projected onto their first two principal component axes before plotting. A two-layer mixing matrix was used to transform data from Zg: to Xg. As increases the data dimensionality, the Slow network performs increasingly better in terms of MCC, although worse in terms of total training loss.
Each estimation method is practically useful in different experimental settings. In the case when the mixing operation is trivially defined (Eq. (4). or when the number of dimensions in match those in x*), the VAE estimator tends to learn a pathological solution. On the other hand, the normalizing flow estimator does not scale well to high dimensional data due the requirement of computing the network Jacobian. Additionally, the framework for constructing normalizing flow estimators assumes the latent dimensionality equal to the data dimensionality Together these results lead choose an estimator based the nature of the problem. For contributed datasets and the DisLib experiments we adopt the VAE framework. However, one aims to perform simplified experiments such as those typically conducted in the nonlinear ICA literature, will often make practical sense to switch to a flow-based estimator.
allow for an invertible transform.
C DISENTANGLEMENT METRICS
Several recent studies have brought to light shortcomings in a number of proposed disentanglement
metrics (Kim and Mnih 2018 Eastwood and Williams 2018 Chen et al. 2018 Higgins et al. 2018 Mathieu al. [2099). many which have been compiled the DisLib benchmark. In addition to the concerns they raise, it is important to note that none of the supervised metrics implemented in DisLib allow for continuous ground-truth factors, which is necessary for evaluating with the Natural Sprites and KITTI Masks datasets, as factors such as position and scale effectively continuous in reality. To rectify this issue without introducing novel metries, we include the Mean Correlation Coefficient (MCC) in our evaluations, using the implementation of Hyvärinen and Morioka (2016), which is described below.
We measure all metrics presented below between 10, 000 samples of latent factors and the cor- responding encoded means of our model (z)) We increase this sample size to 100, 000 for Modularity and MIG stabilize the entropy estimates.
24

---

Published as a conference paper at ICLR 2021
C.I MEAN CORRELATION COEFFICIENT
In addition to the DisLib metrics, we also compute the Mean Correlation Coefficient (MCC) in order perform quantitative evaluation with continuous variables. Because of Theorem T perfect disentanglement in the noiseless case should always lead to a correlation coefficient of 1 or -1, although note that we report 100 times the absolute value of the correlation coefficient. In our experiments, MCC is used without modification from the authors' open-sourced code (Morioka 2018). The method first measures correlation between the ground-truth factors and the encoded latent variables. The initial correlation matrix is then used to match each latent unit with a preferred ground- truth factor. This is an assignment problem that can be solved in polynomial time via the Munkres algorithm, described in the code release from Morioka 2018). After solving the assignment problem, the correlation coefficients are computed again for the vector of ground -tuut factors and the resulting permuted vector of latent encodings, where the output is matrix of correlation coefficients with D columns for each ground-truth factor and D' rows for each latent variable. We use the (absolute value of the) Spearman coefficient as our correlation measure which assumes monotonic relationship between the ground-truth factors and latent encodings but tolerates deviations from a strictly linear correspondence.
In the existing implementation for MCC, the ground truth factors, latent encodings, and mixed signal inputs are assumed have the same dimensionality, i.e. D = D' = N. However, in our case, the ground- -ruth generating factors are much lower dimensional than the signal, N < D. and the latent encoding is higher dimensional than the ground truth factors D' To resolve this discrepancy, we add D' D standard Gaussian noise channels to the ground-truth factors. To compute the MCC score, we take the mean of the absolute value of the upper diagonal of the correlation matrix. The upper diagonal is the diagonal of the square matrix of D ground-truth factors by the top D most correlated latent dimensions after sorting. In this way, we obtain an MCC estimate which averages only over the D correlation coefficients of the D ground truth factors with their corresponding best matching latent factors.
(see Appendix E for details).
C.2 DISLIB METRICS
BetaVAE. (Higgins et al. 2017)
The VAE metric uses biased estimator with tunable hyperparameters, although we follow the convention established in (Locatello et al.] 2018) of using the scikit-learn defaults. For sample in batch, pair of images, (X1, x2), is generated by fixing the value of one of the data generative factors while uniformly sampling the rest. The absolute value of the difference between the latent codes produced from the image pairs is then taken, Zdiff |z| zz). A logistic classifier is fit with batches of Zait variables and the corresponding index of the fixed ground-truth factor serves as the label. Once the classifier is trained, the metric itself is the mean classifier accuracy on a batch of held-out test data. The training minimizes the following loss:
1 W W Σ
n
L
log (og(exp(-yee y + c)) + 1),
(25)
where w and c are the learnable weight matrix and bias, respectively, and y is the index of the fixed ground-truth factor for the batch. The network is trained using the 1bfgs optimizer (Byrd al., 1993), which implemented via the seikit-learn Python package (Pedregosa et al., 2011) in the Disentanglement Library (DisLib, [ocatello et al. 2018). In the original work, the authors argue that their metric improves over a correlation metric such as the mean correlation coefficient by additionally measuring interpretabilitty. However, the linear operation of Z + can perform demixing, which means the measure gives no direct indication of identifiability and thus does not guarantee that the latent encodings are interpretable, especially in the case of dependent factors. Additionally, as noted by Kim and Mnih (2018), BetaVAE can report perfect accuracy when all but one of the ground-truth factors are disentangled, since the classifier can trivially attribute the remaining factor to the remaining latents.
FactorVAR (Kim and Mnih 2018)
For the FactorVAE metric, the variance of the latent encodings is computed for a large (10,000 in DisLib) batch of data where all factors could possibly be changing. Latent dimensions with variance
25

---

Published as a conference paper at ICLR 2021
below some threshold (0.05 in DisLib) rejected and not considered further. Next, the encoding variance is computed again on a smaller batch (64 in DisLib) of data where one factor is fixed during sampling. The quotient of these two quantities (with the larger batch variance the denominator) is then taken to obtain a normalized variance estimate per latent factor. Finally, a majority- is trained to predict the index of the ground-truth factor with the latent unit that has the lowest normalized variance. The Factor score is the classification accuracy for a batch of held-out data.
classifier
Mutual Information Gap (Chen al. 2018
The Mutual Information Gap (MIG) metric was introduced as an alternative to the classifier-based metrics. It provides normalized measure of the mean difference in mutual information between each ground truth factor and the two latent codes that have the highest mutual information with the given ground truth factor. As implemented in DisLib, MIG measures entropy by discretizing the model's 's latent code using histogram with 20 bins equally spaced between the representation minimum and maximum. truth values and the discretized latents using the scikit-learn met r rics. mutual Linff_n__ scrre function (Pedregosa et al. 2011). For the normalization it divides this difference by the entropy of the discretized ground truth factors.
then computes the discrete mutual information between the ground-
Modularity (Ridgeway and Mozer 2018)
Ridgeway and Mozer 2018) measure disentanglement in terms of three factors: modularity, com- pactness, and explicitness. For modularity, they first measure the mutual information between the discretized latents and ground-truth factors using the same histogram procedure that was used for the MIG, resulting in matrix, M e R D'xxD xD with entries for each mutual information pair. Their measure of modularity then
D'
Eド=1 MA max( max(M 1)
1 modularity = D Σe θ
(26)
where max(M3 returns the maximum of the vector of squared mutual information measurements between ground truth and each latent factor. Additionally, a selection function that returns zero for any i where max(M? = 0 and otherwise acts as the identity function.
DCI Disentanglement (Eastwood and Williams, 2018)
The DCI scores measure disentanglement, completeness, and informativeness. which have intu- itive correspondence to the modularity, compactness, and explicitness of (Ridgeway and Mozer 2018), respectively. To measure DCI Disentanglement, D regressors are trained to predict each ground truth factor state given the latent encoding. The DisL.b implementation uses the ensemble Gradient Boost www.ssinnnlasss asllsisssrr function from scikit-learn with default param- eters, which trains D gradient boosted logistic regression tree classifiers. Importance is assigned to each latent factor using the built-in feature watuuurr__mrrrreeeeeee Lmportance, property of the classifier, which computes the normalized total reduction of the classifier criterion loss contributed by each latent. Disentanglement is then measured
D(1-
(27)
1=1
where H is the entropy computed with the stats ent ropy function from scikit-learn, T E R DDD a matrix of the absolute value of the feature importance between each factor and each ground truth, the matrix
and / is a normalized version
Σ/ Lts
i
(28)
km 2 IN.
SAP Score (umaar et al. 2018)
To compute the SAP score, Kumar et al. (2018) first train linear support vector classifier with squared hinge loss and L2 penalty to predict each ground truth factor from each latent variable. In DisLib this is implemented with the svm. LinearsvC function with default parameters from scikit-learn. They construct score matrix S E RDDDD where each entry in the matrix is the
26

---

Published as a conference paper at ICLR 2021
DSprites
Shapes3D
MP13D
Cars3D
SmallNORB
5000
4030
4000
3000
4000
3030
3000
2000
2002
1000
1000
1000
1
2
3
Changed Factors
Changed Factors
Changed Factors
Changed Factors
Changed Factors
Figure 8: Number of changing factors in LAP dataset. For each dataset we sample 10,000 transitions and record the number changing factors. These are indicated the histograms. 1, see Appendix D
batch-mean classifier accuracy for predicting each ground truth given each individual latent encoding. For each generative factor, they compute the difference between the top two most predictive latent dimensions, which are the two highest scores given column of S. The mean (across ground-trum truth factors) of these differences is the SAP score.
D NATURAL DATASETS
We introduce several datasets to investigate disentanglement in more natural scenarios. Here, we provide an overview on the motivation and design of each dataset.
We have chosen to work with pairs of inputs as minimal sequences because we are interested in the first temporal derivative, more specifically in the sparsity of the transitions between pairs of images. Other methods that look at the second temporal derivative, such as work from Hénaff et al. (2019) on straightening, would require triplets as minimal sequences. Extending our approach beyond this minimal requirement would be simple in terms of the resulting ELBO (which would still factorise like in Eq. because of the Markov property). The only additional complexity would be in the data and loss handling.
An issue with evaluating disentanglement on natural datasets is the fact that the existing disentangle- ment metries require knowledge of the underlying generative process of the given data. Although can observe that the world is composed of distinct entities that vary according to rules imposed by physics, are unable determine the appropriate "factors" that generate such scenes. To mitigate this problem, we compile object measurements by calculating the x and y coordinates of the center of mass well the area of object masks in natural video frames. We use these measurements to a) inform new disentanglement benchmarks with natural transitions that have similar complexity to existing benchmarks (Natural Sprites) and b) evaluate the ability of algorithms decode intrinsic object properties (KITTI Masks). We additionally propose simple extension to the existing DisLib datasets in the form of collecting images into pairs that exhibit sparse (i.e. Laplace) transition probabilities.
D.1 UNIFORM TRANSITIONS (UNI)
The UNI extension based on the description given by Locatello et al. (2020), where the number of changing factors is determined using draws from uniform distribution. The key differences between our implementation and theirs is: (i) their code a randomly (with 50% probability) sets A even in the Rnd setting, and (ii) ensure that exactly k factors change. Though we consider these discrepancies minor, we nonetheless label all results reported directly from Locatello et al. (2020) with "I "LOC". as opposed to "UNI", for clarity.
27

---

Published as a conference paper at ICLR 2021
D.2 LAPLACE TRANSITIONS (LAP)
For each of the datasets in DisLib, we collect pairs of images. For each ground-truth factor, the first value in the pair is chosen from a uniform distribution across all possible values in latent space, while the second is chosen by weighting nearby values in latent space using Laplace distributed probabilities (see Eq. 2). We reject samples that would push a factor outside of the preset range provided by the dataset. We call this the LAP DisLib extension. Although the sparse prior indicates that any individual factor is more likely to remain constant, the number of factors that change in a given transition is still typically greater than one. To show this in Fig. X we sampled 10,000 transitions from each DisLib dataset with LAP transitions and computed the number of factors that had changed within pair. This extension of the DisLib datasets provides a bridge from i.i.d. data to natural data by explicitly modeling the observed sparse marginal transition distributions. When training models on the LAP dataset it is possible to reject samples without transitions (i.e. all factors remain constant) since the pair would not result in any temporal learning signal. However, it would arguably be more natural to leave these samples as they would more accurately reflect occurrences of stationary objects in real data. We report the rejection setting in the main text, but found no significant difference between the two settings (see Appendix G
This dataset also introduces hyper-parameted À that controls the rate of the Laplace sampling distribution, while the location is set by the initial factor value. Effectively, when this rate is X = 1 most of the factors change most of the time, whereas for rate of À 10 most of the factors will not change most of the time. Note that this means À (inversely) changes the scale, which results in larger or smaller movements, but does not affect the distribution itself. In other words, the sparsity is unchanged, as the sparsity controlled by the shape a. We fix 1, which yields multiple changes, thus making this dataset fundamentally different both in spirit and in practice, from the UNI dataset.
D.3 YOUTUBE-V VOS
For the YouTube dataset, we download annotations from the 2019 version of the video instance segmentation (Youtube-VIS) dataset (Yang et al. 2019 which is built on top of the video object segmentation (Youtube- VOS) dataset (Xu et al. [2018). The dataset has multi-object annotations for every five frames in a 30fps video, which results in 6fps sampling rate. The authors state that the temporal correlation between five consecutive frames is sufficiently strong that annotations can be omitted for intermediate frames to reduce the annotation efforts. Such a skip-frame annotation strategy enables scaling up the number of videos and objects annotated under the same budget, yielding 131,000 annotations for 2,883 videos, with 4,883 unique video object instances. Although we do not evaluate against YouTube VOS in this study, see it as the logical next step in transitioning to natural data. The large scale, lack of environmental constraints, and abundance of object types makes it the most challenging of the datasets considered herein.
The original image size of the YouTube ToS dataset is 720 x 1280. In order to preserve the statistics of the transitions, choose not to directly downsample to 64 x 64, but instead preserve the aspect ratio by downsampling to 64 x 128. In order to minimize the bias yielded by the extraction method, noting the center bias typically present in human videos, we extract three overlapping, equally spaced 64 x 64 pixel windows with a stride of 32. For cach resulting 64 x 64 x T sequence, where T denotes the number of time steps in the sequence, we filter out all pairs where the given object instance not present in adjacent frames, resulting in 234,652 pairs.
D.4 NATURAL SPRITES
The benchmark is available at http://///eeddoooooo/eeeeeeoeeee
http://eenoooooo
Without metric for disentanglement that can be applied to unknown data generating processes, we are limited to synthetic datasets with known ground-truth factors. Let us take dSprites (Matthey et al. 2017) as example. The dataset consists of all combinations of a set of latent factor values, namely,
Color: white
di
28

---

Published as a conference paper at ICLR 2021
Config
Scale
X
Y
(R. G. B)
Shape
Orientation
Continuous
(2375]
YT (197342)
YT (187112)
(1.0, 1.0, 1.0)
(square, triangle, star 4. spoke_ 4)
(0.9....44..551)
Discrete
YT [6]
YT [32)
YT (32) (1.0, 1.0, 1.0)
(square, triangle, star_4. spoke_4) (0.9....342.3511
Table 4: Natural Sprite Configs. Values in brackets refer to the number of unique values. Shapes presented are predefined in Spriteworld (Watters et al. (019)).
Shape: square, ellipse, heart
Scale: values linearly spaced in [0.5, 1]
Orientation: 40 values in [0, 2T]
Position X: 32 values in [0, 1]
Position Y: 32 values in [0, 1]
Given the limited set of discrete values each factor take on, all possible samples be described by a tractable dataset, compiled and released to the public. But, in reality, all of these factors should be continuous: spectrum of possible colors, shapes, scales, orientations, and positions exist. We address this by constructing a dataset that is augmented with natural and continuous ground truth factors, using the mask properties measured from the YouTube dataset described in Appendix D.3
We can choose the complexity of the dataset by discretizing the 234,652 transition pairs of position and scale into an arbitrary number of bins. In this study, we discretize match the number of possible object states as dSprites, which we present in Table 4 This helps isolate the effect of including natural transitions from the effect of increasing data complexity. We produce a pair by fixing the color, shape, and orientation, but updating the position and scale with transitions sampled from the YouTube measurements. We motivate fixing shape and color by noting that this is consistent with object permanence in the real world. We decided to fix the orientation because we do not currently have a way to approximate it from object masks and we did not want to introduce artificial transition probabilities. To minimize the effect of extreme outliers, frames if the mask area falls below the 5% or above the 95% quantiles, which reduces the number of pairs 207,794. Finally, we use the Spriteworld (Watters et al., 2019) renderer generate the images. Spriteworld allows us to render entirely new sprite objects at the precise position and scale as was measured from YouTube, For example, one would want apply YouTube- transitions to MPI3D (Gondal et al. 2019), this option is unavailable without the associated renderer.
filter out 10% of the data by removing
In relation the Laplace transitions described in section D.2 this update i) produces pairs that correspond to transitions observed in real data, ii) allows for smooth transitions by defining the data generation process as opposed being limited by the given collected dataset (e.g. dSprites), and iii) includes complex dependencies among factors that are present in natural data. We generate the data online, thus training the model to fit the underlying distribution as opposed to sampled finite dataset.
However, as noted previously, all supervised metrics aggregated in DisLib are inapplicable to continuous factors, which is problematic as the generating distribution is effectively continuous with respect to subset of the factors. Therefore, we limit our quantitative evaluation to MCC for continuous datasets. However, we are able to evaluate disentanglement with the standard metrics on the discretized version.
D.5 KITTI MOTS PEDESTRIAN MASKS (KITTI MASKS)
The benchmark is available at https:: zenodo. htt.:pps:/eeood...rrro wwwnddb.org/zecood/33333333
While Natural Sprites enables evaluation of disentanglement with natural transitions, note that any disentanglement framework that requires knowledge of the underlying generative factors is unrealistic for real- world data. Measurements such as scale and position correspond to object properties that ecologically relevant to the observer and can serve as suitable alternatives to the typical generative factors. We directly test this using our KITTI Masks dataset.
To create the dataset, we download annotations from the Multi-Object Tracking and Segmentation (MOTS) Evaluation Benchmark (Voigtlaender et al. 2019 Geiger et al. 2012 Milan al. 2016).
29

---

Published as a conference paper at ICLR 2021
575222 ar
57577 231
X3373 240
K.574 ar 256
Y 346
V.32.0
153
27 209
269
292
ar 393
450
497
a25
ar 505
ar
33.7
3333
X 34,22
34.0
34.7
X 33.6
X. 28.5
X330.2
X 32. y
x.33.6
x.55.2
X.37.9
X K.51.
X.5 52.2.0
X.52.
18.0
y 18.2
22.5
Y 26.77
y
30.8
y
31.55
y 32.0
V
31.3
y 33.7
x.0.6
1.1
1.7
2.8
X-4
6.9
B.2
X-9
30.7
12.1
X13..5 3.5
X 344.7
16.1
3..74
y 37.2
y 38.2 81
36.7
38
Y 38
JULD 89
97.99
Y 37
38.
38.3
30.1
Y 3.378
J0.3
y 39.9
Br
oF 53
OF B9
or IS
or 96
9M
BT
102
or 100
or
11.
A
K
10.5
35.4
36.2 2
36.0
$6.0
36.
36.3
35.8
36.5
$6.
y 3..74
38.3
30.1
y 1.39.4
99.4
4.400
ar 36
43
41
ar 42
ar 33
36
ar 66
ar 74
85
105
ar 137
ar 139
201
4
x1.0
1.7
X 3.8
5..1
6.77
X 8.3
10.2
11
1.3
18.1 y
220.8 & Y
27755
y
y.36.7
y
Y
y
y
113
127
144
Figure 9: KITTI Masks. Each row corresponds to sequential frames from random sequences in the KITTI Mssks dataset. Above each image denote measured object properties where xr y correspond the center of mass position and or corresponds to the area.
which is split into KITTI MOTS and MOTSChallenge) Both datasets contain sequences of pedestri- ans with their positions densely annotated in the time and pixel domains. For simplicity, we only consider the instance segmentation masks for pedestrians and do not the raw data.
The resulting KITTI Masks dataset consists 2, 120 sequences individual pedestrians with lengths between 2 and 710 frames each, resulting in total of 84,626 individual frames. As did with YouTube- VOS, we estimate ground truth factors by calculating the z and y coordinates of the center of mass of each pedestrian mask in each frame. We define the object size the area of the mask, i.e. the total number of pixels. We consider the disentanglement performance for different mean time gaps between image pairs in table and Appendix G.3 For samples and the corresponding ground truth factors see Fig.
The original KITTI image sizes are 1080 1920 or 480 x 640 resolution for MOTSChallenge and between 370 and 374 pixels tall by 1224 and 1242 pixels wide for KITTI MOTS. The frame rates of the videos vary from 14 30 fps, which can be seen in Table 2 of Milan al. (016)). We use nearest neighbor down-sampling for each frame such that the height was 64 pixels and the width is set to conserve the aspect ratio. After down-sampling, we use a horizontal sliding window approach to extract six equally spaced windows of size 64 x 64 (with overlap) for each sequence in both datasets. This results
x x sequence, where denotes the number of time steps Note that here make reasonable assumptions horizontal translation and scale invariance of the dataset. We justify the assumed scale invariance by observing that the data is collected from a camera mounted onto we performed an ablation study on the number of horizontal images. Instead of six horizontal, equally spaced sliding windows, we only do not observe significant changes in the reported data statistics (e.g. the kurtosis of the fit stays within =10% of the previous value for Az transitions). The values of Ay and Aarea do change significantly compared to Table
sequence.
which has varying distance pedestrians. To confirm the translation invariance,
two which leads to differently placed windows. We
For each resulting 64 64 T sequence, where T' denotes the number of time steps in the sequence, we extract all individual pedestrian masks based on their object instance identity and create new sequence for each pedestrian such that each resulting sequence only contains single pedestrian. We ignore images with masks that have less than 30 pixels as they are too far away occluded and were
httpp::/ httts://www.ree http://wwvviiii..... rwth aachen
30

---

Published as a conference paper at ICLR 2021
1
11
13
15
19
25
27
29 33
37 maxiAframes)
39
55 57
150
Figure KITTI Masks Át. Boxes indicate correspondence to physical time for different max( Aframes) in the KITTI Masks datasets. The orange line denotes the median and the green line the mean. The whiskers cover the 5th and 95th percentile of data.
not recognizable by the authors. We keep all sequences of two or more frames, as the algorithm only requires pairs of frames for training.
We leave the maximum distance between time frames within a pair, max (framess) as hyper- parameter. For given max max(arrmms)), we report the mean change in physical time in seconds (denoted by mean At)). We test adjacent frames (max(Aframes) = 1). which corresponds to a mean At 0.05) and max Aframes) = which corresponds to mean( At 0.15). This pro- cedure is motivated by the fact that different sequences were recorded with different frame rates and reporting the mean( (t) in seconds allows for physical interpretation. The relationship be- tween max | Aframes) and mean At) is in Fig. 10
We show results for testing additional values of
mean At) in Appendix G.3
During training, we augment the data by applying horizontal and vertical translations of +55 pixels and rotations of +2 degree. We apply the exact same data augmentation both images within pair to not change any transition statistics.
We note that both YouTube-VOL (Xu al. 2018 Yang al. 2019 and KITTI-MOTS VVoigllenneer et al. 2019 Geiger et al., 2012 Milan et al. 2016) are multi- datasets, although we consider each unique object (mask) separately. Multi-object representation learning and disentanglement are highly connected, in fact they have recently begun to be used interchangeably (Wulfmeier et al. 2020
To briefly comment possible extensions in this direction, we no reason why our prior would not be beneficial to multi-object methods such as MONet (Burgess et al. 2019) and IODINE (Greff et al. 2019), or video extensions such ViMON (Weis et al. 2020) and OP3 /Veerapaneni et al.
2019
E MODEL TRAINING AND SELECTION
We train models on all datasets provided DisLib with the UNI and LAP variants.
All models are implemented in PyTorch (Paszke et al. 2019). To facilitate comparison, the training parameters, e.g. optimizer, batch size, number of training steps, as well as the VAE encoder and decoder architecture are identical to those reported in (Locatello et al.) 2018 2020). We this architecture for all datasets, only adjusting the number input channels (greyscale for dSprites, smalINORB, and KITTI Masks; three color channels for all other datasets).
The model formulation is agnostic to the direction time. Therefore, to increase the temporal training signal at fixed computational cost for each batch of input pairs (xo, x1), optimize the model in both directions i.e. optimizing the model objective for both to = 0, t = as well as 0.
31

---

Published as a conference paper at ICLR 2021
F EXTENDED COMPARISONS AND CONTROLS
F.1 COMPARISON TO NONLINEAR ICA
F.1.1 THEORETICAL COMPARISON
Nonlinear ICA has recently been advanced significantly by several papers from Hyvärinen and colleagues. Of these studies, the two that are most comparable to our work is Hsvvrinne and Morioka (217)) which uses unsupervised contrastive loss for nonlinear demixing and Khemakhem et al. (2020a), which extends the nonlinear ICA framework to include variational autoencoders (VAEs). However, our theory covers an important class of transitions relevant for natural data that is not covered by the identifiability proofs of either of the aforementioned studies.
As specific comparison to the first paper, the non non-Gaussian autoregressive model that their identifia- bility proof rests upon (Eq. in HHvväinen and Morioka (017)) assumes that the second derivative of the innovation probability density function is less than zero satisfy uniform dependence, which is only met for a > for generalized Laplace transition distributions. While they denote (footnote 3) that Laplace distributions (ax 1) are not covered by their theory, they offer a suggestion for smooth approximation. However, they do not demonstrate that this approximation is useful in practice, or offer a solution to fit our data and for our model assumption as it allows for simple parameterization of fits to data (e.g. a 0.5 for natural movie transitions), but is simultaneously quite expressive (Sinz et al. 2009). Though we use a = 1 in practice for our estimation method, we prove identifiability up to permuta- tions and sign flips for any a < 2, covering all sparse distributions under the expressive generalized Laplacian model. In addition, we assume a Gaussian marginal distribution that allows us to derive fundamentally stronger proof of identifiability where we identify up to permutation and sign-flips. Hyvärinen and Morioka (2017) only identify the sources up to arbitrary non-linear element- transformations. Thus they require a subsequent step of ICA (under the typical assumption that at most one marginal source distribution is Gaussian) recover the signal up to permutations and sign flips for a class of distributions where it is unclear whether they account for temporal sparsity.
general class of sparse distributions for a S We chose a generalized Laplacian
The work of Khemakhem et al. (020a) has couple of differences from our own, most notable of which is the form of the conditional prior, p(ze 22-1). They assume that the conditional posterior part of the exponential family, which does not include Laplacian conditionals. Though the exponential family contains the Laplace distribution with fixed mean as its member, it does not allow their approach to model sparse transitions. They assume that the natural parameters of the exponential family distribution are conditioned on Zt-1. meaning that only the scale but not the mean of the Laplace prior for z, can be modulated by the previous time step, thus not allowing for sparse transition probabilities. Additionally, their implementation requires the number of classes (i.e. states of the conditioning variable) to equal the number of stationary segments, which is impractical for the datasets consider.
Thus, we provide a closer match to natural data transitions, with a stronger identifiability result. We provide validation by performing an extensive evaluation leveraging our contributed datasets as well as the models, metrics, and datasets provided by the Disentanglement Library (DisLib, discussed in section 4). We consider methods from the disentanglement literature (Locatello et al. (020) as well as nonlinear ICA (Hyvärinen and Morioka 2017), that are functionally capable of processing transitions.
F.2 EMPIRICAL COMPARISON
Hyvärinen and Morioka 2017) conducted a simulation where the sources in the nonlinear ICA model come from linear autoregressive (AR) model with non-Gaussian innovations. Specifi- cally, temporally dependent 20-dimensional source signals were randomly generated according to log ((s(t)ss((t 1)) - Is(t) 0.7s(t 1)]. Though this generative process was noted to not be covered by the theory presented in (Hyvärinen and Morioka, 2017), the authors demonstrated that PCL could reconstruct the source signals reasonably well even for the nonlinear mixture case. Given our practical use a Laplacian conditional, we found it a valuable comparison to evaluate our theory
in this artificial setting.
32

---

Published as a conference paper at ICLR 2021
Method
I==1
1,=2
L=33
1,=4
L=5
PCL
0.998
0.960
0.950
0.917
0.902
PCL (NF)
0.946
0.918
0.918
0.917
0.876
SlowFlow
0.997
0.987
0.982
0.975
0.975
Table 5: MCC using linear correlation where denotes the number of mixing layers.
Given the discussion in Appendix B we use SlowFlow for these experiments. For computational tractability in demixing highly nonlinear transformations, we consider normalizing flows (Dinh et al. 2017a b, Kingma and Dhariwal 2018), namely volume-preserving flows (Sorrenson et al. 2017), 7). as we find constraining the Jacobian determinant stabilizes learning. To ensure sufficient expressivity, we consider 6 coupling blocks, each containing a 2-layer MLP with 500 hidden units and ReLU nonlinearities. We compare to the PCL implementation presented in (Hyvärinen and Morioka 2017). where an MLP with the same number of hidden layers as the mixing MLP was adopted. We use 100 hidden units as we did not find increasing the value improved performance. To account for the architectural difference serving as a possible confounder, we use the same normalizing flow encoder for optimizing the PCL objective, which we term "PCL (NF)".
While (Hyvärinen and Morioka 2017 used leaky ReLU nonlinearities to make the mixing invertible, involves gradient optimiza- tion of the Jacobian term, and more importantly, unlike PCL, aims to explicitly recover the mixing process. We thus use a smooth version of the leaky-ReLU activation function with hyperparameter (Gresele et al. 2020).
said mixing is non-differentiabll. This is problematic for SlowFlow,
sL()) = ar + (1 a)) log + t*).
(29)
By ensuring the mixing process is smooth, we find that SlowFlow performs favorably relative to PCL (Table 5) when evaluated in the same setting, converging to a better optimum at higher levels of mixing.
F.2 JOINT FACTOR DEPENDENCE EVALUATION
In order consider joint dependencies among natural generative factors, we leverage Natural Sprites to construct modified datasets where time-pairs of factors are shuffled per-factor (e.g. combining the x transition from one clip with the y transition from different clip). This destroys dependencies between the factors, while maintaining the sparse marginal distributions. In Fig. [ (right,, we show 2D marginals before (blue) and after (orange) this shuffling. The additional density on the diagonals in the unshuffled data reveals dependencies between pairs of factors on both datasets. As mentioned in section 3.4 the observed dependency is mismatched from the theoretical assumptions of our model.
We test how robust Slow is to such mismatch by training it the permuted data and re- evaluating disentanglement. In Table 22, we highlight that the improvement of Slow VAE on the permuted (i.e. independent) continuous Natural Sprites is not significant. In Table 21 we surprisingly find an overall improved score with non- non- permuted transitions (i.e. with dependencies). with three out of seven metrics showing significant improvement. This is in line with Fig. 1 in Khemakhem et al. (2020b), where, at least for simple mixing, a model (Khemakhem et al., 2020a) that does not account for dependencies performs well one that does (Khemakhem et al. (020b). We conclude that these preliminary results do not support the hypothesis that SloW disentanglement is reliant upon the model assumption that the factors are independent, but do acknowledge that the empirical effect of statistical dependence in natural video warrants further exploration ('räubbe et al. 2020 Yang et al. 2020).
F.3 TRANSITION PRIOR ABLATION
We consider an ablated model which minimizes a KL-divergence term between the posteriors at time-step and time-step f-1 1. This encourages the model to match the posteriors of both time points as closely as possible, and resembles a probabilistic variant of Slow Feature Analysis (Turner and Sahani 2007). Specifically, set p(z, 24-1) q(z+_1xx(1). replacing the Laplace prior with the
33

---

Published as a conference paper at ICLR 2021
posterior of the previous time step. This is equivalent Gaussian (a = transition prior, where the mean and variance are specified by the previous time step. We ablate over the regularization parameter and provide results in Tables [4 and [3] although note that still use the same hyperparameter values for Slow VAE as in all other experiments. As predicted by our theoretical result, leads to entangled representations in aggregate across evaluated datasets and metrics, even when considering spectrum of y values, resulting in a drastic reduction in scores, particularly on dSprites and Natural Sprites.
G ADDITIONAL RESULTS
G.1 EXTENDED DATA ANALYSIS
KOTT-MOTS
1.0
1.0
1.0
10
-10
-10
0.0
0.0
0.0
-2
2
2
0
-10
10
-10
10
-10
10
YeuTube-vos
1.0
10
arro
area
0.5
0.5
-10
-10
-10
0.0
0.0
0.0
-2
2
-2
0
10
10
-1
30
-10
10
4
Figure 11: Statistics of Natural Transitions. Left) Distribution over transitions for horizontal (Arr and vertical (Ay) position as well mask/object size (Aarea) for both datasets, Orange lines indicate fits of generalized Laplace distributions (Eq. 2). Right) 2D marginal distribution over pairs of factor transitions (blue) and permuted pairs (orange) that indicate the marginal distributions when made independent.
dataset
N
A area
A
A y
KITTI-MOTS
82506
0.45
0.59
0.69
YouTube- VOS
234652
0.44
0.52
0.55
Table 6: Shape parameters (a) of the fitted generalized Laplace distributions in Fig. 11
We report the empirical estimates of Kurtosis in Table 7 We report the log- ligelikelihood scores for the area, A x, A y statistics Tables 8 and [0 respectively for a Normal, a Laplace and a generalized Laplace/Normal distribution. For these distributions,
also report the fit parameters for the A area, A x, A y statistics in Tables [1] [2] and [3] respectively. where the shape parameter of the generalized Laplacian in bold face. As a higher likelihood indicates a better fit, we can see further evidence that natural transitions are highly leptokurtic; a Laplace distribution (a 1) is better fit than a Gaussian (a = 2), while the generalized Laplacian yields the highest likelihood consistently with a z 0.5 for all measurements, as indicated in the main paper. For the plots in Figs. [and II we the standard deviation of each component to 1 and clipped the minimum (-5) and maximum (5) values.
We note that while the marginal transitions appear sparse metries computed from the given object masks, our analysis considers 2D projections of objects instead of the transition statistics in their 3D environment. Understanding the relationship between 3D and 2D transition statistics is compelling question from broader perspective of visual processing, but unfortunately, the KITTI-MOTS masks /Voigtlaender et al. 2019: Geiger et al.) 2012 Milan et al. 2016) lack the associated depth statistics compute relevant, given
data required answer it. Nonetheless, the natural that most computer vision models and vision- based animals see the 3D world as projected onto their 2D receptor arrays.
34

---

Published as a conference paper at ICLR 2021
dataset
N
area
A y
KITTI
82506
68.92
38.50
65.39
YouTube
234652
76.49
39.98
35.59
Table Empirical estimates of Kurtosis for mask transitions per metric for each dataset.
dataset
genlaplace
normal
laplace
KITTI 82506
-3.21e+05
-3.79e+05
-3.35e+05
YouTube
234652
-1.29c+06
-1.45c+06
-1.33c+06
Table 8: Maximum likelihood scores for the considered distributions on area for each dataset.
dataset
N
genlaplace
normal
laplace
KITTI 82506
-8.72e+04
-1.20e+05
-9.25e+04
YouTube
234652
-4.50c+05
-5.64c+05
-4.74c+05
Table 9: Maximum likelihood scores for the considered distributions on Ar for each dataset.
dataset
genlaplace
normal
laplace
KITTI
82506
-7.59e+04
-1.07e+05
-7.86e+04
YouTube
234652
-4.40e+05
-5,45e+05
-4.60e+05
Table 10: Maximum likelihood scores for the considered distributions Ay for each dataset.
dataset N
genlaplace
normal
laplace
KITTI
82506
[4.55e-01, 1.00e+00, 1.01e+00]
[4.53e001, 2.39++01]
[1.00c+00, 1.07c+01]
YouTube 234652
[4.44e-01, 1.47e-16, 5.04c+00]
[2.25c-01, 1.16c+02]
[7.73c-09, 5.28c+01]
Table 11: Parameter fits for the considered distributions on A area for each dataset. The parameters are (alpha, location, scale) for generalized Laplace/Normal, (location, scale) for the other two distributions.
dataset N
genlaplace
normal
laplace
KITTI 82506
[5.87e-01, 4.76e-02, 1.69e-01] [5.34e-02, 1.04e+00)
[5.49e-02, 5.64e-01]
YouTube 234652
[5.15e-01, 2.57e--01 [2.32e-03, 2.68e+00)
[7.54e-09, 1.38e+00)
Table 12: Parameter fits for the considered distributions on A x for each dataset. The parameters (alpha, location, scale) for generalized Laplace/Normal, (location, scale) for the other two distributions.
dataset
N
genlaplace
normal
laplace
KITTI
82506
[6.94e-01, 1.02e-02, 2.32e-01] [3.84e-02, 8.86e-01] [1.7le-02, 4.77e-001
YouTube
234652
[5.48e-01, 2.93e113, 3.08e-01]
[8.81e-03, 2.47e+00)
[9.15e-04, 1.30e+00)
Table 13: Parameter fits for the considered distributions A for each dataset. The parameters are (alpha, location, scale) for generalized Laplace/Normal, (location, scale) for the other two distributions.
35

---

Published as a conference paper at ICLR 2021
28
1
4
4
R
Figure 12: KITTI Masks Latent Representations. We show axis latent traversals along each dimension for the 3-VAE (top) and VAOw (bottom). Here, the latents z; are sorted from top to bottom in ascending order according to the mean variance output of the encoder. With MCC correlation (see e.g. Fig. 20) the known ground truth factors are matched as following: 3-VAE: scale~ 22, x-position
and y-position~ z3; SlowVAE scale~ 20, x-position~ 2 and position 23. With these latent visualizations alone, there no significant difference visible between quantitative difference with the MCC score (see Table
3-VAE and SlowVAE VAE. However, and qualitative difference when directly observing latent embeddings (see Fig. 200)
36

---

Published as a conference paper at ICLR 2021
35
30
85
80
-3
Figure 13: Ablation over mean( At) for Slow VAE. Mean and standard deviation (s.d.) MCC scores
Model
Data
BetaVAE
FactorVAE
MIG
MCC
DC1
Modularity
SAP
SIO
dSprites (Laplace)
100.0 (0.0)
(3.0)
(9.5))
69.8 (2.3)
(3.6)
96.6(
8.1 (3.0)
PM-VAE (16)
dSprites (Laplace)
64.1 (7.0)
44.8 (13.0)
(.2))
(5.5)
(3,))
93.5(
1,7 (0.8)
PM AM-VAR (10)
dSprites (1 aplac)
(7.5)
(91.2)
(.9(188)
49.2
(5.6)
(2.7
(1.7)
PM-VAE (8)
dSprites (Laplace)
82.9 (2.8)
(5.7
(2.
(3.6
(3.5
91.6
41.
PM MA-VAB (4)
dSprites (Laplace)
86.6
64.1 (7.2)
(5.6
(2.0
22.9
90.9
(.7)
PM-VAE (2)
dSprites (Laplace)
86.3 (2.4)
(7.7
(3.9
(3.5)
21.2
92.3
$.5
PM-VAE (1)
dSprites (Laplace)
(5.4)
58.4
(3.6))))
45.9 (4.9)
(5.4)
92.1
4.0
VAE
Natural (Discrete)
(2.6
(4.8)
(5.0
(4.6
(5.5)
88.1 (3.6)
4..3)
PM-VAB (16)
Natural (Discrete)
72.7
(3.7)
(..()
(3.2)
6.9
(1.8
1.27
PM-VAE (10)
Natural (Discrete)
73.6
$2.0
3,8
33.9
7,3
87.0
2.0
(8)
(Discrete)
(3.6
(4.3
(1.
(3.2
7.1
81.8
1.6
PM PM-VAB (4)
Natural (Discrete)
73.8
(5.3)
35.7 (1.5)
6.7
(2.2)
1.6
PM-VAE (2)
Natural (Discrete) (3.1)
(5.3)
2.2(1.1)
3688 (2.4)
6.25
87.4 (1.9)
(.16
PM-VAB (1)
Natural (Discrete)
(3.3
49.7 (5.4)
(1.)
(3.2
(1.8)
86.9 (2.2)
1.8
Table 14: Mean and standard deviation (s.d.) metric scores across random seeds. PM-VAE (7)
refers to replacing the Laplace prior with a KL--divergence term between the (Gaussian) posteriors at time-step and time-step / 1, with conditional prior regularization, 7-
G.2 ALL DISLIB RESULTS
We include results on all DisLib datasets, dSprites (Matthey et al. 2017), Cars3D RRed et 2015). SmalINORB (LeCun et al. 2004), Shapes3D (Kim and Mnih 2018). 8), MPI3D (Gondal et al. 2211), in Tables [6 17
18 19 and 20 respectively. We report both median (a.d.) compare the previous median scores reported in (Locatello et al.) 2020), as well as the the more common mean (s.d.) scores for future comparisons and straightforward statistical estimates significant differences between models. We also consider allowing for static transitions, which denote with "NC". e.g. "LAP-NC", in the tabular results. As mentioned Section we use the same parameter settings for Slow VAE in all experiments, while model selection performed not only per dataset, but per seed, for results from (Locatello et al. 2020).
G.3 KITTI MASKS At ABLATION
As seen in the main text, considering image pairs separated further apart in time appears bene- ficial. Here we evaluate a wider range by taking frames which are further apart in a sequence. max max(Aframes) = N indicates that all pairs differ by at most N frames. We chose upper bound of N, rather than sampling pairs with fixed separation,
account for the variable frame rates and sequence lengths in the original dataset (Milan et al., (016)) without introducing confounding factor of varying dataset size. We report in Fig. 10 how the max (frames) criterion corresponds to the mean time gap between image pairs (mean At)) in seconds. For further details, we refer to Appendix D.5]
In Fig, 13 we visualize ablation mean At). We find that model performance increased initially with larger temporal separation between data points, then plateaued. We also observe Fig. [4] that the measured factor marginals remain sparse, with 1, for all tested settings of mean( At).
37

---

Published as a conference paper at ICLR 2021
Model
Data
MCC
Slow VAB
Natural (Continuous)
49.1 (4.0)
PM-VAE (16)
Natural (Continuous)
35.2 (3.7)
PM-VAE (10)
Natural (Continuous)
33.2 (2.1)
PM-VAE (8)
Natural (Continuous)
32.7 (3.1)
PM-VAE (4)
Natural (Continuous)
33.7 (2.3)
PM-VAE (2)
Natural (Continuous)
32.4
PM-VAE (1)
Natural (Continuous)
34.2 (3.4)
VAB
Kitti (mean At)
-0.05s)
66.1 (4.5)
PM-VAE (16)
Kitti (mean At) -0.05s)
63.1 (9.3)
PM-VAE (10)
Kitti (mean( Ar) = 0.055)
57.4 (8.5)
PM-VAE (8)
Kitti (mean( Ar) 0..55))
(9.66
PM-VAE (4)
Kitti (mean( At) 0.05s)
51.8 (92)
PM-VAE (2)
Kitti (mean( At) - 0.05s)
50.3 (7.4)
PM-VAE (1)
Kitti (mean( At) = 0.05s)
38.4 (6.8)
Slow VAB
Kitti (mean( A1) 0.15s)
79.6
PM-VAE (16)
Kitti (mean( A/ = 0.158)
(5.9
PM-VAE (10)
Kitti (mean At -..55s)
78.2 (6.0)
VAE (8)
Kitti (mean At) = -0.15s)
73.8 (10.0)
VA-VEE (4)
Kitti (mean( Ar) -0.158)
67.9 (10.4)
MM-VAE (2)
Kitti (mean At
= 0..15))
60.7 (8.8)
PM-VAE (1)
Kitti (mean( At) 0.158)
60.9 (9.1)
Table 15: Continuous ground-truth variable datasets. See Table
14 for details.
Model (Data)
BetaVAE
VAC
MIG
DCI
Modularity
SAP
3-VAE (i.i..)
82.3
66.0
10.2
18.6
82.2
4.9
Ada-ML-VAE (LOC)
89.6
70.1
11.5
29.4
89.7
3.6
Ada-GVAN (LOC)
92.3
84.7
26.6
47.9
91.3
7.4
SlowVAE (UND)
89.7 (3.8)
(8.4)
34.5 (9.6)
50.0 (6.9)
87.1 (2.0)
5.1 (1.5)
Slow VAE (LAP)
100.0 (0.0)
(2.3)
28.2)
65.5 (3.1)
96.8 (1.4)
6,0 (2.4)
Slow VAE (LAP-NC)
100.0 (0.2)
97.4 (4.4)
29.1 (7.1)
62.0 (4.2)
97.4 (1.6)
82.9)
Slow VAE (UND)
(510)
(11.1)
28.3 (11.5)
47.7 (8.5)
86.9 (2.8)
4.4 (2.0)
Slow VAE (LAP)
100.0 (0.0)
97.0)
29.5 (9.3)
65.4 (3.6)
96.5 (1.6)
8.1 (3.0)
Slow VAE (LAP-NC)
99.8 (0.6) (6.0)
27.6 (8.6)
61.5 (5.3)
96.8 (1.8)
8.4 (3.4)
Table 16: dSprites. Median and absolute deviation (a.d.) metric scores across 10 random seeds (first three rows are from (Locatello et al.) (2200)). The bottom three rows give mean and standard deviation (s.d.) for the models presented in this paper.
Increasing mean( At) leads increased diversity, and thus more information in the learning signal. However, it is worth noting that since Slow VAE assumes = 1 in the transitions, an increase in from increasing the temporal gap leads reduction in mismatch.
Our results on increasing the temporal difference within pairs of inputs is in agreement with recent work by Oord al. (2018) Table 2), who show increased performance in representation learning for larger separation between positive samples in contrastive objective function. Additional related work from TTscaanenn al. (2019) shows that temporal separation between frame embeddings influences the representation that is learned from videos.
G.4 LATENT SPACE VISUALIZATIONS
We visualize differences in learned latent representations using image embedding in Figures 15 28 We show four different plots for each dataset considered and include all available models. Each figure corresponds to different dataset.
38

---

Published as a conference paper at ICLR 2021
Model (Data)
BetaVAE
FactorVAE
MIG
DCI
Modularity
SAP
3-VAE (i.i.d)
100.0
87.9
8.8
22.5
90.2
1.0
Ada-ML-VAB (LOC)
100.0
87.4
14.7
45.6
94.6
2.8
Ada-GVAE (LOC)
100.0
90.2
15.0
54.0
93.9
9,4
Slow VAE (UNI)
100.0
90.4 (0.4)
15.7 (1.5)
48.9 (1.7)
95.7 (1.0)
1.6 (0.4)
Slow (LAP)
(0.00
(2.5)
(1.1
(2.2
94.4 (1.1)
1.7 (0.9)
Slow VAE (LAP-NC)
(0.00
90.8 (1.1)
9.31
50.0 (2.0)
94.6 (0.9)
0.9 (0.9)
Slow VAE (UNT)
(0.00
(0.5)
(2.2)
(2.4)
95.4 (1.5)
1.6 (0.5)
Slow VAE (LAP)
(0000
03.55
10.4 (1.8)
50.9 (2.7)
94.1 (1.2)
2.0 (1.1)
Slow VAE (LAP-NC)
10.00
(1.2(
9.5 (1.4)
50.2 (2.7)
95.0 (1.2)
1.7 (1.4)
Table 17: Cars3D. Median and absolute deviation (a.d.) metric across 10 random seeds (first three rows are from (Locatello et al.) 2020)). The bottom three rows give mean and standard deviation (s.d.) for the models presented in this paper.
Model (Data)
BetaVAE
FactorVAE
MIG
DCI
Modularity
SAP
3-VAE (i..d.)
74.0
49.5
21.4
28.0
89.5
9.8
Ada-ML-VAR (LOC)
91.0
72.1
31.1
34.1
86.
15.3
Ada-GVAE (LOC)
87.9
55.5
25.6
33.8
78.8
10.6
VAO (UND)
78.8 (2.1)
46.2 (1.9)
23.7 (1.3)
28.8
92.1 (1.6)
7.8 (1.0)
Slow VAE (LAP)
(0.2)
72.9 (0.7)
25.8 (0.5)
42.7
97.7 (0.3)
6.5 (0.4)
Slow VAE (LAP-NC)
86.1 (0.7)
73.7 (0.6)
26.3 (0.5)
42.5
97.6 (0.3)
6.5 (0.9)
Slow VAE (UNT)
(3.8)
(2.9)
23.8 (1.8)
28.7
90.9 (2.1)
7.8 (1.1)
Slow VAE (LAP)
(0.3)
73.1 (0.9)
25.7 (0.6)
(2.9
97.5 (0.3)
6.8 (0.5)
Slow VAE (LAP-NC)
85.7 (1.0)
73.3 (0.8)
26.2 (0.7)
42.6 (0.8)
97.6 (0.5)
6.6 (1.3)
Table 18: SmalINORB. Median and absolute deviation (a.d.) metric scores across 10 random seeds (first three rows are from (Locatello et al. 2020)). The bottom three rows give mean and standard deviation (s.d.) for the models presented in this paper.
In Figures 15 21 we display the mean correlation coefficient matrix and the latent representations for each ground-truth as described in the main text for Fig.
The top row is the sorted absolute correlation coefficient matrix between the latents (rows) and the ground truth generating factors (columns). The latent dimensions are permuted such that the sum on the diagonal maximal. This achieved by an optimal, non-greedy matching process for each ground truth factor with its corresponding latent, described in appendix C As such, a more
A
0.45
alpta
0.550
0.525
0.35
0.60
0.500
0.455
0.30
Ls
mean(tt) [s)
Ay
Aarea
Ay
Aarea
-75
25
-25
0
25
-500
500
-25
25
-25
D
25
-500
500
Figure 14: KITTI Masks Sparseness. We show the sparseness over time of the transitions for horizontal (Az), vertical (Ay) as well as mask/object size (Aarea) in KITTI Masks by plotting the of generalized Laplace fit for different mean( At) (top). To display the quality of the fits we show two exemplary fits mean( At) 0.63 (bottom-left) and At)) 1.02 (bottom-right).
39

---

Published as a conference paper at ICLR 2021
Model (Data)
BetaVAE
FactorVAB
MIG
DC1
Modularity
SAP
3-VAE (i.i.d)
98.6
83,9
22.0
58.8
93.8
6.2
Ada-ML-VAE (LOC)
100.0
100.0
50.9
94.0
98.8
12.7
Ada-GVAE (LOC)
100.0
100.0
56.2
94.6
97.5
15.3
Slow VAE (UNI)
100.0 (0.1)
97.3 (4.0)
(8.4)
82.6 (4.4)
95.5 (1.6)
5.8 (0.9)
Slow (LAP)
100.0
(2.6)
62.5 (3.1)
(4.0)
98.1 (0.6)
8.2 (1.7)
Slow VAE (LAP-NC)
(1.66
97.0
(5.4)
86.7 (4.1)
98.4 (1.4)
(2.1)
Slow VAE (UNT)
(0.3)
95.4 (5.2)
58.8 (13.0)
82.3 (5.4)
95.2 (2.0)
(1.4
Slow VAE (LAP)
(0.00
(3.2)
61.5 (4.5)
85.0 (4.7)
98.3 (0.8)
8.9 (2.6)
Slow VAE (LAP-NC)
98.4 (4.9)
(2.4)
61.6 (10.6)
86.1 (5.2)
98.2 (1.6)
8.2 (2.6)
Table 19: Shapes3D. Median and absolute deviation (a.d.) metric scores across 10 random seeds (first three rows are from (Locatello et al.) 2020)). The bottom three rows give mean and standard deviation (s.d.) for the models presented this paper.
Model (Data)
BetaVAE
FactorVAE
MIG
DCI
Modularity
SAP
8-VAE (i.i.d))
54.6
32.2
7.2
19.5
87.4
3.7
VAL- (LOC)
72.6
47.6
24.1
28.5
87.5
7.4
Ada-GVAB (LOC)
78.9
62.1
28.4
40.1
91.6
21.5
Slow VAE (UNI)
(0.9)
38.6 (2.3)
32.2 (1.0)
29.9 (1.3)
89.2 (2.0)
8.8 (0.8)
Slow (LAP)
67.6
42.4 (6.1)
32.0 (1.8)
35.9 (2.2)
89.5 (1.5)
9.7 (0.8)
SlowVAB (LAP-NC)
(2.7)
39.7)
30.6 (0.7)
(0.7)
85.9 (1.1)
9.3 (0.9)
Slow VAE (UNT)
51.6
38.5 (3.2)
32.2 (1.2)
89.4 31.6
8.7 (1.0)
Slow VAE (LAP)
(6.9)
45.5 (8.3)
32.9 (2.6)
35.5 (2.7)
89.2 (1.9)
9.7 (1.2)
VAE (LAP-NC)
(3.6)
40.3 (2.5)
30.4 (0.8)
(1.0)
86.6 (1.7)
9.3 (1.0)
Table 20: MPI3D. Median and absolute deviation (a.d.) metric scores across 10 random seeds (first
three from (Locatello al.) (2200). The bottom three rows give mean and standard deviation (s.d.) for comparison with other tables.
prevalent diagonal structure corresponds to a better mapping between the ground-trute truuh factors and latent encoding.
The middle set of plots are latent embeddings of random training data samples. The x-axis denotes the ground truth generating factor and the y-axis denotes the corresponding latent factor matched according to the main diagonal of the correlation matrix. For each dataset, we further color-code the latents by categorical variable denoted in each figure.
The bottom set of plots show the ground truth encoding compared to the second best latent as opposed to the diagonally matched latent. This plot be used to judge how much the correspondence between latents is one-to-one or rather one-to-many.
To further investigate the latent representations, we show scatter plot over the best and second best latents in figures 28 Here, the color-coding is matched by the ground truth factor denoted in each row.
When comparing the correlation matrix with the corresponding scatter plots, one can see that embeddings with sinusoidal curves have low correlation, which illustrates shortcoming of the metric. Another limitation is that categorical variables which have no natural ordering have an order-dependent MCC score, indicating the permutation variance of MCC. With VAE, we can infer three different types of embeddings. First, we have simple ordered ground truth factors with non- circular boundary conditions, Here, VAE models often show clear one-to-one correspondence
Model
Data
Permuted?
BetaVAE
FactorVAB
MIG
DC1
Modularity
SAP
Slow VAE
10
6
Natural (Discrete)
Yes
77.6 (4.1)
69.7 (6.5)
8.5 (4.4)
49.9 (3.5)
17.6 (2.8)
89.8 (3.2)
1.8 (0.9)
Slow
(Discrete)
(2.2)
(4.8)
(4.6
(5.5)
(3.6)
Table 21: Impact of removing natural dependence on Discrete Natural Sprites.
40

---

Published as a conference paper at ICLR 2021
Model
7
Data
Permated!'
MCC
Slow
10
Natural (Continuos)
Yes
52.9 (4.2)
SlowVAE
Natural (Continuous)
49.1 (4.0)
Table 22: Impact of removing natural dependence Continuous Natural Sprites.
(e.g. Fig 22 scale, X-position and y-position; Fig 25 O-rotation; Fig 26 -rroatiion). Second, we observe circular embeddings due to boundary conditions for certain factors (c.g. Fig [5 22 3rd row;
Fig 16
23 2nd row). Note that not all datasets with orientations exhibit full rotations and thus do not have circular boundary conditions, e.g. smalINORB. Finally, we have categorical variables, where no
order exists (e.g. Fig- 16 top row, Fig 17 24 top row, Fig 18 25 top row) resulting in separated but not necessarily ordered clusters.
41

---

Published as a conference paper at ICLR 2021
AnnealedVAE
P-TCAAE
B-VAE
DIP-VAE-I 30.7
DIP-VAE-II
Ada-GVAE (LAP)
PCL (LAP)
SlowVAE (LAP)
MCC = 60.6
MCC 65.7
MCC 611
MCC 52.4
MCC 60.8
MCC 67.3
MCC 71.9
GT Factors
GT Shape
GT Scale
GT x Pos
20
GT y-Pos
32
81
Shape
GT Scale
d-Rot
GT y Pos
20
Figure 15: DSprites Latent Representations. Top, MCC correlation matrices. Middle five rows. model latent over highest correlating ground truth factor. Bottom five rows, model latent over second highest correlating ground truth factor. The color-coding corresponds to the shapes: heart/yellow, ellipse/turquoise and square/purple.
42

---

Published as a conference paper at ICLR 2021
AnnealedVAE
B-TCVAE
A-VAE
DIP-VAE-I 25
DIP-VAE-II 24
FactorVAE
SlowVAE (UNI) SloWVAE (LAP)
56
43
59
21
69
72
61
24
51
15
65
52
68
7B
10
10
26
20
22
10
11
20
38
12
35
42
GT Pactors
GT d-Rot
10
20
GT Type
100
GT d-Rot
Type
100
Figure 16: Cars3D Latent Representations. Top, MCC correlation matrices. Middle three rows, model latent over highest correlating ground truth factor. Bottom three rows, model latent over second highest correlating ground truth factor. The color-coding corresponds the 183 different types (GT Types) in the dataset.
43

---

Published as a conference paper at ICLR 2021
AnnealedVAE
A-TCVAE
A-VAE
DIP-VAE-I
DIP-VAE-II
FactorVAE
SloWVAE (UND)
SioWVAE (LAP)
56
11
51
60
2
15
52
14
12
73
2
21
20
12
11
30
27
35
17
53
11
35
26
51
27
45
33
14
1
10
22
14
14
30
15
14
27
26
1
28
14
14
29
20
15
11
12
16
12
12
12
22
10
17
52
15
20
10
27
32
14
GT Category
GT Elevation
GT ror
23
Category
GT Elevation
GT Rot
rot
Figure 17: SmalINorb Latent Representations. Top, MCC correlation matrices. Middle four rows, model latent over highest correlating ground truth factor. Bottom four rows, model latent over second highest correlating ground truth factor. The color-coding corresponds the five different GT categories the dataset.
44

---

Published as a conference paper at ICLR 2021
SloWVAE (UNI)
SlowVAE (LAP)
MCC = 71.2
MCC 69.6
67
76
1
97
100
23 66
44
Factors
GT Floor Col
Wall Col
GT Wall Col
GT Obj Col
[6] Col
GT Size
GT Typ
GT Typ
4
20
GT 6-Rot
10
Figure 18: Shapes3D Latent Representations. Top, MCC correlation matrices. Left two columns, model latent over highest correlating ground truth factor. Right two columns, model latent over second highest correlating ground truth factor. The color-coding corresponds to the four different object types (GT-Type) in the dataset.
45

---

Published as a conference paper at ICLR 2021
SlowVAE
SloWVAE (LAP)
MCC = 27.0
MCC 54,6
24
34
13
4
25
4
1
16
1411
11
14
94
6:12
26
25
22
17
1672
1
13
11
10
4
6
GT Color
GT Color
Shape
2
GT Shape
22
GT Size
0.0
0.5
1.0
Height
Height
GT Bg Col
Bg
GT @-Rot
20
40
Root
GT B-Rot
20
40
B-Rot
20
Figure 19: MPI3DReal Latent Representations. Top, MCC correlation matrices. Left two columns, model latent over highest correlating ground truth factor. Right two columns, model latent over second highest correlating ground truth factor. The color-coding corresponds to the six different object shapes (GT Shape) the dataset.
46

---

Published as a conference paper at ICLR 2021
B-VAE
Ada-GVAE
PCL
SIOWVAE
74.8
MCC 76.0
MCC 62.4
MCC 90.6
0
35
77
26
5
95
2
1
59
9
18
65
25
30
69
1
10
81
79
18
86
13
52
5
95
Latents
4
7
7
26
28
9
16
11
5
3
6
5
3
31
4
25
7
28
11
7
10
8-
17
0
49
28
11
15
12
7
17
2
Factors
1
GT y-Poiition
20
40
GT x- Position
50
4
1000
GT y-ssitioo
GT Position
50
GT Area
1000
Figure 20: KITTI Masks Latent Representations. Top, MCC correlation matrices. Middle three rows, model latent over highest correlating ground truth factor. Bottom three rows, model latent over second highest correlating ground truth factor.
47

---

Published as a conference paper at ICLR 2021
A-VAE (C)
Ada-GVAE (C)
SlowVAE (C)
A-VAE (D)
Ada-GVAE
PCL
SlowVAE
57.4
62.3
67
13
12
74
10
10
17
10
16
76
12
17
80
16
99
15
41
17
28
14
10
23
15
17
GT Factors
GT y-Pos
GT Scale
0.23
0-Rot
22
Shape
GT x-Pos
y-Pas
4
200
GT Shape
Figure 21: Natural Sprites Latent Representations. Top, MCC correlation matrices. Middle five rows, model latent over highest correlating ground truth factor (colored by category). Bottom five rows, model latent over second highest correlating ground truth factor. The left two columns denote the continuous (C) version of Natural Sprites, whereas the right two columns correspond to the discretized (D) version.
48

---

Published as a conference paper at ICLR 2021
caleeVVV
DIP-VAE-11
SlewVAz
AM
Figure 22: DSprites Latent Representations, Best two latents selected from Fig 15 Color-coded by the corresponding ground truth factor.
AnnealedVAE
P-TCUAE
p-VAE
DIP-VAC4
DIP-VAE-I
FactorVAE
slowvaz (UNI)
SlowVer (LAP)
-00
-33
131
40
182
24
Figure 23: Cars3D Latent Representations. Best two latents selected from Fig 16 Color-coded by ground truth.
Categeyy
70
24
630
374
Figure 24: Sobb Latent Representations. Best two latents selected from Fig 17 Color-coded by ground truth.
49

---

Published as a conference paper at ICLR 2021
SlowVAE (UNI)
SlowVAE (LAP)
10
0.38
Zr
25
Z4
82
-1.93
Za
83
Ze
Typ
24
26
14
Zg
Figure 25: Shapes3D Latent Representations. Best two latents selected from Fig 18 Color-coded by ground truth.
SloWVAE (UNI)
SlowVAE (LAP)
Color
20
R
+9.99
0.c9
21
Shape
25
2/
Size
X2
0.60
2
Height
$1
2
Bg Col
24
N
23
25
Z2
0-Rot Z0
0.0
Z7
Figure 26: MPI3DReal Latent Representations. Best latents selected from Fig by ground truth.
Color-coded
50

---

Published as a conference paper at ICLR 2021
Figure 27: KITTI Masks Latent Representations. Best two latents selected from Fig 20 Color- coded by ground truth.
P-VAE
Ade-GVAE
PCL
SlowVAE
PAVAE
Ada-GVAE
SlowVAE 0.98
201
0.005
0.0
GM
3M
0.763
0.067
0.0
Figure 28: Natural Sprites Latent Representations. Best two latents selected from Fig 21 The left four columns denote the continuous (C) version of Natural Sprites, whereas the right four columns correspond to the discretized (D) version. Color-coded by ground truth.
51


Reviews:
[{"review_id": "ikIH9kCP9iG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the SlowVAE to model transitions (of position, size, etc) in single-object videos. A Laplace prior conditioned on the latents at the previous step is used to learn the transitions, which the authors argue are naturally sparse.\n\nMy biggest concern regarding this work is the construction of datasets: the authors effectively “consider pairs of images [x_{t-1}, x_t], which differ effectively in only a few factors of variation due to the sparse prior.” While the marginal distributions of factor transitions may no doubt be sparse in natural settings, I would guess the sparsity is not independent across factors. For example: a video where a camera moves sporadically will show the position factors and size of an object changing sparsely but simultaneously. Could you augment Fig 1 and appendix H to show the joint distributions in addition to the marginals?\n\nMoreover, the LAP procedure has the following rejection rule: “if all factors remain constant (no transition), the sample is rejected as the pair would not result in any temporal learning signal.” Surely there needs to be some static pairs if you want to be close to natural videos? \n\nGiven KITTI is the only dataset you show results on whose transitions are natural (albeit not 3D-natural; see caveat in question 2 below), it would be nice to see more evidence how the SlowVAE would generalize and cope in scenarios different from the ones you’ve constructed. If the LAP procedure indeed samples from marginal transition statistics rather than the joint distributions, I don't believe the samples would resemble \"natural transitions\". Learning disentangled representations from data which is engineered to show changes in one generative feature at a time seems like cheating because we usually don't have easy access to such data (unless an agent actively interacts with an object).\n\nStrengths:\n- Good literature review\n- Comparison to relevant SOTA models, linking to prior work on disentanglement as well as nonlinear ICA, on a variety of metrics.\n- Solid experiments and well-presented results\n\nQuestions:\n1) Have you tried working with sequences rather than pairs of images? I assume your ELBO can be readily extended to sequential data. Are there additional challenges in working with sequences?\n2) Do you think there's room for bias in the natural statistics you computed on Youtube-VOS and KITTI-MOTS? For instance, from the fact that object masks are projections of 3D objects onto 2D frames? Consider the bicycle example (row 1) at https://youtube-vos.org/. In 3D, the position of the bicycle changes smoothly as the cyclist carries it. But in 2D there is a noticeable jump in the center of mass of the bicycle's mask (relative say to the body of the rider). Could this possibly explain why you found a heavier-tailed Laplace distribution to be a good fit for the transitions in Youtube-VOS and KITTI-MOTS?\n3) How do natural transition statistics look like in multi-object datasets? How do you envision extending your work to tackle those?\n4) In 3.3, you suggest alpha = 1 helps break the rotational symmetry of the ELBO by making axis-aligned representations optimal. Could you substantiate this claim? Theorem 1 does not immediately suggest an identifiability advantage for alpha = 1.\n5) On the Natural Sprites dataset, why is it that discrete transitions give SlowVAE an advantage over PCL (MCC score 52.6 versus 50.2 and all other metrics) whereas continuous transitions are disadvantageous (MCC score 49.1 versus 51.7)? Why work with discrete transitions at all?", "extra_scores": {}, "extra_text": {}}, {"review_id": "hffNiYhXXpR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Summary\n\nThis paper introduces a novel VAE-based model with the aim to improve unsupervised disentangling of latent factors in visual data.  This model differs from previous disentangling models in that it takes short (2-frame) videos as input instead of static images.  The model is equipped with a Laplace prior over the dynamics of the video to help it align its representation to axes of sparse temporal dynamics.  The intuition is that the temporal dynamics of natural visual stimuli vary sparsely according to some choice of factors, and that choice of factors is exactly what “disentangled” should refer to.  The authors show that their model achieves better disentangling than previous static-image methods according to a number of metrics.\n\n# Pros\n\n* The interpretation of disentangling as a basis in which the distribution of temporal dynamics of video is sparse is valuable.  Previous approaches to disentangling have been plagued by non-identifiability, and this new interpretation of disentangling is a natural and operational solution.  I fully agree with the authors that temporal information is essential for representation-learning, and hope this paper can help accelerate the field in that direction.\n* The paper is clear and well-written.\n* The experiments are very thorough in terms of comparisons to previous models and evaluation with previous metrics.\n* The application of the Mean Correlation Coefficient (MCC) as a metric for disentangling is good --- I think it is simpler and clearer than many existing disentangling metrics.\n* The latent embedding plots are a nice way to visualize latent representations.  While they don’t show what effects non-matched generative factors have on the latent coordinates, they offer valuable information about the latent embedding that is complementary to the commonly used latent traversals.\n\n# Cons\n\nMy biggest suggestion is to include an ablation study.  Aside from PCL (which isn’t variational so lives in a different world), there are no existing disentangling models on videos to fairly compare the authors’ model to.  Consequently, it is very important to perform ablation experiments.  More specifically, after reading the paper I have a burning question:  How important is the Laplace prior over transitions?  In other words, can the model work with just a KL regularization between the posteriors for consecutive timesteps?  I think it’s quite possible the answer is “yes” --- simply having a KL regularization instead of the Laplace prior should disentangle better than static-image VAEs because the diagonal posterior will be pressured to align with the transition dynamics.  So I wouldn’t be surprised if the model does quite well with just a KL instead of the Laplace, and that would be a simpler model with two fewer hyperparameters.  So please do this experiment --- regardless of the outcome, the results will be very valuable for readers considering using your model.\n\nAside from that, I have only a couple minor suggestions:\n* Figure 2 is a notationally confusing.  You use z subscripts to indicate time in the lower part of the figure, but in the top part of the figure z subscripts indicate component index.  Maybe make the component index a superscript, or at least make the z in the bottom part of the figure bold so the boldness distinguishes vector from scalar.\n* Perhaps make a note that the MCC metric doesn’t work well for discrete latents like shape in dSprites.  For example, in Figure 4 the SlowVAE permutes the shape ordering (e.g. as compared to PCL) and gets a low MCC score for that, but that low score is a drawback of the metric, not the model.  This isn’t unique to MCC --- most other metrics (except MIG) would fall for this too.  But perhaps for discrete latents (ones where we don’t care about the ordering of a discrete set of values), MCC can take the highest over all permutations.  I don’t think it’s necessary to do this, but perhaps you can add a sentence that mentions it so readers understand why the shape score is low.\n\n# Conclusion:\n\nOverall, I recommend this paper to be accepted.  It is very topical, since disentangling has recently been receiving increasing attention, and by incorporating temporal cues into disentangling in VAEs this paper could help steer the field in a productive new direction.  I only hope the authors do the suggested ablation experiment (I think practitioners would appreciate those results).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z-t71YHSz4i", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThe paper addresses the problem of disentangling the underlying generative factors from data with a particular focus on dynamic natural data. It provides evidence that transitions of objects in natural movies can be characterised by temporally sparse distributions. A novel proof based on a sparse prior on temporally adjacent observations is provided, allowing to recover true latent variables up to permutations and sign flips, and improving the disentangling performance over existing methods. Two new datasets with measured natural dynamics are also proposed to enable evaluation on more realistic scenarios.\n\n##################################################################\n\nStrengths:\n- The paper addresses the important problem of unsupervised learning of disentangled representations. Overall, it is well written and easy to read.\n- A new framework of non-linear ICA is introduced, showing evidence to the hypothesis that natural scenes have sparse temporal transitions.\n- Two new benchmarks of natural and unconstrained data are presented which would enable future works on more realistic scenarios.\n- The paper provides extensive experiments and comparisons to relevant baselines, including both qualitative analysis and quantitative results using different metrics.\n\n##################################################################\n\nWeaknesses:\n- The existing datasets present in DisLib are considered unnatural since they assume data is i.i.d and define a prior on the number of factors to be changed. The datasets presented in this work represent incrementally more realistic scenarios where sparse transitions are imposed in addition to data with natural continuous generative factors and data with transitions from unstructured natural videos. In the Laplace transitions dataset, the rate of the Laplace prior λ is sampled from a uniform distribution λ ∼ U (1, 10) which may result in changes in many factors (when λ is small). Although this allows to have fair comparison with Locatello et al., (2020), this setting may sometimes result in non-sparse transitions between image pairs which is not aligned with the statistics of natural transitions as discussed throughout the paper. This would not be expected if one does not read Appendix 4. Hence, it would be good to clarify this particular point in the main paper. \n- In the Natural Sprites dataset, pairs are produced by only varying the position and scale (with real transitions extracted from the YouTube measurements) while color, shape, and orientations are fixed. While fixing the color and shape can be understood to follow natural transitions of objects, it is unclear why the orientation is not varied in this case?\n- For the Kitti masks dataset, continuous natural transitions are considered in all underlying factors. Table 2 shows a clear improvement when the temporal distance between sampled frames ∆t = 5 compared to ∆t = 1. What would be the effect of further varying this parameter?\n- From Figure 4, authors raised up an interesting observation for the rotation factor, where SlowVAE shows three sinusoidal oscillations with different frequencies matching the three distinct rotational symmetries of the shapes present in the dataset. However, from the MCC measurements, we can notice that all methods including SlowVAE struggle in disentangling the rotation factor. Here, it would be interesting to further discuss this limitation. \n\n##################################################################\n\nFinal update: Authors addressed all my comments in the rebuttal making significant improvements to the revision. I've increased my score. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5GOPntXE_l", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper starts with an observation that temporal transitions in sequences of natural images are sparse, which is supported by data collected from two big datasets (youtube-vots and kitti-mots). This suggests using a sparse prior for temporal transitions of latent variables when modelling naturalistic scenes. The authors then introduce SlowVAE, a model for temporal independent component analysis (ICA), that depends on such a sparsity prior, and prove that latent variables are identifiable under this model up to permutation and sign flips, which, they say, is stronger than any previous result. Additionally, this work introduces a number of datasets of increasing complexity (and similarity to natural datasets) for testing disentanglement as well as it performs a large scale and detailed evaluation of the introduced model.\n\nThe paper is very well written and full of convincing arguments. The related work section (#2) is very well thought-through and extensively describes links to the relevant literature. The model (sections 3.1 and 3.3) is well described--I especially liked section 3.4. which describes assumption made by the theory, and how these assumptions can be violated in practice, which is then followed by extensive experimentation showing that the theory can work even when assumptions are validated. It would be of a great benefit to the community if a similar section was present in other papers.\n\nI concede that I did not understand the proof sketch in sec 3.2 nor the corresponding figure 2, and I did not the full proof in the appendix, therefore I cannot speak to the correctness of the identifiability claim.\n\nI have two remarks about eq. 4.\n1) if \\gamma != 0, this is not an ELBO, strictly speaking, which is contrary to what the text suggests.\n2) We can read that the mean \\mu(x_{t-1}) is used as a \"single sample\" to approximate the last term of that equation. Strictly speaking, this can lead to biased estimates of that term, specifically when the mapping from z_{t-1} to the statistics of p(z_t | . ) is non-linear. Additionally, in high-dimensional distribution, the mode itself is a very unlikely sample. Why is this ok to take the mean in the context of this equation?\n\nThe evaluation is extensive (two baselines, one of SOTA from ICA and disentangled representations literature, and 14 datasets). Figure 4, which shows plots of true generative factor vs matched latent variable, is superb. It is a really good method of visualising disentanglement--I agree with the authors that this is a much better way then showing latent traversals. \n\nThe paper seems to be very good, though I am no expert on ICA. I strongly recommend acceptance. The reason I am not giving a higher score is that the significance to the community seems not that high, but please correct me if I am wrong.\n\nSome further questions:\n1) in sec 3.1. you write \"we assume that the noise [in g] is modeled indirectly as a latent variable\". Does that mean that g is itself a latent-variable model?\n2) The appendix includes a \"broader impact\" statement, which strongly suggests that this paper was previously rejected from NeurIPS. May I ask what the main points of criticism were?\n\nSome further remarks:\n* In the first paragraph of intro you write \"although untrue in the literal sense\". Why is that? It seems true to me.\n* \"this problem\" a few lines below the above is unclear.\n* Table 2 appears BEFORE Table 1 in the text. This is very confusing!\n* It is unclear what \"Natural\" in Table 1 refers to. I assumed that is refers to the introduces \"Natural Sprites\" dataset, but maybe I am wrong?\n* A hyphen in latex should be typed as \"---\" and there should be no spaces between the hyphen and the surrounding text; you have one e.g. in  the conclusion.\n\nUPDATE: T`he authors' response addressed all my remarks. I've increased the score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model for learning disentangled representations by assuming the slowness prior over transitions between two frames. The model is well justified theoretically, and evaluated extensively experimentally. The results are good, and all reviewers agree that this paper is among the top papers they have reviewed. For this reason, I am pleased to recommend this paper for an Oral.
Decision: Accept (Oral)