Clearing the Path for Truly Semantic Representation Learning
Authors: Dominik Zietlow
Michal Rolinek
Georg Martius
Venue:  (2021)
Keywords: Representation Learning
Disentanglement
Unsupervised Learning
Semantic Representations
VAE
Causal Representations
PCA

Abstract: The performance of $\beta$-Variational-Autoencoders ($\beta$-VAEs) and their variants on learning semantically meaningful, disentangled representations is unparalleled. On the other hand, there are theoretical arguments suggesting impossibility of unsupervised disentanglement. In this work, we show that small perturbations of existing datasets hide the convenient correlation structure that is easily exploited by VAE-based architectures. To demonstrate this, we construct modified versions of the standard datasets on which (i) the generative factors are perfectly preserved; (ii) each image undergoes a transformation barely visible to the human eye; (iii) the leading disentanglement architectures fail to produce disentangled representations. We intend for these datasets to play a role in separating correlation-based models from those that discover the true causal structure.

The construction of the modifications is non-trivial and relies on recent progress on mechanistic understanding of $\beta$-VAEs and their connection to PCA, while also providing additional insights that might be of stand-alone interest.

Under review as a conference paper at ICLR 2021
CLEARING THE PATH FOR TRULY SEMANTIC REPRE- SENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The performance of 3 Variational Autoencoders (3-VAEs) and their variants on learning semantically meaningful, disentangled representations is unparalleled. On the other hand, there are theoretical arguments suggesting impossibility of unsupervised disentanglement.
this work, we show that small perturbations of existing datasets hide the convenient correlation structure that is easily exploited by VAE-based architectures. To demonstrate this, we construct modified versions of the standard datasets which (i) the generative factors are perfectly preserved; (ii) each image undergoes a transformation barely visible to the human eye; (iii) the leading disentanglement architectures fail produce disentangled representa- tions. We intend for these datasets to play a role in separating correlation- based models from those that discover the true causal structure.
The construction of the modifications is non-trivial and relies on recent progress mechanistic understanding of 3-VAEs and their connection to PCA, while also providing additional insights that might be of stand-alone interest.
INTRODUCTION
The task of unsupervised learning of interpretable data representations has a long history. From classical approaches using linear algebra c.g. via Principle Component Analysis (PCA) (Pearson, 1901) or statistical methods such as Independent Component Analysis (ICA) (Comon, 1994) all the way more recent approaches that rely deep learning architectures.
The comerstone architecture is the Variational Autoencoder (Kingma & Welling, 2014) (VAE) which clearly demonstrated both high semantic quality well as good performance in terms of disentanglement Until today, derivates of VAEs (Higgins et al., 2017; Kim & Mnih, 201 8a; Chen al., 2018; Kumar al., 2017) excel over other architectures in terms of disentanglement metries. The extent of VAE's success even prompted recent deeper analyses of its inner workings (Rolinek al., 2019; Burgess et al., 2018; Chen al., 2018; Mathieu et al., 2018).
If we treat the overloaded term disentanglement to the highest of its aspirations, as the ability to recover the true generating factors of data, fundamental problems arise. As explained bycocalll et al. (2019), already the concept of generative factors is compromised from a statistical perspective: two (or in fact infinitely many) sets of generative factors can generate statistically indistinguishable datasets. Yet, the scores on the disentanglement benchmarks are high and continue to rise. This apparent contradiction needs a resolution.
In this work, we claim that all leading disentanglement architectures can be fooled by the same trick. Namely, by introducing a small change the correlation structure, which, however, perfectly preserves the set of generative factors. To that end, provide alternative version of the standard disentanglement datasets in which each image undergoes a modification barely visible to a human eye. We report drastic drops of disentanglement performance on the altered datasets.
On a technical level, we build on the findings by Rolinck et al. (2019) who argued that VAEs recover the nonlinear principle components of the datasets; in other words nonlinearly computed scalars that are the sources of variance in the sense of classical PCA. The small modifications of the datasets we propose, aim to change the leading principle component by adding modest variance to an alternative
Datasets will be released here: https:/ // //si /ssttes... wetes.goce googl e com/ www.//wwwwweeeeeeeeeeeeeee.ee sem- rep-1ear eearni in

---

Under review as a conference paper at ICLR 2021
candidate. The "to-be" leading principle component is specific to each dataset but it is determined in a consistent fashion. With the greedy algorithm for discovering the linear PCA components in mind, we can see that any change in the leading principle component is reflected also in the others. As a result, the overall alignment changes and the generating factors get entangled leading to low disentanglement scores. We demonstrate that, even though the viewpoint of (Rolinek et al., 2019) only has theoretical support for the (3)-VA,, it empirically transfers to other architectures.
Overall, we want to encourage evaluating new disentanglement approaches on the proposed datasets, in which the generative factors are intact but the correlation structure is less favorable for the prin- ciple component discovery ingrained in VAE-style architectures. We hope that providing a more sound experimental setup will clear the path for new set of disentanglement approaches.
RELATED WORK
The related work can be categorized in three research questions: i) defining disentanglement and metries capturing the quality of latent representations; ii) architecture development for unsupervised learning of disentangled representations; and iii) understanding the inner workings of existing ar- chitectures, as for example of 3-VAEs. This paper is building upon results from all three lines of work.
Defining disentanglement. Defining the term disentangled representation is an open question (Higgins et al., 2018). The presence of learned representations machine learning downstream tasks, such as object recognition, natural language processing and others, created the need to "dis- entangle the factors of variation" " (Bengio al., 2013) very early on. This vague interpretation of disentanglement is inspired by the existence of a low dimensional manifold that captures the variance of higher dimensional data. As such, finding a factorized, statistically independent representation became core ingredient of disentangled representation learning and dates back to classical ICA models (Comon, 1994; Bell & Sejnowski, 1995).
For some tasks, the desired feature of disentangled representation is that it is semantically mean- ingful. Prominent examples can be found in computer vision (Shu et al., 2017; Liao et al., 2020) and in research addressing interpretability of machine learning models (Adel et al., 2018; Kim, 2019). Based on group theory and symmetry transformations, (Higgins et al., 2018) provides the "first prin- cipled definition of disentangled representation" ". Closely related to this concept is also the field of causality in machine learning (Schölkopf, 2019; Suter et al., 2019), more specifically the search for causal generative models (Besserve et al., 2018; 2020).
Architecture development. The leading architectures for disentangled representation learning based on VAEs (Kingma & Welling, 2014). Despite originally developed as a generative modeling architecture, its variants have proven excel at representation learning tasks. First of all the 3-VAE (Higgins et al., 2017), which exposes the trade-off between reconstruction and regularization via the additional hyperparameter, performs remarkably well. Other architectures have been proposed that additionally encourage statistical independence in the latent space, e.g. FactorVAE (Kim & Mnih, 2018b) and 3-TC-VAE (Chen et al., 2018). The VIP-VAE (Kumar et al., 2017) suggests using moment-matching to close the distribution gap introduced in the original VAE paper. As the architectures developed, so did the metrics used for measuring the disentanglement quality of representations (Chen et al., 2018; Kim & Mnih, 2018b; Higgins et al., 2017; Kumar et al., 2017).
Understanding inner workings. With the rising success and development of VAE based architec- tures, the question of understanding their inner working principles became dominant in the commu- nity. One line of work searches for an answer to the question why these models disentangle (Burgess et al., 2018). Another, closely related line of work shows up and tightens the connection between the vanilla (AE objective and (probabilistic) PCA (Tipping & Bishop, 1999) (Rolinek et al., 2019; Lucas et al., 2019). Building on such findings, novel approaches for model selection were proposed (Duan et al., 2020), emphasizing the value of thoroughly understanding the working prin- ciples of these methods. On a less technical side, (Locatello et al., 2019) conducted a broad set of experiments, questioning the relevance of models given the variance over restarts and the choice of byperparameter. They also formalized the necessity of inductive bias as a strict requirement for unsupervised learning of disentangled representations. Our experiments are built on their code-base.
2

---

Under review as a conference paper at ICLR 2021
3 BACKGROUND
3.1 QUANTIFYING DISENTANGLEMENT
Among the different viewpoints on disentanglement, we follow recent literature and focus on the connection between the discovered data representation and set of generative factors. Multiple metrics have been proposed to quantify this connection. Most of them are based on the under- standing that, ideally, each generative factor is encoded in precisely one latent variable. This was captured concisely by Chen et al. (2018) who proposed the Mutual Information Gap (MIG), the mean (over the Nw generative factors) normalized difference of the two highest mutual information values between latent coordinate and the single generating factor:
max I (wii zk) max / (wii (wiizz) N 1= (w:)
(1)
kek
where k' = arg max, I (w;, zn.). More details about MIG, its implementations, and an extension to discrete variables can be found in (Chen et al., 2018; Rolinek et al., 2019). While multiple other metrics were proposed such as SAPScore (Kumar et al., 2017), Factor (Kim & Mnih, 8a) and DCI Score (Eastwood & Williams, 2018) (see the supplementary material of Klindt et al. (2020)), in this work, we focus primarily MIG.
3.2 VARIATIONAL. AUTOENCODERS AND THE MYSTERY OF A SPECIFIC ALIGNMENT
Variational autoencoders hide many intricacies and attempting to compress their exposition would not do them justice. For this reason, we limit ourselves to what is crucial for understanding this work: the objective functions. For well-presented full descriptions of VAEs, we refer the readers to (Doersch, 2016).
As it common in generative models, VAEs aim to maximize the log likeliidd objective
Σ log p(x )))
in which [N = X' is a dataset consisting of N i.i.d. samples x(6) of multivariate random variable X that follows the true data distribution. The quantity p(x(1) captures the probability density of generating the training data point (() under the current parameters of the model. This objective is, however, intractable in its general form. For this reason, Kingma & Welling (2014) follow the standard technique of variational inference and introduce a tractable Evidence Lower Bound (ELBO):
E Ee log p(x(1) 2)) + D x()) p(z)).
Here, are the latent variables used to generate samples from X via parameterized stochastic decoder (xx(") z).
The fundamental question of "How do these objectives promote disentanglementt" was first asked by Burgess et al. (2018). This is indeed far from obvious; in disentanglement the aim to encode fixed generative factor in precisely one latent variable. From a geometric viewpoint this requires the latent representation to be axis-aligned (one axis corresponding to one generative factor). This question becomes yet more intriguing after noticing (and formally proving) that both objective func- tions (2) and (3) are invariant under rotations (Burgess et al., 2018; Rolinek et al., 2019). In other words, any rotation of a fixed latent representation results the same value of the objective function and yet 3-VAEs consistently produce representations that are axis- axis-aligned and in effect isolating the generative factor into individual latent variables.
3.3 RESOLUTION VIA NON-LINEAR CONNECTIONS TO PCA
A mechanistic answer to the question raised in the previous subsection given by Rolinek et al. (2019). The formal argument showed that under specific conditions which are typical for 3 VAEs (called polarized regime), the model locally performs PCA in the sense of aligning the "sources
3

---

Under review as a conference paper at ICLR 2021
(a)
(b)
Input
2
Linear VAE
-2
PCA
Non-linear VAE
Figure 1: (a) The axes of VAEs latent representation, when decoded back to data space, represent non-linear principal components, formalized by Rolinek al. (2019) (figure used with the per- mission of authors). (b) Distribution of latent encodings for an input distributed as depicted in the inset. The linear VAE's encoding matches the PCA encoding remarkably well (left); both focus on aligning with axes based on explained (global) variance. The non-linear VAE is, however, more sensitive to local variance. picks up on the natural axis alignment of the microscopic structure. This discrepancy is the driver of the proposed modifications.
of variance" with the local axes. The resulting global alignment often coincides with finding non- linear principal components the dataset (see Fig. 1). This behavior stems from the convenient but uninformed choice of diagonal posterior, which breaks the symmetry of (2) and (3). This connection with PCA was also reported by Stuehmer et al. (2020), alternatively formalized by Lucas al. (2019) and converted into performance improvements in unsupervised setting by Duan al. (2020).
Since our dataset perturbation method is based this interpretation of the VAE, offer further context. In more technical terms, Rolinek et al. (2019) simplify the closed-form KL-term of the objective (3) under the assumption of polarized regime into the form
(x
Σ =3((( lxg(o)(x())), JEV.
where Vo is the set of active latent variables, (x) is the mean embedding of input x(( and (x(1)) the corresponding term in the diagonal posterior which also plays the role of noise applied to the latent variable j. This form of the objective, when studied independently, lends itself to an alternative viewpoint. When the only "moving part" in the model the alignment of the latent rep- resentation, i.c. application of a rotation matrix, the induced optimization problem has the following interpretation':
Distribute a fixed amount of noise among the latent variables such that the I,2 reconstruction error increases the least.
This extracted problem can be solved in closed form and the solution is based on isolating sources of variance. Intuitively, important scalar factors whose preservation is crucial for reconstruction. need to be captured with high precision (low noise). For that, it is economic to isolate them from other factors. This high-level connection with PCA is then further formalized by Rolinck et al. (2019).
One less obvious observation is that the "isolation" of different sources of variance relies on the accu- racy of the linearization of the decoder at around fixed (x((())) Since in many datasets the local and global correlation structure nearly identical, 3-VAE recover sound global principle components. If, however, the local structure obeys a different "natural" alignment, the Variational autoencoder prefers it to the global one as captured in synthetic experiment displayed in Fig. 1.
The sensitivity of 3-VAE's global representation to small local changes is precisely point of attack.
the precise formal statement be found in (Rolinek al., 2019, Equations (22) and (23))
4

---

Under review as a conference paper at ICLR 2021
4 METHODS
The standard datasets for evaluating disentanglement all have an explicit generation procedure. Each data point x(0) E X' is an outcome of a generative process applied to input w E W. Imagine that g is function rendering a simple scene from its specification w containing as ils coordinates the background color, foreground color, object shape, object size etc. By design, the individual gen- erative factors are statistically independent in W. All all, the dataset A' = (x("), x(2)
x())
is constructed with x g(w()), where y is mapping from the generative factors to the corre- sponding data points.
In this section, we introduce a modification of the generative procedure that barely distorts the resulting data points. In particular, for each x(1) E X, we have
d( x ( ((w())) Se
(5)
under some distance d(.;)..
How to design 9 such that despite an e-small modification VAE-based architectures will create an entangled representation? Following the intuition from Sec. 3.3 and Fig. 1, we "misalign" the local variance with repect to the global variance order to promote an alternative (entangled) latent code.
To avoid hand-crafting this process we can exploit the following observation. VAE based archi- tectures suffer from large performance variance over e.g. random initializations. This hints to the existing ambiguity: two more candidates for the latent coordinate system competing minima of the optimization problem. Some of these solution are "bad" in terms of disentanglement. Below elaborate on how to foster these bad solutions.
should be noted that our dataset modifications are not an implementation of (Locatello et al., 2019, Theorem 1). We do modify the set of generative factors. Rather, slightly perturb the generation process order to target specific subtlety in the inner working of VAEs.
Overall, our modification process has three steps:
(i) Find the most entangled alignment that 3-VAE produces over multiple restarts and re- trieve its first principle component, denoted by s.
(ii) Fix a "noise pattern" that is highly tied with
(iii) Add the noise pattern with suitable magnitude e to each image.
4.1 CHOICE OF FOSTERED LOCAL PRINCIPAL COMPONENT
Over multiple restarts of B-VAE we pick one with the lowest MIG score. This gives us an entangled alignment that is expressible by the architecture. It's first principle component is captured by s: X - R computed as the value of the latent coordinate j that has the least noise of injected (averaged over the dataset):
g(w)
g (w))
wo
W1
s( sxx(1) enc(x j= arg min (02)
arg &
(6)
1122
f(s)
E
This procedure of retrieving the most "important" latent coordinate is consistent with (Higgins et al., 2017) and (Rolinck et al., 2019). The analogy to PCA is that the mapping sxx()) gives the first coordinate of (+) in the new (non-linear) coordinate system.
W3
Figure 2: A schematic visualization of the image generation process. The set of ground truth generating factors stays un- touched by the modification.
4.2 MANIPULATION PATTERNS
We will now describe the modification procedure assuming the data points are r images. The manipulated data-point (() is of the form (() = x() + (x((()) where the mapping f R R" x IR is constrained by |l I for every s. Then inequality (5) is naturally satisfied for
the maximum norm.
5

---

Under review as a conference paper at ICLR 2021
(a) random pattern and f(s
(b) dSprites
(c) Shapes3D
T
Pi Pi,j
s
Figure 3: Dataset modification. (a) Construction the manipulation pattern (transposed visualiza-
tion). (b,c) Visual comparison between original and perturbed samples side-by-side.
At the same time, we aim to produce large local changes. More technically, the local expansiveness factor
[[(s) f(s + 8')ll
(ss()) inf 8'146
[8*]
should be as high as possible everywhere. Additionally, the mapping should be injective so that the resulting modifications contains information about
For the sake of simplicity, we accomplish the above requirements (with high probability) by verti- cally shifting a fixed random pattern. Specifically, for fixed randomly sampled ptj E [1,,1] for i, N and "column factors" Cj E 21, 4], we set f as
( = Pr.js
where i'= round +
This random pattern is moving along the first coordinate with while creating consecutive blocks of length oj in column j in order to incorporate information about on multiple scales.
EXPERIMENTS
In order to experimentally validate the soundness of the manipulations, we need to demonstrate the following:
1. Effectiveness of manipulations. Disentanglement metries should drop the modified datasets across architectures and datasets.
2. Qualitative analysis. The changes in local and global statistics of the dataset should match the intuition fleshed
Sec. 3.3.
3. Comparison trivial modification. Instead the proposed method, we modify with uniform noise of the same magnitude. The disentanglement scores for the algorithms the resulting datasets should not drop significantly.
4. Robustness. The new datasets should be hard to disentangle even after retuning hyperpa- rameters of the original architectures.
5.1 EFFECTIVENESS OF MANIPULATIONS
use scalar s = ()) as described in Sec. 4.1 and embed the data-space via manipulation f(s) described in Sec. 4.2. We deploy this approach on two datasets: Shapes3D (Burgess & Kim, 2018) and dSprites (Matthey et al., 2017), leading to manipulations as depicted in Fig, 3(b,c).
Four VAE based architectures and a regular autoencoder are evaluated on both the original and manipulated dataset using literature regularization strength (or better). Other hyperparameters taken from the disentanglement library (Locatello et al., 2019). For the sake of simplicity and clarity, restricted the latent space dimension to be equal the number of ground truth generative factors.
The resulting MIG scores are listed in Tab. 1. Over all models and datasets, the disentanglement quality is significantly reduced.

---

Under review as a conference paper at ICLR 2021
Table I: MIG Scores for the unmodified and the modified dataset. Each setting was run with 10 distinct random seeds.
AE
B-VAE
TC-3-VAE
Factor-VAte
orig.
0.06 + 0.03
0.23 + 0.08
0.25 + 0.08
0.27 + 0.11
dSprites
mod.
0.05 0.03
0.07 + 0.04
0.14 + 0.05
0.11 + 0.05
orig.
0.09 = 0.06
0.60 + 0.31
0.58 + 0.20
0.27 + 0.18
Shapes3D
mod.
0.05 0.05
0.02 + 0.01
0.28 1 0.23
0.11 + 0.09
5.2 QUALITATIVE ANALYSIS
The proposed manipulations are by definition of constraint (5) small in terms of overall variance, but chosen to have large local expansiveness factor. We want quantify both and the order in which the 3-VAE encodes the generating factors. In addition to the existing generating factors and their influence on the dataset, we also want to evaluate the manipulation applied in the previous experiment. By adding an additional generating factor uniform random variable with zero mean and unit variance, we isolate the effect of the manipulation from the other factors.
We approximate the explained global variance of generating factor w(() by extensively sampling
var E x(0 (w() ). w - (x(1))
Due to the discrete nature of the generating factors, the local expansiveness factor is calculated for equal the step size of the position generating factors. We train a B-VAE on the same modified dataset and report the "information level" (which determines the order of the principle components) of the latent coordinate that encodes each individual generating factor, averaged over 10 restarts. The reported quantity is 1 E (o?), i.e. the lack of noise in the latent coordinate.
Figure (a) and (b) show the local expansiveness factor and the explained global variance for dif- ferent values of e. The local expansiveness of exceeds all other generating factors at E 0.1, long before it's explained global variance becomes dominant. At the same point s becomes the dominant coordinate in the latent encoding, as shown in Fig. 4 (c).
In conclusion, 3-VAE chooses to encode as the first principle component for e 0.1 which is exactly when s starts dominate local statisties but long before it dominates global statistics. This consistent with our prediction in Sec. 3.3.
(a)
(b)
(c)
Local Expansiveness Factor
Shape Scale
Explained Global Variance
Drient.
0.20
0.9
0.012
Pos. X
Pos, Y
0.008
s
0.15
0.004
0.000
0.00
0.05
0.10
0.00
0.05
0.10
0.00
0.05
0.10
Figure 4: A comparison of the local expansiveness factor, the explained global variance and the 3-VAE noise distribution. (a) Sampled local variance for different values of Restricted to the position factors (curves are overlapping) since the sampling in the generating factor space is discrete and varying step sizes are inexpressive.
(b) Variance for generating factors and s. (c) The mean latent noise standard deviation is an indicator for the relevance for reconstruction. Increasing makes the prime encoded coordinate. For & 0.075, was not encoded due to it's noisy nature.
7

---

Under review as a conference paper at ICLR 2021
B-VAE
V-E FactorVAE
x
FactorvAE TC-B-VEE
TC-B-VAE
Score
0.3
MIG Score
0.2
0.10
0.1
0.5
1.0
1.5
2.0
0.5
1.0
1.5
Relative Hyperparameter
Relative Hyperparameter
Figure 5: MIG scores for scaled literature hyperparameters the average number of active units. Less than 5 (left, dSprites) or 6 (right, shapes3D) active units mean the architectures overpruned and can recover the true generative factors.
10 restarts. The dashed lines show
5.3 NOISY DATASETS
We replace our modification by contaminating each image with pixel-wise uniform noise between [-s,e]. The value of is fixed to the level of the presented manipulations. Table provides the results for the same five architectures as used before. The lack structure in the contamination does not affect the performance in guided way and leads to very little effect on 3DShapes. The impact dSprites is, however, noticeable. Due to the comparatively small variance among dSprites images, the noise shifts the balance between the reconstruction loss and the regularizers. We performed grid search over I and recovered the performance on the noisy dataset (3" 16), the same can be expected for the other architectures.
Table MIG scores on original and noisy version the datasets with literature hyperparameters. 3* was tuned to the noisy dataset. Noise cannot explain the full loss in disentanglement.
3-VAE
TC-3-VAE
Pactor-VAB
orig.
0.23 + 0.08
0.25 + 0.08
0.27 + 0.11
dSprites
noisy
0.09 0.05
0.23 0.13
0.20 0.04
0.16 0.08
orig.
0.60 + 0.31
0.58 + 0.20
0.27 0.18
Shapes3D
noisy
0.66 0.05
0.60 0.11
0.33 0.20
5.4 GRIDSEARCH HYPERPARAMETEIS
We run a grid searches over the hyperparameter for each architecture. The results are illustrated in Fig. Overall our modifications seem mostly robust for adjusted hyperparameters. On both datasets, significant increase in the regularization strength allows for some recovery. More thorough analysis reveals that this effect starts only once the models reach a level of over- over-pruning, which is behavior well known to practitioners. The dashed curves mark the region in which the active latent space dimension (number of coordinates for which (a²) < 0.8) shrunk significantly. This effect goes along with decreased reconstruction quality and also intrinsically prevents the models from recovering all true generative factors and as such renders this area uninteresting.
CONCLUSION
We have shown that the success of VAE based architectures is solely based on the structured na- ture of the datasets they are being evaluated on. The goal of truly semantic representation learning requires an alternative benchmark to excel at, free of similar correlation based artifacts. We pro- pose the community set of novel datasets and hope that future architectures become capable of learning truly semantic representation.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tameem Adel, Zoubin Ghahramani, and Adrian Weller. Discovering interpretable representations for both deep generative and discriminative models. In International Conference on Machine Learning, pp. 50-59, 2018.
Anthony J Bell and Terrence J Sejnowski. An informationnmaxxiiizaioon approach to blind separa- tion and blind deconvolution. Neural computation, ((6):119-1159) 1995.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Iransactions on paltern analysis and machine intelligence, 178- 2013.
Michel Besserve, Naji Shajarisales, Bernhard Schölkopf, and Dominik Janzing. Group invariance principles for causal generative models. In International Conference on Artificial Intelligence and Statistics, pp. 557-565. PMLR, 2018.
Michel Besserve, Rémy Sun, Dominik Janzing, and Bernhard Schölkopf. A theory independent mechanisms for extrapolation in generative models. arXiv preprint arXiv:2004.00184. 200000
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://ghhhn.coommmemmmm/////hhs 2018.
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in ß-vae. ArXiv e-prints, abs/ Re//1040499999 2018. URL pattp:/ai /awpiv..iivvvvv////i///a./aa.avv org/ abs 1804. C3599.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018.
Pierre Comon. Independent component analysis, new concept? Signal Processing, 36(3):287 314, 1994. ISSN 0165-1684. doi: https:/doi..og//0.0006666666 16606000666111144499999999999999 URL http: / //w www. sciencedi wtweee/eeeeetttooooo//ooeeeeeeee http:///......eeeeeeeaaaaaa////a
com Higher Order Statistics.
Carl Doersch. Tutorial on variational autoencoders. arXiv preprint
2016.
Sunny Duan, Loic Matthey, Andre Saraiva, Nick Watters, Chris Burgess, Alexander Lerchner, and Irina Higgins. Unsupervised model selection for variational disentangled representation learning. In International Conference on Learning Representations, 2020. URL https: // //openzeview.o net/ dys.mmmm-sdyyxy
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. In International Conference on Learning Representations, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. In International Conference on Learning Representations, ICLR, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.022. 02230.. 2018.
Hyunjik Kim. Interpretable Models in Probabilistic Machine Learning. PhD thesis, University of Oxford, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jemnifer Dy and Andreas Krause (eds.), Proc. ICML, volume 80, pp. 2649-2658, PMLR, 2018a. URL htthtttp///rr/ceddggg m1 mlr. molr.press/vel v80 /iim 8b. html.

---

Under review as a conference paper at ICLR 2021
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. volume 80 of Proceedings of Machine Learning Research, pp. PMLR. URL http:tt/ppptte http::///ccceedinng... mlr. press/v80 seess/vv0/kingg 10000/kimB0 8b html.
Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018b.
D. P Kingma and M. Welling. Auto-Encoding Variational Bayes. ICLR, 2014.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007.109:: 2020.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference disentan- gled latent concepts from unlabeled observations. arXiv preprint arXiv: 1,.000848, 2017.
Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 5871-5880, 2020.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pp. 4114 4124 2019.
James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't 't blame the elbo! a linear vae perspective on posterior collapse. In Advances in Neural Information Processing Systems, pp. 9408-94188 2019.
Emile Mathieu, Tom Rainforth, N. Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational auto-encoders. ArXiv e-prints, abs/ 11111122188,,,., 2018.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https://github..omm// httpp//wwwoooeeppeiee///pssaaaas
2017.
Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, 1901. doi: 10.110001177644400944477777
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue PCA di- rections (by accident). Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12406-1244 12415, 2019.
Bernhard Schölkopf. Causality for machine learning. arXiv preprint arXiv: 2019.
Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, and Dimitris Samaras. Neural face editing with intrinsic image disentangling. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 5541-5550, 2017.
Jan Stuehmer, Richard Turner, and Sebastian Nowozin. Independent subspace analysis for unsuper- vised learning of disentangled representations. volume 108 of Proceedings of Machine Learning Research, pp. 1200-1210. PMLR, 2020. URthtpp:// /ttp:://///ceeeeeeeeee ngs. mll. wwl...press/vv1..
Stuehmer20a html.
Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In Interna- fional Conference on Machine Learning, pp. 6056-6065, 2019.
Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), ((())611-622)) 1999.
10


Reviews:
[{"review_id": "Zu2fawTgjLv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows a new method to hurt the disentanglement of Variational AutoEncoders (VAEs) by only slightly modifying the images. It conducts experiments and shows that the disentanglement metrics drop significantly by this carefully designed perturbation compared to uniformly noise, and the results are consistent across different variants of VAEs and various datasets. These results show that the success of VAEs are based on the structured nature of the datasets, and the community needs to find new datasets to learn the semantic representations.\n \nThe idea proposed by this paper is interesting. It believes that the disentanglement of beta-VAEs comes from the local PCAs, which coincides with global coordinate alignments, and this strange behavior stems from the assumption of diagonal posterior. Thus the global representation of beta-VAEs is sensitive to local changes. Also, this paper comes with a detailed background description, which is very much self-contained. The analysis of the results is also exhaustive, and it well supports the claim of this paper.\n \nHowever, the methodology contribution of this paper is marginal and the method it adopts seems over-simplified. The method is not technical and is nothing but noise to the generation process, and the design of the noise is not elaborate, either. E.g., instead of a detailed noise on all coordinates, the paper only applies this on the \"most important latent coordinate\", which is less convincing and lacks an explanation. \n\nAlso, the conclusions of this paper are already well-discussed by previous works, and the experiments of this paper seem like a verification. The section 3 of this paper revisits many previous results but no new idea is proposed across this paper.\n \nMore metrics should be used. As described in the paper, various metrics have been proposed: SAPScore, FactorVAEScore, DCI Score, but this paper only adopts the Mutual Information Gap, which is neither typical nor representative. Also, the correlation between these metrics is not very much high according to fig2 of Locatello et al. Given all these metrics are already implemented by disentanglement_lib and the codebase of this paper is based on that paper, it's not too hard to show all of them.\n \nSome connections between section 3 and the remainder of the paper are not clear. E.g. this paper argues the importance of local and global alignments in section 3, but it's unclear which part of section 4 corresponds to this point. The term \"local expansiveness\" is used in section 4 but was not discussed in section 3. Some complex ideas were discussed in section 3 but never touched again.\n \nIt is said that the modification to the images is barely visible to human eyes, but clearly, the images of fig 3 are heavily blurred and it's easy to tell the difference. This perturbation is not \"a small modification\" and is human sensible in contrast to other attack/defense papers. The authors should show out a measure of perturbations, e.g. the relative magnitude of epsilon.\n \nNo baseline works are introduced. The authors only compare their results with a model that works with a random baseline, and it's not convincing.\n\nQuestions:\n \nAt the end of section 4, you propose a simplification of the method by a shifting window. Why don't you just apply the modification for the whole image? Is there a specific reason that prevents you from doing so?\n \nPresentation suggestions:\n \nThe paper spends almost 3 pages on background research, but only 1 page on its method. It'd better put more stuff in section 4 and move some content from section 3 to section 2 or do a better summarization.\n \nTypo:\n \nThe third line of section 4.1: it's -> its", "extra_scores": {}, "extra_text": {}}, {"review_id": "WQVTWtyRHqa", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis work aims to demonstrate that VAE-based architectures can take advantage of inherent correlation between data to produce representations. They claim that small perturbations of the data prevent these architectures from taking advantage of such correlations resulting in failure to produce disentangled representations. This work therefore claims that the perturbed datasets should be used to design and learn models that can discover the true structure of the data\n\nThis work highlights a characteristic of leading models for disentanglement. It demonstrates that these models are taking advantage of the correlation in the data and not fully learning the true structure. Therefore, while the current models are certainly useful, models that can learn the true structure might be even more beneficial. \n\nThey take a structured approach by perturbing the data in a manner such that the local variance is `misaligned' from the global variance. They demonstrate empirically that the performance of the models goes down for the data set when it is perturbed in the proposed structured manner. This also shows the merit of the perturbation procedure as adding uniform noise to pixel doe not produce the similar drop in performance.\n\n\nThe work, therefore, does seem to have merit. However, there are some clarifications that are needed.\n\n\nFigure 1 is a bit unclear. If 1a and 1b is supposed to show a distribution then why does the vertical axis have -ve values? I am basically not fully sure what exactly is being depicted by 1a and 1b. They look similar to the raw data in the inset. If its just the principal components then should it not be directions and not a patch? Perhaps the figure can be clarified.  \n\nEq8 is a bit unclear. How is $c_j$ selected from the set $\\{1,2,3,4\\}$. Also assuming s = s(x^{(i)}), is it possible that $i'$ = $i$ for all $i$ ?. Furthermore I am a bit confused about the notations. For example, $f:\\mathbb{R} \\rightarrow \\mathbb{R}^{r} \\times \\mathbb{R}^{r}$. This seems to imply that $f(s) = (v,u)$ where $v,u \\in  \\mathbb{R}^{r}$. In eq8, however it seems that f(s) is a matrix of scalars? This would imply that $f:\\mathbb{R} \\rightarrow P^{2}$ where $P = \\{ -1,1 \\}$. In addition to this, why do i,j belong to \\mathbb{N} (and not related to the size of input x).\n\nIn the context of Figure 4, How are the $1 - \\mathbb{E}(\\sigma_{i}^{2})$ calculated for shape scale orient, PosX and PosY. Are they calculated using the reconstruction? \n\n\nWere any other types of f() explored. What influence does the choice of f() (that abides by all the conditions in Section 4.2) have on the performance. If it is specifically the proposed Eq8 that produces this influence then what might be the major characteristics of the f() that produce the drop in performance?\n\n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TI-wfUHYa6G", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a corruption–of image data in commonly used disentanglement benchmarks– which hurts the MIG disentanglement score of top models. The corruption additively encodes an *entangling* factor into each image. The entangling factor is obtained from a poorly disentangled beta-VAE model trained on the original data.\n\nMy chief criticism of the paper is I’m not convinced it discredits existing disentanglement datasets or models. (Nor does the paper clear the path for truly semantic representation learning, but I will reserve my criticism of the paper title.) Perhaps it discredits the MIG metric, but that does not seem sufficient to me. \n\nThe authors show that models like beta-VAE, TC-beta-VAE, and Factor-VAE are compelled to encode s because it is crucial to reduce the reconstruction error. But s happens to be correlated (and is a deterministic function of) the other generative factors w_j. Therefore it seems redundant to encode. Nevertheless, it is possible to make a case that the models are doing their job correctly (if they do encode the other generative factors as independent latents, which is not clear from the results in the paper). Consider Suter et al. (2019)’s proposal suggesting disentangled representations should reflect causally independent generative mechanisms (influences) on the data. The entangling factor s certainly has an independent generative effect on the images. So perhaps the authors mean to indict mainstream definitions/expectations of disentanglement instead?\n\nHere are some questions and requests for additional details:\n1) How does your s relate to the generative factors (on each dataset)? Could you provide its mutual information with position, scale, and the other factors?\n2) Could you offer a breakdown (across generative factors) of the MIG scores on the corrupted data?\n3) Is there a single latent that encodes for s typically? What if you exclude this latent when computing the corrupted MIG scores?\n4) What would be other reasonable choices of s (other than the most important latent of a poorly disentangled beta-VAE model) which would break disentangling? What choices of s would not break disentangling?\n\nMinor suggestions:\n- The corruption itself could be better motivated and explained. The Related Work and Background sections, though extensive, don’t immediately suggest your corruption function. \n- It would be worth explaining the “local expansiveness factor” and “explained global variance” before the main analysis. Equation 9 is also hard to grasp. Could you please unpack it?\n- “Lack of noise” can be formalized as precision or 1/sigma**2.\n\nStrengths:\n- The analysis around Figure 4 (mainly that epsilon around 0.1 is where s starts to be encoded, despite its relatively low effect on the global variance).\n- Comparison of several models showing a consistent effect of the suggested corruption.\n- Results from varying the regularization hyperparameters.\n\nReferences:\nSuter, R., Miladinovic, D., Schölkopf, B., & Bauer, S. (2019, May). Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning (pp. 6056-6065). PMLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AwE_0Dn3m1N", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors first describe the shortcomings of existing work for disentanged representation learning and the general problem of unidentifiability. Here however, the presentation could be more precise. For example, they state that the log-likelihood objective (2) and the evidence lower bound (3) would be \"invariant under rotations\", and give (Burgess et al., 2018; Rolinek et al., 2019) as a reference. However, the log likelihood or the evidence lower bound themselves are not rotationally invariant per-se. Instead these objectives are invariant under rotations for the standard VAE with a rotationally invariant prior p(z) (e.g. the standard normal distribution).\n\nThe authors then discuss a possible shortcoming of VAE based approaches: their sensitivity to local covariance statistics in the dataset. From this perspective the authors motivate their idea to modify existing datasets by adding a noise pattern to the dataset that is highly correlated with the first principle component identified by the model when trained on the standard dataset.\n\nEssentially the authors crafted a specialised attack against VAE based approaches for \"disentangled\" representation learning by adding another source that is highly correlated with the main source of variation of the data. Any approach similar to PCA (perhaps even to ICA), would fail on this task of separating these two highly correlated signals.\n\nThus I am not convinced about any benefits of the proposed data modification and how this work would \"clear the path for truly semantic representation learning\". The authors do not demonstrate how a method that could decorrelate the added noise signal from the original dataset would have any further benefits.\n\n\nPositive Points:\n\n+ Criticism of current state-of-the-art methods for learning disentangled representations is valid\n\nNegative Points:\n\n- Mathematical background needs to be more precise\n\n- Does not take into account recent advances in self supervised representation learning, e.g. noise-contrastive learning and Nonlinear ICA [1]\n\n- Goal of the paper is not clear\n\n- Benefit of the proposed data modification is not clear\n\n[1] Ilyes Khemakhem, Diederik P. Kingma, Ricardo P. Monti, and Aapo Hyvärinen. Variational Autoencoders and Nonlinear ICA: A Unifying Framework. AISTATS2020", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Since the authors have decided to withdraw this submission, it has been rejected from the conference.
Decision: Reject