Structure by Architecture: Disentangled Representations without Regularization
Authors: Felix Leeb
Giulia Lanzillotta
Yashas Annadani
Michel Besserve
Stefan Bauer
Bernhard Schölkopf
Venue: ICLR 2022 Submitted (2022)
Keywords: Autoencoder
Structure
Disentanglement
Generative
Hybridization

Abstract: We study the problem of self-supervised structured representation learning using autoencoders for downstream tasks such as generative modeling. Unlike most methods which rely on matching an arbitrary, relatively unstructured, prior distribution for sampling, we propose a sampling technique that relies solely on the independence of latent variables, thereby avoiding the trade-off between reconstruction quality and generative performance inherent to VAEs. We design a novel autoencoder architecture capable of learning a structured representation without the need for aggressive regularization. Our structural decoders learn a hierarchy of latent variables, akin to structural causal models, thereby ordering the information without any additional regularization. We demonstrate how these models learn a representation that improves results in a variety of downstream tasks including generation, disentanglement, and extrapolation using several challenging and natural image datasets.

Under review as a conference paper at ICLR 2022
STRUCTURE BY ARCHITECTURE: DISENTANGLED REPRESENTATIONS WITHOUT REGULARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
We study the problem of self-supervised structured representation learning using autoencoders for downstream tasks such as generative modeling. Unlike most methods which rely matching arbitrary, relatively unstructured, prior dis- tribution for sampling, we propose a sampling technique that relies solely on the independence of latent variables, thereby avoiding the trade-off between recon- struction quality and generative performance inherent to VAEs. We design novel autoencoder architecture capable of learning a structured representation without the need for aggressive regularization. Our structural decoders learn a hierarchy of latent variables, akin to structural causal models, thereby ordering the information without any additional regularization We demonstrate how these models learn a representation that improves results in a variety of downstream tasks including gen- eration, disentanglement, and extrapolation using several challenging and natural image datasets.
INTRODUCTION
Deep learning has achieved strong results on a plethora of challenging tasks. However, performing well on a highly specific dataset is usually insufficient to satisfactorily solve real-world problems (Tan Zhuang et al. 2019). This has lead to a particular interest in consistently learning more structured representations with useful properties to help with variety of downstream tasks (Bengio 2013 Tschannen et al. 2018 Bengio et al., 2017] Tschannen et al., 2018 Radhakrishnan jet al. al.
(ct al. 2018
2018). Here, deep learning provides flexible paradigm to train complex architectures based on autoencoders (Ballard 1987 Lange and Riedmiller (010) which are typically latent variable models, thus allowing the representations. However, is still largely an open question as to what kinds of structure in a representation are the most effective for generative modeling and how to learn such structures without
embed powerful inductive biases into the models to further structure
supervision (Locatello et al.) 2018 Khrulkov et al. [/2222]] [hh et al. 2019 Chen et al., 2020 Nic
al., 2020 Mathieu et al. 2019 Vaswani et al. [017]] Kwon and Yel 2021). One direction that may contribute to an answer is causal modeling, as it focuses on the underlying (causal) mechanisms that
generate the observations, instead of relying on (possibly spurious) correlations (Pearl 2009) Peters
et al. 2017 Schölkoppi 2019 Louizos et al. 2017 Mitrovic et al. 2020 Shen et al. 2020).
With the versatility of deep learning on one hand, and the conceptual insights of causality on the other, our contributions herein include:
We propose an architecture called the Structural Autoencoder (SAE), where the struc- tural decoder emulates general acyclic structural causal model to learn a hierarchical representation that can separate and order the underlying factors of variation in the data.
We provide sampling method that can be used for any autoencoder-basee generative models which does not use an explicit regularization objective and instead relies on independence in the latent space.
We investigate how well the encoder and decoder are able to extrapolate upon seeing novel samples.
We release code at *anonymi zed*

---

Under review as a conference paper at ICLR 2022
1.1
RELATED WORK
The most popular autoencoder based method is the Variational Autoencoder (VAE) (Kingma and Welling 2013a) and the closely related (8VEE (Higgins al., 2017). These methods focus on matching the distribution in the latent space to known prior distribution by regularizing the reconstruction training objective (Locatello et al. 2020a Zhou et al., 2020). Although this structure is convenient for generative modeling and even tends to disentangle the latent space to some extent, it comes at the cost of somewhat blurry images due to posterior collapse and holes in the latent space
(Locatello et al. 2018 Higgins al. 2017 Burgess et al. 2018 2020).
Kim and Mnih 2018 Stühmer et al.
To mitigate the double-edged nature of VAEs, less aggressive regularization techniques have been proposed such as the Wasserstein Autoencoder (WAE), which focuses on the aggregate posterior (Tol stikhin al., 2018). Unfortunately, WAEs generally fail produce a particularly meaningful disentangled latent space (Rubenstein et al.) 2018), unless weak supervision is available (Han et al.. 2021).
more structured alternative is the Variational Ladder autoencoder (VLAE) (Zhao et al., 2017) which separates the latent space into separate chunks each of which is processed different levels of the encoder and decoder (called "rungs"). Our proposed architecture takes inspiration from the VLAEs but crucially we do not use the variational regularization, and instead use sampling method based on hybridization (Besserve et al. (2018). We also infuse the information from the latent variables using Str-T'm layers (see section 2.2 which are based on the Ada-IN layers from Style- (Karras et al.) 2019), which significantly improves on the ladder rungs. Lastly, unlike the VLAEs, the SAEs use simple feed-forward CNN encoder (like all the other unstructured models), which simplifies analysis without significantly impacting the ability to structure the representation (see further discussion in section
2
METHODS
2.1 CAUSAL REPRESENTATION LEARNING
Graphical causal modeling represents the relationship between random variables S using directed acyclic graph (DAG) whose edges indicate direct causation and structural equations of the form
S; f((PA,, (i L...,)),, D),
encoding the dependence of variable St on its parents PA, in the graph and on an unexplained noise variable U.. The noises UI...., UD are assumed to be jointly independent. The DAG along with the mechanisms (T) is referred to as Structural Causal Model (SCM) (Pearl 2009). Any joint distribution of the S; can be expressed as an SCM using suitable fi and U.. However, the SCM entails additional assumptions regarding how statistical dependencies between the S, are generated by mechanisms
(1. such that changes due to interventions can be modelled as well (e.g.. by setting some U. S; constants).
Real-world observations are often not structured into meaningful causal variables and mechanisms to begin with. E.g., images are high-dimensional, and it is hard learn objects and their causal relationships from data (Lopez-Paz et al. (017). One may thus attempt to learn a representation consisting of causal variables or disentangled "Tactors" which are statistically independent (Higgins) et al. 20.7). However, in an SCM it is not the S; that should be statistically independent, but the U.. For this reason, our representations will comprise the U, as latent variables, driving causal mechanisms via learned functions Si (.. This embeds an SCM into a larger model whose inputs and outputs may be high-dimensional and unstructured (e.g., images) (Suter al.) 2018a| Schölkopf 2019).
Given (high-dimensional)) X = (X,,..., Xa), our goal is to learn low-dimensional representation U (U..... "i....,U)) (D < d) using encoder Rd RP, and model the generative process (including the SCM) that produced X from the inferred latent variables using decoder RD - Rd If the causal graph of the true generative process were known, the topology of the decoder could be fixed accordingly. However, in the fully unsupervised setting, the generator must be capable of modeling general SCM, so architecture specifies causal ordering (while learning what
2

---

Under review as a conference paper at ICLR 2022
information to embed in the specified ordering), while the edges between parents and children in the SCM are learned implicitly in the computation layers. Specifically, the learned generative process, a.k.a. decoder, produces a reconstruction X X by feeding each of the U; into subsequent computation layers. Here, the root node S in the DAG only depends on U1, while later S, depend on their noise Uj and potentially their parents S < i). Thus the depth in the network corresponds causal ordering (see Supplement for further discussion).
2.2 STRUCTURAL DECODERS
U
A MLP,
This model architecture is implemented by the structural decoder, using D Structural- Transform (Str-Tfmm) layers placed evenly in be- tween the convolution blocks. From the cor- responding latent variable Ug the ith Str-Tfm layer estimates a scale a and bias 3; which are then used to pixelwise transform the intermedi- ate features of the /th layer Ut (as seen in Fig- ure I much like in Ada-IN (Karras al. 2019) except without the preceding normalization.
U,
SeTTinfs" (9.8) nn oxt"" +.8.
Bilinear Upsampling
Static Features
Convolution
Each Str-Tfm layer thus acts like an fi 4 by integrating the information from a latent variable U, to transform the features PA, from earlier layers. This variable decoder depth for each of the latent variables biases high-level non-linear information towards the earlier (and thereby deeper) latent variables, while the model capac- ity is reduced for the later variables so they can only capture relatively low-level linear features with respect the data space. Meanwhile, the random initialization of the MLPs in the Str-Tim layers produce more distinct activations with re- spect to each of the latent variables than if they are all transformed linearly by same dense layers. This architectural asymmetry between latent variables thereby encourages statistical independence and induces a relatively intuitive hierarchical structure of the latent space.
Vector Split
Fully-connected
Figure 1:
The Structural Decoder reconstructs (or generates) a sample from a latent vector U by first splitting U into D variables each of which transforms the image features after the convolu- tion block I of the model using the corresponding Str-Tfm layer (green box where each pixel o/ is transformed by layer specific og and 3, which are extracted from the latent variable Uj by two- hidden-layer network MLP.).
In theory, without some form of supervision or side information, the learned latent variables are not guaranteed to disentangle the true factors of variation, much less that the learned SCM matches the true one (Locatello et al., 2020b). Instead, all we are guaranteed from training on observational data is that the model is optimized to reproduce the same observational distribution as the true generative process. However, for datasets with independent factors of variation, the independent latent variables may align with the true factors, thus disentangling the factors in the representation. Regardless of disentanglement, independence between latent variables also enables principled interventions on individual variables for generating new samples, using hybrid sampling, which is akin to intervening on the learned SCM for generative modeling.
2.3 HYBRID SAMPLING
For generative modeling, it is necessary to sample novel latent vectors that are transformed into (synthetic) observations using the decoder. Usually, this is done by regularizing the training objective so the posterior matches some simple prior (e.g- the standard normal). However, in practice, regularization techniques can fail to match the prior perfectly and actually exacerbate the information bottleneck, leading to blurry samples from holes in the learned latent distribution and unused latent dimensions due to posterior collapse (Dai and Wipi 2019 Stühmer et al.) 2020 Lucas al. 2019 Hoffman and Johnson| 2016). Instead of trying to match some prior distribution in the latent space, suggest alternative sampling method that eliminates the need for any regularization of the loss. Inspired by Besserve et al. (2018), we refer to it as hybrid sampling: the model stores a finite set of N (= 128 in our case) latent vectors, selected uniformly at random from the training set. To generate new latent vector Ü. value for each of the D latent variables Ü, is selected independently from the N
3

---

Under review as a conference paper at ICLR 2022
stored latent vectors { uniformly at random, such that: Ü, 6 U.(i) where j ~ (1,,.., N) This allows the model to generate N distinct latent vectors, and since we assume < d, N can easily be increased up to the full training set size without significant memory costs. Note that hybrid sampling is directly applicable to any learned representation as it does not affect training at all, however the fidelity of generated samples will diminish if there are strong correlations between latent dimensions. Consequently, the goal is to achieve maximal independence between latent variables without compromising on the fidelity of the decoder (i.e. reconstruction error).
Hybrid sampling implicitly relies on the statistical independence of the latent variables since resam- pling the marginal of the aggregate posterior independently breaks any existing correlations. Not only does this align well with the objectives of unsupervised disentanglement methods, but it is also consistent with the causal perspective of the latent variables as independent noises Ug driving an SCM.
3 EXPERIMENTS
We train the proposed architectures and baselines on two smaller disentanglement benchmarking datasets (where d = 64 x 64 3 and the true factors are independent): 3D- 3D-Shapes (Burgess and Kim 2018) and the three variants ("toy", "sim", and "real"" of the MPI3D Disentanglement dataset (Gondal ct al.
2019), well two larger more realistic datasets (where d = 128 x 128 x 3): Celeb-A (Liu
ct al.
2015) and the Robot Finger Dataset (RFD) (Dittadi et al.) 2020).
models a standard 70-10-20 (train-val-test) split of the datasets we evaluate After training
the quality of the reconstructions based on the reconstruction loss (using binary cross entropy loss, same as the optimization objective) and the Fréchet Inception Distance (FID) (Heusel et al. (017) as in Williams et al. 2020). The FID is able to capture higher level visual features and can be used to directly compare the reconstructed and generated sample quality, while the binary cross entropy is purely pixelwise comparison.
Next we compare the performance of the hybrid sampling method to the prior based sampling. Unlike the prior based sampling, which only makes sense for the models that use regularization (VLAE, VAE, VAE, and WAE), the hybrid sampling method can be applied to any latent variable model. Finally we take closer look at the learned representations to understand how the model architecture affects the induced structure and possibly disentanglement.
3.1 MODELS
All models use the same CNN backbone for both the encoder and decoder with the same number of convolution layers and filters, each of which followed by group normalization and MISH nonlinearity (Misral 2019), where the encoder uses max-pooling to down-sample the extracted features and the decoder uses bilinear up-sampling to up-sample features (see the appendix for details). For the smaller datasets, the encoder and decoder have 12 convolution layers each with 64 filters, the latent space has 12 dimensions in total, and the models are trained for 100k iterations, while for CelebA each the encoder/decoder has 16 convolution layers each with twice as many filters, the latent space is 32 dimensional in total, and the models are trained for 200k iterations. For the RFD dataset, the settings are the same as for CelebA, except that there are 256 channels per convolution layer.
We compare four kinds of autoencoder architectures. The first type is our Structural Autoencoders (SAE) which use a conventional encoder and a structural decoder with the latent space split evenly into 2, 4, 6. or 12 variables for the smaller datasets and 16 variables for the larger ones correspond- ing to the labels SAE-2, SAE-3, SAE-4, SAE-6, SAE-12 and SAE-16 16 respectively. The simplest "baseline" architecture uses the traditional "hourglass" architecture, in addition to variety of differ- ent regularization methods: (1) unregularized autoencoders ("AEE",, (2) eeb.seestein-autonncoooe ("WAE") (Tolstikhin et al., 2018). (3) VAES (Kingma and Welling 2013b), and a VVE (Higgins) al. 2017) (with the experimentally determined best setting of 3 2). The next baseline architec- ture is called " "AdaAE" (and referred to as ''Aaptiie"' as it is identical to the SAE models except that all latent variables are passed to each of the Str-Tfim layers, the architecture is effectively the same as using Ada-IN IN layers (Karras et al., 2019). For the purposes of hybrid sampling, each latent dimension of these less structured baselines treated separate latent variable.
4

---

Under review as a conference paper at ICLR 2022
The final type of architecture investigate is the Variational Ladder Autoencoder (Zhao al. 2017) which also uses a structured architecture to learn a hierarchical representation, but unlike our Structural Autoencoders, VLAEs also use the variational regularization and architecture that roughly mirrors the decoder. Another subtle, but significant difference between the SAE and VLAE architecture that the ladder rungs in the VLAE decoder concatenate the features from the latent variable of the corresponding rung in the encoder, instead of using the variable to scale and shift the convolution features directly. Just like for the SAE models, we include variants of the VLAEs with 2, 3, 4, 6, 12, and 16 rungs.
encoder
3.2 EXTRAPOLATION
While the encoder and decoder are generally optimized jointly, the tight coupling combined with the inherent asymmetry in the respective learning tasks begs the question which of the two is the weaker link when it comes to generalization. To this end, we investigate what extent the encoder vs the decoder can extrapolate to novel observations.
First, both the encoder and decoder are trained jointly on a subset of 3D-Shapes where only three distinct shapes exist (instead of four, the ball is missing) for 80k iterations. Then either the encoder only, the decoder only, or both are trained for another 20k iterations on the full 3D-Shapes training dataset. The reconstruction error for samples not seen during any part of training compared for each of the variants and each of the architectures to identify how well the encoder extrapolates compared to the decoder.
4
RESULTS
3D Shapes Reconstruction Quality
MP13D Toy Reconstruction Quality
MP1DD Real Reconstruction Quality
Reconstructoo cioon FID Score
Architecture
Reconstructia ction FID Score
Architscture
Reconstructan FID Score
Architecture
40
Structural
40
Structural
Structural
Baseline
Baseline
Baseline
Adaptive
Adaptive
Adaptive
Ladder
Ladder
Ladder
20
20
10.
SAE
12-12
S4E
SAE-3
SAF
VLAE-4
566
WAR-3
VLAE
VLAE-12
VAE
WAE
WIAE
SML-12
WAL
SAE
AE
WAF-6-
4
WAAE-2 12
VLAE 2
VLAE3 3
VAE
(VVA
AdanE-12
SAE
SAE 12
SAF-4
SAE
SAE-2
WAE
AE
VLAA-12
VLAE-4
VIAE-3
6
VLAC
VAE
(VAE
(a) 3D-phapes
(b) MPI3D Toy
(c) MPI3D Real
CelebA Reconstruction Quality
Reconstruction FID Score
140
Structeull Buasiin Ladder
130
120
110
100
VLAE 16
pVAE
AC
(d) Celeb-A
RFD
Figure 2: Results on the reconstruction quality for all models and datasets. The "Baseline" models correspond traditional "hourglass" CNN architectures, while the "Structural" models novel architectures to further structure the learned representation.
In terms of reconstruction quality, the structural autoencoder architecture consistently outperforms the baselines (seen in figure 2 As expected, unregularized methods like the SAE, AdaAE, and AE tend to have significantly better reconstruction quality. However, the structured architectures SAE and VLAE show improved results compared to their unstructured counterparts.
5

---

Under review as a conference paper at ICLR 2022
3D DD-Shepes Generated Semplee
HPDD Toy Serersted Samplees
HFIIO Real Generated Sa Samples
Samping Nertod
Samplngg Mechod
Sampling Nethod
Habrd Prior
Hybr
Hebrid
Praor
Pdaor
RID
FID
90
MLAF
U1
(a) 3D-Shapes
(b) MP13D Toy
(c) MPI3D Real
IPD Samsko
150
140
123
319
100
Sorrp Methos
hybra Prior
a0
AdM
VAZ
VAE
pusc
VIA-1
BuAZ
VAAE
(d) Celeb-A
(e) RFD
Figure 3: Comparison of the generative models using different sampling methods. Note that our SAE models perform well without having to regularize the latent space towards prior. In fact, even with the conventional "hourglass" architecture (in orange), the hybrid sampling method generates relatively high quality samples.
In comparing the SAE models to the AdaAE architecture, we see that there can be slight penalty in reconstruction quality incurred from separately processing the latent variables. However this is more than made up for in the quality of the generated samples (shown in figure 3 where the SAE models perform significantly better than the baselines. Even the regularized models such VLAE and VAEs usually generate higher quality samples using the hybrid sampling than when sampling from the prior they were trained match. This can qualitatively be observed from figure [b Surprisingly, the AdaAE architecture actually outperforms all other models on CelebA using hybrid sampling. This may be explained by the severity of the information bottleneck experienced when embedding CelebA into only 32 dimensions. Since the AdaAE model produces the best reconstructions, the generated samples have a higher fidelity even though any learned structure between latent dimensions disregarded by the hybrid sampling.
In general, we consider the distribution of the latent variables (i.e., the push-forward of the data distribution into the latent space), then sampling from simple factorized prior introduce at least two types of errors: (1) errors due to not taking into account statistical dependences among latent variables, and (2) errors due to sampling from "holes" in the latent distribution if the prior does not match it everywhere. Whenever (2) is the dominating source of error, hybrid sampling is preferred. Furthermore, learning independent latent variables aligns with the aim towards disentangled represen- tations, while minimizing the divergence between the posterior and prior results in a compromise that does not necessarily promote disentanglement.
4.1 HIERARCHICAL STRUCTURE
To get rough idea of how the representations learned using the structural decoders differ from more conventional architectures, figure 4 shows the one dimensional latent traversals (i.e., each row corresponds to the decoder outputs when incrementally increasing the corresponding latent dimension at a time from the min to the max value observed). The traversals illustrate the hierarchical structure in the representation learned by the SAE models: the information encoded in the first few latent variables can be more nonlinear with respect to the output (pixel) space, as the decoder has more layers to process that information, while the more linear information must be embedded

---

Under review as a conference paper at ICLR 2022
(a) SAE-12
(b) SAE-3
(c) VAE
(d) AdaAE-12 12
Figure 4: Latent traversals of several models trained on 3D-hhapes, in the original order. Note the ordering of the information in the structural decoder models (SAE- 12 and SAE-3) where higher level, nonlinear features (like shape and orientation) are encoded in the first few dimensions, which are located deeper in the network.
in the last few variables. This results in reliable ordering of "high-level" information (such as object shape or camera orientation) first, followed by the "low-level" information (such as color). This means the structural decoder architecture biases the representation to separate and order the information necessary for reconstruction (and generation) meaningful way, and thereby ordering. and potentially even fully disentangling, the underlying factors of variation better.
Figure 3 evaluates how disentangled the representations are using common metries. The table shows the disentanglement scores of the same models discussed above, while the plot on the right sheds light on quality of the representations varies for five different random seeds (used to initialize the network parameters). Most noteworthy is that the SAE-12 12 model consistently achieves very high disentanglement scores. This shows, empirically, that the SAE architecture promotes independence between latent variables (especially SAE-12). We may explain this consequence of splitting up the latent dimensions so that each variable has unique parameterization in the decoder, making different latent variables less likely be processed in the same way.
Unsurprisingly, when there are multiple latent dimensions per variable (like SAE-6, SAE-4, etc.), the dimensions within variable entangled similarly to the baselines like AE adaptive baselines. basis, the
Since all of these disentanglment metrics are computed on a dimeenionn-b--dieesiii resulting scores systematically underestimated. Qualitatively, these SAE models still achieve the same consistent ordering of causal mechanisms, can be seen from figure 4b
For real world demonstration of how well SAE models are able to order information in the latent space, figure 6a shows generated CelebA samples when varying only a quarter of the latent variables time. The labels on the left are empirically chosen by the authors describe roughly what kind of semantic information that quarter of the latent space appears to contain. Although the the inductive biases are not strong enough fully disentangle the factors variation into individual latent dimensions. The hierarchical representation reliably learns diffuse kind of disentanglement where information pertaining to ligeer-level features tend to encoded the first few dimensions while lower level factors of variation show up towards the last few dimensions.
SAE models achieve this structured disentanglement using the Str-Tfm layers as opposed to the standard Ada-In layers. Each Str-Tfm layer only has access one of the latent variables which is not directly seen by any other part of the decoder. contrast, the Ada-In In layers used by the AdaAE allows information from anywhere in the latent vector to leak into any part of the decoder. Consequently, the AdaAE does not disentangle the representation at all (although achieves impressive results for reconstruction nonetheless).

---

Under review as a conference paper at ICLR 2022
SAE12
Model
DCI-D
IRS
MIG
ModExp
VLAE12
AdaAE12
SAE-12
0.999
0.855
0.586
0.968
IRS
SAE-6
0.815
0.712
0.133
0.969
MAE
VAE
SAE-4
0.693
0.560
0.218
0.918
WAE
SAE12
VLAE-12
0.829
0.676
0.662
0.929
VLAE12
VLAE-6
0.785
0.689
0.326
0.929
DCI-D
AdaAE12
VLAE-4
0.690
0.544
0.282
0.900
IVAE
AdaAE-12
0.272
0.477
0.046
0.862
WAE
SAE12
AE
0.326
0.610
0.093
0.880
VLAE12
VAE
0.441
0.712
0.252
0.904
MIG
AdaAE12
AE
IVAE
0.196
0.640
0.107
0.834
WAE
0.205
0.640
0.057
0.958
WAE
0.0
0.2
0.4
0.6
Figure Disentanglement scores for 3D-Shapes. The DCI-D metric corresponds to the DCI- disentanglement score (Eastwood and Williams 2018). IRS is the Interventional Robustness Score (Suter et al. 2018b). MIG is the Mutual Information Gap (Chen et al., 2018), and Mod- (Ridgeway and Mozer 2018) (for all these metrics
Exp refers to the Modularity Explicitness higher better). The figure on the right shows how the scores vary across five models with different random seeds "x"s (and the resulting mean and standard deviation as lines). Note that not only does the SAE architecture consistently achieve high disentanglement scores, but its performance is also less sensitive the random seed.
Oiterattnn Exprosss
friee
Wede
Here hybridized chunks of latent to show how different aspects generated image can be affected using our SAE-16
Proe
resulting
libed
architecture and hybrid sampling technique. For each the corresponding quarter of latent di- mensions (8/32) are hybridized while the remaining 3/4 are fixed. This shows how the SAE architecture is able to order partially disentangled factors of variation from high-level (more nonlinear, like facial expressions and fea- tures) to low-level (such color/lighting) without any additional regularization or supervision.
section 2.3
(b) Samples generated using hybrid and prior- based sampling using several models trained CelebA. Note that the hybrid sampling tends to produce relatively high quality samples both our proposed SAE and AdaAE architectures as well as basclines.
Figure 6: CelebA Controllable Generation and Sampling Comparison
4.2 EXTRAPOLATION
Perhaps unsurprisingly, none of models were particularly adept at zero-shot extrapolation to observa- tions that were not in the initial training data, as consistent with Schoot et al. (2021). Comparing the first two columns the right of figure / without any update, the reconstructed images filter out the novel information the sample (in this case, ball shape), and instead reconstruct similar sample seen during training (a cylinder).
8

---

Under review as a conference paper at ICLR 2022
If only the encoder is updated frozen, then the reconstruction performance increases somewhat, but some deformations and artifacts become visible in the reconstruction. This suggests the frozen decoder struggles to adequately extrapolate, even when the encoder extends the representation to include the ball.
observations with the additional shape while the decoder is
In contrast, when only the decoder is updated, the reconstructions qualitatively look much more similar to the original observations. Although the encoder can be expected to generally filter out any information has not been trained encode into the latent space due the bottleneck, the representation may still extrapolate somewhat provided the decoder can reconstruct any novel features. This underscores the importance of focusing on carefully designing and training the decoder, as embodied by the SAEs, because the decoder is not able to extrapolate as well as the encoder.
Model
Neither
Encoder
Decoder
Both
SAE-12
13.21
7.7
0).42
0.34
SAE-6
13.35
7.99
0.52
0.36
VLAE-12
18.37
7.69
1.55
0.62
VAE
12.97
8.78
0.44
0.46
IVAE
15.49
8.31
1.35
0.52
AE
11.81
7.31
0.38
0.35
WAE
11.68
7.87
0.37
0.35
The table shows the average reconstruction еrror (MSE x1000) on novel observations (example shown on the right) after updating either the encoder, decoder, both, or neither in the extrapolation setting (see section 4.2). Note that all models perform significantly better when updating the decoder than the encoder, and reach a reconstruction quality that is almost indistinguishable from the model when updating both the encoder and decoder. Furthermore, note that the SAE-12 generally outperforms the all variational basclines, suggesting the aggressive regularization VAEs makes updating the representation more difficult.
Figure 7:
CONCLUSION
While VAEs provide a principled approach generative modeling with autoencoders, in practice, the regularization narrows the information bottleneck by penalizing any dependence of the posterior on the observation (Hoffman and Johnson 2016 Tolstikhin al. 2018), resulting in trade-off between reconstruction quality and matching the prior. While this encourages more compact and possibly even disentangled representation, it also tends to result blurry generated samples and relatively poor reconstructions.
This motivated us to look for an alternate sampling method that does not require aggressive regulariza- tion VAžs. Our hybrid sampling technique relies on independence between latent variables instead of expecting the learned posterior to match a (relatively unstructured) prior. This effcetively unifies the goals of achieving disentangled and samplable representation, thereby enabling challenging downstream tasks such as controllable generation.
To that end, propose the structural autoencoder architecture inspired by structural causal models, which orders information in the latent space, while also, as shown by our experiments, encouraging independence. Notably, does so without additional loss terms regularization. The SAE architec- ture produces high quality reconstructions and generated samples, improving extrapolation, as well as achieving a sigmificant degree of disentanglement across a variety of datasets.
While it is encouraging how far one can get with a suitable architectural bias, future work should assay whether the learned models can be structured further by more explicit forms of causal training. For instance, we could explicitly encourage, that hybridization produce realistic samples using an adversarial discriminator, that across domain shifts or actions/interventions external to the network. only sparse set of the latent factors change (cf. the Sparse Mechanism Shift Hypothesis, (Schölkopf al. 202T))

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Fortunately, our project for the most part does not have significant ethical concerns. However, the broader field of representation learning and scene generation does, which we address here. Statistical machine learning has in recent years focused heavily on optimizing the performance of systems on supervised i.i.d. problems. The field of causal learning tries to mitigate the heavy reliance on single i..d. datasets and instead attempts to build systems that transfer knowledge to solve many tasks. This often comes in combination with an element of stronger interpretability is an examples thereof). In the long term, we hope that causal learning will help build machines that approach "common sense" and enable rational and fair decision making in a variety of disciplines.
models (our paper above
A small portion of our results uses the CelebA dataset, which is available for non-commercial research purposes (no license provided). It does show images of individuals, and permission may not have been obtained; however, understanding is that usage of these images is permitted for research purposes since the individuals are celebrities. We will remove those experiments from the paper this is considered inappropriate.
We also note that sophisticated generative models can be misused to synthesize or modify images, reinforcing the need to develop methods that allow reliably detect manipulated content. On the other hand, the interpretability that comes with the modularity our method can yield insights into image analysis, and could therefore facilitate better explanations for model decisions.
REPRODUCIBILITY STATEMENT
Especially in projects that make use of deep learning, particular care must be given the experimental design and supplementary materials to ensure reproducibility of the methods and results.
For the experimental design, have conducted extensive hyperparameter searches to identify which high-level architectural choices consistently perform best. We designed all the models (those using our proposed methods and all baselines) use as many of the same hyperparameters as possible, including adjusting the number of trainable parameters to be comparable. Specifically, we used greedy propose-reject algorithm: each model agnostic hyperparameter setting (primarily architectural and optimization choices discussed in the supplement) proposed was tested on one dataset (in this case 3D-shapes) and only accepted
improved validation set results for all models. This same process was then repeated (to lesser extent) on the other datasets. Next, the model agnostic settings were frozen, and model specific hyperparameters (such as the depth and width of the MLPs in the Str-Tfm layers, VLAE ladder rungs, or 3 in the VAEE) were optimized using the same criterion of the optimization objective on the validation set. Finally, we include results on replica models with different random seeds for the 3D-Shappes dataset to make method.
results not outliers for each
To ensure that our experiments can be reproduced and that interested parties extend our work, have made all code pertaining to this project available online (although the link is anonymized for the review process). For the reviewers, the online repository as well as the code provided for review has been tested and includes walk through guide (a.k.a. "README") to make installation and setup as easy as possible. Although we do not provide any of the dataset we have used for our experiments, they are all freely available for download online (see the corresponding references for more information).
include copy of code supplementary material. Both
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pages 270-279. Springer, 2018.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongho Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. arXiv preprint 19X 1.02685, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-155 1828, 2013.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based representation learning. arXiv preprint arXiv: (a/.:000200009 2018.
Emmanuel Bengio, Valentin Thomas, Joelle Pineau, Doina Precup, and Yoshua Bengio. Independently controllable features, 2017. URL URLarxivv 1703001118
Adityanarayanan Radhakrishnan, Karren Yang, Mikhail Belkin, and Caroline Uhler. Memorization in overparameterized autoencoders. arXiv preprint arXiv:/810. 10333, 2018.
Dana H Ballard. Modular learning in neural networks. In AAAI, pages 279-284, 1987.
Sascha Lange and Martin Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In The 2010 International Joint Conference Neural Networks (IJCNN), pages 1-8. IEEE, 2010.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. arXiv preprint arXiv: 1811:1811 2018
Valentin Khrulkov, Leyla Mirvakhabova, Ivan Oseledets, and Artem Babenko. Disentangled repre- sentations from non-disentangled models. arXiv preprint arXiv:2 102.0644, 2021.
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. arXiv preprint arXiv:1910.09772, 2019.
Shang-Fu Fu Chen, Jia-Wei Yan, Ya-Fan Su, and Yu- Yu-Chiang Frank Wang. Deep representation decom- position for feature disentanglement. arXiv preprint 2rXiv:2011.00785 2020.
Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debnath, Anjul Patney, Ankit Patel, and Animashree Anandkumar. Semi-supervised stylegan for disentanglement learning. In International Conference on Machine Learning, pages 7360-7369. PMLR, 2020.
Emile Mathieu, Tom Rainforth, Nana Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In International Conference on Machine Learning, pages 4402-4412
PMLR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 1998-6008, 2017.
Gihyun Kwon and Jong Chul Ye. Diagonal attention and style-based gan for content-style disentan- glement in image generation and translation. arXiv preprint 16X1v:2103.16146, 2021.
J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, NY. 2nd edition, 2009.
J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA, 2017.
B. Schölkopf. Causality for machine learning, 2019. arXiv:1911.10500.
Christos Louizos, Uri Shalit, Joris Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. arXiv preprint arXiv: 7705. 08821, 2017.
11

---

Under review as a conference paper at ICLR 2022
Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Representa- tion learning via invariant causal mechanisms. arXiv preprint arXiv:2010.07922, 2020.
Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang. Disentangled generative causal representation learning. arXiv preprint arXiv:2010.02637, 2020.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes, 2013a. URL vrxiv;
1312.6114
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning basic visual concepts
with a constrained variational framework. ICLR, 2(5);6, 2017. URL https:
semanticschol oaapaaac.....aa..ee///..00006333 pdf
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look at the unsupervised learning of disentangled representations and their evaluation. Journal of Machine Learning Research (JMLR), 2020a.
Sharon Zhou, Eric Zelikman, Fred Lu, Andrew Y Ng, and Stefano Ermon. Evaluating the disentan- glement of deep generative models through manifold topology. arXiv preprint arXiv:2006.0344 0036000000000 2020.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 3-VAE. arXiv preprini arXvv:/804.03599, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint 1rX.v:1802.05983 2018. URL TRRL/ttpss///xxxx.ooo///////////
Jan Stühmer, Richard Turner, and Sebastian Nowozin. Independent subspace analysis for unsupervised learning of disentangled representations. In International Conference on Artificial Intelligence and Statistics, pages 1200-12 12100111...11111 PMLR, 2020.
I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schölkopf. Wasserstein auto-encoders. In 6th International Conference on Learning Representations (ICLR), May 2018. URL /tttpss://peereeeeee net/Eorum?in 7nl 0b
Paul K Rubenstein, Bernhard Schoelkopf, and Ilya Tolstikhin. On the latent space of Wasserstein auto-encoders. arXiv preprint arXiv: 2018.
Jun Han, Martin Renqiang Min, Ligong Han, Li Erran Li, and Xuan Zhang. Disentangled recurrent wasserstein autoencoder. arXiv preprint arXiv:2101.074990 2021.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep genera- tive models. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 4091-4099. JMLR. 2017.
Michel Besserve, Arash Mchrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. arXiv preprint arXiv:1812.03255, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative
adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. URL. / A_ Style- Based Generator www.eratttorArchtttcuuu...
CVPR 00FR_2011 2019/papers/Karta For Generative Generative_.lvveraaaiaa Adversarial Networks CVPR 2019 paper. pdf
D. Lopez-Paz, R. Nishihara, S. Chintala, B. Schölkopf, and L.. Bottou. Discovering causal signals in images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 58-66, 2017.
Raphael Suter, Djorde Miladinovié, Bernhard Schölkopf. and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. arXiv preprint arXiv: 18/ I. 1.00077, 2018a.
12

---

Under review as a conference paper at ICLR 2022
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference Machine Learning, pages 6348-6359, PMLR, 2020b.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. arXiv preprint arXiv: 1IXvv:190300588, 2019.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Understanding posterior collapse in generative latent variable models. 2019.
Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1. page 2, 2016.
Chris Burgess and Hyunjik Kim. 3d shapes dataset, 2018.
Muhammad Waleed Gondal, Manuel Wüthrich, Djorde Miladinovié, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation the real world: new disentanglement dataset. arXiv preprint arXiv: 0e0./9900022200 03292, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Andrea Dittadi, Frederik Träuble. Francesco Locatello, Manuel Wüthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. arXiv preprint arXiv:2010. 14407, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances neural information processing systems, 2017.
Will Williams, Sam Ringer, Tom Ash, John Hughes, David MacLeod, and Jamie Dougherty. Hierar- chical quantized autoencoders. arXiv preprini arXiv:2002.08111, 2020.
Diganta Misra. Mish: A self regularized non-monotonic neural activation function. arXiv preprint N9NVv:900.00000 4, 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 1rXiv:1312.6114 2013b.
arXiv preprint
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. 2018.
Raphael Suter, Dorde Miladinovic, Stefan Bauer, and Bernhard Schölkopf. Interventional robustness of deep latent variable models. stat, 1050:31, 2018b.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. arXiv preprint arXiv: 1802 2018.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. arXiv preprint arXiv: 0802.053 2018.
Lukas Schott, Julius Kügelgen, Frederik Träuble, Peter Gehler, Chris Russell, Matthias Bethge, Bernhard Schölkopf, Francesco Locatello, and Wieland Brendel. Visual representation learning does generalize strongly within the same domain. arXiv preprint arXiv:2 2021.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEFF, (09(5)6612-63)) 634, 2021.
13

---

Under review as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vitito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019, URL wtt.tttppp/pppprs..eeeeeeeee htw.apperrreeurrs.cccoeoooooeeee pdf
ibbrry.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00844 2017.
14

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 CAUSAL ORDERING
Suppose we are given nigh-dimeessonal) X = 1X,,..,Xa)) (think of X as an image with pixels Xi,..., Xa), from which we should construct ,I,..., Sp (D < d) as well as causal mechanisms
S: :- ((PA,, (i=1...., D).
(2)
To this end, we first use an encoder fenc: RR - RD taking X to latent bottleneck representation
comprising U = (Uz...., UD) The next step is a map f(U) implementing the structural assignments fI....;fD as function of U. We construct depends only on U,. In the step, we evaluate any node j which depends only on its U, and possibly other variables that have already been computed. We iterate until there are no nodes left. This terminates (since the graph is acyclic) and yields a unique f(U), but the order ((i) in which the f. get evaluated need not be unique. It is referred as a causal or topological ordering (Peters et al. (017). satisfying (i) < r(j) whenever j is descendant of i. This embeds the SCM into the network starting from the bottleneck U = (UI,..., UD) with the U; feeding into subsequent computation layers according to causal ordering. This structure reflects the fact that the root node(s) in the DAG only depend on "their" noise variables, while later ones depend on their noise and those of their parents, and
follows: we evaluate the fa of a root node i, i.e., fa
Finally, apply decoder fdee RDD Rd. The system can be trained using reconstruction error to satisfy fdec o f fenc ze id on the observed images.
Recall that for causally sufficient system, the set of noises U1..... Un, assumed to be jointly independent. If, in contrast, only a subset of the causal variables are modelled, then the noises will in the generic case be dependent. We would expect that the architectural bias implemented by the structural decoder, however, may still be a sensible one.
A.2 TRAINING PROCEDURE
A.2.1 ARCHITECTURE DETAILS
As described in the main paper, the basic convolutional backbone of all models is the same. For the smaller datasets, 3D-Shapes and the MPI3D datasets (where observations are 64x64 pixels), the encoder and decoder each have 12 convolutional blocks. Each block has a convolutional layer with 64 channels and kernel size of 3x3 and stride of (unless otherwise specified), followed by group normalization layer and then a MISH nonlinearity (Misra] 2019). In the encoder, the features are downsampled using a 2x2 Max Pooling layer right after the convolution every third layer starting with the first one and the first convolution layer uses kernel size of 5x5. In the decoder, every third convolution layer is immediately preceded by a 2x2 bilinear upsampling. For our structured modules (SAE and AdaAE), the specified number of Str-Tfm layers are placed evenly in between the convolution blocks. For SAE models, the latent space is always split evenly between Str- Str-Tfm layers, and each layer uses a three hidden layer network to process the latent space segment into the scale and bias vectors which are then applied to all pixels individually of the features. For the VLAE models, the inference and generative ladder rungs each also have three hidden layers to process the features into and out of the separate latent space segments respectively.
While the latent space was always 12 dimensional for 3D-Shapes and MPI3D, for Celeb-A use 32 dimensional latent space. For Celeb-A, we also expand the 12 block backbone to 16 blocks and double the filters per convolution layer to 128. The exact sizes and connectivity of the models can be seen in the configuration files of a the attached code, but overall, each of the 3D-Shapes and MPI3D models have approximately 1-1. 1.22M trainable parameters, while for CelebA the models have 6-7M parameters.
A.2.2 TRAINING DETAILS
All models used the same training hyperparameters, which included using an Adam optimizer with a learning rate of 0.0005 and momentum parameters of 31 0.9 and B2 0.999. For the smaller datasets -3D-Shapes MP13D) the models were trained for 100k iterations and a batch size of 128, while for Celeb-A the models were trained for 200k iterations and batch size of 32. The
15

---

Under review as a conference paper at ICLR 2022
hyperparameters for the RFD dataset the same for Celeb-A, except that the number of channels per convolution layer was doubled and the learning rate was decreased by a factor of 10.
The models implemented using Pytorch (Paszke et al. 2019) and were trained on the in in-house computing cluster using Nvidia V100 32GB GPUs, so that training a single model takes about 3-4 hours on the smaller datasets and 7-10 hours for CelebA.
For the V-E,,s, {2. 6, 8, 16) were tested on 3D-Shapes and MPISD, and the model with the smallest loss the validation set was used for subsequent analysis, which was B 2 for all datasets.
A.3
ADDITIONAL RESULTS
A.3.1 3D-SHAPES
3D 3D-Shapes Reconstruction Quality
50
Model
SAE-12
AE
45
SAE-6
VAE
Reconstruction FID Score
SAE-4
BVAE
40
VLAE-12
WAE
AdaAE-12
35
20
30
18
25
16
20
14
15
12
3460
3480
3500
3520
3540
3560
3580
Reconstruction Loss
Figure 8: 3D-Shapes reconstruction quality comparison between models using the reconstruction loss (binary cross entropy) and the Fréchet Inception Distance (FID) (Heusel et al.) 2017) between the original and reconstructed observations (lower is better for both). Each "x" is model trained with a unique random seed using the architectur/reguulaizttioo corresponding to the color. The performance of all the seeds are averaged and plotted as circles "o". Firstly, this plot shows nicely how the reconstruction FID (y-axis) can help quantify the quality of the reconstructed samples when a pixelwise comparison (x-axis) saturated. Next, the multiple seeds help identify two different regimes of performance, one including all the models between the AE (orange) and the SAE-12 (dark blue), while the other regime starts at the 3-VAE (pink) and reaches the VLAFF- 12 (red). These regimes separate the models that the variational regularization loss from the models that only use a reconstruction loss (or a regularization on the aggregated posterior like the WAE), Lastly, the SAE-12 consistently performs best both in terms of reconstruction FID and the pixelwise loss.
16

---

Under review as a conference paper at ICLR 2022
3D-Shapes Generated Samples
80
Sampling Method
X
Hybrid
70
Prior
60
50
FID Score
40
30
20
Model
SAE-12
AE
BVAE
10
VLAE-12
VAE
WAE
AdaAE-12
0
Figure 9: Comparison of the hybrid and prior-based sampling method for all different models. Each bar corresponds to unique random seed. (lower is best)
Model
DCI-d
IRS
MIG
SAP
ModExp
DCI-c
SAE-12
0.999
0.855
0.586
0.172
0.968
0.900
SAE-6
0.815
0.712
0.133
0.029
0.969
0.687
SAE-4
0.693
0.560
0.218
0.118
0.918
0.614
SAE-3
0.765
0.650
0.269
0.056
0.941
0.617
SAE-2
0.577
0.580
0.220
0.078
0.929
0.492
VLAE-12
0.829
0.676
0.662
0.183
0.929
0.832
VLAE-6
0.785
0.689
0.326
0.081
0.929
0.728
VLAE-4
0.690
0.544
0.282
0.115
0.900
0.597
VLAE-3
0.673
0.593
0.365
0.052
0.927
0.699
VLAE-2
0.522
0.548
0.314
0.038
0.870
0.607
AdaAE-12
0.272
0.477
0.046
0.024
0.862
0.210
AE
0.326
0.610
0.093
0.060
0.880
0.260
VAE
0.441
0.712
0.252
0.099
0.904
0.516
OVAE
0.196
0.640
0.107
0.047
0.834
0.215
WAE
0.205
0.640
0.057
0.036
0.958
0.162
Table Disentanglement and Completeness scores for 3D-Shapes. The DCI-d metric corresponds to the DCC-disentanglement score and DCI-c to the completeness score (Eastwood and Williams, 2018), IRS similar disentanglement metric (Suter et al.) 2018b), the MIG the Mutual Information Gap (Chen al. 2018), the SAP score is the Separated Attribute Predictability score (Kumar et al.) 2017), and ModExp refers to the Modularity Explicitness score (Ridgeway and Mozer 2018). (for all these metrics higher is better)
17

---

Under review as a conference paper at ICLR 2022
SAE-12 12
VLAE-12
AdaAE-12
IRS
AE
VAE
BVAE
SAE-11 1-
VLAE-11 12
DCI-D
AdaAE-1 12
AE
VAE
BVAE
WAE
SAE-17
VLAE-12
AdaAE-12
MIG
AE
VAE
BVAE
WAE
SAE-1
VLAE- 12
AdaAE 12
SAP
AE
VAE
BVAE
SAE-1 12
ModExp
VLAE-12 12
AdaAE-12
AE
VAE
BVAE
WAE
0.0
0.2
0.4
0.6
0.8
1.0
Figure 10: Several disentanglement metries for all the models and each of the seeds. (for all these metrics higher is better)
18

---

Under review as a conference paper at ICLR 2022
(a) SAE-12
(b) SAF-6
(c) SAE-4
(d) SAE-3
(c) SAE-2
(f) AdaAE-12
19

---

Under review as a conference paper at ICLR 2022
g VLAF-12
(h) VLAE-6
(i) VLAE-
(i) VAE
(k) 3-VAE
(I) AE
Figure 11: Latent Traversals of several models for 3D-Shapes. Each row shows the generated image when varying the corresponding latent dimension while fixing the rest of the latent vector. For the SAE and VLAE models, the groups of dimensions that are fed into the same Str- ft--Tim layer (or ladder rung) are grouped together. Note the disentangled segments achieved by the SAE models and the consistent ordering of factors of variation.
A.3.2 EXTRAPOLATION
We present results variant of the exptrapolation experiment discussed in section 4.2 modifying the shape in the initial training dataset, we remove the two most extreme camera angles in either direction (removing 4/15 of the full dataset).
Instead of
As seen from figure 12 in this setting the warping and generally lower fidelity experienced by only updating the encoder compared to updating the decoder is very apparent.
20

---

Under review as a conference paper at ICLR 2022
Model
Neither
Encoder
Decoder
Both
SAE-12
4.88
2.65
0.77
0.43
VLAE-12
6.91
3.83
1.61
0.77
AdaAE-12
4.8
2.93
0.59
0.41
AE
4.98
2.94
0.62
0.45
WAE
4.79
3.07
0.69
0,44
VAE
5.17
3.09
1.01
0.51
3VAE
5.7
3.97
1.5
0.82
Figure 12: Same figure 7 except for the camera angle setting. Although the results are generally very consistent, note how much better the SAE-12 model performs than the VLAE-12 this setting.
A.3.3 MP13D-ToY
Model
DCI-d
IRS
MIG
SAP
ModExp
DCI-c
SAE-12
0.642
0.570
0.487
0.287
0.938
0.566
SAE-6
0.454
0.553
0.094
0.060
0.918
0.404
SAE-4
0.316
0.535
0.119
0.078
0.919
0.283
SAE-3
0.380
0.461
0.113
0.068
0.933
0.359
SAE-2
0.252
0.557
0.030
0.011
0.908
0.252
VLAE-12
0.414
0.667
0.323
0.201
0.909
0.518
VLAE-6
0.436
0.623
0.277
0.102
0.927
0.541
VLAE-
0.415
0.591
0.182
0.120
0.936
0.546
VLAE-3
0.301
0.557
0.151
0.084
0.877
0.380
VLAE-2
0.262
0.634
0.130
0.058
0.936
0.311
AdaAE-12 12
0.208
0.546
0.080
0.061
0.919
0.191
AE
0.186
0.632
0.043
0.020
0.911
0.170
VAE
0.093
0.621
0.078
0.044
0.861
0.108
VAE
0.046
0.987
0.004
0.051
0.998
0.051
WAE
0.203
0.633
0.028
0.023
0.904
0.171
Table Disentanglement and Completeness scores for MPI3D-Toy. (for all these metrics higher is
better)
21

---

Under review as a conference paper at ICLR 2022
(a) SAE-12
(b) SAF-6
(c) SAE-4
I
(d) SAE-3
(c) SAF-2
(f) AdaAE-12
22

---

Under review as a conference paper at ICLR 2022
g VLAF-12
(h) VLAE-6
ii VLAE-4
(j) VAE
(k) 3-VAE
()) AE
Figure 13: Latent Traversals several models for MPI3D-Toy. Fach row shows the generated image when varying the corresponding latent dimension while fixing the rest of the latent vector. For the SAE and VLAE models, the groups of dimensions that are fed into the same Str- Tim layer (or ladder rung) are grouped together. Note the disentangled segments achieved by the SAE models and the consistent ordering of factors of variation.
23

---

Under review as a conference paper at ICLR 2022
A.3.4 MPI3D-SIM
Model
DCI-d
IRS
MIG
SAP
ModExp
DCI-c
SAE-12
0.411
0.508
0.238
0.153
0.930
0.389
SAE-6
0.294
0.479
0.052
0.037
0.928
0,300
SAE-4
0.380
0.519
0.139
0.097
0.927
0.356
SAE-3
0.269
0.482
0.046
0.027
0.902
0.256
SAE-2
0.119
0.536
0.019
0.017
0.930
0.121
VLAE-12
0.220
0.634
0.093
0.064
0.863
0.282
VLAE-6
0.290
0.575
0.051
0.041
0.797
0.279
VLAE-
0.372
0.669
0.174
0.167
0.945
0.394
VLAE-3
0.242
0,554
0.135
0.091
0.892
0,302
VLAE-2
0.127
0.662
0.060
0.061
0.868
0.4
AdaAE-12
0.159
0.481
0.022
0.013
0.893
0.129
AE
0.157
0.526
0.033
0.026
0.855
0.143
VAE
0.070
0.850
0.056
0.032
0.828
0.079
VAA
0.060
0.850
0.054
0.014
0.926
0.065
WAE
0.129
0.548
0.033
0.018
0.881
0.120
Table Disentanglement and Completeness scores for MPI3D-Sim
(for all these metrics higher is
better)
24

---

Under review as a conference paper at ICLR 2022
SAE-12
(b) SAF-6
SAE-4
(d) SAE-3
(c) SAE-2
(f) AdaAE-12
25

---

Under review as a conference paper at ICLR 2022
g VLAE-12
(h) VLAE-6
(i) VLAE-
4
4
(i) VAE
(k) 3-VAA
()) AE
Figure 14: Latent Traversals of several models for MPI3D-Sim. Each row shows the generated image when varying the corresponding latent dimension while fixing the rest of the latent vector. For the SAE and VLAE models, the groups of dimensions that are fed into the same Str- Tim layer (or ladder rung) are grouped together. Note the disentangled segments achieved by the SAE models and the consistent ordering of factors of variation.
26

---

Under review as a conference paper at ICLR 2022
A.3.5 MPI3D-REAL
Model
DCI-d
IRS
MIG
SAP
ModExp
DCI-c
SAE-12
0.314
0.543
0.118
0.079
0.908
0.272
SAE-6
0.295
0.535
0.074
0.053
0.879
0.277
SAE-4
0.270
0.523
0.085
0.034
0.876
0.264
SAE-3
0.306
0.495
0.119
0.072
0.884
0.300
SAE-2
0.130
0.527
0.025
0.020
0.908
0.130
VLAE-12
0.291
0.579
0.217
0.129
0.914
0.332
VLAE-6
0.410
0.668
0.279
0.183
0.863
0.414
VLAE-
0.235
0.513
0.113
0.046
0.899
0.223
VLAE-3
0.249
0.611
0.147
0.074
0.897
0.255
VLAE-2
0.126
0.644
0.074
0.037
0.914
0.170
AdaAE-12
0.164
0.530
0.034
0.017
0.952
0.147
AE
0.143
0.563
0.048
0.030
0.858
0.132
VAE
0.080
0.602
0.020
0.004
0.875
0.085
VAE
0.090
0.659
0.021
0.014
0.869
0.108
WAE
0.159
0.587
0.042
0.038
0.837
0.150
Table 4: Disentanglement and Completeness scores for MP13D-Real. (for all these metrics higher is better)
27

---

Under review as a conference paper at ICLR 2022
(a) SAE-12
(b) SAF-6
(c) SAE-4
(d) SAE-3
(c) SAE-2
(f) AdaAF-12
28

---

Under review as a conference paper at ICLR 2022
g VLAE-12
(h) VLAE-6
ii) VLAE-4
(i) VAE
(k) 3-VAA
()) AE
Figure 15: Latent Traversals of several models for MP13D-Real. Each row shows the generated image when varying the corresponding latent dimension while fixing the rest of the latent vector. For the SAE and VLAE models, the groups of dimensions that are fed into the same Str-Tim layer (or ladder rung) are grouped together. Note the disentangled segments achieved by the SAE models and the consistent ordering of factors of variation.
29

---

Under review as a conference paper at ICLR 2022
30

---

Under review as a conference paper at ICLR 2022
A.3.6 CELEB-A
(a) Original samples (from test set)
(b) SAB-16 16 Reconstructions
(c) AdaAF-16 Reconstructions
31

---

Under review as a conference paper at ICLR 2022
(d) VLAE-16 Reconstructions
(c) VAF Reconstructions
(l) AE Reconstructions
32

---

Under review as a conference paper at ICLR 2022
(a) SAE-16 16 Hybrid Sampling
(b) AdaAF-16 16 Hybrid Sampling
(c) VLAE-16 Hybrid Sampling
33

---

Under review as a conference paper at ICLR 2022
(d) VLAE-16 Prior Sampling
(c) VAE Hybrid Sampling
(f) VAE Prior Sampling
34

---

Under review as a conference paper at ICLR 2022
A.3.7 RFD
(a) SAE-16
(b) VLAE-16
35

---

Under review as a conference paper at ICLR 2022
(c) AdaAE-16
(d) VAE
Figure 18: Latent traversals for the RFD dataset. Each row corresponds to ID traversal of the corresponding latent dimension while the other latent dimensions are fixed. Note the ordering of information in the more structured models like the SAE-16 16 and VLAE-16.
36


Reviews:
[{"review_id": "UcydiH3mCGA", "reviewer": "Reviewer_UHD2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is fairly easy to follow, the motivation is generally good, the problem is important (generation and structured unsupervised latent structure / representation learning). The experiments are well-motivated and support their story.\n\nWeaknesses:\nThe main idea of the paper, independence of the latent variables, only seems empirically supported, but it's hard to find convincing motivation in the paper that the precise way they use the latents in generation / algorithm *should* lead to independent latents. This is a minor point I think, but there are some arguments in the paper I either didn't follow or didn't convince me that the results *should* make sense.\n\ne.g., S2.2: \"This architectural asymmetry... encourages statistical independence...\" Can you can explain this more clearly?\n\nSimilarly, with the hybrid sampling, could you clarify: \"This is consistent with the causal perspective of the latent variables as independent noises...\" Do you mean that because we observe this sampling works so well, that this supports the idea that these latents are independent?\n\nOther comments / clarifications:\nSo the latents are a flat vector output of the encoder correct? When you split this latent vector, how do you distribute it amongst the layers which generating when the splits are different? Do you give every layer a latent or if there are less splits do only some get latents?\n\nIs AdaAE your method? From what I can tell, you are just using all of latent vector (not splitting) at every layer? Since this works so well, I'm curious if it's just a matter of hyperparameters that make it inferior sometimes to SAE.\n\nAny intuitions about the ordering of performance of SAE-X? For instance, in Rec FID, SAE-6 performs worse than SAE-12 and SAE-4. Why is this?\n\nThe FID scores are distributional, generated (e.g., using hybrid sampling) vs test, correct? I wonder how close the train distribution is to the samples in FID. Is there any possibility, because you're sampling latents using the training examples (and random mixing) that the generated examples closely resemble training instances? I feel like this needs to be tested.\n\n", "summary_of_the_paper": "This paper structures how latents are used in an autoencoder to improve its performance. They are motivated by causal structure and independence of latent variables. They also sample using a sort of discrete mixup between latent codes from pushing forward data, and show various improvements there.", "summary_of_the_review": "I'm leaning accept, but there are many points of clarification that would need to be sorted out before acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RTlVfU5Uvr", "reviewer": "Reviewer_EdoF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Good points:\n\n* The proposed model gets good empirical results. The authors did extensive experiments and ablation studies. The results and explanations look promising.\n* Overall, this paper is well written and is easy-to-follow. The authors explained the motivation and the techniques clearly.  \n* The authors provide their codes in the supplimentaries. At a glance, the code have a clear structure and there's a good README that provide comprehensible instructions.\n\nMinor issues:\n\n* I feel like Figure 3 can be presented in a better way. There are just too many bars with the same color and it takes a lot of time for me to tell the bar annotations ('X', 'O').\n* Similar issue for Figure 5. I had to go back and forth between the figure and the first paragraph in section 4.1 multiple times to understand the message the authors wanted to deliver. It could be probably better if the texts and figures are close and there are some text annotations or marks in the image to make it easier for readers.\n", "summary_of_the_paper": "The paper proposed a new structure call SAE (structural autoencoder) along with a new sampling technical, named hybrid sampling, to train a encoder-decoder generative model. The authors conducted extensive experiments to show the proposed model and sampling method lead to much better results for generating higher quality images and get more meaningful/smooth images in experiments involving feature disentanglements and extrapolations.  ", "summary_of_the_review": "This paper is well written and its claims are well supported by the experiments, analysis, discussions. I believe this paper will bring values to the broader researcher community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QkGagu5cqc6", "reviewer": "Reviewer_Bkes", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The goal of this paper is important: having architectures that, without carefully engineered regularization losses, learn diverse disentangled representations would be a significant contribution to the literature. Moreover, the approach is fairly simple to implement and natural. However, there are several aspects of this paper that leave me somewhat unconvinced that this work actually accomplishes that goal.\n\nFirst, and most crucially, while the paper emphasizes the goal of disentanglement by its architecture, there is no part of the paper meant to convince the reader that the method accomplishes this aim. Specifically, there is no theoretical justification for why this model should result in the disentangled representations that it advertises. As noted in Subsection 2.2, there is no guarantee of correctly disentangling the true factors of variation (something that’s generally been accepted with unsupervised models following Locatello et al. 2018). Which precise definition of disentanglement is the paper operating under, what is its relationship to the causal ordering in this paper, and how does this method accomplish it? \n\nSecond, while the evaluation is very comprehensive in some aspects (five disentanglement metrics, six (distinct) alternative models, four sources of datasets, with each model trained several times), there are several aspects that take away from this evaluation. \nThe disentanglement metrics presented are not especially convincing. Notably, the MIG scores reported appear to disagree with prior work - for example, 0.107 for $\\beta$-VAE is substantially lower than the performance reported in Khrulkov et al. 2021 (~0.2). Little detail is given about the specific hyperparameter choices for the models which are used for comparison, so it’s difficult to assess whether the comparison is fair. Moreover, the primary proposed model (SAE) is not the best-reported model in terms of MIG for any datasets besides the MPI3D toy and simulated datasets, which is somewhat surprising as Locatello et al. (2018) suggested that DCI-D and MIG are broadly very correlated.\nThe metrics which are most emphasized in the paper seem to be FID and Reconstruction FID. Figure 6 is the only place outside of the appendix where any non-FID metric is shown or discussed. However, FID is broadly inappropriate as a disentanglement metric and is less applicable the less similar a dataset is to ImageNet, as noted in Section 4.2.1 of Barratt and Sharma (2018). FID has been widely criticized by many VAE papers, such as in Section 5.2.1 of Razavi et al 2019, which makes the choice of emphasis on this metric more difficult to understand. Furthermore, in terms of FID, the model proposed by this paper appears visibly less performant than others such as StyleGAN, VQ-VAE, VQ-GAN, BigGAN, LOGAN. \nFID but not disentanglement metrics are reported on Celeb-A or RFD, which are the only datasets evaluated in this paper not consisting of 3D shapes.  There are reasonable arguments which one can make for not evaluating these metrics on these datasets, but I feel like a sentence on this point would be helpful. In addition, if they are excluded due to the general difficulty in interpreting their underlying factors of variation, a disentanglement dataset such as Cars3D (Reed et al. 2015) could be useful in bridging that gap.\nThe models from prior work which are compared against are VAE (Kingma and Welling 2013), $\\beta$-VAE (Higgins et al. 2017), VLAE (Zhao et al. 2017), and WAE (Tolstikhin et al. 2018). However, notably excluded are FactorVAE (Kim and Mnih 2018) or $\\beta$-TCVAE (Chen et al. 2018). Seeing as many of the metrics used here can be calculated on GAN models, it could also have been interesting to see InfoGAN. \n\nThird, the paper discusses Variational Ladder autoencoders (VLAEs) as an inspiration, but the main differences appear to be a different way of injecting the embeddings hierarchically (using Ada-IN layers) and the alternative sampling method/prior - why is this so important? The current argument, as I understand it, is that the variational loss is among the undesirable “aggressive regularization” methods. However, I do not believe that it is traditionally thought of as a disentanglement loss, so this seems fairly unconvincing. I would be interested to see a precise motivation for why this provides an improvement, beyond the general structural advantages of AdaIN as shown in other models. Moreover, there are many ways to enforce causal dependence on an input, many of which are already widely used in the literature. One approach, causal attention masking, appears to also enforce the desired causality constraint and is widely used in the transformer literature (e.g. 3.2.3 of Vaswani et al. 2017). Some variant of causal attention masking has also been applied to generative models, such as “Diagonal Attention and Style-based GAN for Content-Style Disentanglement in Image Generation and Translation” (Kwon et al. 2021) which leveraged a “diagonal spatial attention” layer for hierarchical control of content embeddings. Overall, I’d like to see a much more thorough discussion of the motivation behind the features proposed in this paper. \n\nFinally, I have some concerns over the scalability of this approach. Namely, as the datasets become more complex and necessitate higher-dimensional encodings and deeper models, I would expect the disentanglement benefits of this method to decrease. Specifically, while a causal structure may technically exist here for deeper models and larger embedding spaces, the ability to condition on many prior embeddings may limit their usefulness. The choice to use only 12 and 32 dimensions for the experiments appears to reinforce this. This alone is not a reason to reject this work, as the paper intends to propose a novel architectural approach to disentanglement and further advances would clearly be necessary to make this competitive with the state-of-the-art in generative models in terms of fidelity and diversity - still, it would probably offer an improve understanding of this approach to see a scaling analysis. \n\nA few nitpicks on style: there are many places throughout the paper where references that appear to be intended to be parenthetical are actually in-text (such as Section 1.1, the caption of Table 1, etc. They are easy to find by searching for “(20”. In addition, the paper extensively emphasizes the lack of regularization terms in the loss in the proposed model but it’s not clear exactly what loss was used.", "summary_of_the_paper": "This paper proposes a mechanism to train disentangled generative models without relying on regularization losses. It aims to enforce independence in blocks of the latent representation of a VAE by 1) corresponding different blocks of the latent representation to different depths of the decoder by injecting noise in an Ada-IN-inspired block 2) sampling each block from a fixed learned set of k latent vectors, similar to the codebook of VQ-VAE (Oord et al 2017).", "summary_of_the_review": "The paper has an ambitious goal and some potentially novel ideas but there seem to be serious issues with\n1. Precision: while certainly not every paper in the field needs to be theoretically driven, the work makes claims that would likely benefit from a more rigorous discussion - there is no formal discussion of the intended meaning of disentanglement and no precise argument for why this method would benefit that definition of disentanglement. However, while some additional precision would be helpful, I am mostly evaluating this as an empirical work.\n2. Evaluation:  within the disentanglement (non-FID) metrics and the datasets where a quantitive evaluation of disentanglement is provided, the paper does not consistently show that the proposed approach provides disentanglement advantages over VLAEs. The main text primarily focuses on FID and reconstruction FID which is inappropriate for the datasets used and the way in which the paper presents its contribution. Finally, it would have been helpful to have also seen other standard models like FactorVAE or TCVAE.\n3. Novelty: the main proposed advantage over VLAEs (described as an inspiration) appears to be the lack of variational regularization, but the authors don’t appear to argue that this is a disentanglement loss. It’s unclear to me exactly what argument is being made for the advantage of this approach.\n4. Scaling: it seems that the independence properties should enforce a gradually weaker constraint with deeper models and larger input dimensionalities. Some analysis of the performance of this method as a function of larger input dimensionalities could be beneficial.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7BmiJNTwk-T", "reviewer": "Reviewer_ERAs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong Points:**\n\nThe empirical evaluation has several strong aspects. The authors evaluate their proposed structural autoencoder (SAE) using multiple image datasets of varying complexity. The authors also compare with a wide range of baseline models of varying techniques and latent depth. Experimental results evaluate multiple aspects of the model, including reconstruction and generation performance, disentanglement (via latent traversals and quantitative metrics), and generalization in both the encoder and decoder to unseen data examples. In total, these results demonstrate that SAE obtains a high degree of disentanglement while also yielding high-quality reconstructions, as compared with relevant baselines.\n\nWhile I do not see the extrapolation experiments as particularly essential in demonstrating SAE, these experiments are somewhat novel in systematically evaluating generalization in various components of autoencoder architectures. To the best of my knowledge, these types of experiments do not exist in the current literature. It’s somewhat surprising that the disparity of generalization performance is so large between training the encoder versus the decoder on unseen examples. I see this as an interesting and potentially useful result.\n\nI also see the performance comparison of hybrid and prior sampling (Figure 3) as somewhat interesting and novel, providing another useful analysis for future works.\n\nFor the most part, the paper is clear. Much of the model is described well, results are presented clearly, and the appendix appears to be thorough.\n\n**Weak Points:**\n\nMy largest concern is regarding the validity of some of the design choices in the model. I understand that formulating a proper probabilistic model may not be the authors’ main purpose in developing SAE, however, it’s somewhat difficult to reconcile the presentation of probabilistic models in Section 2.1 with the design choices made in the model. For instance, from what I can tell, the authors use deterministic noise variables without any form of prior distribution. Likewise, the reconstruction is optimized using binary cross entropy, when this is only valid for binary variables, not the RGB values in the image datasets used. To be clear, the authors’ results are still valid. Rather, I’m simply discouraged that the model is not a proper probabilistic model, as this makes it difficult to assess the space of valid design choices.\n\nAnother concern is the novelty of the authors’ insights and the validity of the conclusions drawn. Multiple previous works, some of which the authors cite, have shown that hierarchical latent variable models are capable of learning disentangled, hierarchical representations, e.g., Zhao et al., 2017. Thus, it’s already clear that model architecture can bias representation learning. The authors’ specific claim, then, appears to be that some aspect of the latent independence and structural transform layers is helpful for disentanglement. However, it’s not clear why these techniques necessarily need to be linked, as these seem like independent design choices. One could also feasibly combine SAE with priors, either via fixed or learned priors. Thus, while it’s clear that the presented SAE model performs well, it’s not entirely clear which aspects of the model are responsible or whether techniques from previous works would improve performance further. I remain somewhat skeptical regarding the conclusions drawn by the authors.\n\nSome aspects of the model were not entirely clear to me. Figure 1 presents a helpful diagram of the structural decoder. I had hoped that the authors would include a similar diagram depicting the encoder, as it was not clear whether the $U$ variables were stochastic or deterministic.\n\nThe authors motivate their approach in terms of structural causal models (Section 2.1). However, I don’t see what this adds beyond standard directed graphical models. Motivating the approach from the perspective of causal modeling, in my opinion, only serves to complicate the presentation.\n\n\n**Additional feedback:**\n\nSection 3.2: It’s not clear how the extrapolation sections fit with the rest of the paper. This seems like an orthogonal contribution from the proposed model.\n\nFigure 6: $\\beta$-VAE is lower than VAE on all of the metrics. If $\\beta$ was tuned, as the authors claim, then I wouldn’t expect this to be the case.", "summary_of_the_paper": "This paper presents a hierarchical latent variable model and accompanying sampling procedure for learning disentangled representations. Rather than a latent feature map/vector, the latent variables are used to condition affine transforms in the decoder. The authors combine this with a ‘hybrid’ sampling strategy, effectively sampling from the aggregate approximate posterior. Comparing with several baselines, the authors compare their model in terms of FID and various disentanglement scores. The authors also investigate generalization.", "summary_of_the_review": "This paper has both strengths and weaknesses, as outline above. While aspects of the model presentation, model formulation, and conclusions could be improved, the strength of the empirical evaluation somewhat makes up for these shortcomings. I’m currently borderline, with a tendency toward acceptance. With revisions, I could be convinced to accept this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for structured representation learning using autoencoders. The method has two primary ingredients: (i) encourage independence in latent blocks by feeding different blocks of the latent representation to different depths of the decoder by injecting noise in an Ada-IN-inspired block, (ii) a so-called hybrid sampling, that samples each block from a fixed learned set of k latent vectors, similar to the codebook used in VQ-VAE (Oord et al 2017). The method is claimed to result in higher fidelity reconstruction and generation while also learning representations that are more disentangled compared to VAE and $\beta$-VAE. 

Some limitations that came up in the reviews and later in the discussion among the reviewers are (i) lack of comparison with more advanced disentangled VAEs, which would be helpful in establishing the claim of the paper on better reconstruction but comparable disentanglement performance to regularization based methods (ii) high-level similarity to VLAE and other methods that also use hierarchical latent variables that limits the claims on novelty. Current draft also emphasizes disentanglement which the reviewers found lacking in justification and rigor. The paper is currently not suitable for publication at ICLR but taking into account the comments from reviewers on the presentation aspects will help improve the paper.
Decision: Reject