Structure by Architecture: Structured Representations without Regularization
Authors: Felix Leeb
Giulia Lanzillotta
Yashas Annadani
Michel Besserve
Stefan Bauer
Bernhard Schölkopf
Venue: ICLR 2023 poster (2023)
Keywords: Autoencoder
Structure
Generative
Architecture
Disentanglement
Regularization
Hybridization

Abstract: We study the problem of self-supervised structured representation learning using autoencoders for downstream tasks such as generative modeling. Unlike most methods which rely on matching an arbitrary, relatively unstructured, prior distribution for sampling, we propose a sampling technique that relies solely on the independence of latent variables, thereby avoiding the trade-off between reconstruction quality and generative performance typically observed in VAEs. We design a novel autoencoder architecture capable of learning a structured representation without the need for aggressive regularization. Our structural decoders learn a hierarchy of latent variables, thereby ordering the information without any additional regularization or supervision. We demonstrate how these models learn a representation that improves results in a variety of downstream tasks including generation, disentanglement, and extrapolation using several challenging and natural image datasets.

Published as a conference paper at ICLR 2023
STRUCTURE BY ARCHITECTURE: STRUCTURED REP- RESENTATIONS WITHOUT REGULARIZATION
Felix Leeb ; Giulia Lanzillotta, Yashas Annadani, Michel Besserve, Stefan Bauer, & Bernhard Schölkopf Max Planck Institute for Intelligent Systems, Tübingen, Germany
ABSTRACT
We study the problem of self-supervised structured representation learning using autoencoders for downstream tasks such generative modeling. Unlike most methods which rely on matching an arbitrary, relatively unstructured, prior dis- tribution for sampling, we propose sampling technique that relies solely the independence of latent variables, thereby avoiding the trade-off between recon- struction quality and generative performance typically observed in VAEs. We design a novel autoencoder architecture capable of learning a structured repre- sentation without the need for aggressive regularization. Our structural decoders Iearn a hierarchy of latent variables, thereby ordering the information without any additional regularization or supervision. We demonstrate how these models learn a representation that improves results in variety of downstream tasks includ- ing generation, disentanglement, and extrapolation using several challenging and natural image datasets.
INTRODUCTION
Deep learning has achieved strong results plethora of challenging tasks. However, perform- ing well on a carefully tuned or synthetic datasets is usually insufficient to transfer to real-world problems (Tan al.) 2018: Zhuang al.) (019) and directly collecting real labeled data is often prohibitively expensive. This has led to a particular interest in learning representations without su- pervision and designing inductive biases
achieve useful structure the representation help
with downstream tasks (Bengio et al. [2013] 3 Tschannen et al. 2018 Bengio et al. 2017] Tschannen
al. 2018 Radhakrishnan al. (0118).
Autoencoders (Vincent et al. 2008) have become the de-facto standard for representation learning largely due the simple self-supervised training objective paired with the enormous flexibility in the model structure. In particular, variational autoencoders (VAE) (Kingma & Welling 2013a) have become a popular framework which readily lends itself structuring representations by augmenting the objective or complex model architectures to achieve desirable properties such as matching simple prior distribution for generative modeling (Kingma Welling 2013a Vahdat & Kautz 2020)
Preechakul et al. (022), compression (Yang al. 2022) disentangling the underlying factors of
variation (Locatello et al. [2018] Khrulkov et al. [202]] [Shu et al. Chen
Chen et al. 2020 Nie et al.
2020 Mathieu al. 2019 Kwon & Yel 2021 Shen al. 20200)
However, VAEs exhibit several well-studied practical limitations such as posterior collapse (Lucas
et al.
2019b
Hoffman & Johnson 2016 Vahdat & Kautz 2020). holes in the representation (Li
et al.
2021
Stühmer et al. 2020), and blurry generated samples (Kumar & Poole 2020 Higgins
al.
2017
Burgess al. 2018 Vahdat & Kautz, 2020). Consequently,
seek representation
learning method which serves as a drop-in replacement for VAEs while improving performance on
common tasks such as generation and disentanglement. One promising direction the problem
causal modeling (Pear! 2009 Peters et al.) 2017 Louizos et al.) 2017 Mitrovic et al., 20200] Shen
al. 2020) which formalizes potential tri-irivial structure of the generative process using structural causal models (see appendix A. for further discussion). This powerful and exciting framework
serves as inspiration for our main contributions:
"Email: flech sech@tuebingen.mpg.de

---

Published as a conference paper at ICLR 2023
We propose architecture called the Structural Autoencoder (SAE), where the structural decoder infuses latent information one variable at a time to induce an intuitive ordering of information in the representation without supervision.
We provide sampling method, called hybrid sampling which relies only on independence between latent variables, rather than imposing a prior latent distribution thereby enabling more expressive representations.
We investigate the generalization capabilities of the encoder and decoder separately to bet- ter motivate the SAE architecture and to assess how the learned representation of an au- toencoder can be adapted to novel factors of variation.
1.1 RELATED WORK
The most popular autoencoder based method is the Variational Autoencoder (VAE) (Kingma & Welling 2013a), which has inspired the whole family of disentanglement focused methods includ- ing the 3- VAE (Higgins et al. 2017). Factor (Kim & Mnih 2018) or the 3-TCVAE (Chen ct al.) 2018). These methods aim match the latent distribution larizing the reconstruction training objective (Locatello et al., 2020 Zhou et al.) 2020). Although this structure is convenient for generative modeling and even tends to disentangle the latent space
known prior distribution by regu-
some extent, it comes at the cost of somewhat blurry images due to posterior collapse and holes in the latent space.
To mitigate the double-edged nature of VAEs (Makhzani & Frey 2017 Lin et al. 2022), less ag- gressive regularization techniques have been proposed such the Wasserstein Autoencoder (WAE), which focuses on the aggregate posterior (Tolstikhin et al. 2018). Unfortunately, WAEs generally fail to produce a particularly meaningful or disentangled latent space (Rubenstein et 2018), un- less some supervision is available (Han et al., (201))
Meanwhile, complementary approach to carefully adjusting the training objective is designing model architecture beyond the conventional feed-forward style to induce hierarchy in the repre- sentation. For example, the variational ladder autoencoder (VLAE) (Zhao et al., (017)) separates the latent space into separate chunks each of which is processed at different levels of the encoder and decoder (called "rungs"). However, due to the regularization, VLAEs suffer from the same trade- off's conventional VAEs. Further architectural improvements such FiLM (Perez et al., 2018) or Ada-In layers (Karras et al. 2019) readily learn more complex relationships resulting in more expressive models.
METHODS
Given (high-dimensional) X = (X1...., the encoder learns to encode the observations into low-dimensional representation U -U1... Ua) (d < D), and the decoder models the generative process that produced X from the latent variables to produce reconstruction X. To structure the representation, without loss of generality, some graphical structure between the (unknown) true factors of variation.
may imagine the true generative process follow
If the graphical structure of the true generative process was known, the topology of the decoder could be shaped accordingly (Kipf & Welling | [hang et al. 2219 Lin et al. [202]] Yang et al., 202T]. However, in the fully unsupervised setting, we seek a suffiientlyy general structure on top of the uninterpretable hidden layers to make the representation more structured and interpretable.
2.1 STRUCTURAL DECODERS
Structural decoders integrate information into the generative process one latent variable at a time, resulting in the following structure:
S = ((( 11=1....,dl
where S; is a feature map (So are static sample-independen trainable parameters) that depends on their noise Uz. their parent S.. and indirectly any ancestors S, for j i through S,-1.
2

---

Published as a conference paper at ICLR 2023
We choose f; = ConvBlock" StrTifm where ConvBlock" is some number of convolution blocks (where n depends on the dataset and can freely be adjusted) and StrTff is a Structural- Transform layer. Given the latent variable Us, the ith Str-Tfm layer applies a pixelwise affine transform to the given feature map S as seen in figure [ The Str-Tim layers are based on FiLM layers (Perez al. 2018) and the related Ada-In layers (Karras et al.) 2019) (except, cru- cially, only a single latent variable used for each Str-Tfm layer), which have been shown to infuse latent information into feature maps well compared to naive alternatives such as concate- nation.
U
U
0..A MLL))00)
U,
SeiTinu (..A) - a, of
Bilinear Upsampling
Static Features
Convol ution
Fully-connected
Vector Split
There are three main features of this architec- ture: (1) since each latent variable is infused at different layer of the decoder, the decoder ca- pacity for Ug is larger than for U, when i < j. This biases high-level non-linear information towards the earlier (and thereby deeper) latent variables, while the later variables can only cap- ture relatively low-level linear features with re- spect the observation space. (2) The inde-
The structural decoder reconstructs (or generates) a sample from a latent vector by first splitting U into d variables each of which infuses latent information with an affine transforms of the pixel e/"" in image feature map S; produced by Str- Tfm layer (green box where a, and 3 are the affine parameters that extracted from the latent variable U, by network MLP;)
Figure 1:
pendently initialized hidden layers of each Str-Tfm layer bias the Str-Tfmm layers to be independent of one another (similar to the "lambdas" in [Bllo (202T)) (3) The affine transform for each Str- Tfm layer is initialized to identity, so each latent variable has to independently learn to affect the generative process selecting for informative latent variables. This has a similar effect as the reg- ularization term of the VAE, except instead of penalizing informative latent variables directly and thereby conflicting with the reconstruction objective and giving rise to posterior collapse, it is only weak architectural bias. Overall, this architectural asymmetry between latent variables encourages statistical independence and induces a relatively intuitive hierarchical structure of the latent space.
Note that, without supervision, there is priori no guarantee that the learned structure will recover the true one (Locatello et al., 2018). Instead, the model is only optimized to reproduce the same observational distribution as the true generative process. Disentanglement methods are commonly evaluated with synthetic datasets where the factors of variation are independent by design TTäuble jet al. 202T). For methods penalizing correlations, this implicitly biases the latent variables to align with the true factors. However, the view that disentanglement can be reduced to statistical inde- pendence has been contested (Träuble et al. 2021; Schölkopf et al., 2021), which where a more causal approach may provide additional insight. We treat disentanglement as measured by common metries not as the objective of our model, but merely as tool to better understand the structure of the representation. Independence of latent variables implies that interventions on them do not violate the learned generative process (i.c., the decoder), thereby enabling hybrid sampling as an alternative to variational regularization techniques for generative modeling.
2.2 HYBRID SAMPLING AND OBJECTIVE
For generative modeling, it is necessary to sample novel latent vectors that are transformed into (syn- thetic) observations using the decoder. Usually, this is done by regularizing the training objective so the posterior matches some simple prior (e.g. the standard normal). However, in practice, regu- larization techniques can fail to adequately match the prior and actually exacerbate the information bottleneck, leading to blurry samples from holes in the learned latent distribution and unused latent dimensions due to posterior collapse (Dai & Wipf, 2019 Stühmer et al. 2020 Lucas et al., 2019a Hoffman & Johnson 2016). Instead of trying to match some prior distribution in the latent space, we suggest an alternative sampling method that eliminates the need for any regularization, which in turn also makes it easier to augment the objective with auxiliary loss terms in settings where addi- tional supervision is available. Consequently, the training objective for our models here is only the reconstruction error (specifically, the cross entropy loss).
3

---

Published as a conference paper at ICLR 2023
Inspired by Besserve et al. (2018), we refer to this sampling method hybrid sampling. The goal is to draw new latent samples U from distribution having independent variables with marginals matching those observed in the training set, leading to p(U) s II, p(U)). In practice, we first store set of N (= 128 in our case) latent vectors {
و selected uniformly at random from the training set. We then generate latent samples from by choosing independently the value for each variable Us which is sampled uniformly at random from the corresponding set of values of this variable in the stored vectors { 1==1-
This allows the model to generate a diversity of samples well beyond the size of the training set (Na distinct latent vectors), spanning the Cartesian product of the supports of the (marginal) distribution of each U, on the training data, which includes the support of the joint latent distribution of U ob- served during training. Note that hybrid sampling is directly applicable to any learned representation as it does not affect training at all, however the fidelity of generated samples will diminish if there strong correlations between latent dimensions. Consequently, the goal is to achieve maximal independence between latent variables without compromising on the fidelity of the decoder (i.c., re- construction error), which for SAEs is done through the architecture alone. This also aligns with the approach of existing unsupervised disentanglement methods, which suggests the sampling method may serve as promising alternative the VAE-based prior-based sampling, beyond just SAEs.
3 EXPERIMENTS
We train the proposed methods and baselines on two smaller disentanglement image datasets (where D 64 x 64 3 and the true factors are independent): 3D-Shapes (Burgess & Kim (0188) and the three variants ("toy", "sim", and "real"") of the MPI3D Disentanglement dataset (Gondal et al., 2019. as well as two larger more realistic datasets (where D 128 x 128 x 3): Celeb-A (Liu et al., 2015 and the Robot Finger Dataset (RFD) (Dittadi et al. 2020).
The models are trained with a standard 70-10-20 (train-val-tess split of the datasets where the train- ing objective uses the cross entropy loss (as well as the method specific regularization terms for the baselines). We evaluate the quality of the reconstructions based on the reconstruction loss and the Fréchet Inception Distance (FID) (Heusel et al. (017) as in Williams et al. (2020) on the test set. The FID is able to capture higher level visual features and can be used to directly compare the reconstructed and generated sample quality, while the binary cross entropy is purely pixelwise comparison.
Next we compare the performance of the hybrid sampling method to the prior based sampling. Unlike the prior based sampling, which only makes sense for the models that use regularization the hybrid sampling method can be applied any latent variable model. Finally we take a closer look at the representations to understand how the model architecture affects the induced structure and disentanglement.
3.1 MODELS
Since the datasets involve images, all models use the same CNN backhone for both the encoder and decoder with the same number of convolution blocks (see the appendix for details). For the smaller datasets, the encoder and decoder have 12 convolution blocks, the latent space has 12 di- mensions in total, and the models are trained for 100k iterations, while for CelebA and RFD each the encoder/decoder has 16 convolution blocks each with twice as many filters, the latent space is 32 dimensional in total, and the models are trained for 200k iterations.
We compare four kinds of autoencoder architectures. The first type is our Structural Autoencoders (SAB) which use a conventional deterministic encoder and a structural decoder with the latent space split evenly into 2, 3, 4, 6, or 12 variables for the smaller datasets and 16 variables for the larger ones corresponding to the labels SAE-2, SAE-3, SAE-4, SAE-6, SAE- 12 and SAE-16 respectively. Consequently, the "structural" architecture has an architectural asymmetry between latent variables in the decoder, but not in the encoder. The simplest "baseline" architecture uses the traditional "hourglass" architecture, in addition to variety of different regularization methods: (1) unregu- larized autocncoders ("AE"), (2)
("WAE") (Tolstikhin et al., 2018). (3) VAE (Kingma& & Welling 2013b), (4) ,3VA (Higgins et al. 2017), (5) Factor ("FVAE") (Kim
4

---

Published as a conference paper at ICLR 2023
Archfonlare
Archtadiure Studiied Acapttve
60
solur Acapdhe
Acconstructinn RID Store
1/40
1.0
Rcconddr
100
Arestive
Ladder
SHARO
WY
AC
(a) 3D-Shapes
(b) RFD
(c) Celeh-A
Reansracci SCUEC
Recontreccte RD Score
Strutte
Adsetive
Apct Rocturs
Jadder Zand Pr
ANAE STOPE
(d) MPI3D Toy
(c) MPI3D Sim
( MPI3D Real
Figure 2: Reconstruction quality for all models and datasets (lower is better). "Baseline" models correspond to traditional "hourglass" CNN architectures, while the "Structural" models use our novel architectures to further structure the learned representation.
Mnih 2018), and (6) 3-TCVAE (using the experimentally determined best hyperparameters, see the appendix (.2.2).. The next baseline architecture is called "AdaAE" (and referred to as "Adap- tive"). It identical to the SAE models except that all latent variables are passed to each of the Str-Tfm layers, so the architecture has connections between the latent variables and intermediate layers, but without the architectural asymmetry. For the purposes of hybrid sampling, each latent dimension of these less structured baselines is treated separate latent variable. The final type of architecture we investigate is the variational ladder autoencoder (Zhao et al. (017)) which also learns hierarchical representation, but unlike the structural autoencoders, VLAFS also use the variational regularization and use an encoder architecture that roughly mirrors the decoder, consequently both the encoder and decoder break the architectural symmetry between latent variables. Just like for the SAEs, we include variants of the VLAEs with 2, 3, 4, 6, 12, and 16 rungs.
3.2 EXTRAPOLATION
Since VLAEs structure both the encoder and decoder, while SAEs only structure the decoder, we aim better characterize the relative behaviors of the encoder and decoder. Specifically, this anal- ysis aims to understand to what extent the encoder or decoder is the "weaker link" in regards to integrating new information into the representation.
First, both the encoder and decoder are trained jointly on a subset of 3D-Shapes where only three distinct shapes exist (instead of four, the ball missing) for 80k iterations. Then either the encoder only. the decoder only, or both are trained for another 20k iterations on the full 3D-Shapes training dataset. The reconstruction error for samples each of the variants and each of the architectures to identify how well the encoder extrapolates compared to the decoder, and assay implications for designing novel autoencoder architectures.
seen during any part of training is compared for
RESULTS
In terms of reconstruction quality, the structural autoencoder (SAE) architecture consistently outper- forms the baseline methods (figure 2). As expected, unregularized methods like the SAE, AdaAE, and AE tend to have significantly better reconstruction quality. However, also noteworthy is that the structured architectures SAE and VLAE show improved results compared their unstructured
5

---

Published as a conference paper at ICLR 2023
Sampling lyb/dd
Iryend Phice
cin Hybrnd
2010 Mlay
130
Plaxe
FID Scaee
110
123
Acaptivn
Laidey
10
Ladder
Acaptive
Acaptive
Ladder
Micekie
VAC
(a) 3D-Shapes
(b) RFD
(c) Celeh-A
myson
Pybrin
1000 PMaor
2022 Price
2000 Prior
FD Score
Arckacctur
ArchBacturr
10
Lacder Daselre
Ledder
Ladder
Duseline
Duseinn
(d) MPI3D Toy
(c) MPI3D Sim
(f) MPI3D Real
Figure 3: Quality of the generated samples using different models and sampling methods (lower is better). Note that our SAE models perform well without having regularize the latent space towards a prior. In fact, even with the conventional "hourglass" architecture (in orange), the hybrid sampling method generates relatively high quality samples, often outperforming the more principled prior-bace based sampling.
(a) SAE-12
(b) SAE-3
(c) VAE
(d) AdaAE-112
Figure 4: Latent traversals of several models trained on 3D-Shapes, in their original order. Note the ordering of the information in the structural decoder models (SAE-12 and SAE-3) where higher level, nonlinear features (like shape and orientation) encoded in the first few dimensions, which feed into Str-Tfm layers deeper in the network. Somewhat surprisingly. the SAE-12 even learns to compact the representation by ignoring superfluous latent variables (e.g. first row) resembling the effect of posterior collapse in VAEs (see A.3.I.
counterparts. should be noted that there is some controversy regarding the reliability FID with synthetic datasets (Rzaavi et al., 2019 Barratt & Sharma 2018). However, consistently find strong agreement between the reconstruction FID and the pixelwise metrics (see figure in the appendix for further discussion).

---

Published as a conference paper at ICLR 2023
Model
DCI
MIG
IRS
Mod
Exp
SAE-12
0.974
0.537
0.830
0.967
1.000
VLÁE-12 12
AdaAB 12
SAE-6
0.865
0.225
0.735
0.966
0.999
A6
SAE-4
0.740
0.209
0.654
0.945
0.999
PRRE
VLAE-12
0.832
0.553
0.751
0.914
0.977
FVAE
54E 12
VLAE-6
0.785
0.326
0.689
0.929
0.963
VLAE-4
0.690
0.282
0.544
0.900
0.926
MIG
AE
VIE
STCVAE
0.410
0.237
0.603
0.865
0.923
0998
(OOYAE
FVAE
0.330
0.123
0.725
0.907
0.955
SAB-12
VAA
0.235
0.127
0.593
0.879
0,799
VLAE-12
AdsA2-12
VAE
0.314
0.138
0.607
0.892
0.872
WAE
0.211
0.050
0.621
0.946
0.906
8VA
ATOVAE
AE
0.307
0.092
0.638
0.926
0.943
FVAE
0.0
0.2
0.4
0.8
3.0
AdaAE-12
0.299
0.062
0.503
0.876
0.999
Score
Figure 5: Disentanglement scores for 3D-Shapes. DCI denotes the DCI disentanglement score (Eastwood & Williams|2213 MIG is the Mutual Information Gap (Chen et al., 2018), IRS is the Interventional Robustness Score (Suter et al., (0018), and Mod/Exp refers to the Modular- ity/Exxlicitness scores respectively (Ridgeway & Mozer 2018 (for all these metrics higher bet- ter). The figure the right shows how the scores vary across five models with different random seeds marked with a cross (lines indicate the resulting mean and standard deviation). Both hicrarchi- cal methods, SAE-12 12 and the VLAE-12, 12, outperform all other baselines, and in particular the SAE performs well, despite the lack of regularization.
Comparing the SAE models to the AdaAE architecture, we see that there be slight penalty in reconstruction quality incurred from separately processing the latent variables. However, this more than made up for in the quality of the generated samples (shown in figure 3. where the SAE models perform significantly better than the baselines. Even the regularized models such as VLAE, FVAEs, and VAEs consistently generate higher quality samples using the hybrid sampling than when sampling from the prior they were trained to match (also in figure 6b). Surprisingly, the AdaAE architecture actually outperforms all other models on CelebA using hybrid sampling. This may be explained by the severity of the information bottleneck experienced when embedding CelebA into only 32 dimensions. Consequently, the higher fidelity decoder from the intermediate connections of the latent vector exceed the performance penalty incurred by the hybrid sampling when disregarding the correlations between latent variables.
In general, if consider the latent distribution, then sampling from the approximated factorized prior can introduce at least two types of errors: (1) errors due to not taking into account statistical dependences among latent variables, and (2) errors due sampling from "holes" in the latent dis- tribution if the prior does not match it everywhere. Whenever (2) is the dominating source of error, hybrid sampling is preferred. Furthermore, learning independent latent variables aligns with the aim towards disentangled representations, while minimizing the divergence between the posterior and prior results in a compromise that does not necessarily promote disentanglement.
4.1 HIERARCHICAL STRUCTURE
To get a rough idea of how the representations learned using the structural decoders differ from more conventional architectures, figure 4 shows the one dimensional latent traversals (i.e., each row cor- responds to the decoder outputs when incrementally increasing the corresponding latent dimension at a time from the min to the max value observed). The traversals illustrate the hicrarchical structure in the representation learned by the SAE models: the information encoded in the first few latent variables can be more nonlinear with respect to the output (pixel) space, as the decoder has more layers to process that information, while the more linear information must be embedded in the last few variables. This results in a reliable ordering of "high-level" information (such as object shape or camera orientation) first, followed by the "low-level" information (such as color). This means the structural decoder architecture biases the representation separate and order the information neces- sary for reconstruction (and generation) in a meaningful way, and thereby ordering, and potentially even fully disentangling, the underlying factors of variation better.
7

---

Published as a conference paper at ICLR 2023
QANITN tantre
(a) Here we use hyhridized chunks of latent vec- to show the diffuse disentanglement achieved by the hierarchical structure of the SAE-16 ar- chitecture when combined with the flexible hy- brid sampling. For cach row the corresponding quarter of latent dimensions (8/32) hybridized (sec section 2.2) while the remaining dimensions are fixed. This shows how the SAE architecture is able to order partially disentangled factors of variation from high-level (more nonlinear, like fa- cial expressions and features) to low-level (such as color/lihting) without any additional regular- ization or supervision.
(b) Samples generated using hybrid and prior- based sampling using several models trained on CelebA. Note that the hybrid sampling tends to produce relatively high quality samples both for our proposed SAE and AdaAE architectures as well as baselines.
Figure 6: CelebA controllable generation and sampling comparison
FevuureSevaluates how disentangled the representations are, using common metrics. The table shows the disentanglement scores of the same models discussed above, while the plot on the right sheds light on quality of the representations varies for five different random seeds (used to initialize the network parameters). Most noteworthy is that the SAE- 12 model consistently achieves very high disentanglement scores. This shows, empirically, that the SAE architecture promotes independence between latent variables (especially SAF-12). We may explain this as a consequence of splitting up the latent dimensions so that each variable has unique parameterization in the decoder, making different latent variables less likely to transform the feature map of the decoder in the same way.
Unsurprisingly, when there are multiple latent dimensions per variable (like in SAB-6, SAE-4, etc.), the dimensions within a variable are entangled similarly to the baselines like AE or adaptive base- lines. Since all of these disentanglment metrics are computed on dimension- peeension-byddine by-dimension basis, the resulting scores are systematically underestimated. Qualitatively, these SAE models still achieve the same ordering of causal mechanisms, as can be seen from figure 4b)
For real world demonstration of how well SAE models are able to order information in the latent space, figure 6a| shows generated CelebA samples when varying only quarter of the latent vari- ables at time, with the labels on the left describing roughly the semantic semantic information contained. Note that the inductive biases are not strong enough to fully disentangle the factors of variation into individual latent dimensions. However, the hierarchical structure learns diffuse kind of disentanglemenn where information pertaining to higher-leve level features tend to be encoded in the first few dimensions while lower level factors of variation show up towards the last few dimensions.
SAE models achieve this structured disentanglement using the Str-Tfm layers as opposed to the standard FiLM/Ada-In layers. Each Str-Tfm layer only has access one of the latent variables which is not directly seen by any other part of the decoder. In contrast, the Ada-In layers used by the AdaAE allows information from anywhere in the latent vector to leak into any part of the decoder. Consequently, the AdaAE does not disentangle the representation at all, as seen in the table of figure 3 it achieves impressive results for reconstruction nonetheless, see figure 2
4.2 EXTRAPOLATION
Results of the experiment described in section 3.2 are shown figure 7 Perhaps unsurprisingly, none of the models were particularly adept at zero-shot extrapolation to observations that were not in the initial training data, as is consistent with Schott et al. (002)) Comparing the first two columns
8

---

of the commonly Published as a conference paper at ICLR 2023
on the right, without any update, the reconstructed images filter out the novel information in the sample (in this case, ball shape), and instead reconstruct a similar sample seen during training (a cylinder).
If only the encoder is updated on some observations with the additional shape while the decoder is frozen, then the reconstruction performance increases somewhat, but some deformations and arti- facts become visible in the reconstruction. This suggests the frozen decoder struggles to adequately extrapolate, even when the encoder extends the representation include the ball.
In contrast, when only the decoder is updated, the reconstructions qualitatively look much more similar the original observations. Although the encoder be expected generally filter out any information it has not been trained to encode into the latent space due to the bottleneck, the repre- sentation may still extrapolate somewhat provided the decoder can reconstruct any novel features. This underscores the importance of focusing on carefully shaping the decoder, embodied by the SAEs, because the decoder is not able extrapolate as well the encoder.
Model
Neither
Encoder
Decoder
Both
SAE-12
13.21
7.7
0.42
0.34
SAE-6
13.35
7.99
0.52
0.36
VLAE-12
18.37
7.69
1.55
0.62
VAE
12.97
8.78
0.44
0.46
NAA
15.49
8.31
1.35
0.52
AE
11.81
7.31
0.38
0.35
WAE
11.68
7.87
0.37
0.35
Figure 7: Average reconstruction error (MSE x1000) on novel observations when training either the encoder, decoder, both, or neither on the initially left-out hall shape (see section Example input and reconstructed images shown above. All models perform significantly better when updating the decoder than the encoder, and reach a reconstruction quality that is almost indistinguishable from the model when updating both the encoder and decoder. Furthermore, the SAE-12 generally out- performs all variational baselines, suggesting the aggressive regularization of VAEs makes updating the representation more difficult.
5 CONCLUSION
While VAEs provide a principled approach for generative modeling with autoencoders, in practice, the regularization tends to suppress the dependence of the posterior the observations by minimiz- ing its discrepancy to the prior (Hoffman Johnson 2016 Tolstikhin et al. 2018), resulting in a trade-off between reconstruction quality and matching the prior. While regularization may lead to some disentanglement, this tends to have negative effects on sample quality, and due to the identifi- ability problem, disentanglement cannot be guaranteed (Locatello et al. 2018).
This motivated us to develop an alternative paradigm that improves sample fidelity while still achiev- ing comparable disentanglement. The key is simplify the training objective and exploit the ar- chitectural flexibility of deep models instead. We show that despite the identifiability problem, architectural biases can make the representation more interpretable by focusing on diffuse but intuitive ordering of semantic information, rather than the full dimension-be by-dimnnsinnn disentan- glement. Meanwhile, our hybrid sampling technique leverages the independence between latent variables achieved by both VAE-based and architecture based, rather than requiring the learned ag- gregate posterior to match some unstructured prior. This unifies the goals of achieving a samplable representation and one with independent latent variables, thereby addressing accepted weaknesses of existing VAE-based methods.
While it is encouraging how far one can get with suitable architectural biases, future work should further investigate links between the learned generative process and the true causal mechanisms to make the representation more interpretable and potentially enhance generalization. For instance, could explicitly encourage the realism of hybrid samples, or the sparsity of latent factor changes across domain shifts based on the Sparse Mechanism Shift Hypothesis (Schölkopf et al. 2021).

---

Published as a conference paper at ICLR 2023
REFERENCES
Shane Barratt and Rishi Sharma. A note the inception score. arXiv preprint 1.Xiv:8801.0197 2018.
Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. arXiv preprint arXiv:2102.08600. 2021.
Emmanuel Bengio, Valentin Thomas, Joelle Pineau, Doina Precup, and Yoshua Bengio. Indepen- dently controllable features, 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. JEEE Iransactions on paltern analysis and machine intelligence, 1788)187,, 2013.
Michel Besserve, Arash Mehrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. arXiv preprint arXiv:1812.03253., 2018.
Chris Burgess and Hyunjik Kim. 3d shapes dataset, 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in 3-VAE. arXiv preprint arXiv: 03559, 2018.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. arXiv preprint arXiv: arXiv:/802.049000 2018.
Shang-Fu Fu Chen, Jia-Wei Yan, Ya-Fan Su, and Yu-Chiang Frank Wang. Deep representation decom- position for feature disentanglement. arXiv preprint 1xXiv200110007800 2020.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. arXiv preprint arXiv: 1rXiv:1933.0578: 2019.
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wüthrich. Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. arXiv preprint arXiv:2010. 14407700..144,... 2020.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. 2018.
Muhammad Waleed Gondal, Manuel Wüthrich, Djorde Miladinović, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: new disentanglement dataset. arXiv preprini arXiv: 1906.0::00005 91299900..22... 2019.
Jun Han, Martin Renqiang Min, Ligong Han, Li Erran Li. and Xuan Zhang. Disentangled recurrent Wasserstein autoencoder. arXiv preprint arXiv:2101.0744 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. In Advances in neural information processing systems, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts
with constrained variational framework. ICLR. 2(5):6, 2017. URL httpss://ddss
semant pdf
Matthew D Hoffman and Matthew J Johnson. ELBO surgery: yet another way carve up the variational evidence lower bound. In Workshop in Advances Approximate Bayesian Inference, NIPS, volume 1, pp. 2016.
10

---

Published as a conference paper at ICLR 2023
Tero Karras, Samuli Laine, and Timo Aila. A based generator architecture for gener- ative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. URL http:://pennccess..eeeeeee thecvf. wttp://ee..ooooooeeoetteeo content CVPR CVPR_201 2t19///ppprs///aaaaaaaa A oaraaaaaaaaaaaaasaaeeeeerrrrrrrr for GeeeraativeAAvvrrsaaat L ortpeppreeeeeeaaeeeeaaaaaeeaeeer 20ape_2ep_pppppppcc_.
Valentin Khrulkov, Leyla Mirvakhabova, Ivan Oseledets, and Artem Babenko. Disentangled repre-
sentations from non-disentangled models. arXiv preprint arXhv:2102.06204, 2021.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint WxXiv:802200595. 2018. URLptttss:
http://awwwwww...a......o 802. 119983.pd3
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes, 2013a.
URL arXiv:
1312.6114
Diederik Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
1rXiv:1312.61 2013b.
Thomas N Kipf and Max Welling. arXiv:1611.07308, 2016.
Variational graph auto-encoders.
arXiv preprint
Abhishek Kumar and Ben Poole. On implicit regularization in 8-vaes. In International Conference on Machine Learning, pp. 5480-5490. PMLR, 2020.
Gihyun Kwon and Jong Chul Ye. Diagonal attention and style-based GAN for content-style disen- tanglement in image generation and translation. arXiv preprint arXiv:2103.16145: 2021.
Ruizhe Li, Xutan Peng, and Chenghua Lin. On the latent holes of vaes for text generation. arXiv 2021.
preprint arXiv:21
Fudong Lin, Xu Yuan, Lu Peng, and Nian-Feng Tzeng. Cascade variational auto-encoder for hier- archical disentanglement, 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. arXiv preprint 1rXiv:181..12355 2018.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look the unsupervised learning of disentangled repre- sentations and their evaluation. Journal of Machine Learning Research (JMLR), 2020.
Christos Louizos, Uri Shalit, Joris Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. arXiv preprint VrXiv:/705.0800 2017.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Understanding posterior collapse in generative latent variable models. 2019a.
James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't blame the elbo! linear vae perspective on posterior collapse. Advances in Neural Information Processing Systems, 32, 2019b.
Alireza Makhzani and Brendan Frey. Pixelgan autoencoders. Advances in Neural Information Processing Systems, 30, 2017.
Emile Mathieu, Tom Rainforth, Nana Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders, In International Conference on Machine Learning, pp. 4402-4412. PMLR, 2019.
Diganta Misra. Mish: A self regularized non-mononn monotonic neural activation function. arxiv preprint arXiv: 190::990000000 08681, 4, 2019.
11

---

Published as a conference paper at ICLR 2023
Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Represen- tation learning via invariant causal mechanisms. arXiv preprint arXiv:2010.07922 2020.
Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debnath, Anjul Patney, Ankit Patel, and Ani- mashree Anandkumar. Semi-supervised stylegan for disentanglement learning. In International Conference on Machine Learning, pp. 7360-7369. PMLR, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Gameet (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, NY, 2nd edition, 2009.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA, 2017.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakomn Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10619-10629, 2022.
adityanarayanan Radhakrishnan, Karren Yang. Mikhail Belkin, and Caroline Uhler. Memorization in overparameterized autoencoders. arXiv preprint arXiv:1810.1033. 2018.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in neural information processing systems, pp. 14866-14876, 2019.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. arXiv preprint arXiv: 1802.05312 2018.
Paul K Rubenstein, Bernhard Schoelkopf, and Ilya Tolstikhin. On the latent space of Wasserstein auto-encoders. arXiv preprint arXiv:1802.03777 2018.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-6))) 12441122264222 2021.
Lukas Schott, Julius von Kügelgen, Frederik Träuble, Peter Gehler, Chris Russell, Matthias Bethge, Bernhard Schölkopf, Francesco Locatello, and Wieland Brendel. Visual representation learning does not generalize strongly within the same domain. arXiv preprint 2rX0v22107.08821, 2021.
Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang. Disentangled generative causal representation learning. arXiv preprint 0oXiv:2000.0263:. 2020.
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disen- tanglement with guarantees. arXiv preprint 11X1v:1910.09772. 2019.
Jan Stühmer, Richard Turner, and Sebastian Nowozin. Independent subspace analysis for unsuper-
vised learning of disentangled representations. In International Conference on Artificial Intelli- gence and Statistics, pp. 1200-1210. PMLR, 2020.
Raphael Suter, Dorde Miladinovic, Stefan Bauer, and Bernhard Schölkopf. Interventional robustness of deep latent variable models, 2018. httss//arxiv.org/aabs 1811 11.00/b0/111..000001ll
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pp. 270-279. Springer, 2018.
12

---

Published as a conference paper at ICLR 2023
1. Tolstikhin, O. Bousquet, S. Gelly, and B. Schölkopf. Wasserstein auto-encoders. In 6th In- ternational Conference on Learning Representations (ICLR), May 2018. URL https:// http://wwwnett/ooommeiiii.....
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International Conference on Machine Learning, pp. 10401-10412. PMLR, 2021.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based representation learning. arXiv preprint arXiv: wet. 05069, 2018.
Arash Vahdat and Jan Kautz. Nvae: deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 1ee19966-196771 2020.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Will Williams, Sam Ringer, Tom Ash, John Hughes, David MacLeod, and Jamie Dougherty, Hier- archical quantized autoencoders. arXiv preprint arXiv:2002.08111. 2020.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 9593-9602, 2021.
Yibo Yang. Stephan Mandt, and Lucas Theis. An introduction to neural data compression. arXiv preprint wrXiiv2202.065333 2022.
Si Zhang, Hanghang Tong, Jiejun Xu, and Ross Maciejewski. Graph convolutional networks: comprehensive review. Computational Social Networks, 6(1):1-23, 2019.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep gener- ative models. In Proceedings of the 34th International Conference Machine Learning-Volle Volume 70, pp. 4091-4099. JMLR.org, 2017.
Sharon Zhou, Eric Zelikman, Fred Lu, Andrew Y Ng, and Stefano Ermon. Evaluating the disentan- glement of deep generative models through manifold topology. arXiv preprint arXiv:2006 036800 2020.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprchensive survey on transfer learning. arXiv preprint vrXiv:1911.02600 2019.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 LINKS TO CAUSAL REPRESENTATION LEARNING
The structural decoders can be interpreted as a structural causal model, thereby linking the causal modeling community to representation learning. A structural causal model (SCM) represents the lationship between random variables S, using a directed acyclic graph (DAG) whose edges indicate direct causation and structural assignments of the form
S: = U:), (i Li=
(2)
encoding the dependence of variable S, on its parents PA; in the graph and on an "unexplained" noise variable U; (Pearl, 2009). The noises U......U UD are assumed to be jointly independent. Any joint distribution of the St can be expressed as an SCM using suitable f. and U. Note, that although structural decoders define the relatively simple sequential graphical structure PA, S long as the capacity of S; is sufficiently large, all DAGs can be represented by this structure. Furthermore, the SCM contains additional information regarding how statistical dependencies between the S; are generated by mechanisms (2) such that changes due to interventions can be modelled as well (e.g., by setting some U, constants). From this perspective. hybrid sampling can be interpreted as interventions on the learned generative process. Although thus far we focus on the relatively naive approach of jointly intervening all latent variables for unconditional generative modeling, as shown in figure 6a
We construct the causal ordering as follows: we evaluate the of a root node i, i.e. fi depends only on U,. In the step, we evaluate any node j which depends only on its Uj and possibly other variables that have already been computed. We iterate until there are no nodes left. This terminates (since the graph is acyclic) and yields a unique f(U), but the order (i)) in which the fa get evaluated need not be unique. It is referred to as a causal or topological ordering (Peters al., 2017). satisfying (i)) < Tj) whenever j is descendant of i. This embeds the SCM into the network starting from the bottleneck = (U...... ... UD) with the U, feeding into subsequent computation layers according to causal ordering. This structure reflects the fact that the root node(s) in the DAG only depend on "their" noise variables, while later ones depend on their noise and those of their parents, and so on.
Recall that for a causally sufficient system, the set of noises Ul.... Un are assumed to be jointly independent. If, in contrast, only a subset of the causal variables are modelled, then the noises will in the generie case be dependent. We would expect that the architectural bias implemented by the structural decoder, however, may still be a sensible one.
In this work, we evaluate the proposed architectural biases exclusively image datasets these serve as a particularly intuitive and well-studied setting for disentanglement research due to the high-dimensional observations with relatively few underlying factors of variation. Nevertheless, the structural decoder architecture can directly be applied to data without spatial structure (since the Str-Tfm layers are applied pixel-wise anyway), and the hybrid sampling method is agnostic to the model architecture and training (as long as the learned latent variables are mutually independent).
A.2 TRAINING PROCEDURE
A.2.1 ARCHITECTURE DETAILS
As described in the main paper, the basic convolutional backbone of all models is the same. For the smaller datasets, 3D-Shapes and the MPI3D datasets (where observations are 64x64 pixels), the encoder and decoder each have 12 convolutional blocks. Each block has convolutional layer with 64 channels and a kernel size of 3x3 and stride of (unless otherwise specified), followed by group normalization layer and then a MISH nonlinearity (Misral 2019). In the encoder, the features are downsampled using a 2x2 Max Pooling layer right after the convolution every third layer starting with the first one and the first convolution layer uses a kernel size of 5x5. In the decoder, every third convolution layer is immediately preceded by a 2x2 bilinear upsampling. For our structured modules (SAE and AdaAE), the specified number of Str-Tfmlo layers are placed evenly in between the convolution blocks. For SAE models, the latent space is always split evenly between Str-Tim layers, and each layer uses a three hidden layer network to process the latent space segment into the scale and bias vectors which are then applied to all pixels individually of the features. The output layer
14

---

Published as a conference paper at ICLR 2023
biases of each Str-Tfim network are initialized to 0 and the output corresponding to the scaling is interpreted as logits (just as is done for the standard deviation parameters of the posterior of VAEs), such that a₂ R 1 and 3; se 0 for all Str-Tfmm layers at the beginning of training. For the VLAE models, the inference and generative ladder rungs each also have a three hidden layers to process the features into and out of the separate latent space segments respectively.
While the latent space was always 12 dimensional for 3D-Shapes and MPI3D, for Celeb-A we use 32 dimensional latent space. For Celeb-A, we also expand the 12 block backbone 16 blocks and double the filters per convolution layer to 128. The exact sizes and connectivity of the models can be seen the configuration files of a the attached code, but overall, each of the 3D- Shapes and MPI3D models have approximately 1-1.20 1-12MM trainable parameters, while for CelebA the models have 6-7M parameters. Note that the VIAAE models always have the most parameters due to the hierarchical structure in the encoder that mirrors the decoder, the SAEs and AdaAEs have roughly 5% fewer parameters, and almost the same as the conventional hourglass architecture of the remaining baselines (the Str-Tfim layers contribute negligible additional parameters relative to the CNN backbone).
A.2.2 TRAINING DETAILS
All models used the same training hyperparameters, which included using an Adam optimizer with a learning rate of 0.0005 and momentum parameters of B1 = 0.9 and f3 = 0.999. For the smaller datasets (3D-Shapes,, MPI3D) the models were trained for 100k iterations and a batch size of 128, while for Celeb-A and RFD the models were trained for 200k iterations and batch size of 32. The hyperparameters for the RFD dataset the same as for Celeb-A, except that the number of channels per convolution layer was doubled and the learning rate was decreased by factor of 10.
The models are implemented using Pytorch (Paszke et al. 2019) and were trained on the in-house computing cluster using Nvidia V100 32GB GPUs, so that training single model takes about 3-4 hours on the smaller datasets and 7-10 hours for CelebA.
For the 3- VAES and 3-TCAAEs, 3 E {2, 4, 6, 8, 16). while y E {10, 20, 40, 80} for the FVAE were tested on 3D-Shapes and MPI3D, and the model with the smallest loss on the validation set was used for subsequent analysis, which was 3 = 2 for the 3-VAE and B = 4 for the 3-TCVAE and y 40 for the FVAE. All other method-specific hyperparameters were kept the same in the corresponding papers.
A.3 ADDITIONAL RESULTS
A.3.1
ARCHITECTURALD INDUCED INDEPENDENCE BETWEEN LATENT VARIABLES
A closer look at figure shows traversing several of the latent variables of the SAE-12 has no effect on the resulting image (e.g. the first and second rows) (also observed in the other datasets in fig- ures T1 13 14 15 and [8] This resembles the well-studied phenomenon in VAE-based models of posterior collapse (e.g. first row of figure 4ch where some latent variables are unused/uninformaaive to avoid incurring the regularization cost. However, for SAEs, since there is no regularization loss, instead we can understand this behavior as a consequence of the independence between latent vari- ables due to the Str-Tfin layers.
Specifically, since the beginning of training all Str- Str-Tim layers map roughly identity, the decoder must learn to extract useful information from the latent variable through the Str-Tim layer, otherwise the latent variable has no effect on the decoding process at all. This biases the decoder to only use those latent variables that are in fact informative, otherwise it is preferable for the decoding process to ignore the variable entirely to keep the affine transform of the Str-Tfm layer close to identity. Consequently, the use of Str-Tfm layers biases the decoder treat each latent variable independently and thereby only use the variables that contain information useful for reconstruction not already contained in different variable. Although this is obviously only weak bias, it is evidently sufficient to learn a more compact representation without incurring as much of a cost to the sample fidelity observed with posterior collapse of VAFs.
15

---

Published as a conference paper at ICLR 2023
A.3.2 DH-SHSPES
50
Model
SAE-12
FVAE
WAE
45
SAE-6
BVAE
AE
VLAE-12
VAE
AdaAE-12
Reconstruction FID Score
40
BTCVAE
35
18
25
16
20
15
12
10
3460
3480
3500
3520
3540
Reconstruction Loss
Figure 8: 3D>Shapes reconstruction quality comparison between models using the reconstruction loss (binary cross entropy) and the Fréchet Inception Distance (FID) (Heusel et al.) 2017) between the original and reconstructed observations (lower is better for both). Each "x" is model trained with a unique random seed using the architecture/regullaiiiooo corresponding to the color. The performance of all the seeds are averaged and plotted as circles "o". Firstly, this plot shows how the reconstruction FID (y-axis) can complement the pixelwise comparison (x-axis) to quantify the quality the reconstructed samples. Next, the multiple seeds help differentiate the performance of the regularized vs unregularized methods. These regimes separate the models that use the variational regularization loss from the models that only use reconstruction aggregated posterior like the WAE). Lastly, the AdaAE-12 slightly out performs the SAE-12 12 and both which significantly out perform less structured baselines, which suggests the architectural biases conducive to high fidelity decoders.
(or regularization on the
16

---

Published as a conference paper at ICLR 2023
100
Architecture
Structural
90
Adaptive
x .x...---X
Ladder
80
Baseline
x-xX
XX * of X
| *
70
FID Score
x
XX
60
X
50
X
x x-
X
40
xXX
x
Sampling
30
Hybrid
Prior
X
20
SAE-12
VLAE-12
VLAE-12
SAE-6
VAE
BTCVAE
BTCVAE
FVAE
VAE
BVAE.
AE
WAE
WAE
BVAE
AdaAE-12
FVAE
Figure 9: Comparison of the hybrid and prior-based sampling method for all different models. Each
"x" corresponds unique random seed, the horizontal line corresponds the mean performance and the vertical line signifies one standard deviation above and below the mean. (lower is best)
Model
DCI
MIG
IRS
Mod
Exp
SAE-12
0.974
0.537
0.830
0.967
1.000
SAE-6
0.865
0.225
0.735
0.966
0.999
SAE-4
0.740
0.209
0.654
0.945
0.999
VLAE-12
0.832
0.553
0.751
0.914
0.977
VLAE-6
0.785
0.326
0.689
0.929
0.963
VLAE-4
0.690
0.282
0.544
0.900
0.926
STCVAE
0.410
0.237
0.603
0.865
0.923
FVAE
0.330
0.123
0.725
0.907
0.955
3VAE
0.235
0.127
0.593
0.879
0.799
VAE
0.314
0.138
0.607
0,892
0.872
WAE
0.211
0.050
0.621
0.946
0.906
AE
0.307
0.092
0.638
0.926
0.943
AdaAB-12
0.299
0.062
0.503
0.876
0.999
Table 1: Disentanglement and Completeness scores for 3D-Shapes. The DCI-d metric corresponds to the DCIIddisentanglement score and DCI-c to the completeness score (Fastwood Williams) 2018), IRS is similar disentanglement metric (Suter al. 2018), the MIG is the Mutual Infor- mation Gap (Chen et al. (2018), and Mod/Exp refers to the ModulariyyExpliiinees scores respec- tively (Ridgeway & Mozer 2018) (for all these metrics higher is better)
17

---

Published as a conference paper at ICLR 2023
SAE-12
SAE-D
DCI
BTCVA
AdaAE-
MIG
BTC
AdaAE-
SA
IRS
BTCV
W
AdaAE.
SA
SAP
AdaAE
Mod
WA
AdaAE-
EXp
AdaAE.
0.0
0.2
0.4
0.6
0.8
1.0
Score
Figure 10: Several disentanglement metrics for all the models and each of the seeds. (for all these metrics higher is better)
18

---

Published as a conference paper at ICLR 2023
(a) SAE-12
(b) SAE-6
SAE-4
(d) SAE-3
(c) SAE-2
(f) AdaAF-12
19

---

Published as a conference paper at ICLR 2023
g VLAFF-1
(h) VLAE-6
(i) VLAE-4
(i) VAE
(k) 3-VAE
()) AE
Figure 11: Latent Traversals several models for 3D-Shapes. Each row shows the generated image when varying the corresponding latent dimension while fixing the rest of the latent vector. For the SAE and VLAE models, the groups of dimensions that are into the same Str-Tfmm layer (or ladder rung) are grouped together. Note the disentangled segments achieved by the SAE models and the consistent ordering of factors of variation.
A.3.3 EXTRAPOLATION
We present results variant of the exptrapolation experiment discussed in section 3.2 modifying the shape in the initial training dataset, we remove the two most extreme camera angles in either direction (removing 4/15 of the full dataset).
Instead of
As seen from figure I2 in this setting the warping and generally lower fidelity experienced by only updating the encoder compared to updating the decoder is very apparent.
20

---

Published as a conference paper at ICLR 2023
Model
Neither
Encoder
Decoder
Both
SAE-12
4.88
2.65
0.77
0.43
VLAE-12
6.91
3.83
1.61
0.77
AdaAE-12
4.8
2.93
0.59
0.41
AE
4.98
2.94
0.62
0.45
WAE
4.79
3.07
0.69
0,44
VAE
5.17
3.09
1.01
0.51
3VAE
5.7
3.97
1.5
0.82
Figure 12: Same figure 7 except for the camera angle setting. Although the results generally very consistent, note how much better the SAE- 12 model performs than the VLAE-12 this setting.
A.3.4 MPI3D-TOY
Model
DCI
MIG
IRS
Mod
Exp
SAE-12
0.642
0.487
0.570
0.938
0.946
SAE-6
0.454
0.094
0.553
0.918
0.911
VLAE-12
0.414
0.323
0.667
0.909
0.842
STCVAE
0.091
0.007
0.605
0.858
0.678
EVAE
0.108
0.029
0.680
0.876
0.743
SVAE
0.046
0.004
0.987
0.998
0,621
VAE
0.093
0.078
0.621
0.861
0.659
WAE
0.203
0.028
0.633
0.904
0.859
AE
0.186
0.043
0.632
0.911
0.844
AdaAB-ID 12
0.208
0.080
0.546
0.919
0.931
Table Disentanglement and Completeness scores for MPI3D-Toy. (for all these metrics higher is better)
21

---

Published as a conference paper at ICLR 2023
(a) SAE-12
(b) SAE-6
SAE-4
I
(d) SAE-3
(c) SAF-2
(f) AdaAF-12
22

---

Published as a conference paper at ICLR 2023
g VLAF-11
(h) VLAE-6
ii VLAE-4
(i) VAE
(k) 3-VAE
()) AE
Figure 13: Latent Traversals of several models for MP13D-Toy. Each row shows the generated image when varying the corresponding latent dimension while fixing the rest of the latent vector. For the SAE and VLAE models, the groups of dimensions that are fed into the same Str-Tfm layer (or ladder rung) grouped together. Note the disentangled segments achieved by the SAE models and the consistent ordering of factors variation.
23

---

Published as a conference paper at ICLR 2023
A.3.5 MPI3D-SIM
Model
DCI
MIG
IRS
Mod
Exp
SAE-12
0.411
0.238
0.508
0.930
0.890
SAE-6
0.294
0.052
0,479
0.928
0.877
VLAE-12
0.220
0.093
0.634
0.863
0.816
STCVAE
0.148
0.139
0.688
0.856
0.694
EVAE
0.095
0.044
0.664
0.916
0.722
SVAE
0.060
0.054
0,850
0.926
0.701
VAE
0.070
0.056
0.850
0.828
0.713
WAE
0.129
0.033
0.548
0.881
0.819
AE
0.157
0.033
0.526
0.855
0.805
AdaAE-I 12
0.159
0.022
0.481
0.893
0.905
Table 3: Disentanglement and Completeness scores for MPI3D-Sim (for all these metrics higher is better)
24

---

Published as a conference paper at ICLR 2023
(a) SAE-12
(b) SAE-6
SAE-4
(d) SAE-3
(c) SAF-2
(f) AdaAF-12
25

---

Published as a conference paper at ICLR 2023
g VLAFF-1
(h) VLAE-6
(i) VLAE-4
4
4
(i) VAE
(k) 3-VAE
()) AE
Figure 14: Latent Traversals of several models for MPI3D-Sim. Each row shows the generated image when varying the corresponding latent dimension while fixing the rest of the latent vector. For the SAE and VLAE models, the groups of dimensions that are fed into the same Str-Tfm layer (or ladder rung) grouped together. Note the disentangled segments achieved by the SAE models and the consistent ordering of factors variation.
26

---

Published as a conference paper at ICLR 2023
A.3.6 MPI3D-REAL
Model
DCI
MIG
IRS
Mod
Exp
SAE-12
0.374
0.148
0.564
0.928
0.869
SAE-6
0.295
0.074
0.535
0.879
0.840
VLAE-12
0.291
0.217
0.579
0.914
0.805
STCVAE
0.185
0.100
0.595
0.870
0.699
EVAE
0.095
0.028
0.522
0.904
0.729
SVAE
0.090
0.021
0.659
0.869
0,680
VAE
0.080
0.020
0.602
0.875
0.694
WAE
0.159
0.042
0.587
0.837
0.831
AE
0.143
0.048
0,563
0.858
0.804
AdaAE-I3 12
0.164
0.034
0.530
0.952
0.895
Table 4: Disentanglement and Completeness scores for MP13D-Real. (for all these metrics higher is better)
27

---

Published as a conference paper at ICLR 2023
(a) SAE-12
(b) SAE-6
SAE-4
(d) SAE-3
(c) SAF-2
(f) AdaAF-12
28

---

Published as a conference paper at ICLR 2023
g VLAFF-1
(h) VLAE-6
(i) VIAE-4
(i) VAE
(k) 3-VAE
()) AE
Figure 15: Latent Traversals several models for MP13D-Real. Each row shows the generated image when varying the corresponding latent dimension while fixing the rest of the latent vector. For the SAE and VLAE models, the groups of dimensions that are fed into the same Str-Tfm layer (or ladder rung) grouped together. Note the disentangled segments achieved by the SAE models and the consistent ordering of factors of variation.
29

---

Published as a conference paper at ICLR 2023
A.3.7 CELEB-A
(a) Original samples (from test set)
(b) SAE-16 Reconstructions
(c) AdaAE-16 Reconstructions
30

---

Published as a conference paper at ICLR 2023
(d) VLAE-16 Reconstructions
(c) VAE Reconstructions
(f) AE Reconstructions
Figure 16: Reconstructed samples using several models trained on CelebA. Note the blurriness of the VAE-based baselines compared to SAE and AdaAE.
31

---

Published as a conference paper at ICLR 2023
(a) SAE-16 Hybrid Sampling
(b) AdaAE-16 16 Hybrid Sampling
(c) VIAA--16 Hybrid Sampling
32

---

Published as a conference paper at ICLR 2023
(d) VLAE-16 Prior Sampling
(c) VAE Hybrid Sampling
(f) VAE Prior Sampling
Figure 17: Samples generated using hybrid and brser- sampling using several models trained on CelebA. Note that the hybrid sampling tends to produce relatively high quality samples both for our proposed SAE and AdaAE architectures well as baselines.
33

---

Published as a conference paper at ICLR 2023
A.3.8 RFD
(a) SAE-16
(b) VLAE-16
34

---

Published as a conference paper at ICLR 2023
(c) AdaAE-16
(d) VAE
Figure 18: Latent traversals for the RFD dataset. Each corresponds to ID traversal of the corresponding latent dimension while the other latent dimensions are fixed. Note the ordering of information in the more structured models like the SAE-16 16 and VLAE-16.
35


Reviews:
[{"review_id": "tzPgSPdfX1J", "reviewer": "Reviewer_ECfH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an autoencoder architecture for structured representation learning. In particular, the proposed model incorporates latent variables at different layers of the convolutional decoder through Structural Transform (StrTfm) layers (inspired by FiLM (Perez et al., 2018) and Ada-In (Karras et al., 2019) layers). In essence, an StrTfm layer applies an affine transformation to the feature maps from the preceding convolutional layer. The latent variable associated with an StrTfm layer determines the magnitude and bias of the affine transformation corresponding to the layer. The latent variables are not regularized by a prior distribution and can be sampled independently for generation based on their training distribution. The proposed model is trained on several datasets and performs favorably when compared to various baselines and benchmarks (different versions of VAEs) based on several evaluation metrics. The evaluation metrics include reconstruction quality (measured by reconstruction loss), visual quality of generated samples (measured by FID), and disentanglement of latent features (measured by DCI, MIG, IRS, Mod/Exp). Additional qualitative analysis shows that the latent space of the structured decoder exhibits some hierarchy where latent codes incorporated at the bottom layers of the decoder (farthest from reconstruction) represent high-level information (such as shape) while latent codes at the top layers of the decoder represent more lower-level information (such as color). ", "summary_of_the_review": "This paper proposes a way to learn a hierarchy of latent representations for generation with an autoencoder. The latent variables are not regularized by a fixed prior distribution and can be sampled independently to generate new samples with relatively high visual fidelity when compared to other models. I have some clarifying questions regarding the methodology and assumptions in the paper and tentatively propose a weak accept. \n", "strength_and_weaknesses": "**Strengths**\n\n- The proposed structured decoder which works with a hierarchy of latent variables affecting reconstruction at different scales through StrTfm layers and without regularizing the latent variables to have a pre-set prior distribution is novel.\n\n- The proposed structured decoder is compared to various baselines and shows favorable performance in terms of reconstruction quality, FID, disentanglement, and hierarchy of information encoded in the latent representations.\n\n- The authors provide implementation of their model in the supplementary materials (although I didn’t have a chance to check it closely.)\n\n**Room for improvement**\n\nPlease find below clarification questions regarding some of the statements and methodology in the paper.\n\n- *Statistical significance*: Are confidence intervals available for the results in Figure 2 (comparing reconstruction quality for the different models)? \n\n- *Fairness of comparison*: Is it accurate to say that the SAE model has more parameters than the VAE models it is compared to due to the presence of MLP modules in StrTfm layers? If so, could the favorable performance of SAE models in terms of reconstruction quality and FID be partly due to higher capacity of the structured decoder?\n\n- *Initialization of the affine transformation*: Can the authors please elaborate on how the “affine transform for each Str-Tfm layer is initialized to identity” if the hidden layers (I assume the MLPs) are “independently initialized”? I am probably missing some details but are not $\\alpha_i$ and $\\beta_i$ computed as the outputs of passing $U_i$ through MLP$_i$?\n\n- *Latent traversal*: For the visualizations in Figure 4, when traversing values for one latent dimension, what are the remaining latent dimensions set to? Can the authors elaborate on why there does not seem to be a lot of variation in the generated samples in the top two rows of Figure 4(a)?\n\n- *Information bottleneck acts as regularization*: The latent representations have much lower dimensionality than the inputs ($d\\ll  D$) by design which introduces an informational bottleneck that acts as regularization to avoid trivial solutions. How do authors expect the SAE’s performance will be affected as $d$ increases in terms of the reported metrics?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qxbkDRcpjBU", "reviewer": "Reviewer_HNCN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "An architecture of autoencoders for images that enable disentanglement of the latent variable is proposed. It is contrasted with disentanglement approaches based on regularization. The authors empirically investigate the properties of the proposed method and other disentanglement approaches. One of the conclusions is that the specific architecture of the proposed autoencoder allows the disentanglement of latent representation without any regularization.", "summary_of_the_review": "This is a solid paper with an intensive empirical study on the disentanglement happening in different types of autoencoders.", "strength_and_weaknesses": "### Strengths\n\n- Although the method seems simple, it seems to disentangle the representation as nicely reported with intensive experiments.\n- Not only the proposed method but also other popular approaches for disentanglement are investigated in different aspects.\n\n### Weaknesses\n\n1. The method's applicability is limited to data for which CNNs are meaningful. This is not a serious weakness, but a brief discussion about the (im)possibility of application to non-image data types would make the paper more complete.\n\n2. Related to the above, the discussion starts by assuming that the data are images without mentioning so at all. This might be natural in the disentanglement community, but for a broader range of audience, limiting the data type first, even just for the sake of discussion, would be helpful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pb_PX6RL4Pm", "reviewer": "Reviewer_X8MV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers an autoencoder architecture where the structure in the latent space is induced by architecture of the decoder, which uses latent variables sequentially to generate samples from the target distribution. Rather than imposing a prior distribution on the latent space authors use a “hybrid sampling” approach to generate samples that involves picking components of latent vectors from the cross product of a set of randomly chosen examples from the dataset. Authors study the effectiveness of their approach using FID scores, disentanglement scores and a series of visual inspections on several datasets including (MPI3D, Celeb-A, 3D-Shapes). Authors find that their models perform favorably with respect to other baselines, especially on the disentanglement metrics.\n", "summary_of_the_review": "While the paper presents a new and interesting approach to adding structure to a latent representation by modifying the decoder architecture and how the latent vectors are converted to samples from the data distributions, I’m not convinced that current results offer evidence to support the claim that this is a principled step in a right direction for representation learning. I might have misunderstood parts of the paper in which case I’d love to be convinced otherwise.\n\nMy main concerns are:\n(1) IIUC objective scores like log likelihood of the test dataset are not possible to calculate with hybrid type sampling, while scores like FID are easily fooled by memorization.\n(2) Other generative models have not been considered (e.g. VQ-VAE-2 [1], RG-Flow[2] etc.)\n\n\n\n\n[1] https://arxiv.org/pdf/1906.00446.pdf\n[2] https://iopscience.iop.org/article/10.1088/2632-2153/ac8393/pdf\n", "strength_and_weaknesses": "Strengths:\n* Carries out experiments on several datasets.\n* Paper suggests a simple and clearly explained approach\n\nWeaknesses:\n* Model is evaluated on rather simple test problems\n* Some natural baselines like VQ-VAE are not represented\n* Limitations/caveats arising from the choice of hybrid sampling are not clearly addressed/stated/investigated. (e.g. effect on FID score etc.)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "U4WkdpztQTN", "reviewer": "Reviewer_hCAA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed SAE, which uses structural decoder that infuses latent information one variable at a time to induce an intuitive ordering of information, and provided a sampling method called hybrid sampling which replies only on independence between latent variables without imposing a prior latent distribution. \n\nThe paper compared the metrics with many other baselines, and show improved metrics.", "summary_of_the_review": "The paper proposed a new model architecture and a new sampling method at the same time, it's difficult to see which gives the real contribution. And it's not clear about the diversity of the generated images. ", "strength_and_weaknesses": "Strength\n1. The paper showed SAE has better metrics when compared with 6 baselines, and the authors even compared the model performance when hybrid sampling are used in the baseline models.\n\nWeaknesses\n1. The hybrid sampling method assumes the independence between the latent variables. The model structure uses one latent variable at a time, but given that all the latent variables are trained at the same time. Not sure how independent are these latent states.\n2. Another baseline missing from the paper is autoencoder trained without prior, and sampling based on the proposed hybrid sampling. Because hybrid sampling removed the need of prior distribution for the latent variables, the VAE model can also remove the prior.\n3. Base on the generated images Figure4 and Figure 6, the diversity of generated images seems poor.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster