DAVA: Disentangling Adversarial Variational Autoencoder
Authors: Benjamin Estermann
Roger Wattenhofer
Venue: ICLR 2023 poster (2023)
Keywords: Disentanglement learning
varational auto-encoder
curriculum learning
generative adversarial networks

Abstract: The use of well-disentangled representations offers many advantages for downstream tasks, e.g. an increased sample efficiency, or better interpretability.
However, the quality of disentangled interpretations is often highly dependent on the choice of dataset-specific hyperparameters, in particular the regularization strength.
To address this issue, we introduce DAVA, a novel training procedure for variational auto-encoders. DAVA completely alleviates the problem of hyperparameter selection.
We compare DAVA to models with optimal hyperparameters.
Without any hyperparameter tuning, DAVA is competitive on a diverse range of commonly used datasets.
Underlying DAVA, we discover a necessary condition for unsupervised disentanglement, which we call PIPE.
We demonstrate the ability of PIPE to positively predict the performance of downstream models in abstract reasoning.
We also thoroughly investigate correlations with existing supervised and unsupervised metrics. The code is available at https://github.com/besterma/dava.

Published as a conference paper at ICLR 2023
DAVA: DISENTANGLING ADVERSARIAL VARIATIONAL AUTOENCODER
Benjamin Estermann ETH Zürich
Roger Wattenhofer ETH Zürich Switzerland
Switzerland
este astermann@athz.cr
Setttenhofer@ethz.cc
ABSTRACT
The use of well-disentangled representations offers many advantages for down- stream tasks, e.g. increased sample efficiency, better interpretability. However, the quality of disentangled interpretations is often highly dependent on the choice datasct-specific hyperparameters, particular the regularization strength. To address this issue, we introduce DAVA, novel training procedure for variational auto-encoders. DAVA completely alleviates the problem of hyperparameter selec- tion. We compare DAVA to models with optimal hyperparameters. Without any hyperparameter tuning, DAVA is competitive on diverse range commonly used datasets. Underlying DAVA, we discover a necessary condition for unsupervised disentanglement, which we call PIPE. We demonstrate the ability of PIPE to posi- tively predict the performance of downstream models in abstract reasoning. We also thoroughly investigate correlations with existing supervised and unsupervised metrics. The code is available at github.com/beestrmm/daaa
INTRODUCTION
Real-world data tends to be highly structured, full of symmetries and patterns. This implies that there exists lower-dimensional set of ground truth factors that is able explain significant portion of the variation present real-world data. The goal of disentanglement learning to recover these factors, so that changes in single ground truth factor are reflected only in single latent dimension of a model (see Figure for an example). Such an abstraction allows for more efficient reasoning (Van Steenkiste al.) (2119) and improved interpretability (Higgins et al., 2017a). It further shows positive effects on zero-shot domain adaption (Higgins et al. (2017)) and data efficiency (Duan et al. 2020 Schoot et 2022).
Figure I: Latent traversals of single latent dimension (hair fringes) of DAVA trained on CelebA. DAVA visibly disentangles the fringes from all other facial properties.
the generative ground-trutt factors are known and labeled data is available, one can train a model in supervised manner to extract the ground-truth factors. What if the generative factors are unknown, but one still wants to profit from the aforementioned benefits for a downstream task? This may be necessary when the amount of labeled data for the downstream task is limited training is computationally expensive. Learning disentangled representations in an unsupervised fashion is generally impossible without the of some priors (Locatello et al. (019b). These priors can be present both implicitly in the model architecture and explicitly in the loss function (Tschannen et al.) 2018). An example of such prior present in the loss function is low total correlation between

---

Published as a conference paper at ICLR 2023
latent variables of model (Chen et al. 2018 Kim & Mnih 2018). Reducing the total correlation has been shown to have a positive effect on disentanglement (Locatello et al. (011b)). Unfortunately, as show in more detail in this work, how much the total correlation should be reduced to achieve good disentanglement is highly dataset-specific. The optimal hyperparameter setting for one dataset may yield poor results another dataset. To optimize regularization strength, we need way to evaluate disentanglement quality.
So how can we identify well disentangled representations? Evaluating representation quality, even given labeled data, is no easy task. Perhaps as an example of unfortunate nomenclature, the often-used term "ground- truth factor" implies the existence of a canonical set of orthogonal factors. However, there are often multiple equally valid sets of ground truth factors, such as affine transformations of coordinate axes spanning space, different color representations, or various levels of abstraction for group properties. This poses problem for supervised disentanglement metrics, since they fix the ground truth factors for evaluating representation and judge the models too harshly if they have learned another equally valid representation. Furthermore, acquiring labeled data in a practical setting usually a costly endeavor. The above reasons hinder the usability of supervised metrics for model selection.
In this work, we overcome these limitations for both learning and evaluating disentangled represen- tations. Our improvements are based on the following idea: We define two distributions that can be generated by a VAF. Quantifying the distance between these two distributions yields disentan- glement metric that is independent of the specific choice of ground truth factors and reconstruction quality. The further away these two distributions are, the less disentangled the VAEs latent space is. We show that the similarity of the two distributions is necessary condition for disentanglement. Furthermore, we can exploit this property at training time by introducing an adversarial loss into classical training of VAEs. To do this, we introduce discriminator network into training and use the VAEs decoder as generator. During training, we control the weight of the adversarial loss. We adjust the capacity of the latent space information bottleneck accordingly, inspired by (Burgess et al. 2017). In this way, we allow the model to increase the complexity of its representation long as it is able to disentangle.
This dynamic training procedure solves the problem of dataset-specific hyperparameters and allows our approach to reach competitive disentanglement on a variety of commonly used datasets without hyperparameter tuning.
Our contributions are as follows:
We identify novel unsupervised aspect of disentanglement called PIPE and demonstrate its usefulness in a metric with correlation to supervised disentanglement metrics as well as a downstream task.
We propose an adaptive adversarial training procedure (DAVA) for variational auto anto-enc which solves the common problem that disentanglement performance is highly dependent on dataset-specifie regularization strength.
We provide extensive evaluations on several commonly used disentanglement datasets to support our claims.
2 RELATED WORK
2.1 MODEL ARCHITECTURES
The V-E by Higgins al. (2017a) cornerstone model architecture for disentanglement learning. The loss function of the 3- V-EAE, the evidence lower bound (ELBO), consists of a reconstruction term and KL-divergence term weighted by 8, which forces the aggregated posterior latent distribution to closely match the prior distribution. The KL--dieegeecee term seems to promote disentanglement as shown in (Rolinek et al. 2019). The 3-TCVAE architecture proposed by Chen et al. (2018) further decomposes the KL divergence term of the ELBO into an index-code mutual information, total correlation and a dimension wise KL term. They are able to show that it is indeed the total correlation that encourages disentanglement and propose tractable but biased Monte Carlo estimate. Similarly, the Factor VAE architecture (Kim & Mnih| 2018) uses the density ratio trick with an adversarial network to estimate total correlation. The AnnealedVAF. architecture (Burgess et al. 2017) well
2

---

Published as a conference paper at ICLR 2023
ControlVAE (Shao et al. 2020) build different premise, arguing that slowly increasing the information bottleneck capacity of the latent space leads to the model gradually learning new latent dimensions and thus disentangling them. We will use a similar but optimized approach for DAVA. More recent promising approaches are presented by Wei et al. (202T) using orthogonal Jacobian regularization and by Chen et al. (202T) applying regulatory inductive bias recursively over a compositional feature space.
2.2 INTRODUCTION OF ADVERSARIAL TRAINING TO VAES
When combining autoencoders (AEs) with an adversarial setting, can connect the adversarial network either on the latent space or on the output of the decoder. Makhzami et al. (2016) proposed an adversarial AE (AAAE) that uses adversarial discriminator network the latent space of an AE to match its aggregated posterior to an arbitrary prior. There, the encoder of the AE acts as the generator of the GAN and is the only part of the AE that gets updated with respect to the discriminator's loss. This kind of training is strongly connected to VAE training, with the ad adversarial loss taking on the role of KL divergence in the classical VAE training objective but without the constraint of Gaussian prior. The previously mentioned FactorVAE (Kim & Mnih] (018)) implements an adversarial loss on the latent space to reduce total correlation. Larsen et al. (2016) proposed using discriminator network with the decoder of the VAE acting generator to improve the visual fidelity of the generated images, but with no focus on disentanglement. The difference to DAVA is that they used the discriminator on the real observations, while propose discriminator that only sees observations generated by the decoder of the VAE. Zhuu et al. (2020) introduce an recognition network based loss on the decoder that encourages predictable latent traversals. In other words, given a pair of images where all but one latent dimensions are kept constant, the recognition network should be able to predict in which latent dimension the change occurred. Applied top of a baseline VAE model, this loss slightly improves the disentangiement performance of the baseline VAE. In a semi-supervised setting, Carbajal et al. (02T and Han et al. (2020) propose adversarial losses on the latent space of VAES disentangle certain parts of the latent space from information present in labels. Unfortunately. such an approach does not work in an unsupervised setting with the goal of disentanglement of the complete latent space.
2.3 MEASURING DISENTANGLEMENT
All supervised metries have in common that they are dependent on canonical factorization of ground truth factors. Given access to the generative process of a dataset, the FVAE metric (Kim & Mnih 2018) can be used to evaluate a model. first creates a batch of data generated by keeping aforementioned ground truth factor fixed and randomly sampling from the other ground truth factors. then uses a majority vote classifier to predict the index of ground truth factor given the variance of each latent dimension computed over said batch. Without access to the generative process, but given a number of fully labeled samples, one can use metrics like DCI Disentanglement by [Estwood & Williams (2018) and MIG by Chen al. (2018). While DCI in essence assesses if latent variable of a model captures only a single ground truth factor, MIG evaluates if a ground truth factor is captured only by single variable. As result, DCI compared to MIG does not punish multi-dimensionn representations of a single ground truth factor, for example the RGB model of color or a sine/cosine representation of orientation.
There exists a small number of unsupervised disentanglement metrics. The unsupervised disentan- glement ranking (UDR) by Duan al. (Duan et al.) (020)) evaluates disentanglement based on the assumption that a representation should be disentangled if many models, differently initialized but trained with the same hyperparameters, learn the same representation. To achieve this, they compute pairwise similarity (up to permutation and sign inverse) of the representations of group of models. The score of a single model is the average of its similarity to all the other models in the group. ModelCentrality (MC) (Lin et al., (2020) builds on top of UDR by improving the pairwise similarity evaluation. A drawback is the high computational effort, as to find the optimal hyperparameter setting, multiple models need to be trained for each setting. UDR and MC do not assume any fixed set of ground truth factors. Nevertheless, a weakness of these approaches is that they do not recognize similarity of group of models that each learn different bijective mapping of the ground truth factors. The latent variation predictability (VP) metric Zhu et al. (2020) is based on the assumption that if representation is well disentangled, it should be easy for recognition network to predict
3

---

Published as a conference paper at ICLR 2023
which variable was changed given two imput images with change in only one dimension of the representation. An advantage of the VP metric its ability to evaluate GANs as is not dependent on the model containing dependent on the size of the latent space or need to define additional hyperparameters to recognize inactive dimensions. [stermann et al. (0200) showed that the UDR has strong positive hias for low-dimensional representations, as Now-dimensional representations are more likely to be similar each other. One could see the same issue arise for the VP metric, as accuracy for the recognition network will likely be higher when there is only a low number of thange-inducing latent dimensions available to choose from.
encoder. In comparison to our proposed metric, VP, UDR and MC are
2.4 APPLICATIONS OF DISENTANGLED REPRESENTATIONS
The effects of disentangled representations on downstream task performance are manifold. Van Steenkiste et al. (2019) showed that for abstract reasoning tasks, disentangled representations en- abled quicker learning. We show in Section 4.2 that the same holds for our proposed metric. In work by Higgins et al. (b)1 disentangled representations provided improved zero-shot domain adaption for multi-stage reinforcement-leanee leaanng (RL) agent. The UDR metric (Duan et al. (020) correlated well with data efficiency of a model based RL agent introduced by Watters et al. (Watters et al. 2019). Disentangled representations further seem to be helpful in increasing the fairness of downstream prediction tasks (Locatello et al. 2019a). Contrary to previous qualitative evidence (Eslami et al.) 2018 Higgins et al.] 2018), Montero et al. (2020) present quantitative findings indicating that dis- entanglement does not have an effect on
(OOD) generalization. Recent work by Schout et al. (2022) supports the claims Montrro al. (2020), concluding that disentanglement
shows improvement in downstream task performance but not so in OOD generalization.
POSTERIOR INDIFFERENCE PROJECTION EQUIVALENCE
We first introduce the notation of the VAE framework, closely following the notation used by Kim & i are generated by an unknown process based on
Mnih (2018). We assume that observations some independent ground-truth factors. The goal of the encoder is to represent in a latent vector E Rd. We introduce a Gaussian prior p(z) N N I) with identity covariance matrix on the latent space, The variational posterior for a given observation x is then qe (2|z) = - (x)) where the encoder with weights 9 outputs mean and variance. The decoder with weights é projects from the latent space z back to observation space p& (x|=)) We can
define the distribution q(z).
Definition 3.1 (EP). The empirical posterior (EP) distribution ((2) of VAE the multivariate distribution of the latent vectors: over the data distribution Đ. More formally, q(=) B (1e (=]=:)]-
We can reconstruct an observation x ~ δ the following way: We sample ~ qo (=(xr) and then get the reconstruction ~ (x(2)) We informally call observations generated by this process reconstructed samples and denote them as a.
The decoder is not constrained to only project from 4(2) to observation space. We can sample observations from the decoder by using different distributions on 2. We define a particularly useful distribution.
Definition 3.2 (FP). The factorial posterior (FP) distribution ((z) of a VAE is a multivariate distribu- tion with diagonal covariance matrix. We define it as the product of the marginal EP distribution: ((2) = {(z)). We can use the decoder to project 2 ~ (2) to observation space ă (x(2)) We informally call observations created by this process generated samples and denote them as F.
We can now define the data distributions that arise when using the decoder to project images from either the EP or the FP. Definition 3.3 (DEr, Dre). DEP is generated by the decoder projecting observations from the EP latent distribution q(z), i.c reconstructed samples. DEP = Ez~g(=) [Po (z|=)]. Dpp is generated by the decoder projecting observations from the FP distribution q(=), i.e. generated samples. Drp = Eg--(=) (x|=).
We can now define the core concept of this paper. We look the similarity of two data distributions generated by the decoder, DEp and Drp.
4

---

Published as a conference paper at ICLR 2023
(a) Reconstructed samples.
(b) Generated samples.
(c) Reconstructed samples.
(d) Generated samples.
Figure Shown are two models of dissentanglementt -lib (Locatello et al. 2019b) with different disentanglement performance. The upper part shows poorly disentangled model while the lower part shows a comparatively well disentangled model. In (a) and (c), odd columns show images sampled from the AbstractDSprites dataset, even columns show the corresponding model reconstruction. (b) and (d) show images generated by the decoder of each model given the factorial posterior distribution. While the upper model achieves better reconstructions (a), its generated samples (b) are visibly out of distribution. The bottom model ignores shape in its reconstructions (c), its generated samples (d) capture the true data distribution more accurately.
Definition 3.4 (PIPE). The posterior indifference projection equivalence (PIPE) of VAE represents the similarity of the data distributions DEP and Dyp. In other words, PIPE is measure of the decoder's 's indifference to the latent distribution it projects from.
PIPE(O, σ) ((E (=)l=))))))))))))))))))))))))))) Eo
where
general similarity measure.
Unsupervised disentanglement learning without any inductive biases is impossible as has been proven by Locatello et al. (2019b). Given a disentangled representation, it is always possible to find a bijective mapping that leads to entangled representation. Without knowledge of the ground truth factors, impossible to distinguish between the two representations. We argue that PIPE is necessary for disentangled model, even when it is not sufficient. Suppose model has learned disentangled representation, but DEP happen. The first possibility is that the latent dimensions of the model are not independent. This violates the independence assumption of the ground truth factors, so the model cannot be disentangled. The second possibility is that the model has not learned representation of the ground truth factors and is generating samples that are not represented by the ground truth factors. This model would be disentangled either. We conclude that PIPE is necessary condition for disentanglement.
equivalent to DFp- There are cases where this could
In Figure we show reconstructed and generated samples of two different models to support our chain of reasoning. Generated samples of the entangled model (Figure 2 (b)) look visibly out of distribution compared to the reconstructed samples. Generated samples of the disentangled model (Figure (d)) appear to be equivalent the distribution of the reconstructed samples.
4
PIPE METRIC
We propose way to quantify the similarity metric of PIPE with neural network and call this the PIPE metric. Given a model M with encoder weights 0, decoder weights ø, and corresponding DEP, DFP (see Appendix B.2 for details on how to sample from DEP and Dild)
1. Create a set of observations SgP by sampling from Der- Informally, these are the recon- structed samples.
2. Create a set of observations Spp by sampling from Dyp. Informally, these are the generated samples.
5

---

Published as a conference paper at ICLR 2023
3. Randomly divide SgP U Spp into train and test set.
4. Train discriminator network on the train set.
5. Evaluate accuracy acc of the discriminator on the test set.
6. Since random discriminator will guess half of the samples accurately, we report a score of
2 (1 acc), such that is the best and 0 is the worst score.
We train the discriminator network for 10, 000 steps keep the distinction of SEP and SEP suffi- ciently difficult. We further use a uniform factorial distribution instead of FP for slight improvement in performance. Details the implementation can be found in Appendix B
4.1 RELATION TO EXISTING METRICS
To classify the performance of the PIPE metric, we evaluate correlations with supervised metrics a diverse set of commonly used datasets. We namely consider Shapes3D (Burgess & Kim (0118). AbstractDSprites (Van Steenkiste et al.] 2019) and Mpi3d Toy (Gondal et al. 2019). All datasets are part of di sentangl ement- 1 1 (Locatello et al.) 2019b). which we used to train the models we evaluated our metric on. More details on the implementation of the PIPE metric and the evaluated
models can be found in Appendix B Results are displayed in Figure 3 They show that the PIPE metric correlates positively with existing supervised metrics DCI, MIG and FVAE, surpassing the correlations of the unsupervised baselines. While the performance of UDR on Mpi3d Toy and the performance MC on AbstractDSprites lacking. PIPE demonstrates consistent performance across all datasets. Further, as opposed to UDR and MC, PIPE can be evaluated on a single model only, whereas UDR and MC are only able to evaluate sets of models.
3
AbstraciDpprites
Mpi3d Toy
PIPE (ours)
100
100
100
UDR
45
100
35
100
100
MC
26
80
100
-18
21
100
2
100
Rec
80
41
28
100
58
-37
100
24
-76
100
DCI
48
58
36
14
100
56
4
100
-18
67
-57
100
MIG
64
48
22
36
87
100
64
43
8
43
72
100
24
-11
29
-25
57
100
FVAE
35
17
71
100
42
18
72
100
-30
-22
22
100
PIPE (ours)
UDR
MC
Rec
DCI
MIG
FVAE
PIPE (ours)
UDR
MC
DCI
MIG
EVAB
PIPE (ours)
UDR
MC
Rec
DCI
MIG
FVAB
Figure 3: Spearman rank correlation between different metrics on three different datasets. Correlations take values in the range 1], where 1 means perfect correlation, O means no correlation, and negative values mean anti-correlation. For better readability, we have multiplied all values by 100. PIPE, UDR (Duan et al. 2020) and MC (Lin et al. 2020) the corresponding unsupervised metrics. We include Reconstruction Loss Rec as a trivial unsupervised baseline. DCI, MIG and FVAE are the corresponding supervised metrics. Low and MC show that our metric captures a different aspect of lisentanglement The correlation of our metric with supervised disentanglement metries mostly consistent across datasets. We note that the correlation with supervised metrics on the Mpi3d Toy dataset changes direction for both UDR and Rec, while MC has difficulties with AbstractDSprites.
negative correlations of our metric with UDR
4.2 CORRELATION WITH DOWNSTREAM TASK PERFORMANCE
To further quantify the usefulness of the PIPE metric, we analyze the predictive performance of the metric for an abstract reasoning downstream task. For the downstream task, VAE is trained to learn a disentangled representation of a dataset. The downstream model then only gets access to this representation then trying solve the downstream task. Van Steenkiste al. 2019) evaluated

---

Published as a conference paper at ICLR 2023
different supervised disentanglement metrics in terms of predictive performance of accuracy in an abstract reasoning task on the datasets Shapes3D and AbstractDSprites. They showed that good disentanglement is indicator of better accuracy during the early stages training. We reproduce their experiment on reduced scale by only considering up to 6,000 training steps. As can be seen in Figure 4 metric is par with supervised metrics and clearly outperforms the unsupervised baselines, More importantly this means that PIPE is a positive predictor of accuracy of downstream models few-sample setting and therefore desirable property for unsupervised model selection.
Shupex33
AbstractDSpritts
PIPE (ours)
83
UDR
MC
40
Unsupervised (For comparison)
53
-49
Rec
46
38
71
DCI
76
78
Supervised (For context)
81
72
MIG
79
85
77
75
FVAE
VAc
3-TCVAE
FactorVAE
3-TCVAE
Figure 4: Spearman rank correlation between different disentanglement metrics and downstream
accuracy of the abstract visual reasoning task (Van Steenkiste et al. 2019) after 6,000 training steps for Factor VAE and B-TCVAE. The same metrics as in Figure 3 are evaluated. Correlation for unsupervised metrics are more sensitive model architecture than for supervised metrics. PIPE shows higher correlation with downstream performance than UDR and clearly outperforms the Rec baseline.
TRAINING PROCEDURE FOR VARIATIONAL AUTO-ENCODERS (DAVA)
In the previous chapter have established that PIPE is desirable for a model. We now present how to encourage this property training time by designing an adversarial training procedure. We train discriminator and VAE at the same time. The discriminator needs to differentiate DEP from DFP, which is achieved with the fol- lowing loss:
~ ((z)
Enc
DaciGen
Le
Lre
Dis
Eass
Lais DENDDa [lg((Ds(())) Dis(
Figure 5: Overview of the composition of DAVA. A discriminator attached the decoder of VAE. The goal for the discriminator is to differentiate samples coming from the EP from those coming from the FP.
E D. log(1 (1- - Dis(E))]-
The loss function for the VAE is compromised of reconstruction term with the weighted negative objective of the discriminator:
Lade
Lree -LLdis-
with reconstruction loss
Cree = -E (egge
While this works in practice, it faces the issue of the weight being very dataset-specific. It is also closely related to FactorVAE, with the difference that FactorVAE applies the adversarial loss to the latent space and to the encoder only. Let consider an alternative approach. Work by Burgess et al. (2017) looked at the KL-Divergence from an information bottleneck perspective. They proposed using a controlled latent capacity increase by introducing a loss term the deviation of some goal capacity C' that increases during training:
Le [K(((<\))|lp<())
Such loss provides disentanglement prior complementary the minimizing total correlation prior used 3-TCVAE and Factor VAE. Unfortunately, it too depends on specific choice of hyperparam- eters, mainly the max capacity and the speed of capacity increase. We now demonstrate how
7

---

Published as a conference paper at ICLR 2023
incorporate both the adversarial loss and the controlled capacity increase into single loss function. We call this approach disentangling adversarial variational auto-encoder DAVA (also see Figure 3):
DAVA = Cree - -CC PLdis
The main strength of DAVA is its ability to dynamically tune C and μ by using the accuracy of its discriminator. This yields a model that performs well on a diverse range of datasets. We now outline the motivation and the method for tuning C' and y during training.
The goal of the controlled capacity increase in the AnnealedVAE architecture is allow the model to learn individual factors at a time into individual latent dimensions. The order which they are learned corresponds to their respective contribution the reconstruction loss. In DAVA,
Shapes3D
AlstmcciDSriies Mpi3d Toy
Goal KL Divergence (C)
1.5
we want to encourage the model to learn new factors by increasing C as long as it has learned disentangled representation of the currently learned factors. This the case when the dis- criminator cannot distinguish DEP from Dip.
0.5
As soon as the accuracy of the discriminator increases, we want to stop increasing C' and increase the weight of the adversarial loss to ensure that the model does not learn any new factors of variation while helping it to disen- tangle the ones it has picked up into individual dimensions. As can be seen in Figure 6 this
Training Step (-107)
Figure 6: Information bottleneck capacity C dur- ing training of DAVA for different datasets. De- pending on the complexity and structure of the dataset as well as random initialization of the model, different schedule of C is necessary to achieve best performance. The shaded area denotes one standard deviation across 5 random seeds.
results in dataset-specific schedules of C over
the course of the training. Algorithm T in Ap- pendix C.I describes the training of DAVA in more detail.
5.1 EXPERIMENTS
To quantify the performance of DAVA, perform an extensive study including multiple model architectures and datasets. As a baseline, we include the best disentangling models according to (Locatello et al. 2019b), namely the 3-TCVAE (Chen et al.) 2018) and FactorVAE. (Kim & Mnih 2018). We also include AnnealedVAE (Burgess et al. (017) and ControlVAE (Shao et al. 2020). As discussed earlier, the performance of each of these approaches is highly dependent on the regularization strength. For this reason, we perform a grid-search over a range of values curated by roughly following the work Locatelo et al. (2019h) and adjusting where necessary to ensure best possible performance. The exact hyperparameters considered can be found in appendix C.I The training of DAVA does not rely on such dataset- specific regularization, therefore all hyperparameters for the training of DAVA were kept constant across all considered datasets. All other hyperparameters closely follow (Locatello et al.) 2019b) and were kept consistent across all architectures (baseline and DAVA) and are also reported in appendix C.I We run 5 different random seeds for each configuration to determine statistical significance. We provide summarized results for the most informative datasets in Tables
2 and For the baselines we report the performance of the mean hyperparameter setting according to the average score of all considered metrics. The reader is advised to consult Appendix for complete results of all evaluated datasets.
Table I: Results Shapes3D
Architecture
MIG
DCI
FVAE
PIPE
3-TCVAE
0.26#0.06
0,50+0.03
0.7640.03
0.1130.03
VAc
0.20::0.10
0.7840.04
0.2740.04
AnnealedVAF
0.52+0.02
0,70+0.02
0.92+0.01
0.26+0.03
ControlVAE
0.07=0.03
0.17:0.04
0.68+0.07
0.02+0.00
Ours
0.62:0.05
0.7840.03
0.8240.03
0.6110.04
8

---

Published as a conference paper at ICLR 2023
Table Results AbstractDSprites
Architecture
MIG
DCI
FVAE
PIPE
3-TCVAE
0,12s0.01
0,1840.01
0.45a0.02
0.19.0.01
FactorVAE
0.12=0.02
0.19#0.02
0,55+0.03
0.20#0.03
VnncaledVAE
0.1520.01
0.2120.01
0.54x0.02
0.1840.01
ContooVVE
0.0420.02
0.07a0.01
0.42a0.02
0.06a0.01
Ours
0.23:0.04
0.27+0.05
0.67+0.05
0.35#0.03
Table 3: Results Mpi3d Toy
Architecture
MIG
DCI
FVAE
PIPE
B-TCVAE
0.11=0.02
0,23+0.01
0.39+0.02
0.09+0.01
FactorVAB
0.02:0.01
0.13+0.00
0.38+0.00
(0.99:0. 10)
AnnealedVAE
0.07=0.03
0,23+0.02
0.5011.02
0.02+0.01
ControlVAE
0.04:0.01
0.17+0.02
0.43+0.04
0.03g0.01
Ours 0.1240.09
0.30=0.03
0.41
0.21#0.03
Appendix also contains the results of the best performing hyperparameter choice for each baseline model, demonstrating their variance across datasets. Note that the PIPE scores of FactorVAF Table 3 are exploiting design decision discussed in depth in Appendix B.3] The decoders of these models mapped all latent values to single image, therefore we exclude their scores. The results showcase the strong consistent performance of DAVA across different datasets. Compared to the baselines, DAVA is able to deliver improvements of to 50% for the supervised metrics, and over 100% for PIPE. This allows DAVA to be used practice as consistent performance can be expected.
We further validate DAVA on three real-world datasets. CelebA (Liu al., 2015) is dataset commonly used in the disentanglement litera- ture. It compromises of over 200,000 images of faces of celebrities. As shown in Figure 7 DAVA achieves a competitive FID while still having comparatively high PIPE metric score. This means that DAVA accomplishes similar visual fidelity of its generated samples the best baseline methods while better disentangling its latent space. Results for datasets Mpi3d Real (Gondall et al. (2019) and CompCars Yangg et al. 2015) can be found in Appendix D and spectively.
130
S-TCVAE
AnncaledVAB
120
D FactorVAE
ContrelVAE
FD score
110
DAVA
100
90
80
1.5
2
2.5
PIPE metric (-10-?)
6 CONCLUSIONS
Figure 7: The FID score (Heusel et al. 2017) mea- sures the distance between two different distribu- tions of images and is commonly used in GAN literature evaluate the quality of the generator. low FID indicates that model generates visu- ally convincing images. We evaluate the FID of image samples from DPp of the respective models against the original images. We find that DAVA achieves competitive FID with comparatively good PIPE. We have removed small number of poorly performing outliers from the baseline mod- els to improve readability.
In this work, we presented new way to un- derstand and formalize disentangled representa- tions from unsupervised point of view. The main idea is that the distribution of reconstructed samples of disentangled VAE should be sim- ilar to the distribution of generated samples. We quantify this notion in entanglement metric called PIPE metric for eval- uating disentangled representations. Without access to ground truth, PIPE provides results consistent with supervised metrics. Furthermore, we apply the same idea to DAVA, novel VAE training procedure. DAVA can self-tune its regulariza- tion strength and produces results competitive with existing models that require more dataset-specific hyperparameter tuning. Robust unsupervised disentanglement performance is an important require- ment for practical applicability. With DAVA, we provide a method that can be used by practitioners with relatively low computational requirements.
unsupervised dis-

---

Published as a conference paper at ICLR 2023
REFERENCES
Miguel A Arcones and Evarist Gine. On the bootstrap of u and v statistics. The Annals of Statistics, pp. 655-674, 1992.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. https///ghhhbbcooomcccpimiii/sss 2018.
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in f-vae. Workshop Learning Dis- entangled Representations at the 31st Conference on Neural Information Processing Systems, 2017.
Guillaume Carbajal, Julius Richter, and Timo Gerkmann. Disentanglement learning for variational autoencoders applied to audio-visual speech enhancement. In 202/ IEEE Workshop on Applications of Signal Processing to Audio and Acoustics ( (WASPAA), pp. 126-130. IEEE, 2021.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Yixuan Chen, Yubin Shi, Dongsheng Li, Yujiang Wang. Mingzhi Dong, Yingying Zhao, Robert Dick, Qin Lv, Fan Yang, and Li Shang. Recursive disentanglement network. In International Conference on Learning Representations, 2021.
Sunny Duan, Loic Matthey, Andre Saraiva, Nicholas Watters, Christopher P Burgess, Alexander Ler- chner, and Irina Higgins. Unsupervised model selection for variational disentangled representation learning. 2020.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. International Conference on Learning Representations, 2018.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 360063944:1204-1210, 2018.
Benjamin Estermann, Markus Marks, and Mchmet Fatih Yanik. Robust disentanglement of a few factors at a time. Advances in Neural Information Processing Systems, 33, 2020.
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation the real world: new disentanglement dataset. Advances in Neural Information Processing Systems, 32, 2019.
Mo Han, Ozan Özdenizci, Ye Wang, Toshiaki Koike-Akino, and Deniz Erdoğmuy. Disentangled ad- versarial autoencoder for subject-invariant physiological feature extraction. IEFE signal processing letters, 27:1565-1569, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick.
Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2017a. URL BtLhttts: htttpp::///peeeveiie /oopnreviiievve.e net forum?id
Sy2fzU9qI
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference on Machine Learning, pp. 1480-1490. PMLR, 2017b.
10

---

Published as a conference paper at ICLR 2023
Irina Higgins, Nicolas Sonnerat, Loie Matthey, Arka Pal, Christopher P Burgess, Matko Bošn- jak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. In International Conference on Learning Representations, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658 PMLR, 2018.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using a learned similarity metric. In International conference machine learning, pp. 1558-1566 PMLR, 2016.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. II- 104. IEEE, 2004.
Zinan Lin, Kiran Thekumparampil, Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans. In International Conference on Machine Learning, pp. 6127-6139. PMLR, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. Advances Neural Information Processing Systems, 32, 2019a.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In international conference on machine learning, pp. 4114-4124. PMLR, 2019b.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. Workshop track ICLR, 2016.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement 2017.
testing sprites dataset. http:///gttbb..coommemmmmm///sis
Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bow- ers. The role of disentanglement in generalisation. In International Conference Learning Representations, 2020.
Scott E Reed, Yi Zhang. Yuting Zhang, and Honglak Lee. Deep visual mnalogymmaking. Advances in neural information processing systems, 28, 2015.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea directions (by accident). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12406-12415, 2019.
L. Schott, J. von Kügelgen, F. Träuble, P. Gehler, C. Russell, M. Bethge, B. Schölkopf, F. Locatello, and W. Brendel. Visual representation learning does not generalize strongly within the same domain. In 10th International Conference Learning Representations (ICLR), April 2022. URL wooumroriii 9RUHP11aUda
net wtwprr/wwwweeeeeeeoouummmcc..
Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang. and Tarek Abdelzaher. Controlvae: Controllable variational autoencoder. In International Confer- ence on Muchine Learning, pp. 8655-8664. PMLR, 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2818-2826, 2016.

---

Published as a conference paper at ICLR 2023
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder- representation learning. In Third workshop on Bayesian Deep Learning (NeurIPS 2018), 2018.
BtRt://t://wt nari ee wtw..ethh..cci.. http://:w.cccuuoop///aaaeeeeeeee
Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? Advances in Neural Information Processing Systems, 32, 2019.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher Burgess, and Alexander Lerehner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration. arXiv preprini arXiv: 1e0e::905000000 09275, 2019.
Yuxiang Wei, Yupeng Shi, Xiao Liu, Zhilong Ji, Yuan Gao, Zhongqin Wu, and Wangmeng Zuo. Orthogonal jacobian regularization for unsupervised disentanglement in image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6721-6730, 2021.
Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine-grained categorization and verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3973-3988 2015.
Xinqi Zhu, Chang Xu, and Dacheng Tao. Learning disentangled representations with latent variation predictability. In European Conference on Computer Vision, pp. 684-700. Springer, 2020.
12

---

Published as a conference paper at ICLR 2023
A DATASETS
We provide a short summary of each dataset, including samples and the respective data generating ground truth factors.
A.I
SHAPES3D
The 31 dataset (Burgess & Kim 2018) includes the following ground truth factors: floor hue, wall hue, object hue, scale, shape and orientation.
Figure 8: Samples from Shapes3D.
A.2 DSPRITES
The DSprites dataset (Matthey et al. (017) includes the following ground truth factors: shape, scale, orientation, x- x-position and y-position.
Figure 9: Samples from DSprites.
A.3 ABSTRACTDSPRITES
The AbstractDSprites dataset (Van Steenkiste al. 2019) includes the following ground truth factors: background color, object color, shape, scale, x-position and y-position.
Figure 10: Samples from AbstraciDSprites.
13

---

Published as a conference paper at ICLR 2023
A.4 MP13D Toy
The Mpi3d Toy dataset (Gondal et al., 2019) includes the following ground truth factors: object color, object shape, object size, camera height, background color, first degree of freedom, second degree of freedom. It consists of low-quality renders of the experimental setup.
Figure 11: Samples from Mpi3d Toy.
A.5
MP13D REAL
The Mpi3d Real dataset (Gnnal al. 2019) includes the same ground truth factors as Mpi3d Toy. consists of real pictures of an experimental setup.
Figure 12: Samples from Mpi3d Real
A.6 SMALLNORB
The Smallnorb dataset (LeCun et al. 2004) includes the following ground truth factors: object type, elevation, azimuth, lighting condition.
Figure 13: Samples from Smallnorb.
A.7 NOISYDSPRITES
The NoisyDSprites dataset (Locatello et al., 2019b) includes the same ground truth factors as DSprites. Instead of the background being black, it consists of random noise.
14

---

Published as a conference paper at ICLR 2023
Figure 14: Samples from NoisyDSprites
A.8 CARS3D
The Cars3D dataset (Reed et al. 2015) includes the following ground truth factors: elevation, azimuth, object type.
Figure 15: Samples from Cars3D.
A.9 CELEBA
The CelebA dataset (Liu et al. 2015) includes multiple images of over 10.000 identities with a rich set of facial properties, poses and background clutter.
Figure 16: Samples from CelebA.
15

---

Published as a conference paper at ICLR 2023
A.10 COMPCARS
The CompCars dataset (Yang et al., 2015) contains images of 163 car makes with 1.716 car models, totalling to 136.726 images. The images scraped from the web and composed of the car in different poses, eg. front and side view, with diverse backgrounds. It is uncleaned and might contain occlusions and highly correlated ground truth factors.
Figure 17: Samples from CompCars.
B PIPE METRIC DETAILS
B.1 TRAINING OF EVALUATED MODELS
We used di sent wwg.leeet--eem 1ib (occaeelo et al. 2019b), enabling easy reproducibility of our results, to train all models for Shapes3D, AbstraciDSprites and Mpi3d Toy. We focus on FactorVAE and 3-TCVAE
they are the models generally achieving the highest disentanglement performance in (Locatello et al. 2019b). Following (Locatello et al. 2019b) on reduced scale, we trained 5 random seeds each for different hyperparameters. This leads to a total of 60 model instances per dataset, spanning a broad range of disentanglement performance.
We followed similar approach to reproduce the 30 PactorVAE. and 30 3-TCVAE models for each dataset of the abstract reasoning study (Van Steenkiste et al., | (219)). For each VAE model instance, we then trained 5 downstream models on the abstract reasoning task, based on the representation of the VAE, for 6000 steps and evaluated their performance. We used di aisenangglement -iieenttttiit to compute the metrics for all evaluated models.
B.2 SAMPLING STRATEGIES FOR DEP AND DEP
To sample from DEP, one can simply sample random instances from the data distribution and then use the model M to reconstruct these instances. This implicitly computes the EP latent distribution g(2). To sample from SFP, we first need to estimated ((2). Closely following Algorithm 1 in (Kim & achieve this by sampling a batch from q(=) and then randomly permuting across the batch for each latent dimension (see PermuteDims in Algorithm
Mnih 2018),
[]. This is a standard trick used in the independence testing literature (Arcones Gine 1992). As long as the batch is large enough, the distribution of these samples samples will closely approximate ((z). For the evaluation of the PIPE metric, it is therefore not necessary to compute the marginals over the complete data distribution. To amplify the differences between Sgp and SpP, we then slightly adjusted the sampling strategy for the factorial posterior (FP). For each dimension of the latent representation we first estimate its range while encoding a batch of observations. We then sample values uniformly in that range, separately for each dimension, resulting similar factorial distribution as described earlier, but including a dimension-witn uniform distribution prior. This slightly improves performance of the PIPE metric. The architecture the discriminator used in PIPE is similar to the encoder of the VAE displayed in Table
B.3 THE ROLE OF RECONSTRUCTION LOSS
Consider a model which does not store any information in its latent space, where the decoder outputs the same image regardless of the inputs receives, As the PIPE metric does not (and cannot) have
16

---

Published as a conference paper at ICLR 2023
any knowledge of the ground truth factors, it will give such model a perfect score It is possible to use the reconstruction loss as proxy for how good a model manages to capture the ground truth factors. However, the usefulness of the reconstruction loss is limited by two factors. First, it is strongly affected by noise, which requires at least dataset-specific normalization for to be used in a metric. Second, the contribution of different ground truth factors in the reconstruction loss differ a lot. We define
RePE PIPE - a ReCnorm
with Recuorm denoting the per dataset normalized reconstruction loss. We show results for 0.5 and a = 1 in Figure [8] While adding the reconstruction loss brings improved correlations with supervised metrics for Shapes3D and Mpi3d Toy, it decreases correlations for AbstractDSprites, especially for a = 1. For this reason, we decided to propose PIPE without any reconstruction loss component, as practitioner, depending on the task, might have different requirements. In practice, we suggest to use classical (denoising) VAE to get a feeling for the achievable reconstruction loss on dataset and then use PIPE to select the best disentangling models among those that achieve a satisfactory reconstruction loss.
B.4 RELATION TO FID
The Fréchet inception distance (FID) (Heusel et al. 2017) is metric to quantify the similarity between two image distributions. This is commonly achieved by using a pretrained Inception v3 model (Szegedy al., (016)) without its final classification layer encode the image distributions. The FID is then the Fréchet distance of two Gaussian distributions fitted to the respective latent image distributions. is commonly used to evaluate the performance of GANs where one desires the generated images to be as similar as possible to the original dataset. While it appears that there are strong similarities between FID and PIPE, there exist distinct differences. The contribution of PIPE is to show the importance of the concept of similarity of the DEP and Dyp distributions in light of disentanglement. Compared to our proposed PIPE metric, FID would be another way to quantify PIPE. Figure 19 shows that FID and the PIPE metric have some correlation, while also showing substantial differences for some models. A large advantage of the PIPE metric over FID would be that it is not dependent pretrained model. This opens up the possibility to deploy the PIPE metric on unseen data domains without any pretrained models available.
B.5 COMPARISON TO MODELCENTRALIT
Even though MC outperforms UDR, PIPE still considerably outperforms MC. We discuss possible reasons on why this might be the case. To evaluate similarity of two models, MC samples from the prior of a model, but the EP distribution of the model might be far away from the prior. PIPE does not have this problem because it samples from the EP and FP distributions. To evaluate similarity of two models in MC, the second model is asked to encode generated samples of the first model (very similar (D)Er) This distribution might be far away from the initial data distribution the encoder of the second model is used to. Another issue is that if only very small number of models actually finds a disentangled representation, it is hard for MC to identify these models. PIPE can identify even a single disentangled model among many. Further, due to implicit biases in model architecture, it is possible that models learn similar entangled representations, thus breaking the core assumption of MC that only disentangled representations are similar.
C HYPERPARAMETERS AND DAVA TRAINING DETAILS
C.1 HYPERPARAMETERS
DAVA and all considered baseline models use the same VAE architecture as displayed in Table 4 The architecture of the discriminator and its specific hyperparameters used in FactorVAB are shown in Table σ All models, DAVA and baseline, use the same general hyperparameters displayed Table We present the hyperparameter range explored for each baseline model in Table ? We adjusted the latent space size 7_dim for all models for small number of datasets. We did this in order to give the
S
In fact, this happened with few FactorVAR models Mpi3d Toy, see Table
17

---

Published as a conference paper at ICLR 2023
3
AbstractDSprites
PIPE
Rec
PIPER
PIPEL Rte
Rec
UDR
45
44
21
I0D
UDR
35
28
Rec
80
56
5
41
2041
Rec
16
-14
100
IXT
48
62
65
100
DXT
100
MIG
MIG
55
100
PIPE
PIPEReo (mpgla11
UDR
DC1
MN
FWA
PIPE
PIPB 5)
PIPE (ppple=))
Mo:
DC1
MIG
FYAE
PIPB Stc
Mpi3d Toy
PIPE
PrPple 20.5)
Roc
S4
PIPE
10
Rec
98
100
UDR
74
1001
Rec
100
FVAE
-20
PРE
PIPE ReP
UbK
Roc
DX1
M1G
Rec
Figure 18: Spearman rank correlation between various metrics on three different datasets, similar as
Figure We additionally include PIPE Rec for 0.5 and 1.
120
150
FID score
100
FID score
FID score
100
80
100
60
40
0
0.2 0.4
0.1 0.2 0.3
0.5
PIPE metric
PIPE metric
PIPE metric
(a) Shapex3D
(b) AbstractDSprites
(c) Mpi3d Toy
Figure 19: FID vs PIPE metric quantifying the similarity between DgP and DeP on three different datasets over all evaluated models. both metrics would correlate perfectly, one would expect a line from top left to bottom right corner.
18

---

Published as a conference paper at ICLR 2023
VAE enough latent capacity to capture the complexity of the dataset. More specificallyy, set dim to 20 for Mpi3d Toy and Mpi3d Real, and to 50 for CelebA respectively.
DAVA uses the same architecture and general hyperparameters the baseline models. The dis- criminator of DAVA uses the same architecture as the encoder, with the last FC layer having a single output. To stabilize the adversarial training procedure of DAVA, few measures more commonly used in GAN training. We used label smoothing as well as mixed batches for the training of the discriminator. We clipped the gradient norm for both the VAE and the discriminator at 1. As specified in T we had to use lower weight for applying the adversarial gradient of the discriminator to the decoder. A bigger weight negatively affected reconstruction performance of the VAE. We applied instance normalization to the discriminator, which improved the adversarial gradient signal. We experimented with other types normalization (batch, spectral, layer) for encoder, decoder and discriminator, but did not find consistent improvement for any of them. Further, we improved the mechanism control the KL.-Divergence of VAE proposed by (Burgess et al. 2017). Instead of using the absolute difference, we propose to take the difference to the power of 4. This allows the KL-Divergence to fluctuate in between batches, but strongly punishes big deviations.
had make use of
We summarize the training procedure of DAVA in Algorithm|1 I We use dimension wise permutation (PermuteDims) to approximate the factorial posterior FP, similar FactorVAE (Kim & Mnih 2018) and as explained in B.2 The process of adjusting C' based on the accuracy of the discriminator
described in Algorithm 2
Encoder / DAVA Discriminator
Decoder
Input: [64,64.num channels] 4x4 conv, 2 strides, 32 ReLU
FC, 256 ReLU
FC, 4x4x64 ReLU
4x4 conv, 2 strides, 32 ReLU 4x4 conv, 2 strides, 64 ReLU 4x4 conv, 2 strides, 64 ReLU FC 256, FC 2 =_dim
4x4 upconv, strides, 64 ReLU
4x4 upconv, 2 strides, 64 ReLU
4x4 upconv, strides, 64 ReLU
4x4 upconv, 2 strides, num channels
Table VAE architecture used for models in this study.
Parameter
Value
Batch size
128
Optimizer
Adam
Adam: betal
0.9
Adam: beta2
0.999
Adam: epsilon
Ie-8
Learning rate
0.0001
Training steps
150000 batches
Max. gradient norm
1
AnnealedVAB: y
1000
Annealed VAE: iteration threshold
100000 batches
7_dim
10
Table 5: General hyperparameters.
19

---

Published as a conference paper at ICLR 2023
Discriminator
Parameter
Value
FC, 1000 leaky ReLU
Optimizer
Adam
FC, 1000 leaky ReLU
Adam: betal
0.5
FC, 1000 leaky ReLU
Adam: beta2
0.9
FC, 1000 leaky ReLU
Adam: epsilon
1e-8
FC, 1000 leaky ReLU
Learning rate 0.0001
FC, 1000 leaky ReLU
FC, 2
Table 6: Hyperparameters and architecture the discriminator of FAC
Model
Parameter
Value
3-TCVAE B
[, 2, 8, 16, 32]
FactorVAE
5, 10, 20, 50, 100]
AnnealedVAB
2: 2, 25, 50
VAn
C
[16, 18, 25, 35, 180, 200]
Table 7: Hyperparameter range explored for each model
Parameter
Value
Discriminator: Optimizer
Adam
Discriminator: Adam: betal
0.9
Discriminator: Adam: beta2
0.999
Discriminator: Adam: epsilon
1e-8
Discriminator: Learning rate
0.0001
Discriminator: Max. gradient norm
Dis encoder weight Mene
0.3
Dis decoder weight Pdec
0.001
Y
500
AC
4e-5
Table DAVA hyperparameters
C.2 INDIVIDUAL CONTRIBUTION OF DIFFERENT BUILDING BLOCKS OF DAVA
One might wonder which building blocks of DAVA contribute most to the performance. Results during development show that the adversarial loss of DAVA alone delivers comparable performance to 3-TCVAE and FactorVAE with the same issue of regularization strength being dependent on the dataset. Interestingly, for AnnealedVAE, the optimal C for different datasets closely correspond to the values discovered by DAVA displayed in Figure This also reflects the performance one could expect from DAVA without the adversarial loss.
C.3 COMBINATION OF DAVA WITH EXISTING METHODS
The main contribution of DAVA lies its ability to adapt the latent capacity bottleneck with respect to the current state of the model, evaluated by the discriminator. In that regard, it would be possible to use a different discriminator as the one proposed by DAVA. One such option the discriminator used in FactorVAE, which could be used in similar manner the discriminator in DAVA to guide the latent capacity bottleneck. This would still allow for a dataset-specific annealing scheme without the need for dataset-specifie hyperparameters.
20

---

Published as a conference paper at ICLR 2023
Algorithm 1 Training DAVA
Oone; dees Odis + initialize parameters
while Not Converged do
-- sample random mini-batch from D
+ Enc(x)
f +- Dec(2)
- Lae + (KK c
Oenc & -V VVennhurae
0 doc $ -V
Update VAE
4 Enc(x)
4 4- Dec(2)
> Recreate and & after update
4- Ded Dec(Permutee PermuteDims((
accuracy Dis,
Odis $ -Vo. loa.lo((DDs(ii)) + log(1 - Dis(T))
Di Update Discriminator
C updatec(ace))
Mhoae f- maxx((cc 0.5) 100,0)
Weight grows linearly with higher accuracy
denc Denalbasee Mene log (())
Odec # -V- Hbase Pude log(1 (+))
Update VAE with adversarial loss
end while
Algorithm 2 updatec
Input: accuracy acc if acc < 0.5 then U+Ar AC else ace < 0.51 then U+0
> Only increase discriminator is clueless
D Grace period
else
U+
> Decrease C' if discriminator gets better
end
Output: update U
21

---

Published as a conference paper at ICLR 2023
D COMPLETE QUANTITATIVE RESULTS OF ALL EVALUATED DATASETS
We present results for large range of datasets. We also include results for the best regularization strength for each of the baseline approaches after supervised hyperparameter selection in the upper part of each table. This serves as an additional reference on what can be expected from the completely unsupervised approaches in best case scenario. We further include the results of Recursive Disentan- glement Network (RecurD) by Chen et al. (2021). These are to be interpreted with a grain of salt, as we were not able reproduce them ourselves, but had take the values from their paper. This means that first, only limited set of datasets and metrics were evaluated. Second, there will be differences in model architecture, hyperparameter choice, the range of hyperparameter tuning, general training loop and possibly more things. Please note that there are two aspects to take into consideration in regard of the strong performance of AnnealedVAF. First, the selected range of hyperparameters is especially strong, as we include the hyperparameters discovered by DAVA. Second, explained earlier for DAVA, we use (.)* instead of the absolute value of the difference for controlling the KL-divergence. Further note that the exceptionally high PIPE metric score of Factor VAE NoisyDSprites is caused by the decoder of the Factor VAE models always producing the same output regardless of the latent space. This is design decision of PIPE, described in Subsection B.3 Perhaps surprisingly, these models still seem to convey a small amount of information in the latent space as reflected by the supervised metrics. The same occurred with FactorVAE for Mpi3d Toy, interestingly though not on Mpi3d Real.
Table 9: Results Shapex3D
Architecture
MIG
DCI
FVAE
PIPE
Rec
Best 3-TCVAE (B = 32)
0.39+0.09
0.65+0.03
0.7640.06
0.36#0.03
0.0041#0.0008
Best FactorVAE (y 30)
0.3940.16
0.54+0.12
0.79+0.05
0.30#0.08
0.0032+0.0000
Best AnnealedVAF (C = 1)
0.63+0.04
0.78+0.05
0.94+0.04
0.49+0.05
0.0027 000002
Best ControlVAE (C 18)
0.08+0.03
0.20+0.06
0.68+0.12
0.02+0.00
0.0019=0.0000
Best RecurD (Chen et al. 2021)
(0.31)
(0.58)
(0.0083)
Mean 3-TCVAE
0.26+0.06
0.50+0.03
0.76+0.03
0.11#0.03
0.0023+0.000
Mean Factor
0.20+0.10
0.41#0.07
0.78+0.04
0.22+0.04
0.0027=0. 00005
Mean VnnealedVAE
0.52+0.02
0.70+0.02
0.92+0.01
0.26#0.03
0.0020 1000
Mean ControlVAE
0.07+0.03
0.17+0.04
0.68+0.07
0.02+0.00
0.0019
Ours
0.62+0.05
0.78+0.03
0.82+0.03
0.61#0.04
0.0029#0.00%
Table 10: Results AbstractDSprites
Architecture
MIG
DCI
FVAE
PIPE
Rec
Best 3-TCVAE (3 4)
0.17+0.04
0.22+0.03
0.55+0.03
0.11+0.02
0.0017#0.0001
Best FactorVAE (- = 50)
0.18+0.07
0.26+0.04
0.65+0.05
0.23+0.04
0.0019#0.0011
Best AnnealedVAE (C
0.23+0.04
0.26+0.04
0.65+0.05
0.13+0.02
0.0018+0.0001
Best ControlVAE (C' 18)
0.06#0.01
0.1000.03
0.50+0.05
0.07+0.01
0.001620.0000
Mean 3-TCVAE
0.12+0.01
0.1800.01
0.45+0.02
0.19#0.01
0.0027
Mean Factor
0.12+0.02
0.19+0.02
0.55+0.03
0.20+0.03
0.0018 8000000
Mean AnnealedVAE
0.15+0.01
0.21+0.01
0.54+0.02
0.18+0.01
0.0022+0.000 00000
Mean ControlVAB
0.04+0.02
0.07+0.01
0.42+0.02
0.06+0.01
0.001620.0000
Ours
0.23+0.04
0.27+00.5
0.67=0.05
0.35+0.03
0.0020+0.0002
22

---

Published as a conference paper at ICLR 2023
Table 11: Results Mpi3d Toy
Architecture
MIG
DCI
FVAE
PIPE
Rec
Best 3-TCVAB (3 = 2)
0.23+0.04
0.33+0.01
0.42+0.06
0.07+0.01
0.0005#0.0000
Best FactorVAE (y 5)
0.10#0.05
0.20#0.02
0.46+0.03
0.36#0.16
0.0021#0.0005
Best AnnealedVAE (C 2
0.21+0.07
0.31+0.02
0.41
0.12+0.01
0.0005 0000
Best ControlVAE(C 16)
0.04+0.01
0.18+0.02
0.44+0.04
0.02+0.01
0.0003+0.0000
Mean 3-TCVAE
0.11#0.02
0.23#0.01
0.39+0.02
0.09#0.01
0.0006*000000
Mean FactorVAE
0.02+0.01
0.13+0.00
0.38+0.02
0.99+0.10 10
(0041=0.0000
Mean AnnealedVAE
0.07+0.03
0.23+0.02
0.50#0.02
0.02+0.01
0.0003=000000
Mean ControlVAE
0.04+0.01
0.17+0.02
0.43+0.04
0.03+0.01
0.0003#0.0000
Ours
0.12+0.09
0.30+0.03
0.41#0.04
0.21#0.03
0.0006#0,0000
Table 12: Results DSprites
Architecture
MIG
DCI
FVAE
PIPE
Rec
Best 8-TCVAE (3 = 32)
0.34+0.02
0.50+0.03
0.60+0.00
0.76+0.14
0.0081#0.0000
Best FactorVAE = 50)
0.10+0.04
0.22+0.02
0.73+0.04
0.41+0.05
0.0098#0.0000
Best AnnealedVAE (C
0.36+0.01
0.51+0.01
0.76+0.03
0.42+0.03
0.0060=0.0000
Best Con (C' 18)
0.06+0.02
0.13+0.03
0.67#0.11
0.24+0.03
0.0008
Best RecurD (Chen al. 2021
(0.27)
(0.38)
(0.0047)
Mean 3-TCVAE
0.27+0.05
0.45:0.06
0.84+0.03
0.44+0.08
0.0024=0.000
Mean Factor
0.14+0.06
0.22+0.05
0.72+0.07
0.31+0.03
0.0052+0.0000 0002
Mean Annealed VAE
0.18+0.06
0.28+0.05
0.77+0.04
0.32+0.05
0.0013+0.00 3+0.0001
Mean Control
0.05+0.02
0.13+0.01
0.6940.06
0.20+0.08
0.000800.0000
Ours
0.24+0.09
0.42+0.14
0.74+0.11
0.51
0.0025+0.0005
Table 13: Results Mpi3d Real
Architecture
MIG
DCI
FVAE
PIPE
Rec
Best 3-TCVAE
0.16+0.08
0.35+0.06
0.56+0.05
0.03+0.01
0.0004+0.0000
Best FactorVAE (- = 30)
0.02+0.01
0.18+0.04
0.43+0.03
0.42+0.10
0.0031
Best AnnealedVAE (C =
0.12+0.08
0.30+0.07
0.49+0.08
0.08+0.01
0.0004+0.0000
Best ControlVAE (C 25)
0.06+0.02
0.2240.01
0.54+0.05
0.01+0.00
0.0003+0.0000
Mean 3-TCVAE
0.08+0.04
0.22+0.03
0.26+0.03
0.23+0.07
0.0015+0.000F
Mean Factor
0.05+0.01
0.19+0.01
0.42+0.02
0.12+0.07
0.0022+0.000
Mean AnnealedVAE
0.09+0.03
0.29+0.02
0.52+0.05
0.03+0.01
0.0003+0.0000
Mean ControlVAE
0.04+0.02
0.20+0.03
0.52+0.06
0.01+0.00
0.0003+0.0000
Ours
0.11+0.05
0.27+0.03
0.48+0.05
0.20+0.04
0.0006+0.0000
23

---

Published as a conference paper at ICLR 2023
Table 14: Results Smallnorb
Architecture
MIG
DCI
FVAE
PIPE
Rec
Best 3-TCVAE (3 = 4)
0.16+0.03
0.3440.01
0.60+0.02
0.18+0.03
0.0037+0.0000
Best FactorVAE (7 5)
0.1260.06
0.25#0.03
0.64+0.02
0.24#0.03
0.0054#0.0009
Best AnnealedVAE (C 2
0.20#0.01
0.28+0.01
0.58+0.01
0.22#0.02
0.0037=0.0000
Best ControlVAE (C 180)
0.24+0.01
0.28+0.01
0.62+0.04
0.03+0.00
0.001640.0001
Mean 3-TCVAE
0.18+0.02
0.2940.01
0.57+0.02
0.11#0.01
0.0032+0.0001
Mean FactorVAE
0.03+0.03
0.13+0.01
0.53+0.04
0.17
0.0101#0.0039
Mean AnnealedVAE
0.21=0.01
0.28+0.00
0.60+0.01
0.05+0.00
0.0020#0.000
Mean ControlVAE
0.23+0.02
0.28+0.01
0.62+0.03
0.03+0.00
0.0016#0.0001
Ours
0.25+0.01
0.30+0.01
0.62#0.01
0.25+0.02
0.0030#0.0001
Table 15: Results NoisyDSprites
Architecture
MIG
DCI
FVAE
PIPE
Rec
Best [8- 3-TCVAE (3 4)
0.08+0.03
0.21+0.04
0.63#0.09
0.32#0.01
0.0804+0.0000
Best FactorVAE (7 100)
0.01#0.00
0.10=0.05
0.38#0.09
1,00=0.00
0.0892#0.00M0
Best AnnealedVAE (C = 50)
0.06+0.02
0.13+0.03
0.65+0.08
0.48+0.29
0.0802+0.0000
Best ControlVAE (C 16)
0.02+0.01
0.08#0.01
0.48+0.06
0.43=0.29
0.0801#0.0000
Mean 3-TCVAE
0.05+0.03
0.11
0.44+0.08
0.31
0.0810=0.0000
Mean FactorVAE
0.01
0.06#0.02
0.33+0.03
(1.00=0.00)
0.0892#0.0000
Mean Annealed
0.0920.05
0.18+0.06
0.58+0.09
0.27=0.03
0.0805+0.0000
Mean ControlVAB
0.02+0.01
0.08#0.01
0.49#0.01
0.3520.14
0.0801
Ours
0.0940.05
0.22+0.05
0.55+0.07
0.32+0.04
0.0806#0.0001
Table 16: Results Cars3D
Architecture
MIG
DCI
FVAE
PIPE
Rec
Best 8-TCVAE (3 16)
0.14+0.02
0.33+0.08
0.90+0.02
0.07+0.01
0.000000.0000
Best FactorVAE (7 = 20)
0.10+0.01
0.1640.04
0.88+0.03
0.10#0.02
0.0083#0.000 00005
Best AnncaledVAE (C = 1)
0.1240.01
0.2940.05
0.89+0.02
0.15#0.01
0.0059#0.000
Best ControlVAE. (C 16)
0.05+0.02
0.10+0.02
0.84+0.05
0.03+0.00
0.0045
Best RecurD (Chen al. 2021
(0.17)
(0.0132)
Mean 3-TCVAE
0.11+0.03
0.24+0.01
0.91+0.02
0.06#0.01
0.0049#0.00 0001
Mean FactorVAE
0.06+0.02
0.1240.03
0.74
0.28+0.14
0.0170 0.03
Mean Annealed VAE
0.06+0.02
0.25+0.04
0.86+0.01
0.08+0.01
0.0047
Mean Control
0.03+0.01
0.11#0.01
0.76+0.09
0.03+0.00
0.0045+0.0000
Ours
0.15+0.01
0.23+0.04
0.94#0.01
0.28+0.02
0.0074+0.0001
E FURTHER RESULTS ON REAL-LIFE DATASETS
We further report results of DAVA against the full range of baseline models on the CompCars dataset. As described in the dataset description in A.10 CompCars is an extremely challenging dataset. Performance is most likely limited by the model architecture used, as can also be seen by the bad
FID and PIPE metric scores displayed in Figure 20
24

---

Published as a conference paper at ICLR 2023
400
380
FID score
360
D
340
3-TCVAE
320
AnnealedVAE
FactorVAE
300
Control
0
2
6
8
10
PIPE metric (-10-2)
Figure 20: Similar to Figure 7 we display the FID of image samples from DrP of the respective models versus the original images of CompCars. DAVA achieves competitive FID scores even compared to models exploring a broad range of hyperparameters. FactorVAE models that achieve better disentanglement according the PIPE metric suffer from significantly lower FID scores.
25


Reviews:
[{"review_id": "8P-Uaa0dmTX", "reviewer": "Reviewer_GT7q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel way to encourage disentangling of representations for VAE-like models.\n\nThey introduce 2 distributions over samples x, D_EP which is the empirical distribution one would usually use with a VAE (where $q(z) = E[q(z | x)]$), and D_FP, which replaces the distribution over z with its forced factorial version $\\bar{q}(z) = \\prod_i q(z_i)$. One can generate samples and images from either distributions, and how much they differ is a signal for disentanglement. \n\nThis can then be used as a metric (which they call PIPE, Posterior Indifference Projection Equivalence), or as an extra loss, by using a Discriminator to penalize a model where these two distributions differ. They also show how one can use the same metric to control the capacity of a model on the fly, which is a good addendum as well. Finally, they show good results on a good set of well accepted simple benchmark datasets.", "summary_of_the_review": "I found this work to be clear, well executed, it introduces several good ideas which I think the community can easily build upon, and presents good evidence of its capabilities, hence I would lean towards acceptance.", "strength_and_weaknesses": "1. It might be good to mention that this really only allows for unidimensional disentangling? \n   1. You mention this fact with DCI, but it might be the case that more complex datasets require correlated subspaces, which this metric/regularisation would discourage?\n   2. What happened in your case for the DAVA model, say on AbstractDSprites? How was color or angle represented?\n2. You do not comment on how to sample from D_FP vs D_EP, and on the potential difference in computational cost of both in the main text.\n   1. When looking in the Appendix at Algorithm 1, one can see that you obtain \\bar{z} simply by permuting the dimensions of z, but that wasn’t clear from reading the main text. \n   2. It also doesn’t seem to be strictly the same as what the definition of D_FP implies, i.e. I was wondering if/when you would compute the marginal over the whole dataset?\n   3. This should probably be expanded upon, as it is quite critical.\n3. I found Figure 6 interesting, and the overall idea of automatically adapting the capacity of the VAE based on this schedule is a nice addendum.\n   1. It might be worth comparing it to more standard techniques however, like GECO [1] or simpler heuristics like [2]\n4. The abstract and start of the introduction is clear, and I found the related work to be quite thorough as well. However, I found the end of the introduction, which introduces the main idea of the paper, to be slightly hard to follow (despite being quite clear afterwards).\n   1. More precisely, the section from “if a VAE learned a truly disentangled [...]” till “is a necessary condition for disentanglement” is too compressed to be understood on a first read. It now makes sense to me after having read the paper, but I would recommend simplifying/reworking it.\n5. Section 3 is strong and well-derived, I felt like Figure 2 supported the arguments very clearly too. Similarly, Section 4 and 5 are strong and clear and contained all the experiments and clarifications I wanted to see.\n\n\n[1] https://arxiv.org/abs/1810.00597 \n[2] https://arxiv.org/abs/2002.07514 ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4V30YxK39Ln", "reviewer": "Reviewer_pitB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work developed an adversarial variational auto-encoders to learn the disentangled representations of observed data in an unsupervised manner. Specifically, it introduced a new metric that computes the difference of the reconstruction between EP and FP to measure disentanglement. Experimental results on multiple datasets illustrated the effectiveness of the proposed method.", "summary_of_the_review": "The technical contribution is somewhat limited since the proposed adversarial variational auto-encoders has been developed by the prior work. In addition, this work needs to compare to some latest baselines. The experimental results need to present the reconstruction error.", "strength_and_weaknesses": "* Strength\n1. The introduced PIPE metric is interesting and somewhat novel.\n2. Conduct extensive experiments on benchmark datasets to verify the effectiveness of the proposed approach \n\n* Limitations\n1. Technical innovation is somewhat limited. The proposed model is quite similar to the adversarial variational auto-encoders in prior work [Han et al. 2020] and [Carbajal et al. 2021] except for the metric PIPE. The defined EP and FP are not new concepts as they are already used in prior work such as FactorVAE and \\beta-TCVAE.\n2. Compared to more recent baselines. This work only compared the proposed approach with the baseline methods before 2020. It would be better to compare the proposed method with some latent baselines, such as ControlVAE [Shao et al 2020] and Recursive Disentanglement Network [Chen et al. 2021], and Jacobian regularization [Wei et al 2021]. For instance, ControlVAE also dynamically tunes the hyperparameters in the VAE objective function based on the output KL-divergence to improve the disentanglement.\n3. Some latest works are missing. The authors only introduced the related work before 2019 while missing some latest work on disentangled representation learning, such as ControlVAE, Recursive Disentanglement Network, and Jacobian regularization as mentioned above. \n4. There is a trade-off between reconstruction error and disentanglement. It would be better to compare the reconstruction error with that of prior work, such as FactorVAE and ControlVAE.\n\nReferences:\n[Chen et al. 2021] Recursive Disentanglement Network. In International Conference on Learning Representations, 2021.\n[Wei et al 2021] Orthogonal jacobian regularization for unsupervised disentanglement in image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6721-6730).\n[Han et al. 2020] Disentangled adversarial autoencoder for subject-invariant physiological feature extraction. IEEE signal processing letters, 27, 1565-1569.\n[Carbajal et al. 2021] Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement, https://arxiv.org/abs/2105.08970", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1YdpITYoyQT", "reviewer": "Reviewer_SzgX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new unsupervised metric for evaluating the disentanglement of generative models (named PIPE), and a new approach for training disentangled variational autoencoder (named DAVA). PIPE evaluates the distinguishability of the distribution of reconstructed samples and the distribution of generated samples from factored posterior latent distribution. DAVA involves minimizing the distance of the above two distributions. Experiments show that the PIPE correlates well with supervised metrics, and DAVA achieves state-of-the-art disentanglement results on various datasets.", "summary_of_the_review": "\nThe paper is well-written, and the proposed metric and approach could be useful for the community. However, due to the above problems, I cannot give a positive score at this point. I recommend the authors address the questions in the rebuttal, and I will adjust the score accordingly.", "strength_and_weaknesses": "\nStrength:\n\n* The paper is well-written and easy to read.\n\n* The proposed disentanglement metric can be useful for future work.\n\n* The proposed disentanglement approach shows promising results on several datasets.\n\nWeaknesses:\n\n* Important prior work is missing in the discussion and evaluation.\n    - The paper considers UDR as the baseline unsupervised metric for experimental comparison. ModelCentrality (https://arxiv.org/pdf/1906.06034.pdf) is another unsupervised metric and it was shown that ModelCentrality correlates better with the supervised metrics and is better at selecting disentangled generative models (both VAEs and GANs) than UDR. Given that ModelCentrality is proposed more than 2 years ago and performs better than UDR, it is important to discuss and compare with ModelCentrality in the paper.\n\n* Some details are missing in the main text. \n    - For the experiments in Section 4, which generative model is used in the experiment, and how many models are used?\n    - The paper discusses the automatic process for tuning C and mu during training. Does it mean that we still need to tune gamma manually?\n\nBesides the above points, I have the following questions:\n\n* PIPE depends on training a classifier (neural network). Would it be sensitive to the hyper-parameter choices (e.g., architecture, number of training iterations)? It is better to discuss it in the paper.\n\n* The key idea of DAVA is similar to FactorVAE. The difference is that FactorVAE regularizes the distance between the posterior latent distribution and the factored posterior latent distribution in the latent space using a total variation loss, while DAVA regularizes the distance in the image space (after passing the latents through the decoder) using an adversarial loss. Why is the approach in DAVA better than FactorVAE? I understand that in DAVA the hyper-parameters can be automatically tuned based on the accuracy of the introduced discriminator (section 5), but we can also do the same if FactorVAE's total variation loss is used in place of L_{dis} (i.e., tuning C and mu according to the loss value of total variation).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster