Disentangling Learning Representations with Density Estimation
Authors: Eric Yeats
Frank Y Liu
Hai Li
Venue: ICLR 2023 poster (2023)
Keywords: autoencoder
representation learning
disentanglement
density estimation

Abstract: Disentangled learning representations have promising utility in many applications, but they currently suffer from serious reliability issues. We present Gaussian Channel Autoencoder (GCAE), a method which achieves reliable disentanglement via scalable non-parametric density estimation of the latent space. GCAE avoids the curse of dimensionality of density estimation by disentangling subsets of its latent space with the Dual Total Correlation (DTC) metric, thereby representing its high-dimensional latent joint distribution as a collection of many low-dimensional conditional distributions. In our experiments, GCAE achieves highly competitive and reliable disentanglement scores compared with state-of-the-art baselines.

Published as a conference paper at ICLR 2023
DISENTANGLING LEARNING REPRESENTATIONS WITH DENSITY ESTIMATION
Eric Yeats! Frank Liu
Hai Li!
Department of Flectrical and Computer Engineering, Duke University 2Comuter Science and Mathematics Division, Oak Ridge National Laboratory (eric.yeats, hai.lije @dukee. 1iufy@ornl.gr gov
ABSTRACT
Disentangled learning representations have promising utility in many applica- tions, but they currently suffer from serious reliability issues. We present Gaus- sian Channel Autoencoder (GCAE), method which achieves reliable disentan- glement via flexible density estimation of the latent space. GCAE avoids the curse of dimensionality of density estimation by disentangling subsets of its la- tent space with the Dual Total Correlation (DTC) metric, thereby representing its high-dimensional latent joint distribution collection of many oow-dimeniional conditional distributions. In our experiments, GCAE achieves highly competitive and reliable disentanglement scores compared with state-of-the of-the--rtt baselines.
INTRODUCTION
The notion of disentangled learning representations was introduced by [engii et al. (2013) it is meant be a robust approach to feature learning when trying to learn more about a distribution of data X or when downstream tasks for learned features are unknown. Since then, disentangled learning representations have been proven to be extremely useful in the applications of natural lan- guage processing Jain et al. (2018). content and style separation John et al. (2018), drug discovery Polykovskiy et al. (2018) 1):18:100 ():08))))) Du et al. (2020), fairness Sarhan et al. (2020), and more.
Density estimation of learned representations is an important ingredient to competitive disentangle- ment methods. [Bnniio et al. (2013) state that representations z ~ Z which are disentangled should maintain as much information of the input possible while having components which mutu- ally invariant to one another. Mutual invariance motivates secking representations of Z which have independent components extracted from the data, necessitating some notion of pz (z).
Leading unsupervised disentanglement methods, namely 3- B-V Higgins et al. (2016). FactorVAE Kim & Mnih (2018). and 8-TCVAE Chen et al. (018)) all learn Pz (z) via the same variational Bayesian framework Kingma& & Welling (013). 3) but they approach making pz (z) independent with different angles. 3-VAE indirectly promotes independence in pz (z) via enforcing low DKr. between the representation and a factorized Gaussian prior, 8-TCVAE encourages representations to have low Total Correlation (TC) via an ELBO decomposition and importance weighted sampling technique, and Factor VAE reduces TC with help from monolithic neural network estimate. Other well-know unsupervised methods are Annealed 8- V. et al. (2018), which imposes careful relaxation of the information bottleneck through the VAE DKL term during training, and DIP-VAB I & II Kumar al. (2017), which directly regularize the covariance of the learned representation. For a more in-depth description of related work, please see Appendix
While these VAE-based disentanglement methods have been the most successful in the field, Lo- catello et al. (219)) point out serious reliability issues shared by all. In particular, increasing disen- tanglement pressure during training doesn't 't tend to lead to more independent representations, there currently aren't t good unsupervised indicators of disentanglement, and method consistently dom- inates the others across all datasets. Locatello et al. (2019) stress the need to find the right inductive biases in order for unsupervised disentanglement truly deliver.
We seek to make disentanglement more reliable and gigh-performig by incorporating new inductive biases into our proposed method, Gaussian Channel Autoencoder (GCAE). We shall explain them in

---

Published as a conference paper at ICLR 2023
more detail in the following sections, but to summarize: GCAE avoids the challenge of representing high-dimensional Pz(z) (z) via disentanglement with Dual Total Correlation (rather than TC) and the DTC criterion is augmented with scale-dependent latent variable arbitration mechanism. This work makes the following contributions:
Analysis of the TC and DTC metrics with regard to the curse of dimensionality which motivates use of DTC and new feature-stabilizing arbitration mechanism
GCAE, a new form of noisy autoencoder (AE) inspired by the Gaussian Channel problem, which permits application of flexible density estimation methods the latent space Experiments which demonstrate competitive performance of GCAE against leading dis- entanglement baselines on multiple datasets using existing metrics
BACKGROUND AND INITIAL FINDINGS
To estimate Pz (z). we introduce a discriminator based method which applies the density-ratio trick and the Eadon-Nikodyl theorem to estimate density of samples from unknown distribution. We shall demonstrate in this section the curse dimensionality in density estimation and the necessity for representing pz (z) collection of conditional distributions.
The optimal discriminator neural network introduced by Goodfellow et al. (2014a) satisfies:
Preat
arg max Ex, [log Dxx,)] + ix [log ((xxxx)) & (x) = D(-)
Preat (x) + Prak (x)
where (xx) is discriminator network trained to differentiate between "real" samples x, and "fake" samples Xy. Given the optimal discriminator D* Dxxx,, the density-ratio trick can be applied to yield be supplied conditioning variables to disttibbttionsGGGoodfllllw et al. (2014b); Makhzani al. (2015).
(x) (x)x [x] Furthermore, the discriminator
Pxake(x)
represent ratio of conditional distributions Goodfellow
Consider the case where the "real" samples come from an unknown distribution ~ Z and the "fake" samples come from a known distribution finite and Po (u) is nonzero on the sample space of pz (2), the optimal discriminator can be used to retrieve the unknown density Pz(z) = (xD Pc (z). In our case where is a uniformly distributed variable, this "transfer" of density through the optimal discriminator can be seen as an application of the Radon-Nikodym derivative of pz (z) with reference to the Lebesgue measure. Throughout the rest of this work, we employ discriminators with uniform noise and the density- trick in this way to recover unknown distributions.
U. Permitted that both pz (z) and po (u) are
(*)
This technique can be employed to recover the probability density of an m-dimennional isotropic Gaussian distribution. While it works well in low dimensions (m S 8), the method inevitably fails as m increases. Figure depicts several experiments of increasing m which the KL-divergence of the true and estimated distributions are plotted with training iteration. When number of data samples is finite and the dimension exceeds a certain threshold, the probability of there being any uniform samples in the neighborhood of the Gaussian samples swiftly approaches zero, causing the density-rati ratio trick to fail.
This well-known phenomenon called the curse of dimensionality of density estimation. In essence, as the dimensionality of a joint distribution increases, concentrated joint data quickly be- come isolated within an extremely large space. The limit m is consistent with the limits of other methods such as kernel density estimation (Parzen- Rosenblatt window).
Fortunately, the same limitation does not apply to conditional distributions of many jointly dis- tributed variables. Figure lb depicts similar experiment of the first in which m variables are independent Gaussian distributed, but the last variable 2,, follows the distribution zn ~ N(u
(m - 1)-3 Σm 2;, x² = ) (i.e., the last variable is Gaussian distributed with its mean as the sum of observations of the other variables). The marginal distribution of each component is
Cooee available ahttpp: petttpa://giihe htt:://gtttbb/
com/ http://www..cceeeecccaeaa/aaeeee sennagggeeeeeeeeennnnnnnneeeene
2

---

Published as a conference paper at ICLR 2023
2012
25
2.0
0.2
110
Batce
Banch
Joint distributions
(b) Conditional distributions
Figure 1: Empirical KL divergence between the true and estimated distributions as training itera- tion and distribution dimensionality increase. Training parameters are kept the same between both experiments. We employ Monte-Carlo estimators of KL. divergence, leading to transient negative values when KL near zero.
Gaussian, just like the previous example. While it takes more iterations to bring the KL-divergence between the true and estimated conditional distribution to zero. it is limited by the curse of dimensionality. Hence, we assert that conditional distributions can capture complex relationships between subsets many jointly distributed variables while avoiding the curse of dimensionality.
3 METHODOLOGY
ANALYSIS OF DUAL TOTAL CORRELATION
Recent works encourage disentanglement of the latent space by enhancing the Total Correlation (TC) either indirectly Higgins et al. (016); Kumar et al. (2017) or explicitly Kim & Mnih (2018); Chen al. 2018). TC is a metric of multivariate statistical independence that is non-negative and zero if and only if all elements of z are independent.
(z
TC(Z) E2 log
Σ -((Z))
[1.P2(Z))
Locatello et al. (2019) evaluate many TC-based methods and conclude that minimizing their mea- sures of TC during training often does not lead VAE μ (used for representation) with low TC. We note that computing TC(Z) requires knowledge of the joint distribution pz (z), which can be very challenging model in high dimensions. We hypothesize that the need for model of pz (z) is what leads to the observed reliability issues of these TC-based methods.
Consider another metric for multivariate statistical independence, Dual Total Correlation (DTC). Like TC, DTC is non-negative and zero if and only if all elements of are independent.
I1. Pz. (2))z.)) pz (z)
DTC(z) E,, log
h(Z)
We use 24 to denote all elements of z except the i-th element. At first glance, it appears that DTC(z) also requires knowledge of the joint density p(z). However, observe an equivalent form of DTC manipulated for the i-th variable:
DTC(Z) M(Z) MZ(Z)
{((Z)1Z M(Z)) - E (ZZZlZZ))
(1)
3

---

Published as a conference paper at ICLR 2023
Here, the i-th variable only contributes to DTC through each set of conditioning variables z Hence, when computing the derivative @ DTC( /z,, no representation of pz (z) is required only the conditional entropies hz (اا are necessary. Hence, we observe that the curse of dimen- sionality be avoided through gradient descent the DTC metric, making it more attractive for disentanglement than TC. However, while one only needs the conditional entropies to compute gra- dient for DTC, the conditional entropies alone don't measure how close z is to having independent elements. To overcome this, we define the summed information loss Lzς:
C21 (Z) & Σ 1(Z Σ (ZZ MZZIZZ)) + +h(Z) - h(2) TC( (2)
gradients of each [2,; Zx,) are taken only with respect to Zy then the gradients are equal to @DTC(2) 0z avoiding use of any derivatives of estimates of pz (z). Furthermore, minimizing one ric is equivalent to minimizing the other: DTC(2) = 0 < TC( - 0 < (2) = 0. In our pz: [z, z.)). requiring no further hyperpa- rameters. Details on the information functional implementation are available in Appendix A.1
experiments, we estimate h(2,) with batch estimates Ez,
EXCESS ENTROPY POWER LOSS
We found very helpful to "stabilize" disentangled features by attaching a feature-scale depen- dent term to each 7 I(2;; Z.i)) The entropy power of latent variable z; is non-negative and grows analogously with the variance of z, Hence, we define the Excess Entropy Power loss:
CEEF ())
2Te Σ [(Z); 22(Z2))
which weighs each component of the Cxs loss with the marginal entropy power of each i-th latent variable. Partial derivatives taken with respect the z; subset only, the marginal entropy power only weighs each component. While & + VoLzs in most situations (ó is the set of encoder parameters), this inductive bias has been extremely helpful in consistently yielding high dis- entanglement scores. An ablation study with LEEP can be found in Appendix C The name "Excess Entropy Power" is inspired by DTC's alternative name, excess entropy.
GAUSSIAN CHANNEL AUTOENCODER
We propose Gaussian Channel Autoencoder (GCAE), composed of coupled encoder Ø : + Z and decoder v/ 7 + X, which extracts representation of the data E R in the latent space z _T We assume m < TI, as is typical with autoencoder models. The output of the encoder has bounded activation function, restricting Z (-3,3) in experiments. The latent space is subjected to Gaussian noise of the form = Zg + where each V/ ~ (N(0, o²I) and a is a controllable hyperparameter. The Gaussian noise has the effect of "smoothing" the latent space, ensuring that pz (z) is continuous and finite, and it guarantees the existence of the Radon-Nikodyn derivative. Our reference noise for all experiments is u Unif(-4, 4). The loss function for training GCAE is:
1 1/ n2
LOCAE Exx,,
x11
+ (Z),
where À is a hyperparameter to control the strength regularization, and Va is the Gaussian noise injected in the latent space with the scale hyperparameter The two terms have the following intuitions: the mean squared error (MSE) of reconstructions ensures z captures information of the input while LEEP encourages representations to be mutually independent.
4

---

Published as a conference paper at ICLR 2023
Z
Z
x
p
Zi
Z Z
i-th Discriminator
Figure 2: Depiction of the proposed method, GCAE. Gaussian noise with variance o² is added to the latent space, smoothing the representations for gradient-based disentanglement with LEEP. Discriminators use the density-ratic trick to represent the conditional distributions of each latent element given observations of all other elements, capturing complex dependencies between subsets of the variables whilst avoiding the curse of dimensionality.
EXPERIMENTS
We evaluate the performance of GCAE against the leading unsupervised disentanglement baselines 3-VAE Higgins et al. (2016), FactorVAt: Kim & Mnih (2018), 3-TCVAE Chen et al. (2018), and IIP-VAEEII et al. (2017). We measure disentanglement using four popular supervised disen- tanglement metrics: Mutual Information Gap (MIG) Chen et al.) (2018). Factor Score Kim & Mnih 2018), DCI Disentanglement Eastwood & Williams (2018), and Separated Attribute Predictability
(SAP) Kumar et al. (2017). The four metries cover the three major types of disentanglement metrics
identified by Carbonneau et al. (2020) in order to provide complete comparison of the quantitative disentanglement capabilities of the latest methods.
We consider two datasets which cover different data modalities. The Beamsynthesis dataset Yeats al. (2022) collection 360 timeseries data from a linear particle accelerator beamforming simulation. The waveforms are 1000 values long and are made of two independent data generating factors: duty cycle (continuous) and frequency (categorical). The dSprites dataset Matthey et al. (017) is collection of 737280 synthetic images of simple white shapes
black background. Each 64 x 64 pixel image consists of a single shape generated from the following independent factors: shape (categorical), scale (continuous), orientation (continuous), x-position (continuous), and y-position (continuous).
using the PyTorch framework Paszke et al. 2019) using NVIDIA Tesla V GPUs, and all methods are trained with the same number of iterations. Hyperparameters such as network architecture and optimizer are held constant across all models each experiment (with the exception of the dual latent parameters required by VAE models). Latent space dimension is fixed at m for all experiments, unless otherwise noted. More details are in Appendix
All experiments
In general, increasing À and o led to lower Lsy but higher MSE at the end of training. Figure de- picts this relationship for Beamsynthesis and dSprites. Increasing o shifts final loss values towards increased independence (according to Ls.)) but slightly worse reconstruction error. This is consis- tent with the well-known Gaussian channel as the relative noise level increases, the information capacity of nower-constrained channel decreases. The tightly grouped samples in the lower right of the plot correspond with = and incorporating any > leads to decrease in Cyi and increase in MSE. As is increased further the MSE increases slightly the average Lxy decreases.
Figure Bb plots the relationship between final Lsı values with MIG evaluation scores for both Beam- synthesis and dSprites. Our experiments depict moderate negative relationship with correlation efficient -0.823. These results suggest that LYs is promising unsupervised indicator of successful disentanglement, which is helpful if one does not have access the ground truth data factors.
5

---

Published as a conference paper at ICLR 2023
syril 0.
Beaeryyth OCXE 0 022
04-
Foor (M60)
diprites GEAP m00
Informationn the (We))
03-
03-
asprits GC/E 0.3
02
Legilia
Logoi.
(a) Scatter plot of log(Lrı) MSE for GCAE
(b) Scatter plot of log(Cxz) vs. MIG for GCAE Beamsynthesis and dSprites (both marked with dots). There moderate relationship hetween log (Crr) and MIG (r = -08223,, suggesting log(E<s) is promising indicator of (MIG) disentanglement.
on Beamsynthesis and dSprites. Higher lower
log(Lzz) (through increased disentanglement pres-
sure) tend to increase MSE. However, the increase in
MSF subsides as the model hecomes disentangled.
Figure 3: Scatter plots of log(CEr) vs MSE and MIG, respectively, as a is increased.
EFFECT OF À AND ON DISENTANGLEMENT
DCN 0:
GCAE
(a) Beamsythesis
(b) dSprites
Figure 4: Effect of and on different disentanglement metrics. varied x-axis. Starting from the top left of each subfigure and moving clockwise within each subfigure, report MIG, FactorScore, SAP, and DCI Disentanglement. Noise levels o = {0.2, 0.3} are preferable for reliable disentanglement performance. KEY: Dark lines average scores. Shaded areas one standard deviation.
In this experiment, we plot the disentanglement scores (average and standard deviation) of GCAE as the latent space noise level a and disentanglement strength À vary on Beamsynthesis and dSprites. In each figure, each dark line plots the average disentanglement score while the shaded area fills one standard deviation of reported scores around the average.
Figure 4a depicts the disentanglement scores of GCAE on the Beamsynthesis dataset. All a levels exhibit relatively low scores when À is set to zero (with the exception of FactorScore). In this situation, the model is well-fit to the data, but the representation is highly redundant and entangled, causing the "gap" "separatedness" (in SAP) for each factor to be low. However, whenever À the disentanglement performance increases significantly, especially for MIG, DCI Disentanglement, [0.1, 0.2]. There is clear preference for higher noise levels,
and SAP with generally has higher variance and lower disentanglement scores. FactorScore starts out very high on Beamsynthesis because there just factors of variation, making the task easy.
0.1

---

Published as a conference paper at ICLR 2023
Figure 4b depicts the disentanglement scores of GCAE the dSprites dataset. Similar to the previ- ous experiment with Beamsynthesis, no disentanglement pressure leads to relatively low scores on all considered metrics (~ 0.03 MIG. ~ 0.47 FactorScore, 0.03 DCI, 0.08 SAP), but introduc- ing > 0 signficantly boosts performance on a range of scores ~ 0.35 MIG, ~ 0.6 FactorScore, 0.37 SAP, and 0.45 DCI (for {0.2, 0.3)). Here, there is clear preference for larger o; {0.2, 0.3) reliably lead to high scores with little variance.
COMPARISON OF GCAE WITH LEADING DISENTANGLEMENT METHODS
We incorporate experiments with leading VAE-based baselines and compare them with GCAE o = 0.2. Each solid line represents the average disentanglement scores for each method and the shaded areas represent one standard deviation around the mean.
Disentanglenenn for Methods
Denntangteeeet
Methods (91
10
70
50
100
10
20
50
100
0.40
0.9
0.30
FacterScore (FAC)
0.25
0.20
0.35
0.7
GCAE e 0.2
B-VAL:
0.6
(-TCVAE
0.05
DIPARE-11
0.00
0.4
0.6
2.0
34
0.6
0.8
Disentanglement for GCAE (2)
Disentanglement Strength for GCAE
Disentanglement Strength for VAE Methods (B)
D sssettnngammnt Strength for VAE Mathods (81
10
20
50
100
10
50
100
Separoted Attrbun Pedicaliity (5AP)
0.40
4.4
0.20
0.3
0.15
0.30
0.05
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
Disentanglement Strength for GLAE (A)
Disentanjement Strength for UCAE
Figure 5: Disentanglement metric comparison of GCAE with VAE baselines on Beamsynthesis. GCAE X is plotted on the lower axis, and VAE-based method regularization strength & is plotted on the upper axis. KEY: Dark lines average scores. Shaded areas one standard deviation.
Figure 5 depicts the distributional performance of all considered methods and metrics on Beam- synthesis. When no disentanglement pressure is applied, disentanglement scores for all methods are relatively low. When disentanglement pressure is applied (A, B > 0), the scores of all methods increase. GCAE scores highest or second-highest on each metric, with low relative variance over large range of X. 3-TCVAE consistently scores second-highest FactorVAE and 3- B-V tend to perform relatively similarly, but the performance of 3-VAE appears highly sensitive to hyperparameter selection. DIP-VAE-II performs the worst on average.
average, with moderate variance.
Figure shows a similar experiment for dSprites. Applying disentanglement pressure significantly increases disentanglement scores, and GCAE performs very well with relatively little variance when E [0.1, 0.5). B- 3-VAE achieves high top scores with extremely little variance but only for a very narrow range of B. 3-TCVAE scores very high average for wide range of 9 but with large variance scores. Vac consistently scores highest on FactorScore and it is competitive on SAP. DIP-VAE-II tends to underperform compared to the other methods.
7

---

Published as a conference paper at ICLR 2023
Diventanglement Strenggt
Donntangeemen
10
100
10
50
100
0.10
0.35
0.30
Radtrrooe (9C)
Informatonn
0.23
0.5
0.20
0.15
GCAE dim 0.2
0.3
B-VAE
0.05
DIP VAE
0.00
0.6
Detentanglement Strength
Disentangiement Strength
Oisentanglemenn Strength for VAE Methods
Sennanggemment
Methods (9)
10
20
50
100
10
20
50
100
0.30
Segaraba A1t/R4tt Dodicabilty (5422)
0.40
0.35
0.20
0.25
0.20
0.10
0.30
0.08
0.00
0.4
0.6
Disentangloment ength for (2)
Diser Dinentanglmmnntt GCAE
Figure 6: Disentanglement metric comparison of GCAE with VAE baselines on dSprites. GCAE plotted on the lower axis, and VAE-based method regularization strength 3 is plotted on the upper axis. KEY: Dark lines mean scores. Shaded areas one standard deviation.
DISENTANGLEMENT PERFORMANCE DIMENSIONALITY INCREASES
We report the disentanglement performance of GCAE and FactorVAE on the dSprites dataset m is increased. FactorVAL Kim & Mnih (2018) is the closest TC-based method: it uses a single mono-
lithic discriminator and the density- ratio trick to explicitly approximate TC(Z). Computing TC(Z) requires knowledge of the joint density pz (z), which is challenging to compute as m increases.
Figure depicts an experiment comparing GCAE and FactorVAE
when m 20. The results for
m = 10 are included for comparison. The average disentanglement scores for GCAE m 10 and m 20 are very close, indicating that performance is robust in m. This is not the case for FactorVAE- performs worse on all metrics when increases. Interestingly, FactorVAE seems to recover its performance on most metries with higher / than beneficial for FactorVAE 10. Despite this, the difference suggests that FactorVAE is not robust changes in
20
5 DISCUSSION
Overall, the results indicate that GCAE a highly competitive disentanglement method. It achieves the highest average disentanglement scores on the Beamsynthesis and dSprites datasets, and it has relatively low variance rameters are highly transferable, as À E 0.5) works well on multiple datasets and metrics, and the performance does not change with m, contrary to the TC-based method FactorVAF. GCAE also used the same data preprocessing (mean and standard deviation normalization) across the two datasets. We also find that Cx, is a promising indicator of disentanglement performance.
scores when o {0.2, 0.3). indicating is reliable. The hyperpa-
While GCAE performs well, it has several limitations. In contrast to the VAE optimization process which is very robust Kingma & Welling (2013), the optimization m discriminators is sensitive to choices of learning rate and optimizer. Training discriminators requires a lot of computation, and the quality of the learned representation depends heavily on the quality of the conditional den- sities stored in the discriminators. Increasing the latent space noise seems make learning more
8

---

Published as a conference paper at ICLR 2023
Deeetanglement Strength
(P)
Oeentanglemen strength
20
20
50
0.35
0.70
0.30
Auttrrrore (RAC)
Informatonn
0.50
0.20
0.15
0.10
0.50
0.2
0.4
Desettnnlemeet Strengeh
Dienttanlement Strength
Disertangleeeng for VAE Methods
Disentanglemee Methods
10
20
20
50
Separated AtD Prodit babitabitty (542))
0.40
0.35
0.20
0.30
0.25
0.15
0.20
0.30
0.15
GCAE 0.2 10
0.05
GDAE
0.08
m=
0.3
0.4
for
Disenianglament Strongth GCAE
Figure 7: Comparison of GCAE with FactorVAE on dSprites as m increases. À is plotted below, and B is plotted above. KEY: Dark lines mean scores. Shaded areas one standard deviation.
robust and generally leads to improved disentanglement outcomes, but it limits the corresponding information capacity of the latent space.
6 CONCLUSION
We have presented Gaussian Channel Autoencoder (GCAE), new disentanglement method which employs Gaussian noise and flexible density estimation in the latent space to achieve reliable, high- performing disentanglement scores. GCAE avoids the curse of dimensionality of density estimation by minimizing the Dual Total Correlation (DTC) metric with a weighted information functional to capture disentangled data generating factors. The method is shown to consistently outcompete existing SOTA basclines on many popular disentanglement metrics on Beamsynthesis and dSprites.
ACKNOWLEDGEMENTS
This research supported by grants from U.S. Army Research 191INF2220025 and U.S. Air Force Research Lab FA8750-21-1-1015. We would like to thank Cameron Darwin for our helpful conversations regarding this work.
This research is supported, in part, by the U.S. Department of Energy. through the Office of Advanced Scientific Computing Research's "Data-Driven Decision Control for Complex Systems (DnC2S)" project.
This research used resources of the Experimental Computing Laboratory (ExCL) at ORNL. This manuscript has been authored by UT-Battelle, LLC, under contract DE-ACO5-00OR220 the US Department Energy (DOE). The US government retains and the publisher, by accept- ing the article for publication, acknowledges that the US government retains nonexclusive, paid- up. irrevocable, worldwide license to publish or reproduce the published form of this manuscript, allow others to do so, for US govermment purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan Intt:://enereee www.rrgg.....oooooooorro cads / dooe publ www.ppulcccc---eepep----- access-plan),
with
/

---

Published as a conference paper at ICLR 2023
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3588)11788-1828, 2013.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerehner. Understanding disentangling in f-vae. arXiv preprint MrXiv:1804003599 2018.
Marc-André Carbonneau, Julian Zaidi, Jonathan Boilard, and Ghyslain Gagnon. Measuring disen- tanglement: A review of metrics. arXiv preprint arXiv:2012.09276, 2020.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. Advances neural information processing systems, 31, 2018.
Yuanqi Du, Xiaojie Guo, Amarda Shehu, and Liang Zhao. Interpretable molecule generation via disentanglement learning. In Proceedings of the 11th ACM International Conference on Bioin- formatics, Computational Biology and Health Informaties, pp. 1-8, 2020.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. In International Conference on Learning Representations, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farke Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014a.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577 2014b.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botviniek, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Sarthak Jain, Edward Banner, Jan- Willem van de Meent, lain J Marshall, and Byron C Wallace. Learning disentangled representations of texts with application to biomedical abstracts. In Pro- ceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods Natural Language Processing, volume 2018, pp. 4683. NIH Public Ac- cess, 2018.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. Disentangled representation learning for non-parallel text style transfer. arXiv preprint / 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning. pp. 2649-2651 PMLR, 2018.
Diederik Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
arXiv: 14, 2013.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Proceedings of the 31st international conference on newral information pro- cessing systems, pp. 972-9811 2017.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. arXiv preprint arXiv: 2017.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 41 4114-4124, PMLR, 2019.
Alireza Makhzani. Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint 15/ 2015.
10

---

Published as a conference paper at ICLR 2023
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https://github.com. wttpe///eeemeinddpppsssssaaaeeee datateet 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
Daniil Polykovskiy, Alexander Zhebrak, Dmitry Vetrov, Yan Ivanenkov, Vladimir Aladinskiy, Polina Mamoshina, Marine Bozdaganyan, Alexander Aliper, Alex Zhavoronkov, and Artur Kadurin. En- tangled conditional adversarial autoencoder for de novo drug discovery. Molecular pharmaceu- tics, (5(10):4398 0408,,44055000 2018.
Mhd Hasan Sarhan, Nassir Navab, Abouzar Eslami, and Shadi Albarqouni. Fairness by learning orthogonal disentangled representations. European Conference on Computer Vision, pp. 746- 761. Springer, 2020.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Eric Yeats, Frank Liu, David Womble, and Hai Li. Nashae: Disentangling representations through adversarial covariance minimization. In Computer Vision-ECCV 2022: 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVII, pp. 36-51. Springer, 2022.
A IMPLEMENTATION
A.1 INFORMATION FUNCTIONAL
We estimate the information between each subset of variables T(2;; 23) used in Lys with uniform estimate of the information functional:
I(Z;; Zvi) ~ (b- -.) (x [Eu,-unitto,,, Pz, (u, zz;:) [log Pz, (u:/zya) - log Pz, (u:)]]
where (a, are the bounds of the Uniform distribution (-4 and 4 in our experiments), and Pz, (u,/z,,) (2a:)) the conditional density of the i-th discriminator evaluated with noise from the Uni- form distribution. 50 uniform samples are taken per batch to estimate the functional in all experi- ments. Furthermore, we found beneficial (in terms of disentanglement performance) to estimate the functional using ne (i.e., the noiseless form of 2 Gradient only taken through the pz, (u, (2,))) term with respect to the i variables. The marginal entropy h (())) upper bounds the conditional en- tropy h( / Z with respect to the conditioning variables, so the information functional is a natural path to maximizing 4a Z Zia)) and thereby minimizing DTC.
B MAIN EXPERIMENT DETAILS
Each method uses the same architecture (besides the μ. log heads for the VAE) and receies the same amount of data during training. In all experiments, the GCAE AE and discriminator learning rates are 5e-5 5 and 2e 4, respectively. The VAE learning rate is le-4 and the Factor VAE discrim- inator learning rate is 2e 4. All methods use the Adam optimizer with (81 (2) (0.9, 0.999) for the AE subset of parameters and (8i, 82) = (0.5, 0.9) for the discriminator(s) subset of parameters (if applicable). The number of discriminator updates per AE update & set to 5 when m 10 and 10 when m 20. All discriminators are warmed up with 500 batches before training begins to en- sure they approximate valid density. VAE architectures are equipped with Gaussian decoder for Beamsynthesis and a Bernoulli decoder for dSprites. SELU refers to the SeLU activation function Klambauer et al. (0177).
*Our intuition is that each Bi; comes from one of the "modes" of the corresponding blussian-blurre distribution, ensuring that the loss defined. This avoids the case where the learned conditional distribution is defined when given novel z;c-
11

---

Published as a conference paper at ICLR 2023
Table 1: MLP Architecture
Dataset
GCAE Architecture
VAE Architecture
Beamsynthsis
Linear(n, 1024), SELU Linear( (024, 1024), SELU Linear( I024, 512), SELU Linear(512,, m), SoftSign Linear(m, 512), SELU Linear(512. 1024), SELU Linear( 1024, 1024), SELU Linear(1024, n)
Linear(n, 1024), SELU
BatchSize=64
Linear( (0024, 1024), SELU
Mean/STD Norm 2000 Iterations
Linear(1024 512), SELU
x Linear(512, m) Linear(m, 512), SELU Lincar(512. 1024), SELU Linear(1024, 1024), SELU Linear(1024) n)
dSprites
BatchSize=256
Mean/STD Norm (GCAE) 20000 Iterations
Table Diseriminator Architectures. The FactorVAE architecture follows the suggestion of Kim & Mnih 2018). The GCAE discriminator is much smaller, but there of them compared to just Factor VAE discriminator.
GCAE Discriminator Architecture
FactorVAL Discriminator Architecture
Linear(m, 256), SELU Linear(256, 256), SELU Linear(256, 1), Sigmoid
Linear(m, 1024), SELU
Linear(10244 1024), SELU
Linear(1024, 1024), SELU
Lineer(1024, 1024), SELU
Linear(1024, 1024), SELU
Linear(1024, 1), Sigmoid
C ABLATION STUDY
06
Figure 8: Ablation study: Comparison of MIG scores with and without LEEP- bxs corresponds to direct gradient descent on Lxr-
Figure g depicts an ablation study for training with LEEP vs. directly with Czr- We found that training directly with Lys promotes independence between the latent variables, but the learned variables were not stable (i.c., their variance fluctuated significantly training). The results indicate that LEEP is helpful inductive bias for aligning representations with interpretable data generating factors in a way that is stable throughout training.
12

---

Published as a conference paper at ICLR 2023
D RELATED WORK
D.I DISENTANGLEMENT METHODS
GCAE is unsupervised method for disentangling learning representations hence, the most closely related works are the state-of-the-art unsupervised VAE baselines: f-VAE Higgins et al. (2016). FactorVAE Kim & Mnih (2018), B-TCVAE Chen al. (2018), and DIP-VAE Kumar et al. (2017). All methods rely on promoting some form of independence in pz (z), and we shall cover them more detail in the following sections.
3- B-V
The disentanglement approach of 3- 3-VAE Higgins et al. (2016) is to promote independent codes in by constraining the information capacity of Z. This is done with a VAE model by maximizing the expectation (on x) of the following loss:
C..V = (x(xx) [po(x|z)] 3DRL (2o(z\x)\\po(2)..
where qs (z x) is the approximate posterior (inferential distribution of the encoder), P5 (x (z) is the decoder distribution, (z) is the prior distribution (typically spherical Gaussian), and 3 is hyper- parameter controlling the strength of the "Information Bottleneck" Tishby et al. (2000) induced on
Z. Higher are associated with improved disentanglement performance.
FACTORVAE
The authors of FactorVAE Kim & Mnih (0188) assert that the information bottleneck of 3-VAE too restrictive, and seek improve the reconstruction error vs. disentanglement performance tradeoff by isolating the Total Correlation (TC) component of the DKL [[)zz))]|PP(( term. They employ large discriminator neural network, the density-ratio trick, and a data shuffling strategy to estimate the TC. Factor maximizes the following loss:
FactorVA
(x)()) [x]()] - DKr. (go(z)x)[[pozz) - (2,,
where (Z) the discriminator's estimate of TC(Z). The discriminator is trained to differentiate between "real" jointly distributed and "fake" in which all the elements have been shuffled across a batch.
3-TCVAE
3-TCVAE Chen al. 2018) seeks to isolate TC(Z) via a batch estimate. They avoid significantly underestimating pz [z), by constructing an importance-weighted estimate of h(
E By(4) log qZ)]
log
Σ g(o(Zx)azz)
B
BC
where g(z) is estimate of pz (z), B is the minibatch size, C is the size of the dataset, ((x;) is a stochastic sample from the i-th x, and g(o(z+)12;) is the density of the posterior at o(x;) when
x Zy--
This estimate is used to compute an estimate of TC(Z), and the following loss is maximized:
LY-TvVAE = (2)() [xa(x|z)]
- - Σ DRL. (l(z)))l((z))
where (2; is the "index-code" mutual information, TC, is an estimate of TC(2) com- puted with their estimate of q(z), 3 hyperparameter controlling TC(Z) regularization, and En DKL. (1(2,) p(z,)) is dimension-wise Kullback-Leebllr divergence.
13

---

Published as a conference paper at ICLR 2023
HIP-VAE-II
The approach of DIP-VA is that the aggregate posterior of VAE model should be factorized in order to promole disentanglement Kumar et al. (2017). This is done efficiently using batch estimates of the covariance matrix. The loss be maximized for DIP-VAE-II is:
CDIP-VAE-I =
m
m Σ 1=1 j3t
(x]()) [po(x[z)] DRL (10 (1<(2)))|(pp(z)) - [Cov(za) -1}²
[Cov(2;))]
Hence, the covariance matrix of the sampled representation z should be equal to the identity matrix. is a hyperparameter controlling regularization strength. We did not consider DIP-VAE-I since implicitly assumes knowledge how many data generating factors there are.
D.2 DISENTANGLEMENT METRICS
We evaluate GCAE and the leading VAE baselines with four metrics: Mutual Information Gap (MIG), FactorScore, Separated Attribute Predictability (SAP), and DCI Disentanglement.
MUTUAL INFORMATION GAP
MIG is introduced by Chen et al. (2018) as an axis-aligned, unbiased, and general detector for disentanglemenn. In essence, MIG measures the average gap in information between the latent feature which is most selective for unique data generating factor and the latent feature which is second runner up. MIG is normalized metric on [0, 1], and higher scores indicate better capturing and disentanglement of the data generating factors. MIG is defined as follows:
1 (1(Za: Vi) 1(Zo; V&)), H(V))
MIG V) &
where K the number of data generating factors, ((v)) is the discrete entropy of the k-th data generating factor, and 2, ~ Za and Zo (where b) are the latent elements which share the most and next-most information with Vk ~ Vk, respectively.
For Beamsynthesis, calculate MIG the full dataset using histogram estimate of the latent space with 50 bins (evenly spaced maximum to minimum). For dSprites, we calculate MIG using 10000 samples, and 20 histogram bins following Locatello al. (2019).
FACTOR SCORE
FactorScore is introduced by Kim & Mnih (2018). The intuition is that change in dimension of should result in change of at most one factor of variation. starts off by generating many batches of data in which one factor of variation is fixed for all samples in batch. Then the variance of each dimension on each batch is calculated and normalized by its standard deviation (without interventions). The index of the latent dimension with smallest variance and the index of the fixed factor of variation for the given batch is used as training point for a majority-vote classifier. The test set of data.
score is the accuracy of the classifier
For Beamsynthesis, we train the majority-vote classifier on 1000 training points and evaluate on 200 separate points. For dSprites, we train the majority-vote classifier on 5000 training points and evaluate on 1000 separate points.
SEPARATED ATTRIBUTE PREDICTABILITY
Separated Attribute Predictability (SAP) is introduced by Kumar et al. (2017). SAP involves creating x l score matrix, where ij-th entry is the "predictability" of factor from latent element
14

---

Published as a conference paper at ICLR 2023
For discrete factors, the score is the balanced classification accuracy of predicting the factor given knowledge of the i-th latent, and for continuous factors, the score is the R-squared value of the i-th latent in (linearly) predicting the factor. The resulting score is the difference in predictability of the most-predictive and second-most predictive latents for a given factor, averaged over all factors.
For Beamsynthesis, training size 5000 and a test size of 1000.
training size of 240 and test size of 120. For dSprites,
DCI DISENTANGLEMENT
DCI Disentanglement introduced by Eastwood & Williams (2018). complements other metrics introduced by the paper: completeness and informativeness. The intuition is that each latent variable should capture at most one factor. & decision tree regressors are trained to predict each factor given The absolute importance weights of each decision
the latent codes and inserted as columns in a m x & importance matrix. The rows of the importance matrix are normalized, and the (discrete) k-entropy of each row is computed. The difference of one and each row k-entropy is weighted by the relative importance of each row to compute the final score.
regressor
extracted
For Beamsynthesis, we use 240 training points and 120 testing points. For dSprites, we use 5000 training points and 1000 testing points.
E TRAINING TIME COMPARISON
Table 3: Comparison of training times of the discriminator based disentanglement algorithms on Beamsynthesis, Latent space size is fixed m 10 and discriminator training iterations is fixed to 5.
Method
Average (s)
Standard Deviation (s)
GCAE
955.0
13.6
FactorVAR
1024.4
5.8
15


Reviews:
[{"review_id": "xFr_uO9ej35", "reviewer": "Reviewer_qWQn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the problem of disentangling representations, where we aim to learn representations that have independent features. Authors start with the common learning objective that encourages disentanglement, the Total Correlation (TC), and hypothesize that an important disadvantage of this objective is that it requires an estimate of the latent density. This density is difficult to compute for high-dimensional representations due to the curse of dimensionality. Authors propose an alternative objective, Excess Entropy Power (EEP), that combines TC with Dual Total Correlation (DTC), and can be computed using only the estimates of _conditional_ densities in the latent space. Such conditional densities are often easier to estimate, as also shown by authors in a toy experiment. Authors run experiments on two datasets with known latent factors, observing that EEP improves upon strong VAE-based baselines.", "summary_of_the_review": "The proposed method solves an important problem, is novel, and is evaluated well. However, several choices in the paper are not very well motivated, and the clarity and quality of the manuscript could be improved. As it stands, I consider the paper to be below the bar for acceptance, but would be willing to reconsider if the authors addressed the points raised above.\n\n---\n\nAuthors have responded to some of my points, especially in regards to motivating the choices in the paper, and I have increased my score as a result. I believe the paper can be made stronger still by better connecting to methods in density estimation and studying the pros and cons of autoregressive density estimation in this context, but I'd now support it getting accepted.", "strength_and_weaknesses": "Strengths:\n- The observation that TC depends on the quality of the full density estimate is a keen one, and the solution based on combining TC with DTC is non-trivial and novel.\n- The proposed method is introduced in great detail. The mathematical notation is clear and consistent.\n- Experiments use strong baselines: $\\beta$-VAE and its variants. The proposed method consistently outperforms these baselines.\n- Empirical studies of the role of hyperparameters $\\sigma$ and $\\lambda$.\n- A good critical discussion of the results, as well as the limitations of the method.\n\nWeaknesses:\n- Related work limited to a single paragraph in the introduction. I would've liked to see a broader discussion of disentanglement methods, representation learning methods, and density estimation methods. For density estimation, especially methods that model the density as a sequence of conditional distributions (autoregressive density estimators).\n- The KL divergence values are sometimes _negative_ in Figure 1(b): how is this possible?\n- Several choices in the paper are not clearly motivated, including the density ratio estimation as the density estimation method, the loss weighting scheme in equation 3, and the autoencoder with Gaussian noise as the representation learning method.\n- Density ratio estimation is characterized several times as \"non-parametric\", even though we're parametrizing the density (albeit implicitly) via the discriminator neural net.\n- I am not convinced we're comparing apples to apples in Figure 1: in (a) we're estimating an $m$-dimensional density, while in (b) we're estimating a 1-dimensional density. Estimating the $m$-dimensional density as a series of conditional densities in (b) would be more convincing.\n- Authors do discuss that the method is computationally expensive (need to train $m$ discriminators), but there are no runtime metrics reported in the experiments section.\n\nQuestions:\n- Have you tried using GCAE as a generative model (as you suggest in the discussion)?\n- Do you have intuition for why in Figure 4, (b) part especially, the MIG is going down for $\\alpha > 0.2$? Is it because we're \"over-disentangling\", and the model struggles to capture any info about the true factors?\n\nMinor:\n- Citation style: consider making use of `\\citep` and `\\citet`.\n- Define $m$ in the caption of Figure 1.\n- A figure on page 6 has no caption. It is also not clear what the different dots that have the same color represent.\n- Define $h$ in the first equation of section 3.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "sc5I-R0PYzN", "reviewer": "Reviewer_cTbH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose learning disentangled representation via regularizing Gaussian Channel Autoencoder (GCAE) with Excess Entropy Power (EEP) loss. The authors claim their proposed method can achieve more reliable and high-performing disentanglement.\nTheir methods are largely motivated by the curse of dimensionality difficulty issue of the FactorVAE, one of the leading VAE-based disentangling approaches.  FactorVAE modifies beta-VAE objective by specifically penalizing the dependencies between the latent dimensions towards benefiting the Total Correlation (TC). The authors (of this paper) propose to use Dual Total Correlation (DTC) towards avoiding the curse of dimensionality.\n\nThe authors examined their proposed approaches on two widely used datasets using one metric – Mutual Information Gap (MIG). Experiments show that their approaches achieve (in average) higher MIG compared with other three leading VAE-based disentanglement approaches. \n", "summary_of_the_review": "Overall, the method is novel and demonstrates its effectiveness on learning disentangled representations. However, I do have some concerns/comments as mentioned in my review. I’m slightly not inclined at this moment, but am willing to discuss with authors and change my views accordingly.", "strength_and_weaknesses": "The strengths include but not limited to:\n\n-- The proposed methods are well-motivated. The authors performed preliminary experiments to show the effect of curse of dimensionality for non-parametric density estimation, which is illustrated in Figure one. Based on this fact, the authors analyze the TC and DTC, and show the merits of DTC.\n\n-- The proposed method does achieve higher MIG compared with a few leading VAE-based approaches.\n\n\nThe weaknesses include:\n\n-- There are a lot of quantitative and qualitative ways for evaluating disentangled representations. For quantitative evaluation, there are a few widely used metrics besides MIG, including explicitness (Ridgeway & Mozer, 2018), beta score (Higgins et al., 2017), SAP (Kumar et al., 2018) and DCI disentanglement (Eastwood & Williams 2018).  Typically, previous works (on VAE-based disentanglement) would try to show their methods are consistently better using multiple metrics.\n\n-- Besides the three leading approaches (by 2018) mentioned by the authors, there are also some important works missed by the authors. For example, DIP-VAE (Kumar et al., 2017) and CCI-VAE (Burgress et al., 2017). More importantly, latest works on VAE-based disentangled representation learning has already realized the importance of using inductive biases. A recent work published in ICML (Mita et al., 2021) employs a particular form of factorized prior (also conditionally depends on auxiliary variables) towards learning an identifiable model with theoretical guarantees on disentanglement. Te authors could try to compare with some more (especially recent ones) important works using more datasets.\n\n-- The preliminary experiments do show that higher m may cause difficulty in training/optimization for other works (especially Factor VAE), but the whole set of experiments in the paper are done under a single dimension m=10. It would be interesting to conduct more ablation studies to show the benefits of the proposed methods in terms of curse of dimensionality compared with FactorVAE and Beta VAE.\n\n-- Some comments on ablation studies: In Figure 3, Figure 4 and Figure 5, the best performance is achieved using sigma=0.3 (at least for Beamsythesis), did you further try sigma larger than 0.3? Also, in page 5, you mentioned the importance of adding Gaussian noise – it ensures that p(z) is continuous and finite with respect to uniform density. Did you try the scenario where sigma=0?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qw962vU7yl", "reviewer": "Reviewer_atH7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To address the reliability issue of learning independent representations, the paper proposes to regularize a noisy autoencoder with a new regularization term based on dual total correlation. This results in a new Gaussian Channel Autoencoder, which does not need to estimate the joint distribution of the latent representation and achieves higher disentanglement scores.", "summary_of_the_review": "The paper contributes some new ideas to disentangled representation learning. However, some claims are not well supported and the experiments are weak. The writing could also be improved by clearly defining the notations and clarifying some confusing terms.", "strength_and_weaknesses": "Strength:\n\n+ It is interesting to use the dual total correlation instead of total correlation as the metric for disentangle representation learning. The metric circumvents the curse of dimensionality of estimating the joint distribution of latent representations.\n\n+ The paper is well organized and easy to follow. The motivation is clear.\n\nWeakness\n\n- The paper claims the method can achieve reliable and high-performing disentanglement outcomes. It seems that only the \"reliable\" part is supported by higher disentanglement scores from the experiment. What is the definition of \"high-performing\" and how it is supported by the paper?\n\n- The experimental results are not convincing. As stated in the paper that \"there currently aren’t good unsupervised indicators of disentanglement\", why only the Mutual Information Gap (MIG) is used as the disentanglement score? How about Z-Diff score [1], SAP score [2], and Factor score [3]?\n\n[1] beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017\n\n[2]  Variational inference of disentangled latent concepts from unlabeled observations. In ICLR, 2018\n\n[3]  Disentangling by factorising. In ICML, 2018.\n\n- Some notations are confusing. For example, $DTC(z) = DTC_i(z)$ in Eq. (1). What do $z_{\\forall j\\neq i}$ and $z_{k\\neq j}$ mean and what are their difference? I don't follow  the discussion from $I(z_i; z_{\\forall j\\neq I})$ in Eq.(2) to $I(z_i;z_j)$.\n\n- The motivation for the major loss term in Eq. (3) is not clear. What is the definition of feature-scale dependent term and why it is helpful?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6rU6Mh5D209", "reviewer": "Reviewer_d23T", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is about learning disentanglement using scalable non-parametric density estimation. The author proposes a dual total correlation (DTC) metric for disentangling that divides the joint distribution into many low-dimensional conditional distributions. The authors conduct extensive experiments to show the superiority of their approach against state-of-the-art baselines.\n", "summary_of_the_review": "\nOverall, the paper seems to be a work in progress, and there are many areas where it needs further improvement. Currently, the work has limited novelty and needs more experiments with results to justify the made claims.\n", "strength_and_weaknesses": "Strength\n\n1) The analysis is shown in section 3, which derives the DTC and its comparison against total correlation. DTC in eqn 1. removes the requirement to compute p(z). DTC requires computing the conditional entropies for disentanglement.\n2) Propose a new \"Gaussian Channel Autoencoder\" (GCAE). As the name suggests, gaussian noise with variance sigma^2 is added to smooth the representations. And in Sec 2, the authors have detailed how they use the discriminator-based method, which applies the density-ratio trick and the Radon-Nikodym theorem to estimate the density of samples from an unknown distribution.\n3) The authors conduct experiments on Beamsynthesis (time series dataset) and dsprites (synthetic images) to show GCAE is better than previous baselines. Fig 3a and 3b show the scatter plot of GCAE for summed information loss against mean squared error (MSE) and mutual information gap (MIG). GCAE performs better than baseline models for Beamsynthesis, while in the case of dsprites it does better only for a small range of lambda (hyperparameter of summed information loss).\n\n\n\nWeakness\n\n1) The novelty of the paper is limited. The authors claim that using DTC  removes the constraint of estimating p(z), which is difficult compared to the conditional distribution. However, computing \"m\" conditional distributions needs \"m\" discriminators, which are sensitive to hyperparameters. It also requires a lot of data and time.\n2) The authors use only the MIG metric to show disentanglement. The authors should have compared against other metrics such as DCI, SAP, etc. Please refer to \"Measuring Disentanglement: A Review of Metrics\" to understand why there is not one disentangling metric and each metric explores different aspects.\n3) The datasets used for the experiments are limited. The results work well for Beamsynthesis, not for dsprites. It can be seen in figure 4a and 4b. The scatter plot in figure 3b for beam synthesis shows a negative correlation coefficient of -0.823. There is no mention of dsprites.\n4) No qualitative results were shown for dsprites and no results on the latent traversal on these z axes. Since the authors chose m=10 as the latent dimension, and dsprites has five factors of variation, the qualitative result should have helped understand how the disentanglement happened.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster