Local Disentanglement in Variational Auto-Encoders Using Jacobian $L_1$ Regularization
Authors: Travers Rhodes
Daniel Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: Representation Learning
Sparsity
Variational Auto-Encoders
Disentanglement
Regularization

Abstract: There have been many recent advances in representation learning; however, unsupervised representation learning can still struggle with model identification issues related to rotations of the latent space. Variational Auto-Encoders (VAEs) and their extensions such as $\beta$-VAEs have been shown to improve local alignment of latent variables with PCA directions, which can help to improve model disentanglement under some conditions. Borrowing inspiration from Independent Component Analysis (ICA) and sparse coding, we propose applying an $L_1$ loss to the VAE's generative Jacobian during training to encourage local latent variable alignment with independent factors of variation in images of multiple objects or images with multiple parts. We demonstrate our results on a variety of datasets, giving qualitative and quantitative results using information theoretic and modularity measures that show our added $L_1$ cost encourages local axis alignment of the latent representation with individual factors of variation.

Local Disentanglement in Variational Auto-Encoders Using Jacobian L1 Regularization
Travers Rhodes
Daniel Lee
Department of Computer Science Cornell Tech, Cornell University
Department of Electrical and Computer Engineering Comeel Tech, Cornell University
New York, NY 10044
New York, NY 10044 dd146@cornell.edt
tsr428cornell.em
Abstract
There have been many recent advances in representation learning; however, unsu- pervised representation learning can still struggle with model identification issues related rotations of the latent space. Variational Auto-Encoders (VAEs) and their extensions such 3- VAEs have been shown to improve local alignment of latent variables with PCA directions, which can help to improve model dis- entanglement under some conditions. Borrowing inspiration from Independent Component Analysis (ICA) and sparse coding, we propose applying an L1 loss the VAE's generative Jacobian during training to encourage local latent variable alignment with independent factors of variation images of multiple objects or images with multiple parts. We demonstrate our results a variety of datasets, giving qualitative and quantitative results using information theoretic and modular- ity measures that show our added Lı cost encourages local axis alignment of the latent representation with individual factors of variation.
1 Introduction
Unsupervised representation learning takes a collection of image data from the world and figures out how to organize and find patterns in the data without additional information about how the images were generated. The ideal representation learning algorithm would compress high-dimensional image data into lower-dimensional latent representation that contains relevant information about the ground-trume truth factors variation that generated the image.
Inferring good latent representation from dataset is a difficult problem and is generally under- specified in algorithms. This underspecification is called the "model identification" problem, and one example is the fact that representation learning algorithms often struggle to precise the correct orientation of a latent space. That is, optimization criteria used to learn a representation function might be equally well satisfied by equivalent representation that is just rotation of the latent space by an arbitrary amount.
As commonly implemented (using axis- axis-aligned Gaussian posterior distributions), Variational Auto- Encoders (VAEs) [ and their extension, the 3-VAE [2] solve the rotational part of the model identification issue by tending to ensure that the generation function's 's Jacobian matrix has orthogonal columns (i.e., that the generative Jacobian matrix's right singular values are aligned with the axes of Intuitively, this is because the VAE's stochastic reconstruction cost prefers to budget higher precision (lower embedding noise) the directions along which the generative Jacobian changes most rapidly. Kumar and Poole 4 draw the parallel between this preferred orientation and linear Principal Component Analysis (PCA). However, we note that learning algorithms that resolve rotational identification issues through methods related to PCA will still suffer from an identifiability issue related to rotations that mix the directions for which the generative Jacobian matrix has equal
the latent space) 3 4
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

singular values. Along these directions, the posterior Gaussian would have approximately equal variance and be rotationally symmetric.
We propose adding an I cost to the generation function's Jacobian matrix as a way to resolve that rotational identifiability issue. Since [1 cost is not rotation invariant, L1 regularization creates a preferred latent-space orientation among directions whose singular values are equal. This use of the L1 norm choose an orientation is inspired by similar use in linear models. For example, when using Laplacian prior in Independent Component Analysis (ICA) [S] and applying it to already-whitened data, ICA rotates the data minimize the L1 norm. As shown in Olshausen [6]. that ICA formulation is equivalent to sparse coding using an L1 cost [7]. with the same rotational effect. Sparse PCA, the L1 norm encourages sparsity the loadings/mixing matrix (rather than the principal components/sources as in ICA) & D. similarly encouraging preferred orientations. These techniques for preferring certain orientations using the I, norm are all linear model techniques, and we apply them to non-linear VAEs by regularizing the In norm of the generator Jacobian matrix, thereby encouraging preferred orientation for our non- non-linear models.
The motivation above suggests that an L1 norm on the generator Jacobian can address the rotational identifiability issue, and we think the Li regularization will encourage useful orientations of the latent space for image data. This belief comes from results on the sparse linear coding of images. In particular, Olshausen and Field 7 showed how sparse linear coding on natural images generates local receptive fields similar to those discovered in the mammalian visual processing system. These types of localized receptive fields are shown Figure [ Each image in that figure corresponds to a direction in the latent space, and shows how perturbing the latent value in that direction changes the generated mmage-white pixels means the image gets brighter there, black pixels means the image gets darker. For the ICA basis, we see that perturbations in different latent directions are associated with localized changes to the image, whereas for the PCA basis, the latent directions tend to affect the entire image. Adding the generative Jacobian columns of our model, meaning that perturbations of latent values along individual latent directions should modify as few pixels as possible, leading to latent directions that affect the output image in localized regions. L1 generative Jacobian regularization should therefore disentangle representations of different objects in image. We call the proposed model trained with this additional regularization a Jacobian L, Regularized Variational Auto- -no-Enn (AE)..
L1 penalty to our model should encourage sparsity within
(a) PCA
(b) ICA
(c) 3-VAR
(d) JL.-VAE (ours)
Figure 1: Example columns from generative Jacobian matrices for different modeling techniques on natural image data collected in 77. ICA used 100 latent dimensions, of which 20 samples are shown. 3-VAE used 3 0.01: JLI-VAE used 0.01, 7 0.01, each on ten latent dimensions.
The use of the generative Jacobian in the JLI-VAR implies local linear approximation of the generative function, so order for the JI.I-VAE to be useful, the training image data should lie on a manifold [10]. and should be sufficiently well sampled. Additionally, JLI-VAE contains inductive

---

bias, like every other unsupervised disentanglement algorithm M. As mentioned above, JLI-VAE's regularization of the generative Jacobian encourages small changes in latent values result in sparse (impacting small number of pixels) changes to the resulting image, similar to local receptive fields. This inductive bias is well suited for disentangling motions of different objects in an image, but would presumably not be useful for whole-image changes, such as rotation of the entire image or brightness changes across the whole image.
We apply our novel JLI-VAE framework to a variety of datasets, giving qualitative and quantitative results showing that our added L1 cost can encourage local alignment of the axes of the latent representation with individual factors of variation.
Background
We present brief overview of VAEs and introduce the notation will use throughout this paper. VAEs [ train a model to generate a data distribution that approximately matches the distribution of unlabeled training data. For the VAEs consider in this paper, generating datapoint & R from a trained VAE consists of sampling a latent variable & E R from a standard multivariate Gaussian distribution (00, 1) and applying generator function g : R + R." n
map the latent variable to a generated image (z). Around this generated image we assume a Bernoulli probability of similar images á p(a; g (=)) with the generated image g(=) its mean.
To train the VAE we also define a multi-variate Gaussian embedding distribution ((=[z) for each training image æ with mean h(x:) (using a learned embedding function h : R - R²) and diagonal
covariance (a (1r) (using learned covariance function Egje R + R' that computes the diagonal elements). This embedding distribution ((2(zz) is motivated by desire to approximate the posterior distribution p(=|z))
The objective during training of the VAE is to maximize the Evidence Lower BOund (ELBO), which defined as Σ> L(x), where, for each data point x,
L(x)= = Ezng(z12) (p(()))())))))))))) - KL (v(:)))) 1))
(1)
The ELBO is lower bound on log(p(z)) log point given our model. Thus, maximizing the ELBO is proxy for maximum likelihood estimation. The 3-VAE 2 is an extension the VAE that multiplies the second term in the ELBO by an adjustable hyperparameter 3. The first term in the ELBO is a stochastic reconstruction loss, so we sometimes refer to Σz|= as the "embedding noise," " since it adds noise to the embedding when estimating the stochastic reconstruction loss.
which the likelihood of the data
Kumar and Poole 4 show that the stochastic reconstruction term of the ELBO, Esng(z(a) [log(p(>; (()))) can be approximated using a second-order Taylor expansion as:
Fena(ele) log(p(a;g(=)) logg(>/<())) + ""r
H (L.La(M))))))
Jg is the Jacobian of the generator function, and Hp, (g(h(x))) is the Hessian with respect to g(2) of the log of the generative probability log(p(æ; g(2))) evaluated at g(h(z)). For standard VAE implementations using diagonal Gaussian posteriors and pixel-factorize generative probabilities, Esja(2) (2) and Hp (o(h(z))) are both diagonal.
Equation 2 shows that the stochastic reconstruction loss of the ELBO can be approximated by a deterministic reconstruction loss with a (weighted) T-2 regularization cost on the Jacobian Jg (h(z)). Kumar and Poole 4 use this weighted La regularization in the approximation above to show how the ELBO encourages local alignment of the right singular vectors of Jg to (x). That is, larger values of _x)x(rr (larger embedding noise) in some directions leads to larger L2 regularization on the generator Jacobian in those directions. Equivalently, in directions with large changes in the generator Jacobian, the regularization encourages smaller embedding noise (more precision) in the posterior Egje (x). For this work, we use the presence of an implicit T.2 loss on the generator Jacobian as further motivation for our choice to add an explicit L1 regularization to the generator Jacobian.
3

---

3 Related Work
There are several areas of related work we wish to call to the reader's attention. We note previous work in the analysis of the disentanglement properties of B-VAEs. We delve into previous uses of the term "sparse VAE," as there are (at least) two other common and well studied meanings of "sparse VAE," which we disambiguate from the type of sparsity we study in this work. We note previous VAE work inspired by ICA. We discuss architectural choices that have been shown to improve disentanglement. Finally, we discuss modifications to the VAE objective that have previously been studied to improve disentanglement.
Disentanglement of B-VAEs Mathieu et al. 12] and Rolinck ct al. 33 show that restricting the posterior covariance to diagonal (called the mean-field assumption) breaks the rotational symmetry of 3-VAEs. Rolinek et al. S further show that this encourages the columns of the generator Jacobian to be orthogonal, relating local 3- 3-VAE latent directions with PCA decomposition. In our work, take this a step further, explicitly regularizing the sparsity of the generator Jacobian, to break rotational symmetry between directions with equal singular values in of the generator Jacobian.
Sparsity in VAE codes Some previous work involving L1 regularization and VAEs uses the term "sparse VAEs" to refer to sparsity in the latent values taken on by the latent codes themselves. That is, these works attempt a minimization of something like ||=[|1. This meaning is studied in [3] When
Sparse VAEs:
sparsity in the present work, however, we are not concerned with the values of the latents 2, but rather with how small axis-aligned changes in the latent values affect the output. That is, are concerned with sparsity of Jy (2),,, not x. Our associated cost is [| I| Our use of "sparsity" has to do with local disentanglement, rather than sparsity in latent values, which is a type of global disentanglement.
Sparsity in Network Weights Likewise, other work involving L, regularization and VAEs (and neural networks more broadly) uses "sparsity" to refer to the desire to make many network weights O, with the motivation of reducing the size of the stored neural network architecture. This meaning is seen in [16] 177. In this work, by contrast, we are interested in sparsity in the generator Jacobian, not in the network weights. We note the distinction between sparsity of individual network weights and sparsity in the Jacobian of the overall function. Multiplying sparse matrices does not necessarily result in sparse matrices, and non-lineat activations can allow a sparse Jacobian even if the network weights themselves study in this work is not what is referred to in studies of sparsity of neural networks, which consider regularizations like the Ln cost over network weights.
Sparse VA
not particularly sparse. Thus, the [1]]( cost
ICA within the VAE Literature Independent Component Analysis (ICA) (5] is often mentioned in the VAE literature in relation to the role ICA has played in the theory of identifiability and disentanglement of representations 4 12 3 18 19, Stuehmer et al. (20 propose using a structured, rotationally asymmetric prior to encourage disentanglement in the embedding. This, and other approaches that attempt to globally match the embedding distribution to a desired shape, are very different from the local, Jacobian-basee approach we take in this paper. Khemakhem al. [21] relate nonlinea ICA with VAES in the case where the data has an additionally observed variable, and they give proof that in that case their model is identifiable and correctly disentangles the ground-truth factors of variation. We focus on fully unsupervised training data and assume that we are not given access any data labels. We are not aware of any prior work applying a sparsity cost to the generator Jacobian, which is the inspiration we take from ICA.
Architectures shown to improve disentanglement Previous work has shown impressive results from modifying the network architecture in order to explicitly represent multiple objects by, for example, learning object masks 22). or by modifying how the latent variable is read in to the generative model architecture 23 Ainsworth et al. (24 and Khan et al. (25 both construct model architectures that explicitly encourage sparsity in the generative network weights. In this work, we focus on how we can regularize the objective function to improve disentanglement, rather than studying how different network architectures can improve disentanglement.
Modification of VAE prior Previous work has also investigated modifications to the unit Gaussian prior commonly used in VAEs. Tomezak and Welling (26) use learnable Gaussian mixture prior.

---

Stuehmer et al. (20] use generalized Gaussian distribution (that is not rotationally invariant) as the prior. Bauer and Mnih [27 use rejection sampling form more complicated, con-rotationally- invariant prior. Davidson et al. (88 and Perez. Rey et al. [29] even modify the prior to lie on non-Euclidean surfaces. Kim and Mnih 18] do not explicitly enforce a prior distribution, but rather use a regularization term to encourage the prior distribution ((=) to be factorized distribution. While these approaches pressure the entire embedded distribution to have certain properties, we are instead focused on how to modify the learning objective to give local bias toward disentanglement, rather than using more global methods based on the overall distribution of the embedded dataset.
Regularization of VAFs Several previous works explicitly implicitly L.2 normalization of the generator Jacobian [4 30 Chen et al. 33 regularize by 1 (z) ct el for some
constant c. We are not aware of previous investigations (z) regularization for VAEs.
4 Model Loss Calculation and Architecture
4.1 Loss Calculation
We define a Jacobian L1 Regularized Variational Auto-Encoder (JLI-VAE) VAE that is trained using the B-VAE loss augmented with an L regularization of the Jacobian matrix of the map from latent values mean generated images. The regularization term is modulated by hyperparameter y. Specifically, the maximization objective for the IL1-VAE is the sum over all datapoints x of
(xn Esng(e(2) logp(s)=)->11] -1 (KL()))))) 1))
Since we use a Gaussian posterior (=||=) N (M(+),, Esja(z)), we can use an explicit calculation for the KL-divergence. We estimate the expectation of log p(z|z) and of > using a single sample from the distribution over which are taking the expectation. We estimate the full Jacobian matrix (2) using the finite difference method along each latent dimension. This leads to a runtime that scales roughly linearly with the number of latent variables in the VAE architecture.
4.2 Architecture
We use a convolutional architecture for our VAEs. In particular, our embedding architecture consists of convolutional layers followed by fully connected layer with ReLU activations. This base model is shared between the mean and log variance embedding networks. Each embedding network then appends its own linear fully connected head the shared model. We use diagonal structure for the log variance estimates to reduce the number of parameters we need to estimate. We use a latent dimension of ten for all experiments, though we have seen similar results for other latent dimension sizes. The reconstruction architecture consists of fully connected layers followed by convolutional layers, using ReLU activations, with final sigmoidal activation function. A full set of hyperparameter choices for each experiment can be found in the Appendix. When we compare JL.1VAA with other methods, we ensure consistent architecture choices.
Experiments
5.1
Datasets
To evaluate the ability of JLI-VAE to locally disentangle factors of variation, we apply it to a variety of datasets.
The first are natural images in grayscale taken by Olshausen and Field [7] and cropped to 16 x 16-pixel regions. We do not have labeled "ground-truth factors of variation" for this data, but are able to provide qualitative results by inspecting the columns of the generator Jacobian matrix. This data was made publicty available without specific license,
analyze it under fair use.
The second is a dataset of simulated 64 64-pixel grayscale images of three black dots on white background, inspired by [34). The ground truth factors of variation for this dataset are the x/y
5

---

coordinates of the dot centers. We re- re-implement Zhao et al. [34 s code to generate the dot images and modify the code so that dots can overlap. We note for this dataset that if model were to disentangle individual dot motions into different latent directions, then, by symmetry, we would expect identical singular values of Jg for those directions. Thus, we expect that for this dataset (3 VAS will be unable to isolate individual dot motions, since it has trouble disentangling directions in which the generator Jacobian has equal singular values.
Finally, we also apply our approach to tiled images of a real robotic arm taken from the MPI3D-real dataset [35), licensed under Creative Commons Attribution 4.0 International License. For each data point, we downsample four random images of the robot arm holding large blue square in different locations and tile the random images in a 2x2 pattern to generate new, more complicated 64 x 64-piel image containing four different images of real robotic arm. We call this tiled image dataset MPI3D-Multi.
5.2 Training
For the three-dots and MPI3D-Multi datasets, we train using Bernoulli loss on batches of 64 images over a total of 300,000 batches. We use the Adam optimizer with a learning rate 0.0001 (matching [I.. We use linear annealing from 0 to the final hyperparameter value over the first 100,000 batches for both the beta hyperparameter and JAE's > parameter in our implementations for JL1-VAS VAE and 3-VAE (unlike (II).. We note annealing to be beneficial to avoid model collapse when adding our L1 regularization term. We train these models on a Nvidia Quadro V100 hosted locally and one hosted Google Cloud. Training each JLLI-AE model on ten latent variables takes approximately 2.5 hours, while training each 3-VAE model takes approximately 45 minutes. In total, training ten JL1-VAE models and ten 3-VAE models for quantitative evaluation takes roughly 33 hours.
For the natural image dataset, we train for 100,000 batches of 128 images. We use the Adam optimizer with learning rate of 0.001 and train on a Nvidia Quadro V100s hosted locally. Training takes nine minutes for the V-VAE and 23 minutes for the JLI-VAE.
5.3 Evaluation Metrics
There are several metrics commonly used to measure "disentanglement" of a latent representation. In this work address two common metrics, the Mutual Information Gap (MIG) [36] and Modular- ity (37] and show how we are able to provide extensions to these metrics that give a measure of how well representation locally disentangles factors of variation.
The original MIG and modularity metries measure global disentanglement- that is, they measure across the whole dataset how well each latent variable maps to unique ground-truth factor of variation. Since the JLI-VAE does not add an explicitly global disentanglement incentive to 3-VAEs, but instead is designed to locally encourage disentanglement using the Jacobian of the generative map, we do not expect it to necessarily improve the global disentanglement of factors of variation. For example, the JL1-VAE may assign factors of variation to latent variables using one pairing in one local region of the latent space and a different pairing in a different region of the latent space. This could lead good average local disentanglement, but would not lead to good global disentanglement.
We are therefore interested in defining local disentanglement metrics based on MIG and modularity. We call these metrics "local MIG" and "local modularity," and the general form of their calculation is to compute each metric several times on different random "local" samples from the global dataset and then average the results. As with MIG and modularity, the calculation of these disentanglement metrics requires a generative model of the data from ground-truth factor values.
The key technique for each of our local metrics is to repeatedly compute the disentanglement metric on random local samples of data. To generate a random local sample of data, we we randomly choose a centroid from the ground-truth factor values and then random sample N ground-truth datapoints within an Lox distance p from that centroid. The radius p is a hyperparameter determining how close ground-truth factors of variation need to be in order to be considered "local." We scale the hyperparameter p as a fraction of the total range of available values for each latent variable. This set of N datapoints comprises each local data sample. For our experiments choose N = 10, 000.

---

(a) Example three-dots images
Jacohian matrix columns for 3-VAF
(c) Jacobian matrix columns for JLI-VAE
Figure 2: Qualitative results for the three-dots dataset. We show six Jacobian matrix columns for 3-VAE (3 = 4) and JLI-VAE (3 = 4, 7 = 0.1) evaluated for the leftmost example image.
For each local sample data, we apply the MIG and modularity metries to that sample determine the disentanglement of the latent space in that local region. We use the MIG and modularity calculation implementations from the open-source (Apache License 2.0) disentanglement__u library [11
We repeat this algorithm with 20 different local data samples and report the average the local disentanglement score.
Results
present qualitative results for the three-dots, MPI3D-Multi, and natural image datasets, and quantitative results for the three-dots dataset.
6.1 Qualitative Results
Qualitative results are generated by inspecting the generator Jacobian at the (deterministic) latent embeddings for example images. Each generative Jacobian matrix column is associated with latent direction and shows how the generated image would change from a slight perturbation to the embedding in that latent direction. The generative Jacobian matrix columns for natural images shown in Figure] [ There, we show the largest 20 PCA components, an arbitrary sample of 20 ICA components (from training 100 ICA components using FastICA (3)), and the ten Jacobian matrix columns for the 3-VAE and JL. models. Additional visualizations are included in the Appendix. We discern more localization (results more similar local receptive fields) in the JLI-VAE and ICA results, compared to the 3-VAE and PCA results.
For the three-dots dataset and MPI3D-Mult,, we follow the same procedure to generate qualtitative results. We show results for the three-dots dataset in Figure 2 There, we show the six Jacobian columns with the largest L2 norms for the three-dots dataset for both our JL1-VAE and 3- 3-VAE (all ten Jacobian columns are visualized in the Appendix). Qualitatively, we see that, when evaluating the Jacobian of the generator function for our JL1-VAE, individual dot motions separated into different latent components. The 3- VAE does not exhibit this behavior.
For the MPI3D-Multi dataset, containing tiled images of real robot, we again that the JL1-VAL does a better job separating the four robots contained each image into separate latent variables. These results are presented in Figure
6.2 Quantitative Results
We generate local disentanglement scores for the models trained on the threc-dots images.
To observe the effect of the parameter of our local disentanglement metrics, in Figure we plot the varying local disentanglement scores as we change the P parameter for our VAEE (3 = 4.0 and 0.1) and standard 3- VAE (8 = 4.0). The hyperparameter for 3 was chosen near the middle of the range used [II. We also tried other hyperparameter values and saw similar results. Too large can lead to model collapse, we chose a small enough 7 to avoid that collapse but otherwise large enough to start to see reconstruction performance degradation, so we knew that its regularization was affecting model training.
7

---

(a) MPllD-Multi image
(b) 3-VAE reconstruction
(c) JLL- VAE reconstruction
"
(d) Jacobian matrix columns for 3-VAE
(e) Jacobian matrix columns for a JL1-VAE
Figure 3: Qualitative results for MPI3D- Multi. JLI-VAE shows stronger pressure to locally disentan- gle individual robot motions. Both models used = 0.01. For JL1-VAE y 0.01.
We note quantitatively that JLI-VAE attains higher local disentanglement scores compared to 8- 9-VAEs, which is especially true as we look more localized samples of data, corresponding to a smaller parameter. For p 0.1 compared to 3- VAE (p < 0.001 for T-test), but for p = 1, testing for global disentanglement, we indistinguishable disentanglement scores between the two (p 0.05 for T-Test). Our JL1-VAE able to locally disentangle factors of variation for the three-dots dataset, but does not globally disentangle factors of variation.
significantly higher local disentanglement scores for JL1-VAL ILI-VAE
beta-VAE
beta-VAE
0.14
JL1-VAE
0.9
J.1-AAE
0.8
0.10
local modulleis
tocal MiG
0.08
0.7
0.06
0.6
0.94
0.02
0.00
0.3 Locality
Locality
range!
range)
(a) Local MIG scores
(b) Local modularity scores
Figure 4: Local disentanglement scores varying the locality parameter p. Ten JL1-VAE and 8- 3-VAE models were trained on the three-dots dataset with B = 4 and, for JLI-VAE, 7 = 0.1.
Fixing the parameter to 0.1, we also compute the local MIG and local modularity scores for six different comparative methods, namely 3-VAE, FactorVAE, DIP- VAE-I, DIP- 3-TCVAE and AnnealedVAE, using the implementations of disentanglenmt ement.eeee....m.mmmmmm lib with convolutional architecture. A description of each of these models can be found in [ТО. We trained ten iterations of those models with different random seeds using hyperparameters chosen near the middle of the suggested ranges in that work. That included training ten new 3 VAE models with new random seeds. All models were trained with ten latent dimensions.
Local MIG and local modularity scores are shown in Figure 5
We see a range of disentanglement scores due to the random seeds used to generate our models (ten models for each learning algorithm). Additionally, our local MIG and modularity metrics have some additional stochasticity due to the randomness in sampling 20 local samples of 10,000 points during calculation of those metrics.

---

Nevertheless, we observe significantly higher disentanglement scores (p < 0.001 for T-test) for our JLI-VAE compared to every baseline method.
0.14
0.94
6.17
0.92
lccal medulanty
0.90
MGG
0.10
0.06
0.86
0.84
0.04
0.82
6.02
RLVAE (ours)
beta VAE
factor-VAE
dig dip-AE--
up--AE--
VeE
annealed VAE
R.L-AE (ours)
bata-VAB
factor-AAE
dip dip-VAE-
beta-tt- VAE
anneated- VAE
(a) Local MIG scores
(b) Local modularity scores
Figure 5: Local disentanglement scores for JL1-VAE models and baseline implementations from The baseline implementations use default hyperparameters from that paper, choosing values near the middle when a range of hyperparameters are listed. Fach model is run ten times with new random seeds. Local disentanglement is calculated using p = 0.1 with 20 different local samples.
7 Discussion
In this work, presented JLI-VAE, VAE augmented with L1 regularization to the Jacobian to improve local disentanglement. We extended the MIG and modularity disentanglement metrics to generate metrics that can measure local disentanglement. We evaluated our model on natural images, simulated images of dots, and tiled images of a real robot, and showed qualitatively and quantitatively that our method can improve local disentanglement the generated representation.
Our added L1 regularization to the Jacobian of the generator function is motivated by the use of L, regularization to prefer certain orientations in ICA and sparse coding, by a desire relate each latent direction to sparser pixels (more similar to localized receptive fields), and by the implicit L2 regularization already present in 3-VABs. We show that this L, regularization term can encourage latent axes to locally align with ground-truth factors of variation. While this approach shows promise for local alignment, it does not address global alignment issues. That is, in part of the dataset, the learned representation may assign latent variable é to follow a certain latent factor of variation, and in a different part of the dataset might be a different latent variable eg that follows that latent factor of variation.
Regarding "no free lunch" theorems that show unsupervised disentanglement is impossible without inductive biases [11], we note that L₁ regularization of the generative Jacobian generates an inductive bias. The inductive bias encourages small axis-el perturbations of the latent space to result in sparse changes to the image space, whether that be expressed as localized receptive fields which act only small regions of the image space, as in Figure [ or motions of only single dot robot, as seen in Figures (aaures[]nd]] and Cases where this inductive bias might not add value would include, for example, single robot images from MPI3D, where the primary ground-trutt factors of local variation affect the same object within the same image patch, or images where one factor of variation includes global lighting changes.
This work has only applied this learning method to image data, and we would like to apply this approach to multimodal data gathered from real robotic system understand broader applications.
Finally, this approach computes the full Jacobian of the generator during training, leading to training times that scale linearly with the number of latent dimensions. Future work should look to sampling- based methods to approximate the L, cost by only computing a subset of the Jacobian values in order to speed up training.

---

References
[1] Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2014, 1312.6114.
[2] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. In 5th International Conference on Learning Representations, ICLR 2017 Conference Track Proceedings, 2017.
[3] Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea directions (by accident). In Proceedings of the IEEE Computer Society Conference on Com- puter Vision and Pattern Recognition, volume 2019-June, pages 9781728132938.
2019, ISBN
[4] Abhishek Kumar and Ben Poole. On implicit regularization in 3-VAEs. In Hal Daumé and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 19 of Proceedings of Machine Learning Research, pages 5480-5490, PMLR, 13-18 Jul 2020. URL http: http://prroceedings mlr. wwtpp//rsseeeeeeeeeeeeeeeeeeeeee 200 html
[5] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, (34-5))411-4300 2000.
[6] Bruno Olshausen. Learning linear, sparse, factorial codes. Technical Report A.J. Memo No. 1580, MIT, Septemher 1996. URI.http://dae
mit wtt.:eeeem....etteeemmee htt.eeeaahaanne//11...//// 1/7184
[7] Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381 (6583))6) 1996. ISSN 00280836.
[8] Ian T Jolliffe, Nickolay T Trendañilov, and Mudassir Uddin. A Modified Principal Component Technique Based on the LASSO. Journal of Computational and Graphical Statistics, 12(3): 531-547, 2003. ISSN 10618600.
[9] Hui Zou, Trevor Hastie, and Robert Tibshirani. Sparse principal component analysis. Journal of Computational and Graphical Statistics, 15(2):265-288) 2006. ISSN 10618600.
[10] H. Sebastian Seung and Daniel Lee. The manifold ways of perception. Science, 290(5500)): 2268 2269, 2000, /ttp://////////////.....//////// htttp::/Hrt http:///wwwwww.ieeeeeee.me
200.500...20.0002222 URL 290. 5500 2268
science.
[11] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference Machine Learning, pages 4114 2019.
[12] Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In 36th International Conference on Machine Learning. ICML 2019, volume 2019-June, pages 7744-7754, 2019, 1812.02833. ISBN 9781510886988. URL com htt.://....oot.eeeeaaaaaaaannnnn
[13] Alireza Makhzani and Brendan Frey. k-Sparse autoencoders. In 2nd International Conference on Learning Representutions, ICLR 2014 Conference Truck Proceedings, 2014, 1312.5663.
[14] Andrew Ng. CS294A Lecture Notes Sparse Autoencoder. Cs294a, pages 1-19, 2011, arXiv:1506.03733vl ISSN 19326203. URL http://w// www.:.staffrdd.ooooo htwpafforr.ddd/laaa//ss//2224/
[15] Linxing Preston Jiang and Luciano de Iglesia. Improved training of sparse coding variational autoencoder via weight normalization. CoRR, abs/2101.09453,, 2021, 2101.09453, URL
https://arxiv.. wtwpe/iv.or//abs//1112 httrg/bbs/1101.. 09453
[16] Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through L0 regularization. In 6th International Conference on Learning Representations, ICLR 2018- Conference Track Proceedings. 2018, 1712.01312. ISBN 1712.01312V2.
10

---

[17] Misha Denil, Babak Shakibi, Laurent Dinh, Marc aurelio Ranzato, and Nando De Freitas. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, 2013, 1306.0543.
[18] Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In 35th International Conference on Machine Learning, ICML 2018, volume 6, pages 4153-4171, 2018, 1802.05983. ISBN 9781510866 5701696667
[19] Francesco Locatello, Ben Poole, Gunnar Raetsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 6348 6359. PMLR, 13-18 Jul 2020. URL http::/pppceeeeiingg/e//o//////e mlr wwwpprrsssseeee hwtsss/vvv9////// httl///11///oeee11e1e1 hhmmmmm0..mmmmm11m.
[20] Jan Stuehmer, Richard Turner, and Sebastian Nowozin. Independent subspace analysis for unsupervised learning of disentangled representations. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1200-1210. PMLR, 26-28 Aug 2020. URL htt.htttp//pcceeeeee www.inng..........mmmmmmm.m..m.m mlr. press/ 1008/ stwssmuu 20a. html
[21] lyess Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2207-222 2217. PMLR, URL URL http:://proceeddnggs
26-28 Aug 2020.
moriinngmm.........mmmmmmmmmmmmm mlr. press v108/ www.akhemm0a......mmmmma html
[22] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object rep- resentation learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Leurning, volume 97 of Proceedings of Machine Learning Research, pages 2424-2433, PMLR, 09-15 Jun 2019. URL http:tt::ppppcceeedddee
mlr. press/ prwss/v97 v97 gref fareff19e. 19a. html
[23] Nicholas Watters, Loic Matthey, Christopher P. Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes, 2019, 1901.07017.
[24] Samuel K. Ainsworth, Nicholas J. Foti, Adrian K. C. Lee, and Emily B. Fox. Vi- Output interpretable VAEs for nonlinear group factor analysis. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 119-128. PMLR, 10-15 Jul 2018. URL httt:://proceedings mlr. wwwppppwwsseeeepp http:///nnsswoothh..mmm html
[25] Rayyan Ahmad Khan, Muhammad Umer Anwaar, and Martin Kleinsteuber. Epitomic variational graph autoencoder. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 7203-7210, 2021.
[26] Jakub M. Tomezak and Max Welling. VAE with vampprior. In International Conference on Artificial Intelligence and Statistics, AISTATS 2018, pages 1214-1223, may 2018, 1705.07120. URL http::///axxvv..e wttg//bb//000.07111/01
[27] Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In AIS- TATS 2019- 22nd International Conference on Artificial Intelligence and Statistics, oct 2020, 1810. 14428. htt:: httt::://rrxvvv org/ htse//bs////// 1810. 11428
[28] Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomezak. Hyper- spherical variational auto-encoders. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, volume 2, pages 356-865, apr 2018, 1804.00891. ISBN 9781510871601. URL pttt://arxiv..o abs 1804.00897
[29] Luis A. Perez Rey, Vlado Menkovski, and Jim Portegies. Diffusion variational autoencoders. In IJCAI International Joint Conference on Artificial Intelligence, volume 2021-Janua, -Janua, pages 2704-2710 10, 2020, 1901.0899) ISBN 9780999241165.

---

[30] Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on Machine Learning, ICMI. 20/ I, pages 833-840, 2011. ISBN 978 1450306195.
[31] Dániel Varga, Adrián Csiszárik, and Zsolt Zombori. Gradient Regularization Improves Accuracy of Discriminative Models. Schedae Informaticae, 27:31-45, 45, 2018, 1712.09936. ISSN 20838476.
[32] Judy Hoffman, Daniel A. Roberts, and Sho Yaida. Robust learning with jacobian regularization, 2019, 1908.02729.
[33] Nutan Chen, Alexej Klushyn, Francesco Ferroni, Justin Bayer, and Patrick van der Smagt. Learning flat latent manifolds with VAEs. In 37th International Conference on Machine Learning, ICML 2020, volume PartF16814, pages 1565-1574, 2020, 2002.04881. ISBN 9781713821120.
[34] Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah D.. Goodman, and Stefano Ermon. Bias and generalization in deep generative models: An empirical study. CoRR, abs/181 1.03259, 2018, 1811.03259. URL /tatx:///rxiv owg/ ht////ab//111111 03259
[35] Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovie, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: glement dataset. In H. Wallach, H. Larochelle, A. Beygelzimer, d' Alché-Buc Buch E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019, URL https::ttt/pppt https:///rrccceddnggg neurips ct/ cc/ppaper//019/fillee .pf
disentan-
[36] Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in vaes. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 2615-2625, 2018.
[37] Karl Ridgeway and Michael C. Mozer. Learning deep disentangled embeddings with the f-statistic loss. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 185-194, Red Hook, NY, USA, 2018. Curran Associates Inc.
[38] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python, Journal of Machine Learning Research, 12:2825-2830, 2011.
12


Reviews:
[{"review_id": "xCr0ZJc4xFw", "reviewer": "Reviewer_Bxq2", "summary": "In this paper, the authors try to solve the identifiability issue of representation learning for generative processes with similar eigenvalues. Inspired by ICA, sparse coding, the authors propose to add an L1 regularizer of the Jacobian of decoder mean-mapping to the ELBO. To compare with other methods, the authors also adapt the common evaluation metrics (including MIG and Modularity) to the problem of local disentanglement. Both qualitative and quantitative results show the efficiency of the proposed method compared with baseline models.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This paper discusses the novel problem of local disentanglement, which is interesting and might be applicable to problems such as data manipulation. Also, the solution is rather simple to be used in all VAE-based (even GAN-based) models.\n2. The authors provide a very comprehensive survey of related works, including works on disentanglement and sparse VAE.\n3. Results shown in Sec.6 are interesting. Particularly, the comparison with beta-VAE in Fig.2 is convincing that the proposed method works.\n\nCons:\n1. The correctness of the proposed method is based on the hypothesis that the mapping from latent features to real data in real generation process is locally linear. Otherwise, the L1 loss learn is useless to recover the real generation process. There is no discussion about this hypothesis and experiments are performed on simple datasets where this hypothesis naturally holds.\n2. The ablation study of \\rho in Fig.4 make me feel that the authors are selecting the best parameter of the evaluation metric specifically for their proposed method. It might be unfair. Also, the authors might want to add some experiments on downstream tasks to show the proposed method is useful.\n3. It is not necessary to introduce how to calculate Jacobian in Sec. 4.1. There are packages to do that. Please refer to https://stackoverflow.com/questions/50244270/computing-jacobian-matrix-in-tensorflow/53798621.\n4. Some mathematical mistakes:\n  a. In Eqn.2. H should be the Hessian of logP rather than P. Otherwise, the matric is no longer diagonal.\n  b. In line 169, I think the i-th column of Jacobian matrix should be (g(z_i)-g(z))/epsilon.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper shares the limitations and potential negative social impact of most works on representation learning. As a result, I don't think the authors need to solve these issues in this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "vcLvCMPV9R", "reviewer": "Reviewer_ZTNa", "summary": "The paper presents a novel disentanglement strategy for VAEs based on the induced sparsification of the decoder Jacobian.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem in representation learning and the solution it proposes is simple and, to the best of my knowledge, novel. However, there is a number of issues I found reading the manuscript.\n\n1) Only this one is potentially a critical flaw of the technique and therefore I will start from this. I have some doubt that a sparse decoder Jacobian implicates local disentanglement in the general case. Using the image domain as an example, as done in the paper, a sparse Jacobian means that changes in one element of a latent variable cause changes in the fewest pixels in the generated image. This corresponds to disentanglement only if the generating factors of variations are aligned with this notions, e.g. small objects moving, but it does not for more global factor of variations, e.g. global coloration, shadowing ecc. The experiments in the paper all use datasets that seem to be specifically suited to have this jacobian sparsity-disentanglement correspondence, which I don't think is true in the general case. There would need to be further experiments with more globally affecting sources of variation in the data, or readjust the claims of disentanglement.\n\n2) There may be two typos in the technical section that makes things rather confusing. On line 154: \"we show how penalizing the sparsity of the Jacobian, that is, penalizing the L1 norm...\". Should it be \"encouraging the sparsity\"? Otherwise \"penalising the sparsity\" and \"penalising the L1 norm\" are contradicting. \nThen, in equation 3, should it be a minus sign in front of the gamma*L1_norm term? (or gamma is negative?). This is the ELBO, therefore it is maximised, and I assume you want to minimise the L1 norm.\nThese might just be unfortunate typos, but in combination make the core definition of your method very confusing.\n\nThe reason for my low score is mostly point 1) and I will be happy to higher it if it is convincingly addressed, or alternatively if the claims are readjusted to better defined in which domains we can expect the proposed method to give good disentanglement.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are well defined, aside of that of point 1) above, which I feel is a critical flow, if not addressed properly.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "M2_eDojLvoF", "reviewer": "Reviewer_rKED", "summary": "Authors propose to regularize the Jacobian of the generator in a VAE setting so as to encourage local alignment of the latent axes, i.e. `local disantanglement'. They do so by finite differences, and empirically show on the MPI3D dataset the effectiveness of their approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Strengths\nThe introduced method is quite well motivated and simple to implement. It empirically appears to be better suited than L2 regularization to disentangle the generative factors. These empirical results a fairly convincing.\n\n## Weaknesses\nAs stressed in the paper, this method brings improvement in terms of `local' disantanglement but does not address global alignment. This is fair but there is perhaps a lack of motivation on why local disantanglement matters. This is not completely unsurprising as the Jacobian is by definition a local concept. \nRegarding computational aspects of the method, the regularization term $\\|J_g\\|_1$ is computed via with finite difference. This scales linearly in the number of latent dimensions. In practice, is this a marginal cost?\nIt would be interesting to see how a L2 regularization would perform (as opposed to L1 as proposed), as it is argued to lead to an objective that is invariant to rotation in  subspaces of equal singular vectors. In practice, is it very likely that some singular values be equal?\n\n## Clarity\nThe paper is fairly well-written. I believe that it would be worth improving the explanation of Equation 2 as it is key in the motivation of the introduced method. Additionally, it is claimed that the approach is inspired by ICA, yet it is not detailed how so, nor is ICA really introduced. Regarding Sections 5 and 6, there are lots of details, yet paradoxically it takes a bit of time to understand the empirical settings. I believe that these sections can be streamlined a bit.\n\n## Relation to prior work\nAlthough the Related Work section is pretty good, I believe that a brief discussion of contractive auto-encoders (Rifai et al., 2011) could be useful as these regularize the $l2$ norm of the Jacobian and show that 'this penalty helps to carve a representation that better captures the local directions of variation dictated by the data'. If space is needed, the paragraph 'Sparse VAEs: Sparsity in Network Weights' could be removed since it is not extremely relevant to the topic (as the paragraph itself it stating).\n\n## Reproducibility\nThe code source has been provided in the supplementary material and after a quick inspection, it seems to allow to reproduce their results.\n\n## Additional feedback\n140: \"beta-VAE\"\nEquation 2: Would be useful to give a bit more context on how this equation is derived and what does the approximation symbol means here, as this is a key motivation for the later L1 regularization. Is this specific to $\\beta$-VAE also?\nSection 6.2: Wouldn't it be useful to visualize latent interpolations?\nFigure 4: May be easier to plot error bars / confidence interval instead of each sample.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes limitations are discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a simple and effective way to disentangle factors of variation in VAE representations by penalizing the Jacobian of the decoder with an L1 regularizer. The reviewers unanimously agree that this is a compelling approach, and their concerns were addressed during the discussion period. There are a number of clarifications that should be added to the final draft based on these discussions, but perhaps the main one is to clarify the relationship between the approach and ICA/sparse coding based on learning localized features. Please read through the reviews carefully and ensure that the paper is updated accordingly, as requested by the reviewers.
Decision: Accept (Poster)