Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Authors propose to regularize the Jacobian of the generator in a VAE setting so as to encourage local alignment of the latent axes, i.e. `local disantanglement'. They do so by finite differences, and empirically show on the MPI3D dataset the effectiveness of their approach.
main_review: 
## Strengths
The introduced method is quite well motivated and simple to implement. It empirically appears to be better suited than L2 regularization to disentangle the generative factors. These empirical results a fairly convincing.

## Weaknesses
As stressed in the paper, this method brings improvement in terms of `local' disantanglement but does not address global alignment. This is fair but there is perhaps a lack of motivation on why local disantanglement matters. This is not completely unsurprising as the Jacobian is by definition a local concept. 
Regarding computational aspects of the method, the regularization term $\|J_g\|_1$ is computed via with finite difference. This scales linearly in the number of latent dimensions. In practice, is this a marginal cost?
It would be interesting to see how a L2 regularization would perform (as opposed to L1 as proposed), as it is argued to lead to an objective that is invariant to rotation in  subspaces of equal singular vectors. In practice, is it very likely that some singular values be equal?

## Clarity
The paper is fairly well-written. I believe that it would be worth improving the explanation of Equation 2 as it is key in the motivation of the introduced method. Additionally, it is claimed that the approach is inspired by ICA, yet it is not detailed how so, nor is ICA really introduced. Regarding Sections 5 and 6, there are lots of details, yet paradoxically it takes a bit of time to understand the empirical settings. I believe that these sections can be streamlined a bit.

## Relation to prior work
Although the Related Work section is pretty good, I believe that a brief discussion of contractive auto-encoders (Rifai et al., 2011) could be useful as these regularize the $l2$ norm of the Jacobian and show that 'this penalty helps to carve a representation that better captures the local directions of variation dictated by the data'. If space is needed, the paragraph 'Sparse VAEs: Sparsity in Network Weights' could be removed since it is not extremely relevant to the topic (as the paragraph itself it stating).

## Reproducibility
The code source has been provided in the supplementary material and after a quick inspection, it seems to allow to reproduce their results.

## Additional feedback
140: "beta-VAE"
Equation 2: Would be useful to give a bit more context on how this equation is derived and what does the approximation symbol means here, as this is a key motivation for the later L1 regularization. Is this specific to $\beta$-VAE also?
Section 6.2: Wouldn't it be useful to visualize latent interpolations?
Figure 4: May be easier to plot error bars / confidence interval instead of each sample.


limitations_and_societal_impact: Yes limitations are discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors try to solve the identifiability issue of representation learning for generative processes with similar eigenvalues. Inspired by ICA, sparse coding, the authors propose to add an L1 regularizer of the Jacobian of decoder mean-mapping to the ELBO. To compare with other methods, the authors also adapt the common evaluation metrics (including MIG and Modularity) to the problem of local disentanglement. Both qualitative and quantitative results show the efficiency of the proposed method compared with baseline models.
main_review: Pros:
1. This paper discusses the novel problem of local disentanglement, which is interesting and might be applicable to problems such as data manipulation. Also, the solution is rather simple to be used in all VAE-based (even GAN-based) models.
2. The authors provide a very comprehensive survey of related works, including works on disentanglement and sparse VAE.
3. Results shown in Sec.6 are interesting. Particularly, the comparison with beta-VAE in Fig.2 is convincing that the proposed method works.

Cons:
1. The correctness of the proposed method is based on the hypothesis that the mapping from latent features to real data in real generation process is locally linear. Otherwise, the L1 loss learn is useless to recover the real generation process. There is no discussion about this hypothesis and experiments are performed on simple datasets where this hypothesis naturally holds.
2. The ablation study of \rho in Fig.4 make me feel that the authors are selecting the best parameter of the evaluation metric specifically for their proposed method. It might be unfair. Also, the authors might want to add some experiments on downstream tasks to show the proposed method is useful.
3. It is not necessary to introduce how to calculate Jacobian in Sec. 4.1. There are packages to do that. Please refer to https://stackoverflow.com/questions/50244270/computing-jacobian-matrix-in-tensorflow/53798621.
4. Some mathematical mistakes:
  a. In Eqn.2. H should be the Hessian of logP rather than P. Otherwise, the matric is no longer diagonal.
  b. In line 169, I think the i-th column of Jacobian matrix should be (g(z_i)-g(z))/epsilon.
limitations_and_societal_impact: This paper shares the limitations and potential negative social impact of most works on representation learning. As a result, I don't think the authors need to solve these issues in this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a novel disentanglement strategy for VAEs based on the induced sparsification of the decoder Jacobian.
main_review: The paper addresses an important problem in representation learning and the solution it proposes is simple and, to the best of my knowledge, novel. However, there is a number of issues I found reading the manuscript.

1) Only this one is potentially a critical flaw of the technique and therefore I will start from this. I have some doubt that a sparse decoder Jacobian implicates local disentanglement in the general case. Using the image domain as an example, as done in the paper, a sparse Jacobian means that changes in one element of a latent variable cause changes in the fewest pixels in the generated image. This corresponds to disentanglement only if the generating factors of variations are aligned with this notions, e.g. small objects moving, but it does not for more global factor of variations, e.g. global coloration, shadowing ecc. The experiments in the paper all use datasets that seem to be specifically suited to have this jacobian sparsity-disentanglement correspondence, which I don't think is true in the general case. There would need to be further experiments with more globally affecting sources of variation in the data, or readjust the claims of disentanglement.

2) There may be two typos in the technical section that makes things rather confusing. On line 154: "we show how penalizing the sparsity of the Jacobian, that is, penalizing the L1 norm...". Should it be "encouraging the sparsity"? Otherwise "penalising the sparsity" and "penalising the L1 norm" are contradicting. 
Then, in equation 3, should it be a minus sign in front of the gamma*L1_norm term? (or gamma is negative?). This is the ELBO, therefore it is maximised, and I assume you want to minimise the L1 norm.
These might just be unfortunate typos, but in combination make the core definition of your method very confusing.

The reason for my low score is mostly point 1) and I will be happy to higher it if it is convincingly addressed, or alternatively if the claims are readjusted to better defined in which domains we can expect the proposed method to give good disentanglement.
limitations_and_societal_impact: Limitations are well defined, aside of that of point 1) above, which I feel is a critical flow, if not addressed properly.
review_title: null

Meta-review: This paper introduces a simple and effective way to disentangle factors of variation in VAE representations by penalizing the Jacobian of the decoder with an L1 regularizer. The reviewers unanimously agree that this is a compelling approach, and their concerns were addressed during the discussion period. There are a number of clarifications that should be added to the final draft based on these discussions, but perhaps the main one is to clarify the relationship between the approach and ICA/sparse coding based on learning localized features. Please read through the reviews carefully and ensure that the paper is updated accordingly, as requested by the reviewers.
Decision: Accept (Poster)