Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors study how well they can recover causal structure in inferred latent variables given sparsity in perturbations.
Roughly speaking, assuming a generative process x = g(z), the authors study the effects of applying perturbations z'=z + delta leading to x'=g(z') and whether the delta can be recovered from those under what kind of conditions/assumptions.

In this work, the authors propose a hierarchy of more and more specific assumptions leading them to formalize how more specific structured can be inferred with different types of sparsity, i.e. identification up to affine transformations, blockwise identification, etc.

The paper then has 2 simple simulation examples in which they demonstrate that the theoretical properties hold.

Edit:
Increased score post rebuttal to 6.
strengths_and_weaknesses: Strengths:
-I really enjoyed the hierarchy of assumptions and how these permit making stronger statements quite precisely about recovery of latents under unknown arbitrary latent structure. This core part of the paper seems lucid, well explained, and interesting.
It is also surprisingly simple due to the simplified setup here.
-Another strength of the paper that stands out is that it is well positioned among the current literature, with precise statements about links to contemporary papers and the detailed conditions and assumptions they share or how they differ. 
-Overall the paper is also easy to read and intuitive, which is a positive result.

Weaknesses:
-There is some emphasis on literature from recent years where weakly supervised learning and disentanglement have been around a few more years in the ML community, here there is an implicit belief that after ICA the main new result is Locatello et al. What about older/other work on disentangled representations in both VAEs and RL? Is it not related just because the term causality is used less?
- The empirical setup is almost artificially toy. I have little doubt that the techniques described in this work would be applicable in various domains fitting the assumptions, but find that the experiments chosen are almost insultingly toy and hard to compare with other work.
We have a nice RL simulator from Stefan Bauer's recent work, that might be a useful testbed here in somewhat more ambitious environments. My key issue here is that the experiments are useful, but not really on the level of utility the cited work typically tackles.
- Given the simplicity of the experiments we also see that the encoder structure is simple and suffices here. I would be curious how work like this behaves under noise and uncertainty, the idealized scenario here is noiseless and assumes exact numbers of known latents. 
It would be necessary to start to go beyond that very narrow scenario to study such mechanisms for real world scenarios.


questions: -Can the authors make statements about robustness to uncertainty/noise even in their simplified setting with exactly known number of latents? Even if the theory can't keep up now, I would be curious to see how this shakes out empirically.
- Are there any more 'appealing' experiments the authors could share so we see a less artificial example where this may work?
limitations: no concerns
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose a theoretical framework on representation learning. Different from prior approaches, the authors show a different assumption and theoretically prove that sparse perturbation on latent variables is sufficient to learn the latents that follow any continuous distribution under weak supervision. The authors further discuss more general cases, and propose an estimation procedure based on the theory. Experimental results on two datasets are shown to support the claims in theory.
strengths_and_weaknesses: $\textbf{originality}$ 

This manuscript provides a novel theoretical framework for representation learning which is first used in this area to the best of my knowledge. The related works are adequately cited.

$\textbf{quality}$ 

The authors seem to provide sound theoretical justifcation for the proposed framework. And shows experimental results to support the claim.  

$\textbf{clarity}$ 

The paper is overall clear in presenting intuition and experimental results. The central thoery part is difficult to follow. The authors also adequately discuss the limitation of the proposed methods, and point out future direction for improvement.

$\textbf{significance}$

Lack of applications in real-world. The significance of the proposed method requires further illustration as it might be difficult to find a real-world application given the current assumptions. The applications meet the assumptions might be very specific which undermines the significance of this manuscript.

Overall, I think the paper in the current form might be interesting to the NeurIPS community.
questions: Given the current assumption, how would your method be applied in real-world applications?

For the experiment of image-base experiments, could you consider other baseline representation learning methods as comparison?

limitations: The authors adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper extends the literature on using sparse latent perturbations for identifiable representation learning. It assumes that there is a fixed set of perturbations, each of which only affects a single latent or a block of latents, in which case each block or the smallest overlap between two blocks can be identified up to affine transformations. In contrast to previous work, no independence assumptions on the latent distributions are necessary and the assumptions are a bit closer to reinforcement learning settings.
strengths_and_weaknesses: The paper adds an interesting angle that is amenable to situations in which an agent controls its environment through discrete and finite actions. The manuscript is very well written, the definitions and proofs seem solid and the experiments are mostly sufficient to verify the claims (with one exception, see below). At the same time, some of the assumptions seem quite unrealistic, in particular that all perturbations need to be present for each sample, and that the set of perturbations are completely fixed. Nonetheless, the direction is interesting, in particular because it relaxes the strong assumptions on the marginal that so far had to be in place.
questions: * In one experiment you are selecting the perturbations randomly. What happens if you also perturb the scale of each action, so if each delta_k is scaled with a gamma that is drawn from a narrow Gaussian distribution around 1?
* At least intuitively the assumptions seem quite strong. Do you have an intuition for what assumptions could potentially be relaxed in future work?
* In the image-experiment, why do the latent factors change for all objects upon translation of one? This sounds counter-intuitive and not really in line with the training objective. Also, if this effect is robustly present, it seems to contradict the theory.
limitations: I don't see negative societal implications.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper shows that if given data generated with block-wise (i.e. sparse) perturbations on the latent, then the latents are identifiable up to affine transformation on the blocks.
- In particular, in the special case of block size = 1, i.e. the perturbations change 1 coordinate at a time, then the latent is identifiable up to coordinate-wise permutation and scaling.
- For general blocks, the latent is identifiable up to affine transformations on the set of intersections and differences of the blocks (for non-overlapping blocks, this set contains the blocks themselves).

The paper also presents empirical results on synthetic data to corroborate the theory.

strengths_and_weaknesses: Strengths
- The paper discusses the related work well.
- Assumption 6 (i.e. knowing the group id) is unrealistic but is relaxed in the appendix.
- The paper is well structured and easy to follow.

I have no major complaints about the paper, but I'm concerned about the novelty and significance and hope to hear the authors' comment on this.
- Novelty: It's true that this perturbation setup is different from prior work which rely on auxiliary variables, mechanisms, or assumptions on the structure of the latent. However, I find the idea of coordinate/block-wise perturbation conceptually very similar to the idea of block-wise augmentation (i.e. only act on "style") in Von Kügelgen et. al. 21.
- Significance / applicability of the results: The identifiability results requires the learner to know the type of perturbation, which is often unrealistic.
- I find using the word "sparsity" misleading, since the identifiability results are more about being block-wise, regardless of the size of the block (i.e. sparsity). Perhaps rephrasing it could make the main idea clearer.

=== Update === The authors have addressed my concerns in their response.
questions: Numbers in Table 1-3 need more contexts: how to interpret an imperfect MCC value, e.g. 0.99 vs 0.95 vs 0.85?

Side note: Perhaps considering having the same numbering for theorems in the main paper and the appendix for easier reference.

=== Update === The authors have addressed my concerns in their response.
limitations: The paper discusses limitations and future directions.

There is no direct societal impact of the work.
review_title: null

Meta-review: metareview: This submission theoretically analyzes when it is possible to identify a latent representation, up to various classes of symmetries, under the assumption that one has access to observations corresponding to sparse changes to the latent variables. This is primarily a theoretical paper, with a small amount of empirical validation. Overall, reviewers found the paper interesting and appreciated the granularity of the conclusions. They had concerns about the relationship to some of the more traditional prior work in the area as well as the realism of some of the assumptions, but subsequent discussion resolved the reviewers’ concerns. All reviewers now favor acceptance, so I believe this paper should be accepted.
confidence: Less certain
Decision: Accept