Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper analyses failure cases of deep disentanglement models. It focuses specifically on their ability to deal with combinatorial generalization, i.e. to disentangle at test time attributes that are presented in combinations that have not been observed in the training data.
The paper makes the distinction between encoder errors, corresponding to when the values in the latent variable do not reflect the actual values of the generative attributes, and decoder errors, when the latent code is correctly inferred but the decoder is not able to recombine the values correctly to build the output.

They find, as in previous work, that in the more challenging evaluation conditions, the models fail to disentangle.
They observe that the errors already appear in the latent code (encoder errors) and that some attributes (especially "shape") were more difficult to combine than others.
Those observations seem to carry across different settings: VAE and WAE backbones, with or without SBD in the architecture, and in both their unsupervised and supervised trainings.
strengths_and_weaknesses: Strengths:
- The paper tackles the challenging task of analyzing and studying a not well-understood behavior of generative models (disentanglement).
- The authors try a large combination of settings and make observations that seem to generalize across the settings.

Weaknesses:
- There seems to be no strong new idea, the model, architecture, training, metrics, and experimental setups are all taken from prior work. But the paper is framed as a deeper study of some preliminary observations made in previous works, so it's ok.
- The paper could be presented more clearly. Some important descriptions are ambiguous.
- The observations are interesting but the link between the evidence and the interpretations drawn from the authors seems very weak.
questions: - Regarding non-identifiability (2.1.1): The problem is of course well known and would deserve more attention, but I would point out that 1) in the commonly used settings, many models do seem to work nevertheless 2) it is unclear what in the proposed setup from Montero et al. is dedicated to solving this particular problem. Consequently, I find l.100 a little bit misleading, as it can imply that Motenro et al solve this problem in opposition to other methods that can't. Can the author provide some clarification about what is said in this paragraph?
- Semi-supervised or unsupervised: The setup described in 2.1.1. is referred to as semi-supervised. Indeed, it needs supervision in the sense that labels are necessary to be able to present the correct output given the three inputs. However, in the rest of the paper, there is no mention of the semi-supervised setting. Instead, it seems to be refered to as unsupervised. Moreover, in the supplementary, the authors are mentionning "weak supervision signal" to explain the same setting.
- Following the previous issue, if the presented setting is semi-supervised, then I think the paper also needs to add experiments in a fully unsupervised setting. Indeed, the studied methods derive from the unsupervised disentanglement literature and are aimed to address the specific challenges of this difficult setting. There is value in relaxing some constraints for a study, but it remains necessary to check if the findings hold in a fully unsupervised setting. If the authors wanted to fully address the supervised disentanglement method as well, they would have to encompass a much broader scope of related work, including ELEGANT for instance. Can the author clarify the positionning of the paper regarding supervision, as well as to attribute swapping approaches?
- The authors mention in supplementary that they were not able to train MPI3D with a VAE, which is a little bit surprising considering they can be trained on more complex data, and that other works report some success on this dataset with models of the same family. [2] Can the authors comment on what in there setting makes it impossible?
- In the Discussion section, the author attempt to explain their results with the concept of "interaction" that is not defined. They state that shape and color for instance are attributes that interact together, while color and position do not. But of course, it could be argued that position itself is composed of two variables that interact to produce the output. What about the orientation attribute, should it be considered as interacting with shape? with position? Can the authors explicit what they mean by interacting attributes? and provide some results with the other attributes to make the generalization somewhat more grounded? Currently, it only considers one attribute per dataset, which is not enough to draw any hypothesis. The discussion seems to be very weak to me.

Mainly, the writing of the paper lacks precision. 
I would consider raising the rating if the authors can address this issue by clarifying the mentioned points and showing convincingly that the camera-ready will be less ambiguous.

[1] ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributes. 
Taihong Xiao, Jiapeng Hong, Jinwen Ma. ECCV 2018.

[2] Weakly-Supervised Disentanglement Without Compromises.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, Michael Tschannen. ICML 2020
limitations: The authors do not adress the limitations. They should maybe mention the limited complexity of the data the presented models are able to fit, as it is a general limitation of the field.

---

Post-Rebuttal: My main concerns have been addressed in the revised submission. I now recommend acceptance, and accordingly, change my rating from 3 to 7.

---
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the generalization ability of disentangled representations through a set of empirical results.
Particularly, this paper studies:
- Do the models that disentangle well, fully capture the combinatorial structure of the world? and the answer is probably not!
- So does the encoder fail to correctly map the unseen test examples to latent space or the decoder is the problem? It appears that in cases that the model doesn't generalize, the encoder fails too. So Most probably the encoder's fault.
- Then why does the model learn to properly disentangle on the training set? By associating latent features with perceptual inputs rather than learning how these factors interact with each other.
- Is this problem limited to SSL tasks? No, even in supervised tasks we can see that the encoder fails to map properly to the latent space.
- Then how does it explain other works showing the good performance? They either vary a single factor of variation or factors that do not interact much in the input space.

strengths_and_weaknesses: **Strengths**
- The paper studies a relevant problem, interest of the community.
- It is very well structured. As I read through the paper, some questions raise in my mind and then I find the answer in some later paragraphs.
- The hypothesis, experiment, and then result and intuitions are separated in a clean way. Easy to follow!
- I also enjoyed reading section 4 and the discussion afterwards. Very well concludes the work.

**Weaknesses**
- I detected only some minor errors such as using "it's" instead of "its" and inconsistent citations.

**Disclaimer**
My filed is more generalization rather than disentanglement. Hence, I cannot fully evaluate the significance of the works. From a generalization point of view, I found this work eye-opening and beautifully done. On disentanglement, I am keen to read other reviews on this work.
questions: I would like to know authors' opinions on how one would go about solving the encoder's lack of generalizability?
It appears that the paper claims that the encoder somehow "overfit" on the training distribution. Or it rather undefits and fail to capture the interactions between the factors of variation?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies generalization failures of models with disentangled latent spaces to unseen combinations of factors. The authors present experiments aiming to determine whether the encoder or decoder, or other architectural aspects are causing these failures. To this end, the paper presents latent space visualizations and evaluations on the DCI disentanglement metric. Furthermore, the authors observe that the way the factors interact impacts whether they generalize in combination with other factors.
strengths_and_weaknesses: The paper addresses an important problem in disentanglement learning, namely that of combinatorial generalization. The paper confirms and extends observations from prior work. I see the following as strong points of the paper:
- The authors consider different model/architecture variants (VAE, WAE, spatial broadcasting generator) on top of the composition task from [25] and observe similar generalization failures for all models.
- The observation that factors which “do not interact” with others (e.g. floor color and wall color in 3DShapes) better generalize to unseen combinations than factors that interact (e.g. object position and shape) is interesting, although not extremely surprising.
- The observed failure of supervised models to generalize to unseen combinations is interesting.

I generally feel that the paper could provide deeper insights into why the observed failure modes occur and how they could be mitigated, specifically:
- The concept of encoder/decoder failure is interesting. However, I feel the paper does not provide a comprehensive investigation of this issue, since the encoder and decoder are trained jointly on data with missing factor combinations. What would the results look like for an encoder/decoder pretrained without missing combinations and then trained together with a randomly initialized decoder/encoder on data with missing combinations?
- What is the role of the prior? The fact that the training data does not contain all combinations of the factors should be reflected in the prior. In the current setup, it is possible that enforcing a jointly Gaussian prior could hurt combinatorial generalization.
- In the supervised setting it would be interesting to see whether rebalancing the training set such that the targets are jointly uniformly distributed improves combinatorial generalization.
- In most of the examples, for ordinal factors an interval of values was left out, including the minimum/maximum value. How does including the minimum and maximum values for ordinal factors affect combinatorial generalization? Does seeing the minimum/maximum value improve combinatorial generalization for combinations of ordinal factors (Section 3.2 shows that this is not the case for categorical factors).

In summary, I think the paper addresses an important problem and explores an interesting direction, but could provide more insights and potentially strategies to mitigate combinatorial generalization failures.


Typos:
- L162 achieve instead of access?
- L196 posX


_Post-rebuttal update_

With the investigations added in response to the review the failure cases are now characterized more in-depth and more comprehensively. I therefore adapted my score. I would suggest to summarize the experiments with rebalanced data, frozen decoder, and those suggested by reviewer HV8h in the main paper, with references to the appendix, to ensure that these aspects get noticed by readers who wish to build on top of this work.
questions: I don’t have any specific questions, but I would appreciate comments on the points raised above that I believe could provide deeper insights into the generalization failures observed in the paper.
limitations: I feel that the paper could provide deeper insights into why the observed failure modes occur, please see the “strength and weakness” section above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigates the combinatorial generalization problem in disentangled representation learning. The paper confirms the generalization failures found by previous works, and investigated more detail about the reasons for these failures by considering whether they are due to encoding failure or decoding failure. From experiments, the paper shows that the factor ‘shape’ is the hardest for the models to generalize. This paper concluded with a hypothesis that these failures may be caused by the lack of a causal model that can model the interaction between factors. 
strengths_and_weaknesses: Pros
1. The combinatorial generalization problem is a valuable problem to investigate.
2. The experiments presented in this paper are clear and well-conducted.
3. The visualization of the distribution between learned latent variables and generative factors (e.g. Fig 3 bottom) is beautiful and informative.

Cons
1. Only disentanglement models minimizing total correlation with continuous latent variables are considered. Other models should also be considered such as ControlVAE [1], CascadeVAE [2], and LieGroupVAE [3]. Particularly, as this paper shows that ‘shape’ is the most difficult factor to generalize while ‘shape’ is also a discrete factor, it is more valuable to investigate if models with discrete latent variables (e.g. JointVAE [4] and CascadeVAE [2]) can avoid this type of failure.
2. Paper is simple but somehow lacks novelty to me. The combinatorial generalization problem is known. This paper confirms this finding and shows the failures happen in encoding, but no much further insights can be drawn. This work could be more complete if it can show some potential evidence of solutions to this problem.
3. The paper concludes with a hypothesis that a model may be helpful if it can consider the interaction between factors. The LieGroupVAE [3] tried to model disentanglement from the perspective of matrix composition, which may be seen as an approximation of interaction between generative factors. It would be interesting if this paper can also investigate how this model performs on this combinatorial generalization problem.

[1] ControlVAE: Controllable Variational Autoencoder

[2] Learning Discrete and Continuous Factors of Data via Alternating Disentanglement

[3] Commutative Lie Group VAE for Disentanglement Learning

[4] Learning Disentangled Joint Continuous and Discrete Representations
questions: Some more investigation experiments on JointVAE, CascadeVAE, and LieGroupVAE would be great to address most of my questions mentioned in Weaknesses. 
limitations: Yes
review_title: null

Meta-review: metareview: All reviewers acknowledge that the paper conducts an extensive study of the combinatorial generalization of models, and the additional experiments on CascadeVAE and LieGroupVAE conducted by the authors convinced the reviewers. Despite the lack of a proposed solution, I believe this paper provides novel interesting insights on the failure modes of disentanglement models (for example, the fact that if the factor affect different part of the image combinatorial generalisation is much more likely to occur). In agreement with the reviewers, we recommend acceptance. 


confidence: Certain
Decision: Accept