Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: 
This paper identifies that traditional datasets used for learning disentangled representation have several shortcomings such as no correlation between variables and simple structure.
It proposes a new dataset that has 1M higher-resolution simulated images along with 1K annotated real-world images of the same setup and gives analysis on disentangled representations on the dataset. 
Its results suggest that disentangled representations can result in better out of distribution task performances. 

==================================

Strength:
- Identified weaknesses of the previous datasets and proposed a new dataset that exhibits correlations between different variables. This is an important aspect of real-world scenarios. 
- Provided thorough experiments on disentangled representations and their metrics on the proposed dataset. 

Weakeness:
- Experimental results for showing more disentangled representation results in better OOD task performances is somewhat expected. Unlike the title, it is not clear if the paper shows sufficient transfer of disentangled representations in realistic settings. It would be great to propose a way to make the generalization better for the settings the models have trouble with (OOD2 generalization). 
- This paper tried one approach by adding noise during training, but it leads to my second concern that the real world observation is very similar to the simulated data. With some gaussian noise, they would look similar. Therefore, it might not be sufficient to show that we can use this approach for sim-to-real transfer. 
- The paper claims that the proposed dataset (which is interesting) is challenging and highly complex, but the rendered images look easy enough for a simple neural network model to learn to reconstruct. Datasets like CLEVR, although they might not be used for measuring disentanglement, seems more complex and exhibits occlusions.  

==================================

While the proposed dataset is interesting, I am not confident this paper is showing evidence for the usefulness of disentangled representations for transfer learning in realistic settings. It is also not clear if the dataset is more useful than others because of the reasons stated above. 


review_title: Official review

final_score: rating: 2: Strong rejection
review_content: The authors proposed a unique learning scheme for representation disentanglement. However, unlike infoGAN or ACGAN which explicitly learn disjoint feature representations for describing the attributes of interest (via unsupervised and supervised settings, respectively), the authors chose to address this task in a questionable "weakly supervised setting". More specifically, the authors chose to train AE-like model using pairwise images, which the difference between each pair of the inputs is only associated with one attribute of interest (e.g., angle, position, etc.). 

For some reasons, the authors expected such a training scheme would result in learning feature representation in which only "one" feature dimension would reflect such attribute differences. This is a very strong assumption, since it is very likely that more than one feature dimensions would correspond to such changes. 

Moreover, assuming that precisely one feature dimension would be associated with the attribute of interest by feeding in a pair of images with exactly this attribute change would not be practical either. Most real-world images would be complex and contain multiple attributes. Making this assumption would imply that the training images are not realistic. 

As for the evaluation, there is no comparison to any baseline or SOTA representation disentanglement methods, I found the quantitative metrics selected by the authors not sufficiently informative or supportive either. Most importantly, the authors claimed that the features trained by VAE allowed improved performances (e.g., Figs 3~5). Since VAE are simply trained in a unsupervised way (even the authors called their setting a weakly supervised setting), I see no evidence why the resulting features would be any different from those derived from standard VAEs, and why improved disentanglement results could be achieved.

Based on the above observations and remarks, I feel that the authors would not be able to deliver a work which is technically strong with sufficiently complete evaluation. Therefore, I do not think this paper is above the ICLR standard for acceptance.
review_title: Not technically sound and lack of sufficient evaluation

final_score: rating: 7: Good paper, accept
review_content: The work has sufficient quality, and is sufficiently clear and original. For detail, please see the below pros and cons. 

Overall Pros:
- Dataset contains dependencies in FoV, not artificially induced but present due to the attempt at realism in the simulation, a realistic aspect which hasn't been addressed in previous work. 
-The complete set of factors (e.g. product of number of possible values for each factor), totals to 2.92B, thus the dataset contribution allows for addressing the generalization problem which hasn't been addressed in previous work. 
-Disentanglement metrics notably cannot be evaluated on acquired camera images, so the recording of an annotated dataset under the same conditions in a real-world setup enables researchers to study this problem in a more controlled manner. 
-The observation that deeper and wider autoencoders can arguably scale to datasets with fine-grained factors is an informative observation for the community. 
-Similar to [1], the results collected in the extensive experimental study provide information to the community which can be analyzed and built upon in future work. 
-The OOD evaluation is novel, interesting, and informative. Though the downstream task is quite connected with the disentanglement evaluation itself, the consideration of this setting is a contribution to the community. 
-The negative observation in sim-to-real transfer is a net positive for this work, as it is informative to the community in specifying a problem which can be worked on in future research. 
-The observation that adding input noise in training, a common tool in classic sim-to-real robotics work, is useful for transfer in representations is practical and meaningful. 

Overall Cons:
- For the evaluation prior to OOD evaluation, it appears disentanglement is evaluated on the full collected dataset. If that is the case, though the dataset itself provides the ability to evaluate generalization, the evaluation prior to OOD evaluation does not evaluate generalization. It is not clear then if overfitting played a role in the observed results. 
- Fixing k=1 for the weakly supervised method implies control over the data generating process often unavailable in practical applications. At the same time, said experiment can be extended to test how performance degrades with a mismatch in assumptions. Considering k != 1, as well as PCL [2] and SlowVAE [3] (which assume laplacian transitions) could provide interesting comparisons. 
- The authors state that "many" trained models fully disentangle the factors, but this is "only" possible in the weakly supervised scenario. Clarifying details would be beneficial here, is it true that none of the unsupervised models fully disentangled the factors? If some did, was there an underlying correlation between successful models that can be discussed, or was it simply random? Is using latent traversals alone as a basis sufficient for claiming "full disentanglement"? More specific statements in the description would be desirable. 
- The authors appear to pre-assume that since latent traversals agree with the theory of [1], quantitative metrics which do not score weakly supervised models over unsupervised models are "ineffective at capturing disentanglement in this setting". This reads as confirmation bias and should clarified, either by justifying why metrics which do not agree with latent traversals are "ineffective", or presenting the information without implying judgement on whether the metrics are "ineffective" or not. 
- "We stress that the OOD2 scenario, which is typically not studied extensively, is only possible because the representations are not trained on the entire dataset." Does this mean the representations learned for OOD evaluation are trained on less data than what was considered for disentanglement evaluation? Clarifying details would be beneficial here, as this standalone statement simply yields unanswered questions. 
- How was D1 selected from D, by what sampling process, and how was the number of samples decided upon? For in-distribution generalization, how was the split between training and held-out set in D1 conducted, what was the percentage for the split, and how was it selected? Clarifying detail would be beneficial. 
- "Since the values are normalized, we can take the average of the MAE over all factors (except for the FoV which is OOD)." Why can you not take the average of the MAE over the FoV which are OOD? This statement could be interpreted as some factors being OOD while some not, when a split within the factor set was not mentioned previously. This sentence requires clarification. 
- Many questions brought up by Section 4 (such as the above), are addressed in part in the experimental setup section for Section 5. I'd suggest providing said information earlier so the confusion yielded in Section 4 is mitigated. 
- Notably, the color hues simply mentioned in Table 1, without any explanation on why they are not included within Table 1, appear to be what is ablated upon for OOD1. It would have been beneficial to have earlier clarification on this point. Furthermore, distribution shift in the color hue but all underlying factors being the same is specific type of distribution shift not clarified to the reader. Consideration of discrepancies in renderer would be interesting, but more importantly, discrepancies in the factor value set. The distribution shift considered here is shift in nuisance factors, a useful test but limited in its scope. 
- "Our results therefore suggest that highly disentangled representations are useful for generalising out-of-distribution as long as the encoder remains in-distribution" This statement can be seen as a bit misleading given the data is only out-of-distribution in terms of nuisance factors, not the factors the model was evaluated on disentanglement with respect to. It's intuitive that a highly disentangled representation will capture the ground truth factors well, and thus be more robust to shift in nuisance factors, and it is useful to show this, but the fact that we are looking at shift in nuisance factors should be clear to the reader.
- The justification for training half of the models with gaussian noise is provided in the final page of the paper. As with the previous comments, it would be beneficial to the reader to make this clear when introducing the experimental design decisions.  

Conclusion: This work provides significant contribution in experimental analysis on the performance of disentanglement with the inclusion of realistic complexities. Both the positive and negative observations are clear, interesting, and informative. While I suggested many ablations which could provide the reader with further information, I don't see the lack of inclusion of said ablations as an issue in the work, the experimental study is already extensive. The main issue I see with the work as it is merely comes from a writing standpoint, ensuring the statements made are justified, and ensuring the detail is there in the descriptions for the reader to understand, and most importantly, be able to reproduce, said results. 

Overall, I see this is a valuable contribution, but would like to see improvement in the writing given the points brought up in the revision. 

Questions:
- Can you describe how given a particular state for the factors of variation, how said factors are processed to render the observation? A description of how each factor individually influences the observation would be beneficial in the main paper, if only briefly, such that the reader can see why it is (1) challenging and (2) requires modeling of fine details. 
- "The training set for the VAEs contains 8 randomly chosen color hues." How do these color hues differ from the cube color hues? How do they specifically affect the rendering? Clarifying detail for this statement is needed. 
- Were any ablation studies performed for the model architecture? Many model details are presented without any experimental testing discussed, an ablation study would be informative to the reader. 
- Did you test the behavior of models with latent_dim < factor_dim, e.g. latent space dimensionality of 5, as well as latent_dim = factor_dim, e.g. latent space dimensionality of 7? Such results could be interesting to see if, and how much, performance degrades not only as latent_dim progressively increases from factor_dim, but also when latent_dim progressively decreases from factor_dim, considering the equality case as well. 
- " Finally, note that the BetaVAE and FactorVAE metrics are not straightforward to be evaluated on datasets
that do not contain all possible combinations of factor values." Could you detail why they are not straightforward to evaluate? Clarifying detail would be helpful. 
- Did the authors consider other unsupervised model selection techniques, such as UDR, to see if this under or outperformed the weakly supervised loss selection method?
- Why the choice of MAE? Was this choice ablated on?

[1]: Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Scholkopf, and Olivier ¨
Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, 2019.

[2]: Aapo Hyvärinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Proceedings
of Machine Learning Research, 2017. 

[3]: David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge,
and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding.
arXiv preprint arXiv:2007.10930, 2020.
review_title: Review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: ## Summary

The paper presents a new, more complex, dataset for the use of disentangled representation learning. The dataset is based on real and simulated images of the trifinger robot platform. There are 7 factors of variation with high-resolution measurements of these factors. The dataset contains over 1 million simulated images and another ~1000 annotated images of a real trifinger robot arm.

The authors also present a new neural architecture to scale disentanglement on more complex datasets and present a large empirical study on the performance of various techniques on out-of-distribution downstream task performance.


## Quality & Clarity
The paper itself is well written, with a structure that effectively guides the reader through the work and results.

The dataset, significance and experiments are clearly outlined.

## Originally & Significance

The novelty of the dataset is clear. Providing a complex disentanglement dataset where the underlying factors of variation are inherently correlated. The in- and out-of-distribution experiments are possible because of the presence of both synthetic and real-world data.

The experiments run are repeated multiple times and the results are convincing. They use both unsupervised and weakly supervised approaches and the results are both intuitive and supported by the literature. 

The experiments on out-of-distribution representation transfer are interesting and show that disentangled representations can lead to better transfer to out-of-distribution tasks.


## Outcome Rationale

This dataset is likely to be extremely useful to the community going forward and work disentangled representation learning is likely to benefit from it. The experimental setups are sensible and the largescale benchmarks support the use of disentangled representations when transferring from simulated to real-world scenarios.

review_title: The new standard dataset for complex disentanglement learning

Meta-review: This paper introduces a new dataset for evaluating disentanglement and its impact on out of distribution generalization based on the trifinger robotics platform. Using this dataset, the authors rigorously investigate the performance of beta-VAEs in this setting under a number of conditions, finding that weak supervision is necessary to induce disentangled representations, and that, perhaps surprisingly, disentanglement does not help for sim2real settings despite the similarity between the simulator and the real data. Reviewers were divided on the work, but had a number of concerns related to the claims of novel architecture, comparisons to baselines, and issues with the clarity of the paper, some of which were addressed in the authors' response. I agree with some of these concerns, particularly with respect to the claims of novel architectures since the modifications could simply be viewed as tweaking hyperparameters and are not rigorously compared to baselines. However, I think the novelty of the dataset and the rigorous evaluation of OOD generalization settings is likely to be valuable enough to the community to merit acceptance. I'd encourage the authors, however, to tone down some of the claims regarding the architecture (or provide sufficient baseline comparisons), and instead focus on the dataset and the OOD results. I recommend acceptance. 
Decision: Accept (Poster)