Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: The authors introduced the VIM for learning latent representations of objects and their transition function overtime from video. I find the paper clearly written and the experiments well executed, however, I think if the results can be put in context of the performance of other method, it would be more illustrative
Main review: The reviewer have elementary familiarity with video prediction (graduate level textbook knowledge) and is fairly familiar with unsupervised representation learning.

Detailed Comments:
1)	Modules: I might have missed it, but what is the functional form of the modules (f_\theta)? In the experiments? And what are the common/possible choices in general?

2)	With regard to interpretability, I feel like many alternative ways can achieve similar level of interpretability (e.g. object tracking + unsupervised learning over the trace), so it might be useful to establish some benchmark or show that the module’s abstraction is better than some alternative.

3)	Dynamic Transfer: I feel like this is the most important results since this illustrate the benefit of disentangle the underlying dynamic rules of the generation process. I think it would be useful to compare VIM with prior methods such as SCOFF and GSWM.
review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper proposes Variational Independent Mechanism, a generative model trained to uncover a latent set of objects and disentangled mechanisms that act on these objects. It proposes a generative extension to previous works learning independent mechanisms and tests it on a toy 2D video setup.
Main review: Strengths:
The paper exploits ideas of sparsity and compositionality in latent space to recover mechanisms explaining the data that generalizes out-of-distribution. 
The paper builds on previous work learning independent mechanisms and makes a variational version of them that appears meaningful. The initial experimental results reported are interesting and promising (but they could be more convincing, see below).

Weaknesses:
The ideas motivating the work and the principles behind the architecture are not particularly novel and follow closely previous related works: in particular RIM (Goyal 2019) and SCOFF (Goyal 2020).
The main difference with previous works is that the current paper proposes a generative model. 
What problem does the generative VIM solve that was out-of-reach for RIM-kind of approaches?

This justification appears important because this paper focuses on toy 2D scenarios whereas SCOFF tackles more challenging evaluation setups. Also, this paper focuses on "unary" modules, whereas it seems that SCOFF can deal with more arbitrary relationships between modules and objects.

Thus, it seems that the paper should better justify the advantages brought by the variational mechanisms compared to these previous works. It would be interesting to also discuss (and report) what RIM and/or SCOFF would produce on the same 2D toy setup considered in this work.

The experiments are limited to a 2D setup with moving balls. 
Furthermore, there is no real comparison to baselines and for each experiment, only one run is reported. There is no measure of performance across many test runs. Only Figures 4 and 5 have some notion of global performance, but there is no error bound and we don't know on how many runs this was computed.
In general, the reporting of experimental results could be more thorough and convincing. In particular, 5.2 has only a very shallow description of results.

Minor comments:
If only up/down/left/right are modeled, can the modules encode notions of velocity? Velocities would seem quite important to track objects moving in time but are more abstract than up/down/left/right.

In equation (2), shouldn't it be p(z_0|c) instead of p(z_0^k|c)?
The paragraph "module selection posterior" overflow in the margin
The experiments report results when varying the number of modules, what about varying the number of slots?
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper proposes an extension of current object-centric representation learning method to group transition dynamics per object in a video into multiple independent modules. The model is trained in a generative fashion where the changes in latent space over time are parameterized by those modules.
Main review: The research question that this paper tries to solve is highly relevant for the research field of object-centric representation learning and bringing it closer to the causality domain. The initial idea of the paper is to combine an existing architecture, SCOFF (Goyal et al., 2020), with object-centric video representation learning. However, the model section contains several unclarities which makes it hard to fully grasp the proposed method:
* The latent update prior is said to model the distribution $p(z^{k}\_{t+1}|f(z^{k}\_{t},r^{k}\_{t+1}))$. However, $z^{k}\_{t+1}$ is defined as the output of a Layer Normalization later, with some noise added to the input of the normalization layer. How does the distribution $p(z^{k}\_{t+1}|f(z^{k}\_{t},r^{k}\_{t+1}))$ actually look like then? Why is the noise sampled instead of modeling an actual distribution? Is the distribution not needed in the Gaussian KL term?
* How is the distribution $q(r|x)$ in Equation 5 defined? In Section 3.2, a module selection posterior is mentioned that is of the form $q(r\_t|z\_{t-1},x)$. Is $q(r|x)$ marginalized out over $z$? Or is it a different network?
* The model selection prior $p(r_{t+1}|z_t)$ seems to be independent of previous module selections $r\_{\<t}$. Why is that? Is the module newly chosen at every time step, and not consistent across the time series? 
* Figure 1 shows that $p(x|z)$ is parameterized as a mixture of Gaussians, but images are naturally discrete and/or have a finite set of possible pixel values. Is a mixture of discrete logistics used as replacement, as done in e.g. Salimans et al. (2017)? Or how is a degenerated solution prevented?
* The module selection resembles the mixture of expert setup used in other network architectures, like Fedus et al. (2021). However, the drawback of the proposed method is that all modules need to be run on the all inputs, even during inference, instead of only running the selected modules. Why is that required? Why isn't a learnable key for each module used without the need of running each module during inference?
* In Section 5.1, paragraph Setup, it is mentioned that only during testing, the selection variable $r$ is categorical. Does this mean that during training, $r$ is a one-hot vector with values between 0 and 1, or is it still forced to be a one-hot vector and uses the straight-through gradient estimator?
Considering that no code was submitted, it would be hard to reproduce this architecture.

Another unclarity of this paper is with respect to the motivated inductive bias. In the introduction, the setup of the latent space is stated as the inductive bias. However, the inductive bias actually comes from the fact that the decoder is limited in expressiveness, such that sticking with a single module cannot achieve the same performance. Thus, the proposed model is limited to the setting where given the module, the output distribution is known/simple (e.g. a Gaussian). If more complex distributions would be possible in the output space (e.g. a mixture of logistics), the proposed model would not work anymore. Further, if the input image $x_{t-1}$ would already give clues about the module to use, it can collapse again since $z_{t-1}$ would be sufficient to predict $z_{t}$. Similarly, multiple objects with different appearances and different sets of actions could use the same modules for different actions, degrading the interpretability aspect. Thus, the use cases of this model seem very limited.

The experiments are focused on a single dataset which was created for the purpose of this paper. While it showcases the working of the model, it is very simplistic. The fact that most modules are correctly selected in only 4 out of 5 cases on average (i.e. 80%) is not entirely convincing, considering the simplicity of the dataset. A more realistic dataset should be used to showcase this, although it might be hard to find one considering the drawbacks mentioned above.

In conclusion, while I could see the paper being a valid, small contribution to the object-centric representation learning research field, the current form of the paper contains too many unclarities to fully understand the method, and the use cases seem limited. 

#### Minor comments
* Figure 1: the colors in the legend for inference vs generation class with the chosen slot colors in the picture right below. It suggests that $z_1$ belong to inference, and $z_2$ to generation.
* The citations should be checked for using the correct journal/conference (e.g. Locatello et al. 2020 was published in NeurIPS)

#### References
* Fedus, William, Barret Zoph, and Noam Shazeer. "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity." arXiv preprint arXiv:2101.03961 (2021).
* Salimans, Tim, et al. “PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications.” arXiv preprint arXiv:1701.05517 (2017).
* Goyal, Anirudh, et al. "Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems." arXiv preprint arXiv:2006.16225 (2020).

__Post-rebuttal changes__
The rebuttal of the authors clarified several aspects of the paper. Under the assumption that these clarifications will be included in the final version of the paper, I have updated my score.
review_title: null

Meta-review: This paper is above the bar for acceptance. The authors rebuttal included new empirical studies that allayed the initial concerns.


Decision: Accept (Poster)