Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a model for compositional law parsing that is capable to accurately represent different "laws" of semantic concepts. The laws are built with random functions, and the proposed system can combine these functions to achieve flexibility and generality in different law parsing tasks.

The model uses an encoder-decoder architecture that represents concepts as latent variables, and laws are operationalized as concept-specific functions built with neural processes.

The authors claim that the proposed framework outperforms other methods in scene representation, intuitive physics and abstract visual reasoning tasks. 
strength_and_weaknesses: ## Strengths
- Powerful concept with several meaningful abstractions
- In-depth mathematical formulation explaining the formalities of the proposed framework
- Detailed experiments, including multiple different benchmarks with very different setups and objectives
- Very strong performance against baselines

## Weaknesses
- The abstract is not very good at exemplifying the proposed framework. More information, and possibly examples, describing what each law and concept can be would be great so that the concept becomes clear from the get-go.
- The explanation of the Neural Process in Section 3 is hard to understand. Clarifying that paragraph might be of great help to readers unfamiliar with NPs.
- General clarity. The paper is not easy to follow without substantial previous knowledge of the field, and it is hard to judge the 
- Baselines used: the set of baselines used is small, and an inclusion of other works could improve the experiments section. Importantly, comparing to task-specific networks might be interesting to observe how this general framework competes. 

clarity,_quality,_novelty_and_reproducibility: ## Clarity
The figures are clear and the concept of the paper is understandable. The presentation, however, could be improved: the abstract is not explicit enough to ground the main concepts, sections 3 and 4 are detailed but lack simpler explanations to tie together the main concepts.

## Quality
The quaility of the paper is high. The figures, mathematical formulations and experiments are laid out with care. The overall detail of the paper is high and the framework proposed appears to be carefully crafted and quite powerful.

## Novelty
The ideas appear to be generally novel.

## Reproducibility
The paper appears hard to reproduce from the manuscript alone. Great care is taken to ensure that the mathematical framework is explained in detail, but specific implementation procedures and explanations related to experimental details are lacking.
summary_of_the_review: The paper introduces a novel and powerful concept. The experiments are thorough and the mathematical formulation appears to be thoroughly supported. Its main weakness is tied to clarity: the manuscript assumes large amount of previous knowledge related to NP and other topics, and clarity could be improved to ensure more fluid reading. In general, the paper establishes a powerful framework with strong improved performance over baselines over several benchmarks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a latent variable model that parses dynamic scenes into independent laws represented as neural random functions. The model is trained to reconstruct the image sequence while keeping the distribution of concepts and random functions not changing over time and concepts. The evaluation on various datasets shows that it can predict the target images well. The qualitative examples show that the decomposed laws correlate with the changing visual concepts and can be swapped to generate novel samples.
strength_and_weaknesses: Strength
- This is an interesting paper that focuses on learning disentangled laws to interpret dynamic scenes.
- This paper also proposes a novel approach to learning concept-specific laws with Neural Proces. 
- The paper shows several qualitative examples to show how the results are interpretable and the composition of learned latent functions is meaningful.

Weaknesses
- It is unclear how the authors determined the number of laws for learning. What if we choose too many/little laws? Or how many laws we may need to interpret a realistic video?
- The current quantitative evaluation focuses on MLE and selection accuracy to show the improvement of semantic error but these two don’t really represent the model learns the right laws. The MLE can be a result of better visual encoding, the selection accuracy shows better representation. If the model learns compositional laws, it can generalize better to novel combinations of concepts that are not in training, e.g. new colors. The paper currently only shows one generalization example in the appendix. The evaluation will be much stronger to show a quantitative evaluation of generalization capability.

clarity,_quality,_novelty_and_reproducibility: The paper is well presented and the appendix includes details of datasets, hyperparameters, and network architecture. The author also submitted source code as supplemental material for reproducibility. 

The empirical evaluation is based on simple datasets. It will be beneficial to the community if the authors can discuss the limitation of the proposed model, e.g. what can be represented as compositional laws and what is needed if we apply this approach to more complex scenes.

summary_of_the_review: The paper proposes a novel approach to interpret dynamic scenes. It is well presented and easy to follow. It provides several examples of learned compositional laws. It will be much convincing if the authors can provide quantitative evaluation of compositional generalization.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper brings compositionality into the Neural Process framework. It decomposes the input into multiple concepts, each represented by a latent vector. A Neural Process is learned for each concept to capture its law, so that the concept can be generated at query points given a few context points. The full pipeline is trained end-to-end via variational inference without supervision. Experiments on three diverse datasets show that the proposed model can produce more accurate pixel predictions and more consistent latent space than NP and GQN. It is also qualitatively shown that one can compose different laws by combining the corresponding neural processes.
strength_and_weaknesses: - Strengths
    - The experiment results are quite extensive. The failure of NP and GQN is clearly shown. The law manipulation result (Figure 7) is very interesting and shows that the learned neural processes can indeed be composed.
    - The paper is well organized, and introduces sufficient background knowledge for readers to follow.
- Weaknesses
    - At a high level, the proposed method is similar to [ROOTS](https://www.jmlr.org/papers/v22/20-1176.html), which brings compositionality into GQN and focuses on compositional understanding of 3D scenes. Both of them can be thought of as a compositional NP framework. ROOTS has similarly shown better generation quality than GQN, and can compose novel scenes by combining the object-level GQNs. However, I think there is still merit in this paper, as it seems more general and not limited to 3D scenes. I encourage the authors to highlight their novelty compared to ROOTS.
    - The datasets look visually simple. Some compositional video scene understanding methods (e.g., Kosiorek et al., 2018; Jiang et al., 2019; Lin et al., 2020a as listed in related work) can deal with more complex scenes. Can you clarify why they are not considered baselines?
- Questions and Minor Issues
    - How many concepts did you use and what is the dimension of the latent vector for each concept?
    - The bouncing ball dataset is sometimes referred to as BoBa and sometimes BaBo.
clarity,_quality,_novelty_and_reproducibility: - Clarity
    - The paper is generally well written and easy to follow. The figures are of high quality.
- Quality
    - The evaluation makes sense, and there are some interesting results.
- Novelty
    - The novelty compared to ROOTS is not sufficiently discussed.
- Reproducibility
    - Dataset, architecture details and hyperparameters are provided in Appendix.
summary_of_the_review: I am leaning toward reject for the paper in its current form, mainly because the novelty needs more clarification.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a latent variable model to capture concept-specific laws from images/videos. The laws are encoded as Latent Random Functions (specifically, Neural Processes), and can be composed with one another. 
strength_and_weaknesses: Strengths:
- Multiple datasets with different focus
- Illustration of compositionalilty
- Demonstration of failure cases

Cons:
- Choice of baseline: neither NP nor GQN seem like strong, obvious baselines. Wouldn’t a system like [Aloe (Ding et al. 2020)](https://arxiv.org/abs/2012.08508) offer a better comparison?
- Based on Tables 3 and 4, I worry that the model needs careful tuning to work or several runs for a winner to emerge. The best hyperparameters vary significantly across datasets. 
clarity,_quality,_novelty_and_reproducibility: Overall the writing is easy to follow and the figures are carefully drawn. No specific concerns, but here are a couple of suggestions:
- __On choice of notation__: I found it confusing to think about concepts A and context C. It would be easier if concepts were denoted C, and if you used S for the source/context frames, used to predict targets T. 
- It would be useful to have a __summary table__ describing exactly what $x_t$ is for each dataset (e.g. timestep for BoBA, azimuth/altitute for MPI3D).

summary_of_the_review: The following questions will help me assess the work better:
- Is anything preventing you from __assessing__ the __disentanglement__ of the LRFs? It would be useful to have those metrics (e.g. see https://github.com/google-research/disentanglement_lib). 
- What would be __other potential realizations__ of CLAP which don’t use Neural Processes?
- __Future work__: how do you expect LRFs would perform on the [Abstraction and Reasoning Challenge](https://www.kaggle.com/c/abstraction-and-reasoning-challenge)? Would that be an ideal target domain?
- __How to generate future predictions__: You could generate an extended rollout (e.g. on BoBa) by applying the model autoregressively on the most recent generated frames. For example: to generate 20 target frames after 5 context frames, you could first generate frames 6-10, then use those to generate 11-15, etc. Alternatively, you could evaluate one-shot using a \eta=20/25. Which one do you expect would work better? The former strategy would allow you to re-infer the concepts and global latent at each application.
- On specific figures:
  - Figure 7: The newly generated example on CRPM-T doesn’t have the same color law as matrix 3 right? I expected the triangles in the top row to be dark and identically colored. Rather, the triangles in the top row seem to be following a pattern that is different from all other top rows (from the context image). How is that possible?
  - Figures 4 and 5: could you confirm you’re using a single model on each dataset (but multiple test configurations)?
  - Figure 14: do you learn any latents on CRPM-DT which change the size/rotation of the outer triangle?
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Paper proposes a novel method to decompose a scene into representation of concepts such as object appearance, background, and angle of object rotation. This is done through an encoding-decoding architecture which represents the concepts as latent variables. These concepts are then recomposite via a decoder network to generate the target images.

Paper further proposes to use Neural Processes (CLAP-NP) to instantiated each learnt concepts for inference purposes. Different experiments were ran to validate the model performance against baseline models for the intuitive physics, abstract visual reasoning and scene representation tasks. The proposed method significantly overperforms all baseline methods with the MSE and new derived metric, Selection Accuracy (SA).

Finally, paper also shows that the method can be used to edit the individual concepts to generate new samples.
strength_and_weaknesses: Strengths
1. Paper is well-written and well-motivated.
2. The concepts are clearly explained with formulas and diagrams
3. Claims are sufficiently supported by experiments.

Weaknesses
1. Paper introduces several concepts which are not mainstream. However, this is rectified by providing the reader with references and extensive Appendix material.
clarity,_quality,_novelty_and_reproducibility: 1. Paper is clear and concise.
2. High quality with good references and formulas.
3. Work seems highly original. Although some parts are drawn from previous work.
summary_of_the_review: Overall, this work seems excellent and of high impact. However, due to my unfamiliar with the technique described, it is difficult for me to review this work properly.

The experimental results are convincing. But I note that there is no comparison with any SOTA methods for tasks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: Learn disentangled laws, rules that a domain observes, to reason about that domain in a compositional manner.
Strengths: The method is convincingly demonstrated across numerous domains. Compositionality is of wide interest to the community and this provides a novel way to encode and exploit it. The idea of learning generic laws rather than focusing on objects is important.
Weaknesses: Few baselines are compared against. Although reviewers could not point to a clear missing candidate method. Authors did add a GP-based method to alleviate this concern. The method is only applied to simple datasets and the current scaling properties of the method don't provide an obvious path forward. Reviewers found the background needed to understand the method was partially missing, this could be addressed in an expanded Appendix.

justification_for_why_not_higher_score: While compositionality is of wide interest, the current method doesn't scale well. Future variants of the method, if it can be scaled up and applied to more realistic problems, would merit far more exposure.
justification_for_why_not_lower_score: The method is novel, the problem addressed is interesting and of interest to the community. It is a step forward and one that addresses compositionality in a generic way across many different problems.
Decision: Accept: poster