Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method for imagination (data augmentation) to improve the generalisation of abstract reasoning approaches. The general module is based on adapting slot-attention to work over a sequence of to select the latent function to apply. The augmentation works using Gumbel soft max trick to select the augmentation. The authors additionally propose a new dataset of mathematical problems with values, colours and operations to be resolved. They compare to the prior approach on NPS overfitting to training but improving generalisation on test.
strength_and_weaknesses: Strengths:
- The proposal of the imagination framework is interesting working within the latent space of the model, which should provide increased generalisation without being limited to generation.
- The approach is fully differentiable that is and advantage against some prior methods for exutors.
- The dataset can be useful if well-defined against prior datasets and models
Weaknesses:
- The augmentation strategy can be broadly compatible with data augmentation strategies which don't seem to have been considered in related work or to inspire the approach. 
- The data augmentation is loosely out of distribution, as the numerical dataset has a relatively limited scope with a simple range of numbers and operations. As the explicit function of the functions is unknown, and the logical operations can be deduced as a combinational operation that is unclear if possible in the leave-out strategy.
- It would have been interesting for the approach to be applied to standard Abstract reasoning datasets such as RAVEN. While they are different problems, they could have shown greater generalisation and provided insights into how the model is working. 
- Similar to the above, the dataset seems highly similar to RAVEN and others, just numbers instead of shapes. It isn't clear where this dataset contributes.
clarity,_quality,_novelty_and_reproducibility: The paper is generally well written, however, the notation in Eq.3 and 4 should probably be revised to include T-1 to avoid confusion about the current and prior steps in RNN/GRU.


summary_of_the_review: While the paper works on a challenging problem and shows generalisation across unseen problems based on their proposed baselines. The lack of rigorous evaluation and benchmarks from similar problems means the paper feels incomplete. Additional experiments or justification for their lack of inclusion needs to be provided to position this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on the compositional generalization of visual abstract reasoning, i.e., the  ability  to  compose learned  concepts  in  novel  ways  for new situations. The authors propose an imagination-based learning framework applied on object-centric representations. By training on the imagined tasks, the model is expected to obtain better systematic generalization at test time. 

To evaluate the proposed framework, the authors synthesize a new visual arithmetic reasoning benchmark. Specifically, each example in the benchmark contains a support set of 10 input-output pairs and the outputs of these 10 pairs are obtained by applying a same latent program to the corresponding inputs.  Based on the given support set, the model is required to predict the output of a query input. The inputs are images with three colored MNIST digits placed at three different positions and the colors represent the signs of the digits. The program applied to the inputs is a sequence of arithmetic operations (i.e., addition and subtraction) in a particular positional order. There are three splits in the benchmark to evaluate generalization to new sequences of operations, position orders, and digit-Color configurations.

The used model to solve the proposed benchmark consists of a controller, which generates a latent task embedding based on the given support set, and an executor, which generates a neural program based on the task embeding and then apply the generated neural program to predict the output of the query input. The proposed imagination learning works by (1) randomly sampling neural programs, (2) applying the sampled neural programs to the inputs of original samples to generate new samples, and (3) training the model on the new samples. Experiments show that the imagination learning can improve the generalization to new sequences of operations.





 
strength_and_weaknesses: ## Strengths
The proposed imagination-based learning is interesting and able to improve the systematic generalization over a certain apsect (i.e., sequence of operations)

## Weaknesses

1. It is unclear how to randomly sample a new program, i.e., from what probability distributions the gates, the condition vectors, and the processing modules are sampled.

2. The authors claim the proposed imagination method as "Object-centric Compositional Imagination". However, it seems that the proposed imagination methond (described in Section 4.3) can also be used to non-object-centric representations, e.g., grid representations of images. The authors do not conduct any experiment using non-object-centric features.

3. The proposed visual arithmetic reasoning dataset is similar to RAVEN-style datasets [1], i.e., few-shot learning a rule from a support set and applying it to a new input. The RAVEN-style datasets have been extensively studied by previous works. It is unclear why it is necessary to synthesize such a new dataset. Besides, the authors only conduct experiments on the proposed dataset and do not compare with related works in RAVEN.

[1] Zhang, Chi, et al. "Raven: A dataset for relational and analogical visual reasoning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

4. Figure 3 is the only experimental result presented in this paper and from this figure, we can see that the proposed method has a much larger variance than baselines. There is not any explanation about this phenomenon in the paper.

clarity,_quality,_novelty_and_reproducibility: The proposed method is novel, but the experimental evaluation is insufficient to support the claims.
summary_of_the_review: Although the proposed method is novel, the experimental evaluation is insufficient. Therefore, I recommend rejection.

-------post-rebuttal-----
Thank the authors for the response. The response solves part of my concerns. Still, the main concern about the insufficient experiments is not resolved. Therefore, I will keep my rating. Hope the authors could further improve their work in the future.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a neural program-based method for an abstract reasoning (ARC from Chollet 2019) style task. The model contains an object-centric representation layer (based on slot attention I believe), a controller (encoding the examples and predicting the program), and an executor (which takes the program representation as input and predicts the final answer). The claimed contributions are two-folds. First, the program representation (which is a sequence of module indices, and a sequence of control parameters). Second, the training paradigm, which includes a "dream" stage.
strength_and_weaknesses: The paper presents an approach for neural-program-based approach for solving ARC tasks. The proposed approach is well-motivated, and shows improvements in experiments. My main concerns about the paper are the following three.

First, I think the paper writing can be significantly improved. The major issues are: 
1. Figure 2 is not an overview of the model. For example, it does not contain the object perception module (which the authors claim is an object they want to study in their contribution #3)
2. Section 4 does not contain a full description of the proposed method. I have to look at Algo 3 and based on the names to guess what's used for object perception.
3. The current writing of the paper is very unclear about the contribution of the authors. For example, my understanding that 4.1 and 4.2 are not the contribution of the author (See section 5.1: Our model OCIM can be seen as an implementation of the sparse interaction inductive biases proposed in Neural Production Systems (NPS) Goyal et al. (2021a), but augmented with an imagination-based learning mechanism.)

Second, given that the neural programming model is not the contribution of the paper, my understanding is that the paper focuses on "imagination." While overall I very much like the idea of using trained models to generate new data, and the authors have found an interesting domain to execute the idea, I am a bit worried about the generality of the proposed method.

Specifically, it seems that the dataset is constrained to a "sequential program" structure. And furthermore, almost all possible combinations of learned "modules" can be executed on this dataset (e.g., different orders, different visual positions). However, this is generally not true for many domains. There are at least two cases.

First, consider we want to use a program to query "the color of digit 1." The program assumes the existence of a digit 1 in the data, which, if not satisfied, will lead to unexpected execution results. I can easily imagine that adding those data into the training set will hurt model performance. I think the authors should at least talk about these assumptions about their domain, and discuss why their methods can work in the specific domain. That being said, I think the proposed method is too constrained.

Second, when these modules have dependencies among each other, it is unclear how we can sample programs that are "valid."

Third, the experimental results of this paper are a bit weak, due to the limitation to one dataset curated by the authors themself. Due to the limitations, I described above, I think it is important to extend the results to other domains, for example, other ARC tasks, or tasks such as visual question answering etc.

Other than these limitations, I think the authors should also cite more related works.

For example, the authors have clearly missed several important works in neural programming, such as
- Neural Programmer-Interpreters, https://arxiv.org/abs/1511.06279
- Making Neural Programming Architectures Generalize via Recursion https://arxiv.org/abs/1704.06611
- Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning http://proceedings.mlr.press/v119/li20f/li20f.pdf

The contribution of "disentanglement of object properties in object-centric representation learning" is closely related to many visual representation work. Here are some pointers:
- https://cs.stanford.edu/people/jcjohns/clevr/ Many works have worked on the CLEVR-CoGenT split, which is exactly the setup you have been studying.
- ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. https://arxiv.org/abs/1811.12231
clarity,_quality,_novelty_and_reproducibility: See my weakness comment #1 for my suggestions on paper clarity.
summary_of_the_review: I like the idea of the paper on dreaming with programmatically structured neural networks for generalization. However, the results of the paper are limited. The paper writing can still be improved. Overall my recommendation is WR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes Object-centric Compositional Imagination (OCIM) for visual reasoning tasks. It decomposes the tasks into a series of primitives applied to objects and recomposes these primitives to imaginary tasks. The model learns to reuse the previously-learned concepts to systematically generalize at test time by training on these imagined tasks. The model shows better generalization ability than the other baselines on a synthetic visual arithmetic reasoning benchmark the authors newly proposed.
strength_and_weaknesses: Pros:
1. This work proposes a new synthetic visual reasoning benchmark that provides access to operations, operation order, objects primitives.
2. The novel OCIM model presents better generalization ability over other baselines on the new benchmark.

Cons:
1. The writing needs to be improved, detailed in the clarity part.
2. Lacking ablation studies make it hard to estimate the importance and functionality of each module. May consider the following (you may put details in the appendix and state the ablation study results briefly in the main text if space is not allowed):
* What if $\beta$ (imagination coefficient) is set to zero (disabling the imagination)?
* How important are the warmup epochs?
* Can you modify the max number of steps $T$? $T=2$ seems a very specific design for the benchmark task.
3. The evaluations are only done on a self-proposed dataset that provides access to primitives. It is unclear how this method can be generalized to other more challenging benchmarks or benchmarks that do not have access to these primitives.
4. The overall motivation is clearly stated. However, the motivations behind the specific design of the OCIM model need more explanation.
clarity,_quality,_novelty_and_reproducibility: The main method part (Sec 4) is hard to follow. I would suggest having a running example that helps to explain what each step is doing. Also, the captions of the figures need to be improved by including more explanations to help readers understand better.

Besides, I could not find concrete numerical values of the results in Figures 3 and 4, not even in the appendix. Lacking concrete numbers makes readers hard to judge the extent of the improvement and reproduce the results. It would also benefit the reproducibility if the training curves could be provided (maybe in the appendix).

The idea of object-centric imagination is novel, but not well-supported by experiments.
summary_of_the_review: It is an interesting and novel idea to utilize the object-centric imagination method to improve the OOD generalization of visual reasoning tasks. However, the paper's writing and supporting experiments need to be improved with more effort.

---
Update (2022/11/22): I would keep my score before new results provide.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes Object-centric Compositional Imagination (OCIM) to improve the generalization of visual abstract reasoning. Specifically, it's about "imagination" and the ability to compose concepts to address novel situations. By training on "imagined" data, the premise is that better generalization could be seen at test time. The neural program-like method has an object-centric representation layer, a controller and an executor. It decomposes the tasks into a series of primitives, then composes these to new imaginary tasks. Specifically, it randomly samples neural programs, then applies the programs to the original data in order to create new data, and then train on the new data. In terms of results, performance of the model is better than baselines on a new synthetic benchmark proposed in this paper also.


-- STRENGTHS --

The proposed framework is novel and interesting, and is able to improve generalization (compared to other baselines) on unseen problems, to a certain extent.


-- WEAKNESSES --

There is unanimous agreement that the paper fares poorly in terms of rigorous evaluation. Specific pieces of evidence to support this are:

1) Evaluation is performed only on a dataset created by the authors themselves, without any evaluation on related datasets or tasks (e.g. RAVEN, ARC, etc.)

2) Limited amount of detailed evaluation, such as lack of ablation studies, makes it difficult to understand the characteristics and limitations of the model overall, as well as the contributions of various design decisions.

3) The author-proposed dataset provides access to primitives, which raises questions about whether the proposed method can generalize well to other datasets that do not do so.

The authors have acknowledged these concerns and have not rebutted per se. Their responses have been along the lines of "in order to address the concern about the generality of the proposed we will increase the diversity of tasks and query that can be made and composed in our benchmark to make it more general."

In light of these, it is clear that the paper should not be accepted in its current form.

justification_for_why_not_higher_score: Unanimous agreement that the paper has not performed sufficiently rigorous evaluation, an issue acknowledged by the authors themselves.
justification_for_why_not_lower_score: n/a
Decision: Reject