OCIM : Object-centric Compositional Imagination for Visual Abstract Reasoning
Authors: Rim Assouel
Pau Rodriguez
Perouz Taslakian
David Vazquez
Yoshua Bengio
Venue: Submitted to ICLR 2023 (2023)
Keywords: objects
imagination
visual reasoning
representation learning
inductive biases
compositional generalization

Abstract: A  long-sought  property of machine  learning  systems is  the ability to compose learned concepts in novel ways that would enable them to make sense of new situations. Such capacity for imagination -- a core aspect of human intelligence -- is not yet attained  for machines. In this work, we show that object-centric inductive biases can be leveraged to derive an imagination-based learning framework that achieves compositional generalization on a series of tasks. Our method, denoted Object-centric Compositional IMagination (OCIM), decomposes visual reasoning tasks into a series of primitives applied to objects without using a domain-specific language. We show that these primitives can be recomposed to generate new imaginary tasks. By training on such imagined tasks, the model learns to reuse the previously-learned concepts to systematically generalize at test time. We test our model on a series of arithmetic tasks where the model has to infer the sequence of operations (programs) applied to a series of inputs. We find that imagination is key for the model to find the correct solution for unseen combinations of operations.

Under review as a conference paper at ICLR 2023
OCIM: OBJECT-CENTRIC COMPOSITIONAL IMAGINA- TION FOR VISUAL ABSTRACT REASONING
Anonymous authors Paper under double-blind review
ABSTRACT
A long-sought property of machine learning systems is the ability to compose learned concepts in novel ways that would enable them to make sense of new situations. Such capacity for imagination not yet attained for machines. In this work, we show that object-centric inductive biases can be leveraged to derive an imagination-baste based learning framework that achieves compositional generalization on a series of tasks. Our method, denoted Object-centric Compositional Imagination (OCIM), decomposes visual reasoning tasks into a series of primitives applied to objects without using a domain-specific language. We show that these primitives can be recomposed to generate new imaginary tasks. By training on such imagined tasks, the model learns to reuse the previously-eaamed concepts systematically generalize at test time. We test our model on series of arithmetic tasks where the model has to infer the sequence of operations (programs) applied a series of inputs. We find that imagination is key for the model to find the correct solution for unseen combination of operations.
core aspect of human intelligence - is
INTRODUCTION
Humans have the remarkable ability to adapt to new unseen environments with little experi- ence (Lake al. 2017). In contrast, machine learning systems are sensitive to distribution shifts (Ar- jovsky et al. [2019] Su et al. [ (Egsttrm et al. 2019). One of the key aspects that makes human learning so robust is the ability to produce or acquire new knowledge by composing few learned con- cepts in novel ways, an ability known as compositional generalization (Fodor and Pylyshyn. 1988
Lake et al., 2017). Although the question of how to achieve such compositional generalization in brains or machines is an active area of research (Ruis and Lake, 2022). promising hypothesis is that dreams are a crucial element (Hoel 2021) through the Overfitted Brain Hypothesis (OBH).
Both imagination and abstraction are core to human intelligence. Objects in particular are im- portant representation used by the human brain when applying analogical reasoning (Spelke) 2000). For instance,
infer the properties of new object by transferring properties from similar objects (Mitchell 2021). This realization has inspired a recent body of work that focuses learning models that discover objects in visual scene without supervision (Eslami et al. 6b Kosiorek et al.] [0018] Creff et al. 2017 van Steenkiste et al., 2018) Cireff et af. [2019] Burgess et 2019 van Steenkiste et al. 2019 Locatello al. 2020). Many of these works pro- pose several inductive biases that lead to visual scene decomposition in terms of its constituting objects. The expectation is that such an object-centric decomposition would lead better general- ization since it better represents the underlying structure of the physical world (Parascandolo et al.) 2018 To the best of our knowledge, the effect of object-centric representations for systematic generalization in visual reasoning tasks remains largely unexplored.
knowledge of these
While abstractions, like objects, allow for reasoning and planning beyond direct experience, novel configurations of experienced concepts are possible through imagination. HHoel (202T) goes even further and posits that dreaming, which is a form of imagination, improves the generalization and robustness of learned representations. Dreams do so by producing new perceptual events that are composed of concepts experienced/leareed during wake-time. These perceptual events can be de- scribed by two knowledge types (Goyal et al., 2020 2021b): the declarative knowledge encoding object states (e.g. entities that constitute the dreams), and the procedural knowledge encoding how they behave and interact with each other (e.g. how these entities are processed form the percep-

---

Under review as a conference paper at ICLR 2023
tual event). In this work, we take step towards showing how OBH can be implemented to derive new imagination-barn learning framework that allows for better compositional generalization like dreams do.
We thus propose OCIM, an example of how object-centric inductive biases can be exploited to derive imagination-based learning frameworks. More specifically, we model a perceptual event by its object-centric representations and a modular architecture that processes them to solve the task at hand. Similar to (Ellis al. 2021), we take the program-induction approach to reasoning. In order to solve a task, the model needs to (1) abstract the perceptual input in an object-centric manner (e.g. represent declarative knowledge), and (2) select the right arrangement of processing modules (which can be seen as neural program) that solves the task at hand. In order to generalize beyond direct experience through imagined scenarios, model would have to imagine both of these components (e.g. objects + how to process them). Here we restrict ourselves to imagining new ways to process experienced perceptual objects. We propose to do so by exploiting object-centric processing of inductive biases. The idea is to have neural program (Reed and De Freitas 2015 Cai et al. 2017: Li et al. 2020) composed of modular neural components that can be rearranged (e.g. "sampled"" through selection bottlenecks) to invent tasks. The capacity to generate unseen tasks enables OCIM to generalize systematically to never-seen sever--ee--before before tasks by (1) producing new imagined scenarios composed of learned/experieence concepts and (2) training the model on these imagined samples to predict back their constituting concepts (e.g. used modules that were sampled to produce them). Our contribution threefold:
We propose example of how object-centric inductive biases can be used to derive an imagination- learning framework. Specifically we show that rearranging modular parts of object-centric processing model to produce imagined sample and training the model to predict the arrangement that produced that sample helps with compositional generalization.
We propose visual abstract reasoning dataset to illustrate our imagination framework and evaluate the models along different axis of generalization.
We highlight some drawbacks of current state-of-the-ard (SOTA) object-centric perception model when it comes to disentangling independent factors of variation within single vi- sual object.
2 RELATED WORK
Object-centric Representation. A recent research direction explores unsupervised object-centric representation learning from visual inputs (Locatello al.) 2020 Buuggess all 2019 Greff al., 2019 Eslami et al. 6016a Crawford and Pineau| 2019 Sttlzner et al. 2019 Lin et al. ( Geirhos al. 2019). The main motivation behind this line of work is to disentangle latent representation in terms of objects composing the visual scene (e.g. slots). Recent approaches to slot-based repre- sentation learning focus on the generative abilities of the models; in our case, we study the impact of object-centric inductive biases on systematic generalization of the models in visual reasoning task. We observe that modularity of representations as important as the mechanisms that operate on them (Goyal et al.) 2020 202Tb). Additionally, we show that object-centric inductive biases of both representations and mechanisms allow us to derive an imagination framework that leads to better systematic generalization.
Modularity. Extensive work from the cognitive neuroscience literature (Baars 1997 Dehaene al., 2017) suggests that the human brain represents knowledge in modular way, with different parts (c.g. modules) interacting with a working memory bottleneck via attention mechanisms. Fol- lowing these observations, a line of work in machine learning Goyal and Bengio (200);; Goyal et al. (219 2020 2021b): |Ostapnkko et al. (202T): Goyal and Bengio (2022) has proposed to translate these characteristics into architectural inductive biases for deep neural networks. Recent approaches have explored architectures composed of set of independently parameterized modules that com- pete with each other to communicate and attend or process an input (Goyal et al., 2019 2020 2021b). Such architectures are inspired by the notion of independent mechanisms (Pearl 2009 Bengio et al., 2019; Goyal et al. 2019 Goyal and Bengio 2022). which suggests that a set of inde- pendently parameterized modules capturing causal mechanisms should remain robust to distribution
2

---

Under review as a conference paper at ICLR 2023
shifts caused by interventions, as adapting module should not require adapting the others. The hope is that -uttof--istiibution (OOD) generalization would be facilitated by making possible to sequentially compose the computations performed by these modules, whereby situations can be explained by novel combinations of existing concepts. In this work, we show how modular archi- tectural choices be exploited derive imagination-based learned paradigm that allows better compositional generalization; we do so by explicitly exposing the model to data samples composed of novel combination of learned concepts.
Imagination, Dreaming, and Generalization Dreams are a form of imagination that have in- spired a significant amount of influential work (Hinton et al., 2006 Ellis et al., 2021 Hafner al, 2019 2020 An interesting explanation for such phenomenon is the overfitted brain hypothesis (OBH) (Hoel 2021), which states that dreaming improves the generalization and robustness of learned representations. The idea is that, while dreaming, the brain recombines patterns seen during wake time. This results in artificial data augmentation in the form of dreams. This way, dreams regularize and prevent the brain from overfitting the patterns seen while being awake. In machine learning, data augmentation is long standing technique for tackling the data scarcity problem, whereby new training samples are generated from existing data in order diversify the trained mod- els. Various approaches for augmentation, such as GANs (dos Santos Tanaka and Aranha 2019). VAES (Chadebec and Allassonnière 2021). and diffusion models Ho et al. (2022) have proved to be effective In improving model accuracy and generalizability. As in the case of dreams for the human brain, the strategy of considering imagined data samples broadly compatible with data augmentation objectives. Dreamcoder (Ellis et al., (021) is recent example where training on imagined patterns improve generalization for program induction. Program induction is a challeng- ing problem because the search space is combinatorially large and new unseen programs have low likelihood. To address these challenges, Dreamcoder leverages a wake-sleep algorithm that reduces the search space by learning a domain-specifie language (DSL) while learning to search programs efficiently. During training. Dreamcoder undergoes a dreaming phase where the model learns to dur- solve tasks generated by sampling programs from DSL and applying them to inputs ing the wake phase. Although Dreamcoder is promising for program induction, the DSL a major roadblock solve open-ended visual reasoning tasks where the input consists of raw pixels rather than symbols. In this work, we overcome these challenges by relying on object-centric inductive biases, i.e., architectural choices that both represent and process objects, in order to learn a neural program library.
3
VISUAL ARITHMETIC REASONING DATASET
Operations sequence
Visual Ingot
Progisss to
Osrgut Programs applied to Input
infer
6
(-4)+(-2)-6
2
LRB
Suppet Sct
8
O
(tue)
8+(-4)-0
Positional Order
RBL
ood op hest
train
BLR
ood order trein
(K:4.22)
(2-)
(-(-9)-(-8)
LBR
ood order test
aod order test whichin
RLB
1
Qurry
BRL
answer
= loft. R right, B bottom
Figure 1: Data sample and dataset splits.
Most visual reasoning benchmarks revolve around variations of Raven's Progressive Matrices (RPM) (Jaess 1936 Zhang et al.) 2019: Barrett et al.] 2018 Hoshen and Werman 2017) which are discriminative tasks in which the solver chooses from a set candidate answers. However, in recent survey, Mitchell (2021) recommends evaluating models on generative tasks that focus
3

---

Under review as a conference paper at ICLR 2023
human core knowledge (Spelke 2000). Models trained generative tasks are indeeed less prone to shortcut learning and systems that generate answers are in many cases more interpretable. To that end Chollel (2019) proposes the Abstract Reasoning Corpus (ARC), where the model is given a few examples of Input-Output (I/O) pairs and has to understand the underlying common program that was applied to the inputs obtain the outputs. ARC tasks are meant to rely only on the innate core knowledge systems which include intuitive knowledge about objects, agents and their goals, nu- merosity, and basic spatial-temporal concepts. However, ARC remains unapproachable by current deep learning methods.
We propose to take a step towards solving ARC by designing new generative benchmark in which we evaluate systematic compositional generalization. Like ARC, our dataset is composed of a col- lection of support sets, each having number of input/output pairs, such that the output for every sample (support set) is obtained by applying the same program to the corresponding input. The model then presented with a new query input and evaluated on its ability to predict the right asso- ciated output (i.e. applying the inferred program in the support set to the query input). The inputs are 56 x 56 images with three colored MNIST digits placed at three different positions. These visual digits can have values between -9 and and their color represents their sign. There are six different colors in total (3 of them are negative and the remaining 3 are positive). The program applied to the inputs is sequence of arithmetic operations (we restrict ourselves addition and subtraction and the dataset can further be extended with more complicated queries that involve comparison between the different objects, maximum operations etc..) particular positional order. Since we are inter- ested in the model's ability to generalize compositionally to unseen examples, we create different splits that aim at evaluating different axes of compositional generalization. These three splits are as follows: : (1) OOD seq In this split, during training we leave out some sequence of operations (e.g. (+.-)) and perform the evaluation on samples requiring the excluded sequence., (2) OOD order: where uring training the model only sees programs that take input digits in some particular positional order (e.g. top-left, top-right, down) and is evaluated on unseen orders., and (3) OOD perception which evaluate the perception module ability to disentangle the digit class (e.g. to 9) from its color (representing its sign). We thus consider certain pairs of digit-color configurations during training and evaluate the model unseen pairs.
IOO pairs
Inputs
Controller
Executor
ctack
Slot Attention
select program
Program
library
Task
sample
pinn
program
Executor
p
C(P, pin)
infer imaginary program
Imagined I/O pairs
Controller
Imagination
Figure 2: Task and Imagination pathways in OCIM
4 OBJECT-CENTRIC COMPOSITIONAL IMAGINATION (OCIM)
Our model is designed to generate answers in a sequential way. The design choices that we make reflect the fact that the output answer generative reasoning task can be computed by sequentially updating a working memory whose arguments are obtained from available input (c.g. slots extracted from images in our case). The computation steps the following: (1) Visual inputs mapped
4

---

Under review as a conference paper at ICLR 2023
to N, object-centric slots S = [Si...,Sx Sx.] using Slot Attention module (Locatello et al. 2020 then, (2) the controller takes the support set as input and ouputs a single task embedding z; this task embedding is then (3) translated into sequential neural program (c.g. sequence of transforma- tion to be applied to a working memory); finally, (4) the executor takes this neural program along with input query (i.e. its object-centric slots) and performs the sequential updates. The overall computation paths are given in F
The main contribution of our work resides in the architecture of the executor, its interface with the controller (i.e. how the executor uses the information encoded in the controller) and the derived imagination machinery. The controller scaffold (detailed in the appendix) that we use in all baselines always outputs a single task embedding z and can be adapted depending on the task at hand. In this section, we detail the modeling choices of the executor and the interface between the executor and the controller, which is formulated as a selection bottleneck and the imagination component.
4.1 EXECUTOR
The executor takes a visual query input x R56x56x3 and a neural program: it then updates working memory h E RRP in sequential and structured manner. The visual input is first mapped to a set of N₂ object-centric slots [S,, Sx,] that later used as candidate arguments for each update of the working memory. The executor is composed of library of N, learned modules (e.g. rules, implemented as small GRU cells) and N. condition values. The conditions are expected to encode the way in which to select argument (e.g. among the slots) that is in turn used by module to update the working memory. Both rules (i.e. modules) and conditions are indexed by some learned tags M = [M,,.... MN,1 and C = [C .......: CN,] The neural program that the executor takes as input generates (1) the number T of updates that the executor needs to perform, specified by scalar gate each time step; denote the sequence of such gates by g = [g] ..... ... 9r)) (2) the sequence of modules ..r. (each mh parameterized by small learned GRU RNN) that will perform the I updates of the working memory; and (3) the sequence of conditions [Co, ... ..,,rl (each é being
a condition vector that selects one slot; this slot will be used as an argument to the selected module
each time step). At each time step f, each update in the sequence is done in the following two steps:
Argument selection: given condition vector select an argument of the update from among the input slots of the query. Update: update working memory h4-1 with GRU rule mi, and the selected argument &
Argument Selection. each time step, a slot argument is selected through a key-query attention mechanism. The idea is that the condition vector é is compared against all the input slots to select the one that corresponds best to the features encoded in the condition (e.g. select the slot at the "top- left" of the image). The attention mechanism is thus realized using the condition vector é E Rixd as a query and the N, slots $ = time-step f is given by:
SN.] E R N, as keys such that the selected argument $,
ST IS E R
s GumbelSoftmax
1xd
Vd
The sequence of selected arguments is thus given by , ST].
Sequential Update. Given sequence of processing modules [m, .urr a sequence of input arguments [So, ,8]] and a length given by a sequence gates [gl. the executor updates a working memory whose state at time step is denoted by h, such that:
h ( (1 96+1 (ot1)),,$$$41.. hy) and he So
(2)
For ease of notation, we let Executor(xx x, P) be the result of applying the neural program P to the visual input
4.2 SELECTION BOTTLENECK
In this section, we describe the interface between the controller and the executor: how the task embedding output by the controller is transformed into neural program that the executor then
5

---

Under review as a conference paper at ICLR 2023
takes imput (e.g. sequences of modules, conditions and gates) perform the sequential update.
First the task embedding z E R is transformed into a sequence of embeddings by feeding z as argument to GRU RNN that starts with an empty hidden state [z] ....... Zr| GRU(z).
Both module and condition selections are done through a key-query attention mechanism comparing the task embedding 2; to the N, learned module tags (denoted by M [ Mx,1 E R xd)
and the N₂ learned conditions tags (denoted by C [C],..., Cx.1 E RNexd). The keys are extracted from the condition tags, whereas the query is extracted in both attention operations form
the task embedding 2e (using two MLPs Q and Qo) such that the /-th element of each sequence is obtained with:
W GumbelSoftmax (r( RINN,
(3)
Vd and the resulting update is given by the following weighted sum m,(h,, 8t) Ee Wt Wmlmm, $t). Similarly, the conditions are obtained through:
c = GumbelSoftmax (o(ze) )e
(4)
Vd
with c E R. denoting the set of learned condition vectors.
Finally, the sequence of step gates are obtained directly from the sequence of [Z1,...,2] 2r] such that
gt = MLP(2.)
(5)
For ease of notation, we let Pa Selection (g, è, m) denote the neural program obtained from the task embedding where g, è, rn correspond to the associated step gates, condition vectors and processing module sequences.
4.3 COMPOSITIONAL IMAGINATION
Our main contribution resides in showing how object-centric inductive biases (used in the executor) can be leveraged to induce a new imagination- based learning framework that leads to better composi- tional generalization. The idea is that the same way select sequence of modules, conditions and gates using the task embedding output by the controller, we can also sample them at random (from a uniform distribution) create a new neural program that can be used to create imagined scenarios.
To do so, we sample at random a sequence of gates gmm = .... g a sequence of condition .im) ...) and a sequence of processing modules m im = [m ..... mi]] that cor- respond to the procedural part of the knowledge we have about the reasoning task at hand. Ideally we would also sample the query to process (c.g. the declarative part) but we leave that for future work. Instead, we take visual inputs that are already present in the training data and we apply an imagined neural program to them. Since the goal is to create new samples, we need to apply the same imagined program to set of visual inputs to form a support set.
vectors c.m = [c/m ".-m,...,...
Let Xsuup (xi,..., im cm im) be an imagined program. Then the imagination phase can be split into main steps:
} denote set of visual inputs from the training data, and let pim
Imagined samples: this step consists of applying imagined program pim to a support set of visual input X to obtain an imagined I/O support set Sim expu gmm) } with Omm (xxecutor(x,, pim) for x, E X.upp].
Task embedding inference: this step consists of encoding the imagined support set with the controller produce task embedding z (Sn
Mechanisms Prediction: the last step consists of predicting back the neural programs
(i.e. its components) that produced the imagined sample. This means matching Pat.. Sedection with pim.
The associated loss is called the imagination loss Lim = C(Pgin, pim), which can be split into cross-entropies predicting the step gate values, the conditions vector indices and the processing module indices. During training, we introduce this loss after a warming period during which the model is trained only on the training data available. We detail the hyperparameters in the Appendix.

---

Under review as a conference paper at ICLR 2023
Algorithm Executor Pseudocode
1: function EXECUTOR(X, P)
> The input x and program P > Object-centric perception
2: SlotAttention x)
3: 81..7: m...T, Co. f- P
4: ho f GumbelSoftmax AnS" j
B Working memory initialization
Vd
5:
for (t = 1;t < T;t +) do
6: & f- GumbelSoftmax (48 Bixd
Eq.
vd
7: hr+1 (0t+1)he
9++ (5 hz)
Eq.
8: end for
9: return (hr)
> Task-specific prediction
10: end function
A Algorithm 2 Compositional Imagination
Require: Xoupp
Samples during training & Object-centric perception > Sample a program
1: Ssupp <- SlotAutention (X#upp)
2: pim (g, C, M)
3: O [Executor (X;, pim) for x; E [
4: Sim &- {Xoupp gmm
5: z & Controller
6: P TM SelectionBotllnncck(Z||!
Infer program
7: lossim f- CrossEntropy(P. P2im)
4.4 TRAINING
The training of the whole model can be split into three phases: Step Pretraining of the perception
model such that the next steps start with reasonable latent slots. Step 2: Regular training on the task prediction objective (8-binary-bits digit prediction). Step 3: Imagination, where random modules and conditions are sampled to create new data points and expose the model to potentially OOD samples.
Each of these steps gives rise to a specific objective loss to optimize. The task prediction objective in our case is a simple cross-entropy on the output of the executor, since we treat each bit of the output as binary label to predict. This loss is given by:
Dussk =
Σ tEDmm
(y: log( + +(1 log Y.)).
(6)
The pretraining phase consists of training the Slot Attention module reconstruction task. Dur- ing the imagination phase, new samples are created according to Algorithm 2 and the model is optimized infer the programs that generated these samples and minimize C at the same time. We detail the hyperparameters associated to the different training phases the Appendix.
5 EXPERIMENTS
OCIM has two main components. The perception component and the object processing (i.e. reason- ing) component. Our contribution lies the object processing component, while for the perception component
SOTA slot attention module (Locatello et al. 2020). The goal of this section is two-fold: (1) to evaluate our imagination- based learning paradigm on a set of compositional gener- alization axes and (2) the ability of the perception module to extract symbolic-like like representations that can be used to solve our visual abstract reasoning task.
5.1 BASELINES
Our model OCIM can be seen as an extension of the sparse interaction inductive biases proposed in Neural Production Systems (NPS) Goyal al. (2021a), and augmented with an maainationn-aase

---

Under review as a conference paper at ICLR 2023
learning mechanism. NPS sequentially updates set of slots by choosing at each time-step a primary slot, a secondary slot, and an interaction rule with some key-query attention mechanisms. OCIM sequentially updates the state of shared working memory across time steps (e.g. instead of slot states) from which the final answer can be extracted. As a result, at each time-step the primary argument of an interaction is always the shared memory and the second argument is selected among the input slots. We also compare OCIM (and its variant without imagination that we call OCIM- noim) two other baselines which the executor is parameterized with single monolithic GRU RNN in one case, and with a dense GNN in the other, for which we use the interaction component from the C-SWM (Kipf al. 2019) model (like Goyal al. (202Tb). each of these baselines, input nodes correspond to extracted slots concatenated with the output of the controller. For the GNN baseline, GRU RNN added after computing the interactions between nodes to aggregate the final result. We refer to these two baselines as GRU RNN and GNN respectively, and detail their exact parameterization the Appendix.
test
tost
Va
tast
1.0
1.0
0.8
0.8
accuracy
0.6
0.6
0.6
0.4
0.4
0.4
0.2
odim
ocim-naim model
odim
ocim-noim
ocim
ocim-naim
model
model
OOD op
(b) OOD order
(c) 0OD perception
Figure 3: Validation and test accuracy for the baselines and OCIM on the three axes of compositional generalization described in Section 3 Insights about the high variance of the results for OCIM given in 5.3 when inspecting modules specialization.
5.2 OOD SPLITS
We are interested in 3 axes of compositional generalization that evaluate both our proposed imagination-based learning paradigm and the perception module robustness: (1) We first want to evaluate whether the imagination phase in OCIM can lead to better generalization to arithmetic tasks composed of never-seen sequences of operations during training; (2) we then want eval- uate whether OCIM is able to generalize to never-seen orders in which the input digits are taken perform the sequence of operations (e.g. its ability extract meaningful and general argument selection conditions), and finally,
(3) we want to evaluate whether current object-centric iductive biases as proposed in Slot Attention Locatello et al. (2020) are well suited for disentangling inde- pendent factors of variations within an object (e.g, color and digit class in our case). To that end, we propose to evaluate the models on the three splits described in Section 3
5.3 RESULTS
Imagination and Generalization. In experiments, all the models share the same perception model Locatello et al. (2020) and the same controller. They only differ by their execution compo- nent. In Figure 3 we report the accuracy peformances of our model compared to the baselines of interest across three different splits that aim at evaluating a particular axis of generalization. We observe two main results: (1) Imagination does help to generalize novel sequences of operations as shown in the generalization gap of OCIM between the results of the OOD op split and the other baselines. (2) Current SOTA object-centric perception models like Slot Attention are not quite able to systematically generalize to objects composed of never-seen before arrangements of known ar- guments (such as new pairs of color/shape). This result interesting and suggests that additional inductive biases learning paradigms are needed to learn object-centric representations that dis- entangle independent factors of variations within an object. We did however notice that the choice architecture for the execution component seems have an impact the perception part, and that, surprisingly, both the GNN and GRU baselines perform better than OCIM on the OOD perception split.
8

---

Under review as a conference paper at ICLR 2023
Modules Specialization. We are also interested in analyzing how specialization of the learned modules (i.e. becoming activate for a certain operation) impacts the generalization perfor- mance of OCIM. For each training random seed, we count the number of times each mod- ule was selected for each of the ground-truth operations. Since we use the Gumbel soft- decide
max trick to select modules we use the argmax of the attention coefficient which module is selected. We report these proportions in the heatmap in Figure | for OCIM with and without imagination. The x-axis corresponds to the seed number; the y- axis corresponds to the module indices for both the addition and the subtraction operations. The accuracies reported at the top of the heatmaps correspond to the validation and test accuracies on the OOD op splits (e.g- when evaluating the models on sequences of oper- ations that have not been seen during train-
ing). We note the following three observations: (1) Current inductive biases are not sufficient for specialization to systematically occur since there are some seeds that have overlaps between
Add
Module Number Sub
selected modules. (2) As shown the gen-
eralization results of OCIM-noim, specializa- tion in the modules is not enough for the model to generalize to novel sequences of operations (e.g. modules in seed of OCIM-noim (left column) are specialized yet we do not observe systematic generalization). Finally, (3) We no- tice that imagining new samples is necessary for the model to generalize, but also not suf- ficient it needs the modules to be specialized well. Seeds 0 and 5 of OCIM do have overlap in the module selections and the imagination framework was not successful. This observation explains the variance in terms of performance for the 0OD op split that report in Figure
1 3 5 0
Experiment number Experiment number OCIM-noim
OCIM
Figure 4: Modules specialization.
CONCLUSION AND FUTURE WORK
We have presented OCIM, a method that leverages object-centric representations to decompose visual reasoning tasks into series of learned primitives (operation and object choices). OCIM combines these primitives in novel ways in order to generate and learn from unseen imaginary tasks, which radically improve OOD generalization. We compared OCIM against NPS and two other baselines without imagination apply a sequence operations to colored MNIST digits. found that only OCIM was able to systematically generalize new tasks composed of unseen sequences of arithmetical operations. Interestingly, we observed that specialization among the neural modules seems to be a necessary but not sufficient condition for modular architectures like OCIM and NPS generalize to unseen sequences of operations: imagination seems to be a critical addition to the specialization condition. The effectiveness of imagination in setup raises the question of whether its function is similar in biological brains. An interesting hypothesis is that dreams have regularizing effect in the brain (OBH). While the link between OBH and OCIM superficial, poses an interesting question that might be worth exploring in future work.
a synthetic visual arithmetic reasoning task in which we
Along with OCIM, we have introduced synthetic visual reasoning benchmark assess the extent which imagination improves compositional generalization. Despite the simplicity of the benchmark, we found that SOTA models like NPS fail compose the primitives learned during training novel ways in order to generalize. As research in compositional generalization progresses, the benchmark could be extended with more challenging scenarios by increasing the number of operations, the length of the programs, and the number of objects.
Reproducitbility Statement We reported in the appendix all the model and training hyperparam- eters to implement and reproduce our model (Table different data splits. We will release the code and scripts for both our model, the baselines and the generation of the different dataset splits.
well the detailed content of the

---

Under review as a conference paper at ICLR 2023
REFERENCES
M. Arjovsky, L. Bottou, I. Guirajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint 2019.
B. J. Baars. In the theatre of consciousness global workspace theory, a rigorous scientific theory of consciousness. 1997.
D. Barrett, F. Hill, A. Santoro, A. Morcos, and T. Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning, 2018.
Y. Bengio, T. Deleu, N. Rahaman, R. Ke, S. Lachapelle, O. Bilaniuk, A. Goyal, and C. Pal. A meta-transfer objective for learning to disentangle causal mechanisms. arXiv:1901.10112 2019.
C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner. Monet: Unsupervised scene decomposition and representation, 2019.
J. Cai, R. Shin, and D. Song. Making neural programming architectures generalize via recursion. In International Conference Learning Representations, 2017.
C. Chadebec and Allassonnière. Data augmentation with variational autoencoders and manifold sampling. In S. Engelhardt, I. Oksuz, D. Zhu, Y. Yuan, A. Mukhopadhyay, N. Heller, S. X. Huang, H. Nguyen, R. Sznitman, and Y. Xue, editors, Deep Generative Models, and Data Augmentation, Labelling, and Imperfections, pages 184-192, Cham, 2021. Springer International Publishing.
F. Chollet. On the measure of intelligence. arXiv preprint arXiv: Arxiv-1911.01547,, 2019.
E. Crawford and J. Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In AAAI Conference Artificial Intelligence, 2019.
S. Dehaene, H. Lau, and S. Kouider. What is consciousness, and could machines have it? Science, 358(6362):4466-92,, 2017. doi: 10. 10.1126/cceecce..anng aan8871 URL httpt::////... //www scwssciieeeeeee ence. brg/doi/abs 002.111.11..... 1ttpo/111///ccccccc/
wttp//ccenceee.ccccce.eoo.....ec aan8871
H. K. dos Santos Tanaka and C. Aranha. Data augmentation using gans. Proceedings of Machine Learning Research XXX, 1:16, 2019.
K. Ellis, C. Wong, M. Nye, M. Sablé-Meyer, Morales, L. Hewitt, L. Cary, A. Solar-Lezama, and J. B. Tenenbaum. Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep li- brary learning. In Proceedings the 42nd acm sigplan international conference on programming language design and implementation, pages $35-850, 2021.
L. Engstrom, B. Tran, D. Tsipras, L. Schmidt, and A. Madry. Exploring the landscape of spatial robustness. International conference on machine learning, pages 1802-1811. PMLR, 2019.
S. Eslami, N. Heess, T. Weber, Y. Tassa, D. Szepesvari, K. Kavukcuoglu, and G. E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. arXiv preprint arXiv:1603.08575 2016a.
S. M. A. Eslami, N. Heess, T. Weber, Y. Tassa, D. Szepesvari. K. Kavukeuoglu, and G. E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models, 2016b.
J. A. Fodor and Z. W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3 1988.
R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. Imagenet- trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
A. Goyal and Y. Bengio. Inductive biases for deep learning of higher-level cognition. CoRR, abs/2011. 1,0091 2020.
15091
A. Goyal and Y. Bengio. Inductive biases for deep learning of higher-level cognition. Proc. A, Royal Soc., arXiv:2011. 155091, 2022.
10

---

Under review as a conference paper at ICLR 2023
A. Goyal, A. Lamb, J. Hoffmann, S. Sodhani, S. Levine, Y. Bengio, and B. Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:1909.10892 2019.
A. Goyal, A. Lamb, P. Gampa, P. Beaudoin, S. Levine, C. Blundell, Y. Bengio, and M. Mozer. Ob- ject files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. arXiv preprint arXiv: Arxiv-2006. 200.-2000.16225,, 2020.
A. Goyal, A. Didolkar, N. R. Ke, C. Blundell, P. Beaudoin, N. Heess, M. Mozer, and Y. Bengio. Neural production systems. CoRR, abs/2103.01937, 2021a. URL https://arxiv. org/
abs/2103.00 01937
A. Goyal, A. Didolkar, N. Ke, C. Blundell, P. Beaudoin, Heess, M. Mozer, and Y. Bengio. Neural production systems. arXiv preprint arXiv:2103.01937 2021b.
K. Greff, S. Steenkiste, and J. Schmidhuber. Neural expectation maximization, 2017.
K. Greff, R. I.. Kaufman, Kabra, N. Watters, C. Burgess, D. Zoran, Matthey, M. Botvinick, and A. Lerchner. Multi-object representation learning with iterative variational inference, 2019.
D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.
D. Hafner, T. P. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2020.
G. E. Hinton, S. Osindero, and Y. W. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006.
Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:47-1, 2022.
Hoel. The overfitted brain: Dreams evolved to assist generalization. Patterns, 255):100244, 2021.
D. Hoshen and M. Werman. Iq of neural networks. arXiv preprint arXiv:1110.016922.200 2017.
C. James. Raven. mental tests used in genetic studies: The performance of related individuals on tests mainly educative and mainly reproductive. Unpublished master's thesis. University of London, 1936.
T. Kipf, E. van der Pol, and M. Welling. Contrastive learning of structured world models, 2019. URL htttps://axxiv.oo wwtp:/wwwwwggo///oop/ooooo /1911. 12247
A. R. Kosiorek, H. Kim, I. Posner, and Y. W. Teh. Sequential attend, infer, repeat: Generative modelling of moving objects, 2018
B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Q. Li, S. Huang, Y. Hong, Y. Chen, Y. Wu, and S.-C. Zhu. Closed loop neural-symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning. In International Conference on Machine Learning, pages 5884-5894, PMLR, 2020.
Z. Lin, Y.-. Wu, S. Peri, B. Fu, J. Jiang, and S. Ahn. Improving generative imagination in object- centric world models. In International Conference on Machine Learning, 2020.
Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, Uszkoreit, A. Doso- vitskiy, and T. Kipf. Object-centric learning with slot attention. arXiv preprint arXiv: Arxiv- 2006. 15055, 2020.
M. Mitchell. Abstraction and making in artificial intelligence. Annals of the New York Academy of Sciences, 1505(1):79-101, 2021.
O. Ostapenko, P. Rodriguez, M. Caccia, and L. Charlin. Continual learning via local module com- position. Advances in Neural Information Processing Systems, 14:30298-30312, 2021.
11

---

Under review as a conference paper at ICLR 2023
G. Parascandolo, N. Kilbertus, M. CarullCa and B. Schölkopf. Learning independent causal mechanisms. In International Conference on Machine Learning, 2018.
J. Pearl. Causality: Models, reasoning, and inference. 2nd edition. 2009.
S. Reed and N. De Freitas. Neural programmer-interpreeers arXiv preprint arXiv:1511.06279, 2015.
L. Ruis and B. Lake. Improving systematic generalization through modularity and augmentation. arXiv preprint arXiv:2202. 2022.
E. S. Spelke. Core knowledge. American psychologist, 55(11): 2000.
K. Stelzner, R. Peharz, and Kersting. Faster attend-infer-repeat with tractable probabilistic mod- els. In International Conference on Machine Learning, 2019.
J. Su, D. V. Vargas, and K. Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions Evolationary Computation, 23(5):828 841, 2019.
S. van Steenkiste, M. Chang, K. Greff, and J. Schmidhuber. Relational neural expectation maxi- mization: Unsupervised discovery of objects and their interactions, 2018
S. Steenkiste, K. Greff, and J. Schmidhuber. A perspective objects and systematic general- www.oo/g//.....
ization in model-based RL. CoRR, abs/1906.01035, 2019. URL
1906. 01035
C. Zhang, F. Gao, B. Jia, Y. Zhu, and S.-C. Zhu. Raven: A dataset for relational and analogical visual reasoning. In Conference on Computer Vision and Pattern Recognition, 2019.
A APPENDIX
A.1 MODEL.
Controller. The controller encodes support set of I/O pairs S [X, OJ and outputs task em- bedding z that is used to predict a task-specific output associated to a query input. We formulate the controller in an iterative manner: it starts with random guess Zui (sampled from learned gaus- sian) and refines it. We denote Refine(z, S) a refinement step and T the total number refinement steps. A refinement step timestep l be decomposed into three main steps:
Step 1
Current guess prediction: compute the current predicted outputs Ô associated with each input in the support set Ö |ComputeChuess(X. 2,)
Step 2 Compare current guess Ô, to ground-trui truth outputs O given their associated inputs in S. We denote a, = Compare(O,,, S, S,n.) 2.) the output of this step.
Step 3 Update current task embedding Z Update(a,, 2,)
ComputeGuess(XX (X, 2,) is model-specific and simply consists of predicting the current task output given z, And z1+1 Update(a,, 2,) is parametrized as simple GRU RNN taking a, as input and updating the hidden state 2, We then need detail how obtain a, Compare O S, 2,) This is done in two steps (1) first we compute a representation by for each sample in the support set, then, (2) we aggregate those representations
obtain a,. To do so, each input x; X is transformed into an object-centric set of slots { using the same perception model as in the rest of the model. Each slot then concatenated the current guess 2,, the ground-truth associated input o, and the currently predicted output 6,. The sample-wise result is then obtained with simple GRU RNN. We denote this step by by EncodeSample(x, 2;, O, ô;) and the resulting sequence representation by b. [b:]-
We then need to aggregate the obtained results accross the whole support set. To do so, we concate- nate each sample representation by with the current task embedding 2,, the ground-truth associated input o; and the currently predicted output o; Similarly we aggregate the results with a simple GRU RNN. We denote a, = EncodeSupport(h b,, z,, O, Ó,). The exact parametrization of each of the modules consituting the controller is given in Table 1
12

---

Under review as a conference paper at ICLR 2023
Description
Symbol
Architecture
Task embedding refiner
Update(a 7,)
GRUCelI(64, 64)
Sample concatenation before aggregation Sample represenlation aggregation accross slots Sample-wise wise processing before aggregation Aggregation accross support elements
hy (1)
Concat(s), (s, 2z, ó.))
MLP(128 + 10, 64)
bf Agg( [1g (t)))
GRU(64, 64)
MLP(64, 64)
EncodeSupporthb O.
GRU(64, 64)
Table 1: Controller Hyperparameters
Perception. The perception component is implemented with Slot Attention module. We keep the same hyperparameters the initial version (Locatello et al. 2020) with slot-wise dimension of 64 and using different Gaussian parameters for each slot. We perform all the experiments with 4 slots.
Selection Bottleneck. The selection bottleneck is composed of three parts: (1) the module that transforms a single task embedding z into a sequence of embeddings from which the step-wise elements of the neural program will be predicted, (2) the extraction of keys, queries, and values for the condition and the module selection, and (3) the step-wise gate prediction to determine the number of updates to perform (e.g. neural program length). We detail the parameterizations of these
three parts in Table 2
Description
Symbol/Notation
Architecture/Val Value
Sequence prediction
1.....77 Pred(z)
GRU(64, 64) a (MI.(664, 1))
Gate prediction
Gate(z;
Query prediction for modulc/condition selection Learned Module/Condition Key embedings
(z,), and (z.)..)
MLP(64, 64), MLP(64. 64)
M and C
Embeding(N,, 64) and Embeding(NN. 64)
Max number steps
Gumbel softmax temperature
temp
3
Table 2: Selection Bottleneck Hyperparameters
Executor.
The executor is composed of three main parts: (1) the learned neural program library (e.g. modules and condition vectors), (2) the argument selection part given condition vector, and the (3) task-related output prediction. We detail the parameterization of these part in Table
Description
Symbol/Notation
Architecture/Value
Modules
m... I.N.
[GRU(64, 64)]
Condition vectors
Ci,...C.
Embedding(( Ne, 64)
Condition/Modules keys
C and M (S)
Embedding(NN 64) and Embedding((,, 64)
Key prediction for argument selection
MLP(64, 64)
Output prediction
y (asrPre((br)) N, and Nr
(MLP((64, num bits))
Number conditions/modules
3 and 3
Table Execution component Hyperparameters
A.2 TRAINING
The training of the whole model can be split into three phases:
Step 1: Pretraining the perception model such that the next steps start with reasonable
latent slots.
Step 2: Regular training the task prediction objective (here 8- 8-binary- digit predic- tion).
Step 3: Imagination, where random modules and conditions are sampled to create new data points and expose the model to potentially OOD samples.
13

---

Under review as a conference paper at ICLR 2023
Each of these steps gives rise to specific objective loss to optimize, namely: L L and Lim. The different phases consist of adding progressively these losses to the optimized objective. The total loss is:
L == C + aL + 3
The pretraining the slot attention modules (Step done separately and we initialize with the pretrained weights when adding the task-specific loss Step 2. The coefficient is fixed, whereas introduced after a warm-up period and increased linearly during a certain number of epochs. We report these training-speciin hyperparameters in Table 4
Description
Symbol/Notation
Value
Reconstruction coefficient
0.0002 50
Imagination coefficient
5
Warmup period before imagination Imagination coef schedule
450 epochs
from 0 to /3 in 200 epochs 32
Batch size
Learning rate
2e-4
Table 4: Training Hyperparameters
Dataset For all the different splits, we trained OCIM on 5000 samples where each sample had 10 I/ pairs in the support set and I/O query example. Validation and Test set are composed of 100 and 1000 samples respectively. We describe below the generating steps of the three different splits considered in the experiments:
OOD op: models are trained on sequences of operations [ and evaluated [-+]. The model is also trained and evaluated on one order only: top left, top right, then bottom. Digits, signs and colors are sampled randomly iid.
OOD order: models are trained on all sequences of operations and on one object order (top left, top right, then bottom); they are then evaluated on all operations, and all possible orders in which the visual digits can be considered.
O0D percep: for each digit, only (sampled in uniform way) possible positive col- ors and two negative colors are considered during training. The models are evaluated on the left-out color configurations. Models are trained and evaluated on all sequences of operations [,t
14


Reviews:
[{"review_id": "v0V0cNvydYQ", "reviewer": "Reviewer_zq7u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for imagination (data augmentation) to improve the generalisation of abstract reasoning approaches. The general module is based on adapting slot-attention to work over a sequence of to select the latent function to apply. The augmentation works using Gumbel soft max trick to select the augmentation. The authors additionally propose a new dataset of mathematical problems with values, colours and operations to be resolved. They compare to the prior approach on NPS overfitting to training but improving generalisation on test.", "summary_of_the_review": "While the paper works on a challenging problem and shows generalisation across unseen problems based on their proposed baselines. The lack of rigorous evaluation and benchmarks from similar problems means the paper feels incomplete. Additional experiments or justification for their lack of inclusion needs to be provided to position this paper.", "strength_and_weaknesses": "Strengths:\n- The proposal of the imagination framework is interesting working within the latent space of the model, which should provide increased generalisation without being limited to generation.\n- The approach is fully differentiable that is and advantage against some prior methods for exutors.\n- The dataset can be useful if well-defined against prior datasets and models\nWeaknesses:\n- The augmentation strategy can be broadly compatible with data augmentation strategies which don't seem to have been considered in related work or to inspire the approach. \n- The data augmentation is loosely out of distribution, as the numerical dataset has a relatively limited scope with a simple range of numbers and operations. As the explicit function of the functions is unknown, and the logical operations can be deduced as a combinational operation that is unclear if possible in the leave-out strategy.\n- It would have been interesting for the approach to be applied to standard Abstract reasoning datasets such as RAVEN. While they are different problems, they could have shown greater generalisation and provided insights into how the model is working. \n- Similar to the above, the dataset seems highly similar to RAVEN and others, just numbers instead of shapes. It isn't clear where this dataset contributes.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "m98dQbf9Hk", "reviewer": "Reviewer_65Tv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the compositional generalization of visual abstract reasoning, i.e., the  ability  to  compose learned  concepts  in  novel  ways  for new situations. The authors propose an imagination-based learning framework applied on object-centric representations. By training on the imagined tasks, the model is expected to obtain better systematic generalization at test time. \n\nTo evaluate the proposed framework, the authors synthesize a new visual arithmetic reasoning benchmark. Specifically, each example in the benchmark contains a support set of 10 input-output pairs and the outputs of these 10 pairs are obtained by applying a same latent program to the corresponding inputs.  Based on the given support set, the model is required to predict the output of a query input. The inputs are images with three colored MNIST digits placed at three different positions and the colors represent the signs of the digits. The program applied to the inputs is a sequence of arithmetic operations (i.e., addition and subtraction) in a particular positional order. There are three splits in the benchmark to evaluate generalization to new sequences of operations, position orders, and digit-Color configurations.\n\nThe used model to solve the proposed benchmark consists of a controller, which generates a latent task embedding based on the given support set, and an executor, which generates a neural program based on the task embeding and then apply the generated neural program to predict the output of the query input. The proposed imagination learning works by (1) randomly sampling neural programs, (2) applying the sampled neural programs to the inputs of original samples to generate new samples, and (3) training the model on the new samples. Experiments show that the imagination learning can improve the generalization to new sequences of operations.\n\n\n\n\n\n ", "summary_of_the_review": "Although the proposed method is novel, the experimental evaluation is insufficient. Therefore, I recommend rejection.\n\n-------post-rebuttal-----\nThank the authors for the response. The response solves part of my concerns. Still, the main concern about the insufficient experiments is not resolved. Therefore, I will keep my rating. Hope the authors could further improve their work in the future.", "strength_and_weaknesses": "## Strengths\nThe proposed imagination-based learning is interesting and able to improve the systematic generalization over a certain apsect (i.e., sequence of operations)\n\n## Weaknesses\n\n1. It is unclear how to randomly sample a new program, i.e., from what probability distributions the gates, the condition vectors, and the processing modules are sampled.\n\n2. The authors claim the proposed imagination method as \"Object-centric Compositional Imagination\". However, it seems that the proposed imagination methond (described in Section 4.3) can also be used to non-object-centric representations, e.g., grid representations of images. The authors do not conduct any experiment using non-object-centric features.\n\n3. The proposed visual arithmetic reasoning dataset is similar to RAVEN-style datasets [1], i.e., few-shot learning a rule from a support set and applying it to a new input. The RAVEN-style datasets have been extensively studied by previous works. It is unclear why it is necessary to synthesize such a new dataset. Besides, the authors only conduct experiments on the proposed dataset and do not compare with related works in RAVEN.\n\n[1] Zhang, Chi, et al. \"Raven: A dataset for relational and analogical visual reasoning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n4. Figure 3 is the only experimental result presented in this paper and from this figure, we can see that the proposed method has a much larger variance than baselines. There is not any explanation about this phenomenon in the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lCBoFAH3aM", "reviewer": "Reviewer_Czgi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes Object-centric Compositional Imagination (OCIM) for visual reasoning tasks. It decomposes the tasks into a series of primitives applied to objects and recomposes these primitives to imaginary tasks. The model learns to reuse the previously-learned concepts to systematically generalize at test time by training on these imagined tasks. The model shows better generalization ability than the other baselines on a synthetic visual arithmetic reasoning benchmark the authors newly proposed.", "summary_of_the_review": "It is an interesting and novel idea to utilize the object-centric imagination method to improve the OOD generalization of visual reasoning tasks. However, the paper's writing and supporting experiments need to be improved with more effort.\n\n---\nUpdate (2022/11/22): I would keep my score before new results provide.", "strength_and_weaknesses": "Pros:\n1. This work proposes a new synthetic visual reasoning benchmark that provides access to operations, operation order, objects primitives.\n2. The novel OCIM model presents better generalization ability over other baselines on the new benchmark.\n\nCons:\n1. The writing needs to be improved, detailed in the clarity part.\n2. Lacking ablation studies make it hard to estimate the importance and functionality of each module. May consider the following (you may put details in the appendix and state the ablation study results briefly in the main text if space is not allowed):\n* What if $\\beta$ (imagination coefficient) is set to zero (disabling the imagination)?\n* How important are the warmup epochs?\n* Can you modify the max number of steps $T$? $T=2$ seems a very specific design for the benchmark task.\n3. The evaluations are only done on a self-proposed dataset that provides access to primitives. It is unclear how this method can be generalized to other more challenging benchmarks or benchmarks that do not have access to these primitives.\n4. The overall motivation is clearly stated. However, the motivations behind the specific design of the OCIM model need more explanation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cim3IQFtqA", "reviewer": "Reviewer_Zv6d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a neural program-based method for an abstract reasoning (ARC from Chollet 2019) style task. The model contains an object-centric representation layer (based on slot attention I believe), a controller (encoding the examples and predicting the program), and an executor (which takes the program representation as input and predicts the final answer). The claimed contributions are two-folds. First, the program representation (which is a sequence of module indices, and a sequence of control parameters). Second, the training paradigm, which includes a \"dream\" stage.", "summary_of_the_review": "I like the idea of the paper on dreaming with programmatically structured neural networks for generalization. However, the results of the paper are limited. The paper writing can still be improved. Overall my recommendation is WR.", "strength_and_weaknesses": "The paper presents an approach for neural-program-based approach for solving ARC tasks. The proposed approach is well-motivated, and shows improvements in experiments. My main concerns about the paper are the following three.\n\nFirst, I think the paper writing can be significantly improved. The major issues are: \n1. Figure 2 is not an overview of the model. For example, it does not contain the object perception module (which the authors claim is an object they want to study in their contribution #3)\n2. Section 4 does not contain a full description of the proposed method. I have to look at Algo 3 and based on the names to guess what's used for object perception.\n3. The current writing of the paper is very unclear about the contribution of the authors. For example, my understanding that 4.1 and 4.2 are not the contribution of the author (See section 5.1: Our model OCIM can be seen as an implementation of the sparse interaction inductive biases proposed in Neural Production Systems (NPS) Goyal et al. (2021a), but augmented with an imagination-based learning mechanism.)\n\nSecond, given that the neural programming model is not the contribution of the paper, my understanding is that the paper focuses on \"imagination.\" While overall I very much like the idea of using trained models to generate new data, and the authors have found an interesting domain to execute the idea, I am a bit worried about the generality of the proposed method.\n\nSpecifically, it seems that the dataset is constrained to a \"sequential program\" structure. And furthermore, almost all possible combinations of learned \"modules\" can be executed on this dataset (e.g., different orders, different visual positions). However, this is generally not true for many domains. There are at least two cases.\n\nFirst, consider we want to use a program to query \"the color of digit 1.\" The program assumes the existence of a digit 1 in the data, which, if not satisfied, will lead to unexpected execution results. I can easily imagine that adding those data into the training set will hurt model performance. I think the authors should at least talk about these assumptions about their domain, and discuss why their methods can work in the specific domain. That being said, I think the proposed method is too constrained.\n\nSecond, when these modules have dependencies among each other, it is unclear how we can sample programs that are \"valid.\"\n\nThird, the experimental results of this paper are a bit weak, due to the limitation to one dataset curated by the authors themself. Due to the limitations, I described above, I think it is important to extend the results to other domains, for example, other ARC tasks, or tasks such as visual question answering etc.\n\nOther than these limitations, I think the authors should also cite more related works.\n\nFor example, the authors have clearly missed several important works in neural programming, such as\n- Neural Programmer-Interpreters, https://arxiv.org/abs/1511.06279\n- Making Neural Programming Architectures Generalize via Recursion https://arxiv.org/abs/1704.06611\n- Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning http://proceedings.mlr.press/v119/li20f/li20f.pdf\n\nThe contribution of \"disentanglement of object properties in object-centric representation learning\" is closely related to many visual representation work. Here are some pointers:\n- https://cs.stanford.edu/people/jcjohns/clevr/ Many works have worked on the CLEVR-CoGenT split, which is exactly the setup you have been studying.\n- ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. https://arxiv.org/abs/1811.12231", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject