Systematic Visual Reasoning through Object-Centric Relational Abstraction
Authors: Taylor Whittington Webb
Shanka Subhra Mondal
Jonathan Cohen
Venue: NeurIPS 2023 poster (2023)
Keywords: relational reasoning
object-centric representations
abstract rule learning
relational inductive biases
systematic generalization
TL;DR: A model that combines object-centric and relational inductive biases achieves systematic generalization in abstract visual reasoning tasks.

Abstract: Human visual reasoning is characterized by an ability to identify abstract patterns from only a small number of examples, and to systematically generalize those patterns to novel inputs. This capacity depends in large part on our ability to represent complex visual inputs in terms of both objects and relations. Recent work in computer vision has introduced models with the capacity to extract object-centric representations, leading to the ability to process multi-object visual inputs, but falling short of the systematic generalization displayed by human reasoning. Other recent models have employed inductive biases for relational abstraction to achieve systematic generalization of learned abstract rules, but have generally assumed the presence of object-focused inputs. Here, we combine these two approaches, introducing Object-Centric Relational Abstraction (OCRA), a model that extracts explicit representations of both objects and abstract relations, and achieves strong systematic generalization in tasks (including a novel dataset, CLEVR-ART, with greater visual complexity) involving complex visual displays.

Systematic Visual Reasoning through Object-Centric Relational Abstraction
Taylor W. Webb*;
Shanka Subhra Mondal
Department of Psychology University of California, Los Angeles
Department of Electrical and Computer Engineering
Princeton University
Los Angeles, CA taylor. w. wwwb@gmail.c.mmmm
Princeton, NJ
smondal@@rinnetton..
Equal Contribution
Jonathan D. Cohen
Princeton Neuroscience Institute
Princeton University Princeton, NJ jdc@princeton. edu
Abstract
Human visual reasoning is characterized by ability to identify abstract patterns from only a small number of examples, and to systematically generalize those patterns to novel inputs. This capacity depends in large part our ability to represent complex visual inputs in terms of both objects and relations. Recent work in computer vision has introduced models with the capacity to extract object-centric representations, leading to the ability to process multi-object visual inputs, but falling short of the systematic generalization displayed by human reasoning. Other recent models have employed inductive biases for relational abstraction to achieve systematic generalization of learned abstract rules, but have generally assumed the presence of object-focuse focused inputs. Here, we combine these two approaches, introducing Object- Centric Relational Abstraction (OCRA), model that extracts explicit representations of both objects and abstract relations, and achieves strong systematic generalization in tasks (including novel dataset, CLEVR-ART, with greater visual complexity) involving complex visual displays.
Introduction
When presented with visual scene, human reasoners have capacity to identify not only the objects in that scene, but also the relations between objects, and the higher-order patterns formed by multiple relations. These abilities are fundamental human visual intelligence, enabling the rapid induction of abstract rules from a small number of examples, and the systematic generalization of those rules For example, when presented with the image in Figure T one can easily determine that the objects in the top and bottom rows are both governed by a common abstract pattern (ABA). Furthermore, even those that one has never observed before. Indeed, this is arguably what it means for a pattern to be considered abstract.
to novel inputs (2 25 iK] 27
could easily then generalize this pattern to any potential shapes,
37th Conference Neural Information Processing Systems (NeurIPS 2023).
t Co-first author order is arbitrary and may swapped when citing this work.

---

Feature embeddings
Position embeddings
Relational operator
Relation embeddings
Slot attention
Transformer
FioK
x
Figure I: Object-Ceene Centric Relational Abstraction (OCRA). OCRA consists of three core compo- nents, responsible for modeling objects, relations, and higher-order relations. First, given a visual input x, slot attention extracts object-centric representations, consisting factorized feature em- beddings = and position embeddings m.=11..K. Second, pairwise relation embeddings TAK are computed by first passing each pair of feature embeddings to a relational operator é. and then additively incorporating their position embeddings. This allows visual space serve indexing mechanism for visual relations. Finally, relational embeddings are passed to a transformer to model the higher-order patterns formed by multiple relations.
Though this task appears effortless from the perspective of human reasoning, it exemplifies the type of problems that pose significant challenge for standard neural network methods. Rather than inducing an abstract representation of the underlying rules a given task, neural networks tend to overfit to the specific details of the examples observed during training, thus failing to generalize those
rules to novel inputs 126 3 33
To address this, a number of neural network methods have recently been proposed that incorporate strong inductive biases for relational abstraction 40 51 22 M. also referred as a 'relational bottleneck' [50). The key insight underlying these approaches is that abstract visual patterns are often characterized by the relations between objects, rather than the features of the objects themselves. For example, the pattern seen in Figure T is characterized by the relation between the shape and position of the objects. By constraining architectures to process visual inputs terms of relations between objects, these recent approaches have achieved strong systematic (i.c., out-of -uttt--distribution) generalization of learned abstract rules, given only a small number of training examples [5T 22
However, major assumption of these approaches is that relations are computed between objects. Accordingly, these methods have generally relied on inputs consisting of pre-segmented visual objects, rather than multi-object scenes. Furthermore, these methods can sometimes be brittle to task- lask-irrelevant visual variation, such as spatial jitter Gaussian noise - is thus unclear whether and how these approaches might scale to complex and varied real-world settings.
In this work, we take step toward the development of relational reasoning algorithms that operate over more realistic visual inputs. To do so, we turn to object-centric representation learning 15 G [2], a recently developed approach in which visual scenes are decomposed into a discrete set of object-focused latent representations, without the need for veridical segmentation data. Though this approach has shown success on a number of challenging visual tasks [C 39 60) has not yet been integrated with the strong relational inductive biases that have previously been shown to enable systematic, human-liks generalization Abstraction (OCRA), a model that combines the strengths of these two approaches. Across suite of visual reasoning tasks (including a novel dataset, CLEVR-ART, with greater visual complexity), we find that OCRA is capable of systematically generalizing learned abstract rules from complex, multi-object visual displays, significantly outperforming number of competitive baselines in most settings.
28
51 22 [I. Here, we propose Object-Centric Centric Relational

---

2 Approach
Figure T shows schematic depiction of our approach. OCRA consists of three components, each responsible for a different element of the reasoning process. These three elements can be summarized as extraction of object-centric representations, 2) computation of pairwise relational embeddings, and 3) processing of higher-order relational structures patterns across multiple relations - in order identify abstract rules. We describe each of these three components in the following sections.
2.1 Object-centric representation learning
OCRA employs slot attention 2X to extract object-centric representations from complex, multi-object visual inputs. Given an image æ. the goal of slot attention is to extract a set of latent embeddings (i.e., slots), each which captures a focused representation access to ground truth segmentation data.
single visual object, typically without
To do so, the image is first passed through a series of convolutional layers, generating a feature map feat E IR = XDD Positional embeddings pos E " x are then generated by passing 4d position code (encoding the four cardinal directions) through separate linear projection. Feature and position embeddings are additively combined, and then passed through a series of 1x1 convolutions. This is then flattened, yielding imputs R x (where N x W), over which slot attention performed.
The set of K slots E RRXD is then randomly initialized (from shared distribution with learned parameters), and a form of transformer-style cross-attention is employed to attend over the pixels of the feature map. Specifically, each slot emits a query q(slots) E R (through a linear projection), and each location in the feature map emits a key k k(inputs) E I N and value v(inputs) E I Nx For each slot, a query-key attention operation is then used to generate an attention distribution over the feature map attn softmax( Jo k(imputs) q(slots) and
weighted mean of the values updates = attn v viinputs) is used to update the slot representations using Gated Recurrent Unit [8] followed by a residual MLP. This process is repeated for T iterations, leading to a competition between the slots to account for each pixel in the feature map.
For the purposes of computing relational embeddings, we employed factorized object representation format, in which position is represented distinctly from other object features. For a given input image, after performing T iterations of slot attention, used the final attention map attny E <RKN to compute feature- and position-specific embeddings for each slot:
2g = attny (fetten(feat)) [k]]
m. attny flatton (pos)
(2)
where feat and pos represent the feature- and position-embeedding maps (prior to being combined). As will be seen in the following section, this factorized representation allows visual space to serve as an indexing mechanism for pairwise relations, mirroring the central role of spatial organization in human visual reasoning
2
Although slot attention in principle be learned end-to-end in the service of a specific downstream reasoning task, here we chose to pre-train slot attention using an unsupervised objective. This decision reflects the assumption that human-likk like visual reasoning does not occur in a vacuum, and visual representations are generally not learned from scratch when performing each new task. Instead, visual object representations the effects of this prior experience, we pre-trained slot attention to reconstruct randomly generated multi-object displays, using slot-based autoencoder framework (more details in Section P.3.I] Importantly, for experiments testing generalization of abstract rules to novel objects, we ensured that these objects did not appear in the pre-training data for slot attention.
shaped by wealth of -pnsessc- prior experience. To model
2.2 Relational embeddings
After extracting object embeddings, we employed novel relational embedding method, designed to allow OCRA to abstract over the perceptual features of individual objects (and thus systematically
3

---

generalize learned rules to imputs with novel perceptual features). This embedding method consisted of two steps. First, we applied a relational operator & to compute the pairwise relation between two feature embeddings 2y: and zgz. In the present work, we define this operator as:
olz mR) = (z ZV Wr
(3)
where = and Z are first projected through a shared set of linear weights W₂ E DxD and the dot product between these weighted embeddings is then projected back out into the original dimensionality using another set of weights W, R I Using dot products to model pairwise relations introduces a layer of abstraction between the perceptual features of the objects and the downstream reasoning process, that the dot product only represents the relations between objects, rather than those objects' individual features. We hypothesized that this relational bottleneck would lead to improved generalization to inputs with novel perceptual features. Additionally, the shared projection weights (W₂) enable the model to learn which features are relevant for given task, and to only compute relations between those features.
After applying the relational operator, we then compute the relational embedding for objects k and ke' as:
TAk (Z Zxr) mg W., + my W
(4)
where the position embeddings my and my are each projected through a shared set of linear weight W.. m IR DxD and added the output of Φ. This allows the relation between objects and k' to be indexed by their spatial position, thus endowing OCRA with an explicit variable- criabbl-binding mechanism.
2.3 Processing of higher-order relations
After computing all 2 pairwise relational embeddings, we then pass these embeddings to reasoning process to extract higher-order relations patterns formed between multiple relations. This is an essential component of visual reasoning, abstract rules, such the one displayed in Figure [ are typically defined in terms of higher-order relations. We model these higher-order interactions using a transformer 45
y
(5)
where the inputs are formed by the set of all pairwise relational embeddings TEN (for all k = 1...K and all K' 1)..K), and y is task-specific output. Because we employ symmetric relational operator in the present work, we do not in practice present the full matrix of pairwise comparisons, but instead limit these comparisons to those in the upper triangular matrix.
3 Related Work
A number of methods for annotation- free object segmentation have recently been proposed 15| 28
[1] 20). including the slot attention method employed by our proposed model (28] Though the details of these methods differ, they share the general goal of decomposing visual scenes into a set of latent variables corresponding to objects, without access to ground truth segmentation data. Building on the success of these object-encoding methods, some recent work has developed object-centric approaches to visual reasoning that combine slot-based object representations with transformer- based architectures [0 39 5 30 This work has yielded impressive performance on challenging computer vision tasks, such as question answering from video [TO], or complex multi-object visual analogy problems 30, but has thus far been limited to generalization to similar problem types following extensive tssk-fpec training.
In the present work, our goal was to develop model capable of human- like systematic generalization in abstract reasoning tasks. Our proposed approach builds on the insights of other recent work in this
Note that the linear nature of this transformation is important: a nonlinear transformation of the position embeddings (e.g., via a MLP) could result in the extraction of implicit shape information, and therefore break the relational bottleneck.

---

area, most notably including the Emergent Symbol Binding Network (ESBN) [11 and CoRelNer 122 architectures, both of which employed the concept of a relational bottleneck in different ways. In the ESBN architecture, separate control and perception pathways interact indirectly through key- value memory, allowing the control pathway to learn representations that are abstracted over the details represented in the perceptual pathway. Kerg et al. [22) proposed that the ESBN's capacity for abstraction was due to the implicit encoding of relational information in its memory operations, which are mediated by a propprrouut-based retrieval operation. Kerg et al. also proposed novel architecture, CoReINet, that makes this inductive bias more explicit. The CoReINet architecture operates by explicitly computing the full matrix of relations (based on dot products) between each pair of objects in its input, and then passing these relations to an MLP for further processing. Both ESBN and CoRelNet displayed strong systematic generalization of learned abstract rules from small set of examples. Importantly, however, both of these architectures depended on the presence of pre-segmented objects input. A primary contribution of the present work is combine these inductive biases for relational abstraction (i.e., the relational bottleneck) with object-centric encoding mechanisms, and to introduce the use of spatial indexing mechanism that allows the model to apply these inductive biases to spatially embedded, multi-object visual inputs.
In addition to lacking object encoding mechanisms, it was recently shown that the ESBN can sometimes be overly brittle to task-irrelevant visual variation. Vaishnav and Serre [43] evaluated the ESBN and other reasoning architectures on abstract visual reasoning problems similar to those evaluated by Webb et al. [51], but with the introduction of either spatial jitter or Gaussian noise, finding that this significantly impaired performance in the ESBN (as well as other reasoning architectures, such as the transformer (45) and relation net 35
Vaishnav and Serre also introduced a novel architecture, GAMR, that was more robust these sources of noise, and able to solve problems based on multi-object inputs, likely due to the use of a novel guided attention mechanism. But despite outperforming ESBN and other architectures in this more challenging visual setting, GAMR nevertheless fell short of systematic generalization in these tasks, displaying a significant drop in performance in the most difficult generalization regimes. Here, we directly confront this challenge, demonstrating that OCRA is capable of sy systematic generalization comparable to that displayed by ESBN and CoRelNet, but in more complex visual settings.
It is also worth mentioning that other neural network architectures have been proposed that incorporate relational inductive biases in various ways 35 object-centric representations 48 approach is distinguished by the inclusion of stronger inductive bias toward relational abstraction that is, a relational bottleneck for promoting the development of relational representations that fully abstract over the details of individual objects [51 22 MI To empirically evaluate the importance of the relational bottleneck, we compare our approach with a set of baseline models that capture the key computational properties of previous models, including slot- frootrrnnformer baseline that combines slot attention with a transformer reasoning module [TO 39 52 30), and a
38
34
5, some of which also incorporate 41 Relative to this previous work, our
37
53
14
citterractionnetwott bascline that combines slot attention with an ineracctionnetweer reasoning
module 48
53 2
24
40, 46
To summarize contributions relative to previous work, our proposed model includes:
1. A novel object representation format that is factorized into distinct feature and position embeddings (Equations] and 2 enabling a form of explicit bardable-binding
aadd
2. A novel relational embedding method that implements relational bottleneck (Equations and
3. An architecture that combines these elements with preexisting components (slot attention and transformers) in a novel manner to support systematic visual reasoning from complex visual displays, including novel dataset CLEVR-ART with greater visual complexity.
4 Experiments
4.1 Datasets
We evaluated OCRA on two challenging visual reasoning tasks, ART [51 and SVRT [33 specifically designed to probe systematic generalization of learned abstract rules, as well as a novel dataset, ALEV involving more complex visual inputs (Figure 2
5

---

ART
SVRT
CLEVR-ART
Category
Category
b
Figure 2: Abstract visual reasoning tasks. Three datasets used to evaluate systematic visual reason- ing. Abstract Reasoning Tasks (ART) consist of four separate visual reasoning tasks; same/different, distribution-of-3, and identity rules (pictured), Systematic generalization is studied by training on problems involving a small number of objects (as few as 5), and testing on heldout objects. Synthetic Visual Reasoning Test (SVRT) consists of 23 tasks, each defined by separate abstract rule. Rules are defined based either spatial relations (SR) or same/different relations (SD; pictured). In example problem, category always contains three sets of two identical objects, category 2 always contains two sets of three identical objects. CLEVR-ART is novel dataset consisting of ART problems rendered using more realistic 3D shapes, based on the CLEVR dataset.
ART. The Abstract Reasoning Tasks (ART) dataset was proposed by Webb et al. 55.. consisting of four visual reasoning tasks, each defined by different abstract rule (Figure (S1). In the 'same/different' task, two objects are presented, and the task is to say whether they are the same or different. In the 'relational match-to-sample" task, a source pair of objects is presented that either instantiates 'same' or 'different" relation, and the task is to select the pair of target objects (out of two pairs) that instantiates the same relation. In the "distributionoof-3"" task, a set of three objects is presented in the first row, and an incomplete set is presented in the second row. The task is to select the missing object from set of four choices. In the 'identity rules' task, an abstract pattern is instantiated in the first row (ABA, ABB, or AAA), and the task is to select the choice that would result in the same relation being instantiated in the second row.
The primary purpose of this dataset was to evaluate strong systematic generalization of these rules. Thus, each task consists of number of generalization regimes of varying difficulty, defined by the number of unique objects used to instantiate the rules during training (out of a set of 100 possible objects). In the most difficult regime (m = 95), the training set consists of problems that created using only 5 objects, and the test problems are created using the remaining 95 objects, whereas in the easiest regime (m 0), both training and test problems are created from the complete set of 100 objects (though the arrangement of these objects is distinct, such that the training and test sets are still disjoint). The most difficult generalization regime poses an especially difficult test of systematic generalization, as it requires learning of an abstract rule from a very small set of examples (details shown in Table ST with little perceptual overlap between training and test.
As originally proposed [5T] the ART dataset involved pre-segmented visual objects. Here, we investi- gated version of this task involving multi-object visual displays (see Supplementary Section S2 for details). We also applied random spatial jitter (random translation of up to 5 pixels in any direction) to each object, as this was previously found to significantly impair performance in some previous reasoning models
SVRT. The Synthetic Visual Reasoning Test (SVRT), proposed by Fleuret et al. B consists of 23 binary classification tasks, each defined by a particular configuration of relations. The tasks can be broadly divided into two categories: those that are based primarily
same/different relations ('SD'; tasks 7, 21, 5, 19, 6, 17, 20, 1, 13, 22, 16; example shown in Figure SB(a)). and those that primarily based on spatial relations ('SR"; tasks 12, 15, 14, 9, 23, 8, 10, 18, 4, 3, 11, 2; example shown in Figure b3(b). As with previous work 45 we limit training to a relatively small number of examples (either 500 1000 per task).
CLEVR-ART. We created a novel dataset based on ART using realistically rendered 3D shapes from CLEVR 2I (Figure We focused two specific visual reasoning tasks: relational-matele

---

to-sample and identity rules. Problems were formed from objects of three shapes (cube, sphere, and cylinder), three sizes (small, medium, and large), eight colors (gray, red, blue, green, brown, purple, cyan, and yellow) and two textures (rubber and metal). To test systematic generalization, the training set consisted of problems using only small and medium sized rubber cubes in four colors (cyan, brown, green, and gray) whereas the test set consisted only of problems using large sized metal spheres and cylinders in four distinct colors (yellow, purple, blue, and red). The training and test sets thus involved completely disjoint perceptual features, similar to the original ART.
4.2 Baselines
We compared our model to results from set of baseline models evaluated by Vaishnav and Serre [ including their GAMR architecture, ResNet50 117. and version of ResNet that employed self- attention (Attn-ResNet) 122. To the best of our knowledge, these baselines represent the best performing models on the tasks that we investigated. We also investigated an additional set of baselines that combined our pretrained slot attention module with various reasoning architectures,
including GAMR 43, ESBN [51), Relation Net (RN) (5), Transformer 45]. Interaction Network (IN) [48] and CoRelNet 12 These baselines benefited from the same degree of pre-training and object-centric processing as our proposed model, but employed different reasoning mechanisms. Finally, we investigated a supe--up baseline (a Masked Autoencoder (MAE) (166) to assess the extent to which a generative objective might encourage systematic generalization (see Tables Tab $14 for results). All baselines were evaluated using multi-object displays. See Supplementary Section S for more details on baselines.
4.3 Experimental Details
4.3.1 Pre-training slot attention
We pre-trained slot attention using a slot-based autoencoder framework. For ART, we generated a dataset of random multi-object displays (see Figure (ig utilizing publicly available repository of unicode character image: We eliminated any characters that were identical to, or closely resembled, the 100 characters present in the ART dataset, resulting in training set of 180k images, and validation set of 20k images. For SVRT, we pre-trained two versions of slot attention on all tasks, one using 500 and the other using 1000 training examples from each of the 23 tasks. For CLEVR- we generated a dataset of randomly placed 3D shapes from CLEVR, and pre-trained slot attention using a training set of 90k images, and validation set of 10k images.
To pre-train slot attention, we used a simple reconstruction objective and slot-based decoder. Specifically, we used spatial broadcast decoder [47]. which generates both reconstructed image and a mask for each slot. We then generated a combined reconstruction, normalizing the masks across slots using softmax, and using the normalized masks to compute weighted average of the slot-specific reconstructions.
After pre-training, the slot attention parameters were frozen for training on the downstream reasoning tasks. For ART, we found that despite not having been trained on any of the objects in the dataset, slot attention was nevertheless able to generate focused attention maps that corresponded closely to the location of the objects (see Figures S6 S9p.
4.3.2 Training details and hyperparameters
Images were resized 128 x 128, and pixels were normalized to the range [0, 1]. For SVRT, we converted the images to grayscale, and also applied random online augmentations during training. including horizontal and vertical flips. Tables S2 and S3 describes the hyperparameters for the convolutional encoder and spatial broadcast decoder respectively. For slot attention, we used K = 6 (K = for CLEVR-ART) slots, 3 attention iterations per image, and a dimensionality of D 64. Training details are described in Table S4
Table gives the hyperparameters for the transformer. To solve the SVRT tasks, and the same/different ART task, we used CLS token (analogous to CLS token in ( together with a sigmoidal output layer to generate a binary label, and trained the model with binary cross-entropy
7

---

Same/different
Relational matpleto-sample
Slot-Transformer
Slot-IN
60
60
Slot-RN
Slot-GAMR
Slot: Slot-ESBN
40
40
Slot-CoReiNet
GAMR
20
20
ResNet
85
85
(objects withheld during training)
(objects withheld during training)
Distribution-of-3
Identity rules
80
Accuracy
60
60
40
40
20
20
0
85 (objects withheld during training)
85
(objects withheld during training)
Figure 3: ART results. Results on the Abstract Reasoning Tasks dataset [5]]. OCRA outperformed all baselines for all tasks, with especially strong performance in the most difficult generalization regime (m = 95). Results reflect an average across 10 runs for each model + standard error. The X axis denotes the number objects (out of 100) withheld during training, with no objects withheld in the leftmost condition (m = 0), and almost all objects withheld in the rightmost condition (m = 95).
loss. To solve the RMTS, distribution-of-3, and identity rules tasks, inserted each candidate answer choice into the problem, and used a CLS token together with a linear output layer to generate score. The scores for all answer choices were passed through softmax and the model was trained with cross-entropy loss. For ART and CLEVR-ART, we used a default learning rate of Se 5, and a batch size of 16. The number of training epochs is displayed for each ART task in Table S66 and for each CLEVR-ART task in Table STO The training details for some baseline models were modified as discussed in Section S3 and detailed in Tables S7 59 For SVRT, used learning rate of 4e
a batch size of 32, and trained for 2000 epochs on each task. We used the ADAM optimizer
23 and
implementation was done using the Pytorch library
4.3.3 Model selection
After pre-training slot attention for a fixed number of epochs (number of epochs for each dataset listed Table 54), we selected the version of the model from the epoch with the lowest validation loss (on the unsupervised pre-training task) to use in the visual reasoning tasks, We used a validation set of 20k images for ART (again excluding the objects used in the ART tasks), 4k images per task for SVRT, and 10k images for CLEVR-ART.
When training the primary reasoning tasks, for SVRT, trained for 2000 epochs each task, and selected the version of the model from the epoch with the highest accuracy on the validation set. This is consistent with the approach taken by Vaishnay and Serre [3], who used validation accuracy as a criterion for early stopping. For ART and CLEYR-ART, we did not perform any model (or epoch) selection when training on the primary reasoning tasks.
5 Results
Figure 3 shows the results on the ART dataset (all results are presented in tabular form in Ta- bles ST7 OCRA achieved state-of-the-arr results across all tasks and generalization regimes, outperforming both the previous the model (GAMR), and other strong baselines such as
8

---

CLEVR-ART
SVRT
Same/different
Spatial relations
200
100
OCRA
Slot-Transformer
80
80
80
Slot-IN
Slot-RN
Accuracy
Accuracy
60
Accuracy
60
Slot-GAMR
Slot-ESBN
Slot-CoReiNet
GAMR
20
20
20
Attn-ResNet
ResNet
RMTS
Identity riles
1000
500
1000
500
Task
Number training examples
Number of training examples
Figure 4: CLEVR-ART. Results for systematic generalization on CLEVR-ART tasks. Results reflect test accuracy averaged over 5 trained networks + standard error. SVRT. Results on the Synthetic Visual Reasoning Test (SVRT) [. OCRA and GAMR showed comparable overall performance, with OCRA peforming marginally better on problems involving same different relations, and GAMR performing marginally better on problems involving spatial relations.
the flot-rransformer. This disparity was especially striking in the most difficult generalization regime (m -95), in which models were trained on only 5 objects, and tested on heldout set of 95 objects. Importantly, no component of OCRA (including both the slot attention and relational reasoning components) had been exposed to these heldout objects. OCRA outperformed the next-best baselines in this regime (GAMR and (rotforasformer) by as much as 25%, demonstrating a significant degree of systematic generalization. We also analyzed OCRA's learned relational embeddings, finding that they captured relational information even in this out-of-distribution setting (Figure S5)
Many of the baseline models that we evaluated did not contain relational bottleneck (Transformer, IN, RN, GAMR, ResNet), limiting their ability to generalize relational patterns to novel objects (i.c., higher values of m). The comparison with these models thus demonstrates the importance of the relational bottleneck, which enables systematic generalization of learned relational patterns. The ESBN and CoRelNet architectures did include a relational bottleneck, but they were not designed with multi multi-object inputs in mind, and therefore performed poorly on these tasks in all regimes (even when combined with our slot attention module). This is due primarily to the random permutation of slots in slot attention, which motivated our positional embedding variable-binding scheme (allowing OCRA keep track of which relations correspond to which pairs of objects).
On SVRT, OCRA showed comparable overall performance with GAMR, displaying marginally better performance on tasks characterized by same/different relations, while GAMR performed marginally better on tasks involving spatial relations (Figure 4 right). OCRA also outperformed GAMR on CLEVR-ART (Figure 4 left), demonstrating that its capacity for systematic generalization can be extended to problems involving more realistic visual inputs. Overall, the results on ART, SVRT, and CLEVR-ART demonstrate that OCRA is capable of solving diverse range of relational reasoning problems from complex multi-object displays.
5.1 Ablation Study
We performed an ablation study targeting each of OCRA's major components, focusing on the most difficult regime of the relational match-to-sample and identity nules tasks (Table M. To evaluate the importance of OCRA's capacity for object-centric processing, we tested a version of the model without slot attention ('w/o Slot Attention") "), which the visual feature map was instead divided into a 4 x 4 grid, treating each image patch as though it were an 'object'. We also tested a version in which slot attention was only trained end-to-end on the visual reasoning tasks ('w/o Pre-training"), to test the importance of prior exposure to diverse range of multi-objer displays. To evaluate the importance of OCRA's relational embedding method, we tested five different ablation models. In one model ('w/o Relational Embeddings'), ), slot embeddings were passed directly to the transformer. In another model ('w/o Factorized Representation') ) we did not compute separate feature and position embeddings for each object, and instead applied the relational operator directly to the slot embeddings. In a third model ('w/o Relational Bottleneck'), ) relational embeddings were computed simply by applying MLP to each pair of slot embeddings (allowing the model to overfit to the perceptual

---

Table 1: Ablation study on m 95 generalization regime for two ART tasks. Results reflect test accuracy averaged over 10 trained networks + standard eπтor.
RMTS
ID
OCRA (full model)
85.31 + 2.0
92.80-10.3
w/o Slot Attention
56.58
84.42.+..1
w/o Pre-training
58.69
53.48
w/o Relational Embeddings
74.79
81.21
w/o Factorized Representation
50.26
75,40+10.6
w/o Relational Bottleneck
63.62.1..
76.56
w/o Inner Product
50.70+0.4
48.07+22.6
Replace Inner Product w/ Outer Product
69.58+1.1 1..8411.6
w/o Transformer
51.29+0.2
44.18+0.6
details of these embeddings), thus removing OCRA's inductive bias for relational abstraction. In fourth model ('w/o Inner Product') the dot product in the relational operator (Equation B was replaced with learned bottleneck, in which each pair of slot embeddings was passed to learned linear layer with a one-dimensional output. This model tested the extent to which the inherently relational properties of the dot product are important for relational abstraction (as opposed to resulting merely from compression
single dimension). In a fifth model ('Replace Inner Product w/ Outer Product'), the dot product was replaced with outer product (resulting in D x D matrix, which was then flattened and projected via a learned linear layer to an embedding of size D). Finally, to evaluate the importance of OCRA's capacity to process higher-order relations, instead of passing relational embeddings to a transformer, we instead summed all relation embeddings elementwise and passed the resulting vector to an MLP, is done in the Relation Net ('w/o Transformer'). All ablation models performed significantly worse than the complete version of OCRA, demonstrating that all three elements in model object-centric processing, relational abstraction, and higher-order relations - play an essential role in enabling systematic generalization.
Limitations and Future Directions
In the present work, we have presented model that integrates object-centric visual processing mechanisms (providing the ability to operate over complex multi-object visual inputs) with a relational bottleneck (providing strong inductive bias for learning relational abstractions that enable human- like systematic generalization of learned abstract rules). Though this is a promising step forward, and clear advance relative to previous models of abstract visual reasoning, it is likely that scaling the present approach to real-world settings will pose additional challenges. First, real-world images are especially challenging for object-centric methods due relative lack of consistent segmentation cues. However, there has recently been significant progress in this direction Eo., in particular by utilizing representations from large-scale self-supervised methods [54 7 and should be possible integrate these advances with our proposed framework. Second, the current approach assumes fixed number of slot representations, which may not be ideal for modeling real-world scenes with highly variable number of objects (55] Though we did not find that this was an issue in the present work, in future work it would be desirable to develop method for dynamically modifying the number of slots. Third, OCRA's relational operator is applied to all possible pairwise object comparisons, an approach that may not be scalable scenes that contain many objects. In future work, this may be addressed by replacing the transformer component of our model with architectures that are better designed for high- -igeesnnnnii inputs [8]. Finally, it unclear how our proposed approach may fare in settings that involve more complex real-world relations, and settings that require the discrimination of multiple relations at once. It may be beneficial in future work to investigate a 'multi-head' version of our proposed model (analogous to multi-head attention), in which multiple distinct relations are processed parallel. A major challenge for future work is to develop models that can match the human capacity for structured visual reasoning in the context of complex, real-world visual inputs.
10

---

7 Acknowledgements
Shanka Subhra Mondal was supported by Office of Naval Research grant N00014-22-1-2002 during the duration of this work. We would like to thank Princeton Research Computing, especially William G. Wischer and Josko Plazonic, for their help with scheduling training jobs on the Princeton University Della cluster.
References
[1] A. Altabaa, T. Webb, J. Cohen, and J. Lafferty. Abstractors: Transformer modules for symbolic message passing and relational reasoning. arXiv preprint arXiv:2304. 00195, 2023.
[2] V. Bapst, A. Ganzaez- C. Doersch, Stachenfeld, Kohli, Battaglia, and Ham- rick. Structured agents for physical construction. In International conference machine learning, pages 464 474. PMLR, 2019.
[3] D. Barrett, Hill, A. Santoro, A. Morcos, and Lillicrap. Measuring abstract reasoning in neural networks. In International conference machine learning, pages 5-1-520. PMLR. 2018.
[4] W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzale V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: /80.00/60, 2018.
[5] L. Bergen, T. O'Donell, and D. Bahdanau. Systematic generalization with edge transformers. Advances in Neural Information Processing Systems, 34:1390-1402, 2021.
[6] C. P. Burgess, Matthey, N. Watters, Kabra, Higgins, M. Botvinick, and Ler- chner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 1390, 2019.
[7] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[8] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. 1078, 2014.
[9] J. Devlin, M.-W. Chang, K. Lee, and K. Toottanoa. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018
[10] D. Ding, F. Hill, Santoro, M. Reynolds, and M. Botvinick. Attention over learned object embeddings enables complex visual reasoning. Advances in neural information processing systems, 34:91 12-914,, 2021.
[11] A. Dittadi, S. Papa, M. De Vita, B. Schölkopf, O. Winther, and Locatello. Generalization and robustness implications in object-centric learning. arXiv preprint arXiv:2107.00637 2021.
[12] M. Engelcke, O. Parker Jones, and I. Posner. Genesis-v2: Inferring unordered object repre- sentations without iterative refinement. Advances in Neural Information Processing Systems, 34:8085 8094,,8004,400 2021.
[13] F. Fleuret, T. Li, C. Dubout, E. K. Wampler, S. Yantis, and D. Geman. Comparing machines and humans on visual categorization test. Proceedings of the National Academy of Sciences, 108(43):16211-1723 2011.
[14] A. Gopalakrishnan, S. van Steenkiste, and Schmidhuber. Unsupervised object keypoint learning using local spatial predictability. arXiv preprint arXiv:2011 1.X::20011.12930, 2020.
[15] K. Greff, R. L.. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and Lerchner. Multi-object representation learning with iterative variational inference. In International Conference Machine Learning, pages 2424-2433 PMLR, 2019.

---

[16] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[18) A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651-4664. PMLR, 2021.
[19] M. Janner, S. Levine, W. T. Freeman, J. B. Tenenbaum, C. Finn, and J. Wu. Reasoning about physical interactions with object-oriented prediction and planning. arXiv preprint arXiv:1812.10972, 2018.
[20] J. Jiang, F. Deng, G. Singh, and S. Ahn. Object-centric slot diffusion. arXiv preprint arXiv:2303. 10834, 2023.
[21] Johnson, B. Hariharan, Van Der Maaten, Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901- 2910, 2017.
[22] G. Kerg, S. Mittal, D. Rolnick, Y. Bengio, B. Richards, and G. Lajoie. On neural architecture inductive biases for relational tasks. arXiv preprint arXiv:2206. 05056, 2022.
[23] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[24] T. Kipf, E. Van der Pol, and M. Welling. Contrastive learning of structured world models. arXiv preprint arXiv: 191 122477 2019.
[25] L. Kotovsky and D. Gentner. Comparison and categorization in the development of relational similarity. Child Development, 6776):27772 1996.
[26] B. Lake and M. Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequenne recurrent networks. In International conference on machine learning, pages 2873-2882. PMLR, 2018
[27] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266); 1332-1338, 2015.
[28] Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, Uszkoreit, A. Doso- vitskiy, and T. Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:1 11522-11533 2020.
[29] B. J. Matlen, D. Gentner, and S. L. Franconeri. Spatial alignment facilitates visual comparison. Journal of Experimental Psychology: Human Perception and Performance, 46(5):443, 2020.
[30] S. S. Mondal, T. W. Webb, and J. D. Cohen. Learning to reason over visual objects. In //th International Conference on Learning Representations, ICLR, 2023.
[31] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.
[32] J. C. Raven. Progressive matrices; A perceptual test of intelligence, individual London: Lewis, 1938.
[33] M. Ricci, J. Kim, and T. Serre. Same-different problems strain convolutional neural networks. arXiv preprint 1eXiv:1802.03333 2018.
[34] A. Santoro, R. Faulkner, D. Raposo, J. Rae, M. Chrzanowski, T. Weber, D. Wierstra, O. Vinyals, R. Pascanu, and T. Lillicrap. Relational recurrent neural networks. Advances neural information processing systems, 31, 2018.
12

---

[35] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap. simple neural network module for relational reasoning. Advances in neural information processing systems, 30, 2017.
[36] M. Seitzer, M. Horn, A. Zadaianchuk, D. Zietlow, T. Xiao, C.-J. Gabbie,,, T. He, Z. Zhang, B. Schölkopf, T. Brox, et al. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209. 14860, 2022.
[37] M. Shanahan, K. Nikiforou, A. Creswell, C. Kaplanis, D. Barrett, and M. Garnelo. An explicitly relational neural network architecture. In International Conference on Machine Learning, pages 8593 8603. PMLR, 2020.
[38] P. Shaw, J. Uszkoreit, and A. Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:8003. 02155,, 2018.
[39] G. Singh, Y.-F. Wu, and S. Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. arXiv preprini arXiv:2205. 14065, 2022.
[40] A. Stanić and J. Schmidhuber. R-sqair: relational sequential attend, infer, repeat. arXiv preprint arXiv:1910.0523: 2019.
[41] A. Stanić, S. van Steenkiste, and J. Schmidhuber. Hierarchical relational inference. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9730-9738, 2021.
[42] M. Vaishnav, R. Cadene, A. Alamia, D. Linsley, R. VanRullen, and T. Serre. Understanding the computational demands underlying visual reasoning. Neurul Computation, 3455):1075-1099 2022.
[43] M. Vaishnav and T. Serre. Gamr: A guided attention model for (visual) reasoning. In 11th International Conference on Learning Representations, ICLR, 2023.
[44] S. Van Steenkiste, M. Chang, K. Greff, and J. Schmidhuber. Relational neural expecta- tion maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv: 1802. 10353, 2018.
[45] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[46] R. Veerapaneni, D. Co-Reyes, M. Chang, M. Janner, C. Finn, J. Wu, Tenenbaum, and S. Levine. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pages 1439-1456 PMLR, 2020.
[47] N. Watters, L. Matthey, C. P. Burgess, and A. Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:1901.00017 2019.
[48] N. Watters, D. Zoran, T. Weber, P. Battaglia, R. Pascanu, and A. Tacchetti. Visual interaction networks: Learning a physics simulator from video. Advances in neural information processing systems, 30, 2017.
[49] T. Webb, Z. Dulberg, S. Frankland, A. Petrov, R. O'Reilly, and J. Cohen. Learning represen- tations that support extrapolation. In International conference on machine learning, pages 10136-10166. PMLR, 2020.
[50] T. W. Webb, S. M. Frankland, A. Altabaa, K. Krishnamurthy, D. Campbell, J. Russin, R. O'Reilly, J. Lafferty, and J. D. Cohen. The relational bottleneck as an inductive bias for efficient abstraction. arXiv preprint arXiv:2309.06629, 2023.
[51] T. W. Webb, I. Sinha, and J. D. Cohen. Emergent symbols through binding in external memory. In 9th International Conference on Learning Representations, ICLR, 2021.
[52] Z. Wu, N. Dvoriik, K. Greff, Kipf, and A. Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.0586) 2022.
13

---

[53] V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. Reichert, Lillicrap, E. Lockhart, et al. Relational deep reinforcement learning. arXiv preprint arXiv: 1806.01830, 2018.
[54] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint 1..0222 2021.
[55] R. S. Zimmermann, S. van Steenkiste, M. S. Sajjadi, T. Kipf. and K. Greff. Sensitivity of slot-based object-centric models to their number of slots. arXiv preprint arXiv: 2023.
14


Reviews:
[{"review_id": "azaVEnlYft", "reviewer": "Reviewer_bzvn", "summary": "This work proposed a new method, named OCRA, that combines object-centric presentation learning (for object abstraction) and a relational bottleneck (for relational abstraction). Particularly, OCRA consists of three components: 1) a slot attention to extract object level representations, 2) a relational operator to get pairwise visual relations, and 3) a transformer to model higher-order relations. The slot attention model is pretrained on a large dataset and the relational modules are trained on a small task-specific dataset while freezing the slot attention. Experiments were performed on three datasets: ART, SVRT and CLEVR-ART to show the effectiveness of the proposed method.", "questions": "My major concerns and questions are in how the experiments support the effectiveness and significance of the proposed method. Please see the weaknesses part for more details.", "limitations": "The authors have well addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of combining slot attention with a relational bottleneck (relation operator and transformer) for solving visual relational reasoning problems sounds interesting and also novel to me. \n2. The presentation of the idea and the overall writing are very clear. \n3. Experiments on synthetic datasets (ART and SVRT and CLEVR-ART) shows the proposed method can work for various relational reasoning tasks and it achieves better performance than baselines in many cases.", "weaknesses": "1. For the benchmarks, I have a concern about their simplicity. For example, the performance of many methods on the ART dataset is close to 100%. Similar observations also exist in other two datasets. Does it mean we already have achieved human-level visual reasoning performance or we need better benchmarks to evaluate the success of methods? I think considering some more challenging benchmarks will make the results more convincing. For example, RAVEN [1] and Bongard-HOI [2] are two challenging benchmarks for testing a model's visual relational reasoning abilities. In particular, Bongard-HOI considers the real-world natural images.\n2. In Figure 4, it is a little surprising that OCRA performs worse than ResNet on SVRT - spatial relations with 1000 training examples. Any intuition on this?\n3. For the ablation studies, to test the impact of slot attention, it is too naive to do “feature map divided into a 4x4 grid”. How about comparing slot attention with some standard SSL trained object-centric representation learning methods? Also, from Table 2, we can see that Transformer has a much higher impact than the slot attention and relational bottleneck on both RMTS and ID. Does it mean the most important part of OCRA is the transformer rather than slot attention and relational bottleneck? If so, it downplays the method's significance a little, in my opinion.\n\n[1] RAVEN: A Dataset for Relational and Analogical Visual rEasoNing, CVPR 2019.\n\n[2] Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions, CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful suggestions and comments. We provide a point-by-point reply below:\n\n1. Synthetic vs. real-world benchmarks: \n    - The primary focus of the benchmarks we consider is on the more extreme **out-of-distribution generalization and low-sample regimes**. In the most extreme generalization regime of the ART dataset ($m=95$ in Figure 1 of the attached PDF above), **most baselines perform very poorly, with many displaying near-chance performance**. The baseline performance on CLEVR-ART and the same/different SVRT tasks (especially in the low-sample regime, with only 500 training examples) is similarly poor. Thus, though these tasks don’t involve real-world images, they are nevertheless extremely challenging tests of out-of-distribution visual reasoning.\n    - We also agree with the reviewer that it is important to extend models of visual reasoning so that they are capable of processing real-world visual inputs. We believe that this work takes a significant step in that direction, by proposing a model capable of strong **OOD generalization from multi-object inputs (ART and SVRT) and even photorealistic three-dimensional objects (CLEVR-ART)**. We agree that it will be important in future work to take this even further, by evaluating the model on real-world images, utilizing datasets such as Bongard-HOI. Though this poses a few challenges (which we discuss in the general response above), we do not see any fundamental obstacles to extending our model to these kinds of datasets, and plan to do so in future work.\n2. Relatively poor performance on SVRT - spatial relations: \n    - We found that OCRA’s poor relative performance on these tasks was related to the fact that the learned slot representations were not perfectly object-centric. This was partially remedied by training slot attention for longer. With further pretraining of slot attention, OCRA outperforms ResNet in the low-sample regime (500 training examples), and performs on par with ResNet in the high-sample regime (1k training examples; see Figure 3 in the attached PDF above). Note also that OCRA’s performance on these tasks is generally strong (around 90% test accuracy), and higher than it is for the same/different tasks (which were extremely challenging for many of the baselines). \n3. Testing impact of slot attention and relational bottleneck:\n    - To better test for the impact of slot attention, the reviewer proposes that we compare OCRA with a ‘standard SSL trained object-centric representation learning method.’ We are not sure which specific methods the reviewer has in mind. The purpose of the ‘- Slot Attention’ ablation is precisely to test for the impact of using object-centric representations, so this ablation model should not be object-centric. However, we agree that it is informative to compare to other self-supervised representation learning methods. We have now added a comparison with a state-of-the-art masked autoencoder model (MAE; using the code base from [1]) on the identity rules and distribution-of-3 ART tasks (it is not clear how to formulate the same/different and RMTS tasks in a generative manner). We applied this model to our tasks by masking out the final object in each problem (in the bottom right cell of the input), and training the model to fill in this patch. To select from the set of multiple choices, we then compared the model’s generated output with the four answer choices, and selected the choice with lowest mean-squared error. We found that this model performed considerably worse than OCRA, especially in the most extreme generalization regimes ($m=95$). We have included the results in a table below, and will also add them to the revised paper. If the reviewer had other methods in mind, we would be happy to test these and compare with our model if time permits.\n    - The reviewer notes that the most impactful ablation is the ‘- Transformer’ model. This demonstrates that higher-order relations (relations between pairwise relations, which are extracted by the transformer in OCRA) are essential to the model’s performance. However, we note that ablating either slot attention or the relational bottleneck also has a severe effect on performance. For instance, in the RMTS task, performance drops by ~29% without slot attention, and by ~22% without a relational bottleneck. This demonstrates that all major elements of the model (object-centric representations, relational abstraction, higher-order relations) play an important role.\n\n\n| Distribution-of-3 | $m=0$ | $m=50$ | $m=85$ | $m=95$ |\n| ------- | ------ | ------- | ------- | ------- |\n| OCRA (ours) | **98.86 ± 0.2** | **97.87 ± 0.2** | **96.09 ± 0.4** | **86.42 ± 1.2** |\n| MAE | **99.99 ± 0.0** | 56.47 ± 1.1 | 40.90 ± 1.2 | 28.85 ± 0.9 |\n\n| Identity rules | $m=0$ | $m=50$ | $m=85$ | $m=95$ |\n| ------- | ------ | ------- | ------- | ------- |\n| OCRA (ours) | **99.01 ± 0.0** | **98.01 ± 0.1** | **96.67 ± 0.2** | **92.80 ± 0.3** |\n| MAE | 66.55 ± 0.2 | 44.96 ± 0.6 | 38.39 ± 0.9 | 31.56 ± 1.0 |\n\n[1] He et al. (2022). Masked autoencoders are scalable vision learners. \n\n\n\n"}, {"review_id": "19jg1BhbYs", "reviewer": "Reviewer_1vcp", "summary": "The research topic of this study is the development of a learning machine that achieves systematic generalization in reasoning over complex relations of objects in a visual input (still image). Toward this goal, the authors proposed a new neural network model (OCRA) taking inspirations from the recent results on effective inductive biases for systematic generalization in relational reasoning and methods for obtaining object-centric representations. More concretely, the proposed model comprised of three core components: first component (slot attention mechanism) to extract object-centric representation from a visual input containing multiple objects, the second component to compute pairwise relation embeddings, and the third component (transformer) to provide the final output related to the higher-order relations. The effectiveness of the proposed model was tested with three visual reasoning tasks, two existing and one new, and also compared with various baseline models. As a whole, the proposed model can be said the best in terms of systematic generalization among the compared models.  An ablation study was also conducted to evaluate the roles of the components in the proposed model and pretraining.  ", "questions": "Questions\n\n1. Can the source code of the proposed method be provided at the Author Rebuttal period? In addition, will the source code made publicly available if the paper get accepted?\n\n1.  What are your thoughts on the second point in Weaknesses section above? \n\nSuggestion\n\nIt would be beneficial if the authors could add detailed descriptions about what kind of differences between the problems treated in this study and the real-world vision should be overcome, and also add explanations about the relation between those differences and  self-supervised learning methods mentioned in Section 6 if possible.  ", "limitations": "The authors stated in Section 6 (Conclusion and Future Directions) that the problems addressed in this study are still simple compared with the the real-world vision.  However, currently it is not explained in details what kind of differences between the problems treated in this study and the real-world vision should be overcome. Please also refer to the Weaknesses and Questions sections above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The top-level idea behind the proposed model, that is, combining the inductive biases for relational reasoning and a method for obtaining object-centric representations is clear and reasonable. How to implement the idea with the three core components are explained fairly well. Although the top-level idea might look somewhat straightforward at first glance given the advancement in the two directions, the concrete implementation is not trivial, and the differences from the existing work are described in the paper. \n\nThe effectiveness of the proposed model was tested with fairly rich experiments. Two existing task (ART and SVRT, both created with 2D shapes) and a new task developed based on the CLEVR (CLEVR-ART,  created with 3D shapes) were used and various baseline models including a very recent one (GAMR, accepted at this year's ICLR) are compared with the proposed model. An ablation study gives additional value to this work. \n\nDevelopment of a learning machine that achieves systematic generalization in complex visual reasoning is an important topic in AI. Although there is still a distance to the real-world applications as stated in Section 6 (Conclusion and Future Directions), the proposed method and the evaluation results will be of interest to the NeurIPS audience. ", "weaknesses": "1. A weak point of this submission is lack of the source code of the proposed model as Supplementary Material. It is also unkown whether the source code will be made publicly available if the paper get accepted. These points cast a shadow on the reproducibility.\n\n1. A relatively weak point of the proposed model itself seems to be around the number of slots, $K$. First, as stated in line 117, the proposed model computes all $K^2$ pairwise relations. Second, in the current model, $K$ should be defined in advance. These characteristics can be obstacles in the reasoning over the relations of objects in the real-world visual inputs. Namely, the fist point might affect the applicability (scalability) to complex natural images and there should be some additional mechanism to determine an appropriate $K$ in the first place if the current structure of the proposed model is kept. \n\n1. (Related to 2.) Although it is stated that there exists gap between the problems addressed in this study and the real-world vision in Section 6 (Conclusion and Future Directions), the details are not explained.  \n\n   \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful suggestions and comments. We provide a point-by-point reply below:\n\n1. Lack of source code:\n    - We thank the reviewer for bringing this to our attention. **We have now provided the AC with an anonymized link to the source code for our model. We will also make this code publicly available upon acceptance of the paper.**\n2. Fixed number of slots and scalability:\n    - A desirable feature of slot attention is that the number of slots can be set to the maximum number of expected objects, and slots that don’t correspond to objects will remain largely unutilized (e.g. if there are 6 slots but only 2 objects in an image, typically only 2 slots will end up being used). OCRA inherits this property from slot attention. Therefore, in the ART dataset, in which different problem types contain different numbers of objects (ranging between 2 and 6), we set the number of slots to $K=6$ for all problems. However, we agree that an ideal approach would involve autonomously adjusting the slots based on the number of objects present in an image. We have added some additional discussion of this issue to the paper (see excerpt from revised text below).\n    - We also agree that computing all pairwise relations may pose a challenge when scaling to real-world scenes with many objects. In future work, we believe that this may be addressed by replacing the transformer component of our model with other methods that are better equipped to deal with very high-dimensional inputs, e.g. [1]. We have added additional discussion of this issue to the revised paper (see excerpt below).\n3. Lack of detail on obstacles to scaling:\n    - We agree that this issue deserves further attention.  We have revised the discussion of these issues in the conclusion of the paper as follows (new text in bold, note also reference numbers are not the same as those in the paper):\n\n        ‘In the present work, we have presented a model that integrates object-centric visual processing mechanisms (providing the ability to operate over complex multi-object visual inputs) with a relational bottleneck (providing a strong inductive bias for learning relational abstractions that enable human-like systematic generalization of learned abstract rules). Though this is a promising step forward, and a clear advance relative to previous models of abstract visual reasoning, it is likely that scaling the present approach to real-world settings will pose additional challenges. **First, real-world images are especially challenging for object-centric methods due to a relative lack of consistent segmentation cues. However, there has recently been significant progress in this direction [2], in particular by utilizing representations from large-scale self-supervised methods [3], and it should be possible to integrate these advances with our proposed framework. Second, the current approach assumes a fixed number of slot representations, which may not be ideal for modeling real-world scenes with a highly variable number of objects [4]. Though we did not find that this was an issue in the present work, in future work it would be desirable to develop a method for dynamically modifying the number of slots. Finally, OCRA’s relational operator is applied to all possible pairwise object comparisons, an approach that may not be scalable to scenes that contain many objects. In future work, this may be addressed by replacing the transformer component of our model with architectures that are better designed for high-dimensional inputs [1].**’\n\n[1] Jaegle et al. (2021). Perceiver: General perception with iterative attention.\n\n[2] Seitzer et al. (2022). Bridging the gap to real-world object-centric learning. \n\n[3] Caron et al. (2021). Emerging properties in self-supervised vision transformers. \n\n[4] Zimmermann et al. (2023). Sensitivity of Slot-Based Object-Centric Models to their Number of Slots.\n\n"}, {"review_id": "BconLBGNxZ", "reviewer": "Reviewer_HzsY", "summary": "- The paper seeks to tackle various visual reasoning problems with a focus on systematic generalization.\n- The paper argues that we need explicit inductive bias to extract object-object relationships and express these relationships via a low-capacity representation.\n- The paper seeks to do this by first extracting object vectors (or slots) using pre-trained slot attention. It then performs a dot-product of these slots to obtain a “relation embedding” — which is really a scalar number expressed as an embedding (as far as I understand). These relation embeddings are then given to a transformer and the whole model (except the slot attention) is trained to classify whether the given input is conformant to the true pattern or not.\n- The performance is then evaluated on a systematically OOD test set and compared with various baselines and model ablations on 3 visual reasoning benchmarks: ART, SVRT, and CLEVR-ART. Here, CLEVR-ART is a novel dataset proposed within this paper.", "questions": "1. Also, what part of the architecture is enforcing the “relational bottleneck”? Is it due to the fact that the dot product of two slots results in a scalar number? If yes, then this should be highlighted somehow both in the text and perhaps also in Figure 1. One may also conduct an experiment in which the “bottleneck” dimension is gradually increased (e.g., 1 to 2 to 5 and so on) and show that this gradually worsens the generalization performance. \n2. Line 30: Regarding learning relational abstractions: Is it possible to take a large batch of inputs, extract relation “embedding” and visualize object pairs that have similar relational “embedding”? This would give more credence to the fact that relational abstractions are indeed being inferred. \n3. Line 108: Is it necessary to say “shared” here? IMO, I wouldn’t have assumed that the projection matrices were not shared (based on the equations).\n4. It should be better highlighted in the introduction section and the abstract that CLEVR-ART is a novel contribution. This should also be discussed in the related work relative to the existing visual reasoning benchmarks. It would also be useful to make a statement about whether this dataset will be released to the research community or not.\n5. Table 1: Why are several baselines shown for ART not shown for CLEVR-ART and SVRT? I think these baselines will be useful to show for all datasets. Also, I would suggest using a consistent format for reporting the results of all 3 datasets i.e., picking either the bar-plot format or the tabular format.\n6. L263: In the case of inputs involving multiple images, how are they processed by the network? As I understand, the model does not sequentially consume multiple images. Does “inserted” mean programmatically generating the candidate images, each containing multiple objects?\n7. L283-298. I find the discussion of the results rather small. For instance, what is the rationale for the comparison with ESBN or GAMR? What is the key distinguishing characteristic of those baselines with respect to the proposed one? What can we learn from the result of this comparison i.e., why does the proposed model outperform?\n8. (Minor) Line 34: Would be good to cite the paper(s) that support this statement “By biasing architectures to process visual inputs in terms of relations between objects, these recent approaches have achieved strong systematic (i.e., out-of-distribution) generalization of learned abstract rules, given only a small number of training examples.”\n\nI am giving a score of 6 in the hope that the authors will try and address some of my questions and concerns. If addressed to some degree, I will happily increase the score.", "limitations": "Yes, the limitations are discussed in the last line of the conclusion.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Simple and elegant architecture.\n2. Propose a new dataset for visual reasoning and systematic generalization called CLEVR-ART, a sort of visually complex variant of ART. This appears to be a useful contribution to facilitating progress in the community.\n3. Good systematic generalization performance compared to baselines.\n4. Several useful baseline comparisons. For instance, the paper shows that a standard transformer when applied to slots is not enough to systematically generalize and the proposed relation layer is useful.\n5. Useful ablations. For instance, the paper shows that all of the following are useful: relation bottleneck, relation embedding, object-centric inputs, decomposing position and appearance, etc.", "weaknesses": "I have several questions and possible avenues for improvements that I describe in the “Questions” section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful suggestions and comments. We provide a point-by-point reply below:\n\n1. Mechanism enforcing the relational bottleneck:\n    - The reviewer asks whether the dot product enforces a relational bottleneck only because it compresses inputs to a single dimension. To investigate this, we performed an ablation experiment, in which the dot product was replaced with a linear layer that projected the concatenated pair of input embeddings to a single dimension. We found that this model performed very poorly on the $m=95$ (most difficult) generalization regime for the RMTS (50.7% test accuracy) and ID (48% test accuracy) tasks. Thus, the relational bottleneck does not result from the unidimensional nature of the dot product per se. Instead, we believe that the most important feature of the dot product is that it is inherently relational, in that it is based on a multiplicative interaction between the two input embeddings. By contrast, when using a linear layer (or MLP, as in the ‘- Relational Bottleneck’ ablation model) to model relations, there is nothing to prevent the model from simply copying individual object features into the ‘relational’ embedding (thus making it not really relational). We have included the result of this ablation experiment in the revised paper, along with a discussion of the implications (we are unfortunately not able to include the specific revisions in this reply due to space constraints).\n2. Visualization of relational embeddings:\n    - This is an excellent suggestion. We regrettably did not have time to carry out this analysis during the rebuttal period, but we will try to do so before the end of the discussion period and share the results here time permitting.\n3. Specification of ‘shared’ weight matrices:\n    - We agree that the ‘shared’ nature of the weight matrices (mentioned in line 108) most likely doesn’t need clarification. However, our concern was that some readers may expect these to be separate matrices, in line with the common use of separate ‘key’ and ‘query’ matrices in self-attention. We included this clarification to avoid that potential confusion.\n4. Clarification about novelty of CLEVR-ART and release to public:\n    - We have added the clarifications to the abstract and introduction concerning the novelty of CLEVR-ART. We have also added discussion of CLEVR-ART relative to other visual reasoning benchmarks to the Related Work section (specific revisions omitted here due to space constraints).\n    - We do plan to make the dataset publicly available upon acceptance of the paper.\n5. Systematic testing of baseline and consistent format for results:\n    - We thank the reviewer for these suggestions. We have now evaluated all baselines on all three of these benchmarks (with the exception of Attn-ResNet, for which we do not have source code). The results can be found in Figures 1-3 of the attached PDF. **We found that OCRA performed better than the baseline models in most settings, with the best average performance across all tasks, and especially strong performance in the most extreme generalization regimes.** These results will be included in the final version of the paper, and we will also make sure to report all of these results as figures.\n6. Evaluation on multi-object inputs:\n    - The reviewer is correct that our approach involved ‘programmatically generating the candidate images, each containing multiple objects’. We have described this approach in line 199 of the original submission: ‘*As originally proposed, the ART dataset involved pre-segmented visual objects. Here, we investigated a version of this task involving multi-object visual displays (see Supplementary Section S1 for details).*’\n7. Limited discussion of results:\n    - We agree that more discussion of the results would be informative. Briefly, many of the baseline models do not contain a relational bottleneck (Transformer, IN, RN, GAMR, ResNet), and their ability to generalize is therefore limited to iid settings (e.g., low values of $m$ for the ART tasks). The comparison with these models is thus intended to evaluate the importance of the relational bottleneck, which enables stronger performance in the OOD setting. The ESBN and CoRelNet architectures *do* include a relational bottleneck, but they are not designed with multi-object inputs in mind, and they therefore perform poorly on these tasks in all regimes (even when combined with our slot attention module). This is due primarily to the random permutation of slots in slot attention, which motivated our positional embedding variable-binding scheme (to keep track of which relations correspond to which pairs of objects). We have added discussion of these issues to the Results section (specific revisions omitted here due to space constraints).\n8. Citing papers to support statement about OOD generalization:\n    - This statement is referring to the same papers cited at the beginning of the paragraph (line 30). We have added references to the revised paper to clarify this.\n\n\n\n\n\n"}, {"review_id": "EPJYEXsB7R", "reviewer": "Reviewer_6gd3", "summary": "The paper proposes to combine an object-centric representation model (Slot Attention) with a relational reasoning module to create the Object-Centric Relational Abstraction (OCRA) model. For this model, Slot Attention is first pre-trained in an unsupervised way to represent objects separately. Then, the relational reasoning module, consisting of a \"relational bottleneck\" and a transformer model, is applied to these object-centric representations. This reasoning model is trained in a supervised way on a subset of possible objects. The experiments on three different datasets show that this approach generalizes better to previously unseen objects than comparable baselines.\n", "questions": "As described above, I believe the main weakness of this paper is its current positioning relative to the existing research landscape. If the authors could improve this point, or strengthen the current claims by providing additional experiments, I would be happy to adjust my score.\n", "limitations": "While the relational bottleneck improves the model's ability to generalize to previously unseen objects, I would expect it to also limit the model's applicability to other settings. For example, in the experiments, a separate model is applied for each separate task and thus for each separate relation to be learned. I would expect that the relational bottleneck would harm the model's performance if it had to differentiate several relations at once. Additionally, I would expect it to be more difficult to represent more complex relations between objects, as the slightly worse results on the spatial relations in the SVRT dataset might indicate.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is written well and easy to follow. The introduction provides a strong motivation for this line of research. The proposed approach and conducted experiments are described clearly and with sufficient detail, such that it should be possible to reproduce them. \n\nThe proposed approach seems to be novel and achieves strong generalization performance on previously unseen objects.", "weaknesses": "In my eyes, the main weakness of this paper is that its contribution relative to existing work is not entirely clear. \n\n1) Most importantly, the paper positions itself as proposing an approach with a stronger inductive bias toward relational abstraction, which ultimately leads to better generalization performance (for example, introduction (l.46) and related work section (l. 135, l. 170). However, the experiments do not compare against many existing approaches. Without such a comparison, it remains unclear whether the existing approaches actually suffer from the problem that the proposed approach claims to solve. Thus, ideally, the paper should provide such a comparison, or alternatively adjust the positioning of the proposed approach relative to existing work. \n\n2) Besides this, it remains somewhat unclear which parts of the proposed approach are building on existing work (such as Slot Attention) and which parts are new. For example, do equations (1) and (2) and sections 2.2 and 2.3 describe novel architectural components or have they been used in a similar fashion before?\n\n---\n\n3) The related work section fails to cite many relevant papers. Object-centric representation learning has seen growing interest in recent years, with many papers proposing various solutions. I would recommend taking a look at the related works section of [1], which provides a comprehensive overview of papers that are relevant to this work. Besides this, [2] describes a recent model that combines object-centric representation learning with relational reasoning capabilities and [3] describes generalization properties of Slot Attention to previously unseen objects.\n\n\n4) The experimental results could be strengthened by applying all baselines across all datasets. Additionally, why is the CoRelNet not included in the baselines? It seems like one of the most relevant existing approaches.\n\n---\n\nMinor points:\n\n5) Equation (2): What does adding $pos$ to the position embeddings $m_k$ add for the relational reasoning module? It is not dependent on the input, and cannot be adjusted by the relational reasoning module as it is part of the pre-trained, frozen SlotAttention model. Thus, I would expect that it doesn't provide much useful information.\n\n6) Equation (4): $m_k$ will implicitly contain some information on the object shape, since $attn$ is used in Eq (2). Do you think this could allow the model to circumvent the relational bottleneck, if $m$ would be processed by some more powerful non-linearities?\n\n7) Line 115: \"endowing OCRA with an explicity variable-binding mechanism\". What do you mean with this statement?\n\n8) line 67: what is a \"position-wise fully-connected layer\"? Are you referring to the 1x1 convolutions in Table S2? In general, I think it would be helpful to draw direct connections between the components of the architecture described in Table S2 and the components described in the main text.\n\n9) Eq. (1) + (2): what is the $\\cdot$ symbol referring to here? In Eq. (3), this symbol is used to describe a dot procuct, which would not make much sense here, I think.\n\n10) Lines 255-262: I would move most of this description into the method section.\n\n11) Table 2: Use the same order for the different tested conditions in the table as is used within the text.\n\n---\n\n[1] Jiang, J., Deng, F., Singh, G., & Ahn, S. (2023). Object-centric slot diffusion. arXiv preprint arXiv:2303.10834.\n\n[2] Wu, Z., Dvornik, N., Greff, K., Kipf, T., & Garg, A. (2022). Slotformer: Unsupervised visual dynamics simulation with object-centric models. ICLR 2023\n\n[3] Dittadi, A., Papa, S., De Vita, M., Schölkopf, B., Winther, O., & Locatello, F. (2021). Generalization and robustness implications in object-centric learning. ICML 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful suggestions and comments. We provide a point-by-point reply below:\n\n1. Positioning relative to previous work:\n    - We have endeavored to cite all relevant previous work in the introduction and related work sections. While it was not feasible to implement baselines based on each and every one of these references, we have also done our best to select a representative subset of baselines that capture the important computational features of this previous work, and that are likely to represent the strongest possible baselines in the context of the abstract visual reasoning tasks we consider. Specifically, the slot transformer baseline is extremely similar to models from the references cited on lines 46 and 135, and was modeled closely on the implementation of [1] (ref 25 in the paper). The interaction network in our slot-IN baseline was highly similar to the models from the references cited on line 170, and was based closely on [2] (ref 43 in the paper). We believe that these baselines capture the important properties of the referenced papers, but we would be happy to implement specific additional baselines at the reviewer’s suggestion.\n2. Clarifying novel components of proposed model:\n    - The factorization of slot attention into separate spatial and non-spatial components (equations 1 and 2) is indeed a novel feature of our model. The relational embeddings described in section 2.2 also constitute a novel aspect of the model. Section 2.3 describes passing these relational embeddings to a transformer, which is a novel use of transformers. Overall, the architecture does employ some components from previous work (slot attention, transformers), but they are configured in a novel way to form a broader architecture with very different properties than the individual components alone.\n3. Missing references:\t\n    - We thank the reviewer for bringing these references to our attention. We agree that they are highly relevant to this work, and have added them to the Related Work section, along with a few other missing references that we identified from [3].\n4. Systematic evaluation of all baselines, and addition of CoRelNet baseline:\n    - We agree with the reviewer’s suggestion to more systematically evaluate all baselines on all tasks. We have now included this evaluation in the revised paper (see Figures 1-3 of attached PDF). **We found that OCRA performed better than the baseline models in most settings, with the best average performance across all tasks, and especially strong performance in the most extreme generalization regimes.**\n    - **We have also added a slot-CoRelNet baseline**, by combining slot attention with CoRelNet. This baseline did not perform very well, likely due to the random permutation of objects in slot attention.\n\nMinor points:\n\n5. Role of positional embeddings in relational reasoning:\n    - The positional embeddings are used to index the objects over which relations are computed. In other words, our relational embeddings can be interpreted as representing information such as ‘the object at location $i$ has the relation $R$ with the object at location $j$’, where $i$ and $j$ are spatial locations, and $R$ is a relation such as ‘same shape’. These positional indices are necessary to keep track of which relations correspond to which objects.\n6. Potential to break bottleneck through position embeddings:\n    - It is very important to our formulation that $m_k$ be a *linear* combination of the position embeddings. A nonlinear transformation of the position embeddings could indeed result in the extraction of implicit shape information, and therefore break the relational bottleneck. We have added this important clarification to our description of the model.\n7. Meaning of ‘explicit variable-binding mechanism’:\n    - This refers to the fact that our relational embeddings use positional information as indices, as described in the response to point #5 above (i.e., the relations are bound to explicit indices for the corresponding objects). \n8. 'Position-wise fully-connected layer' vs. '1x1 convolutions':\n    - Yes these refer to the same components. We agree this is confusing, and have revised the paper to consistently use the term ‘1x1 convolutions’.\n9. Clarification about equations 1 and 2:\n    - Thanks to the reviewer for catching this mistake. These are simply matrix multiplications. They have been reformatted as follows (where $\\boldsymbol{attn}_{T} \\in \\mathbb{R}^{K\\times N}$, $\\operatorname{flatten}(\\boldsymbol{feat}) \\in \\mathbb{R}^{N\\times D}$, and $\\operatorname{flatten}(\\boldsymbol{pos}) \\in \\mathbb{R}^{N\\times D}$):\n\n        $z_{k} = \\boldsymbol{attn}_{T} \\operatorname{flatten}(\\boldsymbol{feat})[k]$\n\n        $m_{k} = \\boldsymbol{attn}_{T} \\operatorname{flatten}(\\boldsymbol{pos})[k]$\n\n10. Moving lines 255-262:\n    - We have now moved these lines to the supplementary.\n11. Ordering of entries in Table 2:\n    - We have reordered the table accordingly.\n\nLimitations: \n- It is an interesting question the extent to which the proposed model could simultaneously model multiple relations. In future work, we plan to investigate how the model performs in multi-task settings. One possibility is to develop a 'multi-head' version of the current model, which may be better suited to this setting. We have added some discussion of this issue to the revised paper.\n- In the case of the SVRT spatial relation tasks, the model's performance was primarily limited by suboptimal object segmentation in our slot attention module. This was partly remedied by training slot attention for longer (see updated results in attached PDF). But we agree that it would be interesting in future work to evaluate the model on more complex real-world relations. \n\n[1] Mondal et al. (2023). Learning to reason over visual objects.\n\n[2] Watters et al. (2017). Visual interaction networks: Learning a physics simulator from video.\n\n[3] Jiang et al. (2023). Object-centric slot diffusion. \n"}]

Author rebuttal:
We would like to thank the reviewers for their many insightful suggestions and comments. We have added a number of new experiments to address the concerns raised, and revised the paper to improve clarity and provide further elaboration on some issues. We believe the paper is significantly improved as a result of these changes. Here, we will address the major issues raised by the reviewers, but we also provide point-by-point replies to each reviewers’ comments below.

- More systematic evaluation of baselines: We thank the reviewers for raising this issue, and have now included results for all baselines on all tasks, including six new baseline results for CLEVR-ART and SVRT, and one new baseline result for ART (see figures 1-3 in the attached PDF). These results include the implementation of a new baseline model ‘slot-corelnet’, that combined our pre-trained slot attention module with the corelnet [1] architecture. **We found that OCRA performed better than the baseline models in most settings, with the best average performance across all tasks, and especially strong performance in the most extreme generalization regimes.** Please note that we were not able to evaluate the ‘Attention-ResNet’ baseline on other tasks besides SVRT, as the SVRT result was reproduced from an earlier paper [2], for which no source code was provided.
- Lack of source code in our original submission: **We have now provided the AC with an anonymized link to the source code for our model. We will make this code publicly available upon acceptance of the paper. We will also make the CLEVR-ART dataset publicly available upon acceptance.**
- Concern about simplicity of benchmarks: The primary focus of the benchmarks we consider is **out-of-distribution generalization**. In the case of ART, performance of the baselines in the most extreme generalization regime ($m=95$ in Figure 1 of the attached PDF) is far from ceiling, with most performing at or near chance, whereas OCRA achieves a score of 88%. Performance of the baselines on the CLEVR-ART task (which also involves a significant degree of OOD generalization) and SVRT same-different tasks is similarly poor whereas OCRA achieves a score of 85% on both. Thus, despite the relative visual simplicity of the tasks that we consider, they are nevertheless an extremely challenging test of generalization and abstraction for baselines, whereas OCRA performs well. 
- Importance of extension to real-world visual inputs: We do however agree that it is important to extend models of visual reasoning so that they are capable of processing real-world visual inputs, and indeed that goal is precisely what motivated the present work. Toward that end, we focused in this work on extending abstract visual reasoning methods to deal with multi-object visual inputs. This was also the motivation for developing the CLEVR-ART dataset, which combines OOD generalization and photorealistically rendered three-dimensional objects. In future work, we agree that it would be extremely valuable to take this even further, by evaluating models on OOD generalization in the context of real-world images (e.g. on the Bongard-HOI benchmark [3], see next comment for a discussion of the prospects for doing this).
- Lack of detail on differences between current benchmarks and real-world vision: The primary challenge that we envision in extending this work to more complex settings is the difficulty involved in extracting object-centric representations from real-world images. This is a challenging problem because real-world images contain less consistent segmentation cues than in synthetic tasks like CLEVR. However, there has recently been significant progress in this direction [4], and it should be possible to combine these advances with our proposed model, which is an important goal for future work. Another potential challenge is that real-world scenes often involve a large number of objects, which would yield an exponentially larger number of pairwise relations in our model. We believe that this could be addressed by replacing the transformer component with methods that are better designed for higher-dimensional inputs, e.g. [5]. Thus, although there are certainly challenges to be addressed, we do not envision any fundamental obstacles to further scaling of the proposed approach. We have added these clarifications and additional discussion points to the revised manuscript.

[1] Kerg et al. (2022). On neural architecture inductive biases for relational tasks.

[2] Vaishnav & Serre (2023). GAMR: A guided attention model for (visual) reasoning.

[3] Jiang et al. (2022). Bongard-HOI: Benchmarking few-shot visual reasoning for human-object interactions. 

[4] Seitzer et al. (2022). Bridging the gap to real-world object-centric learning. 

[5] Jaegle et al. (2021). Perceiver: General perception with iterative attention.


Meta-review: This paper proposes ORCA, a model for object-centric relational abstraction and reasoning for visual data. The method combines Slot Attention with a relational reasoning component and proves effective for several challenging synthetic visual reasoning tasks. The paper further introduces a new reasoning task based on CLEVR, called CLEVR-ART, which presents a valuable contribution to this area.

The reviewers agree that this is a paper of high quality that presents a valuable contribution to the field. While the considered benchmark tasks are visually simplistic, the experimental evaluation is very thorough with a solid range of baseline comparisons and model ablations. While the positioning to related work could be expanded, this is a solid submission that in my view meets the bar for acceptance at NeurIPS.
Decision: Accept (poster)