Title: Systematic Visual Reasoning through Object-Centric Relational Abstraction
Venue: NeurIPS 2023
Accepted: yes
Keywords: relational reasoning

object-centric representations

abstract rule learning

relational inductive biases

systematic generalization
Abstract: Human visual reasoning is characterized by an ability to identify abstract patterns from only a small number of examples, and to systematically generalize those patterns to novel inputs. This capacity depends in large part on our ability to represent complex visual inputs in terms of both objects and relations. Recent work in computer vision has introduced models with the capacity to extract object-centric representations, leading to the ability to process multi-object visual inputs, but falling short of the systematic generalization displayed by human reasoning. Other recent models have employed inductive biases for relational abstraction to achieve systematic generalization of learned abstract rules, but have generally assumed the presence of object-focused inputs. Here, we combine these two approaches, introducing Object-Centric Relational Abstraction (OCRA), a model that extracts explicit representations of both objects and abstract relations, and achieves strong systematic generalization in tasks (including a novel dataset, CLEVR-ART, with greater visual complexity) involving complex visual displays.

Systematic Visual Reasoning through Object-Centric Relational Abstraction

Abstract

Human visual reasoning is characterized by an ability to identify abstract patterns from only a small number of examples, and to systematically generalize those patterns to novel inputs. This capacity depends in large part on our ability to represent complex visual inputs in terms of both objects and relations. Recent work in computer vision has introduced models with the capacity to extract object-centric representations, leading to the ability to process multi-object visual inputs, but falling short of the systematic generalization displayed by human reasoning. Other recent models have employed inductive biases for relational abstraction to achieve systematic generalization of learned abstract rules, but have generally assumed the presence of object-focused inputs. Here, we combine these two approaches, introducing Object-Centric Relational Abstraction (OCRA), a model that extracts explicit representations of both objects and abstract relations, and achieves strong systematic generalization in tasks (including a novel dataset, CLEVR-ART, with greater visual complexity) involving complex visual displays.

1 Introduction

When presented with a visual scene, human reasoners have a capacity to identify not only the objects in that scene, but also the relations between objects, and the higher-order patterns formed by multiple relations. These abilities are fundamental to human visual intelligence, enabling the rapid induction of abstract rules from a small number of examples, and the systematic generalization of those rules to novel inputs [32, 25, 13, 27]. For example, when presented with the image in Figure 1, one can easily determine that the objects in the top and bottom rows are both governed by a common abstract pattern (ABA). Furthermore, one could easily then generalize this pattern to any potential shapes, even those that one has never observed before. Indeed, this is arguably what it means for a pattern to be considered abstract .

37th Conference on Neural Information Processing Systems (NeurIPS 2023).

)HDWXUHGLYPH<c=3,font=/BFPIUQ+font000000002aed843c>

HPEHGGLQJV

āGLYPH<c=3,font=/VJEQLJ+font000000002aed843c>āGLYPH<c=3,font=/VJEQLJ+font000000002aed843c>ā

HPEHGGLQJV

ᭋ

N GLYPH<c=20,font=/BFPIUQ+font000000002aed843c>

ᬾ

N GLYPH<c=20,font=/BFPIUQ+font000000002aed843c>

ᭋ

N GLYPH<c=21,font=/BFPIUQ+font000000002aed843c>

ᬾ

N GLYPH<c=21,font=/BFPIUQ+font000000002aed843c>

ᭋ

N .

ᬾ

N .

ᷪ

5HODWLRQDO

RSHUDWRU

GLYPH<c=14,font=/IQCQDL+font000000002aed843c>

U

N GLYPH<c=20,font=/BFPIUQ+font000000002aed843c>GLYPH<c=15,font=/BFPIUQ+font000000002aed843c>N¶ GLYPH<c=20,font=/BFPIUQ+font000000002aed843c>

āGLYPH<c=3,font=/VJEQLJ+font000000002aed843c>āGLYPH<c=3,font=/VJEQLJ+font000000002aed843c>ā

U

N GLYPH<c=20,font=/BFPIUQ+font000000002aed843c>GLYPH<c=15,font=/BFPIUQ+font000000002aed843c>N¶ GLYPH<c=21,font=/BFPIUQ+font000000002aed843c>

U

N .GLYPH<c=15,font=/BFPIUQ+font000000002aed843c>N¶ .

5HODWLRQGLYPH<c=3,font=/BFPIUQ+font000000002aed843c>

HPEHGGLQJV

7UDQVIRUPHU

āGLYPH<c=3,font=/VJEQLJ+font000000002aed843c>āGLYPH<c=3,font=/VJEQLJ+font000000002aed843c>ā

ᭊ

ᭉ

Though this task appears effortless from the perspective of human reasoning, it exemplifies the type of problems that pose a significant challenge for standard neural network methods. Rather than inducing an abstract representation of the underlying rules in a given task, neural networks tend to overfit to the specific details of the examples observed during training, thus failing to generalize those rules to novel inputs [26, 3, 33].

To address this, a number of neural network methods have recently been proposed that incorporate strong inductive biases for relational abstraction [49, 51, 22, 1], also referred to as a 'relational bottleneck' [50]. The key insight underlying these approaches is that abstract visual patterns are often characterized by the relations between objects, rather than the features of the objects themselves. For example, the pattern seen in Figure 1 is characterized by the relation between the shape and position of the objects. By constraining architectures to process visual inputs in terms of relations between objects, these recent approaches have achieved strong systematic (i.e., out-of-distribution) generalization of learned abstract rules, given only a small number of training examples [51, 22, 1].

However, a major assumption of these approaches is that relations are computed between objects . Accordingly, these methods have generally relied on inputs consisting of pre-segmented visual objects, rather than multi-object scenes. Furthermore, these methods can sometimes be brittle to task-irrelevant visual variation, such as spatial jitter or Gaussian noise [43]. It is thus unclear whether and how these approaches might scale to complex and varied real-world settings.

In this work, we take a step toward the development of relational reasoning algorithms that can operate over more realistic visual inputs. To do so, we turn to object-centric representation learning [15, 6, 28, 12], a recently developed approach in which visual scenes are decomposed into a discrete set of object-focused latent representations, without the need for veridical segmentation data. Though this approach has shown success on a number of challenging visual tasks [10, 39, 30], it has not yet been integrated with the strong relational inductive biases that have previously been shown to enable systematic, human-like generalization [49, 51, 22, 1]. Here, we propose Object-Centric Relational Abstraction (OCRA), a model that combines the strengths of these two approaches. Across a suite of visual reasoning tasks (including a novel dataset, CLEVR-ART, with greater visual complexity), we find that OCRA is capable of systematically generalizing learned abstract rules from complex, multi-object visual displays, significantly outperforming a number of competitive baselines in most settings.

2 Approach

Figure 1 shows a schematic depiction of our approach. OCRA consists of three core components, each responsible for a different element of the reasoning process. These three elements can be summarized as 1) extraction of object-centric representations, 2) computation of pairwise relational embeddings, and 3) processing of higher-order relational structures - patterns across multiple relations - in order to identify abstract rules. We describe each of these three components in the following sections.

2.1 Object-centric representation learning

OCRAemploys slot attention [28] to extract object-centric representations from complex, multi-object visual inputs. Given an image x , the goal of slot attention is to extract a set of latent embeddings (i.e., slots ), each of which captures a focused representation of a single visual object, typically without access to ground truth segmentation data.

To do so, the image is first passed through a series of convolutional layers, generating a feature map feat ∈ R H × W × D . Positional embeddings pos ∈ R H × W × D are then generated by passing a 4d position code (encoding the four cardinal directions) through a separate linear projection. Feature and position embeddings are additively combined, and then passed through a series of 1x1 convolutions. This is then flattened, yielding inputs ∈ R N × D (where N = H × W ), over which slot attention is performed.

The set of K slots ∈ R K × D is then randomly initialized (from a shared distribution with learned parameters), and a form of transformer-style cross-attention is employed to attend over the pixels of the feature map. Specifically, each slot emits a query q( slots ) ∈ R K × D (through a linear projection), and each location in the feature map emits a key k( inputs ) ∈ R N × D and a value v( inputs ) ∈ R N × D . For each slot, a query-key attention operation is then used to generate an attention distribution over the feature map attn = softmax( 1 √ D k( inputs ) · q( slots ) /latticetop ) and a weighted mean of the values updates = attn · v( inputs ) is used to update the slot representations using a Gated Recurrent Unit [8], followed by a residual MLP. This process is repeated for T iterations, leading to a competition between the slots to account for each pixel in the feature map.

For the purposes of computing relational embeddings, we employed a factorized object representation format, in which position is represented distinctly from other object features. For a given input image, after performing T iterations of slot attention, we used the final attention map attn T ∈ R K × N to compute feature- and position-specific embeddings for each slot:

where feat and pos represent the feature- and position-embedding maps (prior to being combined). As will be seen in the following section, this factorized representation allows visual space to serve as an indexing mechanism for pairwise relations, mirroring the central role of spatial organization in human visual reasoning [29].

Although slot attention can in principle be learned end-to-end in the service of a specific downstream reasoning task, here we chose to pre-train slot attention using an unsupervised objective. This decision reflects the assumption that human-like visual reasoning does not occur in a vacuum, and visual representations are generally not learned from scratch when performing each new task. Instead, visual object representations are shaped by a wealth of non-task-specific prior experience. To model the effects of this prior experience, we pre-trained slot attention to reconstruct randomly generated multi-object displays, using a slot-based autoencoder framework (more details in Section 4.3.1). Importantly, for experiments testing generalization of abstract rules to novel objects, we ensured that these objects did not appear in the pre-training data for slot attention.

2.2 Relational embeddings

After extracting object embeddings, we employed a novel relational embedding method, designed to allow OCRA to abstract over the perceptual features of individual objects (and thus systematically

generalize learned rules to inputs with novel perceptual features). This embedding method consisted of two steps. First, we applied a relational operator φ to compute the pairwise relation between two feature embeddings z k and z k ′ . In the present work, we define this operator as:

where z k and z k ′ are first projected through a shared set of linear weights W z ∈ R D × D , and the dot product between these weighted embeddings is then projected back out into the original dimensionality using another set of weights W r ∈ R 1 × D . Using dot products to model pairwise relations introduces a layer of abstraction between the perceptual features of the objects and the downstream reasoning process, in that the dot product only represents the relations between objects, rather than those objects' individual features. We hypothesized that this relational bottleneck would lead to improved generalization to inputs with novel perceptual features. Additionally, the shared projection weights ( W z ) enable the model to learn which features are relevant for a given task, and to only compute relations between those features.

After applying the relational operator, we then compute the relational embedding for objects k and k ′ as:

where the position embeddings m k and m k ′ are each projected through a shared set of linear weights 1 W m ∈ R D × D and added to the output of φ . This allows the relation between objects k and k ′ to be indexed by their spatial position, thus endowing OCRA with an explicit variable-binding mechanism.

2.3 Processing of higher-order relations

After computing all K 2 pairwise relational embeddings, we then pass these embeddings to a reasoning process to extract higher-order relations - patterns formed between multiple relations. This is an essential component of visual reasoning, as abstract rules, such as the one displayed in Figure 1, are typically defined in terms of higher-order relations. We model these higher-order interactions using a transformer [45]:

where the inputs are formed by the set of all pairwise relational embeddings r kk ′ (for all k = 1 ...K and all k ′ = 1 ...K ), and y is a task-specific output. Because we employ a symmetric relational operator in the present work, we do not in practice present the full matrix of pairwise comparisons, but instead limit these comparisons to those in the upper triangular matrix.

3 Related Work

A number of methods for annotation-free object segmentation have recently been proposed [15, 6, 28, 12, 11, 20], including the slot attention method employed by our proposed model [28]. Though the details of these methods differ, they share the general goal of decomposing visual scenes into a set of latent variables corresponding to objects, without access to ground truth segmentation data. Building on the success of these object-encoding methods, some recent work has developed object-centric approaches to visual reasoning that combine slot-based object representations with transformer-based architectures [10, 39, 52, 30]. This work has yielded impressive performance on challenging computer vision tasks, such as question answering from video [10], or complex multi-object visual analogy problems [30], but has thus far been limited to generalization to similar problem types following extensive task-specific training.

In the present work, our goal was to develop a model capable of human-like systematic generalization in abstract reasoning tasks. Our proposed approach builds on the insights of other recent work in this

area, most notably including the Emergent Symbol Binding Network (ESBN) [51] and CoRelNet [22] architectures, both of which employed the concept of a relational bottleneck in different ways. In the ESBN architecture, separate control and perception pathways interact indirectly through a keyvalue memory, allowing the control pathway to learn representations that are abstracted over the details represented in the perceptual pathway. Kerg et al. [22] proposed that the ESBN's capacity for abstraction was due to the implicit encoding of relational information in its memory operations, which are mediated by a dot-product-based retrieval operation. Kerg et al. also proposed a novel architecture, CoRelNet, that makes this inductive bias more explicit. The CoRelNet architecture operates by explicitly computing the full matrix of relations (based on dot products) between each pair of objects in its input, and then passing these relations to an MLP for further processing. Both ESBN and CoRelNet displayed strong systematic generalization of learned abstract rules from a small set of examples. Importantly, however, both of these architectures depended on the presence of pre-segmented objects as input. A primary contribution of the present work is to combine these inductive biases for relational abstraction (i.e., the relational bottleneck) with object-centric encoding mechanisms, and to introduce the use of a spatial indexing mechanism that allows the model to apply these inductive biases to spatially embedded, multi-object visual inputs.

In addition to lacking object encoding mechanisms, it was recently shown that the ESBN can sometimes be overly brittle to task-irrelevant visual variation. Vaishnav and Serre [43] evaluated the ESBN and other reasoning architectures on abstract visual reasoning problems similar to those evaluated by Webb et al. [51], but with the introduction of either spatial jitter or Gaussian noise, finding that this significantly impaired performance in the ESBN (as well as other reasoning architectures, such as the transformer [45] and relation net [35]). Vaishnav and Serre also introduced a novel architecture, GAMR, that was more robust to these sources of noise, and able to solve problems based on multi-object inputs, likely due to the use of a novel guided attention mechanism. But despite outperforming ESBN and other architectures in this more challenging visual setting, GAMR nevertheless fell short of systematic generalization in these tasks, displaying a significant drop in performance in the most difficult generalization regimes. Here, we directly confront this challenge, demonstrating that OCRA is capable of systematic generalization comparable to that displayed by ESBN and CoRelNet, but in more complex visual settings.

It is also worth mentioning that other neural network architectures have been proposed that incorporate relational inductive biases in various ways [35, 4, 38, 34, 37, 5], some of which also incorporate object-centric representations [48, 44, 19, 53, 2, 24, 40, 46, 14, 41]. Relative to this previous work, our approach is distinguished by the inclusion of a stronger inductive bias toward relational abstraction - that is, a relational bottleneck - for promoting the development of relational representations that fully abstract over the details of individual objects [51, 22, 1]. To empirically evaluate the importance of the relational bottleneck, we compare our approach with a set of baseline models that capture the key computational properties of previous models, including a slot-transformer baseline that combines slot attention with a transformer reasoning module [10, 39, 52, 30], and a slot-interaction-network baseline that combines slot attention with an interaction-network reasoning module [48, 44, 19, 53, 2, 24, 40, 46, 14, 41].

To summarize our contributions relative to previous work, our proposed model includes:

A novel object representation format that is factorized into distinct feature and position embeddings (Equations 1 and 2), enabling a form of explicit variable-binding.

A novel relational embedding method that implements a relational bottleneck (Equations 3 and 4).

An architecture that combines these elements with preexisting components (slot attention and transformers) in a novel manner to support systematic visual reasoning from complex visual displays, including a novel dataset CLEVR-ART with greater visual complexity.

4 Experiments

4.1 Datasets

We evaluated OCRA on two challenging visual reasoning tasks, ART [51] and SVRT [13], specifically designed to probe systematic generalization of learned abstract rules, as well as a novel dataset, CLEVR-ART, involving more complex visual inputs (Figure 2).

GLYPH<c=20,font=/OUECUQ+font000000002aed843d>

GLYPH<c=21,font=/OUECUQ+font000000002aed843d>

GLYPH<c=22,font=/OUECUQ+font000000002aed843d>

GLYPH<c=23,font=/OUECUQ+font000000002aed843d>

$57

&DWHJRU\GLYPH<c=3,font=/GNHLNX+font000000002aed843d>GLYPH<c=20,font=/GNHLNX+font000000002aed843d>

&DWHJRU\GLYPH<c=3,font=/GNHLNX+font000000002aed843d>GLYPH<c=21,font=/GNHLNX+font000000002aed843d>

6957

&/(95GLYPH<c=16,font=/GNHLNX+font000000002aed843d>$57

ART. The Abstract Reasoning Tasks (ART) dataset was proposed by Webb et al. [51], consisting of four visual reasoning tasks, each defined by a different abstract rule (Figure S1). In the 'same/different' task, two objects are presented, and the task is to say whether they are the same or different. In the 'relational match-to-sample' task, a source pair of objects is presented that either instantiates a 'same' or 'different' relation, and the task is to select the pair of target objects (out of two pairs) that instantiates the same relation. In the 'distribution-of-3' task, a set of three objects is presented in the first row, and an incomplete set is presented in the second row. The task is to select the missing object from a set of four choices. In the 'identity rules' task, an abstract pattern is instantiated in the first row (ABA, ABB, or AAA), and the task is to select the choice that would result in the same relation being instantiated in the second row.

The primary purpose of this dataset was to evaluate strong systematic generalization of these rules. Thus, each task consists of a number of generalization regimes of varying difficulty, defined by the number of unique objects used to instantiate the rules during training (out of a set of 100 possible objects). In the most difficult regime ( m = 95 ), the training set consists of problems that are created using only 5 objects, and the test problems are created using the remaining 95 objects, whereas in the easiest regime ( m = 0 ), both training and test problems are created from the complete set of 100 objects (though the arrangement of these objects is distinct, such that the training and test sets are still disjoint). The most difficult generalization regime poses an especially difficult test of systematic generalization, as it requires learning of an abstract rule from a very small set of examples (details shown in Table S1), with little perceptual overlap between training and test.

As originally proposed [51], the ART dataset involved pre-segmented visual objects. Here, we investigated a version of this task involving multi-object visual displays (see Supplementary Section S2 for details). We also applied random spatial jitter (random translation of up to 5 pixels in any direction) to each object, as this was previously found to significantly impair performance in some previous reasoning models [42].

SVRT. The Synthetic Visual Reasoning Test (SVRT), proposed by Fleuret et al. [13], consists of 23 binary classification tasks, each defined by a particular configuration of relations. The tasks can be broadly divided into two categories: those that are based primarily on same/different relations ('SD'; tasks 7, 21, 5, 19, 6, 17, 20, 1, 13, 22, 16; example shown in Figure S3(a)), and those that are primarily based on spatial relations ('SR'; tasks 12, 15, 14, 9, 23, 8, 10, 18, 4, 3, 11, 2; example shown in Figure S3(b)). As with previous work [43], we limit training to a relatively small number of examples (either 500 or 1000 per task).

CLEVR-ART. We created a novel dataset based on ART using realistically rendered 3D shapes from CLEVR [21] (Figure S4). We focused on two specific visual reasoning tasks: relational-match-

to-sample and identity rules. Problems were formed from objects of three shapes (cube, sphere, and cylinder), three sizes (small, medium, and large), eight colors (gray, red, blue, green, brown, purple, cyan, and yellow) and two textures (rubber and metal). To test systematic generalization, the training set consisted of problems using only small and medium sized rubber cubes in four colors (cyan, brown, green, and gray) whereas the test set consisted only of problems using large sized metal spheres and cylinders in four distinct colors (yellow, purple, blue, and red). The training and test sets thus involved completely disjoint perceptual features, similar to the original ART.

4.2 Baselines

We compared our model to results from a set of baseline models evaluated by Vaishnav and Serre [43], including their GAMR architecture, ResNet50 [17], and a version of ResNet that employed selfattention (Attn-ResNet) [42]. To the best of our knowledge, these baselines represent the best performing models on the tasks that we investigated. We also investigated an additional set of baselines that combined our pretrained slot attention module with various reasoning architectures, including GAMR [43], ESBN [51], Relation Net (RN) [35], Transformer [45], Interaction Network (IN) [48], and CoRelNet [22]. These baselines benefited from the same degree of pre-training and object-centric processing as our proposed model, but employed different reasoning mechanisms. Finally, we investigated a self-supervised baseline (a Masked Autoencoder (MAE) [16]) to assess the extent to which a generative objective might encourage systematic generalization (see Tables S13S14 for results). All baselines were evaluated using multi-object displays. See Supplementary Section S3for more details on baselines.

4.3 Experimental Details

4.3.1 Pre-training slot attention

We pre-trained slot attention using a slot-based autoencoder framework. For ART, we generated a dataset of random multi-object displays (see Figure S2), utilizing a publicly available repository of unicode character images 2 . We eliminated any characters that were identical to, or closely resembled, the 100 characters present in the ART dataset, resulting in a training set of 180k images, and a validation set of 20k images. For SVRT, we pre-trained two versions of slot attention on all tasks, one using 500 and the other using 1000 training examples from each of the 23 tasks. For CLEVR-ART, we generated a dataset of randomly placed 3D shapes from CLEVR, and pre-trained slot attention using a training set of 90k images, and validation set of 10k images.

To pre-train slot attention, we used a simple reconstruction objective and a slot-based decoder. Specifically, we used a spatial broadcast decoder [47], which generates both a reconstructed image and a mask for each slot. We then generated a combined reconstruction, normalizing the masks across slots using a softmax, and using the normalized masks to compute a weighted average of the slot-specific reconstructions.

After pre-training, the slot attention parameters were frozen for training on the downstream reasoning tasks. For ART, we found that despite not having been trained on any of the objects in the dataset, slot attention was nevertheless able to generate focused attention maps that corresponded closely to the location of the objects (see Figures S6-S9).

4.3.2 Training details and hyperparameters

Images were resized to 128 × 128 , and pixels were normalized to the range [0 , 1] . For SVRT, we converted the images to grayscale, and also applied random online augmentations during training, including horizontal and vertical flips. Tables S2 and S3 describes the hyperparameters for the convolutional encoder and spatial broadcast decoder respectively. For slot attention, we used K = 6 ( K = 7 for CLEVR-ART) slots, T = 3 attention iterations per image, and a dimensionality of D = 64 . Training details are described in Table S4.

Table S5 gives the hyperparameters for the transformer. To solve the SVRT tasks, and the same/different ART task, we used a CLS token (analogous to CLS token in [9]) together with a sigmoidal output layer to generate a binary label, and trained the model with binary cross-entropy

Same/different

Relationalmatch-to-sample

100

100

OCRA

80

80

Slot-Transformer

Slot-IN

60-

Slot-RN

Slot-GAMR

40

40-

Slot-ESBN

Slot-CoRelNet

GAMR

20-

20

ResNet

0

0

0

50

85

95

0

50

85

95

m (objects withheld during training)

m (objects withheld during training)

Distribution-of-3

Identity rules

100

100

80

80

60

40

40

20

20

0

0

0

50

85

95

0

50

85

95

m (objects withheld during training)

m (objects withheld during training)

loss. To solve the RMTS, distribution-of-3, and identity rules tasks, we inserted each candidate answer choice into the problem, and used a CLS token together with a linear output layer to generate a score. The scores for all answer choices were passed through a softmax and the model was trained with cross-entropy loss. For ART and CLEVR-ART, we used a default learning rate of 8 e -5 , and a batch size of 16. The number of training epochs is displayed for each ART task in Table S6, and for each CLEVR-ART task in Table S10. The training details for some baseline models were modified as discussed in Section S3, and detailed in Tables S7-S9. For SVRT, we used a learning rate of 4 e -5 , a batch size of 32, and trained for 2000 epochs on each task. We used the ADAM optimizer [23], and implementation was done using the Pytorch library [31].

4.3.3 Model selection

After pre-training slot attention for a fixed number of epochs (number of epochs for each dataset listed in Table S4), we selected the version of the model from the epoch with the lowest validation loss (on the unsupervised pre-training task) to use in the visual reasoning tasks. We used a validation set of 20k images for ART (again excluding the objects used in the ART tasks), 4k images per task for SVRT, and 10k images for CLEVR-ART.

When training on the primary reasoning tasks, for SVRT, we trained for 2000 epochs on each task, and selected the version of the model from the epoch with the highest accuracy on the validation set. This is consistent with the approach taken by Vaishnav and Serre [43], who used validation accuracy as a criterion for early stopping. For ART and CLEVR-ART, we did not perform any model (or epoch) selection when training on the primary reasoning tasks.

5 Results

Figure 3 shows the results on the ART dataset (all results are presented in tabular form in Tables S11-S17). OCRA achieved state-of-the-art results across all tasks and generalization regimes, outperforming both the previous state-of-the-art model (GAMR), and other strong baselines such as

CLEVR-ART

SVRT

Same/different

Spatial relations

100

100

100

OCRA

Slot-Transformer

80

80

80

Slot-IN

Slot-RN

60

60

60

Slot-GAMR

Slot-ESBN

40

40

40

Slot-CoRelNet

GAMR

20

20

20

Attn-ResNet

ResNet

0

RMTS

Identityrules

0

1000

500

0

1000

500

Task

Numberoftrainingexamples

Numberoftrainingexamples

the slot-transformer. This disparity was especially striking in the most difficult generalization regime ( m = 95 ), in which models were trained on only 5 objects, and tested on a heldout set of 95 objects. Importantly, no component of OCRA (including both the slot attention and relational reasoning components) had been exposed to these heldout objects. OCRA outperformed the next-best baselines in this regime (GAMR and slot-transformer) by as much as 25%, demonstrating a significant degree of systematic generalization. We also analyzed OCRA's learned relational embeddings, finding that they captured relational information even in this out-of-distribution setting (Figure S5).

Many of the baseline models that we evaluated did not contain a relational bottleneck (Transformer, IN, RN, GAMR, ResNet), limiting their ability to generalize relational patterns to novel objects (i.e., higher values of m ). The comparison with these models thus demonstrates the importance of the relational bottleneck, which enables systematic generalization of learned relational patterns. The ESBN and CoRelNet architectures did include a relational bottleneck, but they were not designed with multi-object inputs in mind, and therefore performed poorly on these tasks in all regimes (even when combined with our slot attention module). This is due primarily to the random permutation of slots in slot attention, which motivated our positional embedding variable-binding scheme (allowing OCRA to keep track of which relations correspond to which pairs of objects).

On SVRT, OCRA showed comparable overall performance with GAMR, displaying marginally better performance on tasks characterized by same/different relations, while GAMR performed marginally better on tasks involving spatial relations (Figure 4, right). OCRA also outperformed GAMR on CLEVR-ART (Figure 4, left), demonstrating that its capacity for systematic generalization can be extended to problems involving more realistic visual inputs. Overall, the results on ART, SVRT, and CLEVR-ART demonstrate that OCRA is capable of solving a diverse range of relational reasoning problems from complex multi-object displays.

5.1 Ablation Study

We performed an ablation study targeting each of OCRA's major components, focusing on the most difficult regime of the relational match-to-sample and identity rules tasks (Table 1). To evaluate the importance of OCRA's capacity for object-centric processing, we tested a version of the model without slot attention ('w/o Slot Attention'), in which the visual feature map was instead divided into a 4 × 4 grid, treating each image patch as though it were an 'object'. We also tested a version in which slot attention was only trained end-to-end on the visual reasoning tasks ('w/o Pre-training'), to test the importance of prior exposure to a diverse range of multi-object displays. To evaluate the importance of OCRA's relational embedding method, we tested five different ablation models. In one model ('w/o Relational Embeddings'), slot embeddings were passed directly to the transformer. In another model ('w/o Factorized Representation'), we did not compute separate feature and position embeddings for each object, and instead applied the relational operator directly to the slot embeddings. In a third model ('w/o Relational Bottleneck'), relational embeddings were computed simply by applying an MLP to each pair of slot embeddings (allowing the model to overfit to the perceptual

details of these embeddings), thus removing OCRA's inductive bias for relational abstraction. In a fourth model ('w/o Inner Product') the dot product in the relational operator (Equation 3) was replaced with a learned bottleneck, in which each pair of slot embeddings was passed to a learned linear layer with a one-dimensional output. This model tested the extent to which the inherently relational properties of the dot product are important for relational abstraction (as opposed to resulting merely from compression to a single dimension). In a fifth model ('Replace Inner Product w/ Outer Product'), the dot product was replaced with an outer product (resulting in a D × D matrix, which was then flattened and projected via a learned linear layer to an embedding of size D ). Finally, to evaluate the importance of OCRA's capacity to process higher-order relations, instead of passing relational embeddings to a transformer, we instead summed all relation embeddings elementwise and passed the resulting vector to an MLP, as is done in the Relation Net ('w/o Transformer'). All ablation models performed significantly worse than the complete version of OCRA, demonstrating that all three elements in our model - object-centric processing, relational abstraction, and higher-order relations - play an essential role in enabling systematic generalization.

6 Limitations and Future Directions

In the present work, we have presented a model that integrates object-centric visual processing mechanisms (providing the ability to operate over complex multi-object visual inputs) with a relational bottleneck (providing a strong inductive bias for learning relational abstractions that enable humanlike systematic generalization of learned abstract rules). Though this is a promising step forward, and a clear advance relative to previous models of abstract visual reasoning, it is likely that scaling the present approach to real-world settings will pose additional challenges. First, real-world images are especially challenging for object-centric methods due to a relative lack of consistent segmentation cues. However, there has recently been significant progress in this direction [36], in particular by utilizing representations from large-scale self-supervised methods [54, 7], and it should be possible to integrate these advances with our proposed framework. Second, the current approach assumes a fixed number of slot representations, which may not be ideal for modeling real-world scenes with a highly variable number of objects [55]. Though we did not find that this was an issue in the present work, in future work it would be desirable to develop a method for dynamically modifying the number of slots. Third, OCRA's relational operator is applied to all possible pairwise object comparisons, an approach that may not be scalable to scenes that contain many objects. In future work, this may be addressed by replacing the transformer component of our model with architectures that are better designed for high-dimensional inputs [18]. Finally, it is unclear how our proposed approach may fare in settings that involve more complex real-world relations, and settings that require the discrimination of multiple relations at once. It may be beneficial in future work to investigate a 'multi-head' version of our proposed model (analogous to multi-head attention), in which multiple distinct relations are processed in parallel. A major challenge for future work is to develop models that can match the human capacity for structured visual reasoning in the context of complex, real-world visual inputs.

References

A. Altabaa, T. Webb, J. Cohen, and J. Lafferty. Abstractors: Transformer modules for symbolic message passing and relational reasoning. arXiv preprint arXiv:2304.00195 , 2023.

V. Bapst, A. Sanchez-Gonzalez, C. Doersch, K. Stachenfeld, P. Kohli, P. Battaglia, and J. Hamrick. Structured agents for physical construction. In International conference on machine learning , pages 464-474. PMLR, 2019.

D. Barrett, F. Hill, A. Santoro, A. Morcos, and T. Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning , pages 511-520. PMLR, 2018.

P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 , 2018.

L. Bergen, T. O'Donnell, and D. Bahdanau. Systematic generalization with edge transformers. Advances in Neural Information Processing Systems , 34:1390-1402, 2021.

C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390 , 2019.

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

D. Ding, F. Hill, A. Santoro, M. Reynolds, and M. Botvinick. Attention over learned object embeddings enables complex visual reasoning. Advances in neural information processing systems , 34:9112-9124, 2021.

A. Dittadi, S. Papa, M. De Vita, B. Schölkopf, O. Winther, and F. Locatello. Generalization and robustness implications in object-centric learning. arXiv preprint arXiv:2107.00637 , 2021.

M. Engelcke, O. Parker Jones, and I. Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. Advances in Neural Information Processing Systems , 34:8085-8094, 2021.

F. Fleuret, T. Li, C. Dubout, E. K. Wampler, S. Yantis, and D. Geman. Comparing machines and humans on a visual categorization test. Proceedings of the National Academy of Sciences , 108(43):17621-17625, 2011.

A. Gopalakrishnan, S. van Steenkiste, and J. Schmidhuber. Unsupervised object keypoint learning using local spatial predictability. arXiv preprint arXiv:2011.12930 , 2020.

K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and A. Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning , pages 2424-2433. PMLR, 2019.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning , pages 4651-4664. PMLR, 2021.

M. Janner, S. Levine, W. T. Freeman, J. B. Tenenbaum, C. Finn, and J. Wu. Reasoning about physical interactions with object-oriented prediction and planning. arXiv preprint arXiv:1812.10972 , 2018.

J. Jiang, F. Deng, G. Singh, and S. Ahn. Object-centric slot diffusion. arXiv preprint arXiv:2303.10834 , 2023.

J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 29012910, 2017.

G. Kerg, S. Mittal, D. Rolnick, Y. Bengio, B. Richards, and G. Lajoie. On neural architecture inductive biases for relational tasks. arXiv preprint arXiv:2206.05056 , 2022.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

T. Kipf, E. Van der Pol, and M. Welling. Contrastive learning of structured world models. arXiv preprint arXiv:1911.12247 , 2019.

L. Kotovsky and D. Gentner. Comparison and categorization in the development of relational similarity. Child Development , 67(6):2797-2822, 1996.

B. Lake and M. Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning , pages 2873-2882. PMLR, 2018.

B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science , 350(6266):1332-1338, 2015.

F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems , 33:11525-11538, 2020.

B. J. Matlen, D. Gentner, and S. L. Franconeri. Spatial alignment facilitates visual comparison. Journal of Experimental Psychology: Human Perception and Performance , 46(5):443, 2020.

S. S. Mondal, T. W. Webb, and J. D. Cohen. Learning to reason over visual objects. In 11th International Conference on Learning Representations, ICLR , 2023.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.

J. C. Raven. Progressive matrices: A perceptual test of intelligence, individual form . London: Lewis, 1938.

M. Ricci, J. Kim, and T. Serre. Same-different problems strain convolutional neural networks. arXiv preprint arXiv:1802.03390 , 2018.

A. Santoro, R. Faulkner, D. Raposo, J. Rae, M. Chrzanowski, T. Weber, D. Wierstra, O. Vinyals, R. Pascanu, and T. Lillicrap. Relational recurrent neural networks. Advances in neural information processing systems , 31, 2018.

A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap. A simple neural network module for relational reasoning. Advances in neural information processing systems , 30, 2017.

M. Seitzer, M. Horn, A. Zadaianchuk, D. Zietlow, T. Xiao, C.-J. Simon-Gabriel, T. He, Z. Zhang, B. Schölkopf, T. Brox, et al. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209.14860 , 2022.

M. Shanahan, K. Nikiforou, A. Creswell, C. Kaplanis, D. Barrett, and M. Garnelo. An explicitly relational neural network architecture. In International Conference on Machine Learning , pages 8593-8603. PMLR, 2020.

P. Shaw, J. Uszkoreit, and A. Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 , 2018.

G. Singh, Y.-F. Wu, and S. Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. arXiv preprint arXiv:2205.14065 , 2022.

A. Stani´ c and J. Schmidhuber. R-sqair: relational sequential attend, infer, repeat. arXiv preprint arXiv:1910.05231 , 2019.

A. Stani´ c, S. van Steenkiste, and J. Schmidhuber. Hierarchical relational inference. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9730-9738, 2021.

M. Vaishnav, R. Cadene, A. Alamia, D. Linsley, R. VanRullen, and T. Serre. Understanding the computational demands underlying visual reasoning. Neural Computation , 34(5):1075-1099, 2022.

M. Vaishnav and T. Serre. Gamr: A guided attention model for (visual) reasoning. In 11th International Conference on Learning Representations, ICLR , 2023.

S. Van Steenkiste, M. Chang, K. Greff, and J. Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.10353 , 2018.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł . Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

R. Veerapaneni, J. D. Co-Reyes, M. Chang, M. Janner, C. Finn, J. Wu, J. Tenenbaum, and S. Levine. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning , pages 1439-1456. PMLR, 2020.

N. Watters, L. Matthey, C. P. Burgess, and A. Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:1901.07017 , 2019.

N. Watters, D. Zoran, T. Weber, P. Battaglia, R. Pascanu, and A. Tacchetti. Visual interaction networks: Learning a physics simulator from video. Advances in neural information processing systems , 30, 2017.

T. Webb, Z. Dulberg, S. Frankland, A. Petrov, R. O'Reilly, and J. Cohen. Learning representations that support extrapolation. In International conference on machine learning , pages 10136-10146. PMLR, 2020.

T. W. Webb, S. M. Frankland, A. Altabaa, K. Krishnamurthy, D. Campbell, J. Russin, R. O'Reilly, J. Lafferty, and J. D. Cohen. The relational bottleneck as an inductive bias for efficient abstraction. arXiv preprint arXiv:2309.06629 , 2023.

T. W. Webb, I. Sinha, and J. D. Cohen. Emergent symbols through binding in external memory. In 9th International Conference on Learning Representations, ICLR , 2021.

Z. Wu, N. Dvornik, K. Greff, T. Kipf, and A. Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05861 , 2022.

V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. Reichert, T. Lillicrap, E. Lockhart, et al. Relational deep reinforcement learning. arXiv preprint arXiv:1806.01830 , 2018.

J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 , 2021.

R. S. Zimmermann, S. van Steenkiste, M. S. Sajjadi, T. Kipf, and K. Greff. Sensitivity of slot-based object-centric models to their number of slots. arXiv preprint arXiv:2305.18890 , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to combine an object-centric representation model (Slot Attention) with a relational reasoning module to create the Object-Centric Relational Abstraction (OCRA) model. For this model, Slot Attention is first pre-trained in an unsupervised way to represent objects separately. Then, the relational reasoning module, consisting of a \"relational bottleneck\" and a transformer model, is applied to these object-centric representations. This reasoning model is trained in a supervised way on a subset of possible objects. The experiments on three different datasets show that this approach generalizes better to previously unseen objects than comparable baselines.\n\nSTRENGTHS:\nThe paper is written well and easy to follow. The introduction provides a strong motivation for this line of research. The proposed approach and conducted experiments are described clearly and with sufficient detail, such that it should be possible to reproduce them. \n\nThe proposed approach seems to be novel and achieves strong generalization performance on previously unseen objects.\n\nWEAKNESSES:\nIn my eyes, the main weakness of this paper is that its contribution relative to existing work is not entirely clear. \n\n1) Most importantly, the paper positions itself as proposing an approach with a stronger inductive bias toward relational abstraction, which ultimately leads to better generalization performance (for example, introduction (l.46) and related work section (l. 135, l. 170). However, the experiments do not compare against many existing approaches. Without such a comparison, it remains unclear whether the existing approaches actually suffer from the problem that the proposed approach claims to solve. Thus, ideally, the paper should provide such a comparison, or alternatively adjust the positioning of the proposed approach relative to existing work. \n\n2) Besides this, it remains somewhat unclear which parts of the proposed approach are building on existing work (such as Slot Attention) and which parts are new. For example, do equations (1) and (2) and sections 2.2 and 2.3 describe novel architectural components or have they been used in a similar fashion before?\n\n---\n\n3) The related work section fails to cite many relevant papers. Object-centric representation learning has seen growing interest in recent years, with many papers proposing various solutions. I would recommend taking a look at the related works section of [1], which provides a comprehensive overview of papers that are relevant to this work. Besides this, [2] describes a recent model that combines object-centric representation learning with relational reasoning capabilities and [3] describes generalization properties of Slot Attention to previously unseen objects.\n\n\n4) The experimental results could be strengthened by applying all baselines across all datasets. Additionally, why is the CoRelNet not included in the baselines? It seems like one of the most relevant existing approaches.\n\n---\n\nMinor points:\n\n5) Equation (2): What does adding $pos$ to the position embeddings $m_k$ add for the relational reasoning module? It is not dependent on the input, and cannot be adjusted by the relational reasoning module as it is part of the pre-trained, frozen SlotAttention model. Thus, I would expect that it doesn't provide much useful information.\n\n6) Equation (4): $m_k$ will implicitly contain some information on the object shape, since $attn$ is used in Eq (2). Do you think this could allow the model to circumvent the relational bottleneck, if $m$ would be processed by some more powerful non-linearities?\n\n7) Line 115: \"endowing OCRA with an explicity variable-binding mechanism\". What do you mean with this statement?\n\n8) line 67: what is a \"position-wise fully-connected layer\"? Are you referring to the 1x1 convolutions in Table S2? In general, I think it would be helpful to draw direct connections between the components of the architecture described in Table S2 and the components described in the main text.\n\n9) Eq. (1) + (2): what is the $\\cdot$ symbol referring to here? In Eq. (3), this symbol is used to describe a dot procuct, which would not make much sense here, I think.\n\n10) Lines 255-262: I would move most of this description into the method section.\n\n11) Table 2: Use the same order for the different tested conditions in the table as is used within the text.\n\n---\n\n[1] Jiang, J., Deng, F., Singh, G., & Ahn, S. (2023). Object-centric slot diffusion. arXiv preprint arXiv:2303.10834.\n\n[2] Wu, Z., Dvornik, N., Greff, K., Kipf, T., & Garg, A. (2022). Slotformer: Unsupervised visual dynamics simulation with object-centric models. ICLR 2023\n\n[3] Dittadi, A., Papa, S., De Vita, M., Schölkopf, B., Winther, O., & Locatello, F. (2021). Generalization and robustness implications in object-centric learning. ICML 2022\n\nQUESTIONS:\nAs described above, I believe the main weakness of this paper is its current positioning relative to the existing research landscape. If the authors could improve this point, or strengthen the current claims by providing additional experiments, I would be happy to adjust my score.\n\nLIMITATIONS:\nWhile the relational bottleneck improves the model's ability to generalize to previously unseen objects, I would expect it to also limit the model's applicability to other settings. For example, in the experiments, a separate model is applied for each separate task and thus for each separate relation to be learned. I would expect that the relational bottleneck would harm the model's performance if it had to differentiate several relations at once. Additionally, I would expect it to be more difficult to represent more complex relations between objects, as the slightly worse results on the spatial relations in the SVRT dataset might indicate.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper seeks to tackle various visual reasoning problems with a focus on systematic generalization.\n- The paper argues that we need explicit inductive bias to extract object-object relationships and express these relationships via a low-capacity representation.\n- The paper seeks to do this by first extracting object vectors (or slots) using pre-trained slot attention. It then performs a dot-product of these slots to obtain a “relation embedding” — which is really a scalar number expressed as an embedding (as far as I understand). These relation embeddings are then given to a transformer and the whole model (except the slot attention) is trained to classify whether the given input is conformant to the true pattern or not.\n- The performance is then evaluated on a systematically OOD test set and compared with various baselines and model ablations on 3 visual reasoning benchmarks: ART, SVRT, and CLEVR-ART. Here, CLEVR-ART is a novel dataset proposed within this paper.\n\nSTRENGTHS:\n1. Simple and elegant architecture.\n2. Propose a new dataset for visual reasoning and systematic generalization called CLEVR-ART, a sort of visually complex variant of ART. This appears to be a useful contribution to facilitating progress in the community.\n3. Good systematic generalization performance compared to baselines.\n4. Several useful baseline comparisons. For instance, the paper shows that a standard transformer when applied to slots is not enough to systematically generalize and the proposed relation layer is useful.\n5. Useful ablations. For instance, the paper shows that all of the following are useful: relation bottleneck, relation embedding, object-centric inputs, decomposing position and appearance, etc.\n\nWEAKNESSES:\nI have several questions and possible avenues for improvements that I describe in the “Questions” section.\n\nQUESTIONS:\n1. Also, what part of the architecture is enforcing the “relational bottleneck”? Is it due to the fact that the dot product of two slots results in a scalar number? If yes, then this should be highlighted somehow both in the text and perhaps also in Figure 1. One may also conduct an experiment in which the “bottleneck” dimension is gradually increased (e.g., 1 to 2 to 5 and so on) and show that this gradually worsens the generalization performance. \n2. Line 30: Regarding learning relational abstractions: Is it possible to take a large batch of inputs, extract relation “embedding” and visualize object pairs that have similar relational “embedding”? This would give more credence to the fact that relational abstractions are indeed being inferred. \n3. Line 108: Is it necessary to say “shared” here? IMO, I wouldn’t have assumed that the projection matrices were not shared (based on the equations).\n4. It should be better highlighted in the introduction section and the abstract that CLEVR-ART is a novel contribution. This should also be discussed in the related work relative to the existing visual reasoning benchmarks. It would also be useful to make a statement about whether this dataset will be released to the research community or not.\n5. Table 1: Why are several baselines shown for ART not shown for CLEVR-ART and SVRT? I think these baselines will be useful to show for all datasets. Also, I would suggest using a consistent format for reporting the results of all 3 datasets i.e., picking either the bar-plot format or the tabular format.\n6. L263: In the case of inputs involving multiple images, how are they processed by the network? As I understand, the model does not sequentially consume multiple images. Does “inserted” mean programmatically generating the candidate images, each containing multiple objects?\n7. L283-298. I find the discussion of the results rather small. For instance, what is the rationale for the comparison with ESBN or GAMR? What is the key distinguishing characteristic of those baselines with respect to the proposed one? What can we learn from the result of this comparison i.e., why does the proposed model outperform?\n8. (Minor) Line 34: Would be good to cite the paper(s) that support this statement “By biasing architectures to process visual inputs in terms of relations between objects, these recent approaches have achieved strong systematic (i.e., out-of-distribution) generalization of learned abstract rules, given only a small number of training examples.”\n\nI am giving a score of 6 in the hope that the authors will try and address some of my questions and concerns. If addressed to some degree, I will happily increase the score.\n\nLIMITATIONS:\nYes, the limitations are discussed in the last line of the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research topic of this study is the development of a learning machine that achieves systematic generalization in reasoning over complex relations of objects in a visual input (still image). Toward this goal, the authors proposed a new neural network model (OCRA) taking inspirations from the recent results on effective inductive biases for systematic generalization in relational reasoning and methods for obtaining object-centric representations. More concretely, the proposed model comprised of three core components: first component (slot attention mechanism) to extract object-centric representation from a visual input containing multiple objects, the second component to compute pairwise relation embeddings, and the third component (transformer) to provide the final output related to the higher-order relations. The effectiveness of the proposed model was tested with three visual reasoning tasks, two existing and one new, and also compared with various baseline models. As a whole, the proposed model can be said the best in terms of systematic generalization among the compared models.  An ablation study was also conducted to evaluate the roles of the components in the proposed model and pretraining.\n\nSTRENGTHS:\nThe top-level idea behind the proposed model, that is, combining the inductive biases for relational reasoning and a method for obtaining object-centric representations is clear and reasonable. How to implement the idea with the three core components are explained fairly well. Although the top-level idea might look somewhat straightforward at first glance given the advancement in the two directions, the concrete implementation is not trivial, and the differences from the existing work are described in the paper. \n\nThe effectiveness of the proposed model was tested with fairly rich experiments. Two existing task (ART and SVRT, both created with 2D shapes) and a new task developed based on the CLEVR (CLEVR-ART,  created with 3D shapes) were used and various baseline models including a very recent one (GAMR, accepted at this year's ICLR) are compared with the proposed model. An ablation study gives additional value to this work. \n\nDevelopment of a learning machine that achieves systematic generalization in complex visual reasoning is an important topic in AI. Although there is still a distance to the real-world applications as stated in Section 6 (Conclusion and Future Directions), the proposed method and the evaluation results will be of interest to the NeurIPS audience.\n\nWEAKNESSES:\n1. A weak point of this submission is lack of the source code of the proposed model as Supplementary Material. It is also unkown whether the source code will be made publicly available if the paper get accepted. These points cast a shadow on the reproducibility.\n\n1. A relatively weak point of the proposed model itself seems to be around the number of slots, $K$. First, as stated in line 117, the proposed model computes all $K^2$ pairwise relations. Second, in the current model, $K$ should be defined in advance. These characteristics can be obstacles in the reasoning over the relations of objects in the real-world visual inputs. Namely, the fist point might affect the applicability (scalability) to complex natural images and there should be some additional mechanism to determine an appropriate $K$ in the first place if the current structure of the proposed model is kept. \n\n1. (Related to 2.) Although it is stated that there exists gap between the problems addressed in this study and the real-world vision in Section 6 (Conclusion and Future Directions), the details are not explained.\n\nQUESTIONS:\nQuestions\n\n1. Can the source code of the proposed method be provided at the Author Rebuttal period? In addition, will the source code made publicly available if the paper get accepted?\n\n1.  What are your thoughts on the second point in Weaknesses section above? \n\nSuggestion\n\nIt would be beneficial if the authors could add detailed descriptions about what kind of differences between the problems treated in this study and the real-world vision should be overcome, and also add explanations about the relation between those differences and  self-supervised learning methods mentioned in Section 6 if possible.\n\nLIMITATIONS:\nThe authors stated in Section 6 (Conclusion and Future Directions) that the problems addressed in this study are still simple compared with the the real-world vision.  However, currently it is not explained in details what kind of differences between the problems treated in this study and the real-world vision should be overcome. Please also refer to the Weaknesses and Questions sections above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a new method, named OCRA, that combines object-centric presentation learning (for object abstraction) and a relational bottleneck (for relational abstraction). Particularly, OCRA consists of three components: 1) a slot attention to extract object level representations, 2) a relational operator to get pairwise visual relations, and 3) a transformer to model higher-order relations. The slot attention model is pretrained on a large dataset and the relational modules are trained on a small task-specific dataset while freezing the slot attention. Experiments were performed on three datasets: ART, SVRT and CLEVR-ART to show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The idea of combining slot attention with a relational bottleneck (relation operator and transformer) for solving visual relational reasoning problems sounds interesting and also novel to me. \n2. The presentation of the idea and the overall writing are very clear. \n3. Experiments on synthetic datasets (ART and SVRT and CLEVR-ART) shows the proposed method can work for various relational reasoning tasks and it achieves better performance than baselines in many cases.\n\nWEAKNESSES:\n1. For the benchmarks, I have a concern about their simplicity. For example, the performance of many methods on the ART dataset is close to 100%. Similar observations also exist in other two datasets. Does it mean we already have achieved human-level visual reasoning performance or we need better benchmarks to evaluate the success of methods? I think considering some more challenging benchmarks will make the results more convincing. For example, RAVEN [1] and Bongard-HOI [2] are two challenging benchmarks for testing a model's visual relational reasoning abilities. In particular, Bongard-HOI considers the real-world natural images.\n2. In Figure 4, it is a little surprising that OCRA performs worse than ResNet on SVRT - spatial relations with 1000 training examples. Any intuition on this?\n3. For the ablation studies, to test the impact of slot attention, it is too naive to do “feature map divided into a 4x4 grid”. How about comparing slot attention with some standard SSL trained object-centric representation learning methods? Also, from Table 2, we can see that Transformer has a much higher impact than the slot attention and relational bottleneck on both RMTS and ID. Does it mean the most important part of OCRA is the transformer rather than slot attention and relational bottleneck? If so, it downplays the method's significance a little, in my opinion.\n\n[1] RAVEN: A Dataset for Relational and Analogical Visual rEasoNing, CVPR 2019.\n\n[2] Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions, CVPR 2022.\n\nQUESTIONS:\nMy major concerns and questions are in how the experiments support the effectiveness and significance of the proposed method. Please see the weaknesses part for more details.\n\nLIMITATIONS:\nThe authors have well addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}