Neural Slot Interpreters: Grounding Object Semantics in Emergent Slot Representations

## Abstract

Abstract Several accounts of human cognition posit that our intelligence is rooted in our ability to form abstract composable
concepts, ground them in our environment, and reason over these grounded entities. This trifecta of human
thought has remained elusive in modern intelligent machines. In this work, we investigate whether slot representations
extracted from visual scenes serve as appropriate compositional abstractions for grounding and reasoning. We present the
Neural Slot Interpreter (NSI), which learns to ground object semantics in slots. At the core of NSI is an XML -like
schema that uses simple syntax rules to organize the object semantics of a scene into object-centric schema primitives.
Then, the NSI metric learns to ground primitives into slots through a structured objective that reasons over the intermodal
alignment. We show that the grounded slots surpass unsupervised slots in real-world object discovery and scale with scene
complexity. Experiments with a bi-modal object-property and scene retrieval task demonstrate the grounding efficacy and
interpretability of correspondences learned by NSI. Finally, we investigate the reasoning abilities of the grounded slots.
Vision Transformers trained on grounding-aware NSI tokenizers using as few as ten tokens outperform patch-based tokens on
challenging few-shot classification tasks.

## 1 Introduction

> The body is not a thing among things; it is the condition of all things Maurice Merleau-Ponty

Humans possess a repertoire of strong structural biases, a kind of abstract knowledge that enables us to perceive and rapidly
adapt to our environments (Griffiths et al., ). Compositionality is one such structural prior that helps us
systematically reason about complex stimuli as a whole by recursively reasoning about its parts
(Zuberbühler, ; Lake & Baroni, ). We decompose broad motor skills into finer dexterous finger movements, sentences into
words and phrases, and speech into phonemes. In the visual world, the concept of “objectness” serves as a natural
compositional prior, enabling us to decompose novel scenes into familiar objects and reason about their properties
(Lake et al., ). We also have the uncanny ability to connect real-world entities and concepts
to these abstract object-like symbols in our heads, canonically referred to as the grounding problem
(Harnad, ; Greff et al., ). For instance, human infants, while looking at a zebra for the first time, might excitedly
conclude that it is, in fact, “a striped horse.” If grounded object-like representations are fundamental to human-like
compositional generalization, how do we instill these inductive biases into neural network representations?

Unsupervised object-centric autoencoder models (Burgess et al., ; Greff et al., ; ; Locatello et al., ; Engelcke et al., ; ; Singh et al., ; Chang et al., ; Seitzer et al., ; Kori et al., ) have become
increasingly adept at learning object-centric representations called slots from raw visual stimuli. Further work has
demonstrated that learned slots can be flexibly composed together for tasks like scene composition, causal induction,
learning intuitive physics, dynamics simulation, and control (Dedhia et al., ; Jiang et al., ; Wu et al., ; ; Chang et al., ; Jabri et al., ). While object slots hold promise as a compositional
building block for machines that mimic human abstraction and generalization, a key challenge emerges. Unlike humans, these
learned slots lack grounding in real-world concepts. For example, a slot representation of an object like “apple” could
refer to the fruit, the company, or a generic round artifact. Without grounding, a slot-based system cannot disambiguate
these meanings effectively (Haugeland, ) and is fundamentally limited in its embodied reasoning abilities.
Prior works (Locatello et al., ; Seitzer et al., ; Kori et al., ) have tackled learning to ground slots by
predicting object properties (texture, material, category, etc.) from the representations. The grounding objective is,
therefore, implicit within the prediction of object semantics. However, ground truth correspondences between object
concepts and slots are generally unknown, restricting prediction to a set-matching template. Under a set-matching framework,
a single slot predicts object properties of a single object, thereby constraining the grounding information
assimilated per slot. We circumvent the limitations of prediction as a surrogate for grounding by making the grounding
objective explicit in the form of a co-training paradigm that we call the Neural Slot Interpreter (NSI).

Figure: Figure 1: NSI abstracts grounded slots from scenes and enhances object discovery, grounding efficacy, and downstream reasoning abilities of slot representations.
Refer to caption: x1.png

The core insight behind NSI is simple: instead of predicting a single object-concept from a slot, assign
multiple concepts to slots over a shared latent space. Our primary contribution is a similarity metric that explicitly reasons
about the intermodal assignments. Notably, the proposed metric for NSI supplants the one-object-per-slot assumption and
facilitates flexible assignment. Contrastive learning over the similarity objective yields grounded slots that outperform
their ungrounded and set-matched counterparts over a broad swath of tasks (see Fig. 1). We propose an
object-centric annotation schema in Section 4.1 for dense alignment to organize scene annotations for grounding
into slots. We describe the design of a hierarchical transformer-based architecture in Section 4.2.2
to extract neural representations from the schema. To enable slots to ground a wide array of concepts flexibly without relying
on matching templates, we formulate a bi-level scoring metric over a learned latent space in Section 4.2.

Do NSI modules require specialized training recipes? Our experiments demonstrate that NSI is a plug-and-play paradigm
that can be easily augmented to the traditional object-centric learning objective. Does NSI preserve compositionality?
NSI preserves and often improves visual compositionality, as shown in Section 5.2, where we demonstrate
its competitiveness on object discovery benchmarks. Are notions of objects effectively grounded in emergent slot
representations? We validate the efficacy of grounded slots on a bimodal property-image retrieval task in
Section 5.3, where we show that NSI surpasses the state of the art. Are NSI-grounded slots effective
substrates for visual reasoning? In Section 5.4, we train a vision transformer (ViT)
(Dosovitskiy et al., ) for a scene classification task where significantly fewer grounded slot tokens
show improved performance and adaptability over traditional patch-based tokens. Can dense associations learned by
NSI inform real-world reasoning systems? Our experiments described in Appendix E.1 show the usefulness of
learned correspondences in identifying and locating objects in diverse scenes. Concretely, our contributions are as follows:

- 1.
We present NSI, a co-training grounding paradigm for object-centric learners. We also propose an object-centric
annotation schema called visXML for dense slot-label alignment. We formulate a similarity metric that measures
scene-schema similarity by recursively reasoning over the similarity of compositional attributes of the respective modalities.
NSI utilizes the metric to ground slots via a contrastive learning objective.
- 2.
Our experiments demonstrate the efficacy of NSI over a wide array of tasks that encompass (1) object discovery,
(2) scene-property retrieval, (3) few-shot scene classification, and (4) object detection. Overall, we find that grounded
slot representations are key to object-centric perception, property grounding, and downstream adaptability for object-centric
reasoning.

## 2 Related Work

Object-Centric Learning. Researchers have formulated inductive biases for learning composable visual representations
called ‘slots’ from raw visual stimuli (Burgess et al., ; Greff et al., ; Locatello et al., ; Greff et al., ; Engelcke et al., ; ; Singh et al., ; Seitzer et al., ) and auxiliary temporal information (Kipf et al., ; Elsayed et al., ; Singh et al., ). While this line of work demonstrates unsupervised object discovery, the adoption of slot representations
for grounding scenes remains largely underexplored. Prior works have been limited to using the Hungarian Matching Criterion to align
slots to ground-truth property labels for property prediction (Locatello et al., ) or fine-tuning shallow property
predictors on pre-trained backbones (Seitzer et al., ). A recent work (Kori et al., ) improves the
predictive power of slots by learning quantized priors. Fundamentally, the single-object prediction per slot constraint poses
the difficult problem of learning highly specialized representations. In contrast, this work proposes an approach that
explicitly reasons about grounding via a flexible assignment metric.

Visual Tokenizers. Patch-based tokens have been adopted as the standard for visual understanding
(Dosovitskiy et al., ) and generation (Peebles & Xie, ). Variations
of this template include discretized patch tokens (Du et al., ), mixed-resolution patch tokens
(Ronen et al., ), and pruned patch tokens
(Kong et al., ; Tang et al., ). Beyond patches, recent works have explored
region-based tokens (Ma et al., ). However, these tokenizers are inherently
grounding-agnostic, in contrast to humans, who possess the ability to abstract concepts based on linguistic or cultural
grounding priors, (Segall et al., ; Winawer et al., ). To this end, our work explores a grounding-aware tokenizer.

Program Induction. Such problems occur in many guises across computer
vision (Li et al., ; Wu et al., ), natural language processing (Xu et al., ; Devlin et al., ), and cognitive
science (Ellis et al., ). Grounding slots in an annotation schema via NSI is akin to neural program induction in
the schema space.

## 3 Preliminaries

We detail a few essential preliminaries in this section. Readers should refer to the original manuscripts for further details.

### 3.1 Slot Attention

Object-centric learning frameworks decompose scenes by organizing them into semantic object files called slots. Slot Attention
(SA) (Locatello et al., ) is a powerful iterative attention mechanism for learning such slots from perceptual features
extracted from vision backbones. At iteration $t$, for $N$ features $H\in\mathbb{R}^{L\times c}$ and $K$ slots $S^{t}\in\mathbb{R}^{K\times d}$, the slots compete to explain the features as follows:

$$ $\displaystyle M=\frac{K(H)Q(S^{t})^{T}}{\sqrt{D}}\in\mathbb{R}^{L\times K};A_{ ij}=\frac{e^{M_{ij}}}{\sum_{j\in\{1,\cdots,K\}}e^{M_{ij}}}$ (1) $\displaystyle S^{t+1}=W^{T}V(S^{t})\quad\text{where}\quad W_{ij}=\frac{A_{ij}} {\sum_{i\in\{1,\cdots,N\}}A_{ij}}$ (2) $$

Here, $Q,\;K,\;V$ are learned query, key, and value matrices, respectively. Note that we present only the attention
operations of SA and refer the reader to the original article by (Locatello et al., ) to understand the complete rollout.

### 3.2 DINO Vision Transformer

DINO (Caron et al., ) is a self-supervised knowledge distillation technique that leads to emergent objectness biases
in Vision Transformers (ViTs). These inductive biases make them excellent candidates for learning object representations.
A recent object-centric learning method (Seitzer et al., ) uses the DINO backbone to scale to complex real-world
datasets and reports state-of-the-art results. Motivated by these findings, we use DINO to extract perceptual features for SA.

## 4 Neural Slot Interpreters

Recall that the goal of NSI is to ground concepts into slot representations such that the objects contained within the slot
align with the embodied notions of the object. We begin by discussing our proposed Extensible
Markup Language (XML) for annotating object concepts.

### 4.1 visXML Object-Centric Annotation Schema

Figure: Figure 2: visXML description of a real-world scene. The dotted arrows show correspondences between primitives and the objects they annotate.
Refer to caption: x2.png

visXML is a simple markup syntax for abstracting scene labels as a collection of objects and their associated
properties. Instances in visXML comprise multiple object-centric primitives. Primitives, in turn, are
instantiated with respective object properties that form atomic units of the language. Each primitive starts with an
<element> tag that identifies a unique object in the scene. The <$p_{j}$> tags form the children
of the <element> tags and capture properties $p_{1},\cdots,p_{J}$ of the parent object. Some examples of
<$p_{j}$> tags are, but not limited to, shape, material, category, and object position. Thus, instance
primitives naturally capture the notion of an object and neural representations extracted from primitives are, as such,
well suited for being grounded in slots. In Section 5.1, we demonstrate the straightforward application of
visXML organization on popular datasets. See Fig. 2 for an example instance and
Appendix A.3 for more examples. On a more practical note, such markup languages are commonly-used software
abstractions and can be ubiquitously interfaced with graphics engines, web APIs, or even large language models for semantic
understanding (Dunn et al., ; Bubeck et al., ).

### 4.2 NSI Grounding Technique

Scenes and their corresponding visXML instances capture object-centric representations through slots and primitives,
respectively. NSI learns to align the object-centric representations of the respective modalities, i.e., slots
and primitives, by grounding neural representations of schema primitives into slots (see Fig. 3). The grounding is learned by optimizing a
contrastive learning objective over ground-truth scene-schema pairs (see Fig. 4). We describe the scene
and schema encoder next.

#### 4.2.1 Learning Scene Representations

A given scene $I_{x}$ is represented via $K$ slots $S_{x}^{1:K}\in\mathbb{R}^{K\times d}$ abstracted from its perceptual features
$H_{x}\in\mathbb{R}^{L\times c}$. For a given feature extractor $\mathcal{E}_{\phi}(.)$, the slots are obtained as

$$ $\displaystyle H_{x}=\mathcal{E}_{\phi}\left(I_{x}\right)\quad\Rightarrow\quad S _{x}^{1:K}=SA\left(H_{x}\right)\rightarrow\text{Slot Attention}$ (3) $$

A spatial broadcast decoder $\mathcal{D}_{\theta}(.)$ (Locatello et al., ) reconstructs the
features from slots, with the reconstruction error used as a learning signal:

$$ $\displaystyle\hat{H}_{x}=\mathcal{D}_{\theta}\left(S_{x}^{1:K}\right)\quad \Rightarrow\quad\mathcal{L}_{recon}=\left\lVert H_{x}-\hat{H}_{x}\right\rVert^ {2}$ (4) $$

#### 4.2.2 Learning Schema Representations

A bi-level architecture (see Appendix B.2) learns neural representations of the visXML
primitives. First, a lower-level primitive encoder learns property-specific dictionaries $D(.)$ and embeds the
property features into neural primitive representations. For discrete-valued properties, $D(.)$ is modeled as a simple lookup
table of learnable weights, while continuous-valued properties are embedded via multi-layered perceptrons (MLPs). Let the
dictionary $D_{j}(.)$ learn features for property $p_{j}$. Then, a primitive embedding $Z_{prim}$ is computed as:

$$ $\displaystyle Z=concat\left[D_{1}(p_{1}),\cdots,D_{J}(p_{J})\right]\quad \Rightarrow\quad Z_{prim}=MLP(Z)$ (5) $$

Note that these lower-level representations are schema-agnostic and only capture object-specific features. Then an upper-level
schema encoder uses a bidirectional schema Transformer to further embed primitives, endowing representations with the
overall schema context. For a given schema instance $P_{y}$ with $N$ primitives, the final representations
$Z_{y}^{1:N}\in\mathbb{R}^{N\times d}$ are computed via Transformer $\mathcal{T}_{schema}(.)$ as:

$$ $\displaystyle Z_{y}^{1:N}=\mathcal{T}_{schema}\left(Z_{prim}^{1},\cdots,Z_{ prim}^{N}\right)$ (6) $$

Figure: Figure 3: NSI Architecture. NSI augments object-centric learning autoencoders with a contrastive learning objective over a batch of scene-schema pairs. A DINOSAUR backbone (Seitzer et al., ) extracts slot representations $\mathcal{S}_{x}^{1:K}$ from a batch of scenes and a schema encoder extracts neural primitives $\mathcal{Z}_{y}^{1:N}$ from their corresponding schema pair. The slots are then passed to a decoder for reconstruction and the slot-primitive neural pairs are passed to the contrastive learning objective.
Refer to caption: x3.png

#### 4.2.3 Compositional Score Aggregation

Recall that we want to ground entities $Z_{y}^{1:N}$ into the entities $S_{x}^{1:K}$. As a first step, we project these embeddings
into a shared semantic space $Y\in\mathbb{R}^{d_{proj}}$. The projection head $\mathcal{H}_{scene}(.)$ for slots is modeled
as the following residual network:

$$ $\displaystyle\tilde{Y}^{k}_{x}=W_{proj}S^{k}_{x},\;W\in\mathbb{R}^{d_{proj} \times d}\quad\Rightarrow\quad Y^{k}_{x}=\tilde{Y}^{k}_{x}+MLP\left(LayerNorm \left(\tilde{Y}^{k}_{x}\right)\right)$ (7) $$

Here, $LayerNorm$ denotes the layer normalization operation. A separate residual head $\mathcal{H}_{schema}(.)$ projects the
primitive representations $Z_{y}^{1:N}$ into the semantic embeddings $Y^{1:N}_{y}$. Next, we supplant the traditional single object
per slot assumption by assigning each primitive to its nearest slot in the latent space, as measured by dot-product similarity.
The similarity score $\mathcal{S}_{xy}$ between a scene $x$ and primitive $y$ is the sum of nearest-neighbor
similarities resulting from the primitive-slot assignment.

$$ $\displaystyle k^{*}_{n}=\operatorname*{arg\,max}_{k\in\{1,\cdots,K\}}{Y^{k}_{x }}^{T}{Y^{n}_{y}}\quad\Rightarrow\quad\mathcal{S}_{xy}=\sum_{n\in 1\cdots N} \max_{k\in\{1,\cdots,K\}}\left({Y^{k}_{x}}^{T}{Y^{n}_{y}}\right)$ (8) $$

#### 4.2.4 Contrastive Learning Objective

The modality-specific embeddings and the resultant grounding are learned by optimizing a contrastive learning objective. More
precisely, given a $B$-sized batch of {scene, schema} pairs, we use the $S_{xy}$ scores to distinguish the $B$ correct pairs
from the $B^{2}-B$ incorrect pairs. The probability of correctly classifying schema $P_{x}$ as the true pairing for scheme
$I_{x}$ (and conversely predicting $I_{x}$ from $P_{x}$) is formulated as follows:

$$ $\displaystyle\mathbb{P}^{schema}_{x}=\frac{exp\left(\mathcal{S}_{xx}/\tau \right)}{\sum_{y\in\{1,\cdots,B\}}exp\left(\mathcal{S}_{xy}/\tau\right)}\quad \Rightarrow\quad\mathbb{P}^{scene}_{x}=\frac{exp\left(\mathcal{S}_{xx}/\tau \right)}{\sum_{y\in\{1,\cdots,B\}}exp\left(\mathcal{S}_{yx}/\tau\right)}$ (9) $$

Here, the calculated scores are interpreted as logits and $\tau$ denotes the temperature parameter.
The cross-entropy losses for scene and schema prediction are given by:

$$ $\displaystyle\mathcal{L}_{schema}=-\sum_{x\in\{1,\cdots,B\}}log\left(\mathbb{P }^{schema}_{x}\right);\mathcal{L}_{scene}=-\sum_{x\in\{1,\cdots,B\}}log\left( \mathbb{P}^{scene}_{x}\right)$ (10) $$

The global contrastive learning objective is based on a symmetric cross-entropy (CE) loss as follows:

$$ $\displaystyle\mathcal{L}_{contrastive}=(\mathcal{L}_{scene}+\mathcal{L}_{ schema})/2$ (11) $$

The overall training objective for NSI is given by:

$$ $\displaystyle\mathcal{L}_{train}=\beta_{1}\times\mathcal{L}_{contrastive}+ \beta_{2}\times\mathcal{L}_{recon}$ (12) $$

Note that $\beta_{1}=0.0,\beta_{2}=1.0$ corresponds to traditional autoencoder object-centric learning frameworks. Algorithm 1 contains Python-style pseudocode for the NSI metric and subsequent contrastive learning
objective.

Figure: Figure 4: NSI similarity metric. (a) The inner loop of the metric computes the score $S_{xy}$ between compositional abstractions of an image $I_{x}$ and a schema $P_{y}$. Object slots and schema primitives are projected onto a shared embedding space and every latent primitive is assigned to its nearest slot for score aggregation. (b) The $S_{xy}$ scores obtained from local entities are used to optimize a global contrastive learning objective in the outer loop over a batch of image-schema pairs.
Refer to caption: x4.png

Figure: Algorithm 1 Neural Slot Interpreter Contrastive Learning Pseudocode

## 5 Experiments

In this section, we set up experiments to answer (1) whether the NSI objective leads to the emergence of slots that bind
to raw object features, (2) whether the slots are concurrently and coherently imbued into the notion of the object properties,
and (3) if the slots are effectively grounded, do they enable improved reasoning over objects.

### 5.1 visXML Instantiation and Architecture Backbone

Our experiments encompass different tasks on scenes ranging from synthetic renderings to in-the-wild scenes viz. (1) CLEVr
Hans (Stammer et al., ): objects scattered on a plane, (2) CLEVrTex (Karazija et al., ): textured objects
placed on textured backgrounds (3) MOVi-C (Greff et al., ): photorealistic objects on real-world surfaces, and
(4) MS-COCO 2017 (Lin et al., ): a large-scale object detection dataset containing real-world images. In a
pre-processing step, we organize scene labels into the visXML schema (Section 4.1). The property
tags <$p_{j}$> that populate schema primitives in each dataset are listed in Table 1.
We use the schema instances to ground object information in their corresponding scenes via NSI. See
Appendix A.3 for visXML instances.

**Table 1: Property tags used to instantiate and ground schema primitives.**
| Dataset | Property tags <$p_{j}$> |
| --- | --- |
| CLEVr Hans | <color>, <shape>, <material>, <size>, <3D position> |
| CLEVrTex | <texture>, <shape>, <size>, <3D position> |
| MOVi-C | <category>, <scale>, <2D position>, <bounding box> |
| MS-COCO 2017 | <category>, <bounding box> |

We followed the DINOSAUR (Seitzer et al., ) recipe for learning slot representations. DINOSAUR uses
semantically-informative DINO ViT (Caron et al., ) features as an autoencoding objective, significantly improving
real-world object-centric learning abilities. More specifically, we train an MLP decoder to reconstruct features from slots
for all our experiments. Training details and hyperparameters are given in Appendix C.2.

### 5.2 Object Discovery

Object-centric frameworks have been traditionally used to bind neural network representations to distinct objects within a
scene. Here, we evaluate whether grounded slots are more adept at discovering objects in the context of visual segmentation.
To systematically probe the effect of grounding, we use a common backbone trained within the same compute budget and ablated
across the grounding continuum: (a) ungrounded slots derived from the autoencoder objective, (b) set-matching
prediction via Hungarian Matching Criterion (HMC), and (c) latent semantic assignment via NSI.

Figure: Figure 5: Object discovery results. (a) Segmentation results on three datasets. We report scores of $FG$-$ARI$ (higher is better) and $mBO$ (higher is better). We also show scores relative to the ungrounded baseline on top of each bar. (b) Visualization of attention masks learned by different models on instances of the datasets. See Appendix D.1.1 for error bars and more results.
Refer to caption: x5.png

For each method, we extracted object masks from the backbone derived from slot-attention clusters and reported two
segmentation metrics in Fig. 5(a): (1) Foreground Adjusted Rand Index ($\mathbf{FG}$-$\mathbf{ARI}$):
measures the accuracy of clustering foreground objects into their respective segments and (2) Mean Best Overlap
($\mathbf{mBO}$): assesses the best overlap between predicted and ground truth object masks. We report both instance
($mBO^{i}$) and class ($mBO^{c}$) level $mBO$ scores for COCO. Mask instances are shown in Fig. 5(b). We observed that:

- (a)
NSI endowed slots meaningfully segment objects. Object masks generated by NSI are competitive on synthetic
scenes and markedly improve object discovery on COCO scenes. We posit that contrastive learning via NSI enhances
symmetry breaking of the slot attention backbone for challenging real-world scenes, effectively binding slots to raw visual
features.
- (b)
HMC matching obscures object discovery on COCO. Constraining slots to predict a single object forces the
backbone to develop specialized representations for each object. In real-world scenes, this imposes a difficult learning
problem and causes slots to deteriorate.
- (c)
NSI is biased towards semantic classes over instance classes. On COCO instances, semantic segmentation scores
($mBO^{c}$) are higher compared to the baseline. We attribute this to the NSI metric that biases slots to represent broader
categories by grounding multiple objects.

### 5.3 Grounded Compositional Semantics

Grounded concepts should be effectively aligned to the slots they represent. To this end, we set up a bimodal scene-property
retrieval task that evaluates the grounding efficacy of various methods, i.e., the degree to which they
learn to associate object properties with their representations. In the first half, models retrieve a set of object
properties from a database, given the scene, using their respective alignment scores. In the second half, the task is inverted,
where the models search for scenes, given the object properties. We use the test split across datasets as a retrieval database.
For NSI, we use similarity scores $\mathcal{S}_{xy}$ as the retrieval metric and also evaluate various comparisons.

- (a)
CLIP embeddings: Such embeddings (Radford et al., ) form a strong
non-slot baseline. Here, we encode the schema as a string and measure the similarity between the text-image CLIP embeddings.
- (b)
Ungrounded slots (Seitzer et al., ): We freeze the backbone and fine-tune a shallow predictor on
it for property prediction using HMC (Kuhn, ). Subsequently, we use HMC scores for retrieval.
- (c)
HMC matching: We fine-tune the slot architecture end-to-end to predict object properties from slots.
We use the optimal HMC scores of fine-tuned slots for retrieval.
- (d)
Ablations: On NSI, where NSI-ResNet 34 replaces DINO ViT with the ResNet backbone, as
described in (Elsayed et al., ), and on NSI-Schema Agnostic where schema primitives are encoded without the
schema Transformer (Appendix B.2).

In Fig. 6(a),(b),(c), we report Recall@(1/5), which denotes the accuracy with
which the correct scene (property) is among the top (1/5) retrieved entities.

- (a)
Grounded slots via NSI significantly improve semantic alignment compared to set matching: NSI and its
ablations that explicitly reason about compositional semantics enable improved retrieval compared to slots that rely on
set-matching prediction. The performance gap widens as we evaluate the methods on more realistic datasets like MOVi-C and
COCO.
- (b)
The full model is essential: Schema-agnostic encoders that encode object primitives independently
perform sub-par compared to NSI. The pre-trained DINO backbone is crucial for textured objects and real-world generalization,
as evidenced by CLEVrTex/COCO results.
- (c)
Non-compositional embeddings inadequately capture object semantics: CLIP embeddings fall short in
capturing object properties beyond basic semantic categories. Moreover, natural language appears insufficiently equipped for
object-centric grounding (Chandu et al., ).

Qualitative interpretation: In Fig. 6(d),(e),(f), we visualize the slot-object
pairs inferred by our scoring metric. Interpretable and dense correspondences emerge from NSI contrastive learning. In
real-world COCO scenes, we find that slots ground as many as ten objects. See Appendix D.2.5 for
more results.

Figure: Figure 6: Retrieval results. (a), (b), (c) Property and scene retrieval results. We report Recall$@1/5$ (higher is better). Standard deviation (over five seeds) was $<0.3$ across all model instances and retrieval tasks. (d), (e), (f) Visualization of correspondences learned by the NSI similarity metric. The colored arrows show the respective correspondences of schema primitives to the slots. Each schema instance is chunked and color-coded by the slot to which its primitives are assigned.
Refer to caption: x6.png

### 5.4 Grounded Slots as Visual Tokens

Figure: Figure 7: Few-shot classification results. (a) (b) Few-shot classification accuracy on Hans 3 and Hans 7 datasets, respectively. (c) Task accuracy on NSI-CLEVr tokenizers trained on different numbers of grounding examples. Standard deviation (over five seeds) was $<$ 0.03 across tasks and methods. (d) Visual rationales generated across the grounding continuum by extracting attention maps from the final ViT layer. See Appendix D.3.2 for more examples.
Refer to caption: x7.png

Grounding-agnostic patch-based tokens are the de facto standard for transformer-based models. On the other hand,
humans can flexibly abstract out entities free of rigid geometric templates. Here, we investigate the ability of grounded
slots to bridge this abstraction gap by training a ViT architecture on slot-based tokenizers for a few-shot classification
task. The test suites are derived from the CLEVr-Hans reasoning benchmark (Stammer et al., ) that consists of multiple
classes based on object attributes and relations. Moreover, the true membership properties are confounded with other
attributes in the train split. We explore the following tokenization schemes:

- (a)
Traditional patch tokens extracted from $14\times 14$ image patches.
- (b)
Ungrounded slot tokens learned from autoencoder object-centric learning on CLEVr.
- (c)
Conditional Slot Attention (CoSA) tokens (Kori et al., ) derived from set-matching.
- (d)
NSI-CLEVrTex slot attention trained on semantically-similar CLEVrTex via NSI. The backbone is frozen and
subsequently used to infer CLEVr slots, making it partially grounded.
- (e)
Fully grounded NSI-CLEVr slot attention trained on CLEVr via NSI.

In Fig. 7(a),(b), we report test accuracy against the $k-$shot training sweep for CLEVr-Hans 3 and
CLEVr-Hans 7. We found that

- (a)
Grounded slots facilitate improved reasoning and surpass patch-based tokenizers across data regimes
using $25\times$ fewer tokens. While the performance of the patch-tokens saturates, grounded slots deconfound object
attributes with greater ability with increasing data.
- (b)
Partial grounding is often sufficient: NSI-CLEVrTex slots that are grounded in a different dataset
demonstrate competitive reasoning capabilities.
- (c)
Property prediction ability does not necessarily transfer to class prediction: CoSA slots trained on
object-attribute prediction show weaker transfer on the Hans-3 dataset.

How many visXML annotations are essential? In Fig. 7(c), when ablating the number of
annotations used to train the NSI-CLEVr tokenizer, we observe that inductive biases instilled by grounding are key, as seen
from the sensitivity of the performance to the number of examples. On the other hand, significant annotation of scenes is not
necessary. Annotating just 100 visXML examples yields performative accuracy within a $3\%$ margin of the tokenizer
trained on the complete set.

Qualitative interpretation: We probe the attention maps from the final layer of the ViT to generate visual
rationales. Fig. 7(d) visualizes the maps across the grounding continuum. The ungrounded slots show little
to no correlation with the class rule. On the other hand, ViT trained on the NSI-CLEVr slots weighs slots pertinent to the
class rule with greater attention. Grounded slots provide abstractions for reasoning that are not only compute-efficient
substrates but also yield interpretable visual rationales.

## 6 Conclusion

This work introduced NSI, which grounds object semantics into slots for object-centric understanding. It uses a simple
schema abstraction to define object concepts and learns to flexibly associate neural embeddings of the schema primitives
with object slots via contrastive learning. Unlike set-matching approaches, which struggle when scaled to real-world scenes,
NSI enhances object discovery compared to ungrounded counterparts. Further, NSI facilitates interpretable grounding in
slot representations. Whereas natural language-grounded embeddings struggle to retrieve granular object properties, we find
that NSI embeddings abstracted from the slot-schema intermodal alignment are key representations for such tasks. Finally,
we demonstrated the usefulness of NSI as a grounding-aware visual tokenizer that improves the few-shot visual reasoning
abilities of ViTs on a hard classification task. While annotations can be prohibitively expensive and laborious, we also
demonstrated performative reasoners under practical annotation settings. Slot representations have traditionally been
associated with visual stimuli, but object concepts transcend perception to other sensorimotor experiences like audio,
tactile signals, and motor behaviors. To this end, NSI lays the groundwork for multimodal object-centric learning. Future
work involves adopting NSI to ground common object-centric concepts into different sensorimotor experiences as a step
towards a modular human-like understanding of the world.

## 7 Acknowledgment

This work was supported by NSF under grant no. CCF2203399. The experiments reported in this article were performed on the computational resources managed and supported by Princeton Research Computing at Princeton University. We also express gratitude to The Sacred Grounds Cafe and Karma Cafe in San Francisco for brewing the best coffee North of the Panhandle.

## Appendix A Data

This section contains additional information on the datasets, visXML schema space instantiation, and
visXML examples.

### A.1 visXML Schema Space Description

Table 2 lists the various object properties used to create visXML description of scenes.

**Table 2: visXML schema space across various datasets.**
| Dataset | Property | Discrete/Continuous | Size |
| --- | --- | --- | --- |
| CLEVr Hans (Johnson et al., ; Stammer et al., ) | Material | Discrete | 2 |
| Color | Discrete | 8 |  |
| Shape | Discrete | 3 |  |
| Size | Discrete | 2 |  |
| Object Position | Continuous | 3 |  |
| CLEVrTex (Karazija et al., ) | Texture | Discrete | 60 |
| Shape | Discrete | 4 |  |
| Size | Discrete | 3 |  |
| Object Position | Continuous | 3 |  |
| MOVi-C (Greff et al., ) | Object Category | Discrete | 17 |
| Object Size | Continuous | 1 |  |
| Object Position | Continuous | 2 |  |
| Bounding Box | Continuous | 4 |  |
| MS-COCO 2017 (Lin et al., ) | Object Category | Discrete | 90 |
| Bounding Box | Continuous | 4 |  |

### A.2 Dataset Splits

The dataset splits used in this work are detailed in Table 3.

**Table 3: Dataset splits used in experiments.**
| Name | Train Split Size | Validation Split Size | Test Split Size |
| --- | --- | --- | --- |
| CLEVr-Hans 3 | 9000 | 2250 | 2250 |
| CLEVr-Hans 7 | 21000 | 5250 | 5250 |
| CLEVrTex | 37500 | 2500 | 10000 |
| MOVi-C | 198635 | 35053 | 6000 |
| MS COCO 2017 | 99676 | 17590 | 4952 |

### A.3 visXML examples

Figs. 8-11 show visXML descriptions of instances from all the datasets.

Figure: Figure 8: CLEVr-Hans instance with its corresponding visXML description.
Refer to caption: extracted/5869070/figs/clevr.png

Figure: Figure 9: CLEVrTex instance with its corresponding visXML description.
Refer to caption: extracted/5869070/figs/clevrtex.png

Figure: Figure 10: MOVi-C instance with its corresponding visXML description.
Refer to caption: extracted/5869070/figs/movi.png

Figure: Figure 11: COCO instance with its corresponding visXML description.
Refer to caption: extracted/5869070/figs/coco.png

## Appendix B Neural Slot Interpreter

Next, we present details of NSI training modules.

### B.1 NSI Scene Encoder

Fig. 12 shows the DINOSAUR encoder for learning slot representations. Here, the slot attention
mechanism operates on pre-trained DINO ViT embeddings. A spatial broadcast MLP reconstructs the feature embeddings from the
slots.

Figure: Figure 12: A DINOSAUR encoder (Seitzer et al., ) learns to represent images via slots.
Refer to caption: x8.png

### B.2 NSI Schema Encoder

The NSI schema encoder architecture is shown in Fig. 13. The lower-level primitive encoder represents
individual object attributes via primitives and their property tags. However, objects in scenes also exist in their relation
to other objects. To this end, a higher-level Schema Transformer represents the primitives jointly.

Figure: Figure 13: A bi-level schema encoder learns a representation of schema primitives. The primitive encoder embeds the object properties of each schema primitive. Then, a Transformer learns embeddings that assimilate the entire schema context.
Refer to caption: x9.png

## Appendix C Hyperparameters

We list the hyperparameters for NSI and other methods used in our experiments, which were all performed on Nvidia A100 GPUs.

### C.1 Ungrounded and HMC Matching Backbone Hyperparameters

The hyperparameters for ungrounded and HMC matching backbones are given in Table 4.

**Table 4: Hyperparameters for the ungrounded and HMC matching method used in our experiments. In the ungrounded case, the backbone and slot attention modules are trained solely on the reconstruction objective and frozen.**
| Module | Hyperparameters | CLEVr-Hans | CLEVrTex | MOVi-C | MS COCO 2017 |
| --- | --- | --- | --- | --- | --- |
| DINO Backbone | Image Size | 224 | 224 | 224 | 224 |
| Patch Size | 8 | 8 | 8 | 8 |  |
| Num. Patches | 784 | 784 | 784 | 784 |  |
| Num. Layers | 8 | 8 | 8 | 8 |  |
| Num. Heads | 8 | 8 | 8 | 8 |  |
| Hidden Dims. | 192 | 192 | 192 | 192 |  |
| Slot Attention | Num. Slots | 10 | 10 | 10 | 93 |
| Iterations | 3 | 3 | 3 | 3 |  |
| Hidden Dims. | 192 | 192 | 192 | 192 |  |
| Broadcast Decoder | Num. MLP Layers | 3 | 3 | 3 | 3 |
| MLP Hidden Dims. | 1024 | 1024 | 1024 | 1024 |  |
| Output Dims. | 785 | 785 | 785 | 785 |  |
| Prediction Head | MLP Hidden Layers | 2 | 2 | 2 | 2 |
| MLP Hidden Dims. | 64 | 64 | 64 | 64 |  |
| Output Size | 18 | 71 | 25 | 95 |  |
| Training Setup | Batch Size | 128 | 128 | 128 | 128 |
| LR Warmup steps | 10000 | 10000 | 10000 | 30000 |  |
| Peak LR | $4\times 10^{-4}$ | $4\times 10^{-4}$ | $4\times 10^{-4}$ | $1\times 10^{-4}$ |  |
| Dropout | 0.1 | 0.1 | 0.1 | 0.1 |  |
| Gradient Clipping | 1.0 | 1.0 | 1.0 | 1.0 |  |
| Inference Configuration | Num. Slots | 10 | 10 | 10 | 30 |
| Training Cost | GPU Usage | 40 GB | 40 GB | 40 GB | 40 GB |
| Days | 1 | 3 | 3 | 5 |  |
| ViT | Num Layers | 2 | - | - | - |
| Hidden Dims | 64 | - | - | - |  |
| Num Heads | 4 | - | - | - |  |

### C.2 NSI Hyperparameters

The hyperparameters for the NSI alignment model are listed in Table 5. The ablated architectures follow the same setup without the ablated module.

**Table 5: Hyperparameters for the NSI alignment model instantiation and training setup.**
| Module | Hyperparameters | CLEVr Hans | CLEVrTex | MOVi-C | MS COCO 2017 |
| --- | --- | --- | --- | --- | --- |
| DINO Backbone | Image Size | 224 | 224 | 224 | 224 |
| Patch Size | 8 | 8 | 8 | 8 |  |
| Num. Patches | 784 | 784 | 784 | 784 |  |
| Num. Layers | 8 | 8 | 8 | 8 |  |
| Num. Heads | 8 | 8 | 8 | 8 |  |
| Hidden Dims. | 192 | 192 | 192 | 192 |  |
| Broadcast Decoder | Num. MLP Layers | 3 | 3 | 3 | 3 |
| MLP Hidden Dims. | 1024 | 1024 | 1024 | 1024 |  |
| Output Dims. | 785 | 785 | 785 | 785 |  |
| Slot Attention | Num. Slots | 10 | 10 | 10 | 15 |
| Iterations | 3 | 3 | 3 | 3 |  |
| Hidden Dims. | 192 | 192 | 192 | 192 |  |
| Schema Encoder | Num. Layers | 8 | 8 | 8 | 8 |
| Num. Heads | 8 | 8 | 8 | 8 |  |
| Hidden Dims. | 192 | 192 | 192 | 192 |  |
| Max. Schema Len. | 10 | 10 | 10 | 93 |  |
| Projection Heads | Embedding Dims. | 64 | 64 | 64 | 64 |
| MLP Hidden Layers | 2 | 2 | 2 | 2 |  |
| MLP Hidden Dims. | 256 | 256 | 256 | 256 |  |
| Training Setup | Batch Size | 128 | 128 | 128 | 128 |
| LR Warmup steps | 10000 | 10000 | 10000 | 30000 |  |
| Peak LR | $4\times 10^{-4}$ | $4\times 10^{-4}$ | $4\times 10^{-4}$ | $1\times 10^{-4}$ |  |
| Dropout | 0.1 | 0.1 | 0.1 | 0.1 |  |
| Gradient Clipping | 1.0 | 1.0 | 1.0 | 1.0 |  |
| $\beta_{1},\beta_{2}$ | 0.5, 0.5 | 0.5, 0.5 | 0.5, 0.5 | 0.5, 0.5 |  |
| Training Cost | GPU Usage | 40GB | 40 GB | 40 GB | 40 GB |
| Days | 1 | 3 | 3 | 5 |  |
| ViT | Num Layers | 2 | - | - | - |
| Hidden Dims | 64 | - | - | - |  |
| Num Heads | 4 | - | - | - |  |

## Appendix D Experiments

### D.1 Object Discovery

#### D.1.1 Object Discovery Results

**Table 6: Object Discovery results. We use the DINO backbone and an MLP decoder across methods. The standard deviation was calculated over five random seeds.**
| Dataset | Metric | Ungrounded | HMC Matching | NSI |
| --- | --- | --- | --- | --- |
| CLEVrTex | FG-ARI | 87.79 $\pm$ 0.12 | 88.37 $\pm$ 0.12 | 89.89 $\pm$ 0.01 |
| mBO | 44.86 $\pm$ 0.04 | 45.23 $\pm$ 0.23 | 46.60 $\pm$ 0.02 |  |
| MOVi-C | FG-ARI | 65.53 $\pm$ 0.15 | 65.61 $\pm$ 0.31 | 66.41 $\pm$ 0.12 |
| mBO | 36.79 $\pm$ 0.03 | 36.79 $\pm$ 0.41 | 38.52 $\pm$ 0.23 |  |
| COCO | FG-ARI | 40.12 $\pm$ 0.29 | 32.18 $\pm$ 0.45 | 44.24 $\pm$ 0.27 |
| mBO^i | 27.20 $\pm$ 0.31 | 18.32 $\pm$ 0.51 | 28.12 $\pm$ 0.25 |  |
| mBO^c | 26.54 $\pm$ 0.25 | 19.61 $\pm$ 0.82 | 32.10 $\pm$ 0.31 |  |

Table 6 contains the complete set of object discovery results with the standard deviations.

#### D.1.2 Slot Sweep over COCO scenes

Fig. 14 shows the NSI object discovery results on COCO scenes as the number of
slots is increased. We observe that segmentation improves until 15 slots and then slowly tapers off.

Figure: Figure 14: Ablation on the number of slots for COCO object discovery using NSI.
Refer to caption: x10.png

#### D.1.3 Property Ablations on COCO scenes

In Fig. 15, we ablate the properties used to form schema primitives. Ostensibly, the slots weigh
bounding box coordinates more than categories, as the performance drop is steeper when the former is ablated as opposed to
the meagre loss when the latter is ablated.

Figure: Figure 15: Ablation on the visXML properties for NSI.
Refer to caption: x11.png

#### D.1.4 Mask Visualizations

Figs. 16, 17, 18 visualize slot masks over scenes from CLEVrTex, MOVi-C, and COCO, respectively.

Figure: Figure 16: Object discovery results on CLEVrTex scenes.
Refer to caption: x12.png

Figure: Figure 17: Object discovery results on MOVi-C scenes.
Refer to caption: x13.png

Figure: Figure 18: Object discovery results on COCO scenes.
Refer to caption: x14.png

### D.2 Grounded Compositional Semantics

#### D.2.1 Top Learned Slots

Figs. 19-21 show the top 50 slots learned by the image encoder over each dataset. Each
slot is weighted by the $l_{2}$ norm of its embeddings $Y^{k}_{x}$. The top slots follow an interesting distribution. On CLEVrTex
(Fig. 19), slots with larger objects and a clean segmentation are assigned a higher magnitude.
Intuitively, these slots are the most discriminative when assigning primitives that contain shape, texture, and size, and
benefit from large, cleanly segmented objects. On the other hand, the emergent distribution in MOVi-C
(Fig. 20) and COCO (Fig. 21) weights edge artifacts more. We posit that since the object
annotations of these datasets contain object positions in the image, edge objects tend to be more discriminative.

Figure: Figure 19: Top 50 slots (left to right, top to bottom) ranked by magnitude from test split of the CLEVrTex dataset.
Refer to caption: x15.png

Figure: Figure 20: Top 50 slots (left to right, top to bottom) ranked by magnitude from test split of the MOVi-C dataset.
Refer to caption: x16.png

Figure: Figure 21: Top 50 slots (left to right, top to bottom) ranked by magnitude from test split of the COCO dataset.
Refer to caption: x17.png

#### D.2.2 Top Learned Primitives

Figs. 22-24 show the t-SNE visualization (van der Maaten & Hinton, ) of
representations learned by the schema encoder on the visXML primitives. In each plot, primitives in the test split
are encoded context-free. The learned embeddings exhibit clustering effects on property categories.

Figure: (a) t-SNE scatter plot labeled with object shapes.
Refer to caption: x18.png

Figure: Figure 23: t-SNE scatter plots of the top 10000 MOVi-C primitives weighted by the $l_{2}$ norm of their embeddings. The scatter points are labeled with the object categories. While there are multiple local clusters, a larger clustering effect based on object categories are not apparent.
Refer to caption: x21.png

Figure: Figure 24: t-SNE scatter plots of the top 10000 COCO primitives weighted by the $l_{2}$ norm of their embeddings. The scatter points are labeled with the object categories. A category-based clustering pattern is emergent.
Refer to caption: x22.png

#### D.2.3 Searching over Slots

We found that the NSI metric, which learns to match entire schemas to entire images, can also be used zero-shot to
reliably retrieve individual slots from primitives. We demonstrate two instances of slot search in
Figs. 25 and 26. A query in the form of a single primitive is embedded using the
schema encoder. The query embedding ranks the slot of a database formed from the test split. In
Figs. 25 and 26, we show the property search and position search results,
respectively.

Figure: Figure 25: Searching over slots with property queries. The rows represent a shape and the columns represent an object material. The position is fixed at $[0,0,0]$ and the size is set to ‘Large.’ The alignment model reliably retrieves slots with desired object properties if they exist in the database.
Refer to caption: x23.png

Figure: (a) Position search on cubes.
Refer to caption: x24.png

#### D.2.4 Slot Sweep for COCO Retrieval Task

Fig. 27 shows the effect of the number of slots on the recall rates. Recall is highest at
15-20 slots for COCO scenes and the model overfits as the number of slots increases further.

Figure: Figure 27: Ablation on the number of slots for scene-property retrieval task. The standard deviation over five seeds was $<$ 0.3 across datasets.
Refer to caption: x27.png

#### D.2.5 Additional Qualitative Results

Fig. 28 shows additional results on the associations inferred by NSI on MOVi-C and MS COCO 2017.

Figure: Figure 28: Correspondences inferred by NSI on MOVi-C and COCO scenes.
Refer to caption: x28.png

### D.3 Grounded Slots as Visual Tokens

#### D.3.1 Confusion Matrices

Figs. 29 and 30 show the classification confusion matrices for the CLEVr-Hans 3
and CLEVr-Hans 7 tasks. In the few-shot setting, the model often predicts scenes into under-specified and less discriminative
classes, like “cyan object in front of two red objects.” Similarly, it also tends to mispredict into classes containing
large objects like “large cube and large cylinder” that contain reasoning over larger objects and are easier to tokenize.

Figure: Figure 29: Confusion matrices for NSI prediction on the CLEVr-Hans 3 task.
Refer to caption: x29.png

Figure: Figure 30: Confusion matrices for NSI prediction on the CLEVr-Hans 7 task.
Refer to caption: x30.png

#### D.3.2 Visual Rationales

Fig. 31 simultaneously visualizes the rationales across data settings and attention layers. On an average,
we observe that rationales tend to get stronger and more accurate as the number of training examples increases and the
ViT depth increases.

Figs. 32 and 33 demonstrate rationales across the grounding continuum for Hans 3 and
Hans 7 classification tasks, respectively.

Fig. 34 demonstrates attention maps where the model prediction is correct, but the rationale is not
entirely dispositive.

Figure: Figure 31: NSI visual rationales across different data regimes and attention depth.
Refer to caption: x31.png

Figure: Figure 32: Visual rationales on Hans 3 across different methods.
Refer to caption: x32.png

Figure: Figure 33: Visual rationales on Hans 7 across different methods.
Refer to caption: x33.png

Figure: Figure 34: Some failure cases where NSI makes the correct prediction, albeit using incorrect or ambiguous support slots.
Refer to caption: x34.png

## Appendix E Object Detection with NSI

In this section, we demonstrate an architecture that uses inferred correspondences by NSI to perform object detection. We
run preliminary experiments and explore the use of slots in real-world visual reasoning systems.

### E.1 NSI Schema Generator Model

We formulate the NSI schema generator (see Fig. 35) to predict, from each slot, its corresponding
visXML primitives. To this end, we modify an encoder Transformer by interleaving cross-attention blocks with the
self-attention layers to assimilate the context from the encoded slots. The input to the model is simply learned positional
embeddings $\mathcal{P}^{1:N}$ that are decoded by $L$ attention-ensemble stacks to output primitive representations. We
call this architecture $SET_{\theta}(.)$. The Slot Encoder Transformer (SET) is a stack of $L$ blocks, each computing
(a) self-attention over inputs followed by (b) cross-attention of inputs over the context slot. Let $Z_{prim,l-1}^{dec,1:T}$
be the sequence representation at layer $l-1$ and $S$ denote the slot. Then

$$ $\displaystyle Z_{prim,L}^{dec,1:N}=SET_{\theta}(\mathcal{P}^{1:N},S)$ (13) $$

The cross-attention implementation of block $l$ is shown in Algorithm 2.

Figure: Algorithm 2 Cross-Attention for Block $l$ of SET

Using an encoder-transformer-styled predictor has two advantages: (1) primitives can be generated in parallel and (2) each
primitive representation is aware of the overall prediction context. $MLP$ property heads predict the object properties from
the $Z_{prim,L}^{dec,1:N}$ representations. At each training iteration, the predicted properties $\hat{p}^{1:N}$ are optimally
matched to their ground truth labels $p^{1:N}$ via an ordering $\sigma(1:N)$ obtained from Hungarian matching (HM)
(Kuhn, ) on the property prediction loss $\mathcal{L}_{properties}$. Note that $\mathcal{L}_{properties}$ is a
per-primitive loss obtained from the sum of individual property prediction losses. We use the cross-entropy loss for discrete
properties, mean-squared error for continuous properties, and augment bounding-box regression with the Intersection over Union
(IoU) loss. The training objective $\mathcal{L}_{gen}(.)$ is formulated as follows:

$$ $\displaystyle\sigma(1:N)=HM\left(\mathcal{L}_{properties}\left(p^{1:N},\hat{p} ^{1:N}\right)\right)$ (14) $\displaystyle\mathcal{L}_{gen}=\sum_{i=1}^{N}\mathcal{L}_{properties}(p_{i}, \hat{p}_{\sigma(i)})$ (15) $$

At training time, we pad the aligned ground-truth instances with no-object labels $\Phi$ to account for representations without
object predictions. In practice, bipartite matching for a single-slot instance is more computationally feasible than the overall
image instance because per-slot object instances are significantly fewer.

Figure: Figure 35: NSI schema generator model: A Slot Encoder Transformer attends to individual slots via cross attention and, in parallel, decodes tokens into schema primitives. At training time, a Hungarian set matching procedure assigns predictions to primitives associated with the slot. The prediction error is aggregated over assignments to compute $\mathcal{L}_{gen}$.
Refer to caption: x35.png

### E.2 NSI Schema Generator Hyperparameters

The hyperparameters for the NSI schema generator are listed in Table 7.

**Table 7: Hyperparameters for the NSI schema generator model instantiation and training setup.**
| Module | Hyperparameters | MOVi-C | COCO-10 | COCO-30 |
| --- | --- | --- | --- | --- |
| SET | Num. Layers | 2 | 2 | 2 |
| Num. Heads | 2 | 2 | 2 |  |
| Hidden Dims. | 192 | 192 | 192 |  |
| Predictions per Slot ($T$) | 5 | 8 | 8 |  |
| Inference | top-$M$ predictions | 10 | 10 | 30 |
| Non-Max Suppression/Threshold | Yes (0.75) | Yes (0.75) | Yes (0.75) |  |
| Training Setup | Batch Size | 64 | 64 | 64 |
| LR Warmup steps | 10000 | 30000 | 30000 |  |
| Peak LR | $4\times 10^{-4}$ | $1\times 10^{-4}$ | $1\times 10^{-4}$ |  |
| Dropout | 0.1 | 0.1 | 0.1 |  |
| Training Cost | GPU Usage | 40 GB | 40 GB | 40 GB |
| Days | 2 | 4 | 4 |  |

### E.3 Experiments

The correspondences from the train split are used to learn the NSI schema
generator, as outlined in Appendix E.1. The schema generator decodes $T$ primitives from each slot,
including the confidence level of each object prediction. The overall schema for a single image is obtained as the top $M$
confident primitives out of predictions from all $K$ slots of that image. The NSI schema generator enables solving of
downstream tasks with slots by making use of predicted properties from primitive tags. We demonstrate the usefulness of NSI
for object detection.

Object Detection Results: In the context of real-world object detection on images, the visXML schema
properties can be used to identify and locate objects in diverse scenes from the MOVi-C and COCO datasets. To this end, we
extract the (<cat>) category and (<bbox>) bounding box fields from the generated primitives that depict the
object category and location, respectively. For COCO, we test on two variants of the dataset: (1) COCO-10, a simple
subset of the test split containing ten objects at a maximum, and (2) COCO-30 that contains as many as 30 objects in
a scene. We report the $AP_{IoU}$ metric across all three benchmarks. It denotes the area under the precision-recall curve for
a certain IoU threshold (in $\%$). We also run comparisons against the methods outlined in the retrieval experiments.
Fig. 36 shows the experimental results and Figs. 37, 38 visualize
object detection across various predictors. Our comparison baselines include:

- (a)
Vanilla Slot Attention (Locatello et al., ): The model is trained from scratch to resolve
slot-object assignments through HMC.
- (b)
DINOSAUR (Locatello et al., ): It uses the DINO ViT backbone to learn slots for unsupervised object
discovery by reconstructing perceptual features. The architecture is frozen while we train shallow predictors on top to
detect objects.
- (c)
DINOSAUR-FT: We use the DINOSAUR model but fine-tune the architecture end-to-end on the prediction task.

We make the following observations:

- (a)
NSI outperforms prior set-matching slot predictors, especially by significant margins (20-30%) at lower
IoU thresholds. Grounding object concepts in slots a priori improves the predictive power of slots. In comparison,
matching the set of slots against the entire set of object annotations of the image yields poor generalization. In addition,
large-scale pre-training and end-to-end fine-tuning are crucial ingredients, as evidenced by the subpar performance of the
Vanilla and DINOSAUR methods on COCO.
- (b)
The performance disparity between NSI and DINOSAUR-FT widens as the complexity of scenes increases from
COCO-10 to COCO-30. The inability of DINOSAUR to predict more than one object per slot necessitates modeling and learning to
match up to 30 different slots, which generalizes poorly on novel scenes.
- (c)
HMC slots deteriorate for COCO-30 where the backbone is tasked with matching with 30 different objects at a
time.

Figure: Figure 36: Object detection performance of various prediction methods on the MOVi-C, COCO-10, and COCO-30 benchmarks. We report $AP@IoU$ (higher is better) for different IoU thresholds. Standard deviation is reported over five random seeds.
Refer to caption: x36.png

Figure: Figure 37: Object detection results on COCO scenes. NSI schema generator can flexibly detect multiple objects from the same slot, as evidenced by detections on COCO images. For example, a single slot predicts multiple ‘donuts,’ ‘chair and dining table,’ and ‘person and tennis racket’.
Refer to caption: x37.png

Figure: Figure 38: Object detection results on MOVi-C scenes.
Refer to caption: x38.png

Figure: Figure 39: Effect of slot cardinality on object detection performance.
Refer to caption: x39.png

Slot Sweep on COCO Scenes: Next, we investigate the effect of slot count on object detection. The number of
slots is adjusted across a [5-30] range for the NSI schema generator and DINOSAUR-FT on the COCO-30 task.
Fig. 39 presents the results of this experiment. For the DINOSAUR predictor, the number of slots imposes a
strict upper limit on the number of object detections. As a result, the performance sharply deteriorates as the slot count
is lowered. On the other hand, the NSI schema generator can detect multiple objects over a single slot and the performance
drop is more graceful as the number of slots decreases.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: Request the authors to address the weaknesses to demonstrate the quantitative benefits of the current approach compared to all existing methods (even if it does not have superior performance).
soundness: 2
strengths: (S1) The problem tackled is interesting as it builds on top of existing object-centric slot learning to ground object concepts in them.

(S2) The proposed approach to use a visXML-based schema and measure scene-schema similarity is intuitive and simple to understand.

(S3) The manuscript is mostly well-written with sufficient motivation, relevant discussion of the literature, and clear technical details to understand the proposed approach, spread across the main and the appendix. I thoroughly enjoyed reading the manuscript.
summary: The current manuscript tackles the problem of grounding object-centric semantics in the slot representations that emerge via a visual reconstruction task. Towards this, it proposes a schema, called visXML, to represent the scene semantics in a structured form, and aligns it with the emergent slots via a contrastive loss within a given batch. The proposed approach, called Neural Slot Interpreter (NSI), results in a more grounded representation and outperforms some of the existing methods on bi-modal objective-property and scene retrieval tasks, in addition to outperforming patch-based tokens on few-shot classification tasks.
weaknesses: (W1) Insufficient comparisons to prior work. The experiment tables do not report any numbers from the literature (e.g., [A, B, C, D]). 
* In particular, [A] reports performance on slot discovery for MS COCO dataset as 35.27 (mBO_i) and 46.28 (mBO_c), while current approach reports 28.12 and 32.10 respectively. 
* Further, if [C] is the Ungrounded baseline, the numbers do not match with those shown in the literature, mBO_c of 31.1 [C] vs 26.5 (reported in Appendix E.1.1). Request the authors to include necessary comparisons and fix these inconsistencies.

(W2) Limited experimentation on real world datasets. Though the results on synthetic datasets like CLEVr Hans, CLEVrTex, and MOVi-C motivate the problem, the real-world applications are more guided by experimentation on real world datasets. This also helps understand the efficiency of the approach beyond the sim2real domain difference. While the paper includes results on MS COCO, results on PASCAL VOC (often reported in prior works in the area) are missing.

Without these results, the quantitative benefits of the proposed approach are unclear.

(W3) A minor concern is the lack of relevant pointers in the main, when referring to modules from prior work. For instance, it is unclear how slots are obtained in Sec 3.2.1 without reference to either the appendix A.1 or prior work. Request the authors to go through the manuscript carefully to provide these details.

**References:**
* [A] Guided Latent Slot Diffusion for Object-Centric Learning. https://arxiv.org/pdf/2407.17929
* [B] Object-Centric Slot Diffusion. https://arxiv.org/pdf/2303.10834
* [C] Bridging The Gap To Real-World Object Centric Learning. https://arxiv.org/pdf/2209.14860 
* [D] SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models. https://arxiv.org/pdf/2305.11281

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: I don't have particular questions about the paper.
soundness: 2
strengths: The paper introduced a novel contrastive learning technique to align slot representations with visXML representations, which are essentially object-centric symbolic representations associated with the scene.
The evaluation was performed on a variety of tasks: (1) object discovery, (2) scene-property retrieval, (3) few-shot scene classification, and (4) object detection. The results demonstrate the efficacy of NSI.
summary: The paper presents the Neural Slot Interpreter (NSI), a model that aims to bridge the gap between visual perception and object-based reasoning by grounding object semantics in emergent slot representations. NSI organizes visual scenes using an XML-like schema (visXML) that assigns semantics to specific object-centered slots, allowing structured grounding through a contrastive learning objective.
weaknesses: The main weakness of the paper, to me, is the unclarity of the problem setting. VisXML, based on my understanding, is essentially some kind of symbolic scene representation similar to works such as http://nsd.csail.mit.edu/papers/nsd_cvpr.pdf I think, in order to obtain such kind of representations, one needs to train models with supervision on object masks/bounding boxes, and object labels (or maybe off-the-shelf models like large vision-language models or object segmentation models can already do the job). If that's the case, why do we still need to learn object-centric representations? Why don't we just directly supervise the learning of object segmentation etc. using these visXML labels...? I could imagine ways that combined supervised and unsupervised learning could help, but I don't think the paper shows sufficient evidence about their advantages.

Concretely, the paper should compare with a baseline that directly uses VisXML to do all these tasks.

Also, VisXML should not be listed as a contribution. See the NSD paper cited above.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - Figure 7 shows only 100 grounding examples can have similar benefits on the downstream reasoning task compared with annotating the full dataset.  Does object discovery performance have the same trend? 
- Can the model handle cases where only a subset of the objects in every image are annotated?
soundness: 2
strengths: 1. The paper is easy to follow.
2. The paper includes the necessary details for reproducing the results.
summary: The paper addresses the problem of grounding concepts to objects within images by training on paired images and object descriptions. It proposes an architecture built upon unsupervised object discovery models. During training, object features are first extracted by an object discovery module, and then these features are aligned with object property features through contrastive learning. Experiments demonstrate that the proposed method outperforms baseline models in object segmentation performance. Additionally, the learned representations benefit downstream applications, such as concept retrieval and concept reasoning.
weaknesses: - Using semantic grounding to enhance object discovery is not novel. For instance, [1] leverages question-answering and captions paired with images to improve the object discovery capabilities of Slot Attention. Author should also consider discussing other related works in concept grounding in images, such as [2] and [3].
- The experimental setting is somewhat unrealistic, as the paper assumes access to all objects and their attributes for every training image. In particular, the use of object position information (such as bounding boxes and 3D locations) during training significantly undermines the experimental results, as it contradicts the core motivation of object discovery. According to Figure 15, the proposed model performed worse than baseline models when trained without object position information.
- Referring to the semantic representation in the paper as VisXML is confusing, as there are no recursive structures involved. Only objects and their attributes are considered, which is not a substantial contribution.

[1] Language-Mediated, Object-Centric Representation Learning. Wang, et al. ACL Findings 2021.

[2] GroupViT: Semantic Segmentation Emerges from Text Supervision. Xu, et al. CVPR 2022.

[3] MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding. Kamath et al. ICCV 2021.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 3
questions: Points 1 and 2 in Weaknesses.

Given the recent improvements in token-based visual encoders, do we still need Slot Attention? Couldn’t we just use the tokens from ViT-CLIP or DINOv2, compress them into a smaller set of tokens using a cross-attention layer (like in Perceiver and PerceiverIO) and treat these tokens as slots? It is possible that your schema based loss makes the spatial biases that Slot Attention encodes redundant. So, your method could potentially be used without Slot Attention, which would overcome the scaling limitations and instabilities of SA.
soundness: 4
strengths: 1. The authors propose to use visXML descriptions of objects and their properties to better ground object slots in Slot Attention. This is a novel combination of the spatial biases in Slot Attention and the semantic biases in object-centric annotations.
2. The authors propose a nearest neighbor slot to semantic embedding assignment mechanism that is shown to lead to better representations than hungarian matching. The key insight is to assign object properties to slots without enforcing an exclusive 1-to-1 mapping.
3. The paper includes a number of evaluations of the quality of the learned slot representation, including unsupervised segmentation, scene retrieval and few-shot classification. Each experiment has reasonable baselines.
summary: This work focuses on grounding slots – tokens that have a learned correspondence to objects in images. The authors build upon prior work, specifically Slot Attention and DINOSAUR. Their contribution is in using datasets of object properties to further ground slots computed with Slot Attention. They do so by proposing a novel contrastive loss. This loss pairs object representations based on object properties with object representations based on visual slots and then applies a standard InfoNCE objective over correct and incorrect pairings. The correct pairings are computed by greedy assignment based on the feature vector dot product. The model is evaluated in unsupervised object segmentation, object and scene retrieval and few-shot classification across synthetic and real-world datasets.
weaknesses: 1. It is not clear to me why the object properties get concatenated and processed with an MLP in Equation 3. Do we assume that each object has the same set of properties? Are there any missing properties? Some kind of permutation-invariant model (e.g., a Transformer or a graph neural network) would be more flexible with respect to these questions.
2. As far as I understand, the Transformer used in Equation 4 processes an entire scene. Doesn’t this result in a loss of distinction between individual objects to a certain extent, as we are presumably computing self-attention across all objects in the scene? I suppose this decision is already ablated in NSI-Schema Agnostic, which performs worse, but I am still curious about the rationale here. Would a per-object Transformer instead of a scene-level Transformer work?
3. The matching mechanism in Equation 6 might fail due to bad initialization, especially since Slot Attention is also sensitive to initialization. I would like to see a discussion of this.
4. The $S_{xx}, S_{xy}, S_{yx}$ notation in Equation 7 can be confusing.

---

confidence: 2
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. What's the motivation for using two losses in Eq. 10? Those equations are introduced but not explained well. What's the connection of such a loss to contrastive losses used in prior works?

2. The bi-level architecture should be explained in the main text. What motivates learning the slot attention rather than learning a simple tokenizer? It seems that the authors wish to present a new tokenizer but it is neither simple nor scalable (to some other architecture).

3. What's the novelty of this work against general compositional learning works? The slot representations are not very well-defined and it's hard to claim novelty there.
soundness: 3
strengths: 1. This paper provides a contrastive learning objective that learns from image-schema pairs. This specific contrastive learning algorithm looks novel.

2. The proposed structured learning seems to benefit and outperform pure slot attention learning.

3. The experiment combining with vision transformers demonstrates slots as a better representation than patches.
summary: This paper proposes neural slot interpreters, which provide an XML-like schema that uses simple syntax rules to organize the object semantics of a scene into object-centric schema primitives. It can learn to ground primitives into slots through an objective that reasons over the intermodal alignment. Experiments demonstrate that NSI tokenizers can achieve fewer tokens than other patch-based tokens.
weaknesses: 1. The XML representation looks unnecessary, given a strong LLM model. Nowadays, multi-modal LLMs develop really fast, so as long as you have any testified representation, those language models can help you structure it into XML. In other words, the advantage of the proposed representation over the naive text representation (e.g., a caption model or a CLIP model) is very marginal.

2. The proposed experiments are limited in scale. CLEVR is too small to verify such an idea generally. The authors should experiment and report numbers on ImageNet. Also, ViT is not the SOTA, and it's interesting to see whether slot-based representations are better than patch-based ones.

3. Those slots are not accurately defined and it seems that those are just vector embeddings. Therefore, the reviewer doubts the motivation to research such slot-like representations.

4. The writing is not very clear. In fact, the methods are not explained very well (see questions below).

Meta-review: This paper proposes a method (NSI) for object-centric learning from visual data, while grounding this using an XML schema (which the authors called visXML). The method uses DINOSAUR (slot-attention based backbone) additionally trained on scene-schema pairs using contrastive loss. The authors perform experiments on CLEVr Hans, CLEVrTex, MOVi-C, and COCO, on tasks including object discovery, few-shot classification, and scene property retrieval. NSI is compared to ungrounded slots, as well as a baseline using Hungarian Matching Criterion (to use the visXML as supervision for slot predictions) to show improvements in performance. While the results in the paper is promising, and represents an interesting application of contrastive loss for grounding properties in object-centric learning, concerns are raised by reviewers on the choice of baselines and experimental setup. The main concern raised by reviewers is that the proposed approach requires annotations and supervision to ground the slots, and reviewers did not consider HMC to be a sufficient baseline for comparison (e.g. one could directly train a model to predict these properties; weakly-supervised grounding was brought up by one of the reviewers). The AC agrees that in its current form, with the level of annotations required for NSI, comparison and including a discussion of this work in relation to weakly supervised approaches would be valuable + experiments in the main paper on varying levels of annotations, would better demonstrate the significance of this approach. I encourage the authors to take into account helpful comments raised by reviewers with remaining concerns and I think this work could definitely make a strong paper with these additional discussions and experiments.

Decision: Reject