Neural Slot Interpreters: Grounding Object Semantics in Emergent Slot Representations
Authors: Bhishma Dedhia
Niraj Jha
Venue: Accepted by TMLR ()

Abstract: Several accounts of human cognition posit that our intelligence is rooted in our ability to form abstract composable concepts, ground them in our environment, and reason over these grounded entities. This trifecta of human thought has remained elusive in modern intelligent machines. In this work, we investigate whether slot representations extracted from visual scenes serve as appropriate compositional abstractions for grounding and reasoning. We present the Neural Slot Interpreter (NSI), which learns to ground object semantics in slots. At the core of NSI is a nested schema that uses simple syntax rules to organize the object semantics of a scene into object-centric schema primitives. Then, the NSI metric learns to ground primitives into slots through a structured contrastive learning objective that reasons over the intermodal alignment. Experiments with a bi-modal object-property and scene retrieval task demonstrate the grounding efficacy and interpretability of correspondences learned by NSI.  From a scene representation standpoint, we find that emergent NSI slots that move beyond the image grid by binding to spatial objects facilitate improved visual grounding compared to conventional bounding-box-based approaches. From a data efficiency standpoint, we empirically validate that NSI learns more generalizable representations from a fixed amount of annotation data than the traditional approach. We also show that the grounded slots surpass unsupervised slots in real-world object discovery and scale with scene complexity. Finally, we investigate the downstream efficacy of the grounded slots. Vision Transformers trained on grounding-aware NSI tokenizers using as few as ten tokens outperform patch-based tokens on challenging few-shot classification tasks.

Published in Transactions on Machine Learning Research (05/2025)
Neural Slot Interpreters: Grounding Emergent Slot Representations
Object Semantics in
Bhishma Dedhia
Niraj K Jha
Department of Electrical and Computer Engineering
Princeton University { bdedhia, jhaj @erincetrnnccccomm edu
Reviewed on OpenReview: https: // openreview. net/ fort/? id- LyaRBPmam
Abstract
Several accounts of human cognition posit that our intelligence is rooted in our ability to form abstract composable concepts, ground them in our environment, and reason over these grounded entities. This trifecta of human thought has remained elusive in modern intelligent machines. In this work, we investigate whether slot representations extracted from visual scenes serve as appropriate compositional abstractions for grounding and reasoning, We present the Neural Slot. Interpreter (NST), which learns to ground object semantics in slots. At the core of NSI is a nested schema that uses simple syntax rules to organize the object semantics a scene into object-centric schema primitives. Then, the NSI metric learns to ground primitives into slots through structured contrastive learning objective that reasons over the intermodal alignment. Experiments with bi-modal object-property and scene retrieval task demonstrate the grounding efficacy and interpretability of correspondences learned by NST. From scene representation standpoint, we find that emergent NST slots that move beyond the image grid by binding to spatial objects facilitate improved visual grounding compared to conventional bounding-box-base based approaches. From a data efficiency slandpoint, we empirically validate that NST learns more generalizalle representations from a fixed amount of annotation data than the traditional approach. We also show that the grounded slots surpass unsupervised slots in real-world object discovery and scale with scene complexity. Finally, investigate the downstream efficacy of the grounded slots. Vision Transformers trained on grounding-aware NSI tokenizers using few as ten tokens outperform patch-based tokens on challenging few-shot classification tasks.
1 Introduction
Humans possess repertoire of strong structural biases, kind of abstract knowledge that enables us to per- ceive and rapidly adapt to our environments (Griffiths et al., 2010). Compositionalily is one such structural prior that helps us systematically reason about complex stimuli as whole by recursively reasoning about its parts (Zuberbühler, 2019; Lake & Baroni, 2023). We decompose broad motor skills into finer dexterous finger movements, sentences into words and phrases, and speech into phonemes. In the visual world, the concept of "objectness" serves as a natural compositional prior, enabling us to decompose novel scenes into familiar objects and reason about their properties (Lake et al., 2016). We also have the uncanny ability to connect real-world entities and concepts to these abstract object-like symbols in heads, canonically referred to as the grounding problem (Harnad, 1990; Greff et al., 2020). For instance, human infants, while looking at a zebra for the first time, might excitedly conclude that it is, in fact, "a striped horse." " If grounded object-like representations are fundamental to luman-like compositional generalization, how do we instill these inductive biases into neural network representations?
Unsupervised object-centric autoencoder models (Burgess al., 2019; Greff al., 2019; 2020: Locatello al., 2020; Engelcke et al., 2019; 2021; Singh et al., 2021; Chang et al., 2023b; Seitzer et al., 2023; Kori

---

Published in Transactions on Machine Learning Research (05/2025)
et al., 2024a) have become increasingly adept at learning object-centric representations called slots from raw visual stimuli. Further work has demonstrated that learned slots can be flexibly composed together for tasks like scene composition, causal induction, learning intuitive physies, dynamics simulation, and control (Dedhia et al., 2023; Jiang ct al., 2023; Wu et al., 2023a;b; Chang et al., 2023a; Jabri et al., 2023). While object slots hold promise as a compositional building block for machines that mimic humnan abstraction and generalization, a key challenge emerges. Unlike humans, these learned slots lack grounding in real-world concepts. For example, a slot representation of an object like "apple" could refer to the fruit, the company, generic round artifact. Without grounding, a slot-based system cannot disambiguate these meanings effectively (Haugeland, 1985) and is fundamentally limited in its embodied reasoning abilities. Then the central question that guides our work is:
"Can slots serve as effective representations to ground object semantics in visual scenes?"
Grounding, in the context of slot representations, refers to the ability to associate labels (for e.g., identi- fying objects and their properties) with slot representations learned from a scene's visual features. Prior works (Locatello et al., 2020; Seitzer et al., 2023; Kori et al., 2024a) have tackled learning to ground slots by predicting object properties (texture, material, category, etc.) from the representations. The grounding objective is, therefore, implicit within the prediction of object semantics. However, ground truth correspon- dences between object concepts and slots generally unknown, restricting prediction to a set-matching template. Under a set-matching framework, single slot predicts object properties of single object, thereby constraining the grounding information assimilated per slot. We circunvent the limitations of prediction as a surrogate for grounding by making the grounding objective explicit in the form of a co-training paradigm that we call the Neural Slot Interpreter (NSI).
The core insight behind NSI is simple: instead of predicl- ing single object-concept from slot, assign multiple concepts to slots over shared latent space. Our primary contribution is similarity metric that explicitly reasons about the intermodal assignments. Notably, the proposed metric for NSI supplants the one-object-per-slot assump- tion and facilitates flexible assignment. Contrastive learn- ing over the similarity objective yields grounded slots that outperform their ungrounded and set-matched counter- parts over broad swath of tasks (see Fig. 1). We propose an object-centric annotation schema in Section 4.1 for dense aligument to organize scene annotations for ground- ing into slots. We describe the design of a hicrarchical transformer- architecture in Section 4.2.2 to ract neural representations from the schema. To enable slots to ground wide array of concepts flexibly without rely- ing on matching templates, we formulate bi-level scoring metric over a learned latent space in Section 4.2.
Ungrounded slots
Set Match Slots
NSI Slots
Hans 3
1000-shot
Hans 7
COCO propertyreer.
1000-hhot
MOVIC
KG-ARI
COCO
scene gtrr
10
OVV-C
mBo
MOVI-C property yetr.
30
cOCO FGARI
MOViC
18
scene refr.
76
-OCC
COCO m80
MB0
Does NSI necessitale training slot-centric archileclures from scratch? Our experiments demonstrate that slots learned from pre-trained object-centric backbones (Seitzer et al., 2023) are easily adaptable to the NSI objec- tive. Are notions of objects effectively grounded in emer- gent slot representations? We explicitly evaluate grounding efficacy through bi-modal retrieval tasks (Sec- tion 5.2.1), where models must retricve corresponding object properties given scene, and vice versa. Our experiments demonstrate that NSI outperforms approaches based on ungrounded slots, set-matching, and non-compositional embeddings. How does NSI compare against traditional weakly supervised visual ground- ing? In Scction 5.2.2 we show that conventional bounding box abstractions are incffective at grounding object semantics compared to NSI. Can NSI be effective across different annotation size regimes? We train NST on different annotation levels and discuss it in Section 5.2.2. We find that NSI enables grounded slots to
NSI abstracts grounded slots from scenes and enhances object discovery, ground- ing elficacy, and downstream reasoning abilities of slot representations.
Figure 1:
2

---

Published in Transactions on Machine Learning Research (05/2025)
be data-cfficient, especially real-world data. Do slots that emerge from the NSI objective preserve object discovery abilities? NST slots preserve and often improve object discovery, as discussed in Section 5.3, where we demonstrate its competitiveness on object discovery benchmarks. Can NSI-grounded slots be effective substrates for dounstream tasks? In Section 5.4, we discuss training of a vision transformer (ViT) (Dosovit- skiy et al., 2021) for a scene classífication lask where significantly fewer grounded slot tokens show improved performance and adaptability over traditional patch-based tokens. Can dense associations learned by NST inform real-world reasoning systems? Our experiments described in Appendix E.1 show the usefulness of learned correspondences in identifying and locating objects in diverse scenes.
Concretely, our contributions are as follows:
1. We present NSI (Section 4) co-training grounding paradigm for object-centric learners. We use a nested object-centric annotation schema for dense slot-label alignment (Section 4.1). We formulate a similarity metric that measures scene-schema similarity by recursively reasoning over the similarity of compositional attributes of the respective modalities (Section 4.2.3). NSI utilizes the metric ground slots via rastivvt learning objective (Section 4.2.4).
2. Our model demonstrates the elfectiveness of NSI-trained slot. representations at grounding object proper- ties via a bimodal retrieval task (Section 5.2.1). NSI significantly improves retrieval performance compared to Hungarian Matching Critcrion (HMC) matching slots, ungrounded slots, and non-compositional cm- bedding baselines. Moreover, dense and interpretable correspondences between slot masks and object properties emerge from the NST similarity metric.
3. We also compare NST to conventional methods in weakly supervised grounding where practitioners tradi- tionally utilize bounding box regions to encode visual object features (Section 5.2.2). Overlapping objects within coarser bounding box regions impede accurate object property assignment, a problem mitigated by learned slot masks via NST. We also probe the data efficiency of both approaches and find that for a given set of annotations, NSI is more adept at aligning object semantics to scenes.
4. Our experiments also demonstrate that slots grounded via NSI improve slot performance over a wide array of tasks that encompass (1) object discovery (Section 5.3), (2) few-shot scene classification (Section 5.4), and (3) object detection (Appendix E.1). Overall, we find that grounded slot representations are key to object-centric perception, property grounding, and downstream adaptability for object-centric reasoning.
2 Related Work
Object-Centric Learning. Rescarchers have formulated inductive biases for lcarning composable visual representations called 'slots' from raw visual stimuli (Burgess et al., 2019; Greff et al., 2019; Locatello et al., 2020; Greff et al., 2020; Engeleke et al., 2019; 2021; Singh et al., 2021; Seitzer et al., 2023) and auxiliary temporal information (Kipf al., 2021; Elsayed et al., 2022; Singh et al., 2022). While this line of work demonstrates unsupervised object discovery, the adoption of slot representations for grounding scenes re- mains largely underexplored. Prior works have been limited to using the HMC to align slots to ground-truth property labels for property prediction (Locatello et al., 2020) or fine-tuning shallow property predictors on pre-trained backbones (Seitzer et al., 2023). A recent work (Kori et al., 2024a) improves the predictive power of slots by learning quantized factorised priors to foster invariance of slots with object properties they repre- sent. However, these methods, often relying on HMC for training or evaluation, fundamentally operate under a one-object-per-slot prediction constraint. This limits the richness of grounding information assimilated per slot and poses challenges in learning highly specialized representations, particularly in complex scenes. An orthogonal body of work aims to learn identifiable and interpretable slots without supervision. For instance, Kori et al. (2024b) focus on providing theorctical identifiability guarantees for unsupervised object-centric learning through probabilistic slot atlention. Stanić et al. (2023) investigate synchrony-based models using complex-valued slot representations and contrastive learning for unsupervised object discovery. Similarly, Baldassarre & Azizpour (2022) incorporate contrastive losses to facilitate symmetry-breaking between slots. Furthermore, efforts like Zhang et al. (2023) focus on improving the core slot attention mechanism itself, using optimal transport to enhance tie-breaking in dynamic scenes. In contrast to these directions that focus
3

---

Published in Transactions on Machine Learning Research (05/2025)
unsupervised identifiability, our NSI method directly tackles the challenge of explicit semantic grounding through annotation-driven contrastive alignment. Moreover, intepretable slot features can potentially be integrated within the NSI framework to improve alignment.
Visual Grounding. Several bodies of works have grounded text into scenes by pre-training large-scale transformers using Internet-scale natural language supervision (Chen al., 2020; Desai & Johnson, 2021; et al., 2019; 2020; Radford et al., 2021). Subsequent work has improved modeling of the multimodal streams by explicitly imbibing compositional synergy between visual concepts and text phrases. A prominent line of work trains predictive models on labeled text- bost-bounding box correspondences (Chen al., 2020; Kamath et al., 2021; Gan et al., 2020; Lu et al., 2019) but is expensive due to its fully supervised nature. To reduce the reliance on pre-specified correspondences for training, like NSI, several works propose weakly supervised visual grounding (Karpathy & Fei-Fei, 2015; Gupta et al., 2020; Wang et al., 2021a; Liu et al., 2024), but these approaches are still reliant on object detectors for bounding box proposals. Morcover, bounding boxes underspecify overlapping or occluded objects, thus making them a poor choice for grounding object semantics beyond categories. A recent work by Xu et. al. (2022) goes beyond image patches to hierarchically discover semantic masks but relies careful stacking of ViI's (Dosovitskiy et al., 2021) to achieve necessary object granularity. As opposed to completely neural methods, neuro-symbolic approaches use symbolic domain- specific languages to reason about image concepts (Johnson et al., 2017; Yi et al., 2018; Wang et al., 2021b). Such methods are extremely effective at question-answering but rely heavily on well-engineered logic priors and fail to generalize beyond synthetic data.
Visual Tokenizers. basch-based tokens have been adopted as the standard for visual understanding (Dosovitskiy al., 2021) and generation (Peebles & Xie, 2023). Variations of this template include discretized patch tokens (Du et al., 2024), mixed-resolution patch tokens (Ronen et al., 2023), and pruned patch tokens (Kong et al., 2022; Tang et al., 2023). Beyond patches, recent works have explored region-based tokens (Ma et al., 2024). However, these tokenizers are inherently grounding-aguostic, in contrast to humans, who possess the ability to abstract concopts based on linguistic or cultural grounding priors, (Sogall et al., 1966; Winawer et al., 2007). To this end, our work explores a grounding-awe tokenizer.
3 Preliminaries
We detail a few essential preliminaries this section. Readers should refer to the original manuscripts for further details.
3.1 Slot Attention
Object-centric learning frameworks decompose scones by organizing them into compact representations called slots. Slot Attention (SA) (Locatello el al., 2020) is a powerful iterative attention mechanism for learning such slots from perceptual features extracted from vision backbones. At iteration t, for L features H g2xe and K slots St E Rxxd the slots compete to explain the features follows:
M ((H)Q(S*) VD
Mas
E BRIK, = Eje(1 K) My
(1)
Ag
Update = WT'(S") where Wi Eie(L,..N) Aij
(2)
D denotes the dimension of the atlention head. Here, Q(-) E R K()) ReXD V(.) dx are learned query, key, and value matrices, respectively. After attention- weighted aggregation of features, Gated Recurrent Unit g() (Cho et al., 2014) updates the slots as follows:
s++1_
G(state St input Update)
The aggregation and updates are performed over multiple iterations to obtain the final representations. Slots have been empirically shown to bind to the features of the same object as the forward iteration proceeds, yielding per-object representations. We refer the reader to the original article by Locatello et al. (2020) understand the complete rollout.
4

---

Published in Transactions on Machine Learning Research (05/2025)
3.2 DINOSAUR Slot Attention
The original slot attention could learn slots on synthetic objects but failed to scale to real-world scenes. A recent object-centric learning method by Seitzer et al. (2023), called DINOSAUR, uses the DINO vision backbone (Caron et al., 2021) to scale slot representation learning to complex real-world scenes. The funda- mental insight of the work lies in using features from strong, pre-trained encoders perform slot attention instead of pixel-level reconstruction used by prior works. The authors use the rich representations from DINO a substrate for learning grouped slots that capture spatial objectness features. Motivated by these findings, we adopt the pre-trained DINOSAUR our slot-attention backbone for NST and fine-tune it for semantic grounding.
3.3 Hungarian Matching Criterion (HMC) for Prediction using Slots
Learning object properties from slots a challenging problem because the ground truth correspondences between objects and slots are unknown. Practitioners circumvent this problem by assuming that every slot contains single object and minimize the costs associated with the bijection mapping from the set of slots to the set of objects using the Hungarian Algorithm (Kuhn, 1955). Given set of slots 81:K object properties pi:k and assigument cost function C(-), the Hungarian algorithm is polynomial-time algorithm that finds the optimal assignment (+):::
Σ C(S", pUci) iE(
U = arg min
(4)
HMC is the primary method utilized by prior methods (Locatello et al., 2020; Seitzer et al., 2023; Kori et al., 2024a) to predict object properties from slots.
4 Neural Slot Interpreters
Recall that the goal of NST is to ground concepts into slot representations such that the objects contained within the slot align with the embodied notions of the object. We begin by discussing the proposed organi- zation for scene labels.
4.1 Nested Object- Annotation Schema
We propose a simple nested schema for expressing scene labels as a collection of objects and their associated properties. At a higher level, instances in the schema comprise multiple object- centric primitiies (<element>). Primitives, in turn, contain re- spective object properties that form alomic units of the schema. Each primitive identifies a unique object in the scene. The nested properties <p,> (pi,...,,P)) form the children of the parent objects <element>. Some examples of <pj> properties are, but not limited to, shape, material, category, and object position. Thus, instance primitives naturally capture the no- tion of an object, and neural representations extracted from primitives are, such, well-suited for being grounded in slots, In Section 5.1, we demonstrate the straightforward application of the schema lo organize labels on popular datasets. See Fig. 2 for an example instance and Appendix A.3 for more examples. On a more practical note, such schemas are commonly used software abstractions and can be ubiquitously interfaced with graphics engines, web APIs, or even large language models for semantic understanding (Dunn et al., 2022; Bubeck et al., 2023).
solament:11d00ccmm www.i//wwwtpeeeee//eeeeeeee Abbsad .... <78boa0k //lement
Coloanat Libb-ceattooe//aaa .... </bboa0 /alemett
Falleeet:: 1.S catroeeche catt
colomeett Lufoorattpp coac/aaat
calement: dato
Description of a real-world scene using the nested schema. The dotted ar- rows show correspondences between primi- tives and the objects they annotate.
Figure 2:
5

---

Published in Transactions on Machine Learning Research (05/2025)
4.2 NSI Grounding Technique
Scenes and their corresponding schema instances capture object-centric representations through slots and primitives, respectively. NSI learns to align the object-centric representations of the respective modalities, i.e, slots and primitives, by grounding neural representations of schema primitives into slots (see Fig. 3). The grounding is learned by optimizing a contrastive learning objective over ground- ground-truth scene-schema pairs (see Fig. 4). We describe the scene and schema encoder next.
I1.B
Encoder
S1:K
H1.B
Object-Centric Learning
Slot Atlention
SA(.)
(.)
Contrastive Learning
Figure 3: NSI overview. NSI augments object-centric learning autoencoders with contrastive learning objective batch of scene-schema pairs. A DINOSAUR backbone (Seitzer et al., 2023) extracts slot reprosentations S.KK from batch of scenes and a schema encoder extracts neural primitives 2 from their corresponding schema pair. The slots are then passed to a decoder for reconstruction and the slot-primitive neural pairs are passed to the contrastive learning objective.
4.2.1 Learning Scene Representations
A given scene Ia is represented via K slots 51:K E RR xd abstracted from its perceptual features H, E R (see Fig. 4 (a)) For given feature extractor E,(.), the slots are obtained as
H. (Ix)
S SA(H.) - Slot Attention
A spatial broadcast. decoder Do(-) (Locatello al., 2020) reconstructs the features from slots, with the reconstruction error used a learning signal:
fl. = Do D (S
Lrecon = Hr Al
(6)
4.2.2 Learning Schema Representations
A bi-level architecture (see Fig 4 (b)) learns neural representations of the schema primitives. First, a lower- level primitive encoder learns property-specific dictionarics D(.) and embeds the property features into neural primitive representations. For discrete-valued properties, D(.) modeled as a simple lookup table of learnable weights, while continuous-valued properties are embedded via multi-layered perceptrons (MLPs). Let the dictionary (.).) learn features for property Pj- Then, primitive embedding Zprim is computed as:
= concal (Di(1),, (D(t))) Zprim = MLP(Z)
Note that these lower-level representations are schema-agnosiic and only capture object-specific features. Then an upper-level schema encoder uses a bidirectional schema Transformer to further embed primitives, endowing representations with the overall schema context. For a given schema instance Pg with N primitives, the final representations Z1:N Bgxx are computed via Transformer Tachema (.) as:
71:N 1:N T Tschema (Zprsm 7(prim)

---

Published in Transactions on Machine Learning Research (05/2025)
4.2.3 Compositional Score Aggregation
Recall that we want to ground entities Z3a0 into entities S1:K, As a first step, we project these embeddings into a shared semantic space Y E Rdos, The projection head (.)ene(.) for slots is modeled as the following residual network:
/ W Wp W E 1R
Y = Y MLP (LayerNorm (V:))
Here, Layer Norm denotes the layer normalization operation. A separate residual head Hachema projects the primitive representations Z into the semantic embeddings Y Next, we supplant the traditional single object per slot assumption by assigning each primitive to its nearest slot in the latent space, measured by dot-product similarity. The similarity score Sny between a scene and primitive is the sum of neigest-n similarities resulting from the primitive-slot assignment (see Fig. 4 (c)).
arg max y Y (1.,,KKK K)
Say =
E
max kE(L,-. .] ( (x**) y
(10)
nE(l.",,N) .N
4.2.4 Contrastive Learning Objective
The mopelity-specific embeddings and the resultant grounding are learned by optimizing a contrastive learn- ing objective (Fig 4 (d)). More precisely, given a B-sized batch of {scene, schema) pairs, we use the Say scores lo distinguish the B correct pairs from the B B incorrect pairs. The probability of correctly clas- sifying schema Pg as the true pairing for scene Le (and conversely predicting Ly from P2) is formulated as follows:
exp (Suz/T)
exp (Suz/T)
pachema
paccen
(11)
yE exp (Smy/T)
yE ,B) exp (Syz/T)
Here, the calculated scores are interpreted as logits and r denotes the temperature parameter. The cross- entropy losses for scene and schema prediction are given by:
Lschema =
Σ log (pschema)
Escene
Σ
log (IpzceRe)
(12)
B)
The global contrastive learning objective is based on the InfoNCE loss (van den Oord et al., 2019) follows:
Lcontrastine = (Lscene + Lschema))2
(13)
The overall training objective for NSI is given by:
Lirain = B1 X Ccontrastive + B2 Lrecon
(14)
Note that 31 0.0, 1.0 corresponds to traditional autoencoder object-centric learning frameworks. The learning objective can be interpreted as estimating a lower bound the Mutual Information (MI) between the scene and schema. distribution by optimizing the compatibility of their compositional embeddings. Let the MT between the joint scene and schema distribution p(-) be defined as:
MI(I, P) = ((I,PP))o((..))) log p(I,, Pv)
(15)
p(L))p(P,).
van den Oord et al. (2019) show that the optimized InfoNCE distributions IP" Pr and Poeeee provide a low variance estimator of MI (which generally intractable for high-dimensional data). The InfoNCE lower bound MI given by
MI(I, P) log(B) Lschema MT(I,P) 2 log( B) Lacene
(16)
MI(IPP) (og(B) Lackema + Lacene = log (B) - Loontrastive
(17)
2
which becomes tighter as becomes larger. Therefore, minimizing the NSI contrastive loss maximizes a lower bound on MI. Algorithm 1 presents the NSI pseudocode for aggregating local alignments and computing the contrastive loss.
7

---

Published in Transactions on Machine Learning Research (05/2025)
(a) Scene Encoder
S1:K
H
Patch Pokenizer
Actantion
Vir
DINO
E.(.
(b) Schema Encoder
2
22
2
2g
Schema Encoder Transformer Ta(.)
<tboas Phloec/elennii enwiic
prins
2 pries
2
Z
calaeant: dioosatooonbbccat cato
</boos></elemmntt
calecant: www.aaatooookkccaaaaa..a.
Primitive Encoder
D(.)
ObooP... </tooo2</slement>
Primitive tive
Primi tive
Primi tive
Primi Eive
(c)
Compositional Score Aggregation
(d)
Contrastive Learning Objective
mazsel...
Zowar
811
y/ur
Figure 4: NSI method. (a) A DINOSAUR encoder (Seitzer et al., 2023) learns to represent images via slots, (b) A bi-level schema encoder learns representation of schema primitives. The primitive encoder embeds the object properties of each schema primitive. Then, Transformer learns embeddings that assimilate the entire schema context. (c) The inner loop of the metric computes the score Szy between compositional abstractions of an image Ig and a schema Py- Object slots and schema primitives are projected onto a shared embedding space and every latent primitive is assigned to its nearest slot for score aggregation. scores obtained from locul entities are used to optimize a global contrastive learning objective the outer loop over batch of image-schema pairs.
(d) The Szy
5 Experiments
In this section, we set up experiments to (1) answer whether the slots are coherently imbued into the notion of the object properties, (2) study how the grounding compares to traditional weakly supervised contrastive

---

Published in Transactions on Machine Learning Research (05/2025)
Algorithm Neural Slot Interpreter Contrastive Learning Pseudocode
Require: Projection heads Nscene (-), Hachema
2: Require: Batch of slot embeddings from the slot encoder { K h:BB
3: Require: Batch of primitive embeddings from the schema encoder 1 J1.8
4: # Compositional Score Aggregation
5: [Y-: J1:B Hacene ((S). 11
b Project slots
6: (Yg h:B = Hschema (1Zy- 11:8)
v Project primitives
for all i (1,... ... B} do
Compute S R
8: Sij Ene(I,.....)) maxke(1.".. K} YkT Yn
9: end for
10: # Contrastive Learning
11: labels arange(B)
12: Lschema Cross Cross Entropy Loss (labels; S)
13: Lscene = Cross (CroosEnnropyLov (labels, S²)
14: Leontrastive (Lachema + Lsoene) /2
15: return Lcontrastive
learning, (3) estimate the quality of alignment under different annotation regimes, (4) probe whether the NSI objective leads to the emergence of slots that bind to raw object features, and (5) answer the slots are effectively grounded, do they enhance downstream performance?
5.1 Schema Instantiation and Architecture Backbone
Our experiments encompass different tasks scenes ranging from synthetic renderings to in-the-wild scenes viz. (1) CLEVr Hans (Stammer et al., 2020): objects scaltered on plane, (2) CLEVrTex (Karazija et al., 2021): textured objects placed on textured backgrounds (3) MOVi-C (Greff et al., 2022): photorealistic objects on real-world surfaces, and (4) MS-COCO 2017 (Lin et al., 2015): a large-scale object detection dataset containing real-world images. In pre-processing step, we organize scene labels into the proposed schema (Section 4.1). The property tags <p,> that populate schema primitives in each dataset are listed in Table 1. We use the schema instances ground object information in their corresponding scenes via NSI. See Appendix A.3 for schema instances.
Dataset CLEVT Hans CLEV MOVi-C MS-COCO 2017
Property tags <p,>
<color>, <shape>, <material>, <size>, <3D position <texture>, <shape>, <size>, <3D position> <category>, <scale>, <2D position>, <bounding box> <category>, <bounding box>
Table 1: Property tags used instantiate and ground schema primitives.
We followed the DINOSAUR (Seitzer et al., 2023) recipe for learning slot representations. DINOSAUR uses semantically-inormaatee DINO ViT (Caron et al., 2021) features autoencoding objective, significantly improving real-world object-centric learning abilities. More specifically, we train an MLP decoder to re- construct features from slots for all our experiments. Training details and hyperparameters are given in Appendix C.2.
5.2
Grounded Compositional Semantics
5.2.1
Scene-Schema Alignment Evaluation
Grounded concepts should be effectively aligned the slots they represent, and we operationalize this ability through the lens of retrieval, task providing a direct and quantifiable test of this concept-slot alignment. To this end, set up bimodal scene-property retrieval task.

---

Published in Transactions on Machine Learning Research (05/2025)
This task consists of two databases annotation database and scene database. The annotation database contains collection of object properties, and the other contains collection of scene images. Each set of properties in the annotation database has a corresponding scene in the scene database. In the first half, we measure property retrieval, where models are tasked with retrieving the correct set of object properties, given the scene, using their respective alignment scores. In the second half, the task is inverted to perform scene retrieval, where the models search for scenes the scene database, given the object properties. We set up the task on CLEVrTex, MOVi-C, and MS-COCO datasets using their test split to instantiate the databases. The task contains 10000, 6000, and 4952 instances in both retrieval databases for CLEVVTTex, MOVi-C, and MS-COCO, respectively.
Experiment Setup:
Baseline Setup:
We consider several baselines for the retrieval task spanning the compositionality contin-
uum.
(a) CLIP embeddings: (Radford et al., 2021) CLIP embeddings have been explicitly trained for retrieval on image-text pairs and form a strong non-compositional bascline. Here, we encoded the schema as a string and measured the similarity between the text-image CLIP embeddings.
(b) GroundingDINO (Liu et al., 2024): We consider GroundingDINO, which is an open set vision- language model pre-trained a large corpus to align text and images on grounding objective. Unlike CLIP, which is non-compositional the alignment in GroundingDINO is dependent the compositional sub-alignment between text phrases and image regions. We directly mcasure the similarity score outputs obtained from the model on a (schema string, image) pair to perform retrieval.
(c) Ungrounded slots (Seitzer al., 2023): Our first slot-based baseline utilizes the slots from a frozen pre-trained DINOSAUR backbone with shallow predictor fine-tuned for property prediction using HMC (Kuhn, 1955). Subscquently, used HMC scores for retrieval. See Appendix C.1 for training details.
(d) HMC matching: For this baseline, we fine-tune the ungrounded slot architecture, including the backbone end-to-en,d to predict object properties from slots. We used the optimal HMC scores of fine-tuned slots for retrieval.
(c) Ablations: On NSI, where Net-ResNet 34 replaced DINO VIT with the ResNet backbone, described in (Elsayed et al., 2022), and on NSI-Schema Agnostic where schema primitives were encoded without the schema Transformer (Fig 4 (b)).
Metrics: We report Recall@K (k E [1,5)), which measures the fraction of times correct item was retrieved among the top K results. The results of the retrieval task are in Fig. (l)), and Fig. 5(d).(o).() visualizes the alignment learned by NSI. We now highlight the major observations.
(a) Explicitly grounding slots through NSI significantly enhances semantic aligument com- pared to set matching approaches: HMC matching slots that rely on set-matching of predicted slots exhibit notably weaker performance across all datasets in both property and scene retrieval tasks. NSI and its ablations, by design, learn to associate individual slots with semantically meaningful object attributes. This explicit grounding mechanism proves crucial, especially plex and realistic datasets like MOVi-C and MS- MS-COCO, where the performance gap between NSI-based models and sel-matching methods widens considerably.
move towards more com-
(b) The full model is essential: Schema-agnostic encoders that encode object primitives independently perform sub-par compared to NSI. This underscores the importance of contextualizing object semantics in the overall schema. The pre-trained DINO backbone is crucial for textured objects and real-world generalization, evidenced by CLEVITeX/COCO results.
(c) Vision-Language Models grounded in text inadequately capture object semantics: CLIP embeddings and GroundingDINO both fall short in retrieving object properties beyond basic semantic categories, evidenced by the weaker performance MOVi-C and CTxV datasets. This limitation
10

---

Published in Transactions on Machine Learning Research (05/2025)
Property Retrieval
Scene Retrieval
(a)
(d)
100
ceize>Wediumc/
cna enaterial-Sanda c
<ahape>Cube</ahapa
CLEvrTex
Percent:
Kaise>Hediunk//ise
Kna
<chapeWoolees</ehhppp>>
<aite>Maduun</iioo>
<ahappocyLinderc//hhhpp
(b)
100
KoatagoryyToccaaaeeerr <acale>1 33</scale>>>
80
MOVi-C
Percent
60
<scale>1 89</scale>
<acale>2 21<sac/ee
KateporyXonnuurroocccttt
<acale>0 0tte//sl..///sle.
(c)
<catepor>porsonc//aateea www.:/rrscccccaacprr....... ccatapprypparooss/aatppppyy
MS- MSSCOCO
cateepcyyboxk//sstapppyyy
kontege:ppagpoor///attggegg
<atteporpsbokk///tttprrrr
categprrpbottlle///aaagggg
<catoporyadoor//attepryy
CUP
GroundingDINO
Ungrounded
HMC Matching
N5I-ResNat34
NSI-Schema Agnestic
NSI
Figure 5: Retrieval results. (a), (b), (c) Property and scene retrieval results. We report Recall@1/5 (higher is better). The standard deviation (over five seeds) was < 0.3 across all model instances and retrieval tasks. The lighter shade shows Recall@5 while the darker shade shows Recall@1. (d), (c), (f) Visualization of cor- respondences learned by the NSI similarity metric. The colored arrows show the respective correspondences of schema primitives to the slots. Each schema instance chunked and color-coded by the slot to which its primitives are assigned.
further emphasized by their relatively weak performance across the two datasets compared to methods that explicitly model object compositionality in scene and annotation representations. We posit that while natural language is effective in grounding object properties pertaining broad categories (as evidenced by the stronger performance MS-COCO), it appears insufficiently equipped for object- centric grounding (Chandu et al., 2021).
Qualitative: In Fig. (e)),, we visualize the slot-object pairs inferred by our scoring metric. Inter- pretable and dense correspondences emerge from NSI contrastive learning. Notably, in complex real-world COCO scenes (Fig. 5(f)), NST slots successfully ground to a diverse set of objects, even capturing instances where single slot might be associated with multiple related objects (c.g., multiple bowls and spoon associated with slot). See Appendix D.1.6 for more results.
5.2.2 Comparison against Traditional Weakly-Superviser Visual Grounding
Weakly supervised visual grounding, by learning to link phrases in caption to specific regions in an image, is well-ostablished area of rescarch (Karpathy & Fei-Fei, 2015; Ych et al., 2018; Wang & Specia, 2019; Gupla et al., 2020). These same region-phrase aligument techniques can be easily adapted to the scene- schema grounding problem by replacing phrase embeddings with schema representations. However, a key distinction between these approaches and NSI how scenes are represented. Traditional imagc-caption alignment methods rely on pre-trained models propose bounding boxes around objects, while NSI learns slot. representations concurrently with the grounding objective. In this section, then explore (1) whether
11

---

Published in Transactions on Machine Learning Research (05/2025)
Property Retrieval
Scene Retrieval
(a)
Data-Scarce
Data-Flich
Data-Scarce
Data-Rich
BBox QKY
80
MOVI-C
Percent.
60
NSI
(b)
1%
10%
50%
100%
1%
10%
50%
100%
R@ 1
Ms-COCO
Percent
60
R@5
20
1%
50%
100%
10%
50%
100%
Fraction Training Data
Fraction Training Data
(c)
MOVi-C
MS-COCO
10% Training Data
Training
10% Training Data
Training
InfoNCE Lower Bound
20
-46-
80-
110
-100
3006 2006 3900 6259 se65 roce 7000 9003
5000
19000
15/0e
20002
Iterations
Iterations
Iterations
Ittraaionns
Figure 6: Comparison against traditional weakly supervised visual grounding. We compared NST against a state-of-the-art bounding box-based phrase grounding method (Gupta et al., 2020). (a) and (b) compare the performance of either method on the property-scent retrieval task on MOVi-C and COCO datasets, respectively. Despite being trained on the same InfoNCE objective, NST outperforms region-based ground- ing, demonstrating that slot representations emergent from the NSI objective are key to grounding object semantics. Moreover, NSI is also data-efficient in real-world COCO scenes.
(c) plots the InfoNCE lower bound attained on the test set as training progresses (smoothened for improved visualization). NSI has a stronger lower bound and more effectively captures the scenc-schema MI via object-centric abstractions.
bounding box representations are equally effective as slot representations at grounding object properties, and (2) how the elficacy of each method compares across different annolation levels.
Experiment Setup: We investigated the efficacy the contrastive learning- phrase grounding method by Gupta et al. (2020) and trained it on scene-schema pairs from the MOVi-C and COCO datasets. This method utilizes bounding box proposals to encode scenes, and we refer to it as BBox QKV. Like NST, BBox QKV optimizes the modality encoders to maximize the lower bound on MI via the InfoNCE loss. In addition, this method trains extra Query, Key, and Value heads to compute similarity scores between image regions and schema primitives via the attention method.
Bounding box-based grounding methods rely on detectors like Fast R-CNN (Girshick, 2015) to extract image regions. To simplify the setup and eliminate errors arising from the detection pipeline, we directly utilize ground truth bounding box annotations provided in the schema. The bounding boxes are then represented by directly averaging the DINO features contained within the box and used to ground scenes into schema. labels.
Baseline Sctup:
We further sought to understand the data efficiency of NSI against the conventional bounding box-based approach. To this end, we controlled for training data size and explored four training regimes for each method: {1%, 10%, 50%, and 100% } of the training dataset, ranging from data-scarce to data-rich.
Dataset Setup:
We evaluated BBox QKV trained on various dataset sizes on the bimodal retrieval task setup from the previous section. The retrieval results are summarized in Fig. 6. Our major findings were:
Evaluation Setup:
(a) Slot encoding outperforms Bounding Box encoding. NSI consistently outperforms BBox QKV in both Property and Scene Retrieval tasks on either dataset (see Fig. 6(a) and (b)). This performance
12

---

Published in Transactions on Machine Learning Research (05/2025)
BBox QKV
NSI
(a)
10% training data
50% training data
training data
50% training data
MOVi-C
<oategory"Bagccceeeo tegory? <acalab2, 36c.ab/leec///ll..
<eategory-Sheeek//ee cat//otteegy
<attserysShoorraateepry <acale>2 43</scale>
<catago:pSShoac/zaaappae <ocale>2 43</scale>>
<scale>2 43//soae>>
SerpoorpsShoe</catpprr. <scale>2 43</scale>
SattaggryyBag//atagggry cecale>2, 66</scale>
<csteeprrppaa//caaaagee <scale>2. 65</scale>
<csteparpagg</attppr>> <ecale>2 684/scale>
coateposyxooss<eatteppo <scale>2. 67</scale>
ccatapoyyNiona//aataere <scale>2 <scale>2. 57</scale>
<ca (catagoryyHone<//ee category? <scale>2. 67</scle>>
ccataporyxMons//aaaeppee <scale>2 67</scale>
(b)
10% training data
50% training data
10% training data
50% training data
MS-COCO
ccategosyRRotted
<catagozy<Rotted
<cateporysPotted
<ontegorysPotted Plant</eategoz>>
>lants/catwgory>
>lant</eategory>
Flant</sategor>>
<ateppryboo</aategppp
cateepry2ook/esteeerr
<atepprysPrroor//aaappry
<oa casteeccphhessec//aaaapapy
Cateegry>Rersokkaattppyy
<category>Teddy
?
Bearc/category>
SattegorySoog/cctteprr
<cateporysboor/ccaaeppa
<ostagory>
coatagoryoPeddy
<category>-eddy Bear</catagory>
Bearc/category>
Bearc/categorp.
castepory2TK/cattppor
<categosy/TK<cattegryy
kateeprysPerronn/aattoooy
<astepory/VK/aateepry
<csteposyTKK/ategppp
?
Figure 7: BBox QKV vs NSI retrioval results. After being trained on two different settings, we demonstrate the emergent correspondences from BBox QKV and NSI. For NSI, at the 10% setting, the slots are less robust than the 50% sctting, however, the correspondences between objects in the slot and their labels are accurate overall. On the other hand, we observe multiple failure modes for BBox-QKV at. low-data settings where overlapping regions are assigned incorrectly the model fails to ground objects at all (see (a) and (b), BBox QKV 10% training data). The issue with disambiguating occluded and overlapping objects persists even with greater training data.
difference ariscs despite both methods employing identical InfoNCE estimation and scoring functions, differing solely in image encoding: NSI utilizes slots, while BBox QKV relies on bounding boxes. This suggests that coarser bounding boxes may underspecify objects they contain, leading to weaker semantic grounding and diminished retrieval efficacy than slot-based counterparts. On the other hand, the slot- attention mechanism explicitly encourages symmetry breaking between slots such that they encode non-overlapping fine-grained spatial features that facilitate improved alignment to object features.
13

---

Published in Transactions on Machine Learning Research (05/2025)
(b) NSI is data-efficient real-world scenes. NSI exhibits remarkable data efficiency, demonstrating robust performance even in data-scarce scenarios, notably on MS-COCO. Performance remains strong with only 10% (~ 10,000 examples) of MS-COCO training data. Despite limited data, this robustness highlights that grounding via the NSI objective facilitates a strong transfer of discriminative object semantics to slots that originally only contained spatial biases. Moreover, such transfer is significantly more data-efficient compared to the performance of bounding box abstractions across dataset sizes.
(c) NSI achieves a higher InfoNCE lower bound. We plot the InfoNCE lower bound (Eq. 17) obtained from the validation split across the training iterations in Fig. 6(c). NSI consistently achieves much stronger (highor) lower bound compared BBox QKV for both MOVI-C and MS-COCO datasets and across annotation levels. A higher InfoNCE Lower Bound for NST indicates that it is more effectively extracting the MI between schema and scene pairs. This further suggests that the slot- based representations learned by NSI allow for a more discriminative representation of the relationship between scenes and their object semantics.
Qualitative: Fig. 7 (a) and (b) demonstrate the correspondences learned by NSI and BBox QKV across the datasets and the 10% and 50% training data fractions. The qualitative results further illuminate the limitations of bounding box representations, particularly in scenarios with object overlap or occlusion. As seen in MOVI-C (a) and MS-COCO (b), BBox QKV exhibits failure modes when objects are not distinctly isolated, often misattributing objects (e.g., see MOVI-C 10% data) or failing to ground objects altogether. These instances suggest that bounding boxes, being granular and encompassing potentially overlapping regions, struggle to disambiguate objects and their semantics, especially when trained on fewer instances.
5.3 Object Discovery with NSI
Object-centric frameworks have been traditionally used to bind neural network representations to distinct objects within scene. Here, we evaluate whether grounded slots are more adept at discovering objects in the context of visual segmentation.
Experiment Setup: We extract object attention masks from the backbone derived from slot-attention clusters. The masks are then upsampled from the feature resolution to image resolution. We compared these masks to the ground truth segmentation across the CLEVrTex, MOVi-C, and MS-COCO datasets on the test split scenes.
Baselines Setup: We compare masks from the DINOSAUR scene encoder trained on three different ground- ing objectives.
(a) Ungrounded slots derived from the autoencoder reconstruction objective, without any semantic grounding.
(b) HMC matching grounded slots learned from predicting object properties from slots via set-matthing,
(c) NSI grounded slots that explicitly ground object properties in slots via latent semantic assignment.
Metrics: We report the quality of object masks obtained from the slot representations based on two segmentation metries: (1) Foreground Adjusted Rand Index (FG-ARI): measures the accuracy of clustering foreground objects into their respective segments and (2) Mean Best Overlap (mBO): assesses the best overlap between predicted and ground truth object masks. We report both instance (mBO*) and class BOBO)) level mOO scores for COCO. The segmentation results can be found in Fig. 8(a). We observe that:
(a) NSI endowed slots meaningfully segment objects. Object masks generated by NSI are com- petitive on synthetic scenes and markedly improve object discovery on COCO scenes. We posit that contrastive learning via NSI enhances symmetry breaking of the slot attention backbone for challenging real-world scenes, effectively binding slots to raw visual features.
(b) HMC matching obscures object discovery on COCO. Constraining slots to predict a single object forces the backbone develop specialized representations for each object. This imposes a difficult learning problem on real-world scenes and causes slots to deteriorate.
14

---

Published in Transactions on Machine Learning Research (05/2025)
(c) NSI is biased towards semantic classes over instance classes. On COCO instances, semantic segmentation scores (mBOs) are higher compared to the baseline. We attribute this to the NST metric that biases slots to represent broader categories by grounding multiple objects.
(a)
90
0.0 0.58..8.. 2.10
Ungrounded
HMC Matching
NSI
Scores
0.88
60
0.37 174
4.12
0.0
0.00 1.73
-7.94
5:56
30
0.0
0.0
20
-8.88
6.93
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
CLEVrTex
MOVi-C Ungrounded
MS-COCO
(b)
Original
Ground Truth
HMC Matching
NSI
CeEVrTex
MS-COCO MOVi-C
Figure 8: Object discovery results. (a) Segmentation results on three datasets. We report scores of FG-ART (higher is botter) and mBO (higher better). We also show scores relative the ungrounded bascline on top of each bar. (b) Visualization of atlention masks learned by different models instances of the datasets. See Appendix D.2.1 for error bars and more qualitative results.
Qualitative: Fig. 8(b) visualizes the object masks that emergent across the various methods. We notice that NSI grounding of object semantics is helpful in disambiguating masks that violate object boundaries or contain more than one object. For example, in the CLEVrTex example, the yellow mask learned by the Ungrounded instance contains two objects, but the NSI instance, which encourages each slot to map to its distinct semantics, separates the two objects. Similarly, in the case of the COCO example, the boundaries of the bench and person are obfuscated and ambiguously overlapped. However, in the NSI instance, the learned mask of the person is non-contiguous and allows for the full bench mask to form.
15

---

Published in Transactions on Machine Learning Research (05/2025)
CLEVr-Hans
(d) Rule
Scene
Ungrounded
NSI-CLEV/Tex
NSI-CLEVr
(a)x
Patch
CoSA
Uhgrounded
Large
Accuracy 186
NSI CLEVriex
Large Cylinder
NSI-CLEVr
and Small Sphore
(b)
CLEVr-Hans
Accuracy
Yellow Sphere
Hars 100
Hans 7 1020-shot
Hans Put
Cyan Object
(c)
Hanss
A5S--U.5 Accuraa y (S
Hanss
Two Red Objects
Three Spheres Left Side
Number of Grounding Examples (log scale)
Figure 9: Few-shot classification results. (a) (b) Few-shot classification accuracy on Hans 3 and Hans 7 datascts, respectively. (c) Task accuracy on NSI-CLEVr tokenizers trained on diffcrent numbers of grounding examples. The standard deviation (over five seeds) was < 0.03 across lasks and methods. (d) Visual rationales are generated across the grounding continuum by extracting attention maps from the final ViT layer. See Appendix D.3.2 for more examples.
5.4 Grounded Slots as Visual Tokens
Grounding-agnostic patch-based tokens the de facto standard for transformer-based models. On the other hand, humans can flexibly abstract out entities free of rigid geometric templates. Here, we investigate the ability of grounded slots to bridge this abstraction gap on a downstream classification task.
We consider the CLEVi-Hans classification benchmark (Stammer et al., 2020). The data set comprises images derived from the CLEVr scenes, which are partitioned into distinct classes. Class membership is determined by predefined combinations of object attributes and their inter-object relation- ships (e.g., 'Large cube and large cylinder'). Notably, specific classes within this collection are intentionally designed with confounding factors that consists of multiple classes based object attributes and rela- tions. Moreover, the true membership properties are confounded with other attributes in the train split. Consequently, achieving accurate classification on test images necessitates that the model effectively decon- found the attribute correlations inherent in the training set. Here, we maintain a common VIT classifier (Dosovitskiy et al., 2021) backbone and train on tokens obtained from different tokenizers. We conduct experiments on CLEVi-Hans and CLEVi-Hans benchmarks containing three and seven classification categories, respectively. We further sweep across training dataset size across 100, 1000, and the full training set.
Experiment Setup:
Baselines Setup: We explore the following tokenization schemes:
aa) Traditional patch tokens extracted from 14 14 image patches of the scene.
16

---

Published in Transactions on Machine Learning Research (05/2025)
(b) Ungrounded slot tokens learned from autoencoder object-centric learning on CLEVr.
(c) Conditional Slot Attention (CoSA) slot tokens (Kori et al., 2024a) derived from set-malching slots to property labels on the CLEVr dataset.
(d) NSI-CLEVrTex slot attention trained on CLEViTex via NSI. The backbone is frozen and subsequently used to infer CLEVr slots. This setup makes the slots partially grounded because CLEVr and CLEViTee share common attribute types (shape, size, texture), and the scenes and the scenes across the datasets are also structurally similar.
(e) Fully grounded NSI-CLEVr slot attention trained on CLEVr via NSL.
Metrics: In Fig. (b),,b)) we report lest accuracy against the k-shot training sweep for CLEVi-Hans 3 and CLEVi-Hans 7. We find that:
(a) Grounded slots facilitate improved reasoning and surpass patch- tokenizers across data regimes using 25x fewer tokens. While the performance of the patch-tokens saturates, grounded slots de-confound object attributes with greater ability with increasing data.
(b) Partial grounding is often sufficient: NSI-CLEVTTex slots grounded in a different dataset demon- strate competitive reasoning abilities. CLEViTex and CLEVr capture the common structural notion of objocts in scene but significantly differ in their texture semanties. However, tokenizer trained on CLEVrTex still facilitates downstream reasoning on the CLEVr scenes. This result further corroborates the strong transfer effect of explicit grounding observed in Section 5.2.2 where grounded slots assimilate generalizable representations with annotations from the same scene and also facilitate rapid transfer across distinct stimuli.
(c) Property prediction ability does not necessarily transfer to class prediction: CoSA slots trained on abject-attribute prediction learned from HMC matching show weaker transfer on the Hans-3 dataset. This points out another limitation of the one-to-one template imposed by HMC where slots are predictive of individual objects but lack the contextual abstractions for downstream inference.
How many schema annotations are essential? In Fig. 9(c), when ablating the number of annotations used to train the NSI-CLEVr tokenizer, we observe that inductive biases instilled by grounding are key, as soen from the sensitivity of the performance to the number of examples. On the other hand, significant annotation of scenes is not uecessary. Annotating just 100 schema examples yields performative accuracy within a 3% margin of the tokenizer trained on the complete set.
Qualitative: We probed the attention maps from the final layer of the ViT to generate visual rationales. Fig. (d) visualizes the maps across the grounding continuum. The ungrounded slots showed little to no correlation with the class rule. The partially ground slots in CLEVVTTe are more discriminative than the ungrounded slots but often yield incorrect rationales. On the other hand, ViT trained on the NSI-CLEVr slots and weighed slots pertinent to the class rule with greater attention. The visual rationales can be valuable in interpreting the predictions of the model.
6 Conclusion
This work introduced NSI, which grounds object semantics into slots for object-centric understanding. Tt uses a simple schema abstraction to define object concepts and learns to flexibly associate neural embeddings of the schema primitives with object slots via contrastive learning, NSI facilitates interpretable grounding in slot representations. Whereas natural language-groudedd embeddings struggle to retrieve granular object properties, NST embeddings abstracted from the slot-schema intermodal alignment: are key representations for such tasks. The effectiveness of the representations is also highlighted when pre-specified bounding boxes trained with the same contrastive learning objective demonstrate weaker grounding. Furthermore, unlike set-matching approaches, which struggle when scaled to real-world scenes, NSI enhances object discovery compared to ungrounded counterparts. Finally, we demonstrated the uscfulncss of NSI as a grounding-awaro visual tokenizer that improves the few-shot visual reasoning abilities of ViTs on a hard classification lask. The major limitation of the work lies in requiring annotations that can be prohibitively expensive and laborious. However, we empirically showed the data efficiency of the method on real-world scenes where using as little as 10% training data yielded grounding comparable to the full setting. We also demonstrated performative reasoners under practical annotation settings. Slot representations have traditionally been associated with
17

---

Published in Transactions on Machine Learning Research (05/2025)
visual stimuli, but object concepts transcend perception to other sensorimotor experiences like audio, tactile signals, and motor behaviors. To this end, NSI lays the groundwork for multimodal object-centric learning. Future work involves adopting NSI to ground common object-centric concepts into different sensorimotor experiences as a step towards a modular human-like understanding of the world.
7 Acknowledgment
This work was supported by NSF under grant no. CCF2203399. The experiments reported in this article were performed on the computational resources managed and supported by Princeton Research Computing at Princeton University. We also express gratitude The Sacred Grounds Cafe and Karma Cafe in San Francisco for brewing the best coffee North of the Panhandle.
18

---

Published in Transactions on Machine Learning Research (05/2025)
References
Federico Baldassarre and Hossein Azizpour. Towards Sell-Supervised Learning of Global and Object-Centric Representations. CoRR, (bss/220300599,, 2022.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Fee Kamar, Pe- ter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of Artificial General Intelligence: Early Experiments with GPT-4. CoRR, abs/2303 12712, 2023.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised Scene Decomposition and Representation. CoRR, abs/2303.087// 08774, 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. CoRR, abs/2104.14294, 2021.
Khyathi Raghavi Chandu, Yonatan Bisk, and Alan W Black. Grounding 'Grounding' NLP. CoRR, abs/2106.02n 02192, 2021.
Michael Chang, Alyssa L. Dayan, Franziska Meier, Thomas L. Griffiths, Sergey Levine, and Amy Zhang. Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Re- arrangement. CoRR, abs/2303.1 11377,, 2023a.
Michael Chang, Thomas L. Griffiths, and Sergey Levine. Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation. CoRR, abs/2207.0077/ 2023b.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: UNiversal Image-TExt Representation Learning. CoRR, abs 115//199..17740, 2020.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua. Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Sta- tistical Machine Translation. CoRR, abs/ 1408.1078, 2014.
Bhishma Dedhia, Michael Chang, Jake Snell, Thomas L. Griffiths, and Niraj K. Tha. Im-Promptu: In- Context Composition from Image Prompts. In Thirby-seventh Conference on Neural Information Process- ing Systems, 2023.
Karan Desai and Justin Johnson. Virtex: Learning Visual Representations from Textual Amnotatios.. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognation, pp. 11162-11173, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. CoRR, hts//2000119229 2021.
Tianqi Du, Yifei Wang, and Yisen Wang. On the Role of Discrete Tokenization in Visual Representation Learning. In The Twelfth International Conference on Learning Representations, 2024.
Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew S. Rosen, Gerbrand Ceder, Kristin Persson, and Anubhav Jain. Structured Information Extraction from Complex Scientific Text with Fine-Tuned Large Language Models. CoRR, abs/2212.05238, 2022.
Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf. +: Towards End-to-End Object-Centric Learning from Real-World Videos. CoRR, Abs/2226.07766, 2022.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations. CoRR, abs/ abs/1197.13052, 2019.
19

---

Published in Transactions on Machine Learning Research (05/2025)
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS-VE V2: Inferring Unordered Object Rep- resentations without Iterative Refinement. CoRR, abs/2104409958,, 2021.
Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes Challenge: A Retrospective. International Journal of Computer Vision, 111(1):98 136, 2015.
Zhe Gan, Yen- Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale Adversarial Training for Vision-andd-anguaae Representation Learning. Aduances in Neural Information Processing Systems, 33:6616 6628, 2020.
Ross Girshick. Fast R-CNN. CoRR, abs/1504.08088, 2015.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-Object Representation Learning with Iter- ative Variational Inference. CoRR, abs/ 1903.0003, 2019.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the Binding Problem in Artificial Neural Networks. CoRR, als/2001.05508,, 2020.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragssam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh Ti Liu, Hemning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mchdi S. M. Sajjadi, Matan Scla, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: Scalable Datasct Generator. CoRR, abs/2203.0368 0335700 2022.
Thomas L. Griffiths, Nick Chater, Charles Kemp, Amy Perfors, and Joshua B. Tenenbaum. Probabilistic Models of Cognition: Exploring Representations and Inductive Biases. Trends in Cognâtive Sciences, 14 (9):357-36, Aug. 2010.
Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem. Contrastive Learning for Weakly Supervised Phrase Grounding. CoRR, abs/2006. /2006.09920, 2020.
Stevan Harnad. The Symbol Grounding Problem. Plysico D: Nonlinear Phenomena, 42(1 3), June 1990.
John Haugeland. Artificial Intelligence: The Very Idea. MIT Press, Cambridge, 1985.
Allan Jabri, Sjoerd van Steenkiste, Emiel Hoogeboom, Mehdi S. M. Sajjadi, and Thomas Kipf. DORSal: Diffusion for Object-centric Representations of Scenes et al. CoRR, abs/2306.08065 08068, 2023.
Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-Centric Slot Diffusion. CoRR, abs/2333.10884 2023.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. CoRR, abs/ abs/1612.06890 2016.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zit- nick, and Ross Girshick. Inferring and Executing Programs for Visual Reasoning, In Proceedings of the IEEE International Conference on Compuler Vision, pp. 2989 2999, 2017.
Aishwarya Kamath, Mannat. Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. ADETR-modulated Detection for End-to-end Multi-modal Understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1780 1790, 2021.
Laurynas Karazija, Iro Laina, and Christian Rupprecht. ClevrTex: A Texturo-Rcch Benchmark for Unsu- pervised Multi-Object Segmentation. CoRR, abs/2111.10265, 2021.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Andrej Karpathy and Li Fei-Fei. Deep Visual-Semantic Alignments for Generating Image Descriptions. CoRR, abs/1412.2306, 2015.
Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. CoRR, abs/2111.12590 2021.
Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Mengshu Sun, Wei Niu, Xuan Shen, Gong Yuan, Bin Ren, Minghai Qin, Hao Tang, and Yanzhi Wang. SPVIT: Enabling Faster Vision Transformers via Soft Token Pruning. CoRR, hts/2112.13890, 2022.
Avinash Kori, Francesco Locatello, Fabio De Sousa Ribeiro, Francesca Toni, and Ben Glocker Grounded 2024a.
Object Centric Learning. CoRR,
Avinash Kori, Francesco Locatello, Ainkaran Santhirasekaram, Francesca Toni, Ben Glocker, and Fabio De Sousa Ribeiro. Identifiable Ubject-Centric Representation Learning via Probabilistic Slot Attention. CoRR, abs/2406.07144 2024b.
Harold W. Kuhn. The Hungarian Method for the Assignment Problem. Noval Research Logistics (NRL), 52, 1955.
Brenden M. Lake and Marco Baroni. Human-like Systematic Generalization through Meta-Learning Neural Network. Nature, 623(7985):115--2, 2023.
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building Machines that Learn and Think Like People. CoRR, abs/1604.00283. 2016.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Chang. Visualbert: A Simple and Performant Baseline for Vision and Language. arXiv preprint Xiv::1998. 03557, 2019.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Dong, Furu Wei, et al. Oscar: Object-semantics Aligned Pre-training for Vision-languag Tasks. In Proceedings of the 16th European Conference on Computer Vision, Part XXX 16, pp. 121-137. Springer, 2020.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongic, Lubomir Bourdev, Ross Girshick, James Hays, Pictro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. CoRR, abs/ hts/1405.03122 2015.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with Grounded Pre-Training 2024.
for Open-Set Object Detection. CoRR, abs/2303.06
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-Centric Learning with Slot Attention. CoRR, abs abss2200.15055, 2020.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining Task-agnostic Visiolinguistic Representations for Vision-and Viston-and-language Tasks, Advances in Neural Information Processing Systems, 32, 2019.
Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models. CoRR, abs/ Abs/2404.13013 2024.
William Peebles and Saining Xie. Scalable Diffusion Models with Transformers. CoRR, abs 2bs/2222.09748, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. CoRR, abs/2103.00020, 2021.
21

---

Published in Transactions on Machine Learning Research (05/2025)
Tomer Ronen, Omer Levy, and Avram Golbert. Vision Transformers with Mixed- Tokenization. CoRR, abs//334.002877, 2023.
Marshall H. Segall, Donald T. Campbell, and Melville Herskovits. The Influence of Culture on Visual Perception. Bobbs-Merrill Indianapolis, 1966.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk,, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong Hc, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the Gap to Real-World Object-Centric Learning. CoRR,
2023.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-E Learns to Compose. CoRR, abs/2110..110 2021.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos. CoRR, abss2225.14000 14065, 2022.
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations. CoRR, abs/2011.12854 2020.
Aleksandar Stanié, Anand Gopalakrishnan, Kazuki Trie, and Jürgen Schmidhuber. Contrastive Training of Complex- Complex-Valued Autoencoders for Object Discovery. CoRR, abs/2305.15001, 2023.
Quan Tang, Bowen Zhang, Jiajun Liu, Fagui Liu, and Yifan Liu. Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation. CoRR, abs/2308.01045, 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. CoRR, abs/1807 11077033778 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9(86):2579 2509-260,, 2008.
Josiah Wang and Lucia Specia. Phrase Localization Without Paired Training Examples. abs/1908.07553, 2019.
CoRR,
Liwei Wang, Jing Huang, Yin Li, Kun Xu, Zhengyuan Yang, and Dong Yu. Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 140000, 2021a.
Ruocheng Wang, Jiayuan Mao, Samuel J. Gershman, and Jiajun Wu. Language-Mediated Object-Centric Representation Learning. CoRR, abs/ hbs/2012.15814, 2021b.
Jonathan Winawer, Nathan Witchoft, Michael C. Frank, Lisa Wu, Alex R. Wade, and Lera Boroditsky. Russian Blues Reveal Effects of Language on Color Discrimination. Proceedings of the National Academy of Sciences, 777(19):770--7785 2007.
Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Flomer Unsupervised Visual Dynamics Simulation with Object-Centric Models. CoRR, abs/2210.05861,, 2023a.
Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. SlotDiffusion: Object-Centrie Generative Modeling with Diffusion Models. CoRR, has/2305.1128/ 2023b.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. GiTupViT: Semantic Segmentation Emerges from Text Supervision. CoRR, abs/2202.11094, 2022.
Raymond A. Yeh, Minh N. Do, and Alexander G. Schwing, Unsupervised Textual Grounding: Linking
Words to Image Concepts. CoRR, Abs/1803.11188 2018.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmect Kohli, and Josh Tenenbaum. Neural- symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. Advances in Neural Information Processing Systems, 31, 2018.
22

---

Published in Transactions on Machine Learning Research (05/2025)
Yan Zhang, David W. Zhang, Simon Lacoste-Julien, Gertjan J. Burghouts, and Cees G. M. Snock. Unlocking Slot Attention by Changing Optimal Transport Costs. CoRR, abs/2301.13119) 2023.
Klaus Zuberbühler. Syntax and Compositionality in Animal Communication. Philosophical Transactions of the Royal Society B: Biological Sciences, 375(1789), November 2019.
23

---

Appendix
Data
24
Nested Schema Space Description
24
Dataset Splits
24
Schema examples
24
Neural Slot Interpreter
30
Hyperparameters
30
Ungrounded and HMC Matching Backbone Hyperparameters NST Hyperparameters
30
30
Experiments
32
Grounded Compositional Semantics
32
Top Learned Slots
32
Top Learned Primitives
32
Searching over Slots
32
Slot Sweep for COCO Retrieval Task
33
Retrieval with underspecified annotations Additional Qualitative Results
35
36
Retrieval results across Property Types Object Discovery
37
39
Object Discovery Results
39
Slot Sweep over COCO scenes
................................
Property Ablations on COCO scenes
................................
39
Grounding Example Ablation COCO scenes
40
Mask Visualizations
40
Grounded Slots as Visual Tokens
45
Confusion Matrices
45
Visual Rationales
45
Object Detection with NSI
50
NSI Schema Generator Model
50
NSI Schema Generator Hyperparameters Experiments
50
50
A Data
This section contains additional information on the datasets, nested schema space instantiation, and examples of the annotation organization.
A.1
Nested Schema Space Description
Table lists the various object properties used to create scherna description of scenes.
A.2
Dataset Splits
The datasct splits used in this work are detailed in Table 3.
A.3
Schema Examples
Figs. 10-13 show schema descriptions of instances from all the datasets.
24

---

Published in Transactions on Machine Learning Research (05/2025)
Dataset
Property
Discrete Continuous
Size
Material
Discrete
Color Shape
Discrete
CLEVr Hans (Johnson et al., 2016; Stammer et al., 2020)
Discrete
Size
Discrete
Object Position
Continuous
Texture
Discrete
Shape
Discrete
CLEViTex (Karazija et al., 2021)
Size
Discrete
Object Position
Continuous
Object Category
Discrete
17
Object Size
Continuous
MOVi-C (Greff et al., 2022)
Object Position
Continuous
Bounding Box
Continuous
Object Category Bounding Box
Discrete
MS-COCO 2017 (Lin al., 2015)
Continuous
Table Schema space across various datasets.
Name
Train Split Size
Validation Split Size
Test. Split Size
CLEVr Hans 3
9000
2250
2250
CLEVr Hans
21000
5250
5250
CLEVVTex
37500
2500
10000
MOVi-C
198635
35053
6000
MS COCO 2017
99676
17590
4952
Table 3: Dataset splits used in experiments.
25

---

Published in Transactions on Machine Learning Research (05/2025)
<element: 0>
<size> small</size>
<shape> cube</shape>
<color> gray</colo>>
<material> metal</material]
<pos> 76, -0.79, 0.35) </pos>
< /element>
Kelement: 1>
<size> small</size>
<shape> sphere</shape>
<color> yellow</color>
<material> metal</material)
<pos> (-0.14, 1.75, 0.35) </pos>
</ </element>
<element: 2>
<size> large</size>
<shape> cube</shape>
<color> green</color>
<material> rubber</material))
<pos> (1.24, 2.12, 0. </pos>
</ /eeement>>
<element: 3>
<size> small</size>
<shape> cylinder</shape>
<color> gray</color>
<material> rubber</material
<pos> (-1.11, 1.79, 0.35) </pos>
</ </element>
< <element: 4>
<size> large</size>
<shape> cube</shape>
<color> red</color>
<material> rubber</materiat
<pos> (2.73, -2.23, 0. </pos>
</element>
Figure 10: CLEVr Hans instance with its corresponding schema description.
26

---

Published in Transactions on Machine Learning Research (05/2025)
<element: Delement:0><500 DXssiiiemmmmddiun/iiieee
<shape> sphere</shape>
<material> #
<pos> (1.58, -2.81, 0.60) </pos>
</ /eeemment>
celement: wtssssssesssemmll///is.a.
<shape> cylinder</shape>
<material> polyhaven_aeva aerial_mud__ /
<pos> (-0.23, 2494, 0.40) </pos>
</ /elemet>>
<element: :><sizaeemeduummssiieee
<shape> cylinder</shape>
<material> polyhaven_forress forrest_groune Baaes/goooonddd/ttttrrieee 01</material>>>>
<pos> (2.67, 2.78, 0.60) </pos>
</ /element>
e Kelement: 3><Gizee>medimmk/sizez>
<shape> monkey</shape>
<material> polyhaven cracked www.erteeenlll//eeeeeoneeeee
polyhaven_craaked
<pos> (-0.63, 1.98, 0.60) </pos>
</ </element>
<shape> cube</shape>
<material> polyhaven_brite brick wwwcc_cralloOO0l0 wtw._a005///maeerraaaaa
<pos> (-2.81, 0.52, 0.42) </pos> </ </element>
<element: bxsii=>>rrarree//iiii>>>>>>
<shape> sphere</shape>
<material> polyhaven_ laaprree__rrreeree___ee trerttlleeells/tttrrraalll
<pos> (-2.66, 2.94, 0.90) </pos>
</ </elemen>>
<element: : 6><sisessmmmlll//sseee
<shape> cylinder</shape>
<material> Setihavven_lavees.ee foreet_grrommkmtttrii
<pos> (1.12, 2.49, 0.40) </pos> </ /element>>
2allament:<Pxssizzeeemllllse
<shape> monkey</shape>
<material> polyhaven
polyhaven_aerial aerial aerial_rocks__ee
<pos> (1.98, 0.84, 0.40) </pos>
</ /leement>
<el telement: s>siizemmmdimmm/siizzz
<shape> cube</shape>
<materia>> Leeeavenwwodd_laeee pllankssgrey//mttrraaalll
<pos> (0.78, -1.23, 0.42) </pos>
</element>
Figure 11: CLEVrTex instance with its corresponding schema description.
27

---

Published in Transactions on Machine Learning Research (05/2025)
<element: orlleet:t0kkateerryyh/////terr
<scale> 1.93 </scale>
<position> (0. 68, 0. 0.80) </position>
<bbox> (0.70, 52, 0. 98, 0.84) </bbor>
< </element>
<element: Decterrrrraaeeerrroouuuurrrrr Goods</category>
<scale> 2.25 </scale>
<position> (8,28, 0.28) </position>
<bbox> (0.18, 0. 14, 0.39, 0.42) </bbox>
</
/leement>>>
<element 2XXcategory/Noeee//ttggg???
<scale> 1.98 </scale>
<position> (0.67, 0.1 </position>
<bbox> (0.02, 0.60, 0.29, 0.77) </bbor>
</ ellement>>
<element oxcaeee:teoccateerrrrroaaaaaa Goods</category>
<scale> 1.97 </scale>
<position> (0. 50, 0.20) </position> <bbox> (0.09, 0.41, 0.32, 0.58) </bbox>
< </element>
<element: /tter:/seerccerrrrr////////tgr
<scale> 1.45 </scale>
<position> (0, 7.75) </position> <bbox> (0.63, 0.27, 0.86, 0.39) </bbox>
</
</eleeent>
F:leuennt:bxccattooryllllll Cases</category> <scale> 0.80 </scale>
<position> 20,20, 0.25) </position> <bbox> (0.21, 0.15, 0.29, 0.26) </bbox>
< /lemmett>
<element : SxcttpprrrNNNNn///ttttgg
<scale> 1.02 </scale> <position> (0.62) 62, 0.33) </position>
<bbox> (0.28, 0. 0.54, 0.38, 0.66) </bbox>
</element>
Figure 12: MOVi-C instance with its corresponding schema description.
28

---

Published in Transactions on Machine Learning Research (05/2025)
<element:0>
<cat>tv<ccat>
<bbox> (0.34, 0.26, 0. 0.18, 0. </bbox>
</element>
<element:1>
<cat>chair</cat>
<bbox> (0.03, 0.32, 0.27, 0.33) </bbor>
</element>
<element:2>
<cat>book</cat>
<bbox> (0.70, 0.67, 0.30, 0.24) </bbox>
</element>
<element:3>
<cat>vase</cat>
<bbox> (0.59, 0.46, 0. 0.10, 0.32) </bbox>
</element>
<element :4>
<cat>charr</cat>
<bbox> (0.72, 0.27, 0.08, 0.22) </bbox>
</e </eleentt>
<element:5>
<cat>dining cable</cat>
<bbox> (0.79, 0.32, 0.07, 0.22) </bbox>
</ </element>
<element:6>
<cat>remtte<<cat>
<bbox> (0.34, 0.62, 0.06, 0.04) </bbox>
</elemett>
<element:7>
<cat>book</cat>
<bbox> (0.71, 0. 0.77, 0.07, 0.09) </bbox>
</element>
<element:8>
<cat>chair</cat>
<bbox> (0.80, 0.28, 0.09, 0.21) </bbox>
</element>
Figure 13: COCO instance with ils corresponding schema description.
29

---

Published in Transactions on Machine Learning Research (05/2025)
B Neural Slot Interpreter
Next, present details of NST training modules and summarize the NST pseudocode.
C
Hyperparameters
We list the hyperparameters for NST and other methods used in experiments, which were all performed Nvidia A100 GPUs.
C.1
Ungrounded and HMC Matching Backbone Hyperparameters
The hyperparameterss for ungrounded and HMC matching backbones are given in Table
Module
Hyperparameters
CLEVr Hans
CLEViTex
MOVi-C
MS COCO 2017
Image Size
224
224
224
224
Patch Size
8
8
Num. Patches
784
784
784
784
DINO Backbone
Num. Layers
8
8
Num. Heads
8
8
Hidden Dims.
192
192
192
192
Num. Slots
10
10
10
93
Slot Attention
Iterations
3
3
3
3
Hidden Dims.
192
192
192
192
Num. MLP Layers
3
3
Broadcast Decoder
MLP Hidden Dims.
1024
1024
1024
1024
Output Dims.
785
785
785
785
MLP Hidden Layers
2
2
Prediction Head
MLP Hidden Dims.
64
64
64
64
Output Size
18
71
25
95
Batch Size
128
128
128
128
LR Warmup steps
10000
10000
10000
30000
Peak LRR
10-4
x 10-4
10-4
1 x 10-4
Training Sctup
Dropout
0.1
0.1
0.1
0.1
Gradient Clipping
1.0
1.0
1.0
1.0
Inference Configuration
Num. Slots
10
10
10
30
GPU Usage
40 GB
40 GB
40 GB
40 GB
Training Cost
Days
1
3
Num Layers
2
VIT
Hidden Dims
64
Num Heads
4
Table 4: Hyperparameters for the ungrounded and HMC matching method used in our experiments. In the ungrounded case, the backbone and slot attention modules are trained solely on the reconstruction objective and frozen.
C.2 NSI Hyperparameters
The hyperparameters for the NSI alignment model are listed in Table 5. The ablated architectures follow the same setup without the ablated module.
30

---

Published in Transactions on Machine Learning Research (05/2025)
Module
Hyperparameters
CLEVr Hans
CLEViTex
MOVi-C
MS COCO 2017
Image Size
224
224
224
224
Patch Size
8
8
8
8
Num. Patches
784
784
784
784
DINO Backbone
Num. Layers
8
8
Num. Heads
Hidden Dims.
192
192
192
192
Num. MLP Layers
3
3
3
3
Broadcast Decoder
MLP Hidden Dims.
1024
1024
1024
1024
Output Dims.
785
785
785
785
Num. Slots
10
10
10
15
Slot Attention
Iterations
3
3
3
3
Hidden Dims.
192
192
192
192
Num. Layers
8
8
Num. Hcads
8
8
Schemna Encoder
Hidden Dims.
192
192
192
192
Max. Schema Len.
10
10
10
93
Embedding Dims.
64
64
64
64
Projection Heads
MLP Hidden Layers
2
2
2
2
MLP Hidden Dims.
256
256
256
256
Batch Size
128
128
128
128
LR Warmup steps
10000
10000
10000
30000
Peak LR
x 10
4 10
x 10
1 10-1
Training Setup
Dropout
0.1
0.1
0.1
0.1
Gradient Clipping
1.0
1.0
1.0
1.0
31,32
0.5, 0.5
0.5, 0.5
0.5, 0.5
0.5, 0.5
GPU Usage
40GB
40 GB
40 GB 3
40 GB
Training Cost
Days
1
3
Num Layers
2
VIT
Hidden Dims
64
Num Heads
4
Table 5: Hyperparameters for the NSI alignment model instantiation and training setup.
31

---

Published in Transactions on Machine Learning Research (05/2025)
D Experiments
D.1 Grounded Compositional Semantics
D.1.1 Top Learned Slots
Figs. 14-16 show the top 50 slots learned by the image encoder over each dataset. Each slot is weighted by the la norm of its embeddings Y The top slots follow an interesting distribution. On CLEViTex (Fig. 14), slots with larger objects and a clean segmennatiio are assigned higher magnitude. Inluitively, these slots are the most discriminative when assigning primitives that contain shape, texture, and size, and benefit from large, cleanly segmented objects. On the other hand, the emergent distribution in MOVi-C (Fig. 15) and COCO (Fig. 16) weights edge artifacts more. We posit that since the object annotations of these datasets contain object positions the image, edge objects tend to be more discriminative.
Figure 14: Top 50 slots (left to right, top to bottom) ranked by magnitude from test split of the CLEVrTex dataset.
D.1.2
Top Learned Primitives
Figs. 17-19 show the (-SNE visualization (van der Maaten & Hinton, 2008) of representations learned by the schema encoder on the schema primitives. each plot, primitives in the test split are encoded context-free frree. The learned embeddings exhibit clustering effects on property categories.
D.1.3 Searching over Slots
We found that the NSI metric, which learns to match entire schemas to entire images, can also be used zero-shot to reliably retrieve individual slots from primitives. We demonstrate two instances of slot search in Figs. 20 and 21. query the form of single primitive is embedded using the schema encoder. The query embedding ranks the slot of database formed from the test split. In Figs. 20 and 21, we show the property search and position search results, respectively.
32

---

Published in Transactions on Machine Learning Research (05/2025)
Figure 15: Top 50 slots (left to right, top bottom) ranked by magnitude from test split of the MOVi-C dataset.
Figure 16: Top 50 slots (left to right, top to bottom) ranked by magnitude from test split of the COCO dataset.
D.1.4 Slot Sweep for COCO Retrieval Task
Fig. 22 shows the effect of the number of slots on the recall rates, Recall highest at 15-20 slots for COCO scenes and the model overfits the number of slots increases further.
33

---

Published in Transactions on Machine Learning Research (05/2025)
cube
cylinder
sphere
monkey
medium lange
t-SWE
K-SNE
1-SNE
(a) t-SNE scatter plot. labeled with object
t.SNE
shapes.
(b) t-SNE scatter plot labeled with object sizes.
yhaver wood floce deck
polyhaven_
poyhaven careee_carra__o wond yolyhvven plankflooring 022 polyhaven
payhaver jeree_deee__arrre
polyhaven aeral rockz
2-N 2
polyhaven
eayhaver_ro____ 07
2phaven_arphait.02
D-SNE
(c) t-SNE scatter plot labeled with object materials.
Figure 17: t-SNE scatter plots of the top 10000 CLEVrTex primitives weighted by the l2 norm of their embeddings. The embeddings are clcarly clustered by the shape type. Interestingly, when looking at the object size, only large and medium-sized objects are represented in the top primitives.
Action Figures
Bag
Board Games
Bottles and Cans and Cups
Camera
Car Seat
Goods
t-SNE 2
Headphones
Keyboard
Legos
Media Cases
Mouse
None
Shoe
Stuffed Toys
Toys
t-SNE 1
Figure 18: t-SNE scatter plots of the top 10000 MOVi-C primitives weighted by the la norm of their embeddings. The scaller points are labeled with the object categories. While there are multiple local clusters, a larger clustering effect based on object categories are not apparent.
34

---

Published in Transactions on Machine Learning Research (05/2025)
person
chair
tv
bowl
bus
t-SNE 2
pizza
train
dog
couch
truck
bed
horse
motorcycle
sink
clock
giraffe
t-SNE
Figure 19: t-SNE scatter plots of the top 10000 COCO primitives weighted by the norm of their em- beddings. The scatter points are labeled with the object categorios. A category-based clustering pattern is emergent.
cyllider
5phere
monkey
whitemarble
stone wall
cracked concrete, wall
leather red 02
slab tiles
wood Nloor deck
coral, fort wall 02 large sandstone blocks
Figure 20: Searching slots with property queries. The object material. The position is fixed at [0, 0, 0] and the size is set to 'Large' The alignment model reliably retrieves slots with desired object properties if they exist in the database.
represent. shape and the columns represent
D.1.5 Retrieval with Underspecified Annotations
We investigated the efficacy of grounding under the selting where the annolations of scenes were under- specified. Figs. 23(a) and (b) show the retrieval results under two settings: (a) maximum of ten objects annotated scene and (b) a maximum of five objects annotated in scene. First, we observed that the drop in recall rates for the ResNet backbone was significant compared to the full setting. Second, CLIP and NSI were resilient to the limited annotation, with NSI outperforming other methods. CLIP benefited
35

---

Published in Transactions on Machine Learning Research (05/2025)
Pos
Pos-X
Pos-X
Pos-X
(a) Position search cubes.
(b) Position search spheres.
(c) Position search cylinders.
Figure 21: Searching over slots with position queries. We adjust the 'Pos-'' for each shape on the z-axis and Pos- Y on the y-axis. The top-ranked slot clearly reflects the position adjustment.
Property Retrieval
Scene Retrieval
FornAgg
80
60
Percent.
40
20
0
20
25
30
5
10
25
30
Number Slots
Number Slots
Figure 22: Ablation the number of slots for icene-property retrieval task. The standard deviation over five seeds was 0.3 across datascts.
from the vast training corpora that helped it generalize the underspecifiaation On the other hand, the compositional grounding with strong backbones endows NSI with strong retrieval despite being trained limited data.
MS- AS-CCCC retrieval with ten annotations
(b)
MS-COCO Retrieval with five annotations Property Retrieval
Property
Retrieval
Scane Retrieval
Becollg
Kersiog
Reeniigi
Weraligs
60
Porcant.
20
0
HMC
Resnel
GLIP
HAGG
NS
Figure 23: Retrieval results on underspecified scene annotations settings.
D.1.6 Additional Qualitative Results
Fig. 24 shows additional results the associations inferred by NSI MOVi-G and MS COCO 2017.
36

---

Published in Transactions on Machine Learning Research (05/2025)
D.1.7 Retrieval Results across Property Types
We trained and compared the MOVi-C retrieval via. NSI and BBox QKV two separate tasks where schemas only contain non-positional object properties (category, size) and one where schemas only contain object positions (center coordinates, bounding boxes). The results are shown in Tab. 6. First, we found that retrieval results across both models and both instances are worse off compared to the completely populated schemas. However, we observe that BBOx QKV relies on positional properties more than non- positional semantics for retrieval and even outperforms NSI on the former type. On the other hand, NSI depends more on non-positional properties and scores significantly higher on retrieving them.
Model NST
R5cal1@5 (%) Non Positional Properties Recal1@5 (%) Positional Properties
58.51
43.73
BBox QKV
38.25
45.53
Table 6: MOVi-C scene retrieval results on schemas populated by only specific object property types
37

---

Published in Transactions on Machine Learning Research (05/2025)
rxcales 2.04 </aclles//teeett
belowest: www.eeeeeeeeeeeeeooooooooooo
recaden 2. Ctt//odee///aeeee
rolament www.eeeerroooooorr.....o
casass> 2.36 cresaledd alseon
celesset: Boocchagingn Easeaner CeddacCatappny
acalend Pericaae c/atamsnts
www.a//gggogyaae/e//eeegrgg acalled www.///aeeeee/eeeeeeeeaa
Lement 2weratiireeeee//eeei hwtpe//wwwweeeeeeeeeeeeeeeeeeeee /twe////nenaaeeetttt
lamestt: Exccatiaaa//eeee///eeeeeea
Endeat Ligeneci ea enrti tnteemeeee
ralanase Excracrepooo cartii relenaa
racenest accaabeectiee the eelemess
na Halnnss aoccatooappc cas
Valament: www.ataaaaaeeeeoooc.o... 22 DaCC- /2lemenns
Alament:
Figure 24: Correspondences inferred by NSI on MOVi-C and COCO scenes.
38

---

Published in Transactions on Machine Learning Research (05/2025)
D.2
Object Discovery
D.2.1
Object Discovery Results
Datasct
Metric
Ungrounded
HMC Matching
NSI
FG-ARI
87.79 t 0.12
88.37 + 0.12
89.89 + 0.01
CLEViTex
mBO
44.86 + 0.04
45.23 + 0.23
46.60 + 0.02
FG-ARI
65.53 + 0.15
65.61 = 0.31
66.41 + 0.12
MOVi-C
mBO
36.79 + 0.03
36.79 = 0.41
38.52 + 0.23
FG-ARI
40.12 + 0.29
32.18 + 0.45
44.24 + 0.27
COCO
mBO'
27.20 + 0.31
18.32 + 0.51
28.12 0.25
mBO
26.54 + 0.25
19.61 = 0.82
32.10 0.31
FG-ARI
20.42 0.13
15.14 + 0.09
21.97 0.17
PASCAL VOC 2012 (Zero-Shot)
mBO
35.97 +: 0.15
25.15 + 0.08
36.98 0.19
mBO
37.94 + 0.17
27.02 + 0.12
39.06 + 0.22
Table 7: Object discovery results, We use the DINO backbone and an MLP decoder across methods. The standard deviation was calculated over five random sceds.
Table 7 contains the complete set of object discovery results with the standard deviations. We also conducted zero-shot evaluation of the models trained on MS-COCO on Pascal VOC 2012 (Everingham et al., 2015). We observed that the benefits of grounding the model via NSI on one real-world dataset were transferred to the other for object discovery, with the grounded model also improving mask segmentation over its ungrounedd counterparts for Pascal VOC.
D.2.2 Slot Sweep over COCO Scenes
Fig. 25 shows the NSI object discovery results on COCO scenes as the mumber of slots is increased. We observe that segmentation improves until 15 slots and then slowly tapers off.
Numberof
15
20
30
Score
FG-ARI
m8o
mB0
Figure 25: Ablation on the number of slots for COCO object discovery using NSI.
D.2.3 Property Ablations on COCO Scenes
In Fig. 26, we ablate the propertics used form schema primitives. Ostensibly, the slots weigh bounding box coordinates more than categories, the performance drop is steeper when the former is ablated opposed to the meager loss when the latter is ablated.
39

---

Published in Transactions on Machine Learning Research (05/2025)
Score
22
w/o category
w/o bounding box
Figure 26: Ablation on the schema properties for NSI.
D.2.4 Grounding Examples Ablation on COCO Scenes
Fig. 27 shows the results ablating over the mumber of grounding examples used for NSI co-training. We observe that the ART metric and the class-wise mBO are sensitive to grounding, with as few as 100 annotations improving object discovery via segmentation masks.
42.5
mB0*
3.12 %
37.5
Score
32.5
1.76%
27.5
Ungrounded
100
1000
Full
Number of grounding examples
Figure 27: Ablation on number of grounding examples used to train NSI MS COCO.
D.2.5 Mask Visualizations
Figs. 28, 29, 30, 31 visualize slot masks over scenes from CLEVETex, MOVi-C, COCO, and Pascal VOC, respectively.
40

---

Published in Transactions on Machine Learning Research (05/2025)
Original
Ground Truth
Ungrounded
HMC Matching
NSI
Figure 28: Object discovery results on CLEVrTex scenes.
41

---

Published in Transactions on Machine Learning Research (05/2025)
Original
Ground Truth
Ungrounded
HMC Matching
NSI
Figure 29: Object discovery results on MOVi-C scenes.
42

---

Published in Transactions on Machine Learning Research (05/2025)
Original
Ground Truth
Ungrounded
HMC Matching
NSI
Figure 30: Object discovery results on COCO scenes.
43

---

Published in Transactions on Machine Learning Research (05/2025)
Original
Ground Truth
Ungrounded
HMC Matching
NSI
Figure 31: Zero-shot object discovery results on Pascal VOC 2012 scenes.
44

---

Published in Transactions on Machine Learning Research (05/2025)
D.3 Grounded Slots as Visual Tokens
D.3.1 Confusion Matrices
Figs. 32 and 33 show the classification confusion matrices for the CLEVi-Hans 3 and CLEVi-Hans 7 tasks. In the few-shot setting, the model often predicts scenes into under-specified and less discriminative classes, like "cyan object in front of two red objects." Similarly, it also tends to mispredict into classes containing large objects like "large cube and large cylinder" that contain reasoning over larger objects and are easier to tokenize.
Predicted Label
Predicted Label
Predicted Label
2
True Label
2
0.25
2
0.00
0.31
S
0.22
0.07
0.24
0.00
2.02
6.00
0.00
0.00
0.33
100 shot
1000 shot
Full
Accuracy: 54.53 score:
Accuracy: 74.44 F1 score: 74.14
Accuracy: 94.04 score: 94.05
Figure 32: Confusion matrices for NSI prediction on the CLEVr-Hans 3 task.
Label
Predided Labe
Prediceed Label
5
CUB
2.00
2.00
2.00
XXS
33
641
640
0.6
0.0
0.6
500
3.0
33
23
513
3.0
33
3M
640
641
6.6
036
0.6
Trus Labal
600
60L
630
330
23
230
641
646
04
0.
6000
6/6
6:6
636
230
23
230
641
640
6/6
6.6
A/S
6MM
6/6
3.6
5
Aspumnyy3300
P1 sppr: 25.15
I11 accee 50.45
Accusnry
Figure 33: Confusion matrices for NST prediction on the CLEVi-Hans 7 task.
D.3.2 Visual Rationales
Fig. 34 simultancously visualizes the rationales across data scttings and attention layers. On an average, we observe that rationales tend to get stronger and more accurate as the number of training examples increases and the ViT depth increases.
Figs. 35 and 36 demonstrate rationales across the grounding continuum for Hans 3 and Hans classification tasks, respectively.
Fig. 37 demonstratos attention maps where the model prediction is correct, but the rationale is not entiroly disposilive.
45

---

Published in Transactions on Machine Learning Research (05/2025)
100 shot
1000 shot
100 shot
1000 shot
Leyer
Layer
Layer
"Large Cube and Large Cylinder"
"Small Metal Cube and Small Metal Sphere"
100 shot
1000 shot
Full
300 shot
1000
Layer
Layer 2
"Large Blue Sphere and Small Yellow Sphere"
"Large Blue Sphere and Small Yellow Sphere"
100 shot
1000 shot
Full
100 shot
1000 shot
Tull
Layer
Layer
Layer 2
Loyer
"Cyan Object in front Two Red Objects"
"Three Spheres on the Left Side"
Layer 1
Layer
Layeer
Layer
"Large Blue Sphere and Small Yellow Sphere"
"Small Metal Cube and Small Metal Sphere"
46
Figure NST visual rationales across different data regimes and attention depth.

---

Published in Transactions on Machine Learning Research (05/2025)
Scene
Ungrounded
NSI-CLEV/Tex
NSI-CLEVr
Small Metal Cube and Small Sphere
Large Cube and Large Cylinder
Large Blue Sphere and Small Yellow Sphere
Figure 35: Visual rationales on Hans 3 across different methods.
47

---

Published in Transactions on Machine Learning Research (05/2025)
Scene
Ungrounded
NSI-CLEVrTex
NSI-CLEVr
Three Spheres on Left Side
Three Spheres Left Side
and Three Metal Cylinders
Right Side
Cyan Object in Front of Two Red Objects
Three Metal Cylinders Right Side
Figure 36: Visual rationales on Hans 7 across different methods.
48

---

Published in Transactions on Machine Learning Research (05/2025)
Scene
Ungrounded
NSI-CLEV/Tex
NSI-CLEV
Small Metal Cube and Small Sphere
Large Blue Sphere Small Sphere
Three Spheres Left Side or Three Spheres on Left Side
and Three Metal Cylinders
on Right Side
Cyan Object In Front of Two Red Objects
Three Metal Cylinders Right Side
Figure 37: Some failure cases where NSI makes the correct prediction, albeit using incorrect or ambiguous support slots.
49

---

Published in Transactions on Machine Learning Research (05/2025)
E Object Detection with NSI
In this section, demonstrate architecture that uses inferred correspondences by NST to perform object detection. We run preliminary experiments and explore the use of slots in real-world visual reasoning systems.
E.1
NSI Schema Generator Model
We formulate the NSI schema generator (see Fig. 38) predict, from each slot, its corresponding schema primitives. To this end, modify an encoder Transformer by interleaving cross-attention blocks with the sclf-attention layers to assimilate the context from the encoded slots. The input to the model is simply learned positional embeddings pl:N that are decoded by L attention-ensemble slacks to oulput primitive representations. We call this architecture SET,(.). The Slot Encoder Transformer (SET) is stack of L blocks, each computing (a) self-attention over inputs followed by (b) cross-attention of inputs over the context slot. Let primll1 be the sequence representation at layer 1 - and S denote the slot. Then
SET&(PLNN ,S)
(18)
The cross-attention implementation of block / is shown in Algorithm 2.
Algorithm
for Block of SET
Require: z E RRxx primitive embeddings from self-attention of layer
Require: Se Rd. Context Slot
Get query tokens: Q M (2de..11))
Get keys, values of slot S: K, V = MLPxv(S)
Compute attention values: M softmax(Q"
Get output: 2 M X
Using an encoder teantssrrnnooomestttytte styled predictor has two advantages: (1) primitives can be generated in parallel and (2) each primitive representation is aware the overall prediction context. MLP property heads predict the object properties from the z representations. At each training iteration, the predicted properties fl:N are optimally matched to their ground truth labels p via an ordering a(1 N) obtained from Hungarian matching (HM) (Kuhn, 1955) on the property prediction loss Eproperties- Note that Epropertios is per-primitive loss obtained from the sum of individual property prediction losses. We use the cross- entropy loss for discrete properties, mean-squared error for contimuous properties, and augment bounding-box regression with the Intersection over Union (IoU) loss. The training objective Lgen (.) is formulated as follows:
o N) HM (Eproperties IN pliN))
(19)
ggn Σ Eproperties (o..P(()))
(20)
At training time, pad the aligned ground-truth instances with no-object labels + to account for rep- resentations without object predictions. In practice, bipartite matching for single-slot instance is more computationally feasible than the overall image instance because per-slot object instances are significantly fewer.
E.2
NSI Schema Generator Hyperparameters
The hyperparameters for the NSI schema generator are listed in Table 8.
E.3
Experiments
The correspondences from the train split are used learn the NSI schema generator, as outlined in Ap- pendix F.1. The schema generator decodes T primitives from each slot, including the confidence level of
50

---

Published in Transactions on Machine Learning Research (05/2025)
aa) Learned Correspondences
(b) Schema Generator
<cat>person<cea>> </bbox>>
<oat>person</cat> Pi <bbox> </bbox>
<cat>donut<cctt> P1
$ Ps
<bbox>... </bbox>
<cat>domt<<<eat> <bbox> ... </bbox>>
yen
<cat>person<cca>> <bbox>... <bboxo...//bbo <//bbox>>
P1
P2
Prediction Heads
<cat>person</cat cat> <bbox>.... .//bbon
Z prim,
2 prém
2
<oat>book<cat>>
<bbox> <bbbo>..<//
S
Slot Encoder Trans former
<cathclock</cat>
tbboxo..</bboxx
<cat>couch</eat>
pl
p
p
<bbox>..</bbbr>
Figure 38: NSI schema generator model: (a) Learned correspondences between slot and schema primitives are used to Irain the schema generator. (b) A Slot Encoder Transformer attends to individual slots via cross- attention and, in parallel, decodes tokens into schema primitives, At training time, a Hungarian set matching procedure assigns predictions to primitives associated with the slot. The prodiction error is aggrogated over assignments to compule Lyen.
Module
Hyperparameters
MOVi-C
COCO-10
COCO-30
Num. Layers
2
2
2
Num. Heads
2
2
2
SET
Hidden Dims.
192
192
192
Predictions per Slot (T))
5
8
8
top-M predictions
10
10
30
Inference
Non-Max Suppression/ Thhreshold
Yes (0.75)
Yes (0.75)
Yes (0.75)
Batch Size
64
64
64
LR Warmup steps Peak LR.
10000
30000
30000
Training Setup
10-4
10-4
10-4
Dropout
0.1
0.1
0.1
GPU Usage
40 GB
40 GB
40 GB
Training Cost.
Days
2
4
4
Table Hyperparameters for the NSI schema generator model instantiation and training setup.
each object prediction. The overall schema for a single image is obtained as the top M confident primitives out of predictions from all K slots of that image. The NSI schema generator enables solving of downstream tasks with slots by making use of predicted properties from primitive tags. We demonstrate the usefulness of NSI for object detection.
51

---

Published in Transactions on Machine Learning Research (05/2025)
Object Detection Results: In the context of real-world object detection images, the schema properties can be used to identily and locate objects in diverse scenes from the MOVi-C and COCO datasets. To this end, we extract the (<cat>) category and (<bbox>) bounding box fields from the generated primitives that depict the object category and location, respectively. For COCO, we tost on two variants of the dataset: (1) COCO-10, simple subset of the lest split containing len objects at a maximum,, and (2) COCO-30 that contains as many
objects scene. We report the APIoU metric across all three benchmarks. denotes the area under the precision-recal curve for a certain IoU threshold (in %). We also run comparisons against the methods outlined in the retrieval experiments. Fig. 39 shows the experimental results and Figs. 40, 41 visualize object detection across various predictors. Our comparison baselines include:
(a) Vanilla Slot Attention (Locatello al., 2020): The model is trained from scratch resolve slot- objoct assignments through HMC.
(b) DINOSAUR (Locatello et al., 2020): It uses the DINO VIT backbone to learn slots for unsupervised object discovery by reconstructing perceptual features. The architecture is frozen while we train shallow predictors on top to detect objects.
(c) DINOSAUR-FT. We use the DINOSAUR model but fine-tune the architecture end-to-end on the prediction task.
We make the following observations:
(a) NSI ourperorms prior set-matching slot predictors, especially by significant margins (20-30%) at lower IoU thresholds. Grounding object concepts in slots priorí improves the predictive power of slots. In comparison, matching the set of slots against the entire set of object annolations of the image yields poor generalization. In addition, large-scale pre-training and end-to-end fine-tuning are crucial ingredients, as evidenced by the subpar performance of the Vanilla and DINOSAUR methods on COCO.
(b) The performance disparity between NSI and DINOSAUR-FT widens as the complexity of scenes increases from COCO-10 COCO-30. The inability of DINOSAUR to predict more than one object per slot necessitates modeling and learning to match up to 30 different slots, which generalizes poorly novel scenes.
(c) HMC slots deteriorate for COCO-30 where the backbone is tasked with matching with 30 different objects at a time.
50
Vanilla DINOSAUR DINOSAUR-P7 NSI
00
AP [in %)
20
APs
Alto MOVi-C
APa
AB3 COCO-10
AP
APp
Al COCO-30
AP%
Figure 39: Object detection performance of various prediction methods the MOVi-C, COCO-10, and COCO-30 benchmarks. We report APÂIoU (higher is better) for different IoU thresholds. Standard devia- tion reported over five random seeds.
52

---

Published in Transactions on Machine Learning Research (05/2025)
Ground Truth
Vanilla
DINOSAUR
DINOSAUR-FT
NSI
dining Table
person
chair
person
person
dining table
toilet
cake
dining table
dining table
donut
didenusitt
oout
cup
person
person
person
person
Fire hydrant
person
person
person
clock
tenzis
umbr
racckkt
tennis rack ket
person
potted plant
chaje
couch
Cell g dining table dining table
teddy
sC remote
person souch
person
oven
book
book
eeddy bear
bed
pe pesufboard
surfboard
Cow
person
person
person
person
person
surfhoard
surfboard
person
perssont
cake utt
wine glass
person
person
chair
person
person
chair
pizza
cup
dining
Aenit
person
sports ball
pers
tennis racket
person
person
rennis racket
racket
son
sports ball
person ennis racket
person
tennis rackets racket
Figure 40: Object detection results on COCO scenes. NSI schema generator can flexibly detect multiple objects from the same slot, as evidenced by detections on COCO images. For example, single slot predicts multiple 'donuts,' 'chair and dining table,' and 'person and tennis racket!.
53

---

Published in Transactions on Machine Learning Research (05/2025)
Ground Truth
Vanilla
DINOSAUR
DINOSAUR-FT
Consumer Goods
Cons shoe
Sonds
None
Nome
Consumer Goods
Shoe
None
Cons
Shop
Shoe
Shoe
Nome
None
None
Cups
Shoe
Consumer Goods
Shoe ner Goods
Toys
Toys
Shoe
Bottles and Cans and Cups
Toys
Consumer GolNone
Bottles and Cans and Cups
Bottles and Cans and Cups
Consui
None
None
Figures
Figures
Action Figures
Toys
shoe
Toys
Toys Shoe
Nóme sumer Gb
Consumer Goods
Toys
Shoe
Shoe
Shoe
Toys
Shoe
Toys
Shoe
Tays
Action Figures
Toys
Action
Toys
Nome
con-sho an Glods
Co Shoehr Goods
Ehou
Shou.
Toys
Bottles and CupShole
Shoe
Toys
Media Cases
Consumer Goods
Media Cases
Figure 41: Object detection results on MOVi-C scenes.
54


Reviews:
[{"review_id": "ZjMAqxqLoB", "reviewer": "Reviewer_SoPX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper is generally well-written and the proposed method was explained clearly.\n- The combination of slot attention co-trained with object-level labels is relatively underexplored from my understanding, so this work presents findings not found in previous work.\n- The experiments are generally well-done and show the advantages of NSI, especially when compared to the previously used HMC.\n\n\n**Weaknesses / Questions:**\n\n- In Section 5.2.1, it would be informative to include the standard property prediction task results as comparison.\n- For the property retrieval task, what is the breakdown of the task by property type? For Section 5.2.2, it seems that maybe BBox QKV would perform well retrieving the location of the object, but not other properties?\n- Related to the previous question, what happens in the case where the object labels do not fully capture the objects? For example, in the CLEVR dataset, what if we only had labels for object color, but not object shape? If we train our model with these labels, do we still effectively capture shape in the slots? Since we still have the Slot Attention reconstructive objective, I imagine that shape will still be captured, but how does it compare to training without the contrastive objective? This is relevant to more complex scenes because the labels may not completely capture the objects and all properties of objects in the scene. We would ideally still want to capture concepts that are not present in the labels.\n- Does it make sense to add GroupVIT [1] as a baseline to some of the experiments? They similarly train with a contrastive objective, but use natural language instead of the structured object labels used in this work.\n- Are the results in Figure 9(d) cherry-picked? Is it possible to quantify how well the attention maps capture the contents of the task across the entire evaluation dataset? If not, I would soften the language about interpretability related to this figure.\n- I do not see the value in presenting the object annotations with XML. It seems this could be represented with any other nested structure. In fact, I feel the introduction of visXML actually adds a bit of confusion and may be more appropriate as an implementation detail in the appendix.\n\n\n[1] GroupViT: Semantic Segmentation Emerges from Text Supervision. https://arxiv.org/abs/2202.11094", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Neural Slot Interpreter (NSI), which combines Slot Attention with weak supervision in the form of object-level labels to ground object semantics in slots. The supervision is presented in an XML-like schema called visXML and the model is co-trained with the traditional Slot Attention objective and a newly introduced contrastive learning objective. In experiments, the authors show the advantages of their proposed approach when compared to traditional Slot Attention, using Hungarian Matching to ground the slots, as well as several other baselines. Specifically, NSI is shown to effectively ground object object semantics in slots, enhance object discovery, and effectively tokenize the image for the CLEVR-Hans reasoning benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Addressing the weaknesses / questions in the above section could strengthen the work, but are not critical to securing my recommendation for acceptance.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vmjnibfSkY", "reviewer": "Reviewer_MjoJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The ability to discovery and ground objects is an important direction\n- The proposed method is interesting and shown to be effective in experiments\n\n**Weaknesses**\n- It was difficult to get the main contributions of the work and how it related to prior work on slot attention and object-centric learning\n- The claims / questions in the introduction are not properly substantiated by the experiments \n- The writing and presentation of the work need to be improved considerably so that the contributions, method, and experimental setup are clear and self-contained (without the reader having to refer repeatedly to prior work) \n- Even though the work is following up on a line of work that aims to perform object discovery, it appears to require training.  While there are experiments comparing against weakly supervised visual grounding, it is not clear whether it should also compare against fully supervised visual grounding methods. \n- For experiments, it is unclear whether suitable baselines are included.  For instance, for weakly supervised visual grounding methods, perhaps it would make sense to compare against more recent methods such as [1] or zero-shot methods such as [2][3]. The reviewer feels that the direction of this work doesn’t necessarily necessitate such comparisons, but if the manuscript claims to have compared against a state-of-the-art weakly supervised visual grounding method, then it is important that it does so.\n\n[1] Weakly Supervised Open-Vocabulary Object Detection [Lin et al. 2024]\n\n[2] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection [Liu et al. 2023]\n\n[3] GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection [Shen et al. 2023]", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Neural Slot Interpreters (NSI) which builds on slot attention and DINOSAUR, which uses DINO as the vision backbone for slot attention and aims to reconstruct the features instead of the raw pixels.  NSI builds on DINOSAUR by using an annotation scheme for objects and their properties, a similarity metric to assign multiple concepts to slots (thus breaking the constraint of one-object-per-slot of prior work), and introduces contrastive learning to ground slots, as well as a hierarchical transformer-based architecture.  A series of experiments are conducted to show the effectiveness of the proposed NSI and different design choices (e.g. using Hungarian matching for slot matching, and different visual encoders).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The reviewer feels that the manuscript needs considerable amount of rewriting for clarity, especially wrt to the introduction, related work and experiments.\n\n1. The introduction needs to be reworked to be clearer with claims / hypotheses be reworked to better match what is shown by the experiments. \n       \n     a. There is a disconnect because the questions being asked and text that follows: *Can slot representations, when explicitly grounded in object semantics, serve as effective compositional abstractions for grounding and reasoning* \n         \n     - The reviewer expected this to be followed by a description of a set of experiments to answer this question, but instead there is a description of the proposed method.  A logical flow to this reviewer would be to describe the proposed method, which key challenges inspired the key modifications to the method, and then start to pose the questions which will be answered by experiments.\n\n    b. In the introduction, the manuscript poses the following questions:\n            \n      - *Do NSI modules require specialized training recipes?* \n               The reviewer is uncertain what is meant by a “specialized training recipe”.  The manuscript indicates that the “NSI is a plug-and-play paradigm”.  The reviewer does not find the question and statement to be logically connected as it is possible for a framework to be plug-and-play, but specific training recipes to be necessary for the framework to be effective (e.g. some models need to be trained before other parts, or parts need to be pretrained, or certain training strategies - optimizers, learning rate scheduler need to be applied).  Also, the reviewer believes that a specific training recipe is followed for the set of experiments provided.  \n\n      - *Are notions of objects effectively grounded in emergent slot representations?* The paper states that “NSI surpasses the state of the art” in section 5.2.1  Due to the limited discussion of prior work, it is not clear to the reviewer whether NSI indeed surpasses the state-of-the-art methods, and what the appropriate set of methods would be considered for this particular setting.  Section 5.2.1 does compare different ways of matching and different visual encoders and shows the effectiveness of the proposed approach.\n      -  *Does NSI preserve compositionality?* (Section 5.3) The reviewer does not find competitiveness in object discovery to be necessarily a sign of compositionality, as it is possible to discover objects without notions of compositionality.\n      -  *Are NSI-grounded slots effective substrates for visual reasoning?* The experiments in section 5.4 also does not demonstrate visual reasoning.  The setup of the Clevr-Hans experiments are not clearly specified, but it does not appear to follow the reasoning that is needed for the question answering that was designed for the original Clevr dataset.   Instead, the task appears to be a classification task with posthoc explanations.\n\n   c. The presentation of visXML as a contribution feels a bit odd.  While it is reasonable to have well-defined object properties, the more important part is what object properties are vs the specific format (e.g. XML) used to store the properties.  To the reviewer, it seems more appropriate to discuss the concepts vs the specifics of the XML tags. \n\n2. Related work need to have better coverage of prior work on slot attention and object centric learning\n- The relation of this work to recent works on improved slot attention for grounding and compositionality [1,2] is unclear\n- As there as been prior work that used contrastive learning for object-centric learning [3,4] and different ways to match features and slots [5] it would be good to discuss how these work relate more.\n- The relation of this work to program induction is not clear to the reviewer.  It seems unnecessary to include program induction in the related work.  The reviewer would recommend a more thorough review of prior work on using slot attention and object-centric learning.\n\n3. Improved description of the method\n\n- Overall a bit more introduction of slot attention and DINOSAUR would be helpful, and a more clear explanation of how this work builds on DINOSAUR.\n\n- Section 3.3 - this section is missing the most relevant piece of information - i.e. that HMC is typically the main method that prior work uses for matching objects to slots.\n\n4. Introduce the experiments setup for each experiment so the reader can have a clearer understanding of each task.\n\n5. Several of the figures should be improved so that the colors are more distinct and easier to interpret.\n\n   a. Figure 1 - The colors used for \"Set Match Slots\" and \"NSI Slots\" are extremely similar.  More distinct colors are recommended.\n\n   b. Figure 5 - Use more distinct colors for different methods (e.g. CLIP and NSI), with same color family for related methods (e.g. NSI variants).  \n\n   c. Figure 5,6 - Indicate in caption that higher saturation is R@1, lighter saturation is R@5.\n\n   d. Figure 5, it is difficult to tell the what material the objects are in the picture\n\n6. Some of the equations hard to follow\n\n- Eq. 1: K is used for both the number of slots and the key matrix.  \n- Eq. 1: It is not clear how the notation K(.), Q(.), V(.) should be interpreted - should K,Q,V be interpreted as matrices or functions?\n- Eq. 1: D should be specified as the dimension of the key, query, value.\n\nIn general, a brief description in text of what all the symbols are and how the equations are related would be helpful\n\n6. Writing should be improved to be more concise and clearer\n\n- \"a similarity metric that explicitly reasons\" => It not clear that a similarity metric is capable of reasoning\n\n- Section 3.1 - the terminology \"semantic object files\" is not introduced.  \n\nFor simplicity, the reviewer recommend avoiding using this terminology.  Potentially by rewording  the sentence to \n\n\"Object-centric learning frameworks decompose scenes by organizing them into semantic object files called slots\" => \"Object-centric learning frameworks decompose scenes by using \\textit{slots} to organize and represent semantic objects.\"\n\nAlternatively, the authors can properly introduce the concept of semantic object files - this is perhaps in reference to the object files introduced by Kahneman et al [6]?\n\n**References** \n\n[1] Grounded Object-Centric Learning. [Kori et al. 2024]\n\n[2] Identifiable Object-Centric Representation Learning via Probabilistic Slot Attention. [Kori et al. 2024]\n\n[3] Contrastive Training of Complex-Valued Autoencoders for Object Discovery [Stanic et al. 2023]\n\n[4] Towards Self-Supervised Learning of Global and Object-Centric Representations [Baldassarre and Azizpour 2022]\n\n[5] Unlocking Slot Attention by Changing Optimal Transport Costs [Zhang et al. 2023]\n\n[6] The reviewing of object files: Object-specific integration of information. [Kahneman et al. 1992]", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g6D2F6VTNN", "reviewer": "Reviewer_3va7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* NSI proposes to adapt the DINO features aggregated by slot attention and align them with visXML text, in a compositional manner. This is interesting. \n* The paper provides abundant evaluation of the proposed method on various downstream tasks, showing different aspects and applications of the proposed methods. \n* This paper provides many technical details and qualitative results. They help understand the experiment setup and model performance. \n\n\nMajor weaknesses:\n\n* NSI poses a strong assumption that a correct schema that corresponds to the scene exists and is given. This limits the usage of the proposed method compared to the common-used (either open-set or close-set) object detection or segmentation models, which can generate the set of objects for each input image without any assumptions. In such a case, how is NSI useful in practical computer vision tasks? Or, **in what case should I use NSI rather than an object detection(-segmentation)-featurization-captioning pipeline with pretrained models?** (e.g. https://github.com/ttengwang/Caption-Anything)\n* Following the above point, the \"Scene-Schema Alignment Evaluation\" task looks very *artificial* to me. It requires you to have a set of scenes and their corresponding schemas and requires you to shuffle them to construct such a task. I cannot see how it's useful in practice. \n* The central question of this paper is “Can slot representations, when explicitly grounded in object semantics, serve as effective compositional abstractions for grounding and reasoning?\". Yet I doubt whether this paper addresses this question. Because the slot attention representations are essentially the weighted mean of the underlying featurizer (DINO in this paper). In this regard, this paper shows that DINO features are very good at distinguishing different objects, and taking the mean of them is still semantically meaningful. I wonder how relevant is this to \"slot representations\". For example, how about simply pooling DINO features according to object detection bounding boxes/masks and aligning them with text? \n* How does NSI ground \"primitives\" into \"slots\"? The contrastive loss in Section 4.2.4 only explains how the \"schemas\" are grounded into \"scenes\". I was expecting another contrastive loss between primitives and slots (as depicted in Figure 3) but it's not mentioned at all in the paper. Without this, how could NSI possibly learn to relate primitives with slots? \n* While the paper compares to the CLIP model and claims the advantage of NSI in compositionality, I think a VLM baseline is needed. While the CLIP encoder has \"bag-of-word\" behavior, some of the recent VLMs do not and have a better performance in understanding texts structured in the visXML-like format. \n\nMinor weaknesses and questions:\n\n* This paper uses a lot of $\\Rightarrow$ in sequences of equations that do not actually involve \"imply\" relationships (Eq. 4, 5, 6, 8, 9). Please use a comma or semicolon when they're simply sequential operations on the input variables. \n* I'm not sure why and how visXML is important for this work. Seems that visXML can be replaced by any dictionary-like data structure and it does not affect the method or its performance. If this is the case, I suggest to hide/omit such implementation details in the paper. \n* I wonder how NSI handles \"background slots\". As exemplified in Figure 5(d), the background (floor) also takes up two slots (cyan and gray), but seems somehow they are not matched to any primitives. How is NSI able to do this? \n* In Figure 5(f), what do two repeating \"<category>person</category>\" tags mean?\n* In Figure 7, the top row, 3rd image, seems the slots are not accurately aligned with the object. Why is the bounding box still accurate? \n* The writing of the paper could be improved. I found myself lost multiple times and struggled to understand when reading this paper. \n* In Section 3.1, \"N features\" -> \"L features\". \n* Figures 14 and 15 are informative. I think it's worth to move them to the main text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Neural Slot Interpreters (NSI), a method to ground semantic information to objects learned by slot attention. The semantic information is represented visXML, a markup syntax for object categories and properties. The grounding is achieved by a structured contrastive loss, which projects slot features and visXML text into a shared latent space and aligns them. NSI is evaluated on multiple tasks, including scene-schema alignment, weakly-supervised visual grounding, object discovery, and learned visual token. The Proposed NSI shows better performance than the compared baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Please address my questions in the \"Major weaknesses\" section, which I think are cirtical to securing my recommendation. \n* I also strongly recommend a re-write of the experiment section, clarifying the input/output of each experiment and how exactly each baseline is used in each experiment. Currently, they look crumbled and are not good at conveying the information. \n* Please also address the minor weaknesses I mentioned above, which I think will strength the work in my view.", "broader_impact_concerns": "I see no outstanding concerns about ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers are leaning towards accepting the paper after the revision. The AC agrees with the reviewers and recommends acceptance.
Decision: Accept (poster)