Low-shot Object Learning with Mutual Exclusivity Bias
Authors: Ngoc Anh Thai
Ahmad Humayun
Stefan Stojanov
Zixuan Huang
Bikram Boote
James Matthew Rehg
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: low-shot learning
self-supervised learning
object learning
mutual exclusivity bias
TL;DR: We proposed a novel low-shot learning problem that leverage mutual exclusivity bias phenomenon observed in human word learning.

Abstract: This paper introduces Low-shot Object Learning with Mutual Exclusivity Bias (LSME), the first computational framing of mutual exclusivity bias, a phenomenon commonly observed in infants during word learning. We provide a novel dataset, comprehensive baselines, and a SOTA method to enable the ML community to tackle this challenging learning task. The goal of LSME is to analyze an RGB image of a scene containing multiple objects and correctly associate a previously-unknown object instance with a provided category label. This association is then used to perform low-shot learning to test category generalization. We provide a data generation pipeline for the LSME problem and conduct a thorough analysis of the factors that contribute to its difficulty. Additionally, we evaluate the performance of multiple baselines, including state-of-the-art foundation models. Finally, we present a baseline approach that outperforms state-of-the-art models in terms of low-shot accuracy. Code and data are available at https://github.com/rehg-lab/LSME.

Low-shot Object Learning with Mutual Exclusivity Bias
Anh Thai'
Ahmad Humayun? Stefan Stojanov' Zixuan Huang?
Bikram Boote James M. Rehgl.3
Georgia Institute of Technology. Google Deepmind,
University of Illinois, Urbana- Chamm-Caaap
Abstract
This paper introduces Low-shot Object Learning with Mutual Exclusivity Bias (LSME), the first computational framing mutual exclusivity bias, a phenomenon commonly observed in infants during word learning. We provide a novel dataset, comprehensive baselines, and a state-of-the of-tth--artttt art method to enable the ML com- munity tackle this challenging learning task. The goal of LSME is to analyze an RGB image of a scene containing multiple objects and correctly associate a previously -unoouww object instance with provided category label. This associa- tion is then used to perform low-shot learning to test category generalization. We provide data generation pipeline for the LSME problem and conduct thorough analysis of the factors that contribute to its difficulty. Additionally, we evaluate the performance of multiple baselines, including state-of-the-arr foundation models. Finally, we present baseline approach that outperforms state-of-the-ar models in terms of low-shot accuracy. Code and data are available https::/github.com/re
lab/LSME
Standard Low-shot Learning
Low-shot Learning with Mutual Exclusivity Bias
sandwich
car
Boon
sandwich
lion
Training
Input: Single object image Learn classify object
Training
Input: Scene image with seen and novel objects Task: Identify object
Testing
Single object image Task; Classify object
Input:
Testing
Input: image known objects Task: Localize and classify novel objects
Figure I: We propose Low-Shot Object Learning with Mutual Exclusivity Bias (LSME), novel realistic and significantly challenging than the standard low-shot object learning
setting that setting. Given scene with multiple objects including seen and novel objects, the goal of LSME is to use mutual exclusivity bias to correctly identify the unfamiliar object and generalize this association to other instances of the novel category at testing time.
'Equal contribution.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Introduction
Toddlers are incredible learners, acquiring an average of 10 to 20 new words per week during a period known the vocabulary spurt [29]. This is possible in part because of effective inductive biases that help them to overcome the inherent ambiguity that arises in inferring which weal-world object spoken object name (e.g.. provided by caregiver), is referring
Inductive biases like shape 35 36], due in part to widespread interest in the link between 3D object shape and categorization. In contrast, the key inductive bias of mutual exclusivity has
bias have received more interest 44
27 28] Mutual exclusivity leverages the assumption that each object has only one label. Thus when presented with a scene containing multiple objects, some familiar and some unfamiliar, mutual exclusivity guides the child to bind new object names to the unfamiliar objects. For example, consider a toddler playing with three toys: her favorite Ducky, a Mickey Mouse figure, and toy that she does not know the name of. When the caregiver says, "Look at the dinosaur," the toddler uses mutual exclusivity bias to rule out Ducky and Mickey, as she already knows their names. She then associates the word "dinosaur" with the unfamiliar object. After this mapping is made, the toddler can retain the category label "dinosaur," " and use it later to label other dinosaur-like objects. The goal of this work is to create a novel dataset and associated baselines to spur the MI. community in developing computational models for mutual exclusivity, as a further step towards modeling the inductive biases that underlie human object learning performance.
not yet received significant attention [1]
Base Categories
Low-shot Query
candy
mig
puitze
robul
"Where ockapus
"Look the octopus"
Low-shot Support
Mutual
Exclndiity Blas
Ouipul
Familiar
octopus?
X
octopus!
X
octogu??
Figure 2: Our proposed LSME (Low-shot Object Learning with Mutual Exclusivity Bias) setting. We leverage mutual exclusivity bias observed in humans to associate a novel word label with an unfamiliar object in scene. Given set of familiar categories (base classes), the task is use this bias to correctly identify the unfamiliar object and generalize this association to other instances of the novel category.
This paper introduces Low-shot Object Learning with Mutual Exclusivity Bias (LSME), com- putational framing of mutual exclusivity as an inference problem, with low shot learning as the downstream measure of generalization performance (see Fig. for an overview and Table [ for related settings). The LSME task begins with a set of already-known object categories. The input an RGB image containing multiple object instances from known categories and a single object instance of an unknown category, along with novel category label. Solving the LSME problem requires three steps: 1) Localize the object instances within the scene via segmentation: 2) Associate the novel category label with the unknown category instance that it refers to; and 3) Solve a low-shot learning task by classifying other object instances from the novel category during inference. The ability to reliably solve LSME would be an important step in creating agents that can learn with minimal supervision. For example, houschold robotics is a long-standing goal of AI. Our work could ultimately yield robot capable of rapidly learning the names of novel objects in the home, without the need for a burdensome supervised training stage.
L.SE can be partitioned into three sub- tasks: 1) object localization, 2) open- world recognition, and 3) low-shot learning. The model must first localize each object in the scene. It must then distinguish between learned and unknown categories to correctly assign the novel label to the unknown instance. Finally, the model must generalize to new instances of the novel categories after only one or a few labeled samples. At the heart of LSME is the challenge of associating novel object instances with their category labels. In supervised pre-training, there is a danger of leakage the datasets used in pre- pre-training overlap with the soo-shot category labels used in testing. In order to prevent this,

---

Table Comparative analysis of LSME and other related settings. LSME requires a comprehensive
understanding of scenes that involve the presence of multiple objects.
Settings
I Low-shot Recognition
Open-set Detection
Low-shot Detection
Object Discovery
GCD
NCDL
Open Category Detection
Image Conditioned Detection
LSME
Properties
Instance Localization
X
x
Mutual Exclusivity Blas
x
x
x
x
X
Discover Novel Classes
x
x
X
Novel Classes
x
x
X
Zero/Low-shot
x
Pretrained
constrain the LSME problem design so that pre-training prior to the low-shot phase cannot include any object-label association data (e.g. as in CLIP). This constraint further aligns with the investigation of key question in developmental psychology: How can child rapidly learn words in the absence
of prior knowledge of object semantics? [51 26
It's natural to ask LSME can be solved by simply combining existing SOTA models. We find that this is not the case. Errors in the first two sub-tasks propagate to the low-shot stage and degrade accuracy. For example, complex spatial interactions (c.g. occlusions) between objects make reliable instance segmentation challenging, leading to poor feature extraction for each object. Furthermore, any errors in identifying novel vs. known objects will result in label noise that degrades low-shot recognition. In addition, pre-trained feature representations from foundation models (e.g. DINOv1 and DINOv2 (33]) are not sufficient to solve our ask effectively (see Sec. 4
One challenge we address in this paper is the creation of suitable LSME datasets. Since real-world data is not available, we introduce a generic data generation pipeline that can take any categorical 3D dataset input and render training and testing data with realistic backgrounds, lighting variations, and realisic object poses. Our dataset is structured with increasing levels of complexity in order to aid in understanding the shortcomings of learning approaches.
Additionally, we benchmark the performances of various baselines on LSME, including SOTA foundation models with robust visual representations such DINOv2 333, ImageBind [12], and CLIP 377. Interestingly, we find that the performance of these baselines degrades significantly when there occlusions/incompltte instance masks (see Sec.
From these observations, we propose a baseline that involves pretraining the object representations on cluttered scenes with occlusion. Following the success of prior works that show performance
improvement when training on multi-view inputs 43 [7]. we propose self-supervised training on the large-scale multi-view multi-object ABC 21 dataset using contrastive learning. Our baseline composed the robust 2D representations from foundation models and 3D features from multi-view inputs demonstrates improvement in accuracy compared to existing models when occlusions present by approx. 10%. To verify that this gain significant, we evaluate our approach on the real-world dataset CO3D 38 and show the benefit of multi-view training with occlusion.
In summary, our contributions are 4-fold: 1) The first to provide a computational framing of mutual exclusivity via LSME; 2) A dataset generation pipeline that enables the creation of progressively more challenging LSME tasks using any set of 3D models; 3) Performance benchmarking of multiple baselines including foundation models the SOTA on LSME.
novel task; and 4) A novel baseline method that defines
2 Related Work
Because we the first to provide computational framing of mutual exclusivity (ME), there is no direct prior work to compare to. [IT demonstrated that existing deep models fail at ME, but did not provide comprehensive framing or SOTA methods. We now review three bodies of related work.
Self-supervised and Weakly-Supervised Learning Self-supervised learning aims to leverage the inherent structure in visual data to train representations suitable for downstream tasks, without relying on labels. Contrastive methods 7 [5] g work by treating image pairs under different data augmentations as positive examples, and other sample pairs as negatives. DINOv1 [5) enforces "global-local" correspondences by using multiple image crops, while other methods 48 55
focus pixel-level matching combine both objectives [33]. On the other hand, VISPE [17].

---

Pretraining: ABC
Low-shot: Toys4k
Low-shot: ShapeNetCore..g
Figure 3: Rendered scenes used for LSME. First column has scenes of ABC [2] objects that are used for pretaining. Second and third columns show data for low-shot evaluation on Toys4k [4] and ShapeNetCore..v v2 [6]
DOPE 43 and VSA 24 incorporate multi-view data by treating samples captured from different viewpoints as positive pairs. These approaches leverage 3D structure to gain improvements.
learning foundation models such as CLIP 37 and Image- Bind [12] take advantage of multi-modal signals, including language and audio in addition to visual information. Another body of work [4 [2] (50) attempts learn the underlying visual structure of data by learning to reconstruct images from heavily masked inputs. In this work, we study the performance of these methods our novel task. Moreover, we extend these representations by incorporating multi-view multi-object information. By integrating data from multiple viewpoints and considering information the object level instead of the scene or pixel level, we aim to learn object representations that can reason about the spatial layout and category composition of scenes with multiple objects.
In contrast, recent
Related Settings Vaze et al. proposed Generalized Category Discovery (GCD) 45 a setting that allows input come from both known and unknown distributions, and further aims to label novel data. We differ from this setting in that we do not assume object localization information. We further consider scene-centric data that provides more holistic view of the entire scene instead of object-centric data as input. This enables our model to take into account the relationships between multiple objects within the scene.
Fomenko et al. 9 introduce the novel class discovery and localization (NCDL) setting, which closely relates to our proposed LSME task. Both tasks share the goal of localizing and classifying unknown objects in a scene. However, we operate in a low-shot setup, where only a limited number of samples from the novel categories available. This contrasts with NCDL, which focuses on balancing long-tail distribution of novel classes with varied number of samples per class. In addition, our setting requires mutual exclusivity bias-explicitly learning ambiguous word-object associations.
have recently gained popularity for its generalization ability beyond the trained vocabulary. This task focuses on using large pre-trained vision- lision-language models to detect and segment known and novel objects via prompting the LLMs. In contrast, our task focuses specifically on learning the association between objects and their corresponding labels. These labels may not necessarily align with the vocabulary typically used for LLM pretraining (e.g. using pseudo names for toy objects like "dax" commonly seen in psychology experiments [2]). LSME can flexibly accommodate various vocabulary systems that potentially is a challenge for pretrained LLMs.
Open-categors object detection and segmentation 553 19]
Image-conditioned object detection The goal of this setting is to detect objects that share the same semantic characteristics as the template objects given as one or a few input images. Our setting solves a more general and challenging task where the model is required to identify the unknown object of interest before generalizing to other object instances from the novel category. LSME is further related to low-shot learning 43 [4 25).
30
34
a related setting where language input is not required.
open-world learning 46 4. and object instance segmentation [4 49 24 settings as it integrates
these components into a holistic solution. (See Tab.
Computational Frameworks Motivated by Developmental Psychology Recent works have in- vestigated various learning scenarios to explore the strategies children employ when acquiring new concepts 18 3 16]. This work complements these efforts by introducing a comprehensive bench- mark that studies the mutual exclusivity bias commonly observed in infants during the initial stages of word learning.

---

Synthetic 3D Datasets and Dataset Generators Synthetic 3D datasets and environments have been explored study a wide range of tasks 44
50 1 where real-world data might
I0 41 and image not be available at large scale. Recent realistic data rendering engines [3]
generation models [39 40 have significantly reduced the sim2real domain gap, enabling more effective generalization to real-world settings. In this work, we leverage the benefits of synthetic data to tackle novel problem where real-world datasets are unavailable. This allows us to create diverse and customizable scenarios that closely resemble real-world settings to study the problem in a controlled and scalable manner.
Single-Obip Object Sctting
Multi-Objeet Setting
Calcany Stot desenad
Categney Single Object
Category Single Object Varying Pose
Caaggry Onject
Skoe Minal Dedrediyy
Single Object
Base
classes
MF required
Support
Support
Query
Query
X
X
Localizaaiion
X
Mutual Exclusivity
X
X
Mutual Exclusivity
X
Low-shot
X
Low-shor
Figure 4: Different variants of LSME setting. Left: Single-object case where only single object is present in the scene for both supports and queries and Right: Multi-object setting where multiple objects present in the scene. Bounding boxes and texts indicate given localization information and labels respectively. The difficulty levels of these variants increase from left to right, with the hardest setting proposed LSME on the rightmost column requiring the models to achieve all properties: localization, mutual exclusivity bias, and low-shot.
Low-shot Categorization Using Mutual Exclusivity Bias (LSME)
3.1 Task Formalization
Consider RGB scene containing multiple objects, including known instances and single novel instance that lacks any explicit localization information and a word for its label. Our task can be partitioned into solving three well-known subtasks:
Object Localization. The first step is to localize the objects in an RGB scene.
Open-worke Recognition. The second step is to diffcrentiate between known and novel categories. The objective identify the instance within each scene that belongs novel category. To enable this, we assume the availability of a collection of labelled images from known categories, referred to as the base classes. Once the novel object identified it is associated with the novel word label, which we refer to support object.
Low-shot Learning. The last step is low-shot generalization. Given objects from the novel categories, the goal to correctly classify them based on the support objects.
few query images of
Data Generation Pipeline. Given any 3D categorical dataset, following the convention of low-shot learning, we first partition these object categories into disjoint sets: base classes and low-shot classes. While base classes commonly used to learn robust feature representations, low-shot classes are for evaluating the generalization ability of the models a low data setting.
Data Rendering. To generate each scene, we first randomly select a subset of objects. The rotational poses for the objects are obtained using rigid body simulation. The objects are scaled and placed into the scene at random locations making sure collisions do not occur. Scenes are generated with varying background, illumination, and camera viewpoint. Please refer to the Supplement for more detailed descriptions of the generating process.
5

---

Maxiced Ofjects
Festune Extrater
"Lock octupeo
Object Ensheddings
max
0.7
Support
laput
max
0.8
argmin
Extmetor
Ocupus
max
0.4
Compute similanties and take max
Fosture
Classes
Figure 5: The process of Mutual Exclusivity Bias in our models. The input is the support a novel word label. The model first needs to 1) localize the objects in the scene, 2) extract feature embeddings for these objects and compare with the features of the objects from the base classes using cosine distance, 3) take the maximum similarity score between each object embedding and the objects the feature bank, 4) determine the unknown object by taking the object with the minimum similarity score and then assign the novel word label to this object.
with
3.2 Data Variants
To support comprchensive analysis and gain insights into the limitations of different methods, generate data with increasing difficulty levels of variability (please see Fig.
The first group of variants considers scenes with a single object (-SObj) (left side of Fig. 4). The (as-SSObii-eeoggnnn
simplest setting concerns object instance recognition on few images in the same pose. Our second group of variants requires models to generalize to different instances of the same category (Categ-SObi). while our third setting also takes into consideration random initial object pose sampling (Categ-SObj-Poseeaa) Note that Categ-SObj similar to the standard low-shot learning setting 44 These settings do not require mutual exclusivity bias.
the same object based
side The second group of has multiple objects (-MObj) (right side of Fig. 4. For the first, labels for the shots are given (Ooje.-MOb). This is similar to Categ-SObji SObj-Pose Var, except the objects might be partially occluded. The second variant requires using mutual exclusivity bias to assign the label to the correct object before low-shot generalization Stateg-MObj SSpppssSppsssi)n)) Finally, LSME also requires object localization.
Note that while the first data variant (Inst-SObject) yields a straightforward task that does not require category generalization, all subsequent variants CCbjeg-SObject, Categ-SObject-Pose and Categ- MObject) require eestance-to-catgnne Although not requiring mutual exclusivity bias, these variants are significantly more challenging in comparison to the standard low-shot learning formulation, because they introduce pose variability and include multiple objects, as described in Sec.
generalization, as in the standard low-shot learning setting.
3.3 Our Approach
Representation Learning. Motivated by the understanding that multi-view observations provide rich visual information about the 3D environment that aids downstream tasks [43 17 we adopt strategy of pretraining the feature extractor by leveraging contrastive learning on multi-View scenes. Unlike most self-supervised approaches that primarily operate on the scene level even when the scene consists of multiple objects, we focus on the object-level representation learning. Specifically, given scenes of multiple objects captured from various viewpoints, along with the corresponding instance masks for each object, we learn to match the representation of the same object across different views.
We use tre-trained backbones, (e.g. DINOv1 [5), DINOv2 momentum encoder [15]. Given two views of the same scene, V1 and U2, we first use the instance mask associated with each object in the scene to eliminate the background and other objects. Subsequently, we extract the query object feature by performing a forward pass of the image encoder on vi. For each query feature, we minimize the InfoNCE 32 loss function. The positive sample is the feature of the same object in 12 while the negative set consists of object features from the memory queue as in MoCo-v2 [8] and different objects from the same scene. This approach is inspired by the VISPE+
33 contrastive training strategy with

---

Table 2: Low-shot recognition on the Toys4k dataset the single object setting. All methods
consistently drop in accuracy when evaluated on the harder data variants.
DINOv1-S/8
DINOw2-S/14
DINOv2-B/14
I-shot
I-shot
1-shot
I-shot
1-shot
1-shot
Variants
5-way
10-way
5-way
10-way
5-way
10-way
Inst-SObj
95.80
92.37
95.75
93.06
96.50
94.22
Categ Categ-SObj
73.06
60.73
77.11
66.62
79.69
69.55
Sateg-SOhj-Poseyee 68.84
57.45
73.07
61.44
75.18
66.30
Table 3: Results low-shot recognition on the Toys4k dataset in multi-object setting, All methods
consistently experience a significant drop in low-shot accuracy when mutual exclusivity is required. and further decrease when instance segmentation is involved.
DINOv1
DINOv2
DINOv2
CLIP-Img ViT B/16
ImageBind viT H/16
VIT S/8
ViT S/14
VIT B/14
Variants
LSA
SA
LSA
SA
LSA
SA
LSA
SA
LSA
SA
Categ-MObj
56.99
N/A
56.95
N/A
57.92
N/A
56.76
N/A
60.49
N/A
Categ-MObj
SSppAssign
40.21
51.68
41,26
52.28
43.21
54.96
41.22
51.64
45.91
58.58
LSSM
36.44
46.92
37.08
48.16
39.24
50.88
38.25
48.96
38.85
50.24
baseline in 43 For each input view pair, we ensure to only train on objects that are visible in both views (e.g. with instance segmentation area greater than some threshold pixels)
Unsupervised Instance Segmentation. We use fine-tuned FreeSOL. 47 model as our segmenter, benefiting from its strong performance instance segmentation tasks. Fine tuning is done on 1K scenes from the ABC dataset, which have been annotated with instance segmentation masks.
Low-shot Learning. We apply the following support assignment and low-shot generalization technique in all methods.
We first collect a feature library consisting of object features extracted from the base classes as learned objects. During the low-shot training phase, for each input scene, our goal to determine which object should be assigned the novel label. For every object in the scene, we calculate its similarity with each object in the feature library using cosine distance. To determine the final similarity score with the feature library, for each object in the scene, we select the maximum similarity measure across all objects in the library. Finally, the object in the scene with the lowest similarity score assigned the novel label. Under this procedure, a learning agent would associate the novel label to the object that it is least familiar with (Fig. 5)
Support Assignment:
Low-shot Generalization: Given query scene, we compute the cosine distance between each object in the query scene and every support object identified in the support assignment phase. We then determine the nearest neighbor for each query object within the support set and assign the label of the query object to its closest counterpart in the support set.
Experiments
4.1 Evaluation Setup
and the convention in low-shot learning, we partition the categories of ShapeNetCore. v2 66 and Toys4k (44). These datasets are divided into two disjoint subsets: base classes, for learning object-label associations, and test classes, for evaluating low-shot object recognition performance. We generate 1K scenes each for support and query sets, and 500 base scenes for multi-object experiments. Each scene in our dataset consists of 3 objects rendered in 20 views. To study mutual exclusivity bias, we ensure that each scene in the support set only contains 1 object from a novel category while the remaining objects come from base classes. We further incorporate a subset of CO3D 38) that shares 13 categories with the test classes in Toys4k to investigate the effectiveness of occlusion-
Dataset. Similar to prior works 44 43
feature representations in the real-world scenario. Since CO3D is an object-centric dataset, it is not possible to directly test LSME CO3D. We only evaluate this dataset on the standard low-shot setting where mutual exclusivity bias is not considered.
We pre-train our models using the 3D object instance dataset ABC 21 without any category structure. We use subset of ABC that consists of 100K objects. We render 10K scenes with 20 views per scene (8K for training and 2K for validation), each containing 2-3 objects with random object poses
7

---

Single-abject Settings Toys4k
Multi-object Settings Toys4k
DENONS AN
IENONA NOW
Low abot Accuracy (S5
Culan Poofle
Setting
Setting
Figure 6: Low-shot accuracy DINOvI and DI- NOv2 pre-trained models on the Single-object set- tings of Toys4k. The size of the markers corre- sponds to the number of parameters of the models.
Figure 7: Performance of pre-trained models on the Multi-object settings of Toys4k. Circle and triangle markers represent self-superrvsed vision models and models that have linguistic input respectively.
Table Performance of different methods on Toys4k under Sangg-SOb-P-eee Var and Categ-MObj settings. These settings solve similar problem, with Categ-MObj having object occlusions present both support and query objects. Performance of all methods drops significantly when faced with occlusion cases.
Method
DINOv1
DINOv2 S/14
DINOv2 B/14
Categ-SObj -Pose Var Categ-MObj
68.84
73.07
75.18
56.99
56.95
57.92
and scales. We further randomize the textures and surface materials of the objects in each scene (refer to Fig. 3
Evaluation Protocols. We evaluate the performance of the baselines using the following metrics: 1) support assignment accuracy (SA) which quantifies the percentage of accurately identifying the novel instance within the scene, and 2) low-shot accuracy (LSA) for measuring low-shot performance, and 3) mloU for instance segmentation. In this work, we follow the standard framing of low-shot inference, such that "1-shot 5-ways" means that each episode has novel categories, each with only I object during the low-shot training phase. Unless stated otherwise, the experiments in this paper evaluated on the 1-shot-5-way setup with 500 episodes. We report the confidence intervals and experiments on other setups in the Supplement. We make sure that novel objects are clearly visible during both support assignment and low-shot generalization phases (e.g. with instance segmentation area greater than threshold pixels).
Representation Learning Baselines. Our main design constraint is that pre-training prior to the low-shot phase cannot incorporate any object labels, order to avoid any concerns of label leakage. We focus analyzing the performance of models initialized from the SOTA self-supervised vision models DINOv (5) and DINOv2 [33. This allows us to gain further insights into whether LSME be tackled without prior language inputs. We further investigated the performance of large-scale vision-language foundation models, in order to characterize the difference between self-supervised and pretraining approaches. We present findings using CLIP (37) and ImagelBind [12 pre-trained models. Note that these models violate the constraint by having unrestricted (in terms of category composition) image captions their pre-training.
4.2 LSME and Related Tasks on Synthetic Data
We first analyze the performance of pre-trained representations on settings with varying difficulty. building to LSME.
Single Object Setting. We present the results for the single object setting Toys4k in Table 2 and Figure 6 1 1-shot 5-way and I-shot 10-way set ups. We observe a decrease in performance when tested the harder variants for all models. While the difference between Categ-SObj-Posss Var and Categ-SObj is not significant, we observe a more prominent gap between Categ-SObj and Inst-SObj. This indicates the challenge faced by the models when generalizing from instance category level.
Multiple Objects Setting. We report these results in Table and Figure 7
The performance of all models exhibits significant decrease (~ 15%) when mutual exclusivity bias is required (2nd row).

---

Table 3: Performance of DINOv2 and our method fine-tuned on Toys4k and ABC on Toys4k under LSME setting. All methods use VIT B/14 as the backbone and our method is initialized with pretrained DINOv2 weights. Training on ABC improves the performance significantly, surpassing the model that was trained on the base classes of Toys4k with the same number of scenes.
Method
LSA
DINOv2 Toss
39.24
50.88
43.62
53.44
OuS-DDLNO2-AACC
47.70
61.32
Table 6: Performance of baseline finetuned with different backbones on Toys4k under LSME
settings with four object segmenters. Best performance is highlighted in bold while underline represents second best performance. The quality of the instance masks plays significant role in the low-shot and shot assignment performance for all methods.
Ours Burs-DINOv1
Ours- Burs-DINOv2
Ours- Durs-DINOv2 B/14-ABC
mloU
S/8-ABC
S/14-ABC
Support
Query
LSA
SA
LSA
SA
LSA
SA
FreeSOLO 47
0.52
0.54
32.31
43.24
33,99
44.84
35.50
48.92
CutLer
0.61
0.63
34.62
42.40
36.34
46.08
39.42
52.04
SAM 20
0.72
0.73
35.72
48.96
38.58
52.04
42.38
56.92
FreeSOLO-ABC
0.81
0.83
41.19
55.40
43.92
57.88
47.70
61.32
Table 7: Performance of DINOv2 and ours using ViT B/14 at LSME on ShapeNetCore.v2. Our
method demonstrates an improvement in both low-shot and shot assignment accuracy.
Method DINOv2
LSA SA
32.55
47.68
Ouss-DINOV-AAEE 41.03
58.92
This decline can be attributed to the challenges associated with imperfect support assignment, which negatively affects the low-shot accuracy across all models.
Effect of Segmentation Quality. Table provides additional evidence of the significance of the quality of the instance masks generated by the segmenter. demonstrates that a decrease of 0.1 in mloU (between the last 2 rows) result in substantial impact on performance, with approximately 6 - 7% decrease in low-shot and support assignment accuracy. This emphasizes the critical role that accurate instance masks play in achieving seigh-eeffrmance results in both LSA and SA metrics.
Effect of Occlusion. Table highlights the impact of occlusion caused by other objects in the scene on the low-shot performance of the models. While Categ-SObj-Ponn Var and Categ-MObj address a similar problem, Categ-MObs considers potential occlusion both support and query objects due to multiple objects being in the scene. We observe a decrease in performance for all models when occlusion is present. Specifically, DINOv1 experiences an average decrease of 11.85% while DINOv2 models show greater decreases of 12% and 17.26% respectively.
These findings indicate the challenge posed by occlusion in low-shot learning scenarios. The presence of occlusion introduces additional complexity and ambiguity, making it more difficult for the models accurately associate novel objects with their corresponding categories. This highlights the importance of developing methods purpose-built for the task of low-shot learning with mutual exclusivity.
Contrastive Finetuning on ABC Improves Performance. In Table we show the advantage of contrastive finetuning on the ABC (21) dataset. While prior work (43 has demonstrated the benefits of pretraining feature representations on ABC, we are the first to show the advantages of ABC in multi-object setting. Additionally, we train our model on an equal number of scenes from the base classes of Toys4k. Although this approach enhances the low-shot generalization performance on the test classes of Toys4k compared to DINOv2, its performance is inferior to the model trained on ABC. This highlights the benefits of leveraging ABC in the context of multi-object understanding and its potential for improving the capabilities of models in complex scenarios.
In addition, the performance comparison between DINOv2 with VIT B/14 backbone and our model fine-tuned ABC is presented in Tab. 7 on the ShapeNetCore.v2' dataset. Notably, our model consistently outperforms DINOv2 across both the LSA and SA metrics, showcasing an improvement of approximately 10%.
4.3 Low-shot Generalization on CO3D Dataset

---

We investigate the generalization capabilities of pretraining on the multi-object nulli-view ABC dataset, extending beyond the synthetic data domain to real-world datasets. Consider- ing the negative impact of object occlusions and incomplete object masks on the performance of models, we hypothesize that pretraining ABC scenes with multiple objects where occlu- sions are present improves the models' ability to handle such scenarios. To test this hypothesis, conduct experiment where we randomly mask the instance segmentations and evaluate the models these masked segmentations in the standard low-shot setting. Figure [ show- cases the performance of different approaches on CO3D dataset with varying mask ratios. The models finetuned on ABC exhibit a better per- formance compared to other models the mask ratios increase. Even mask ratio of 0.5, our ABC-finetuned model with DINOv1 ViT S/8 pretrained weights performs on par with DINOv2 ViT G/14. Note that DINOv2 ViT G/14 has significantly more parameters and was trained on an order of magnitude more data than our model. These results demonstrate the benefit of occlusion- feature representations.
Segmentation Masking CO3D
Lowabot Accursev
1414
Masking
Figure 8: Performance of different methods on the segmentation masking experiment on CO3D. Low-shot performance of these methods decreases as the mask ratio increases. Models pretrained on ABC exhibit a slower decrease rate.
4.4 Limitations & Future Work
One important limitation of our work is the absence of uncertainty reasoning. In real-world scenarios, agents often encounter multiple unknown objects, requiring the integration of novel labels acquired in diverse contexts to correctly associate objects with their corresponding labels. This more complex setting requires continuous integration of new information and reasoning in anbiguous situations. Another challenging real-life infant-lcarning scenario is where objects might have multiple names "dog" and "husky" both referring to the breed "husky"). Due the procedural nature of our data generation system, we have the ability to increase the task complexity as solutions to LSME improve beyond the current baselines. Future research can explore more challenging settings and developing approaches that incorporate uncertainty reasoning to improve performance in ambiguous scenarios.
Negative Societal Impact. Training and evaluating large-scale self-supervised learning models as well as generating data require extensive GPU usage, which negatively impacts the environment. Advancements in hardware design and techniques for optimizing deep models offer potential solutions to mitigate this impact.
5 Conclusion
We present a novel setting called Low-shot Object Learning with Mutual Exclusivity Bias (LSME), which requires comprehensive reasoning about scenes with complex object interactions. We conduct thorough analysis of the challenges present LSME and their impact on SOTA models by generating various problem variants. Based these insights, propose pretraining strategy that outperforms SOTA baselines on both synthetic and real-world data. Additionally, we release our open-source data generation pipeline and the generated datasets for further research.
Broader Impact
The challenging LSME task provides a framework for devising algorithms that can learn from limited data input. We hope that LSME is an example computational framework that serve as a template to formulate learning tasks based on other insights from the developmental psychology community. Future development of this setting can provide a learning environment that closely resembles real- world scenarios, effectively for further studies of both computer vision and developmental psychology communities.
10

---

Acknowledgement
This work was supported by NIH RO1HD
References
[1] Blender, https://blender.org/..
[2] Poly haven, https//pollyavven.om/
[3] Harsh Agrawal, Eli A. Meirom, Yuval Atzmon, Shic Mannor, and Gal Chechik. Known unknowns: Learning novel concepts using -yy-oiinnn-b-lllmiaaiinnnn.. In Cassio de Campos and Marloes H. Maathuis, editors, Proceedings of the Thirty-Sevetth Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 504-514, PMLR, 27-30 Jul 2021.
[4] Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1893-1902, 2015.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégon, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-super vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[6] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1r1iv:551..03012. 2015.
[7] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved baselines with momentum contrastive learning. arXiv preprini arXiv:2003. 04297, 2020.
[9] Vladimir Fomenko, Ismail Flezi, Deva Ramanan, Laura Leal--Taxxé and Aljosa Osep. Learning to discover and detect objects. Advances in Neural Information Processing Systems, 33:8746-8759, 2022.
[10] C Gan, J Schwartz, S Alter, Schrimpf. J Traer, J De Freitas, Kubilius, Bhandwaldar, N Haber, M Sano, et al. Threedworld: A platform for interactive Newral Information Processing Systems (NeurIPS), 2021.
physical simulation. Advances in
[11] Kanishk Gandhi and Brenden M Lake. Mutual exclusivity as a challenge for deep neural networks. Advances in Neural Information Processing Systems, 33:14182-14192, 2020.
[12] Rohit Girdhar, Alaaeldin Fl-Nouhy, Zhuang I Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3180-15190, 2023.
[13] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann Thomas Kipf. Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: a scalable dataset generator. 2022.
[14] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009 2022.
[15] Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[16] Felix Hill, Olivier Tieleman, Tamara Von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. arXiv preprint arXiv:2009.01719 2020.
[17] Chih Hui Ho, Bo Liu, Tz- Yz-Y Wu, and Nuno Vasconcelos. Exploit clues from views: Self-supervised and regularized learning for multiview object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9090-9100, 2020.
11

---

[18] Guangyuan Jiang, Manjie Xu. Shiji Xin. Wei Liang, Yujia Peng. Chi Zhang, and Yixin Zhu. Mewl: Few-shot multimodal word learning with referential uncertainty. arXiv preprint arXiv:2306.00500 00503, 2023.
[19] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5830-5840, 2021.
[20] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tcte Xiao, Spencer Whitchead, Alexander C. Berg, Wan-Yen Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304. 02643, 2023.
[21] Sebastian Koch, Albert Matveev, Zhongshi Jiang. Francis Williams, Alexey Artemov, Evgeny Burnaev, Mare Alexa, Denis Zorin, and Daniele Panozzo. Abe: A big cad model dataset for geometric deep learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9601-9611. 2019.
[22] H.W. Kuhn. The Hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-9) 97, 1955.
[23] Shaoteng Liu, Xiangyu Zhang, Tao Hu, and Jiaya Jia. Self superviee learning by view synthesis. arXiv preprint Xrv::22044.1 1330, 2023.
[24] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold. Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:1 13111152--1538 2020.
[25] Orchid Majumder, Avinash Ravichandran, Subhransu Maji, Alessandro Achille, Marzia Polito, and Stefano Soatto. Supervised momentum contrastive learning for few-shot classification. arXiv preprint arXiv:2101.1105,, 2021.
[26] Ellen M Markman. Constraints on word meaning in early language acquisition. Lingua, 92:199-227, 1994.
[27] Ellen M Markman and Gwyn F Wachtel. Children's 's use of mutual exclusivity to constrain the meanings of words. Cognitive psychology. 20(2): 1988.
[28] Ellen M Markman, Judith Wasow, and Mikkel B Hansen. Use of the mutual exclusivity assumption by young word learners. Cognitive psychology, 47(3):241-775, 2003.
[29] Bob McMurray. Defusing the childhood vocabulary explosion. Science, 317
[30] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenhorn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open- vocabulary object detection. In European Conference on Computer Vision, pages 728-755. Springer. 2022.
[31] Pedro O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsu- pervised learning of dense visual representations. Advances in Neural Information Processing Systems, 33:4489-4500, 2020.
[32] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 180700378, 2018.
[33] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07799 2023.
[34] Anton Osokin, Denis Sumin, and Vasily Lomakin. Os2d: One-stage one-shot object detection by matching anchor features. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pages 635-652. Springer, 2020.
[35] Deepan Chakravarthi Padmanabhan, Shruthi Gowda, Elahe Arani, and Bahram Zonooz. Lsfsl: Leveraging shape information few-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4970-4979, 2023.
[36] Shitala Prasad, Yiqun Li, Dongyun Lin, and Aiyun Guo. Implicit shape biased few-shot learning for 3d object generalization. In 2022 IEEE International Conference on Image Processing (ICIP), pages 3436-3440. IEEE, 2022.
12

---

[37] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-8763, PMLR. 2021.
[38] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision. pages 10901-10911, 2021.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[40] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
[41] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9339-9347, 2019.
[42] Linda Smith and Chen Yu. Infants rapidly learn word-referent mappings via cross-situationa statistics. Cognition, 106(3): (06(3):5558-15 1558-1668, 2008.
[43] Stefan Stojanov, Anh Thai, Zixuan Huang, and James M. Rehg. Learning dense object descriptors from multiple views for low-shot category generalization. In Advances in Neural Information Processing Systems, pages 12566- 2022.
[44] Stefan Stojanov, Anh Thai, and James Rehg. Using shape to categorize: Low-shot learning with an explicit shape hias. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1798-1808, 2021.
[45] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7492-7501, 2022.
[46] Weiyao Wang, Matt Feiszli, Heng Wang, Jitendra Malik, and Du Tran. Open-world instance segmentation: Exploiting pseudo ground truth from learned pairwise affinity. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 4422 22224 2022.
[47] Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. Freesolo: Learning segment objects without annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14176-14186, 2022.
[48] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3024-3033, 2021.
Li. Dense contrastive learning for
[49] Xudong Wang. Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 3124-3134, 2023.
[50] Philippe Weinzaepfel, Vincent Leroy. Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav ARORA. Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jerome Revaud. Croco: Self-supervised pre- training for 3d vision tasks by cross-view complction. In Advances in Neural Information Processing Systems.
[51] Amanda L Woodward and Ellen M Markman. Early word learning. 1998.
[52] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 0653-9663, 2022.
[53] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. ODISE: Open- Vocabulary Panoptic Segmentation with Text-to- Image Diffusion Models. arXiv preprint arXiv: 2303. 0003.04833,00 2023.
13

---

[54] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings the IEEE/CVF conference computer vision and puttern recognition, pages 8808-8817, 2020.
[55] Wenwei Zhang, Jiangmiao Pang. Kai Chen, and Chen Change Loy. Dense siamese network for dense unsupervised learning. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXX, pages 464-480. Springer, 2022.
14

---

APPENDIX
This appendix is structured as follows: We first provide more details about data information in Section A
We then show additional results in Section B Finally, we provide additional training details about our baseline models in Section
A
Data
A.1 Datasets
In our work, performed experiments and analysis using three datasets: Toys4K 44), ShapeNet- Core.v2 [6]. ABC [2I], and CO3D [8 In the following section, we provide comprehensive details about each of these datasets.
Toys4K 44). This dataset consists of 4,1 object instances in 105 categories. We use the base and low-shot splits provided by Stojanov et al. 44 In particular, the base classes consist of 40 categories while the low-shot classes have 55 categories. Objects this dataset were collected under Creative Commons and toyalty-free licenses. (Please refer to Table g for basc/low-shot split compositions).
ShapeNetCore..2 [6]. This dataset consists of 52K objects in 55 categories. We partition these categories into 25 base and 30 low-shot classes (see Table. 8). The terms of use for ShapeNet are specified on their website, which can be accessed at bttps://shapennt.org/eenm
ABC (11). For pretraining our representation learning models, we used subset of 100K object instances from ABC, which contains a total of 750K instances. Note that this dataset lacks categorical structures. The dataset is distributed under the MIT license. More licensing information is available https://bee---eoneerrgggiiii ioa.beoiobbbbbbbbbb wo//abb-daatatt//Hcceeee
CO3D (38.. We chose the 13 classes out of classes that overlap with Toys4K for low-shot validation, detailed in Table 8 The terms of use for CO3D specified
at http://afffcccoooocoommmcs/////d
A.2 Data Generation
Software. We used Blender 2.93 with ray-tracing renderer Cycles for data generation and rendering.
Assets. Objects are placed on top a plane that simulates the ground/floor with PBR materials and image-based lighting from HDRI environment maps are used illuminate scenes. We collected these assets from PolyHaven (2). The list of assets used shown in Table 9
Scene Generation. Given any 3D categorical dataset, we first partition these object categories into disjoint sets: base classes and low-shot classes. For each object in the dataset, we preprocess by simulating rigid body drop using Blender [] This simulation process is repeated 16 times, allowing us to collect metadata and initial rotational poses for each object. These collected data are used in the subsequent stages of scene generation.
To generate each scene, we first choose a subset of objects from the dataset. Their initial rotational poses are determined by randomly choosing from the preprocessed poses. Objects are then scaled and placed into the scene at random locations. We ensure that collisions do not occur by maintaining minimum margin of A > ( between each pair of objects. We randomize the scene background by randomly choosing a pair of PBR material and HDRI environment map from the assets.
Data Renderingg To render each view of the scenes, we first determine the camera position. The camera's position in the scene is specified by three parameters: 0 E [0, 2TT. r [r Tmann Tiaas] > and 2 E [2min, Zmaz] > O where 0 is the rotational angle, is the distance from the origin in the XY-plane, and 2 denotes the world Z-coordinate of the camera. Note that Tmin; Tmary 2mins Zmaz are preset parameters. The world coordinate of the camera is computed by (r cos(f), sin(f), 2). To determine the camera's orientation, it set to point towards a location on the XY-plane that within small distance from the mean locations of the objects in the scene. This is done by rotating the camera in the world XY and YZ-planes. We then randomize illumination intensity, consistently for all the views of each scene.
15

---

2
m
Figure 9: Rendered scenes for LSME on Toys4k
Generated Data for LSME. We generated IK scenes for each of support and query sets, with each scene consisting of views. The data generated for LSME evaluation can be found Additionally, the code for data generation is available on Detailed parameters for scene generation
at http:://tinyull..oo3//rrrrrs GitHub repository at httpss//gihhbb..c//hhhlllaaall can be found in To
A.3 Data Augmentation for Contrastive Training
To augment the data, we applied various transformations, including random horizontal flips and brightness and color jittering. Following [43, we employed random object masking, where the object instance mask was used to eliminate the background. Additionally, we applied rotations and translations to the foreground object and incorporated background randomization techniques.
A.4
More Data Visualizations
Figure 9 showcasss additional examples of rendered scenes from the Toys4K dataset 44 These examples highlight the diversity found in the background, illumination conditions, and object poses within the scenes.
In Figure TO we demonstrate the instance mask prediction the FreeSOL. model finetuned on IK scenes of ABC. The quality of the predicted masks is essential to solving LSME
16

---

ShapeNerCCoe..22
Toys4k
CO3D
Base
Low-shot
Base
Low-shot
Low-shot
chair
piano
candy
boar
TV
table
train
flower
mouse
bathtub
file
dragon
whale
cubinel
pistol
apple
cupeake
loaster
lamp
motorcycle
guitar
train microwave
car
printer
tree
pizza
donut
bus
mug
glass
marker
orange
cellular
rocket
cup
cookie
sandwich
guitar
skatehoard
pig
sandwich
hicycle
bench
bed
cal
octopus
banana
bottle
ashcan chair
monkey
bowl
laptop
washer
ice-cream
fries
motorcycle
jar
bowl
hat
violin
pizza
loudspeaker
bag
deer mouse
mushroom
bookshelf'
mailbox pillow
penguin
closet
faucet
ball
tractor
vessel
carphone
fox
submarine
clock
camera
dog
butterfly
airplane
basket
knife pear
pot
remote
laptop bicycle
rifle
stove
dolphin
display
microwave
bunny
knifc
microphone
plate
coin
telephone
cap
chess picce
radio
sofa
dishwasher
cake
grapes
keyboard
frog
banana
tower
ladder
cow
helmet
keyboard sofa
donut
birdhouse
stove
can
trashcan
dinosaur
orange
bottle
elephant
chicken
pencil
hamburger
key
piano
monitor
light bulb
hammer
spade
screwdriver
crah
rubot
sheep
bread
toaster
motorcycle
mouse
bus
helicopter
microwave
cell buttery
drum
panda
TV
car
helmet
fridge
bowl
Table 8: Split composition of ShapeNetCoore..2,, Toys4K and CO3D
Additional Experiments
B.1 Evaluation Metric Details
We evaluate the performance of the baselines using the following metrics: 1) support assignment accuracy (SA) which quantifies the percentage of accurately identifying the novel instance within the scene, and 2) low-shot accuracy (LSA) for measuring low-shot performance, and 3) mean
17

---

PBR
HDRT
Carpet001
Aft Lounge
Carpe1005
Anniversary Lounge
Carpet006
Balcony
Carpet007
Cabin
Carpet008
Cayley Interior
Carpe1009
Children's 's Hospital
Carpet013
Colorful Studio
Carpett14
Entrance Hall
Fabric024
Fireplace
Fabric025
Hotel Room
Fabric028
Kiara Interior
Marblc012
Lapa
Planks001
Lehombo
Planks009
Lythwood Lounge
Planks011
Lythwood Room
Planks013
Moonlit Golf
Planks014
Music Hall
Planks018
Photo Studio
Terrazzo001
Reading Room
Tiles001
Roof Garden
Tiles027
Small Emply House
Tiles071
Spiaggia Di Mondello
Tiles072
St Fagans Interior
WoodFlourO0S
Umhlanga Sunrise
WoodFloor022
Wooden Lounge
Table List of assets used data generation.
Parameter
Value
Camera r"
[.0, 1.1)
Camera 8
[0.3, 0.5)
Camera jittering
0.01
Object scale
0.3550.45)
Object location
0.5,
Illumination intensity
00.6,088
Object margin A
Table 10: Data rendering parameters.
Input
GT
Pred
GT
Pred
GT
Pred
Figure 10: Segmentation prediction results on Toys4K model
using FreeSOLO 47 fine-tuned on ABC
intersection-ovve-uiion (mloU) for instance segmentation as detailed below, For each episode,
SA
Σ 1 =
N.
2=1
18

---

Table 11: Results on low-shot recognition on the Toys4k dataset in single object setting. All methods consistently experience significant drop in accuracy when being evaluated on the harder data variants.
DINOv 1-S/8
DINOv2-S/14
DINOv2-B/14
I-shot
1-shot
I-shot
1-shot
1-shot
I-shot
Variants
5-way
10-way
5-way
10-way
5-way
10-way
Inst-SObj
0..8
92.37:0042
95.7540.44
93.0620.41
96.5020.45
94.2220.37
Categ-SOby
73.06
60.7310.76
17.11
66.62.0.78
79.69
69.55.0.77
Categ-SOb) S-b---ObboP---e-e Pose Var
68. 1.04
57.4510.77
73:07.1.03
61.4410.50
75.18
66.3010.79
Table 12: Results on low-shot recognition on the Toys4k dataset in multi-object setting. All methods
consistently experience a significant drop in low-shot accuracy when mutual exclusivity is required, and further decrease when instance segmentation involved.
DINOv1 ViT S/8
DINOv2 ViT S/14
DINOv2 VIT B/14
CLIP-Img ViT B/16
ImageBind ViT H/16
Variants
LSA
SA
LSA
SA
LSA
SA
LSA
SA
LSA
SA
Categ-MObj
56.99
N/A
56.95
N/A
$7.92
N/A
56.76
N/A
60.49
N/A
20.90
#1.04
#1.00
Categ-MOhj
40.21
51.68
41.26
52.28
43.21
54.96
41.22
51.64
45.91
58.58
SuppAssign
#1.15
+1.86
#1.25
36.44
46.92
37.08
48.16
39.24
50.88
38.25
48.96
38.85
50,24
LSME
+1.08
2200
1.00
+1.87
#1.17
#1.91
11.14
22.03
111.4
$1.98
Table 13: Performance of DINOv2 and our method fine-tuned on Toys4k and ABC on Toys4k under LSME setting. All methods use ViT B/14 as the backbone and our method is initialized with pretrained DINOv2 weights. Training on ABC improves the performance significantly, surpassing the model that was trained on the base classes of Toys4k with the same number of scenes.
Method DINOv2
LSA
SA
39.24=1.17
50.88
Ours Buss-DINOv2-Toy
43.62
53.44=1.89
Ours-DINOv2-ACC
47.70
61.3221.86
where ô, and N, are ground truth object, predicted object, and the number of support objects
respectively (e.g. in the 1-shot-5-way setup N, = 5 since there are 5 support objects in the episode.)
N.. ME 1(i yak)
LSA Ng
where ý and y are predicted and ground truth labels respectively. The number of query objects is denoted Ng while No is the number of classes (e.g. in the 1-shot-5-way setup, N 5 since there are 5 novel classes.)
m. m
mIOU
m; m;
where m, m, and N denote the ground truth mask, predicted mask, and number of objects respectively.
B.2 Main Manuscript Results
In this section, we report the confidence intervals of the experiment results in the main manuscript (Please see Tables [ scenes for each episode.
12 13
14
and 15). We evaluate our models with 500 episodes and 15 query
B.3 Other Low-shot Setups
Table 16 presents the results of DINOv2 ViT B/14, both pre-trained and fine-tuned on ABC, in various low- setups, including 1-shot-5-way, 5-shot-5-way, 1-shot-10-way. 10-way, and 5-shot-10- 10yy under LSME setting on Toys4k.
While the support assignment accuracy (SA) remains consistent across all low-shot setups, the low-shot accuracy shows notable improvement in the 5-shot scenarios with an approximate 16% increase in low-shot accuracy in both 5-way and 10-way setups.
19

---

Table 14: Performance of different methods on Toys4k under Satee-SOOj-a Var and Categ-MObi settings. These settings solve similar problem, with Categ-MObj having object occlusions present in both support and query objects. Performance of all methods drops significantly when faced with occlusion cases.
Method
DINOvI S/8
DINOv2 S/14
DINOv2 B/14
Categ-SOhj Pose--ssho--P Var
68.84 #1.04
73.07 #1.03
75.18 #1.04
Categ-MObj
56.99 #0097
56.95 20.99
57.92 #1.04
Table 15: Performance of our baseline finetuned with different backbones on Toys4k under LSME
settings with four object segmenters. Best performance is highlighted in bold while underline represents second best performance. The quality of the instance masks plays a significant role in the
low-shot and shot assignment performance for all methods.
Ours-DINOvI S/8-ABC
Ours- DINOv2
Ours-D MIrS-DINOv2 B/14-ABC
mIoU
S/14-ABC
Support
Query
LSA
LSA
SA
LSA
SA
32.31
43.24
33,99
44.84
35.50
48.92
FreeSOLO 47
0.52
0.54
20.93
#1.94
20.93
#1.95
$0.99
#1.88
34.62
42.40
36.34
46.08
39.42
52.04
CutLer 49
0.61
0.63
11.96
#1.01
35.72
48.96
38.58
52.04
42.38
56.92
SAM 20
0.72
0.73
1:1.0
22,06
1.10
1:2.01
1.117
1.990
41.19
55.40
43.92
57.88
47.70
61.32
FreeSOLO-ABC
0.81
0.83
#1.16
22.03
1:23
1..97
#1.26
11.86
Table 16: Results on low-shot recognition on the Toys4k dataset in multi-object setting. All methods consistently experience a significant drop in low-shot accuracy when mutual exclusivity is required, and further decrease when instance segmentation involved.
DINOv2 VIT B/14
DINOv2 VIT B/14-ABC
Low-shot Sctup
LSA
SA
LSA
SA
1 1-shot-5-way
39.24.1.17
50.88
47.70=1.26
61.32=1.86
5-shot-5-way
55.03+0.99
50.22:0.99
63.52
60.60
1-shot-10- 10-way
28.3244073
51.32+1.46
35.66=0.82
61.10::1.00
5-shot-100 10-way
43.26:10.70
50.62:10.99
51.72=0075
60.85=0.74
C Models
Representation Learning Models: We use pre-trained backbones, (e.g. DINOv1 5 DINOv2 33
contrastive training strategy with momentum encoder Given two views of the same scene, Vi and 02, first use the instance mask associated with each object in the scene to eliminate the background and other objects. Subsequently, we extract the query object feature by performing forward pass of the image encoder on V1. For each query feature, we minimize the InfoNCE [32 loss function.
explq k+/T)
Cy = - log explq +
The positive sample k4 is the feature the same object in U2 while the negative set {k_} consists of object features from the memory queue in MoCo-v2 [8] and different objects from the same scene. For each input view pair, we ensure to only train on objects that are visible in both views (e.g. with instance segmentation area greater than some threshold
30 pixels).
In our approach, we omit the projector and predictor components present most contrastive learning approaches [15] 15 7 since found empirically that this gave better performance. We trained our model using Adam W optimizer with initial learning rate 5e and weight decay 0, batch size 32 on RTX 2080 GPUs for 50 epochs. Training took approximately 5 hours in clock time. Our pretrained weights can be found at hhtps::/tinyrll.com http://wwl.oo//a///ao3/a and the training code on our GitHub repository https://githubb.c www.://w........o.eeeeee/ee Larehggllb/LMMM All pre-trained weights for other models directly loaded from the corresponding released codebases.
Segmentation Models: We finetuned the pretrained FreeSOLO 47 model on 1K scenes of ABC
dataset with instance mask annotations. To obtain the predicted instance masks for low-shot, we performed forward pass of the fine-tuned model on our low-shot data. From the output masks,
20

---

we retained the ones with confidence score above 0.5, To handle overlapping masks, we merged those with an IoU greater than 0.7. Finally, we employed the Hungarian matching algorithm [22] to associate each predicted mask with its corresponding ground truth mask. We finetuned FreeSOI.O with batch size RTX 2080 GPUs for 30K epochs.
21


Reviews:
[{"review_id": "OdmlPlgjDA", "reviewer": "Reviewer_BMRQ", "summary": "", "questions": "", "limitations": "1. It mentions that the proposed task is far from the real-world scenario where multiple objects in a scene can be unfamiliar.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The proposed task is meaningful for measuring generalization.\n1. L98 The related settings paragraph provides clear distinction from the existing tasks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "vGuWdqNKNA", "reviewer": "Reviewer_xYTB", "summary": "", "questions": "", "limitations": "Currently, there is only one novel object in a scene. This setting might be too easy and can not reflect the real world.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The first to explore mutual exclusivity via Low-shot Object Learning with Mutual Exclusivity Bias(LSME);\nMultiple baselines are evaluated;", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JUUddb4Xjj", "reviewer": "Reviewer_FRCv", "summary": "", "questions": "", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work. No foreseeable negative societal impact.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper effectively utilizes the concept of inductive biases, specifically the mutual exclusivity principle, that humans use to learn and recognize objects in their environment. This concept is cleverly framed into a computational problem (LSME) in the context of machine learning.\n\n- The problem setting and the task are well-motivated and relevant. Modeling such inductive biases could be essential for improving machine learning models' abilities to understand and interact with their environment. The authors provide a clear explanation of the mutual exclusivity bias and how it can be applied to machine learning tasks, bridging the gap between cognitive science and artificial intelligence. \n\n- The authors experiment with several state-of-the-art models and are candid about where these models fall short. They propose an alternate baseline that considers the shortcomings of existing models and provides improved performance.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Ht93Jd5zra", "reviewer": "Reviewer_HVNj", "summary": "", "questions": "", "limitations": "As mentioned in the submission, this work is somehow limited to one single unknown object in the scene image. Further technical improvement is required for extending LSME to multiple unknown objects.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- This work studies a novel low-shot learning setting, in which the learner needs to understand a new object concept with mutual exclusivity bias. A few scene images are given to the learner, which include both seen objects and unseen objects. The learner needs to reason the existence of previously not learned objects and build a connection between the unseen object and the new concept. This learning task presents a new challenge, as well as new opportunities for various real-world applications such as robot learning and continual learning.\n\n- Various pre-trained visual representations are evaluated, including vision-only representations (e.g., DINO) and vision-language representations (e.g., CLIP). These different representations can help understand the role of language-based learning in gaining new object concepts.\n\n- The proposed LSME solution is general and intuitive. It mainly includes three sub-solutions, object localization, open-world recognition, and low-shot learning. This framework can be extended when better solutions in these related topics are available.\n\n- The writing is clear and easy to follow.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: In this paper, the author proposes a new benchmark method and evaluates its performance. The Performance benchmark testing is performed on multiple baseline models, including methods based on the basic model.  And a new benchmark method is proposed to achieved state-of-the-art results in LSME tasks. Specifically, the author evaluates the performance of the benchmark method by measuring metrics such as support allocation accuracy (SA), low sample accuracy (LSA), and mIoU for instance segmentation.
Decision: Accept (Poster)