SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anh Thai $^{1}$  Ahmad Humayun $^{2*}$  Stefan Stojanov $^{1*}$  Zixuan Huang $^{3}$
Bikram Boote<sup>1</sup> James M. Rehg<sup>1,3</sup>
<sup>1</sup>Georgia Institute of Technology, <sup>2</sup>Google Deepmind,
<sup>3</sup>University of Illinois, Urbana-Champaign
This paper introduces Low-shot Object Learning with Mutual Exclusivity Bias (LSME), the first computational framing of mutual exclusivity bias, a phenomenon commonly observed in infants during word learning. We provide a novel dataset, comprehensive baselines, and a state-of-the-art method to enable the ML community to tackle this challenging learning task. The goal of LSME is to analyze an RGB image of a scene containing multiple objects and correctly associate a previously-unknown object instance with a provided category label. This association is then used to perform low-shot learning to test category generalization. We provide a data generation pipeline for the LSME problem and conduct a thorough analysis of the factors that contribute to its difficulty. Additionally, we evaluate the performance of multiple baselines, including state-of-the-art foundation models. Finally, we present a baseline approach that outperforms state-of-the-art models in terms of low-shot accuracy. Code and data are available at https://github.com/rehglab/LSME.
sandwich
Training
Standard Low-shot Learning
car
Input: Single object image  
Task: Learn to classify object
Low-shot Learning with Mutual Exclusivity Bias
sandwich
Training
Input: Scene image with seen and novel objects  
Task: Identify novel object
Testing
Figure 1: We propose Low-Shot Object Learning with Mutual Exclusivity Bias (LSME), a novel setting that is more realistic and significantly challenging than the standard low-shot object learning setting. Given a scene with multiple objects including seen and novel objects, the goal of LSME is to use mutual exclusivity bias to correctly identify the unfamiliar object and generalize this association to other instances of the novel category at testing time.
Input: Single object image  
Task: Classify object
Input: Scene image with known and novel objects  
Task: Localize and classify novel objects
Toddlers are incredible learners, acquiring an average of 10 to 20 new words per week during a period known as the vocabulary spurt [29]. This is possible in part because of effective inductive biases that help them to overcome the inherent ambiguity that arises in inferring which real-world object a spoken object name (e.g., provided by a caregiver), is referring to. Inductive biases like shape bias have received more interest [44, 35, 36], due in part to widespread interest in the link between 3D object shape and categorization. In contrast, the key inductive bias of mutual exclusivity has not yet received significant attention [11, 27, 28]. Mutual exclusivity leverages the assumption that each object has only one label. Thus when presented with a scene containing multiple objects, some familiar and some unfamiliar, mutual exclusivity guides the child to bind new object names to the unfamiliar objects. For example, consider a toddler playing with three toys: her favorite Ducky, a Mickey Mouse figure, and a toy that she does not know the name of. When the caregiver says, "Look at the dinosaur," the toddler uses mutual exclusivity bias to rule out Ducky and Mickey, as she already knows their names. She then associates the word "dinosaur" with the unfamiliar object. After this mapping is made, the toddler can retain the category label "dinosaur," and use it later to label other dinosaur-like objects. The goal of this work is to create a novel dataset and associated baselines to spur the ML community in developing computational models for mutual exclusivity, as a further step towards modeling the inductive biases that underlie human object learning performance.
Figure 2: Our proposed LSME (Low-shot Object Learning with Mutual Exclusivity Bias) setting. We leverage mutual exclusivity bias observed in humans to associate a novel word label with an unfamiliar object in a scene. Given a set of familiar categories (base classes), the task is to use this bias to correctly identify the unfamiliar object and generalize this association to other instances of the novel category.
This paper introduces Low-shot Object Learning with Mutual Exclusivity Bias (LSME), a computational framing of mutual exclusivity as an inference problem, with low shot learning as the downstream measure of generalization performance (see Fig. 2 for an overview and Table 1 for related settings). The LSME task begins with a set of already-known object categories. The input is an RGB image containing multiple object instances from known categories and a single object instance of an unknown category, along with a novel category label. Solving the LSME problem requires three steps: 1) Localize the object instances within the scene via segmentation; 2) Associate the novel category label with the unknown category instance that it refers to; and 3) Solve a low-shot learning task by classifying other object instances from the novel category during inference. The ability to reliably solve LSME would be an important step in creating agents that can learn with minimal supervision. For example, household robotics is a long-standing goal of AI. Our work could ultimately yield a robot capable of rapidly learning the names of novel objects in the home, without the need for a burdensome supervised training stage.
LSME can be partitioned into three sub-tasks: 1) object localization, 2) open-world recognition, and 3) low-shot learning. The model must first localize each object in the scene. It must then distinguish between learned and unknown categories to correctly assign the novel label to the unknown instance. Finally, the model must generalize to new instances of the novel categories after only one or a few labeled samples. At the heart of LSME is the challenge of associating novel object instances with their category labels. In supervised pre-training, there is a danger of leakage if the datasets used in pre-training overlap with the low-shot category labels used in testing. In order to prevent this, we
Table 1: Comparative analysis of LSME and other related settings. LSME requires a comprehensive understanding of scenes that involve the presence of multiple objects.
Settings Properties	Low-shot Recognition	Open-set Detection	Low-shot Detection	Object Discovery	GCD [45]	NCDL [9]	Open Category Detection	Image Conditioned Detection	LSME
Instance Localization	X	✓	✓	✓	X	✓	✓	✓	✓
Mutual Exclusivity Bias	X	X	X	X	X	X	X	X	✓
Discover Novel Classes	X	✓	X	X	✓	✓	✓	X	✓
Label Novel Classes	✓	X	✓	X	✓	✓	✓	X	✓
Zero/Low-shot	✓	X	✓	✓	X	X	✓	✓	✓
No Pretrained LLM	✓	✓	✓	✓	✓	✓	X	✓	✓
constrain the LSME problem design so that pre-training prior to the low-shot phase cannot include any object-label association data (e.g. as in CLIP). This constraint further aligns with the investigation of a key question in developmental psychology: How can a child rapidly learn words in the absence of prior knowledge of object semantics? [51, 26]
It's natural to ask if LSME can be solved by simply combining existing SOTA models. We find that this is not the case. Errors in the first two sub-tasks propagate to the low-shot stage and degrade accuracy. For example, complex spatial interactions (e.g. occlusions) between objects make reliable instance segmentation challenging, leading to poor feature extraction for each object. Furthermore, any errors in identifying novel vs. known objects will result in label noise that degrades low-shot recognition. In addition, pre-trained feature representations from foundation models (e.g. DINOv1 [5] and DINOv2 [33]) are not sufficient to solve our ask effectively (see Sec. 4).
One challenge we address in this paper is the creation of suitable LSME datasets. Since real-world data is not available, we introduce a generic data generation pipeline that can take any categorical 3D dataset as input and render training and testing data with realistic backgrounds, lighting variations, and realistic object poses. Our dataset is structured with increasing levels of complexity in order to aid in understanding the shortcomings of learning approaches.
Additionally, we benchmark the performances of various baselines on LSME, including SOTA foundation models with robust visual representations such as DINOv2 [33], ImageBind [12], and CLIP [37]. Interestingly, we find that the performance of these baselines degrades significantly when there are occlusions/incomplete instance masks (see Sec. 4.3).
From these observations, we propose a baseline that involves pretraining the object representations on cluttered scenes with occlusion. Following the success of prior works that show performance improvement when training on multi-view inputs [43, 17], we propose self-supervised training on the large-scale multi-view multi-object ABC [21] dataset using contrastive learning. Our baseline composed of the robust 2D representations from foundation models and 3D features from multi-view inputs demonstrates improvement in accuracy compared to existing models when occlusions are present by approx.  $10\%$ . To verify that this gain is significant, we evaluate our approach on the real-world dataset CO3D [38] and show the benefit of multi-view training with occlusion.
In summary, our contributions are 4-fold: 1) The first to provide a computational framing of mutual exclusivity via LSME; 2) A dataset generation pipeline that enables the creation of progressively more challenging LSME tasks using any set of 3D models; 3) Performance benchmarking of multiple baselines including foundation models on our novel task; and 4) A novel baseline method that defines the SOTA on LSME.
Because we are the first to provide a computational framing of mutual exclusivity (ME), there is no direct prior work to compare to. [11] demonstrated that existing deep models fail at ME, but did not provide a comprehensive framing or SOTA methods. We now review three bodies of related work.
Self-supervised and Weakly-Supervised Learning Self-supervised learning aims to leverage the inherent structure in visual data to train representations suitable for downstream tasks, without relying on labels. Contrastive methods [7, 15, 8] work by treating image pairs under different data augmentations as positive examples, and other sample pairs as negatives. DINOv1[5] enforces "global-local" correspondences by using multiple image crops, while other methods [48, 55, 31] focus on pixel-level matching or combine both objectives [33]. On the other hand, VISPE [17],
Figure 3: Rendered scenes used for LSME. First column has scenes of ABC [21] objects that are used for pretaining. Second and third columns show data for low-shot evaluation on Toys4k [44] and ShapeNetCore.v2 [6]
DOPE [43], and VSA [23] incorporate multi-view data by treating samples captured from different viewpoints as positive pairs. These approaches leverage 3D structure to gain improvements.
In contrast, recent weakly-supervised learning foundation models such as CLIP [37] and Image-Bind [12] take advantage of multi-modal signals, including language and audio in addition to visual information. Another body of work [14, 52, 50] attempts to learn the underlying visual structure of data by learning to reconstruct images from heavily masked inputs. In this work, we study the performance of these methods on our novel task. Moreover, we extend these representations by incorporating multi-view multi-object information. By integrating data from multiple viewpoints and considering information at the object level instead of the scene or pixel level, we aim to learn object representations that can reason about the spatial layout and category composition of scenes with multiple objects.
Related Settings Vaze et al. proposed Generalized Category Discovery (GCD) [45], a setting that allows input to come from both known and unknown distributions, and further aims to label novel data. We differ from this setting in that we do not assume object localization information. We further consider scene-centric data that provides a more holistic view of the entire scene instead of object-centric data as input. This enables our model to take into account the relationships between multiple objects within the scene.
Fomenko et al. [9] introduce the novel class discovery and localization (NCDL) setting, which closely relates to our proposed LSME task. Both tasks share the goal of localizing and classifying unknown objects in a scene. However, we operate in a low-shot setup, where only a limited number of samples from the novel categories are available. This contrasts with NCDL, which focuses on balancing a long-tail distribution of novel classes with varied number of samples per class. In addition, our setting requires mutual exclusivity bias—explicitly learning ambiguous word-object associations.
Open-category object detection and segmentation [53, 19] have recently gained popularity for its generalization ability beyond the trained vocabulary. This task focuses on using large pre-trained vision-language models to detect and segment known and novel objects via prompting the LLMs. In contrast, our task focuses specifically on learning the association between objects and their corresponding labels. These labels may not necessarily align with the vocabulary typically used for LLM pretraining (e.g. using pseudo names for toy objects like "dax" as commonly seen in psychology experiments [42]). LSME can flexibly accommodate various vocabulary systems that potentially is a challenge for pretrained LLMs.
Image-conditioned object detection [30, 34] is a related setting where language input is not required. The goal of this setting is to detect objects that share the same semantic characteristics as the template objects given as one or a few input images. Our setting solves a more general and challenging task where the model is required to identify the unknown object of interest before generalizing to other object instances from the novel category. LSME is further related to low-shot learning [43, 54, 25], open-world learning [46, 4], and object instance segmentation [47, 49, 24] settings as it integrates these components into a holistic solution. (See Tab. 1)
Computational Frameworks Motivated by Developmental Psychology Recent works have investigated various learning scenarios to explore the strategies children employ when acquiring new concepts [18, 3, 16]. This work complements these efforts by introducing a comprehensive benchmark that studies the mutual exclusivity bias commonly observed in infants during the initial stages of word learning.
Synthetic 3D Datasets and Dataset Generators Synthetic 3D datasets and environments have been explored to study a wide range of tasks [44, 43, 50, 13, 41] where real-world data might not be available at a large scale. Recent realistic data rendering engines [13, 10, 41] and image generation models [39, 40] have significantly reduced the sim2real domain gap, enabling more effective generalization to real-world settings. In this work, we leverage the benefits of synthetic data to tackle a novel problem where real-world datasets are unavailable. This allows us to create diverse and customizable scenarios that closely resemble real-world settings to study the problem in a controlled and scalable manner.
Figure 4: Different variants of LSME setting. Left: Single-object case where only a single object is present in the scene for both supports and queries and Right: Multi-object setting where multiple objects are present in the scene. Bounding boxes and texts indicate given localization information and labels respectively. The difficulty levels of these variants increase from left to right, with the hardest setting—our proposed LSME on the rightmost column requiring the models to achieve all 3 properties: localization, mutual exclusivity bias, and low-shot.
3 Low-shot Categorization Using Mutual Exclusivity Bias (LSME)
3.1 Task Formalization
Consider an RGB scene containing multiple objects, including known instances and a single novel instance that lacks any explicit localization information and a word for its label. Our task can be partitioned into solving three well-known subtasks:
Object Localization. The first step is to localize the objects in an RGB scene.
Open-world Recognition. The second step is to differentiate between known and novel categories. The objective is to identify the instance within each scene that belongs to a novel category. To enable this, we assume the availability of a collection of labelled images from known categories, referred to as the base classes. Once the novel object identified it is associated with the novel word label, which we refer to as a support object.
Low-shot Learning. The last step is low-shot generalization. Given one or a few query images of objects from the novel categories, the goal is to correctly classify them based on the support objects.
Data Generation Pipeline. Given any 3D categorical dataset, following the convention of low-shot learning, we first partition these object categories into disjoint sets: base classes and low-shot classes. While base classes are commonly used to learn robust feature representations, low-shot classes are for evaluating the generalization ability of the models in a low data setting.
Data Rendering. To generate each scene, we first randomly select a subset of objects. The rotational poses for the objects are obtained using rigid body simulation. The objects are scaled and placed into the scene at random locations making sure collisions do not occur. Scenes are generated with varying background, illumination, and camera viewpoint. Please refer to the Supplement for more detailed descriptions of the generating process.
Figure 5: The process of Mutual Exclusivity Bias in our models. The input is the support scene with a novel word label. The model first needs to 1) localize the objects in the scene, 2) extract feature embeddings for these objects and compare with the features of the objects from the base classes using cosine distance, 3) take the maximum similarity score between each object embedding and the objects in the feature bank, 4) determine the unknown object by taking the object with the minimum similarity score and then assign the novel word label to this object.
3.2 Data Variants
To support a comprehensive analysis and gain insights into the limitations of different methods, we generate data with increasing difficulty levels of variability (please see Fig. 4).
The first group of variants considers scenes with a single object (-SObj) (left side of Fig. 4). The simplest setting concerns object instance recognition (Inst-SObj)—recognizing the same object based on a few images in the same pose. Our second group of variants requires models to generalize to different instances of the same category (Categ-SObj), while our third setting also takes into consideration random initial object pose sampling (Categ-SObj-PoseVar). Note that Categ-SObj is similar to the standard low-shot learning setting [44, 43]. These settings do not require mutual exclusivity bias.
The second group of has multiple objects (-MObj) (right side of Fig. 4). For the first, labels for the shots are given (Categ-MObj). This is similar to Categ-SObj-PoseVar, except the objects might be partially occluded. The second variant requires using mutual exclusivity bias to assign the label to the correct object before low-shot generalization (Categ-MObj-SuppAssign). Finally, LSME also requires object localization.
Note that while the first data variant (Inst-SObject) yields a straightforward task that does not require category generalization, all subsequent variants (Categ-SObject, Categ-SObject-PoseVar, and Categ-MObject) require instance-to-category generalization, as in the standard low-shot learning setting. Although not requiring mutual exclusivity bias, these variants are significantly more challenging in comparison to the standard low-shot learning formulation, because they introduce pose variability and include multiple objects, as described in Sec. 4.
3.3 Our Approach
Representation Learning. Motivated by the understanding that multi-view observations provide rich visual information about the 3D environment that aids downstream tasks [43, 17], we adopt a strategy of pretraining the feature extractor by leveraging contrastive learning on multi-view scenes. Unlike most self-supervised approaches that primarily operate on the scene level even when the scene consists of multiple objects, we focus on the object-level representation learning. Specifically, given scenes of multiple objects captured from various viewpoints, along with the corresponding instance masks for each object, we learn to match the representation of the same object across different views.
We use pre-trained backbones, (e.g. DINOv1 [5], DINOv2 [33]) contrastive training strategy with a momentum encoder [15]. Given two views of the same scene,  $v_{1}$  and  $v_{2}$ , we first use the instance mask associated with each object in the scene to eliminate the background and other objects. Subsequently, we extract the query object feature by performing a forward pass of the image encoder on  $v_{1}$ . For each query feature, we minimize the InfoNCE [32] loss function. The positive sample is the feature of the same object in  $v_{2}$  while the negative set consists of object features from the memory queue as in MoCo-v2 [8] and different objects from the same scene. This approach is inspired by the VISPE++
Table 2: Low-shot recognition on the Toys4k dataset in the single object setting. All methods consistently drop in accuracy when evaluated on the harder data variants.
DINOv1-S/8	DINOv2-S/14	DINOv2-B/14
Variants	1-shot 5-way	1-shot 10-way	1-shot 5-way	1-shot 10-way	1-shot 5-way	1-shot 10-way
Inst-SObj	95.80	92.37	95.75	93.06	96.50	94.22
Categ-SObj	73.06	60.73	77.11	66.62	79.69	69.55
Categ-SObj-PoseVar	68.84	57.45	73.07	61.44	75.18	66.30
Table 3: Results on low-shot recognition on the Toys4k dataset in multi-object setting. All methods consistently experience a significant drop in low-shot accuracy when mutual exclusivity is required, and further decrease when instance segmentation is involved.
DINOv1
ViT S/8	DINOv2
ViT S/14	DINOv2
ViT B/14	CLIP-Img
ViT B/16	ImageBind
ViT H/16
Variants	LSA	SA	LSA	SA	LSA	SA	LSA	SA	LSA	SA
Categ-MObj	56.99	N/A	56.95	N/A	57.92	N/A	56.76	N/A	60.49	N/A
Categ-MObj
-SuppAssign	40.21	51.68	41.26	52.28	43.21	54.96	41.22	51.64	45.91	58.58
LSME	36.44	46.92	37.08	48.16	39.24	50.88	38.25	48.96	38.85	50.24
baseline in [43]. For each input view pair, we ensure to only train on objects that are visible in both views (e.g. with instance segmentation area greater than some threshold  $\sigma$  pixels)
Unsupervised Instance Segmentation. We use a fine-tuned FreeSOLO [47] model as our segmenter, benefiting from its strong performance in instance segmentation tasks. Fine tuning is done on 1K scenes from the ABC dataset, which have been annotated with instance segmentation masks.
Low-shot Learning. We apply the following support assignment and low-shot generalization technique in all methods.
Support Assignment: We first collect a feature library consisting of object features extracted from the base classes as learned objects. During the low-shot training phase, for each input scene, our goal is to determine which object should be assigned the novel label. For every object in the scene, we calculate its similarity with each object in the feature library using cosine distance. To determine the final similarity score with the feature library, for each object in the scene, we select the maximum similarity measure across all objects in the library. Finally, the object in the scene with the lowest similarity score is assigned the novel label. Under this procedure, a learning agent would associate the novel label to the object that it is least familiar with (Fig. 5).
Low-shot Generalization: Given a query scene, we compute the cosine distance between each object in the query scene and every support object identified in the support assignment phase. We then determine the nearest neighbor for each query object within the support set and assign the label of the query object to its closest counterpart in the support set.