SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset. Similar to prior works [44, 43] and the convention in low-shot learning, we partition the categories of ShapeNetCore.v2 [6] and Toys4k [44]. These datasets are divided into two disjoint subsets: base classes, for learning object-label associations, and test classes, for evaluating low-shot object recognition performance. We generate 1K scenes each for support and query sets, and 500 base scenes for multi-object experiments. Each scene in our dataset consists of 3 objects rendered in 20 views. To study mutual exclusivity bias, we ensure that each scene in the support set only contains 1 object from a novel category while the remaining objects come from base classes. We further incorporate a subset of CO3D [38] that shares 13 categories with the test classes in Toys4k to investigate the effectiveness of occlusion-aware feature representations in the real-world scenario. Since CO3D is an object-centric dataset, it is not possible to directly test LSME on CO3D. We only evaluate this dataset on the standard low-shot setting where mutual exclusivity bias is not considered.
We pre-train our models using the 3D object instance dataset ABC [21] without any category structure. We use a subset of ABC that consists of 100K objects. We render 10K scenes with 20 views per scene (8K for training and 2K for validation), each containing 2-3 objects with random object poses
Figure 6: Low-shot accuracy of DINOv1 and DINOv2 pre-trained models on the Single-object settings of Toys4k. The size of the markers corresponds to the number of parameters of the models.
Figure 7: Performance of pre-trained models on the Multi-object settings of Toys4k. Circle and triangle markers represent self-supervised vision models and models that have linguistic input respectively.
Table 4: Performance of different methods on Toys4k under Categ-SObj-PoseVar and Categ-MObj settings. These settings solve a similar problem, with Categ-MObj having object occlusions present in both support and query objects. Performance of all methods drops significantly when faced with occlusion cases.
Method	DINOv1 S/8	DINOv2 S/14	DINOv2 B/14
Categ-SObj -PoseVar	68.84	73.07	75.18
Categ-MObj	56.99	56.95	57.92
and scales. We further randomize the textures and surface materials of the objects in each scene (refer to Fig. 3).
Evaluation Protocols. We evaluate the performance of the baselines using the following metrics: 1) support assignment accuracy (SA) which quantifies the percentage of accurately identifying the novel instance within the scene, and 2) low-shot accuracy (LSA) for measuring low-shot performance, and 3) mIoU for instance segmentation. In this work, we follow the standard framing of low-shot inference, such that "1-shot 5-ways" means that each episode has 5 novel categories, each with only 1 object during the low-shot training phase. Unless stated otherwise, the experiments in this paper are evaluated on the 1-shot-5-way setup with 500 episodes. We report the confidence intervals and experiments on other setups in the Supplement. We make sure that novel objects are clearly visible during both support assignment and low-shot generalization phases (e.g. with instance segmentation area greater than threshold  $\sigma$  pixels).
Representation Learning Baselines. Our main design constraint is that pre-training prior to the low-shot phase cannot incorporate any object labels, in order to avoid any concerns of label leakage. We focus on analyzing the performance of models initialized from the SOTA self-supervised vision models DINOv1[5] and DINOv2 [33]. This allows us to gain further insights into whether LSME can be tackled without prior language inputs. We further investigated the performance of large-scale vision-language foundation models, in order to characterize the difference between self-supervised and weakly-supervised pretraining approaches. We present findings using CLIP [37] and ImageBind [12] pre-trained models. Note that these models violate the constraint by having unrestricted (in terms of category composition) image captions in their pre-training.
4.2 LSME and Related Tasks on Synthetic Data
We first analyze the performance of pre-trained representations on settings with varying difficulty, building up to LSME.
Single Object Setting. We present the results for the single object setting on Toys4k in Table 2 and Figure 6 on 1-shot 5-way and 1-shot 10-way set ups. We observe a decrease in performance when tested on the harder variants for all models. While the difference between Categ-SObj-PoseVar and Categ-SObj is not significant, we observe a more prominent gap between Categ-SObj and Inst-SObj. This indicates the challenge faced by the models when generalizing from instance to category level.
Multiple Objects Setting. We report these results in Table 3 and Figure 7. The performance of all models exhibits a significant decrease ( $\sim 15\%$ ) when mutual exclusivity bias is required (2nd row).
Table 5: Performance of DINOv2 and our method fine-tuned on Toys4k and ABC on Toys4k under LSME setting. All methods use ViT B/14 as the backbone and our method is initialized with pretrained DINOv2 weights. Training on ABC improves the performance significantly, surpassing the model that was trained on the base classes of Toys4k with the same number of scenes.
Method	LSA	SA
DINOv2	39.24	50.88
Ours-DINOv2-Toys	43.62	53.44
Ours-DINOv2-ABC	47.70	61.32
Table 6: Performance of our baseline finetuned with different backbones on Toys4k under LSME settings with four object segmenters. Best performance is highlighted in bold while underline represents second best performance. The quality of the instance masks plays a significant role in the low-shot and shot assignment performance for all methods.
mIoU	Ours-DINOv1 S/8-ABC	Ours-DINOv2 S/14-ABC	Ours-DINOv2 B/14-ABC
Support	Query	LSA	SA	LSA	SA	LSA	SA
FreeSOLO [47]	0.52	0.54	32.31	43.24	33.99	44.84	35.50	48.92
CutLer [49]	0.61	0.63	34.62	42.40	36.34	46.08	39.42	52.04
SAM [20]	0.72	0.73	35.72	48.96	38.58	52.04	42.38	56.92
FreeSOLO-ABC	0.81	0.83	41.19	55.40	43.92	57.88	47.70	61.32
Table 7: Performance of DINOv2 and ours using ViT B/14 at LSME on ShapeNetCore.v2. Our method demonstrates an improvement in both low-shot and shot assignment accuracy.
Method	LSA	SA
DINOv2	32.55	47.68
Ours-DINOv2-ABC	41.03	58.92
This decline can be attributed to the challenges associated with imperfect support assignment, which negatively affects the low-shot accuracy across all models.
Effect of Segmentation Quality. Table 6 provides additional evidence of the significance of the quality of the instance masks generated by the segmenter. It demonstrates that a decrease of 0.1 in mIoU (between the last 2 rows) can result in a substantial impact on performance, with approximately a  $6 - 7\%$  decrease in low-shot and support assignment accuracy. This emphasizes the critical role that accurate instance masks play in achieving high-performance results in both LSA and SA metrics.
Effect of Occlusion. Table 3 highlights the impact of occlusion caused by other objects in the scene on the low-shot performance of the models. While Categ-SObj-PoseVar and Categ-MObj address a similar problem, Categ-MObj considers potential occlusion in both support and query objects due to multiple objects being in the scene. We observe a decrease in performance for all models when occlusion is present. Specifically, DINOv1 experiences an average decrease of  $11.85\%$ , while DINOv2 models show greater decreases of  $16.12\%$  and  $17.26\%$  respectively.
These findings indicate the challenge posed by occlusion in low-shot learning scenarios. The presence of occlusion introduces additional complexity and ambiguity, making it more difficult for the models to accurately associate novel objects with their corresponding categories. This highlights the importance of developing methods purpose-built for the task of low-shot learning with mutual exclusivity.
Contrastive Finetuning on ABC Improves Performance. In Table 5 we show the advantage of contrastive finetuning on the ABC [21] dataset. While prior work [43] has demonstrated the benefits of pretraining feature representations on ABC, we are the first to show the advantages of ABC in a multi-object setting. Additionally, we train our model on an equal number of scenes from the base classes of Toys4k. Although this approach enhances the low-shot generalization performance on the test classes of Toys4k compared to DINOv2, its performance is inferior to the model trained on ABC. This highlights the benefits of leveraging ABC in the context of multi-object understanding and its potential for improving the capabilities of models in complex scenarios.
In addition, the performance comparison between DINOv2 with ViT B/14 backbone and our model fine-tuned on ABC is presented in Tab. 7 on the ShapeNetCore.v2 dataset. Notably, our model consistently outperforms DINOv2 across both the LSA and SA metrics, showcasing an improvement of approximately  $10\%$ .
4.3 Low-shot Generalization on CO3D Dataset
We investigate the generalization capabilities of pretraining on the multi-object multi-view ABC dataset, extending beyond the synthetic data domain to real-world datasets. Considering the negative impact of object occlusions and incomplete object masks on the performance of models, we hypothesize that pretraining on ABC scenes with multiple objects where occlusions are present improves the models' ability to handle such scenarios. To test this hypothesis, we conduct an experiment where we randomly mask the instance segmentations and evaluate the models on these masked segmentations in the standard low-shot setting. Figure 8 showcases the performance of different approaches
on CO3D dataset with varying mask ratios. The models finetuned on ABC exhibit a better performance compared to other models as the mask ratios increase. Even at a mask ratio of 0.5, our ABC-finetuned model with DINOv1 ViT S/8 pretrained weights performs on par with DINOv2 ViT G/14. Note that DINOv2 ViT G/14 has significantly more parameters and was trained on an order of magnitude more data than our model. These results demonstrate the benefit of occlusion-aware feature representations.
4.4 Limitations & Future Work
One important limitation of our work is the absence of uncertainty reasoning. In real-world scenarios, agents often encounter multiple unknown objects, requiring the integration of novel labels acquired in diverse contexts to correctly associate objects with their corresponding labels. This more complex setting requires continuous integration of new information and reasoning in ambiguous situations. Another challenging real-life infant-learning scenario is where objects might have multiple names (e.g., "dog" and "husky" both referring to the breed "husky"). Due to the procedural nature of our data generation system, we have the ability to increase the task complexity as solutions to LSME improve beyond the current baselines. Future research can explore more challenging settings and developing approaches that incorporate uncertainty reasoning to improve performance in ambiguous scenarios.
Negative Societal Impact. Training and evaluating large-scale self-supervised learning models as well as generating data require extensive GPU usage, which negatively impacts the environment. Advancements in hardware design and techniques for optimizing deep models offer potential solutions to mitigate this impact.
We present a novel setting called Low-shot Object Learning with Mutual Exclusivity Bias (LSME), which requires comprehensive reasoning about scenes with complex object interactions. We conduct a thorough analysis of the challenges present in LSME and their impact on SOTA models by generating various problem variants. Based on these insights, we propose a pretraining strategy that outperforms SOTA baselines on both synthetic and real-world data. Additionally, we release our open-source data generation pipeline and the generated datasets for further research.
The challenging LSME task provides a framework for devising algorithms that can learn from limited data input. We hope that LSME is an example computational framework that can serve as a template to formulate learning tasks based on other insights from the developmental psychology community. Future development of this setting can provide a learning environment that closely resembles real-world scenarios, effectively for further studies of both computer vision and developmental psychology communities.
Figure 8: Performance of different methods on the segmentation masking experiment on CO3D. Low-shot performance of these methods decreases as the mask ratio increases. Models pretrained on ABC exhibit a slower decrease rate.
This work was supported by NIH R01HD104624-01A1.
[54] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8808-8817, 2020.
[55] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. Dense siamese network for dense unsupervised learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX, pages 464-480. Springer, 2022.
This appendix is structured as follows: We first provide more details about data information in Section A. We then show additional results in Section B. Finally, we provide additional training details about our baseline models in Section C.
In our work, we performed experiments and analysis using three datasets: Toys4K [44], ShapeNet-Core.v2 [6], ABC [21], and CO3D [38]. In the following section, we provide comprehensive details about each of these datasets.
Toys4K [44]. This dataset consists of 4,179 object instances in 105 categories. We use the base and low-shot splits provided by Stojanov et al. [44]. In particular, the base classes consist of 40 categories while the low-shot classes have 55 categories. Objects in this dataset were collected under Creative Commons and royalty-free licenses. (Please refer to Table 8 for base/low-shot split compositions).
ShapeNetCore.v2 [6]. This dataset consists of 52K objects in 55 categories. We partition these categories into 25 base and 30 low-shot classes (see Table. 8). The terms of use for ShapeNet are specified on their website, which can be accessed at https://shapenet.org/terms.
ABC [21]. For pretraining our representation learning models, we used a subset of 100K object instances from ABC, which contains a total of 750K instances. Note that this dataset lacks categorical structures. The dataset is distributed under the MIT license. More licensing information is available at https://deep-geometry.github.io/abc-dataset/#license.
CO3D [38]. We chose the 13 classes out of 51 classes that overlap with Toys4K for low-shot validation, detailed in Table 8. The terms of use for CO3D are specified at https://ai.facebook.com/datasets/co3d-downloads/.
A.2 Data Generation
Software. We used Blender 2.93 [1] with ray-tracing renderer Cycles for data generation and rendering.
Assets. Objects are placed on top of a plane that simulates the ground/floor with PBR materials and image-based lighting from HDRI environment maps are used to illuminate scenes. We collected these assets from PolyHaven [2]. The list of assets used is shown in Table 9.
Scene Generation. Given any 3D categorical dataset, we first partition these object categories into disjoint sets: base classes and low-shot classes. For each object in the dataset, we preprocess it by simulating a rigid body drop using Blender [1]. This simulation process is repeated 16 times, allowing us to collect metadata and initial rotational poses for each object. These collected data are used in the subsequent stages of scene generation.
To generate each scene, we first choose a subset of objects from the dataset. Their initial rotational poses are determined by randomly choosing from the preprocessed poses. Objects are then scaled and placed into the scene at random locations. We ensure that collisions do not occur by maintaining a minimum margin of  $\Delta > 0$  between each pair of objects. We randomize the scene background by randomly choosing a pair of PBR material and HDRI environment map from the assets.
Data Rendering. To render each view of the scenes, we first determine the camera position. The camera's position in the scene is specified by three parameters:  $\theta \in [0,2\pi ]$ ,  $r\in [r_{min},r_{max}] > 0$  and  $z\in [z_{min},z_{max}] > 0$  where  $\theta$  is the rotational angle,  $r$  is the distance from the origin in the XY-plane, and  $z$  denotes the world Z-coordinate of the camera. Note that  $r_{min},r_{max},z_{min},z_{max}$  are preset parameters. The world coordinate of the camera is computed by  $(r\cos (\theta),r\sin (\theta),z)$ . To determine the camera's orientation, it is set to point towards a location on the XY-plane that is within a small distance  $\epsilon$  from the mean locations of the objects in the scene. This is done by rotating the camera in the world XY and YZ-planes. We then randomize illumination intensity, consistently for all the views of each scene.
Figure 9: Rendered scenes for LSME on Toys4k [44]
Generated Data for LSME. We generated 1K scenes for each of support and query sets, with each scene consisting of 20 views. The data generated for LSME evaluation can be found at https://tinyurl.com/3a9r83z9. Additionally, the code for data generation is available on our GitHub repository at https://github.com/rehg-lab/LSME. Detailed parameters for scene generation can be found in Table 10.
A.3 Data Augmentation for Contrastive Training
To augment the data, we applied various transformations, including random horizontal flips and brightness and color jittering. Following [43], we employed random object masking, where the object instance mask was used to eliminate the background. Additionally, we applied rotations and translations to the foreground object and incorporated background randomization techniques.
A.4 More Data Visualizations
Figure 9 showcases additional examples of rendered scenes from the Toys4K dataset [44]. These examples highlight the diversity found in the background, illumination conditions, and object poses within the scenes.
In Figure 10, we demonstrate the instance mask prediction of the FreeSOLO [47] model finetuned on 1K scenes of ABC. The quality of the predicted masks is essential to solving LSME.
ShapeNetCore.v2	Toys4k	CO3D
Base	Low-shot	Base	Low-shot	Low-shot
chair	piano	candy	boat	TV
table	train	flower	lion	mouse
bathtub	file	dragon	whale	car
cabinet	pistol	apple	cupcake	toaster
lamp	motorcycle	guitar	train	microwave
car	printer	tree	pizza	donut
bus	mug	glass	marker	orange
cellular	rocket	cup	cookie	sandwich
guitar	skateboard	pig	sandwich	bicycle
bench	bed	cat	octopus	banana
bottle	ashcan	chair	monkey	bowl
laptop	washer	ice-cream	fries	motorcycle
jar	washer	hat	violin	pizza
loudspeaker	bowl	deer mouse	mushroom
bookshelf	bag	penguin	closet
facet	pillow	ball	tractor
vessel	earphone	fox	submarine
clock	camera	dog	butterfly
airplane	basket	knife	pear
pot	remote	laptop	bicycle
rifle	stove	pen	dolphin
display	microwave	mug	bunny
knife	microphone	plate	coin
telephone	cap	chess piece	radio
sofa	dishwasher	cake	grapes
keyboard	frog	banana
tower	ladder	cow
helmet	keyboard	donut
birdhouse	sofa	stove
can	trashcan	sink
dinosaur	orange
bottle	saw
elephant	chicken
pencil	hamburger
key	piano
monitor	light bulb
screwdriver	spade
robot	sheep
bread	toaster
lizard
motorcycle
mouse
pc mouse
bus
helicopter
microwave
cell battery
drum
panda
TV
car
helmet
fridge
bowl
Table 8: Split composition of ShapeNetCovre.v2, Toys4K and CO3D
B.1 Evaluation Metric Details
We evaluate the performance of the baselines using the following metrics: 1) support assignment accuracy (SA) which quantifies the percentage of accurately identifying the novel instance within the scene, and 2) low-shot accuracy (LSA) for measuring low-shot performance, and 3) mean
PBR	HDRI
Carpet001	Aft Lounge
Carpet005	Anniversary Lounge
Carpet006	Balcony
Carpet007	Cabin
Carpet008	Cayley Interior
Carpet009	Children's Hospital
Carpet013	Colorful Studio
Carpet014	Entrance Hall
Fabric024	Fireplace
Fabric025	Hotel Room
Fabric028	Kiara Interior
Marble012	Lapa
Planks001	Lebombo
Planks009	Lythwood Lounge
Planks011	Lythwood Room
Planks013	Moonlit Golf
Planks014	Music Hall
Planks018	Photo Studio
Terrazzo001	Reading Room
Tiles001	Roof Garden
Tiles027	Small Empty House
Tiles071	Spiaggia Di Mondello
Tiles072	St Fagans Interior
WoodFloor005	Umhlanga Sunrise
WoodFloor028	Wooden Lounge
Table 9: List of assets used in data generation.
Parameter	Value
Camera r	[1.0, 1.1)
Camera z	[0.3, 0.5)
Camera jittering ε	0.01
Object scale	[0.35, 0.45)
Object location	[-0.5, 0.5)
Illumination intensity	[0.6, 0.8)
Object margin Δ	0.4
Table 10: Data rendering parameters.
Figure 10: Segmentation prediction results on Toys4K [44] using FreeSOLO [47] fine-tuned on ABC model
intersection-over-union (mIoU) for instance segmentation as detailed below. For each episode,
$$
S A = \frac {1}{N _ {s}} \sum_ {i = 1} ^ {N _ {s}} \mathbb {1} \{\hat {o} _ {i} = o _ {i} \}
$$
Table 11: Results on low-shot recognition on the Toys4k dataset in single object setting. All methods consistently experience a significant drop in accuracy when being evaluated on the harder data variants.
DINOv1-S/8	DINOv2-S/14	DINOv2-B/14
Variants	1-shot 5-way	1-shot 10-way	1-shot 5-way	1-shot 10-way	1-shot 5-way	1-shot 10-way
Inst-SObj	95.80±0.46	92.37±0.42	95.75±0.44	93.06±0.41	96.50±0.43	94.22±0.37
Categ-SObj	73.06±0.96	60.73±0.76	77.11±0.89	66.62±0.78	79.69±0.99	69.55±0.77
Categ-SObj-PoseVar	68.84±1.04	57.45±0.77	73.07±1.03	61.44±0.80	75.18±1.04	66.30±0.79
Table 12: Results on low-shot recognition on the Toys4k dataset in multi-object setting. All methods consistently experience a significant drop in low-shot accuracy when mutual exclusivity is required, and further decrease when instance segmentation is involved.
DINOv1
ViT S/8	DINOv2
ViT S/14	DINOv2
ViT B/14	CLIP-Img
ViT B/16	ImageBind
ViT H/16
Variants	LSA	SA	LSA	SA	LSA	SA	LSA	SA	LSA	SA
Categ-MObj	56.99 ±0.97	N/A	56.95 ±0.99	N/A	57.92 ±1.04	N/A	56.76 ±1.01	N/A	60.49 ±1.00	N/A
Categ-MObj -SuppAssign	40.21 ±1.10	51.68 ±1.95	41.26 ±1.15	52.28 ±1.86	43.21 ±1.21	54.96 ±1.89	41.22 ±1.16	51.64 ±1.87	45.91 ±1.25	58.58 ±2.00
LSME	36.44 ±1.08	46.92 ±2.04	37.08 ±1.05	48.16 ±1.87	39.24 ±1.17	50.88 ±1.91	38.25 ±1.14	48.96 ±2.03	38.85 ±1.14	50.24 ±1.98
Table 13: Performance of DINOv2 and our method fine-tuned on Toys4k and ABC on Toys4k under LSME setting. All methods use ViT B/14 as the backbone and our method is initialized with pretrained DINOv2 weights. Training on ABC improves the performance significantly, surpassing the model that was trained on the base classes of Toys4k with the same number of scenes.
Method	LSA	SA
DINOv2	39.24±1.17	50.88±1.91
Ours-DINOv2-Toys	43.62±1.29	53.44±1.89
Ours-DINOv2-ABC	47.70±1.26	61.32±1.86
where  $o$ ,  $\hat{o}$ , and  $N_{s}$  are ground truth objects, predicted object, and the number of support objects respectively (e.g. in the 1-shot-5-way setup  $N_{s} = 5$  since there are 5 support objects in the episode.)
$$
L S A = \frac {1}{N _ {q}} \sum_ {i = 1} ^ {N _ {q}} \sum_ {k = 1} ^ {N _ {w}} \mathbb {1} \left\{\hat {y} _ {i k} = y _ {i k} \right\}
$$
where  $\hat{y}$  and  $y$  are predicted and ground truth labels respectively. The number of query objects is denoted as  $N_{q}$  while  $N_{w}$  is the number of classes (e.g. in the 1-shot-5-way setup,  $N_{w} = 5$  since there are 5 novel classes.)
$$
m I o U = \sum_ {i = 1} ^ {N} \frac {\hat {m} _ {i} \cap m _ {i}}{\hat {m} _ {i} \cup m _ {i}}
$$
where  $m$ ,  $\hat{m}$ , and  $N$  denote the ground truth mask, predicted mask, and number of objects respectively.
B.2 Main Manuscript Results
In this section, we report the confidence intervals of the experiment results in the main manuscript (Please see Tables 11, 12, 13, 14, and 15). We evaluate our models with 500 episodes and 15 query scenes for each episode.
B.3 Other Low-shot Setups
Table 16 presents the results of DINOV2 ViT B/14, both pre-trained and fine-tuned on ABC, in various low-shot setups, including 1-shot-5-way, 5-shot-5-way, 1-shot-10-way, and 5-shot-10-way under LSME setting on Toys4k.