SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While the support assignment accuracy (SA) remains consistent across all low-shot setups, the low-shot accuracy shows a notable improvement in the 5-shot scenarios with an approximate  $16\%$  increase in low-shot accuracy in both 5-way and 10-way setups.
Table 14: Performance of different methods on Toys4k under Categ-SObj-PoseVar and Categ-MObj settings. These settings solve a similar problem, with Categ-MObj having object occlusions present in both support and query objects. Performance of all methods drops significantly when faced with occlusion cases.
Method	DINOv1 S/8	DINOv2 S/14	DINOv2 B/14
Categ-SObj-PoseVar	68.84 ±1.04	73.07 ±1.03	75.18 ±1.04
Categ-MObj	56.99 ±0.97	56.95 ±0.99	57.92 ±1.04
Table 15: Performance of our baseline finetuned with different backbones on Toys4k under LSME settings with four object segmenters. Best performance is highlighted in bold while underline represents second best performance. The quality of the instance masks plays a significant role in the low-shot and shot assignment performance for all methods.
mIoU	Ours-DINOv1 S/8-ABC	Ours-DINOv2 S/14-ABC	Ours-DINOv2 B/14-ABC
Support	Query	LSA	SA	LSA	SA	LSA	SA
FreeSOLO [47]	0.52	0.54	32.31 ±0.93	43.24 ±1.94	33.99 ±0.95	44.84 ±1.95	35.50 ±0.99	48.92 ±1.88
CutLer [49]	0.61	0.63	34.62 ±0.87	42.40 ±1.96	36.34 ±0.96	46.08 ±1.96	39.42 ±1.04	52.04 ±1.97
SAM [20]	0.72	0.73	35.72 ±1.06	48.96 ±2.06	38.58 ±1.10	52.04 ±2.01	42.38 ±1.17	56.92 ±1.90
FreeSOLO-ABC	0.81	0.83	41.19 ±1.16	55.40 ±2.03	43.92 ±1.23	57.88 ±1.97	47.70 ±1.26	61.32 ±1.86
Table 16: Results on low-shot recognition on the Toys4k dataset in multi-object setting. All methods consistently experience a significant drop in low-shot accuracy when mutual exclusivity is required, and further decrease when instance segmentation is involved.
DINOv2
ViT B/14	DINOv2
ViT B/14-ABC
Low-shot Setup	LSA	SA	LSA	SA
1-shot-5-way	39.24±1.17	50.88±1.91	47.70±1.26	61.32±1.86
5-shot-5-way	55.03±0.99	50.22±0.99	63.52±1.02	60.60±1.13
1-shot-10-way	28.32±0.73	51.32±1.46	35.66±0.82	61.10±1.30
5-shot-10-way	43.26±0.70	50.62±0.69	51.72±0.75	60.85±0.74
C Models
Representation Learning Models: We use pre-trained backbones, (e.g. DINOv1 [5], DINOv2 [33]) contrastive training strategy with a momentum encoder[15]. Given two views of the same scene,  $v_{1}$  and  $v_{2}$ , we first use the instance mask associated with each object in the scene to eliminate the background and other objects. Subsequently, we extract the query object feature by performing a forward pass of the image encoder on  $v_{1}$ . For each query feature, we minimize the InfoNCE [32] loss function.
$$
\mathcal {L} _ {q} = - \log \frac {\exp (q \cdot k _ {+} / \tau)}{\exp (q \cdot k _ {+} / \tau) + \sum_ {k _ {-}} \exp (q \cdot k _ {-} / \tau)}
$$
The positive sample  $k_{+}$  is the feature of the same object in  $v_{2}$  while the negative set  $\{k_{-}\}$  consists of object features from the memory queue as in MoCo-v2 [8] and different objects from the same scene. For each input view pair, we ensure to only train on objects that are visible in both views (e.g. with instance segmentation area greater than some threshold  $\sigma = 30$  pixels).
In our approach, we omit the projector and predictor components present in most contrastive learning approaches [15, 15, 7] since we found empirically that this gave better performance. We trained our model using AdamW optimizer with initial learning rate  $5e^{-6}$  and weight decay 0, batch size 32 on 3 RTX 2080 GPUs for 50 epochs. Training took approximately 5 hours in clock time. Our pretrained weights can be found at https://tinyurl.com/3a9r83z9 and the training code is on our GitHub repository at https://github.com/rehg-lab/LSME. All pre-trained weights for other models are directly loaded from the corresponding released codebases.
Segmentation Models: We finetuned the pretrained FreeSOLO [47] model on 1K scenes of ABC dataset with instance mask annotations. To obtain the predicted instance masks for low-shot, we performed a forward pass of the fine-tuned model on our low-shot data. From the output masks,
we retained the ones with a confidence score above 0.5. To handle overlapping masks, we merged those with an IoU greater than 0.7. Finally, we employed the Hungarian matching algorithm [22] to associate each predicted mask with its corresponding ground truth mask. We finetuned FreeSOLO with batch size 6 on 3 RTX 2080 GPUs for 30K epochs.