Detect Everything with Few Examples
Authors: Xinyu Zhang
Yuhan Liu
Yuting Wang
Abdeslam Boularias
Venue: CoRL 2024 (2024)
Keywords: Robot Vision
Object Detection and Recognition
Few-shot Learning
TL;DR: We introduce DE-ViT, a few-shot object detector without the need for finetuning, which establishes new state-of-the-art on all few-shot detection benchmarks (Pascal VOC, COCO, LVIS), and we evaluate DE-ViT with a real robot in sorting novel objects.

Abstract: Few-shot object detection aims at detecting novel categories given only a few example images. It is a basic skill for a robot to perform tasks in open environments. Recent methods focus on finetuning strategies, with complicated procedures that prohibit a wider application. In this paper, we introduce DE-ViT, a few-shot object detector without the need for finetuning. DE-ViT's novel architecture is based on a new region-propagation mechanism for localization. The propagated region masks are transformed into bounding boxes through a learnable spatial integral layer. Instead of training prototype classifiers, we propose to use prototypes to project ViT features into a subspace that is robust to overfitting on base classes. We evaluate DE-ViT on few-shot, and one-shot object detection benchmarks with Pascal VOC, COCO, and LVIS. DE-ViT establishes new state-of-the-art results on all benchmarks. Notably, for COCO, DE-ViT surpasses the few-shot SoTA by 15 mAP on 10-shot and 7.2 mAP on 30-shot and one-shot SoTA by 2.8 AP50. For LVIS, DE-ViT outperforms few-shot SoTA by 17 box APr. Further, we evaluate DE-ViT with a real robot by building a pick-and-place system for sorting novel objects based on example images. The videos of our robot demonstrations, the source code and the models of DE-ViT can be found at https://mlzxy.github.io/devit.

Detect Everything with Few Examples
Xinyu Zhang (xz653, y11834, yw632, ab1544) @ereeerruuttr.... edu Rutgers University
Yuhan Liu
Yuting Wang Abdeslam Boularias
Abstract: Few-shot object detection aims at learning novel categories given only a few example images. a basic skill for a robot that performs tasks in open environments. Recent methods focus on finetuning strategies, with complicated procedures that prohibit wider application. In this paper, we introduce DE-ViT. few-shot object detector without the need for finetuning. DE-VIT's 's novel ar- chitecture is based new region-propagation mechanism for localization. The propagated region masks are transformed into bounding boxes through learn- able spatial integral layer. Instead of training prototype classifiers, we propose to use prototypes to project ViT features into a subspace that is robust to overfitting on base classes. We evaluate DE-ViT on few-shot, and one-shot object detection benchmarks with Pascal VOC, COCO, and LVIS. DE- ViT establishes new state- of-the-art results on all benchmarks. Notably, for COCO, DE- ViT surpasses the few-shot SoTA by 15 mAP 10-shot and 7.2 mAP on 30-shot and one-shot SoTA by 2.8 AP50. For LVIS, DE-ViT outperforms few-shot SoTA by 17 box APr. Further, we evaluate DE-ViT with real robot by building a pick-and-place system for sorting novel objects based on example images. The videos of our robot demonstrations, the source code and the models of DE- EiT can be found at https://mlzxy, gi btthub.io//evv..
Keywords: Robot Vision, Object Detection and Recognition, Few-shot Learning
Introduction
Object recognition and localization are two core skills an autonomous robot operating new unstructured environment. Few-shot object detection is promising approach for training a robot to detect novel categories based on a small of support images [1]. However, most recent few-shot detection methods rely fine-tuning dious procedures that limit the practical use of these methods and that results in a large accuracy gap between the base and the novel classes [3]. Pretrained vision transformers (ViTs) [4, 5] can be used to overcome the limitations of fine-tuning. However, despite their rich semantical representations, pretrained ViT features lack the coordinates information that required to perform a bounding box regression. As we show in Appendix C, naively applying conventional regression on ViT features yields poor localization results, while unfreezing the ViT backbone leads to accuracy collapse on novel classes, by completely overfitting the base classes.
both base and novel classes [2]. with complicated and te-
To address these issues, we propose novel localization architecture based on region-propagation In this architecture, object proposals are expanded by a fixed ratio. Objects are localized by per- forming a mask prediction within the expanded proposals instead of bounding-box regression. To accurately derive bounding boxes from the propagated regions, we propose the spatial integral layer, learnable mask-to-box transformation. To further narrow the accuracy gap between base and novel classes, propose construct prototypes not classifter weights, as shown in Fig. 2(a), but to project VIT features into subspace that is used network inputs. Our empirical studies demonstrate that the projected features are more robust to overfitting on base classes
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Zaga
bektse
CHEEZ-IT
Pringle
SCTE
Figure I: Demonstration of our method on YCB objects [6]. The model is trained on only the base categories LVIS, A few example images of YCB objects provided novel categories during inference only.
With these proposed techniques, we introduce DE-Vis ViT, a few-shot detector that uses example im- ages to detect novel objects without the need for any finetuning or further training. An example of the results returned by DE-ViT is shown in Fig. 1. We evaluate DE-VIT RI-VIT few-shot, and one-shot object detection benchmarks with Pascal VOC [7]. COCO [8] and LVIS [9] datasets. DE-VIT establishes new state-of state-of-thh-art (SoTA) results on all benchmarks. For COCO, DE-ViT surpasses the SoTA LVC [10] by 15 mAP on 10-shot and by 7.2 mAP on 30-shot, and also surpasses the one-shot SOTA BHRL [11] by 2.8 AP50. For Pascal VOC, DE-ViT surpasses the SoTA NIFF [12] by 2.0 nAP50. For LVIS, which has been regarded as highly challenging dataset [13], DE- ViT outperforms the SoTA DiGeo [14] by 17 box APr. Notably, our method achieves faster inference time while having better accuracy. Further, evaluate DE- VIT with real robot in novel-object sorting system.
2 Method
2.1 Problem Formulation
We use C to denote the of classes. In few-shot object detection (FSOD), C is composed of set of base classes, denoted by Crase. and a set of novel classes, denoted by Cnovel. Thus, C Chase U Crovel and Caase n Cnovel 2. During training, a large number of examples are provided for the base classes. During testing, only k labeled samples are provided for each novel class. The samples for novel classes are referred to as support images. The goal is to leverage the training data of Crase to learn a detector that can detect objects of Cnovel given the k-shot support images.
Novel class examples
Novel class examples
Novel and background class examples
Projected
Prototypes
Prototypes
Clasaifler
Matching Network
Region Propagation
Tapry Mccowon
Losiig Michoaor
Network
Query image
(a)
Query image
(b)
Query image
(c) Our method
Figure 2: Existing gestallearnng--aas FSOD methods can be divided into two categories. Methods in the first category (a) build prototypes from novel class examples and these prototypes as the classifier weights a detection network. Despite simplicity. this strategy exhibits inferior accuracy [2]. Methods the second category (b) learn to match the proposal regions in the query image and novel examples through a matching network. This strategy is computationally hcavy due to dense feature interactions across multiple images and usually requires finetuning inerease accuracy in novel classes [15, 16]. In contrast, method (c) applies subspace that robust overfitting on base
a dot dot-productt with the prototypes to project VíT features classes, and then applies a region propagation network to refine the localization and derive class score. Our method does not employ any finetuning for base or novel classes. Details of related work are in Appendix B.

---

2.2
Region Propagation Network
Model Overview
Propagation Layer
Predicted
Output
box
Predicted box
Output region
features
Class score
Class
Spataal Integral
Propagated
Region
output region
Sofa
Layer
Sigmoid
Linear
Conv #out ch=
Propagation Layer (PL)
input region
txtxxx
Concate
oulput region
Concate
Conv--n-reeu (lloul ch=S).
Weighted Avg over H. W
input region
Propagation (PL)
Initial
Inpul regions
Input
Region
8 8
2. Feature Subspace Projection
8x x
Expanded
Festure Subspace
Channel Reorder
ROI Align
Proposal
Projection
Subspace Featuuees
1 + (8)
DxH
Dot Product
Proposal box
(12 (80)
Prototypes VIT Features
Protolypes
VIT Features
Proposal
Figure 3: Overview the proposed method. Given proposal box query image, we extract the initial mask region (ycllow) and expand the proposal by constant ratio. Next, the region within the expanded proposal is gradually propagated and refined to fit the object area through sequence of propagation layers, Each propagation layer accepts previous regions and features input while returning updated features and region outputs. The final predicted region is transformed bounding-box coordinates through learnable spatial integral layer, as detailed Sec. 2.3. predicted region also serves as spatial attention average features along height and width. The averaged features are then mapped to class scores. The projected features are the dot products between the VIT features of query image and class prototypes, as detailed in Sec. 2.4.
Despite having rich semantics information, pretrained ViT features lack the coordinates information required for bounding box regression. As shown in Appendix C, naively applying a conventional regression on VIT features yields poor localization results. natural solution is to learn this local- ization capability by finetuning the ViT backbone during the training of the detector with the base classes. However, observed that finetuning results in completely overfitting the base classes and in accuracy collapse novel classes. This was observed when integrating DINOv2 ViT into the framework of Meta RCNN [17], standard prototype-basd based FSOD, as shown in Appendix C. This suggests that harnessing the generalization power of strong ViT backbones for FSOD is non-trivial. The question here how to produce accurate localization with pretrained VIT features.
Given an object proposal, we use region propagation network that gradually propagates the pro- posal region accurately cover and fit the object by refining object mask. Unlike bounding- box regression, mask prediction localizes objects without coordinate outputs. The propagated region is then transformed into bounding box through learnable spatial integral layer. We use an off- the-shelf region proposal network (RPN) to generate the initial region proposals, as class-agnostic proposals are shown to generalize well to novel classes [18]. Each proposal expanded by fixed ratio in order delimit the propagation boundaries. The overall framework shown Fig. 3.
Propagation Layer. We propose the region Propagation Layer (PL), a new type network module designed for object detection. PL serves as the central building block of our method. An example is shown in Fig. The t-th PI. block takes all previous regions fo:-1 E x and the previous PL block features h kSxXxK input, where denotes the number of PL blocks, denotes the number of feature channels, and K denotes the feature spatial size. The t-th PL block outputs K the updated region E x features h2 E SxKK bounding box by R2 and class score

---

0
0
0
0
O
object
0
1
0
0
0
0
0
0
0 0
0
0
0 0
(cw)
rel) Ch
Σ
K' K
softmax(hreginn)
(3)
h=4
0
0
S-=
0
0
(hregion (())) g
0
O
0
K
(4)
Calumn
a (hregion Jili) 0h
Sum
Figure 4: The max row sum and column sum are used as the box width and height of the triangular object.
K
E R. Each PL block works as small detection network and can be stacked to improve accuracy.
The update rule explained in Eq. I and illustrated in the third part of Fig. 3.
h4 = fupdats, i (concat
hx-1): 0), he = fregion, (1eg 0) (To: (roocaattrot--0
T₁ = o (h4,region)) b, (he, 9)
(1)
Ct = felass, (Weighted (h4, Fz));
In Eq. 1, fupdate.t denotes the conv-bn-relu block with S output channels that updates the hidden features, and concat denotes channel-wise concatenation. fregion, denotes the conv block with single channel output that predicts the output region logits h, regrin R x denotes the sigmoid function. fintegral, denotes the spatial integral layer detailed in Sec. 2.3. h; is aggregated over spatial dimensions to WeighledAvggoool(h,, rt) Sxixxl with weights feluss, denotes the linear block that maps WeightedAvgPPoo((n: rx) to class scores. 0 denotes network parameters. During training, we use focal loss and LI regression loss for the output class score c and bounding box b. For the output region rz, we apply BCE loss and Dice loss [19]. The region labels during training are generated by the ground-truti object region within the expanded proposals.
2.3 Learnable Spatial Integral
Converting masks to bounding boxes is a widely-used transformation in instance segmentation net- works [20] post-processing step. The standard solution is find the top-left and bottom- right foreground pixels and use their positions the bounding box coordinates [21]. However, this approach has major limitations. Firstly, this mask-to-box conversion assumes the availability of ground-truth instance masks, which are much more expensive to obtain than bounding boxes [8]. Moreover, this approach is non-differentiable and also prone to outliers. Therefore, the question is how to accurately derive bounding boxes from the region-based localization results using a learnable and differentiable function.
Ch Let your = (cour out our Joout) denote the output bounding box, where c E [0, W), wo E [0, W]
and ch E [0, h [0, H]. Instead of predicting fou directly, we propose to first predict relative bounding box yel (crel c%. wtl hell) E [0, 1]4. that can be transformed gaut according to Eq. 2,
(woat, /uuu)) (w rel hoxp ho (cret o,, = (c SP
Ch 0P5 + (c Tx rel heT)),
exp
where Josp = (CWP exp xxp (xx)) denotes the expanded proposal. Thus, gel is normalized bound- ing box relative to 1oxp. Let hregson E K denote the output region logits, where we skip the notation of 1-th block and the channel of for simplicity. Our spatial integral layer fintegral estimates gel with Eq. and 4. An illustrative example is given in Fig. 4.
To motivate Eq. 3 and 4, consider the toy example of converting binary triangle mask to bounding box in Fig, 4. A reasonable approach is to compute the mask center the bounding box center and

---

Table 1: Results COCO 2014 few-shot benchmark. Our method outperforms existing work in detecting novel classes by significant margin. Results surpassing the SoTA are indicated in bold.
Method
Requires
10-shot
30-shot
Finetune
bAp
nAP
nAPS0
nAP75
bAP
nAP
nAP50
nAP75
FSRW [22]
X
5.6
12.3
9.1
7.6
Meta R-CNN [17]
x
5.2
6.1
19.1
6.6
7.1
9.9
25.3
10.8
TFA [13]
V
33,9
10
19.2
9.2
34.5
13.5
24.9
13.2
Multi-Relation Det [23]
X
16.6
31.3
16.1
FSCE [24]
11.9
10.5
16.4
16.2
Retentive RCNN [25]
V
39.2
10.5
19.5
9.3
39.3
13.8
22.9
13.8
IeteroGraph [26]
11.6
23.9
9.8
16.5
31.9
15.5
FsDetView [27]
6.4
7.6
9.3
12
Meta Faster RCNN [15]
12.7
25.7
10.8
16.6
31.8
15.8
LVC [10]
28.7
19
34.1
19
34.8
26.8
45.8
27.5
Cross CrossTrans Tramerormer [16]
17.1
30.2
17
21.4
35.5
22.1
NIFF [12]
39
18.8
39
20.9
DiGeo [14]
39.2
10.3
18.7
9.9
39.4
14.2
26.2
14.8
VIT-S/14
27.1
43.1
28.4
24.2
26.9
43.1
28.5
DE-VIT
VIT-B/14
X
28.3
33.2
51.4
35.5
28.5
33.4
51.4
35.7
(Ours)
VIT-L/14
X
29.4
34.0
52.9
37.0
29.5
34.0
53.0
37.2
Proposal Box
Initial Repion
Propagated Region
Predicted Box
Figure 5: Region propagation class "fridge". The proposal, expanded proposals, and final predicted boxes are colored transparent yellow, yellow, and green. respectively. The three propagated regions (from left right) are sampled from the output of the first three PL blocks in ascending order.
the max expected position of the spatial distribution softmax (Aregion) as the center of fist in Eq. 3. We com- pute the row and column sums of the output region as Σø=1 (lhregioo)) )ij and Σ a (hregioon); in Eq. 4. Instead of picking the maximum, apply soft aggregation to average all row column sums in terms of magnitude rank. The aggregation is done by sorting the and column sums, and then computing the weighted average. This explains the use of order statistics notation (i) and (j)- Parameters 8h E IR dW E R are learnable aggregation weights.
and column
width and height. Inspired by this insight, we compute the
2.4 Feature Subspace Projection
The main challenge of FSOD is to generalize to novel classes that are unseen during training. How- ever, despite numerous attempts to solving this problem, by using margin- baaged regularization for example [14], there persists a considerable accuracy gap between base and novel classes [1]. This disparity indicates that network trained with base classes would inevitably overfit on patterns that only present among the base classes. A classic technique to reduce overfitting consists in repre- senting data in low-rank subspace [28]. We explore in this work the construction of subspace of pre-trained VIT features that reduces the accuracy gap between base and novel classes.
Prototypes are class representatives built from support images. Given the support images of each class, we compute the ViT features, crop the features with object bounding boxes and use the average feature as the class prototype [17]. Let Pc R denote the prototypes of classes from set C, where D denotes the channel dimension. Let hos D denote the ViT features of the query image. We assume that both prototypes and features are normalized to unit length at the channel dimension. Then Avir E x can be interpreted as a subspace projection with Pc being the basis. However, this subspace construction has two limitations. Firstly, only using prototypes of classes set C can be too limited to sufficiently capture the feature information. Secondly, permutation of creates a different but equivalent subspace, yet designing permutation- invariant networks is a highly challenging problem [29]. For the first limitation, we introduce set B of background classes, BnC Ø, with p8 | being the prototypes of B. to preserve more information from hese. For the second limitation, we propose to build separate subspace for each

---

Table 2: nAPS0 results on Pascal VOC few-shot benchmark. Results surpassing the SoTA are indicated in bold. (*) denotes that implementation is not publicly available.
Metiod
Novel Splic
Novel Split
Novel
10
10
TEA (13)
39.8
36.1
46.7
55.7
56,0
23.5
26.9
34.1
35.1
39.1
30.8
34.8
42.8
69.5
69.8
39,9
FaDetView
20.4
36.3
32.4
29.8
39.8
Multi Relation (23)
56.5
58.5
22.5
30.6
31
43.7
49.8
Retentive RCNN (2S)
45.8
45.0
53.7
56.1
21.7
27.8
37.0
40.3
30.2
43
40.7
50.1
Meta Faster R-CNN [15]
43.0
$4.5
60.6
66.1
654
27.7
35.5
46.
47.5
51.4
40.6
46.4
53.4
59.9
58.6
50.5
LVC [10)
54.5
53.2
56.8
63.2
65.7
32.8
29.2
50.7
69.8
$0.6
48.4
52.7
55
59.6
59.6
52.3
Tranfforrr [16]
49.9
57.1
$7.9
63.2
67.1
27.6
54.5
43.7
49.2
51.2
39.5
54.7
52.3
58.7
50.9
HeterGGraph
51.9
55.7
62.6
25.9
57.8
51.1
35.2
47.8
53.3
48.0
[14]
39.4
30.4
54.7
NIFF ()
67.3
68.0
70.3
68.8
35.4
42.9
54.0
56.4
56.4
61.2
63.9
DE-VI (Ours)
B/14
VIT:5714
47.5
64.5
57.0
73.9
68.5
72.8
67.3
43.1
45.3
34.1
47.3
58.2
49.7
56.7
59.8
00.8
52.5
58.6
62.1
60.7
62.7
61.4
67.8
64.5
56.7
VIT-1/14
70.9
class E C, and reorder other classes C \ to resolve permutation ambiguity. The feature subspace projection is explained in Eq. 5 and illustrated in Fig. 3.
Naubsppcce.c = concat(p. huse, channel-reorder(peee llsst) PS Hosst)
(5)
In Eq. 5, haubepace,e E S denotes the subspace feature for class C, and function channel-reorder sorts the |C| - 1 channels of input tensor Pele hygt [[( by magnitude each spatial location, and then linearly interpolates the tensor (S - 1 I x H X W, where S is a constant hyperparameter. In practice, we use example im- ages of non-object stuff classes, e.g., sky, road, floor, to construct B. As shown in Appendix C, feature subspace projection significantly reduces the accuracy gap between base and novel classes. On the other hand, creating a subspace for each class e E C introduces costly per-class inference. However, the per-class inference cost can be reduced by finding the top T most likely classes with a lightweight prototype classifier [17] and only performing inference for these T classes. As shown in Appendix C, our method achieves a faster inference speed and surpasses SoTA when T =
pre-defined size
3 Experiments
We comprehensively evaluate our method on few-shot and one-shot benchmarks. Furthermore, we compare the efficiency of our method against SoTA solutions, study few-shot performance for different numbers of shots, compare it to language-based detectors, and provide qualitative results. We conduct ablations to study the contributions of the proposed components to the performance of our method. Our source code and the pretrained models are included in the supplementary material and will be publicly released upon acceptance.
Evaluation Metrics and Datasets. Few-shot and one-shot evaluations split classes into base and novel classes. Base classes are seen during training and novel classes are unseen. The performance on novel classes is more important. For COCO and Paseal VOC, nAP, nAP50, and nAP75 represent mAP, AP50, and AP75 in novel classes. bAP and bAP50 represent mAP and AP50 in base classes. One-shot evaluation conventionally divides 80 classes of COCO into four even partitions, and alter- natively uses three as base classes and one partition novel classes [30]. There are 4 base/novel splits in total, named 1/2/341 For LVIS, APr, APc, and APF represent AP on rare, common, and frequent categories. Rare categories are used as novel classes. Metrics on LVIS, such as box APr and mask APr, are computed on bounding boxes and on instance segmentation masks sepa- rately. We evaluate our method on Pascal VOC [7], COCO [8], and LVIS-v1 [9]. We follow the conventional base/novel classes split and use the same support images of novel classes [13, 11].
Implementation Details. We adopt a standard two-stage object detection framework, similarly to Mask R-CNN [31]. We use the same off-the-shelf RPN with existing work [32] to generate object proposals. The RPN is trained separately for each dataset using only base classes. We use DINOv2 [33] VIT as the backbone, and report results in V/T-S/B/L (small, base, large) model sizes. The VIT backbones are kept frozen during detector training. We adopt the prototype extraction

---

DE-ViT (Ours)
Meta-Faster RCNN
DE-VIT (Ours)
W
V
Jobi HAT
mularopde
Natter
(a) COCO
(b) LVIS
Figure 6: Qualitative visualization of our method DE-ViT on COCO (a) and LVIS (b), and compar- ison to Meta-Faster RCNN [15]. Boxes base and novel classes are colored green and yellow.
procedure of Meta RCNN [17] using ViT features. During training, the model can access only the prototypes base classes, After training, the prototypes novel classes are appended while the remaining parameters are unchanged. During evaluation, the model is evaluated on images that contain objects of both base and novel classes. Prototypes of background classes B are extracted from images of stuff (non-object) classes, e.g., sky, road, from COCOStuff [34] unless specified. Similar to [17], the top T most likely classes for each proposal are determined by the distances between prototypes and the average proposal feature. We set I to 10 unless specified, where T is the number of classes to create subspace features and perform inference as explained in Sec. 2.4. We apply 3 PL blocks for experiments on Pascal VOC and COCO, and PL blocks for those on LVIS.
3.1 Main Results
Tab. shows our results the few-shot COCO benchmark. Our method DE-ViT outperforms the previous SoTA LVC [10] by significant margin (+15 nAP on 10-shot, +7.2 nAP on 30-shot). It is worth noting that LVC requires over ten stages for self-training and pseudo-labeling procedures on novel classes [35], while our method DE-V ViT trained once on the base classes and used directly on the novel classes. A pretrained model for LVC has never been released. We plot the nAP50 of our method and the SoTAs with different numbers of shots in Fig. 7.
2014 nAP50
Table 3: Performance comparison with existing FSOD methods on the LVIS dataset. We report
LVC
8
the box AP.
IVC
Method
APr
APc
APГ
AP
HeleoGraphi Mrto-Prster RCNN
DiGeo [14]
16.6
22.8
28
24.4
Ours
DE-VI VIT
VIT-S/14
23.4
22.8
22.5
22.8
10
HetwoGraph
Existing SOTAs
VYT-B/14
26.8
26.5
25.3
26
(Ours)
ViT-L/14
33.6
30.1
30.7
30.9
Shots
Performance of DE-ViT and few-shot SoTA with different numbers of shots.
Figure 7:
Tab. shows our results on the few-shot Pascal VOC benchmark. Our method DE- ViT outperforms the previous SoTA NIFF [12] by +2.0 averaged nAP50. It is worth noting that the implementation of NIFF has not been released. LVIS has been regarded
highly challenging benchmark for
FSOD [13] with 337 novel classes and only DiGeo [14] reports few-shot results on LVIS v1. Tab. 3 shows that our method outperforms DiGeo in all metrics and a significant boost in the accuracy of detecting novel objects (+20 box APr).
Tab. shows our results on the one-shot COCO benchmark. DE-ViT surpasses the previous SoTA BHRL by 6 bAP50 and 2.8 nAP50. One-shot detection task follows single-class detection setting. Therefore, adapt method DE-ViT by detecting each class separately during evaluation. OWL-
7

---

Table 4: Results on COCO 2017 one-shot benchmark. DE-VIT outperforms existing work and is limited single class detection and single support image as other one-shot methods.
bAP50
nAP50
Split-1
Split-2
Split-3
Split-4
Avg
Split-1
Split-2
Split-3
Split-
Avg
SiamMask (30)
38.9
37.1
37.8
36.6
37.6
15.3
17.6
17.4
17
16.8
CoAE [37]
42.2
40.2
39,9
41.3
40.9
23.4
23.6
20.5
20.4
[38
50.1
47.2
45.8
46.9
47.5
26
26.4
22.3
22.6
24.3
SaFT [3]
49.2
47.2
47.9
49
48.3
27.8
27.6
21
23
24.9
BHRL [11)
52.1
52.6
53.4
53.6
26.1
22.7
24.5
25.6
DE-VIT (Ours, VIT-L/14)
59.4
57.0
61.3
60.7
59.6
27.4
33.2
27.1
26.1
28.4
VIT [36] also reports one-shot results on COCO. However, OWL-ViT's results are obtained with an ensemble of sanguage-based based detection and one-shot pipelines without providing an implementation or isolated measurements. Therefore, we exclude OWL-ViT from the one-shot comparison.
3.2 Real Robot Experiment
Table 5: Success rates of DE-
ViT on sorting novel objects.
Object
Success
(Qurs)
Overall (%)
97%
Chips
8/10
Tomato Can
9/10
SAM
Ball
Crackers
groe pose
Example Image
Brick
Figure 8: Overview of the system for sorting novel objects built upon our DE-ViT Our system receives few example images novel classes and instantly detects new objects within the same category with- out tuning or further learning.
Cup
10/10
Mustard Bottle
Sugar Box
Orange Cleanser Bottle
Setup. To evaluate our DE-VI DET in real-world robotic tasks, we develop pick-and-place system for sorting novel objects based example images. The system is outlined in Fig. 8 with an example of sorting Rubik's cubes. First, the front RGB camera takes images of the example objects, which are then segmented with SAM [39] and built as class representative prototypes. Next, our DE- ViT detects objects and generates instance segmentation masks from the RGB image of a side- view RGBD camera. Then, the instance segmentation masks are combined with the depth image to produce the point-cloud each object instance. The grasp pose for each object is generated with heuristic-based pose generator [40]. We use Kuka LBR iiwa robot. Note that our system receives example images of novel classes and detects novel objects instantly it does not require finetuning.
Results. Tab. shows the success rates of our system in object sorting. We adopt ten YCB objects such as crackers and mustard bottles. For each run, all objects are placed on the table, and we ask the robot to pick and place objects based on given order of the novel classes. We use our DE-ViT with the VTT-I. backbone. We use example images for each object class, except the mustard bottle and tomato soup can, for which we images improve their detection accuracy. Our system achieves an overall 97% success rate out of 100 independent picks for all objects. The layout of all objects is randomized for each pick. Of the three failed cases, two involve mistaking chips for tomato can and vice versa. In the third case, the chips are detected correctly, but the instance mask is not precise enough to enable a successful grasp. We believe this is because our DE-ViT model trained exclusively on the base categories the LVIS dataset without using YCB images or any datasets on retail products.
4 Final Remarks
We conducted extensive analyses and ablation studies on model efficiency, comparisons to DINOv2 in Meta RCNN, comparisons to language- layers and shots. Details of these studies and results are included in Appendix C due to space limit.
basea based detectors [32], and the effects of the number of PL

---

Acknowledgement
This work is partially supported by NSF awards 1846043 and 2132972.
References
[1] S. Antonelli, D. Avola, Cinque, D. Crisostomi, L. Foresti, F. Galasso, M. R. Marini,
A. Mecca, and D. Pannone. Few-shot object detection: survey. ACM Computing Surveys (CSUR), (4(115):113333 2022.
[2] M. Köhler, M. Eisenbach, and H.-M. Gross. Few-shot object detection: a comprehensive survey, IEEE Transactions on Neural Networks and Learning Systems, 2023.
[3] Y. Thao, X. Guo, and Y. Lu. Semantic-aligned fusion transformer for one-shot object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7601-7611, 2022.
[4] L. Melas-Kyriazi, C. Rupprecht, I. Laina, and A. Vedaldi. Deep spectral methods: surpris- ingly strong baseline for unsupervised semantic segmentation and localization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8364-8375, 2022.
[5] O. Siméoni, C. Sekkat, G. Puy, A. Vobecky, é. Zablocki, and P. Pérez. Unsupervised object localization: Observing the background to discover objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3176-3186, 2023.
[6] B. Calli, A. Singh, A. Walsman, Srinivasa, P. Abbeel, and M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In International Conference Advanced Robotics (ICAR), pages 510-517. IEFF, 2015.
[7] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, $8:303-338, 2010.
[8] T.-Y. Lin, M. Maire, Belongie, Hays, P. Perona, D. Ramanan, P. Dollár, and L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755. Springer, 2014.
[9] A. Gupta, P. Dollar, and R. Girshick. Lvis: A dataset for large vocabulary instance segmenta- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pages 5356-5364, 2019.
[10] P. Kaul, W. Xie, and A. Zisserman. Label, verify, correct: simple few shot object detec- tion method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1237-14247, 2022.
[11] H. Yang, S. Cai, H. Sheng, B. Deng. J. Huang, X.-S. Hua, Y. Tang, and Y. Zhang. Balanced and hierarchical relation learning for one-shot object detection. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7591-7600, 2022.
[12] K. Guirguis, J. Meier, G. Eskandar, M. Kayser, B. Yang, and J. Beyerer. Niff: Alleviating forgetting in generalized few-shot object detection via neural instance feature forging. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24193-24209 2023.
[13] X. Wang, T. E. Huang, T. Darrell, E. Gonzalez, and F. Yu. Frustratingly simple few-shot object detection. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.

---

[14] J. Ma, Y. Niu, J. Xu, S. Huang, G. Han, and S.-F. Chang. Digeo: Discriminative geometry- aware learning for generalized few-shot object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3208-3218 2023.
[15] G. Han, S. Huang, J. Ma, Y. He, and S.-F. Chang, Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 780-789, 2022.
[16] G. Han, J. Ma, S. Huang, L. Chen, and S.-F. Chang. Few-shot object detection with fully cross- transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5321-5330, 2022.
[17] X. Yan, Z. Chen, A. Xu, X. Wang. Liang, and L. Lin. Meta r-cnn: Towards general solver for instance-level low-shot learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 9577-9586, 2019.
[18] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and lan- guage knowledge distillation. International Conference on Learning Representations, 2021.
[19] C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, and M. Jorge Cardoso. Generalised dice over- lap as a deep learning loss function for highly unbalanced segmentations. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, pages 240-248. Springer, 2017.
[20] T. Ren, S. Liu, F. Li, H. Zhang, A. Zeng. J. Yang, X. Liao, D. Jia, H. Hi, H. Cao, J. Wang, Z. Zeng, X. Qi, Y. Yuan, J. Yang, and L.. Zhang. detrex: Benchmarking detection transformers, 2023.
[21] T. maintainers and contributors. Torchvision: Pytorch's computer vision library. https: //gi giittbb.comm www.com///pppe httmm/ptooch//is//ee 2016.
[22] B. Kang, Z. Liu, X. Wang, F. Yu, J. Feng, and T. Darrell. Few-shot object detection via feature reweighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8420-8422 2019.
[23] Q. Fan, W. Zhuo, C.-K. Tang, and Y.-W. Tai. Few-shot object detection with attention -ppn and multi-relation detector. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4013-4022, 2020.
[24] B. Sun, B. Li, S. Cai, Y. Yuan, and C. Zhang. Fsce: Few-shot object detection via contrastive proposal encoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7352-7362, 2021.
[25] Z. Fan, Y. Ma, Z. Li, and J. Sun. Generalized few-shot object detection without forgetting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4536. 2021.
[26] G. Han, Y. He, S. Huang. J. Ma, and S.-F. Chang. Query adaptive few-shot object detection with heterogeneous graph convolutional networks. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 2263-3272, 2021.
[27] Y. Xiao, V. Lepetit, and R. Marlet. Few-shot object detection and viewpoint estimation for objects in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3): 3090-3106, 2022.
[28] C. Schittenkopf, G. Deco, and W. Brauer. Two strategies avoid overfitting feedforward networks. Neural Networks, 10(3):005-516, 1997. ISSN 0893-6080. doi:htps://doi..o 101106650893-408000000066666 URL
ariicl//pi//0093600000088
10

---

[29] F. Hensel, M. Moor, and B. Rieck. A survey of topological machine learning methods. Fron- fiers in Artificial Intelligence, 4:681 108, 2021.
[30] C. Michaelis, I. Ustyuzhaninov. M. Bethge, and A. S. Ecker. One-shot instance segmentation. arXiv preprint arXiv: I.
[31] K. He. G. Gkioxari, Dollár, and R. Girshick. Mask r-cmn. In Proceedings of the IEEE International Conference on Computer Vision pages 2961-2969, 2017.
[32] Y. Zhong, Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, al. Regionclip: Region- basod language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pages 16793-16803, 2022.
[33] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07799 2023.
[34] H. Caesar, J. Uijlings, and V. Ferrari. Coco-stuff: Thing and stuff classes in context. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1209- 1218, 2018.
[35] Official repo: "label, verify, correct: A simple few-shot object detection method"" https: // gggthuu... www.ub..coo/ooooomemeeeeeeeeeee. http///aaaaalll/a/aaaaaa (Accessed on 1 11/17/2023).
[36] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Ma- hendran, A. Arnab, M. Dehghani, Z. Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pages 728-755. Springer, 2022.
[37] T.-. Hsieh, Y.C.. Lo, H.TT. Chen, and T.-L.. Liu. One-shot object detection with co-attention and co-excitation. Advances in Neural Information Processing Systems, 32, 2019.
[38] D.-J. Chen, H.-Y. Hsieh, and T.-L. Liu, Adaptive image transformer for one-shot object detec- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 12247-12256, 2021.
[39] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015 2023.
[40] M. Gualtieri, A. ten Pas, K. Saenko, and R. Platt. High precision grasp pose detection in dense clutter, 2016.
[41] X. Li, L. Zhang, Y. P. Chen, Y.-W. Tai, and C.-K. Tang. One-shot object detection without fine-tuning. arXiv preprint arXiv:2005.0383 2020.
[42] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF Interna- lional Conference on Computer Vision, pages 9650-9660, 2021.
[43] X. Zhang and A. Boularias. Optical flow boosts unsupervised localization and segmentation. In 2023 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7635-7642. IEEE, 2023.
[44] A. Ziegler and Y. M. Asano. Self-supervised learning of object parts for semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pages 14502-14511, 2022.
[45] M. Salehi, E. Gavves, G. Snoek, and Y. M. Asano. Time does tell: Self-supervised time- tuning of dense image representations. In Proceedings the ence Computer Vision, pages 16536-16547, 2023.
the IEEE/CVF International Confer-

---

[46] Y. Wang, X. Shen, S. X. Hu, Y. Yuan, J. L. Crowley, and D. Vaufreydaz. Self-supervised transformers for unsupervised object discovery using normalized cut. Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14543-14553, 2022.
[47] A. Gupta, S. Narayan, K. Joseph, S. Khan, F. S. Khan, and M. Shah. Ow-detr: Open-wone world detection transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9235- 9244, 2022.
[48] 7. Xin, S. Chen, T. Wu, Y. Shao, W. Ding, and X. You. Few-shot object detection: Research advances and challenges. Information Fusion, page 102307, 2024.
[49] S. Kim, C. Wang. B. Li, and S. Scherer. Robotic interestingness via human-informed few- shot object detection. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2022.
[50] Y. Li, H. Zhu, S. Tian, Feng, Ma, C. Teo, C. Xiang, P. Vadakkepat, and T. H. Lee. Incre- mental few-shot object detection for robotics. In 2022 International Conference Robotics and Automation (ICRA), pages 8447-8453. IEFE, 2022.
[51] Y.-W. Chao, Y. Xiang, et al. Fewsol: A dataset for few-shot object learning in robotic envi- ronments. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 9140-9146. IEEE, 2023.
[52] B. Griffin. Mobile robot manipulation using pure object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications Computer Vision, pages 561-571, 2023.
[53] Z. Liu, Y. Lin, Y. Cao, H. Hu, Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVE In- ternational Conference on Computer Vision, pages 10012-10022, 2021.
[54] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[55] J. Wu, Li, X. H. Yuan, H. Ding, Y. Yang, X. Li, J. Zhang, Y. Tong, X. Jiang, B. Ghanem, et al. Towards open vocabulary learning: A survey, arXiv preprini arXiv:2306. 2023.
[56] B. Landau, L. R. Gleitman, and B. Landau. Language and experience: Evidence from the blind child, volume 8. Harvard University Press, 2009.
[57] Jolicoeur, M. Gluck, and S. M. Kosslyn. Pictures and names: Making the connection. Cognitive psychology, 16(2):243-275, 1984.
[58] Dorogovtsev and J. F. Mendes. Language as an evolving word web. Proceedings of the Royal Society of London. Series Biological Sciences, 266(1485):20332266 2001.
[59] Y. Zang, W. Li, K. Zhou, C. Huang, and C. C. Loy. Open-vocabulary detr with conditional matching. In European Conference on Computer Vision, pages 106-122. Springer, 2022.
[60] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision, pages 350-368. Springer, 2022.
[61] P. Kaul, W. Xie, and A. Zisserman. Multi-modal classifiers for open-vocabulary object detec- tion. In International Conference on Machine Learning, pages 15946-15969, PMLR, 2023.
[62] X. Wu, F. Zhu, Zhao, and H. Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7031-7040, 2023.
12

---

[63] D. Kim, A. Angelova, and W. Kuo. Region-aware pretraining for vpcabbocyyylary object detection with vision transformers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11144-1114, 2023.
[64] C. Ma, Y. Jiang, X. Wen, Z. Yuan, and X. Qi. Codet: Co-occurrence guided region-word alignment for open-vocabulary object detection. Advances in Neural Information Processing Systems, 36, 2024.
[65] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012.
[66] X. Chen, Fang, T.-Y. Lin, R. Vedantam, Gupta, P. Dollár, and C. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 1504.00500000335 2015.
[67] Y. Fang. Q. Sun, X. Wang. T. Huang, X. Wang. and Y. Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303. 11311, 2023.
[68] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recog- nition, pages 633-641, 2017.
[69] W. Kuo, Y. Cui, X. Gu, A. Piergiovanni, and A. Angelova. Open-vocabulary object detection upon frozen vision and language models. In The Eleventh International Conference Learn- ing Representations, 2023. URL
wtwpriewwwee/fforrtlt 1d-M1Myy4kh91f.
[70] Q. Liu, Xu, G. Bertasius, and M. Niethammer. Simpleclick: Interactive image segmentation with simple vision transformers. arXiv preprint arXiv:2210.11006, 2022.
[71] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213- 229. Springer, 2020.
13

---

A Appendix
A.1 Code and Pretrained Models
The source code of our method DE-VIT is included in the supplementary folder Code. Please check Code/READOE md for instructions on installation, dataset setup, and downloading pretrained models from an anonymous server.
B Related work
Few-shot Object Detection (FSOD) aims at detecting objects of novel classes by utilizing a few support images from novel classes training samples [2]. Existing approaches can be broadly classified into finetuning-based [13, 25, 24, 27, 12] and meta-learning-basee strategies [17, 22, 23]. Finetuning-bed based methods, despite their prevalence, suffer from large accuracy gap between the base and novel classes, as well as practical limitations due to redundant multi-stage procedures [3]. Meta-leaming- based methods avoid finetuning through online adaptation. Early works on meta- learning FSOD (Fig. 2.a) construct prototypes from novel class examples and use prototypes as input to network that classifies query images [17, 22]. Recent methods (Fig. 2.b) design interaction mechanisms of dense spatial features between query and support images [15, 16, 26]. In contrast, method (Fig. 2.c) computes dot dot-products with the prototypes project their features into network instead
subspace, instead of using prototype classifier, and applies of a dense feature matching network. One-shot Object Detection (OSOD) is an extreme case of FSOD with only one example per novel class, it reduces the setting to single-class detection [11]. Prior approaches primarily focus on designing interaction mechanisms of dense spatial features between support and target images [30, 37, 41]. However, the OSOD formulation restricts the use of additional support and requires separate inference per class [3, 38]. Compared with existing work, method does not per-class inference and only utilizes class-level prototypes without dense feature interactions.
Vision Transformer (ViT) is recent architecture that demonstrates stronger performance and more interpretable features than traditional convolutional architectures. There is growing trend of apply- ing self-supervised ViTs, such as DINO [42, 33], to unsupervised object discovery [43, 44, 45, 5]. TokenCut [46] applies graph-cut over DINO features to separate the primary foreground object. DeepSpectral [4] predicts segmentation masks through unsupervised spectral clustering over DINO features. OW-DETR [47] uses DINO ViT to discover unknown objects in an open-world setting.
Robotics Application of Few-shot Object Detection. FSOD has been increasingly deployed in robotics applications [48]. AirInteraction [49] designs a robot exploration strategy based on human- informed interestingness, leveraging the learning from examples ability of few-shot detectors. CI- FSOD [50] designs a lightweight few-shot adaptor module that suitable for robotics deployment on edge devices. Fewsol [51] studies the existing few-shot detector performance in typical robotics environments. TFOD [52] designs a mobile manipulator application purely based on the predictions of few-shot detector with online annotations.
C Analysis and Ablation Studies
Efficiency. We compare the inference time of DE-VIT with different values of T' against recent few-shot works in Tab. 1 in COCO 10-shots setting, where Swin denotes Swin Transformer [53], RN101 denotes ResNet101 [54], MERCNN denotes Meta Faster RCNN [15], and CrossT denotes Cross Transformer [16]. T is the number of classes to create subspace features and perform in- ference. Tab. Al shows that our method DE-ViT can achieve the smallest inference time while having better accuracy. The inference times of all the compared methods are measured the same machine.
14

---

115
CO00 2014 nAP50
Table Al: Ablation study different values of and inference time comparison with existing methods. N/A: The pretrained model is not pub- licly available for evaluation.
42
38
Method
Backhone
T
nAP50
Secs/Img
36
0.22
DE-VIT (Ours)
ViT-I/14
3
52.5
0.33
34
33.9
10
52.9
0.83
Number of PL blocks
LVC [10]
Swin-S
34.1
N/A
MERCNN [15]
RNI01
25.7
0.61
Figure A1: Ablation study of our method on different numbers of PL blocks with ViT-S/14 backbone.
CrossT [16)
Custom
30.2
3
Table A2: Comparison to naively applying DINOv2 with Meta RCNN, and ablation studies on the feature subspace projection.
Meta RCNN + DINOv2 Backhone
Feature Subspace Projection
Novel
Base
c
C C\c
nAP50
nAP75
hAP50
bAP75
4.5
2.2
48.9
22.5
26.2
9.7
29.3
12
38.4
23
43.4
26.8
1
39.5
24.1
42.3
25.9
Feature subspace projection. We examine the component effects for feature subspace projection in Tab. A2. The first column represents the integration of raw features of DINOv2 VIT into Meta RCNN [17], standard prototype-base based few-shot detector. Tab, A2 shows that the model has curacy collapse novel classes with raw ViT feature inputs by completely overfitting base classes. The c, B. and C ) represents different groups of prototypes used in feature projection, as in Eq. 5. By projecting features to Pe, general detection ability emerges (nAP50: 4.5 + 26.2), then signifi- cantly improves after adding background classes prototypes Pg (nAP50: 26.2 ~ 38.4), and further improves with Pete (nAP50: 38.4 + 39.5). Results Tab. A2 and A3 are obtained on COCO 2017 with ViT-S/14.
Region propagation network. We study the impacts of cictonn-oppgtiion---o based localization in Tab. A3, The conventional regression network exhibits poor localization accuracy on novel classes (nAP75: 14.6). Note that set of learned 2d positional embeddings is added to the VIT features be- fore feeding into the conventional regression network. When using proposed region-propagalie network, the localization accuracy has significant boost (nAP75: 14.6 - 24.1). Since our region- propagation network expands proposals, the accuracy gain may come from the spatial features of larger areas. Therefore, we ablate this by expanding proposals within the conventional regression network. This results in even lower performance than the conventional regression network alone (nAP75: 14.6 + 12.5). This shows our region propagation network is the main contributing fac- the performance boost. Note that the results in Tab. A3 obtained by only changing the localization architectures while keeping the feature subspace projection.
Number of propagation layers. We study the impacts of the number of PL blocks in Fig. A1, which shows that stacking more PL. blocks consistently improves accuracy. The accuracy saturates around 5 blocks and there is still non-negligeble improvement from - 4 blocks. Note that use 3 blocks in our COCO and Pascal VOC experiments and 5 blocks in LVIS ones.
Comparison to language-based detectors. Tab. A4 compares method DE-VIT against language-based detectors the COCO and LVIS-v1 datasets. Language-bar based detection, also
15

---

Table A3: Ablation studies on the region propagation based localization.
Novel
Base
Configuration
nAP50
nAP75
bAP50
bAP75
Conventional regression network Conventional regression network expanded proposal
37.7
14.6
46.5
23.9
35.6
12.5
41.3
19.8
Region propagation network
39.5
24.1
42.3
25.9
known as open-vocabulary detection [55], does not assume access to support images novel classes but requires the knowledge of the novel class names. The class names then used to discover images of novel classes from large collection of image-text pairs. Therefore, language-based de- tectors have the following limitations. First, many objects lack clear names [56], e.g., objects that are specific to certain contexts [57]. Second, the association between visual concepts and language evolving and not static [58]. contrast, few-shot object detection (FSOD) does not assume any knowledge of class names, and novel classes are described with support images only. FSOD's setting therefore more general and arguably more challenging, and aims emulate humans' capability to recognize objects by their consistent appearance.
Comparison language-based detectors on LVIS and COCO 2017. f: use customized pretrained backbone instead of public ones. "L": result that was not reported.
Table A4:
Use Extra
LVIS
COCO nAP50
Method
Backbone
Training Set
mask APr
box APr
ViLD [18]
EffNet-B7
X
26.3
27
27.6
RegionCLIP [32]
RN50x4
22
39.3
OV-DETR [59]
ViT-B/32
X
17.4
29.4
Detic [60]
RN50
17.8
27.8
MM-OVOD [61]
RNSO
25.8
MM-OVOD (10-shots) [61]
RN50
27.3
OWL-ViT [36]
ViT-L/14
X
25.6
OWL-ViT [36]
ViT-L/14t
31.2
CORA* [62]
RN50x4
28.1
43.1
Ro-ViT [63]
ViT-I/14
X
31.4
33
Co-Det [64]
Swin-B
29.4
30.6
F-VLM [63]
RN50x64
X
32.8
28.0
ViT-S/14
20.6
19.6
28.3
shots
ViT-B/14
26.4
25.0
36.1
ViT-L/14
28.5
27.9
38.3
ViT-S/14
21.8
21.9
36.1
DE-ViT (Ours)
x
10 shots
ViT-B/14
29.9
28.1
42.4
ViT-L/14
32.4
31.9
46.2
ViT-S/14
24.2
23.4
39.5
30 shots
ViT-B/14
28.5
26.8
45,4
VIT-L/14
34.3
33.6
50
In COCO, our method DE-VIT outperforms the previous SoTA CORA+ by 6.9 AP50. Our method only trains on COCO while CORA* uses ImageNet-21K [65] and COCO Captions [66] as additional training data. LVIS, DE- ViT outperforms the previous SoTA on mask APr (+1.5 over F-VLM) and box APr (+2.4 over OWL-ViT). Note that we report the Co-Det [64] result with Swin-B back- bone instead of EVA02-L backbone [67] because the latter includes the base and novel classes of COCO in its training set. Moreover, observe high variance in the performance of language- based detectors. F-VLM achieves mask APr of 32.8 LVIS but only has 28 nAP50 on COCO. While CORA* has 43.1 nAP50 on COCO but only box APr of 28.1 on LVIS. In contrast, our DE-
16

---

VIT outperforms existing solutions on both LVIS and COCO, MM-OVOD (10-shots) [10] denotes the MM-OVOD language detector with 10 images per novel classes provided as examples.
Background classes set B. We study the impacts of using different background classes sets B in Tab. A6. The first and second rows represent using the stuff (non-objed) classes, e.g., floor, sky, of COCOstuff [34] and ADE20k [68] to construct B. Note that COCOstuff has 53 stuff classes and ADE20K has 35 stuff classes, which explains the minor accuracy drop from ADE20K. The third row denotes using all classes of ADE20K as B. This indicates that adding thing (object) classes to B may be beneficial. Tab. A6 shows that changing B only results in small accuracy variations. This suggests that our proposed feature subspace projection does not rely specific prototypes.
Table A6: Ablation studies the background classes set B.
Table A5: Ablation studies on annotation types used to build prototypes.
Support Images Annotation
nAP50 10-shot
Background Classes B
nAP50
bAP50
5-shot
30-shot
COCOstuff [34]
43.1
45.6
mask bbox
43.1
43.1
43.1
ADE20k [68]
42.5
45.8
42.6
43.1
ADE20K (all)
42.1
44.7
More Shots. We study the model performance with different numbers of shots Fig. 7 and Fig. A2, on COCO 2014 and COCO 2017, correspondingly. For COCO 2014, the numbers of shots are set from to 10, 15, 20, and 30. To align with existing work, previous work [13] for shots 20,,5, For other shots, we sample within the support images mentioned above. For COCO 2017, we follow the conventional base/novel class splits of the one- shot benchmark. To measure with more robustness, we randomly select support images within the validation set COCO 2017 for each query image, and compute nAP50 for all four novel class splits. The reported nAP50 the average among all splits and choices of support images. The numbers of shots are set from to 10, 15, 20, 30, 40, 50, 75, 100.
the support images by
Table A7:
Comparison of training epoch and param- eter size to other detectors trained LVIS.
43r8 201 oox0
Total Params
Truined Purams
Epochs
APr
OWL-VIT [36]
433M
433M
1800
31.2
3
F-VLM [69]
445M
118
32.8
DE VIT (Ours)
14.4
34.3
Slint
Figure A2: Detection accuracy under different num- bers shots in COCO 2017.
Using Boxes or Masks Build Prototypes. We study the effects of annotation types such as bounding boxes masks for prototype construction in Tab. A5. We observe that using bounding boxes to build prototypes yields almost indistinguishable performance compared to using instance masks at even 5-shot in COCO.
Training Epochs and Parameter Sizes. In Tab. A7, compare the parameter sizes and training epochs of detectors trained the large-scale dataset LVIS. Tab. A7 shows that method DE-Vit only has 23M trainable parameters, and is trained orders of magnitude faster than F-VLM [69] and OWL-ViT [36].
Hyperparameters. We report the hyperparameters used in experiments in Tab. A8.
17

---

Table A8: Hyperparameters used in our experiments.
Hyperparameter
Value
Max image size
800 1333
Batch size
16
Learning rate
0.002
Optimizer
SGD
Warmup steps
5000
Learning rate schedule
stepwise decay
Learning rate decay steps Learning rate decay gamma Training steps
60000, 80000
0.1
90000
Evaluation frequency
5000 steps for COCO, 10000 steps for LVIS
Vision transformer size (layers, channel size)
small (12, 384), base (12, 784), large (24, 1024)
Channels of propagation network
256
Channels of feature subspace projection (S)
256
Classes in feature subspace projection (7)
10
Number of PL layers
for COCO, 5 for LVIS
Preparation of YCB Objects. Fig. shows the detection results of DE-ViT YCB objects, standard set of objects widely used in robotic manipulation benchmark [6]. There mis- classifications and inaccurate boxes, e.g., the white skillet is mistaken as can, all round-shape fruits are recognized orange, while the red one is an apple. However, we believe the over- all result is encouraging. The specification of YCB objects at the time this paper is written includes 72 categories. We use a total of 33 by selecting and merging certain categories. The categories in use are apple, ball, banana, bowl, brick, can, cheez-it, chips, clamp, cleanser bottle, coffee jar, comet pine, cups, drill, glass, lego, lemon, marker, mug, mustard, orange, peach, pear, peg-hole, pitcher, plate, screadriver, skillet, spray bottle, sugar box, toy airplane box, utensil, wood blocks jar. The source image in Fig. is taken from the banner picture of yebbenchmarks.com Google Image Search to collect a few sample images. Fewer than four images on average are gathered per category. We annotate the corresponding objects by instance masks in each image using the software provided by SimpleClick [70]. Similar to SAM, SimpleClick generates instance masks automatically from user clicks, which significantly simplifies and accelerates the annotation procedure. Our annotator feedback indicates that annotating masks with SimpleClick is even easier and more accurate than drawing bounding boxes. An NVIDIA 3060 GPU is used for SimpleClick software. Class prototypes for YCB objects are built from the annotated example images. During DE-ViT inference, we replace prototypes of LVIS categories with those of YCB objects in order to detect these new categories. During postprocessing, We apply class-agnostic NMS and filter small bounding boxes. The data used for demonstration will released upon acceptance.
For each category, we use
D Limitations
In this work, propose DE-VIT, few-shot detector that example images to detect novel classes without any finetuning. We demonstrate that DE-ViT establishes new state-of-the-art in few- shot and one-shot benchmarks. One of the limitations is that our method DE-ViT uses a hybrid architecture of ViT and RCNN. The region proposal network (RPN) of RCNN is trained with only base classes and frozen during inference. This potentially limits the detection performance of novel objects. To overcome this limitation, one solution to implement the RPN with our proposed framework as well. In doing so, the object proposals can be generated by conditioning the prototypes of novel objects. Another solution is to adopt full transformer architecture such as DETR [71] to eliminate the object proposal stage.
18

---

Our method DE-ViT adopts prototype-feature interactions, instead of dense spatial feature interac- tions that are computationally heavy. In doing so, our method Di- archives greater accuracy and efficiency. However, class representative prototypes can potentially earry stronger bias from the pretrained ViTs such as DINOv2 than low-level spatial features. This would affect the model performance on long-taildd domains where the existing feature extractors underperform. A promis- ing future direction is design more adaptive and awar- architecture balance the efficient prototype-feature interactions and expensive spatial feature interactions.
19


Reviews:
[{"review_id": "pydelgen6Q", "reviewer": "Reviewer_dhu9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces DE-ViT, a novel few-shot object detection method that leverages pretrained vision transformers (ViTs) without the need for fine-tuning. The main idea is to address the limitations of traditional few-shot detection methods, which often require fine-tuning on both base and novel classes, leading to complex procedures and significant accuracy gaps between these classes. DE-ViT utilizes ViT features to detect novel objects based on a small set of support images, overcoming the need for fine-tuning by employing a dot-product with prototypes and a region propagation network to refine localization and derive class scores.  Experiment results show that DE-ViT achieves new state-of-the-art results on several benchmarks, including Pascal VOC, COCO, and LVIS, surpassing previous methods by significant margins in terms of both accuracy and inference time. It also demonstrates the effectiveness of DE-ViT in a real-world robotic task, showcasing its ability to detect and sort novel objects based on example images without any further training.", "summary_of_recommendation": "The overall logic of the paper is relatively clear, and a relatively complete method is designed based on the shortcomings of fine-tuning. In particular, the paper designed a large number of experiments, which achieved good results on datasets such as COCO and LVIS, and designed a series of ablation experiments to demonstrate the rationality of the module design. At the same time, this work also conducted experiments in the fields of actual robot operation and open-vocabulary object detection, and achieved good results. Therefore, this work has made good progress in the fields of robot perception and computer vision. But considering the relavant with robotics, I will not argue for my recommendation if other reviewers have a different opinion.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "1. There is an article named “One-shot object detection without fine-tuning”, which is also for the purpose of avoiding finetune, and can be compared with the results of this article if necessary. [1]\n2. In your Table A4, the model using 30-shot is compared with open-vocabulary detectors. However, 30-shot is generally too much for vision-language models (e.g. [2] only used 10 visual examples for LVIS), so it’s better for you if you can provide experimental results with fewer shots and compare them to other methods in your Table A4.\n3. In your Table A3, you conducted a series of ablation experiments. However, the experiments on individual parts need to be supplemented, such as solely Region Propagation Network.\n\n[1] Li X, Zhang L, Chen Y P, et al. One-shot object detection without fine-tuning[J]. arXiv preprint arXiv:2005.03819, 2020.\n[2] Kaul P, Xie W, Zisserman A. Multi-modal classifiers for open-vocabulary object detection[C]//International Conference on Machine Learning. PMLR, 2023: 15946-15969.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y4ZEyzyOsI", "reviewer": "Reviewer_dSpH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a new approach to few-shot object detection without fine-tuning. The approach leverages a dot-product between  ViT features + prototypes to project features into a subspace relevant for the task. Importantly, the authors note that ViT features do not contain good information for regressing bounding boxes and therefore the authors propose a region propagation  layer that can be stacked to ultimately produce accurate 2d bounding boxes. The experimental analysis showcases the effectiveness of the approach on standard few-shot detection benchmarks compared to alternative approaches as well as vision-language detection models. Finally, the paper shows a downstream robotics application of the approach to sort new objects.", "summary_of_recommendation": "Overall, the paper presents an interesting new approach to few-shot object detection without fine-tuning. While I am not an expert in this area, the experimental analysis appears to be demonstrate the effectiveness of the method and I believe the paper, experiments and code would be useful to the community. I am open to revising my opining given other data, but for now I would argue for a weak accept.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- What are the details of the channel re-order operation? (see weaknesses above)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UMn5akJEMF", "reviewer": "Reviewer_3v8u", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel few-shot object detection (FSOD) framework that achieves SOTA on various benchmarks and is demonstrated on YCB objects in a pick-and-place setting. The FSOD method uses an iterative proposal refinement process for localization and class prototypes to project pretrained ViT features into a subspace for processing and final classification. These prototypes can be of novel classes which enables few-shot object detection. This is achieved with novel “Propagation Layer” and “Feature Subspace Projection” modules, taking advantage of strong feature representations from a pretrained ViT.", "summary_of_recommendation": "This paper presents a novel few shot learning approach with strong performance on benchmarks. Furthermore, the authors create a framework for application in a pick-and-place setting demonstrating the viability of immediate application in robotics settings. However, there is a lack of a limitations section.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "Does Table A1’s secs/img include latency/cost of the lightweight prototype classifier to find the T classes (line 145)?\n\nWere position embeddings (learned or sinusoidal) added post-hoc to the extracted ViT features for the “conventional regression” localization experiments (Table A3)?\n\nIs ROI Align necessary in the “feature subspace projection”? Could the expanded proposal be aligned to pixels, rather than the original proposal box?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method for few-shot object detection that leverages pre-trained ViT models. A new region propagation network is designed to refine the location of region proposals.

Strengths

- The reviewers agree on the novelty of the proposed method for  few-shot object detection.
- The experimental results in the paper are convincing with comparisons to previous works.

Weaknesses
- The reviewers have raised several questions regarding clarifications of details in the paper and missing citations.

Post-Rebuttal
- The authors have successfully addressed the concerns from the reviewers.
Decision: Accept