Prototypical VoteNet for Few-Shot 3D Point Cloud Object Detection
Authors: Shizhen Zhao
XIAOJUAN QI
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D Point Cloud Object Detection
Few-Shot Learning
Geometric Prototype
Class-Specific Prototype

Abstract: Most existing 3D point cloud object detection approaches heavily rely on large amounts of labeled training data. However, the labeling process is costly and time-consuming. This paper considers few-shot 3D point cloud object detection, where only a few annotated samples of novel classes are needed with abundant samples of base classes. To this end, we propose Prototypical VoteNet to recognize and localize novel instances, which incorporates two new modules: Prototypical Vote Module (PVM) and Prototypical Head Module (PHM). Specifically, as the 3D basic geometric structures can be shared among categories, PVM is designed to leverage class-agnostic geometric prototypes, which are learned from base classes, to refine local features of novel categories. Then PHM is proposed to utilize class prototypes to enhance the global feature of each object, facilitating subsequent object localization and classification, which is trained by the episodic training strategy. To evaluate the model in this new setting, we contribute two new benchmark datasets, FS-ScanNet and FS-SUNRGBD. We conduct extensive experiments to demonstrate the effectiveness of Prototypical VoteNet, and our proposed method shows significant and consistent improvements compared to baselines on two benchmark datasets. 

Prototypical VoteNet for Few-Shot 3D Point Cloud Object Detection
Shizhen Zhao, Xiaojuan Qi' The University of Hong Kong {zhaosz, xjq)) hku. hk
Abstract
Most existing 3D point cloud object detection approaches heavily rely on large amounts of labeled training data. However, the labeling process is costly and time-consuming. This paper considers few-shot 3D point cloud object detection, where only few annotated samples of novel classes are needed with abundant samples of base classes. To this end, we propose Prototypical VoteNet to recognize and localize novel instances, which incorporates Vote Module (PVM) and Prototypical Head Module (PHM). Specifically, as the 3D basic geometric structures can be shared among categories, PVM is designed to leverage class-agnostic geometric prototypes, which are Iearned from base classes, refine local features of novel categories. Then PHM is proposed utilize class prototypes to enhance the global feature of each object, facilitating subsequent object localization and classification, which trained by the episodic training strategy. To evaluate the model in this new setting, we contribute two new benchmark datasets, NS-ScanNet and FS-SUNRGBD. We conduct extensive experiments to demonstrate the effectiveness of Prototypical VoteNet, and our proposed method shows significant and consistent improvements compared baselines on two benchmark datasets. This project will be available at https; //shi fehizhen-ao zhao github. io/FS3D_page/
new modules: Prototypical
Introduction
3D object detection aims to localize and recognize objects from point clouds with many applications in augmented reality, autonomous driving, and roboties manipulation. Recently, number of fully supervised 3D object detection approaches have made remarkable progress with deep learning [22 18 31 24). Nonetheless, their success heavily relies on large amounts of labeled training data, which are tine-consuming and costly to obtain. On the contrary, a human can quickly learn to recognize novel classes by seeing only few samples. To imitate such human ability, we consider few-shot point cloud 3D object detection, which aims to train model to recognize novel categorizes from limited annotated samples of novel classes together with sufficient annotated data of base classes.
Few-shot learning has been extensively studied in various 2D visual understanding tasks such as object detection [39 40
32 and semantic segmentation 23 Early attempts 9 38 employ meta-learning to learn transferable knowledge from
43 45
image classification [4] 9
21 48 99
collection of tasks and attained remarkable progress. Recently, benefited from large-scale datasets (e.g. ImageNet (6] and advanced pre-training methods [27 49 10, (54), finetuning large-scale pre- trained visual models down-stream few-shot datasets emerges
effective approach to address
this problem [33 39 (55). Among different streams of work, cetottpe-bbsce methods 42 [53] 20 [7]
have been incorporated into both streams and show the great advantages, since they can capture the
*Correponniigg author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

3D Basic Geometric Structure
Figure 1: Illustration of the basic geometry of 3D objects, which can be shared among classes.
representative features of categories that can be further utilized for feature refinement 45
classification [26] 32)
This motivates us to explore effective 3D cues to build prototypes for few-shot 3D detection. Different from 2D visual data, 3D data can get rid of distortions caused by perspective projections, and offer geometric cues with accurate shape and scale information. Besides, 3D primitives to constitute objects can often be shared among different categories. For instance, as shown in Figure rectangular plates and corners can be found in many categories. Based these observations, in this work, propose Prototypical VoteNet, which employs such robust 3D shape and primitive clues to design geometric prototypes to facilitate representation learning in the few-shot setting.
Prototypical VoteNet incorporates two new modules, namely Prototypical Vote Module (PVM) and Prototypical Head Module (PHM), to enhance local and global feature learning, respectively, for few-shot 3D detection. Specifically, based on extracted features from a backbone network (i.e. PointNet++ (25 PVM firstly constructs a class-agnostic 3D primitive memory bank to store geometric prototypes, which are shared by all categories and updated iteratively during training. To exploit the transferability of geometric structures, PVM then incorporates a multi-head cross-attention module to associate geometric prototypes with points in a given scene and utilize them to refine their feature representations. PVM is majorly developed to exploit shared geometric structures among base and novel categories enhance feature learning of local information in the few-shot setting. Further, to facilitate learning discriminative features for object categorization, PHM is designed employ multi-heas cross- aotssnatttttoon module and leverage class-specific prototypes from a few support samples to refine global representations of objects. Moreover, episodic training 32 38 is adopted to simulate few-shot circumstances, where PHM is trained by a distribution of similar few-shot tasks instead of only one target object detection task.
Our contributions are listed as follows:
We are the first to study the promising few-shot 3D point cloud object detection task, which allows a model detect new classes, given few examples.
We propose Prototypical VoteNet, which incorporates Prototypical Vote Module and Proto- typical Head Module, to address this new challenge. Prototypical Vote Module leverages class-agnostic geometric prototypes enhance the local features of novel samples. Proto- typical Head Module utilizes the slass-specific prototypes to refine the object features with the aid of episode training.
We contribute two new benchmark dataset settings called FS Se-ScanNet and FS-SUNRGBD, which are specifically designed for this problem. Our experimental results on these two benchmark datasets show that the proposed model effectively addresses the few-shot 3D point cloud object detection problem, yielding significant improvement over several compet- itive baseline approaches.
2 Related Work
3D Point Cloud Object Detection. Current 3D point cloud object detection approaches can be divided into two streams: Grid Projection/Voxelzzaiin
41 and point- based (30, 22 18 2] The former projects point cloud to 2D grids or voxels so that the
based [46
15
35
52
advanced convolutional networks can be directly applied. The latter methods take the raw point cloud

---

Pruiotypical Vote Module
Prototypical Head Medule
Query
Object Features
Cross Attention Module
Backbone
Add Norz
Samping Grouping
Attention Module
Predectonn
Layes
Norma
Protesyne
Support
Class Prulutypes
Shared
Pooling
VouNee Sylle Conpaeeen
Network
Figure 2: Illustration of Prototypical VoteNet. Prototypical VoteNet introduces two modules for few-shot 3D detection: 1) Prototypical Vote Module for enhancing local feature representation of novel samples by leveraging the geometric prototypes, 2) Prototypical Head Module for refining global features of novel objects, utilizing the class-specific prototypes.
feature extraction network such as PointNet++ [25 to generate point-wise features for the subsequent detection. Although these fully supervised approaches achieved promising 3D detection performance. their requirement for large amounts of training data precludes their application in many real- real-world scenarios where training data costly or hard to acquire. To alleviate this limitation, we explore the direction of few-shot 3D object detection in this paper.
Few-Shot Recognition. Few-shot recognition aims elassify novel instances with abundant base samples and few novel samples. Simple pre-training and finetuning approaches first train the model on the base classes, then finetune the model on the novel categories B 7.. Meta-learning based methods 19 16 I1
are proposed to Iearn classifier across tasks and then transfer to the few-shot classification task. The most related work Prototypical Network (32), which represents class as one prototype so that classification can be performed by computing distances to the prototype representation of each class. The above works mainly focus on 2D image understanding. Recently, some few-shot learning approaches for point cloud understanding [29 51 47 are proposed. For instance, Sharma er al. [51] propose graph- method to propagate the knowledge from few-shot samples to the input point cloud. However, there is work studying few-shot 3D point cloud object detection. this paper, we first study this problem and introduce the spirit of Prototypical Network into few-shot 3D object detection with 3D geometric prototypes and 3D class-specific prototypes.
38 32
2D Few-shot Object Detection. Most existing 2D few-shot detectors employ meta-learning 440 14 (5) or fine-tuning based mechanism [44 43 26 36). Particularly, Kang et al. [14) propose one-stage few-shot detector which contains meta feature learner and feature re-weighting module. 45 presents meta-learning over Rol (Region-of Intereest) features and incorporates it
Meta R-CNN
2 and Mask R-CNN I TFA 39 reveals that simply fine-tuning the box classifier and regressor outperforms many meta- meta-learning based methods. Cao et al. [I] improve the few-shot detection performance by associating each novel class with a well-trained base class based on their semantic similarity.
into Faster R-CNN
Our Approach
In few-shot 3D point cloud object detection (FS3D), the object class set C is split into Crase and Cnovel such that C Cruse U Cnovel and Cluse n C = 0. For each class E C, its annotation dataset T, contains all the data samples with object bounding boxes, that is T, = {(u, P)lu E R Nx3 Here, (u, P) is 3D object bounding box (x, y, 2, h, w. representing box center locations and box dimensions, a point cloud scene P.
There are only few examples/shots for each novel class E Cnwell which are known support samples. Besides, there are plenty of annotated samples for each base class r E Chuse- Given the above dataset, FS3D aims to train model to detect object instances in the novel classes leveraging

---

such sufficient annotations for base categories Caaee and limited annotations for novel categories Caovel-
In the following, we introduce Prototypical VoteNet for few-shot 3D object detection. We will describe the preliminaries of our framework in Section 3.1] which adopts the architecture of VoteNet- style 3D detectors [24] 50 2). Then, we present Prototypical VoteNet consisting of Prototypical Vote Module (Section 3.2. and Prototypical Head Module (Section 3.2.2) to enhance feature learning for FS3D.
3.1 Preliminaries
VoteNet-style 3D detectors 24 50, [ takes point cloud scene P. as input, and localizes and categorizes 3D objects. As shown in Figure 2 it firstly incorporates a 3D backbone network (i.e.
PointNet++ (25)) parameterized by 0, with downsampling layers for point feature extraction as Equation
where N and M represent the original and subsampled number of points, respectively, P, R represents an input point cloud scene i, and F; E R x (3+d) is the subsampled scene points (also called seeds) with d-dimensional features and 3-dimensional location coordinates.
Then, F; is fed into the vote module with parameters 02 which outputs 3-dimensional coordinate
offset Ad; = (Az,, Auj, Azy) relative to its corresponding object center c = (Ca: Cys C2) and a residual feature vector ffy for each point j in F; {ff}: in Equation (2.
(Adj, Af,): } (F;;
(2)
Given the predicted offset Adj. the estimated corresponding object center ej = (Car,) Cysy C2,) that point belongs to be calculated Equation
Cxz = Fi + Ary Cyy = yJ + Ayj, C2 = 2j + Azj.
Similarly, the point features are updated as F. +- F, + AF; where AF, (Affk.
Next, the detector samples object centers from {[ez, Cy. Cz. ) using farthest point sampling and group points with nearby centers together (see Figure 2 Sampling & grouping) to form a set of object proposals O = {oe}: J:- Each object proposal is characterized by feature vector fo, which is obtained by applying a max pooling operation on features of all points belonging to
Further, equipped with object features {for Ji, the prediction layer with parameters 03 is adopted to yield the bounding boxes 6, objectiveness scores 8,, and classification logits ry for each object proposal o following Equation
Sb, = Oz)..
3.2 Prototypical VoteNet
Here, we present Prototypical VoteNet which incorporates two new designs Prototypical Vote Module (PVM) and Prototypical Head Module (PHM) improve feature learning for novel categories with few annotated samples (see Figure 2 Specifically, PVM builds a class-agnostic memory bank of geometric prototypes g = {9k 1 with size of K, which models transferable class-agnostic 3D primitives learned from rich base categories, and further employs them to enhance local feature representation for novel categories via multi-head cross-atte cooss-attention module. The enhanced features are then utilized by the Vote Layer to output the offset of coordinates and features as Equation (2). Second, to facilitate learning discriminative features for novel class prediction, PHM employs an attention-bart based design to leverage dlass-speciffe prototypes 3 ) extracted from the support set Duupoon with R categories
refine global discriminate feature for representing each object proposal (see Figure 2) The output features are fed to the prediction layer for producing results as Equation ④ To make the model more generalizable to novel classes, we exploit the episodic
training B2 38 strategy to train PHM, where a distribution of similar few-shot tasks instead of
only one object detection task is learned in the training phase. PVM and PHM are elaborated in the following sections.

---

1300
1240
1550
1300
Toilet
1900
590
500
510
500
60
90
100
100
20
40
60
100
1.00
20
90
180
20
40
60
100
1.20
Chair
Desk
1933
Cubinct
Hathtup
1000
1160
1900
500
500
11
Figure 3: Visualization of the assignment of object point features to the geometric prototypes, where the horizontal axis represents the index of geometric prototypes and the vertical axis represents the number of assignments.
3.2.1 Prototypical Vote Module
Given input features F; extracted by a backbone network, Prototypical Vote Module constructs class-agnostic geometric prototypes G = {9x ) and then uses them to enhance local point features with an attention module.
Geometric Prototype Construction. At the beginning, g = {g E R K is randomly initialized. During training, G is iteratively updated with a momentum based on point features of foreground objects. Specifically, for each update, given G = {gx } and all the foreground points J where Mf the number of foreground points in the current batch, we assign each point to its nearest geometric prototype based on feature space distance. Then, for each prototype Jk, we have a group of points {pm h with features represented as {fm h& assigned to it. Point features in one group averaged to update the corresponding geometrie prototype as Equation (5).
My {pm)m= with features [
gh - 7 * + (1-7)Jk- where J, = average((fm)ss)
(5)
Here E [0.1] [] is the momentum coefficient for updating geometric prototypes in a momentum manner, serving moving average over all training samples. Since one point feature is related to one geometric prototype, we call this one-hot assignment strategy as hard assignment. An alternative the hard assignment the soft assignment, which calculates the similarity between point features with all geometric prototypes. Empirically, we found that hard assignment results in more effective grouping versus soft assignment. More details can be found in the supplementary material.
Geometric Prototypical Guided Local Feature Enrichment. Given the geometric prototypes = [gk) kk and point features F; {fz); of scene i, PVM further employs multi-head cross-attention module (37] to refine the point features. Specifically, the multi-head attention network uses the point features F: } as query, geometrie prototypes G
) as key and value where linear transformation functions with weights represented as Qn, Un, V,, are applied to encode query, key and value respectively. Here, h represents the head index. Then, for each head h, the query point feature is updated by softly aggregating the value features where the soft aggregation weight is determined by the similarity between the query point feature and corresponding key feature. The final point feature f, is updated by summing over outputs from all heads as Equation (
K
expl(Qn.A) K EE es
fa + Cross conis_At(fff, (ox3)
= Σ
WA
Alie-Vngr) Vn.g),, where A 1=1
1=1
Here, A/e, is the soft aggregation weight considering the similarity between the j-th query point feature and the k-th key feature and used to weight the k-tth value feature. Through this process, the point feature is refined using geometric prototypes in a weighted manner where prototypes similar to the query point feature will have higher attention weights. This mechanism transfers geometric prototypes learned from base categories with abundant data to model novel points. The multi-head design enables the model to seek similarity measurements from different angles in learnable manner
5

---

Dable
Tubli
Tedett
(a)
(b)
Figure 4: t-SNE visualization of the effect of PHM. (a) shows the features without being processed
by PHM. (b) shows the features processed by PHM using class-specific prototypes.
to improve robustness. Additionally, in both PHM and PVM, the multi-head attention layer combined with feed forward FC layers. After refining point features (ff PVM predicts the point
offset and residual feature vector { (Ad,, AFj): ) stated in Equation (2) Adj is explicitly supervised by a regression loss Lvate used in [25
What do the geometric prototypes store? To shed more insights on what the geometric prototypes represent, we visualize the frequency of geometry prototypes in different categorics using the "assignment". The number of "assignment" 's of object point features to the geometric prototypes is shown in Figure where a point is assigned to the geometric prototype with the highest similarity. In each histogram, the horizontal axis represents the index of geometric prototypes and the vertical axis represents the number of assignments. Note that the first row is the novel classes and the second row is the base classes. Figure shows that visually similar categories have a similar assignment histogram since they share the basic geometric structures. This indicates that the memory bank of geometric prototypes successfully learns the 3D basic geometric knowledge, which can be bridge to transfer the geometric knowledge from base classes to novel ones.
3.2.2 Prototypical Head Module
given object proposals O, fo with features (So } from Sampling & Grouping module, PHM module leverages class-specific prototypes (ex) to refine the object features fo, for the subsequent object classification and localization. Moreover, for better generalizing to novel categories, PHM is trained by the episodic training scheme, where PHM learns a large number of similar few-shot tasks instead of only one task. Considering the function of PHM, we construct the few-shot tasks that, in each iteration, PHM refines the object features {fo, Ji with the aid of class-specific prototypes, which are extracted from randomly sampled support samples.
As shown in Figure 2
In each few-shot task, class- spasss- prototypes are built based on support sets that are randomly sampled. For class r, the class-specific prototype e, is obtained by averaging the instance features for all support samples in class r. The instance feature is derived by applying max pooling operation that instance. As shown in Figure 2 with class-specific prototypes 3 = { R for a total of R categories, РHM further employs a multi-head cross-attention module refine object features {fo, J. Here, the object features [for serve as the query features, class-specific prototypes are used to build value features and key features similar as what has been described in Section I.2.1 Then, the representation fo, of each proposal o, is refined using the outputs of the multi-head attention module, which are weighted sum over the value features and the weight is proportionally the similarity between the query feature and corresponding key features. This process can be formulated as Equation (7). which is similar to Equation (6).
over the features of all points belonging
Jo, Cross_Att( (er)).
Until now, fo, is refined using class-specific prototypes, injecting class-specific features from given support samples into object-level features. Finally, the refined object features {for J are fed into the prediction module following Equation
What does PHM help? Figure visualizes the effect of feature refinement by PHM. The experiment conducted on FS-ScanNet. Figure shows the object features, which have not been processed by PHM. Figure 4b) shows the object features processed by PHM. We could observe that, after

---

the feature refinement by PHM. the features of each classes become more compact compared to the non-PHM one, which further validates the effectiveness of PHM.
3.3 Model Training
The model is trained by the episodic training strategy 32 38
The detailed training strategy is included in the supplementary material. We use the standard cross entropy loss Las and smooth- L loss (28] Leg to supervise the classification and the bounding box regression, respectively. As for the objectness prediction, if a vote is located either within 0.3 meters to ground truth object center or more than 0.6 meters from any center, it is considered to be positive (24), which is supervised by a cross entropy loss Labj- Therefore, the overall loss for Prototypical VoteNet is given by,
Lat Las (r Lreg a22 + or aryev
(8)
where 0 02, 0g is the coefficients balance the loss contributions.
4 Experiments
To our best knowledge, there is no prior study of few-shot point cloud object detection. Therefore, setup new benchmark which is described in Section & 4.2 Then, we conduct experiments and compare our method with baseline methods in 4.3 hird, a series of ablation studies are In addition, the implementation
performed for further analyzing Prototypical VoteNet in Section details are included the supplementary material.
4.1 Benchmark Setup
Datasets. We construct two new benchmark datasets FS-SUNRGBD and FS- FS-ScanNet. Specifically,
ES-SUNRGED derived from SUNRGBD (34. SUNRGBD consists of 5K RGB-D training images annotated, and the standard evaluation protocol reports performance on 10 categories. We randomly select 4 classes as the novel ones while keeping the remaining ones as the base. In the training set, only K annotated bounding boxes for each novel class are given, where k equals 1, 2, 3, and 5. FS-ScanNet is derived from ScanNet [5] ScanNet consists of 1,513 point clouds, and the annotation of the point clouds corresponds to 18 semantic classes plus one for the unannotated space. Out of its 18 object categories, we randomly select 6 classes as the novel ones, while keeping the remaining the base. We evaluate with 2 different base/novel splits. In the training set, only K annotated bounding boxes for each novel class are given, where k equals 1, 3 and More details about the new benchmark datasets can be referred to the supplementary material.
Evaluation Metrics. We follow the standard evaluation protocol
in 3D point cloud object detection by using mean Average Precision(mAP) under different IoU thresholds (i.e. 0.25, 0.50), denoted as AP25 and 50 In addition, in the inference stage, we detect both novel classes and base classes. The performance on base classes is included in the supplementary material.
4.2 Benchmark Few-shot 3D Object Detection
We build the first benchmark for few-shot 3D object detection. The benchmark incorporates competitive methods, and three of them are built with few-shot learning strategies, which have been shown to be successful in 2D few-shot object detection.
Baseline: We abandon PVM and PHM, and train the detector on the base and novel classes together. In this way, it can learn good features from the base classes that are applicable for detecting novel classes.
TFA [39]: We incorporate a well-designed few-shot object detection method TFA (39 with VoteNet. TFA first trains VoteNet on the training set with abundant samples of base classes. Then only the classifier and the regressor are finetuned with small balance set containing both base classes and novel classes.
VoteNet+PT++TF: The pretraining is proven to be important in 2D few-shot learning, as it learns more generic features, facilitating knowledge transfer from base classes to novel classes. Therefore, we add pretraining stage, which is borrowed from a self-supervised point cloud contrastive learning method [13]. before the training stage of VoteNet+TFA.
7

---

Novel Split
Novel Split 2
Method
I-shot
3-shot
5-shot
1-shot
3-shot
5-shot
AP25
APso
AP2s
AP50
AP25
AP3o
AP25
AP50
AP25
APso
AP25
AP50
Baseline
9.21
3.14
22.64
9.04
24.93
12.82
4.92
0.94
15.86
3.15
20.72
6.13
VoteNet+
0.48
0.09
8.07
1.03
16.36
7.91
1,00
0.15
2.64
0.22
5.71
2.40
PoIeNet+PT+TFA
2.58
1.04
10.37
2.13
17.21
8.94
2.13
0.56
4.85
1.25
7.25
2.49
Meta VoteNet
11.01
4.20
25.73
10.99
26.68
14,40
6.06
1.01
16.93
4.51
23.83
7.17
Ours
15.34
8.25
31.25
16.01
32.25
19.52
11.01
2.21
21.14
8,39
28.52
12.35
Table 1: Results on FS-ScanNet using mean Average Precision (mAP) at two different IoU thresholds of 0.25 and 0.50, denoted as 25 and AP50-
1-shot
2-shot
3-shot
4-shot
5-shot
Method
AP25
AP50
AP25
APso
AP25
AP50
AP25
AP5o
AP25
AP5o
Baseline
5.46
0.22
6.52
0.77
13.73
2.20
20.47
4.50
22.99
5.90
VoteNet+TTFA
1.41
0.03
3.70
0.78
4.03
1.09
7.91
2.10
8.50
2.81
VoteNet+PT+TT
3,40
0.51
5.13
1.22
7.94
2.31
10.05
3.12
11.32
4,01
Meta VoteNet
7.04
0.98
9.23
1.34
16.24
3.12
20.10
4.69
24.41
6.05
Ours
12.39
1.52
14.54
3.05
21.51
6.13
24.78
7.17
29.95
8.16
Table 2: Results on FS-SUNRGED using mean Average Precision (mAP) at two different IoU thresholds of 0.25 and 0.50, denoted as AP and AP50-
Meta RCNN develops a meta-learning prediction head, which leverages class prototypes to multiply the Rol features and then predicts if Rol feature is the category that the class prototype belongs to. In VoteNet, the grouped point features after the vote stage can be treated as the Rol features.
Meta VoteNet:
Meta VoteNet is inspired by Meta RCNN 45
4.3 Main Results
FS-ScanNet. We first compare the proposed approach with the basclines on FS-ScanNet in Table I These results show: 1) VoteNet+TFA has the worst performance. For example, it only achieves 8.07% AP25 and 1.03% AP50 on 3-shot in Novel Split-1. The reason is that VoteNet+TTAA is trained from scratch, and only a few layers are funetuned by novel samples. Therefore, it tends to be overfitting with poor generalization ability novel classes. 2) Even adding pretaining stage to VoteNet+TFA, which is termed as VoteNet+PT+TTA, it still performs poorly. For example, it only contributes the improvement of +2.30% AP25 and 10% AP to VoteNet+TTFA is different from 2D few-shot object detection. This is potentially caused by the lack of large-scale datasets for pre-training in 3D, while the 2D backbone ResNet [12) has dataset of over one million images for pre- pre-training. 3) Besides our designed method, Meta VoteNet is superior. For example, Meta VoteNet reaches 25.73% AP25 and 16.01% AP on 3-shot in Novel Split- This shows the meta-learning prediction head, derived from Meta RCNN 45). helps to generalize from base classes to novel classes. 4) Our new model Prototypical VoteNet surpasses all competitors by significant margins. For example, Prototypical VoteNet achieves 32.25% AP25 and 16.01 % AP30 on 3-shot in Novel Split- 1. This is because our proposed method equipped with a more generic vote module by learning geometric prototypes, and leverage class prototypes to promote the discriminative feature learning.
3-shot in Novel Split-1. This
ES-SUNRGBD. Table 2 shows the result comparison on FS "S-SUNRGBD. FS-SUNRGBD, we find that VoteNet+TFA and VoteNet+PT+TFA still have large gap with other methods due to the overfitting and insufficient pretraining. Meanwhile, Meta VoteNet outperforms other baseline methods. For example, on 3-shot, Meta VoteNet surpasses VoteNet+PT+TFA by +8.34% AP25 and +0.81 31% AP and achieves 16.24% AP25 and 3. 3.12% AP Finally, our designed Prototypical VoteNet outperforms all the methods. For instance, on 3-shot, Prototypical VoteNet surpasses Meta VoteNet by +5.37% AP25 and +3.01 % APs0, which further validates the effectiveness of our proposed PVM and PHM.
8

---

4.4 Further Analysis
3-shot
5-shot
Method
Prototype
APn.25
AP0.50
AP25
AP50
AP25
AP
Geometric
31.25
16.01
Baseline
22.64
9.04
24.93
12.82
PVM
Self-learning
28.34
14.01
+PVM
27.43
13.63
28.44
16.45
Class
31.25
16.01
+PHM
28.76
14.04
30.13
17.51
РHM
Self-learning
27.45
13.67
+PVM+PHM
31.25
16.01
32.25
19.52
Table 4: Ablation study of Prototypes.
Table 3: Ablation study of individual components.
Contributions of individual components. We conduct ablation studies on 3-shot and 5-shot in split-1 of FS-ScanNet with results shown in Table The results show that both of them are effective on their own. For example, on 3-shot, PVM contributes the improvement of +4.19% AP25 and +4.59% APso. and PHM contributes the improvement of +6. 12% AP25 and +5.00% AP50. Moreover, when combined, the best performance, 31.25% AP25 and 16.01 % AP is achieved.
Effectiveness of Prototypes. Table shows the effectiveness of two kinds of prototypes. order to validate the geometric prototypes, we displace them by the self-learning ones, which are randomly initialized and updated by the gradient descend during the model training. Table f results show that the performance significantly degrades. To validate the effectiveness of class prototypes, we also alter them by the randomly initialized self-learning prototypes. Similarly, the performance drops drastically due to the lack of class prototypes.
Prototype
APo.25
AP
Coefficient m
APo.25
AP0,50
K = 30
29.98
15.01
7 0.2
29.50
14.65
K 60
30.24
15.54
7 = 0.9
30.55
15.15
K 90
31.10
15.98
Y 0.99
30.71
15.30
K -120
31.25
16.01
7 = 0.999
31.01
15.89
K 150
31.01
15.89
Y 0.9999
30.90
15.79
Table 5: Ablation study of memory bank size.
Table 6: Ablation study of coefficient 7
Size of Memory Bank. Table studies the size of the memory bank containing the geometric prototypes. This ablation study is performed on 3-shot in split-1 of FS-ScanNet. The value of K is set to (30, 60, 90, 120, 150]. For = 30, the memory bank only contains 30 geometric prototypes, which only achieves 29.98% AP25 and 15.01 % AP Furthermore, when using more prototypes (i.c., K 120), there will be an obvious performance improvement, which reaches 31.25% AP25 and 16.01% AP However, when continuous increasing K, there will be no improvement. Therefore, we set the size of the memory bank to 120.
Coefficient Table 6 shows the effect of momentum coefficient (y in Equation ( ment is performed on 3-shot in split of FS-ScanNet. The results show that, when using a relatively large coefficient (i.e., 7 E (0.999, 0. 0.9999)). the model performs well, compared with the model using a small momentum coefficient (i.e. 7 E (0.9, 0.99). Moreover, the performance drops when using a small value of 7 0.2. The is potentially because small momentum coefficient might bring about unstable prototype representation with rapid prototype updating.
The experi-
Novel Split
Novel Split 2
Method
I-shot
I-shot
5-shot
I-shot
3-shot
5-shot
AP25
AP50
AP25
AP50
AP25
AP50
AP25
AP50
AP25
APso
AP25
AP50
TOFA
0.48
0.09
8.07
1.03
16.36
7.91
1.00
0.15
2.64
0.22
5.71
2.40
TOAA
10.38
3.96
23.77
9.83
26.02
13.96
5.12
0.95
16.23
3.72
21.89
6.76
Ours
15.34
8.25
31.25
16.01
32,25
19.52
11.01
2.21
21.14
8.39
28.52
12.35
Table 7: Results of VoteNet+TFA" on FS-ScanNet using mean Average Precision (mAP) at two different loU thresholds of 0.25 and 0.50, denoted as AP25 and AP30-
More Analysis on TFA. TFA 33 a simple yet effective method few-shot 2D object detection. Specifically, TFA first trains detector the training set with abundant samples of base classes

---

3-shot
5-shot
Method
AP25
AP50
AP25
AP50
GroupFree + DeFRCN
25.22
10.90
26.42
14.01
GroupFree + FADI
25.73
11.02
27.12
14.32
3DETR + DeFRCN
26.01
10.95
26.88
14.45
3DETR + FADI
26.24
11.12
26.93
15.22
Ours
31.25
16.01
32.25
19.52
Table 8: More Methods Borrowed From 2D Few-Shot Object detection
and then finetunes the classifier and the regressor on a small balanced set, which contains both base classes and novel classes. More details be referred in [39] However, shown in Table I and Table 2 in the main paper, VoteNet+TFA performs poorly in few-shot 3D object detection. As discussed in Section 4.3 in the main paper, the reason that few layers are finetuned by the samples of novel classes. A question arises here: why does TFA in few-shot 2D object detection only need train the classifier and the regressor on the novel samples? We speculate that the large-scale pre- pre-training on ImageNet [6] helps. To overcome this problem, in the first stage of TFA, we train the VoteNet on the training set with both novel classes and base classes. Then, the second stage, we use small balanced set containing both novel classes and base classes to finetune the classifier and the regressor. We denote this new baseline as VoteNet+TFA As shown Table VoteNet+TFF" boosts the performance significantly. For example, in 3-shot in split-1 of FS FS-ScanNet, VoteNet+TFA can achieve 26. 26.02% AP25 and 13.96% AP50- 50. while VoteNet+ only reaches 16.36% AP25 and 7.91% % AP
More Methods Borrowed From 2D Few-Shot Object detection. We combine two SOTA 2D few-shot object detection techniques (i.e. DeFRCN 226.. FADI (I) and two SOTA 3D detectors (i.e. GroupFree [18), 3DETR (22)). These two few-shot techniques are plug-in-play modules and can be casily incorporated into the different detection architectures. We conducted this experiment on 3-shot and 5-shot in split-1 of FS-ScanNet. The results Table R show that our method still surpasses these methods by a large margin. This is potentially because, in the 2D domain, they often build their model upon a large-scale pre-trained model on ImageNet. However, in the 3D community, there does not exist large-scale dataset for model pre-training, which requires future investigations. Therefore, these 2D few-shot object detection techniques might not be directly transferable to the 3D domain. For future works, we might resort the pre-training models in the 2D domain to facilitate the few-shot generalization on 3D few-shot learning and how these techniques can be combined with our method.
Concluding Remarks
In this paper, we have presented Prototypical VoteNet for FS3D along with a new benchmark for evaluation. Prototypical VoteNet enjoys the advantages of new designs, namely Prototypical Vote Module (PVM) and Prototypical Head Module (PHM), for enhancing feature learning in the few-shot setting. Specifically, PVM exploits geometric prototypes learned from base categories to refine local features of novel categories. PHM is proposed to utilize class-specific prototypes to promote discriminativeness of object-level features. Extensive experiments on two new benchmark datasets demonstrate the superiority of our approach. We hope our studies on 3D propotypes and proposed new benchmark could inspire further investigations in few-shot 3D object detection.
Acknowledgments and Disclosure of Funding
This work has been supported by Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621). HKU Startup Fund, HKU Seed Fund for Basic Research, and Tencent Research Fund. Part of the described research work is conducted in the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust.
10

---

References
[1] Cao, Y. Wang, J., Jin, Y. Wu, T., Chen, K., Liu, Z., Lin, D.: Few-shot object detection via association and discrimination. In: Advances in Neural Information Processing Systems (2021)
[2] Chen, Lei, B., Song, Q.. Ying, H., Chen, D.Z., Wu, J:: A hierarchical graph network for 3d object detection on point clouds. In: IEEE Conference on Computer Vision and Pattern Recognition (2020)
[3] Chen, W.Y., Liu, Y.C., Kira, Z., Wang, Y.C.F., Jin-Bin, H.: A closer look at few-shot classifica- tion. In: International Conference on Learning Representations (2019)
[4] Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-view 3d object detection network for au- tonomous driving. In: IEER Conference on Computer Vision and Pattern Recognition (2017)
[5] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: Scannet: Richly- annotated 3d reconstructions of indoor scenes, In: IEEE Conference on Computer Vision and Pattern Recognition (2017)
[6] Deng, J., Wei, D., Richard, S., Li-Jia, L., Kai, L., Li, F.F: Imagenet: A large-scale hierarchical image database. In: IEEE Conference Computer Vision and Pattern Recognition (2019)
[7] Dhillon, G.S., Chaudhari, P., Ravichandran, A., Soatto, S.: A baseline for few-shot image classification. arXiv preprint arXiv:1912.03083 (2019)
[8] Engelmann, F. Bokeloh, M. Fathi, A., Leibe, B., Niessner, M.: 3d-mpa: Multi-proposal aggregation for semantic instance segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (2020)
[9] Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International Conference on Machine Learning (2017)
[10] Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H.. Qiao, Y: Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544 (2021)
[11] He, K., Gkioxari, G.. Dollar, P., Girshick, R.: Mask r-cnn. In: IEEE International Conference on Computer Vision (2017)
[12] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE
Conference Computer Vision and Pattern Recognition (2016)
[13] Hou, J., Graham, B., Niessner, M.. Xie, S.: Exploring data-efficient 3d scene understanding with contrastive scene contexts. In: IEEE Conference on Computer Vision and Pattern Recognition (2021)
[14] Kang, B., Liu, Z., Wang, X., Yu, F., Feng, J., Darrell, T.: Few-shot object detection via feature re- re-weighting. In: IEEE International Conference Computer Vision (2019)
[15] Lang, A.H., Vora, S., Caesar, H., Zhou, L.,, Yang. J., Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)
[16] Lee, K., Maji, S., Ravichandran, A., Soatto, S.: Meta-learning with differentiable convex optimization. In: IEEE Conference Computer Vision and Pattern Recognition (2019)
[17] Liu, J., Song, L., Qin, Y.: Prototype rectification for few-shot learning. In: European Conference on Computer Vision (2022)
[18] Liu, Z.,, Zhang, Z., Cao, Y., Hu, H., Tong, X.: Group-free 3d object detection via transformers. In: IEEE International Conference on Computer Vision (2021)
[19] Lu, Z.,, He, S., Zhu, X., Zhang, L., Song, Y.Z., Xiang, T.: Simpler is better: Few-shot semantic segmentation with classifier weight transformer. In: IEEE International Conference on Computer Vision (2021)

---

[20] Ma, J., Xie, H., Han, G., Chang, S.F., Galstyan, A., Abd-Almageed W.: Partner-assiste learning for few-shot image classification. In: IEEE International Conference on Computer Vision (2021)
[21] Min, J., Kang, D., Cho, M.: Hypercorrelation squeeze for few-shot segmentation. In: IEEE International Conference on Computer Vision (2021)
[22] Misra, I., Girdhar, R., Joulin, A.: An end-to-end transformer model for 3d object detection. In: IEEE International Conference on Computer Vision (2021)
[23] Nguyen, K., Todorovic, S.: Feature weighting and boosting for few-shot segmentation. In: IEEE International Conference on Computer Vision (2019)
[24] Qi, C.R., Litany, O.,, He, K., Guibas, L.J.: Deep hough voting for 3d object detection in point clouds. In: IEEE International Conference on Computer Vision (2019)
[25] Qi, C.R., Yi, L., Su, H., Guibas, L.J.; Pointnet++:: Deep hierarchical feature learning point
sets in a metric space, In: Advances Neural Information Processing Systems (2017)
[26] Qiao, L., Zhao, Y., Li, Z., Qiu, X., Wu, J., Zhang, C.: Defren: Decoupled faster r-cnn for few-shot object detection. In: IEEE International Conference on Computer Vision (2021)
[27] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.000 (2021)
[28] Ren, S., He, K., Girshick, R., Sun, Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems (2015)
[29] Sharma, C., Kaul, M.: Self-supervised few-shot learning on point clouds. In: Advances in Neural Information Processing Systems (2020)
[30] Shi, S., Guo, C., Jiang, L., Wang. Z., Shi, J., Wang, X., Li, H.: Pv-renn: Point-voxel feature set abstraction for 3d object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (2020)
[31] Shi, S., Wang, X., Li, H.: Pointrenn: 3d object proposal generation and detection from point cloud. In: IEEE Conference Computer Vision and Pattern Recognition (2019)
[32] Snell, J., Swersky, K., Zemel, R.S.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems (2017)
[33] Song, H., Dong, L.,, Zhang, W.N., Liu, T., Wei, F: Clip models are few-shot learners: Empirical studies on vqa and visual entailment. In: Association for Computational Linguistics (2022)
[34] Song, S., Lichtenberg, S.P., Xiao, J.: Sun rgb-d: rgb-d scene understanding benchmark suite. In: IEEE Conference on Computer Vision and Pattern Recognition (2015)
[35] Song, S., Xiao, Deep sliding shapes for amodal 3d object detection in rgb-d images. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)
[36] Sun, B., Li, B., Cai, S., Yuan, Y., Zhang, C.: Fsce: Few-shot object detection via contrastive proposal encoding. In: IEEE Conference on Computer Vision and Pattern Recognition (2021)
[37] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I. Attention is all you need. In: Advances in Neural Information Processing Systems (2017)
[38] Vinyals, O.,, Blundell, C., Lillicrap, T., Kavukeuoglu, K., Wierstra, D.: Matching networks for one shot learning. In: Advances in Neural Information Processing Systems (2016)
[39] Wang, X., Huang, T.F.,, Darrell, T., Gonzalez, J.F., Yu, Frustratingly simple few-shot object detection. In: International Conference on Machine Learning (2020)
12

---

[40] Wang, Y.X., Ramanan, D., Hebert, M.: Meta-learning to detect rare objects. In: IEEE Interna- tional Conference on Computer Vision (2019)
[41] Wang, Y., Fathi, A., Kundu, A., Ross, D.A., Pantofaru, C., Funkhouser, T., Solomon, J.: Pillar- based object detection for autonomous driving, In: European Conference on Computer Vision (2020)
[42] Wu, A., Han, Y., Zhu, L., Yang. Y: Sniversal-prototyyp enhancing for few-shot object detection. In: IEEE International Conference Computer Vision (2021)
[43] Wu, A., Zhao, S., Deng, C., Liu, W.: Generalized and discriminative few-shot object detection
via svd-dictionary enhancement, In: Advances Neural Information Processing Systems (2021)
[44] Wu, J., Liu, S., Huang, D., Wang. Y.: Multi-scale positive sample refinement for few-shot object detection. In: European Conference Computer Vision (2020)
[45] Yan, X., Chen, Z., Xu, A., Wang, X., Liang, X., Lin, L.: Meta r-emn: Towards general solver for
instance-level low-shot learning. In: IEEE International Conference on Computer Vision (2019)
[46] Yan, Y., Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors (2018)
[47] Ye, C., Zhu, H., Liao, Y., Zhang, Y., Chen, T., Fan, J.: What makes for effective few-shot point cloud classification? IEEE Winter Conference on Applications of Computer Vision (2022)
[48] Zhang, B., Xiao, J., Qin, T.: Self-guided and cross-guided learning for few-shot segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (2021)
[49] Zhang, R., Fang, R., Zhang, W., Gao, P., Li, K., Dai, J., Qiao, Y., Li, H.: Tip-adapter: Training- free clip-adapter for better lisionel modeling. arXiv preprint arXiv:2111 11122111109999 (2021)
[50] Zhang, 7.,, Sun, B., Yang, H., Huang, Q.: H3dnet: 3d object detection using hybrid geometric primitives. In: European Conference Computer Vision (2020)
[51] Zhao, N., Chua, T.S., Lee, G.H.: Few-shot 3d point cloud semantic segmentation, In: IEEE Conference on Computer Vision and Pattern Recognition (2021)
[52] Zhou, Y., Tuzel, O.: Voxelnet: End-to-end learning for point cloud based 3d object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (2018)
[53] Zhu, K., Cao, Y., Zhai, W., Cheng, J., Zha, Z.J.: Self-promoted prototype refinement for few-shot classinncremenal learning. In: IEEE Conference on Computer Vision and Pattern Recognition (2021)
[54] Zhu, K., Yang, J., Li, H., Chen, C.I., Wang, X., Liu, Z.: Conditional prompt learning for vision-language models. In: IEEE Conference Computer Vision and Pattern Recognition (2022)
[55] Zhu, X., Zhu, J., H., Wu, X., Wang, X., Li, H., Xiaohua, W., Dai, J.: Uni-perceiver Pre- training unified architecture for generic perception for zero-shot and few-shot tasks. In: IEEE Conference on Computer Vision and Pattern Recognition (2022)
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes]. [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
13

---

Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] we have included limitation
discussion in the supplementary material Section A.7.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "q0i9Wignlw0", "reviewer": "Reviewer_M2Gz", "summary": "This work explores the problem of few-shot learning in 3D object detection. It appears to be the first work in the field, and contributes 2 dataset-settings, based on SUNRGB and ScanNet, as well as 4 benchmarks, mostly based on VoteNet. The authors contribute their own method, which extends VoteNet to incorporate prototypes in each of the feature representation “stages”. The prototypes are moving averages of the closest features. The method is simple, yet effective, showing strong performance gains compared to the other VoteNet baselines.", "questions": "Were other methods of updating the prototypes tried? Does setting the prototype at the end (no updates) perform well? (not necessarily a weakness, just a curious possible experiment)", "limitations": "The authors did not discuss any limitations.", "rating": 7, "confidence": 2, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work, to the best of my knowledge, is one of the first works to tackle the problem of few-shot learning in a 3D object detection setting. In addition to their own method, the authors also contribute a set of benchmarks and a proposed dataset setting to evaluate future few-shot learning methods. This is a good contribution to the community as a whole. The method proposed by the author appears to achieve strong performance compared to proposed benchmarks. The method appears to be informed, and empirical evidence backs up each module’s necessity. By initializing the prototypes used for the features (“geometric”) and for the classes with the feature-space centroids and updating with a moving average, it seems to prevent the few-shot learner from overfitting as much (table 1,2). Experimentally, the ablation section is detailed and I do not have any questions after reading it. Nice to see the prototypes didn’t collapse (fig 3).\n\nThe conclusion in section 3.2, L162 that the prototypes are learning basic 3D geometric shapes is a bit misleading, since distribution of features in any pre-trained detection model should be similar for similar objects as well (fig 3). Considering the prototypes are updated in the feature space, there is nothing to indicate that there is 3D geometric interpretation; one way to visualize if it truly is learning basic geometric shapes is to optimize a shape input to maximize one prototype. This would give a better “geometric” interpretation of what that prototype is representing. However, this is not necessarily a weakness, since feature-space prototypes are just as useful, but I would recommend either changing the claim, or backing up the claim with visualized representations of what each prototype corresponds to in 3D input space. \nOne weakness is the benchmarks proposed are all built on top of VoteNet. A benchmark that would be good to include is KNN assignment to the closest support class. This would be able to leverage other detectors.\n\nMiscellaneous:\nL121: “Taking a point cloud scene P_i as input… localize and categorize…” → “VoteNet takes a point cloud scene P_i as input and localizes and categorizes…”\nFig 4 caption: “Tsne” → “t-SNE”\nL225: “Till now” → “Until now”", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fTI8V3h_jf3", "reviewer": "Reviewer_Eu2W", "summary": "This paper proposes a new task, few-shot 3d point cloud object detection. It is a combination of two well-studied topics, few-shot learning and 3d point cloud detection. And naturally, two well-known methods from both sides, prototypical learning and VoteNet, are combined to tackle the new task.\n\nWhile the combination seems straight forward, the paper further shows that it is not trivial. The implementation of PVM and PHM disentangles the feature embedding and detection. And successfully ", "questions": "Questions are listed in weaknesses.", "limitations": "The limitations of the work has not been addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It is overall a good work. Strengths are already covered in summary. Here are some weaknesses.\n\n1. As a general problem when a few-shot detection benchmark is composed, do we have novel classes in training set but annotated as background? What is is ratio?\n\n2. Can you visualize some typical structures of geometric prototypes? For example, the neighborhood of the point with features close to each prototype?\n\n3. The abbreviation FSOD is usually used for few-shot object detection (2D). As a novel task for 3D, the name could be more specific, such as FS3D.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cexKDfbqTwv", "reviewer": "Reviewer_ymyp", "summary": "The paper presents a method for few-shot 3D point cloud object detection. The method extends VoteNet by incorporating two new modules, prototypical vote module (PVM) and prototypical head module (PHM). The experimental results show that the proposed prototypical VoteNet improves the performance for few-shot t 3D point cloud object detection.", "questions": "- How are the new components different from existing work, such as [37]?\n- How are the hyperparameters such as alpha_1, alpha_2, etc. set?\n", "limitations": "The limitations of the paper are not sufficiently discussed. Given the relatively low performance, the paper should include more discussions of limitations beyond just resolution difference. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper addresses a new problem, which is potentially useful.\n- The method is plausible and seems to work better than existing methods.\n\nWeaknesses:\n\n- The technical components are largely adapted from existing papers, so technical novelty is somewhat limited.\n- The few-shot setting is not sufficiently convincing. The categories chosen for few-shot learning are not really rare categories, so it is not sufficiently meaningful. The performance is also rather low for practical use.\n- The datasets are straightforward adaptations of well-known datasets, so they should be declared as a contribution. \n- The paper only compares the baseline VoteNet, not follow-up improvements which could give better performance. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TOEAOs1OYrC", "reviewer": "Reviewer_ZU9c", "summary": "This paper proposes a few-shot framework for 3D point cloud indoor object detection. They propose prototypical votenet to recognize\nand localize novel instances based on the PVM and PHM modules. PVM leverages class-agnostic geometric prototypes learned from base classes to refine local features of novel categories. PHM is designed to utilize class prototypes to enhance the global feature of each object. This paper also provides two new benchmark datasets, FS-ScanNet and FS-SUNRGBD. They conduct extensive experiments to demonstrate the effectiveness of their method, which shows a promising performance compared to several self-designed baselines on two benchmark datasets.", "questions": "All the questions are mentioned in Weakness.\n\n-I want to see more baseline approaches widely used in 2D few-shot object detection.\n\n-I want to see the improvement of the whole class by the proposed model. (Compared with VoteNet).\n\nIf the authors solve my concerns, I will consider raising my score.", "limitations": "No", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-This paper is well-written and easy to follow.\n\n-The idea is interesting. This paper considers utilizing geometric learning and object prototypes to assist few-shot 3D object detection, which will help our community. \n\n-The results of their experiment are promising with a remarkable margin.\n\nWeakness:\n\n-The baselines of this paper are not well-designed. Few-shot object detection in 2D community is well-studied. Why didn't this paper compare some classical few-shot approaches in 2D community? I don't think it's difficult to transform some 2D techniques into the 3D domain in this task. It's difficult to judge their contribution based on these weak baselines. \n\n-If you statistics them, both of these datasets i.e. ScanNet V2 and SUN RGB-D are unbalanced. In ScanNetV2, the number of chairs is 4357 and 113 of bathtubs. If this paper focuses on solving the imbalanced problem of different classes by considering the prototype learning, it would contribute more to the 3D community. So I want to see the improvement of fully training based on the votenet and proposed PVM&PHM. That means the base class is the whole class (18 classes) and no novel class.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept