Prototypical Variational Autoencoder for 3D Few-shot Object Detection
Authors: Weiliang Tang
Biqi YANG
Xianzhi Li
Yun-Hui Liu
Pheng-Ann Heng
Chi-Wing Fu
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D Point Cloud Object Detection
Few Shot Learning
Computer Vision
Geometric Prototype

Abstract: Few-Shot 3D Point Cloud Object Detection (FS3D) is a challenging task, aiming to detect 3D objects of novel classes using only limited annotated samples for training. Considering that the detection performance highly relies on the quality of the latent features, we design a VAE-based prototype learning scheme, named prototypical VAE (P-VAE), to learn a probabilistic latent space for enhancing the diversity and distinctiveness of the sampled features. The network encodes a multi-center GMM-like posterior, in which each distribution centers at a prototype. For regularization, P-VAE incorporates a reconstruction task to preserve geometric information. To adopt P-VAE for the detection framework, we formulate Geometric-informative Prototypical VAE (GP-VAE) to handle varying geometric components and Class-specific Prototypical VAE (CP-VAE) to handle varying object categories. In the first stage, we harness GP-VAE to aid feature extraction from the input scene. In the second stage, we cluster the geometric-informative features into per-instance features and use CP-VAE to refine each instance feature with category-level guidance. Experimental results show the top performance of our approach over the state of the arts on two FS3D benchmarks. Quantitative ablations and qualitative prototype analysis further demonstrate that our probabilistic modeling can significantly boost prototype learning for FS3D.

Prototypical Variational Autoencoder for Few-shot 3D Point Cloud Object Detection
Weiliang Tang"
Biqi Yang'
The Chinese University of Hong Kong wltang21@cse. cuhk. edu. hk
The Chinese University of Hong Kong bqyang@cse. cuhk. edu.
Xianzhi Li
Pheng-Ann Heng The Chinese University of Hong Kong pheng@cse. cuhk. edu. hk
Huazhong University of Science and Technology xzli@hust.. wwwhhut.edu....... cn
Yunhui Liu
Chi-Wing Fu
The Chinese University of Hong Kong yhliu@mae. cuhk. edu hk
Department of CSE and SHIAE The Chinese University of Hong Kong cufuêcse. cuhk. edu. hk
Abstract
Few-Shot 3D Point Cloud Object Detection (FS3D) is challenging task, aiming to detect 3D objects of novel classes using only limited annotated samples for training. Considering that the detection performance highly relies on the quality of the latent features, design VAE-based prototype learning scheme, named prototypical VAE (P-VAE), to learn a probabilistic latent space for enhancing the diversity and distinctiveness of the sampled features. The network encodes multi-center GMM- like posterior, which each distribution centers at a prototype. For regularization, P-VAE incorporates reconstruction task to preserve geometric information. To adopt P-VAE for the 3D object detection framework, we formulate Geometric- informative Prototypical VAE (GP-VA VAB) to handle varying geometric components and Class-specific Prototypical VAE In the first stage, we harness GP-VAE aid feature extraction from the input scene. In the second stage, we cluster the geometric-informatie per-instance features and use CP-VAE to refine each instance feature with category- level guidance. Experimental results show the top performance of our approach over the state of the arts on two FS3D benchmarks. Quantitative ablations and qualitative prototype analysis further demonstrate that our probabilistic modeling can significantly boost prototype learning for FS3D.
to handle varying object categories.
features into
Introduction
3D point cloud object detection is fundamental scene understanding task. aiming at recognizing and localizing objects in point-based scenes. Recent deep-learning methods are mostly fully supervised, requiring extensive labeled data support the model training. Such data-hungry setting is, however,
Equal contributions to the works. This work was supported by Shenzhen Portion Shenzhen- -oog Kong Science and Technology Innovation Cooperation Zone under HŽQB-KCZYB KC22B-2020089, This work was supported by the Hong Kong Centre for Logistics Robotics, the Research Grants Council-Genede Fund (No. 142016200 620). This work was supported by InnoHK of the Govermment of Hong Kong via the Hong Kong Centre for Logistics Robotics. This work supported by project MMT-p2-21 of the Shun Hing Institute Advanced Engineering (SHIAE).
Research
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Reconstruction
proo
Encoder
prog
Decoder
Input
Output
Mpro
GMM Distribution Space
Geomettt-infermaaie Prototypes
Point Cloud Soene
Point Cloud Objects
spectfs Protolypes
Diverse Sampled Features
[a]
(b)
(c)
Figure 1: (a) The basic structure of our proposed Prototypical VAE, where we learn GMM-like distribution for feature sampling that can be regularized by 3D reconstruction tasks. (b) At the scene level, each point cloud scene can be reconstructed by composing local geometric components, we call them be represented in a unified fashion by class prototype, named the class-specific prototype.
prototypes. (c) At the object level, objects of the same category can
impractical in real applications, considering the heavy and costly loads to densely annotate the data. Inspired by humans' capability of recognizing novel objects by learning from limited data, Few-shot Learning (FSL) I 2 3
has been proposed to enable neural networks to handle novel data by training on large labeled data of base classes and small labeled data of novel classes.
Prototype learning is an effective approach to achieve FSL [2] 6. 7 8 9 10 III
It has been applied [4, semantic segmentation [15] 19 20 and instance segmentation 221 22 In prototype learning, some principles first defined to separate the prototypes, e.g., by category 23
successfully to various 2D vision tasks, e.g., object detection [12] 13
[6 17
18
by partial These prototypes can be viewed as clustering centers, such that we can group the latent features based on our pre-defined principles. Importantly, the prototypes provide transferable knowledge that allows us to generalize a model for handling novel classes, thereby facilitating representation learning for downstream tasks.
similarity [I] or by intrinsic characteristics [12 15 17 18 20
In this work, exploit prototype learning for Few-Shot 3D Point Cloud Object Detection, named FS3D. We aim to predict accurate 3D bounding boxes on objects of novel classes, given only a small amount of labeled novel-class exemplars. To address FS3D with prototype learning, Zhao et al. [IO] propose a pioneering work. Yet, their learning scheme lacks fine-level supervision, as the intermediate features simply averaged to update the prototypes, which are then used augment features for sequential detection. There are two fundamental issues. First, without strong regulation on prototype learning, the prototypes can lose substantial 3D information and become less geometric-infone innommtri-iffoormativee.. Second, due to the data imbalance, prototypes of novel classes are clearly underrepresented than those of the base classes, so the latent space can easily overfit the base ones.
To address the above issues, we propose a variational autoencoder approach particularly designed for prototype learning, named Prototypical VAE (abbr. VAE). As Fig. (T) illustrates, instead of directly learning the features, our P-VAE learns the distribution parameters, by which we can construct posterior for sampling features from the probabilistic latent space. For issue 1, we propose perform reconstruction tasks with an encoder-decoder architecture in our P-VAE framework to better preserve the geometric information in the intermediate features. For issue 2, our P-VAE allows to sample diverse and varied features controlled by a multi-center distribution. Hence, even with limited novel samples, our approach can still gather richer information to generate good-quality representative prototypes for the novel classes.
Gaussian Mixture Model
To enhance the object detection task with the proposed P- PAV, we formulate two crucial extensions: eeometric-informativv Prototypical VAE (GP-VAE) and Class-specific Prototypical VAE (CP-VAE). As Fig. shows, we can view real-world 3D objects from two aspects. In a broader sense, all the objects, regardless of their class labels, can be reasoned by a of local geometric components; see Fig. bb). For GP-VAE, each component is geometric-informativ prototype, such that given point cloud scene can be constructed by combining these prototypes. Narrowing the scope to each object, objects of the same category have similar shapes, while objects of different categories look more different; see Fig. (... Hence, we propose CP-VAE to obtain class-specific prototypes, such that each prototype can serve as strong shape prior to guiding the object-level feature learning. We incorporate the two modules in our object detection pipeline, leveraging GP-VAE to account for scenc-level feature extraction and CP-VAE for object-level feature refinement; see Fig. 2

---

We conduct extensive experiments on two FS3D benchmarks, FS-ScanNet and FS-SUNRGB [[]] The results in Sec. 4.2 show that our method significantly outperforms the SOTA approaches in various few-shot settings. More analysis on prototypes in Sec. [.33 demonstrate the effectiveness of leveraging VAE for prototypical learning. To summarize, our contributions are listed as follows:
We propose the Prototypical VAE (P-VAE) to learn multiple distributions centering at prototypes, such that it can construct a probabilistic latent space that allows us to effectively sample diverse features with higher controllability and flexibility.
To harness V-VAE for object detection, we extend it for ceometrie-informattio and class- specific prototypical VAEs (i.e., GP-VAE. & CP-VAE) based on our observations.
Results demonstrate the superiority of our new approach for FS3D and further studies analyze the prototypes and explain the effectiveness in detail; see Sec. 4 annd Appendix.
2
Related Works
Variational Autoencoder (VAE) is first introduced by Kingma et al. 24 to learn a generative model for constructing a distribution based latent space. It provides a probabilistic representation learning scheme with theoretical support. Bornschein et al. [25] assume that features follow a multi- center prior distribution, which encourages unsupervised deep clustering in the embedding space. Norouzi et al. (66) select exemplars from the training data and use them to model the embedding space with GMM. To strengthen the robustness,
(2
28 29) balance between reconstruction authenticity
and generation diversity;
30 31 32
aims at preventing VAE models from collapse; [3 34 35
explore usages of different regularizations. VAE widely adopted in 3D generation tasks, c.g., 38) and scene reconstruction [39] 40]. However, it has not been fully explored for 3D perception tasks, especially for the challenging FS3D task.
point cloud generation [36 37
Few-shot 2D Image Object Detection (FS2D) methods can be classified into two categories. The first family aims to address the data imbalance problem, for example, by fine-tuning with balanced data [4T,, augmenting objects by different scales 42 augmenting novel samples with generative networks [43], learning reasonable and unbiased latent space (44). etc. The second family focuses on learning support (base)-novel (query) correlations. Kang et al. [45] reweight the query features given the support features; Han et al. [46] fuse the support and query features by spatial alignment; Fan et al. [47 and Yan al. [48 exploit deep support information to guide the region proposal network and the detection heads; some other works incorporate support information learned from the backbone
to guide the feature extraction of the query images, using transformers [49 50 51 52 53 or graph convolution networks 44 An intuitive solution is to adopt these FS2D strategies to tackle FS3D, as some our comparison methods in Sec. 4.2 However, their detection results (see Tab. and are far worse than our P-V The reason is that 3D point clouds are intrinsically more disordered and complex than pixel-grid 2D images, thus naturally lacking abundant data with full annotations.
FSL. for 3D Point Cloud under-explored, as we discussed before. As pioneer work, Fan et al. [54] tackle the one-shot point cloud classification, exploring base-novel sample relations with graph convolution. Cheraghian et al. (55) introduce a transductive zero-shot learning method with a triplet loss to obtain a better classifier. Zhao al. [I]] incorporate FSL for 3D semantic segmentation with a transformer for multi-prototype transductive inference. To our best knowledge, Prototypical VoteNet [10] is the first and only work on FS3D that harnesses prototype learning. However, their experimental results, refer to the indoor scenes in Fig. 3 are not satisfactory. First, it lacks a strong regulation for prototype learning, like our reconstruction task, to preserve latent geometric information. Second, the quality of their novel-class prototypes is impaired due to data imbalance, whereas we propose a new VAE-based approach to sample features with significant diversity.
3 Method
In this section, we first define the FS3D task and give a modular overview of our pipeline in Sec. 3.1 Then, we present the formulation of our novel Prototypical VAE (P- (P-VAE) to learn a probabilistic- distribution-basser feature space in Sec. 3.2 Building upon the P-VAE, we develop (i) Geometric- informative Prototypical VAE VAE) (Sec 3.3) and (ii) Class-specific Prototypical VAE (CP-VAB) (SecB3.4), which are customized for learning prototypes of varying geometric components and varying object categories, respectively.
3

---

Samp
Scene-level Reconstruction
Centerr ProorpO0
geily
Xg
Scene Encoder
Scene
Decoder
geo
inpurpoint doud
Ny-denter Poster Q(T))
Ruttedtructed point cleud
Calibration
Object-level Reconstructin
Mae p(W)
elsy
My
approximation
elle
M4
Object Encoder
Object Decoder
GP-UAE
Reconstrusted 5D objects
((N))))
Calibration
Cluster N. features Mase
[W],
results
Figure 2: Illustration of pipeline. We incorporate crucial tasks facilitate the framework to learn representative prototypes: a scene-level reconstruction task with GP- SAP and an object-level reconstruction task with CP-VAE.
3.1 Problem Definition and Framework Overview
In the FS3D task, the full class set C is split into base set Chase and novel set Cnovel- where Cbase Cnovel = 0. Each base class in Chose has plenty of annotated objects, while the amount of labeled samples is very limited for each novel class in Cnoeet The FS3D annotation for an instance of class-i (pos, dim,,c) G,), denoting the box center point, 3D box dimensions, and the object class label. Following the FSL definition, the K-shot setting means only K annotated objects can be used for training in each novel class. Our goal
enable model pre-trained on aannntttttyyannotateed Cbase samples to generalize over new classes using only K labeled objects per class in Cnovel.
Fig. 2 shows the overall pipeline. Built upon the detection framework, we perform two reconstruction tasks during the training to regulate the prototype learning and model the probabilistic latent space:
(i) Scene-level reconstruction, in which we incorporate our proposed GP- VAE to learn geometric prototypes {geo} with variational encoder-decoder architecture. From the learnt distribution, sample features {z} } for decoding and further refine {z} to {z'} with the geometric prototypes for down-stream object detection (see Sec. [33); and
(ii) Object-level reconstruction, in which we adopt a voting-based object encoder to cluster the scene-level features {='} into object-level features and incorporate the proposed CP-VAR to learn class-specific prototypes {cls} as distribution centers. Similar to (i), we sample features {w) for decoding and output calibrated features (w²) (see Sec. 3.4).
In the end, we use multiple feed-forward network heads, which are fed with {w'), to generate instance- 3D box positions and dimensions as the detection results. During the inference, we can skip the reconstruction decoders and adopt the pre-trained prototypes for forward data flow.
3.2 Prototypical Variational Autoencoder
Our proposed Prototypical VAE (P-VAF) is built upon standard VAE as its backbone to learn high-level prototypes. Referring to Fig. [[),, the overall architecture consists of an encoder and decoder. X is the variable that represents our data and the encoded latent space follows certain prior distribution p(Z). The encoder f parameterized by 0y takes the point cloud æ E N.,x3 x3 as the input and generates parameters 4 E xd and RN.xd, where N₂ the number of parameters {ui, J; and dl is the latent space dimension. We adopt them to obtain a set of predicted posteriors qe, XX) ~ N(,,,, o.),, where = 1...N..
Based these N. normal distributions, can sample N₂ new features {2//2 adopt decoder which is parameterized as fg. taking each 2; to reconstruct a point cloud with Np points p: E RNxx3, Finally, we combine these reconstruction results to form our point cloud prediction p. Given the grouhd-truth point cloud the optimization goal for P- P-VAE is
Rd Then,
Ri
arg min (CD(p, 0,
KL(ge, (p())))(
(1)
N

---

where CD denotes the Chamfer Distance that supervises the reconstruction task and KL denotes the Hullback-Leible Divergence to regularize the posteriors 90, X,, = I...N; latent space prior p(Z). By far, we can learn this distribution space standard VAE term.
stay close the
Importantly, we focus on obtaining prototypes based on latent distributions. Denoting (pro,lpro, Rr) Naro as Npro prototypes, our goal to use them as clustering centers to group the latent embeddings. Hence, we define pl p(Z) following an approximate GMM, which is composed of N Npro independent normal distributions that each centers at pro, and has variance I:
Nara exp( 1 2112 pro,
-2=
1 arg min || pro,ll2).
A(2x) )d/2
Compared with the standard GMM, our approximate GMM is designed with two strategies. First, we adopt a fixed weight 1 (g pZ = z)d2 = 1) instead of distinctive learned weights for these normal distributions, such that each prototype can be equally accessed. This design is for imitating real-world situation, where all the geometric components share the same probability to be observed
among common geometric structures. Second, adopt a hard regularization in Eq. (2) such that each sampled feature 2; can be directly assigned to its nearest prototype. In this way, our intermediate features are more organized, since they discriminatively grouped by the prototypes. By introducing pro, we modify the posterior distribution fa, (Z; = x| = Aexp(- -21l= 1= f [[ll==ro,,l 112 S
|| proxl12 where A is the normalization term and pro, is the closest prototype to μ4,
With Eq. 2 we can optimize the KL term of Eq. (T) and rewrite Eq. ①1
as
Npro
pro
arg min f) 0
111.:12 112
log of + arg min| pro
Σ Py
I| - M -1 1[2)1:
Term Pro
where B, = {ux |llax pro,lle I l - proyIl2 .j = I...NN indicating the set of features that are assigned prototype pro; by least Euclidean distance. Please Appendix for detailed derivations.
To find the optimal prototypes J is to solve the 'Term Pro' in Eq. (3), which can be viewed i=1 as a clustering task to seek Npro clustering centers, such that the mean distance between all pairs of the sampled features to their closest prototypes is minimized. We adopt the K-means algorithm updated with Exponential Moving Average (EMA) to enhance the training efficiency. Overall, at the tt training iteration, pro; is estimated as
Σ
Pk
pro,* + apro + (1 a) 2 ELEB
B, = (mullle pro,lla S llpx pro,ll,, Vj= = 1,.... Nproh:
B
(4)
where a is the smoothness hyperparameter and all the prototypes are initialized with the Gaussian random initialization before the training. After we obtain the new prototypes in an iteration, we take them back to Eq. (3) to get our loss for training the network:
Nara
Naro
L CD(
Σ
log of]h]]+
Σ
Ipro,
13112),
M
where the first term is the reconstruction loss and the last term is the clustering loss for encouraging each sampled feature stay close its associated nearest prototype.
3.3 Scene Reconstruction with Geometric Prototypical VAE (GP-VAE)
The detection framework starts by extracting features from the input scene. Here, we incorporate GP-VAE to enhance the feature quality. We reconstruct the whole point cloud scene with GP- VAE, and obtain geometric-informatii prototypes denoted (geo, Ngeo These prototypes represent
basic geometric components of common real-world 3D objects.
Please refer Fig. 2 for GP-VAE We use the PointNet++ (56) as our encoder facene. takes raw point cloud RN.x3 x3 as input and outputs parameters {uu, N. construct the posterior normal distributions A (ju, aF)), = I...Ng We sample (z, N_ from the learned distributions and build decoder Jscene to reconstruct the original point cloud scene. For Jacene, we adopt the architecture
5

---

of NotdingNet [57]. which transforms plane into a 3D object surface. Specifically, we use furthest point sampling (FPS) to obtain V points on a 0.4m x 0.4m plane B RN,x2, For each 2;, we repeat the feature Np times then concatenate them with plane B, followed by two 2-layer MLPs to
generate Pt E RN,x3, We use Eq. 4 to obtain the geometric prototypes {geo: Ngeo
H We combine the Nv predicted point clouds construct the scene-level result p N,N,N, then use Eq. (5) to supervise the scene-level reconstruction, where the ground truth is the input point cloud x.
Instead of directly using the sampled features {=.) to cluster the per-object features, we calibrate {=:) with the geometric prototypes via cross attention layer. This calibration enriches the information diversity of each feature by not only its nearest prototype but also other geometric prototypes. For brevity, use geo E IR RR- geo as the united representation of the prototypes:
Softmax (z2,) K (geo)
V(geo),
(6)
Vd
where Q, K, and V are three MLPs to generate the query, key, and value in the attention framework. In this way, we can obtain robust embedding features (=
3.4 Object Reconstruction with Class-specific Prototypical VAE
CP-VAE to reconstruct each object and obtain class-specific prototypes as shown in F C)). We build voting-based [58] encoder foöject- For detection purpose, we need per-object features { J==1 where Nins is the number of the predicted instances. Therefore, we use I-layer MLP to predict the feature offsets and point offsets from {= then adopt FPS on the shifted points to obtain Nine votes. For each vote, we collect maximum number of 16 shifted features within 0.3m from the vote, then use max pooling these features to generate the associated object-level feature &, Another built MLP is then fed with Nu ( to output parameters {/4 ) which can be used to construct the posterior truncated normal distributions N (14,,0 o3) where = 1...N We sample [w:
Given the features {=) generated in Sec. 3.3
Ni and build a FoldingNet decoder Jobject (Sec. 3.3 for object-level reconstruction. For each wi Pobject outputs an unique 3D object pi.
According to the sampling and grouping mechanisms in feature extraction, we reversely map each w/ to set of corresponding points in x (we will detail this operation in the Appendix). which constructs our reconstruction target p.. Therefore, we modify the reconstruction term of Eq. (5)t to
Nen
Nete
Nei
C =
Neno Σ
CD(p..P:) +
1
Σ 11o, - log of
[1]+ +
|clss-M,l333 (7)
2Noi
MyE
where B; = {peelll 19 cls;ll2 122 < |i -... Importantly, we further modify the prototype update formula in Eq. (4) to customize for class spass-specificcccc prototype learning. As discussed before, we can reversely map each μa to a set of input points. During the training, we use this point set to find the nearest ground-truth instance by the least center Euclidean distance, then assign the corresponding category label to μı: We collect features of the same category to update the class- class-specific prototypes accordingly. Denoting ((-) the operation to get the category index for each feature, to obtain precise and convincing class class-specine specific prototypes, Eq. (4) is rewritten as
Σ PJ
cls A+1 acls! + (1 (1- a) -/)
RYEB.
B. = (H;(((()))) i}.
(8)
B.
Similar to Sec. 3.3
we refine the sample features (n attention augmentation, where we adopt cls E RRNax xd as the united representation of the prototypes:
Nime with the prototypes by using a cross-
w/ Softmax( Q(w)) (cls) (cls).
Vd
Experiment
4.1 Experiment Setup
We conduct experiments on two benchmarks FS-ScanNet and FS-SUNRGBD (10. The FS-ScanNet dataset consists of 18 object categories and 1.513 point cloud scenes in total. This benchmark has two

---

Prototypical Votenst
Ours
Figure 3: Example detection results by method and Prototypical VoteNet [00 (SOTA) FS- ScanNet split-1 I with K = 5. The red bounding boxes are the ground truths, while the green bounding boxes the predictions.
different base/novel splits, each including 6 novel classes and 12 base classes. The FS-SUNRGED dataset contains 5,000 point cloud scenes, covering 10 object categories. It has only one base/novel split, including novel classes and base classes. Following the standard 3D object detection protocol, comparison results on both datasets are quantitatively evaluated mean Average Precision (mAP) under IoU thresholds 0.25 and 0.5, denoted as AP25 and 50 More details on the splits of both datasets, the network architecture, and the training scheme can be found in the Appendix.
split-1 3-shot
split-2
Method
T-shot
5-shot
I-shot
3-shot
5-shot
AP25
AP50
AP25
AP
AP25
AP50
AP25
APs0
AP25
AP50
2P
APs0
Beseline 58
11.72
8.02
21.13
9.57
28.63
15.69
8.79
1.71
18.19
5.52
22.68
11.64
Generalized
FS3D
59
12.03
8.19
24.90
10.26
29.29
16.67
9.19
1.87
19.41
6.80
25.18
12.74
CanCContttttVoteNeee 60
12.59
8.52
20.12
11.16
25.83
15.49
9.55
1.97
18.44
5.23
20.06
10.19
Fractal-VoteNN 61]
11.81
7.57
21.38
10.11
24.66
14.73
9.16
1.68
15.65
4.88
20.35
10.26
Meta-Det3D 62
10.28
4.03
23.42
10.64
25.65
13.88
5,21
1.32
15.44
4.37
22.13
7.09
Prototypical VoteNet 10
15.34
8.25
31.25
16.01
32.25
19.52
11.01
2.21
21.14
8.39
28.52
12.35
Ours
16.00
10.22
31.60
19.37
32.84
22.39
12.66
4.15
21.27
10.09
31,70
14.43
Table 1: Comparing the FS3D performance of our method against existing works on FS-ScanNet.
4.2 Comparison Results
Comparison Approaches. We quantitatively compare our method with VoteNet [88 baseline and five recent methods that utilize different FSL techniques to tackle FS3D. For the baseline, we train the VoteNet detector and then directly infer its performance Catteonotttttootteeeet
the novel test data without FSL. The other five recent methods are (i) Generalized FS3D [59]. an incremental fine-tuning method with a
sample adaptive balance loss built upon TFA 41); (ii) PointContrast- VoteNet [60], which obtains
generic and transferable features by self-supervised pre-training: (ii)) Practal-VoteNe
[6]], which adopts a pre-training similar to (ii). whereas it learns features from fractal geometry 3D patterns (this concept can be viewed as simplified geometric prototype without any regulation principle); (iv) Meta-Det3D 62), which uses meta detector [63 to output class- seeess-c re- reiehgtt vectors to refine Rol features (this is similar to our CP-VAE but their class vectors are negatively affected by data imbalance); (v) Prototypical VoteNet 10 the most recent SOTA approach that groups the latent features to obtain prototypes.
s-shot
2-shot
3-shot
4-shot
5-shot
Method
AP2s
APse
AP2s
APso
AP25
AP50
AP25
AP1o
AP25
AP50
Baseline 58
5.46
0.22
6.52
0.77
13.73
2.20
20.47
4.50
22,99
5.90
Generalized FS3D [59
6.81
1.58
12.21
2.02
17.52
4.69
22.12
5.97
22.84
6.76
PointContrast- VoteNet [60
7.03
1.17
8.16
2.33
20.32
4.19
21.13
4.49
21.03
6.71
Fractal-VotcNer VoteNet [6]]
7.54
1.39
9.16
3.01
21.08
4.25
21.23
5.68
22.01
6.77
Meta-Det3D [62]
6.77
0.73
8.29
1.21
15.37
2.99
19.60
4.67
24.22
5.68
Prototypical VoteNet 10
12.39
1.52
14.54
3.05
21.51
6.13
24.78
7.17
29.95
8.16
Ours
14.36
2.42
22,28
4.30
27.70
8.73
31,55
13,84
33,21
13,98
Table 2: Comparing the FS3D performance of our method against existing works on FS-SUNRGBD.
FS-ScanNet Benchmark. Tab. [ summarizes our quantitative comparison results on FS-ScanNet.
Despite their achievements in FS2D, the improvements of fine-tuning and pre-training (59 60 61 over the baseline are marginal for FS3D. They rely on a robust feature space pre-trained with large- scale datasets (ImageNet (64) for 2D), which are hard to obtain and annotate for the 3D scenes.
7

---

As we discussed before, [6] and 62 use the concept of prototypes but their learning schemes are less in-depth. Our proposed P- PAE consistently outperforms all the above methods, thanks to the proposed distribution-base based prototype learning scheme adopted in GP-VAE and CPE. Fig. 3 shows convincing qualitative results compared with SOTA [10], our predictions (green) better overlap with the ground truths (red), while 10 may produce more false positives and box errors.
FS-SUNRGED Benchmark. Tab. 2 summarizes our quantitative comparison results on FS- SUNGRBD. Compared with FS-ScanNee, FS-SUNGRBD is more challenging with higher clutter levels and more severe object diversity. Under such cases, our method still consistently surpasses SOTA [0] in all the few-shot settings, especially on the more strict AP50 5o metric.
Comparison on Base Classes. We fur- ther test only on base classes to see if our prototype learning scheme for novel samples may impair the features learned on the base samples. Please see Tab. 3 specifically de- signed for base classes, Compared with it, Prototypical VoteNet (Proto) suffers from consistent precision drops, which means the extracted knowledge from the novel samples is less incompatible with one of the base classes, thus the base feature space is hurt by those chaotic signals. Our VAE is able to better preserve the base feature structures and improves the origin detection performance on all the benchmarks. This observation demonstrates that our novel and base features are better aligned and can be simultaneously enhanced by our distribution-batt based prototype learning.
I'S-ScanNet-1
Ve--ScaNNe-2
FS-SUNRGED
Method
APas
AP50
APa5
AP5a
AP25
AP5o
Baseline 558
57.96
32.60
54.63
35.76
47.77
26.78
Proto 10
53.83
28.20
51.22
32.11
46.07
25,26
Ours
58.16
32.69
56.05
38.77
47.92
27.75
VoteNet (Baseline)
Table 3: Comparing base-class results on three datasets.
4.3 Ablation Studies and Detailed Analysis
Method
3-shot
5-shot
Module
PL scheme
AP25
3-shot
APaa
AP25
5-shot
APan
AP25
AP50
AP25
APs0
Baseline
21.13
9.57
28.63
15.69
GP-VAE
proto
AB
29.57
25.71
13.57
14.48
33.14
29.04
17.17
19.16
GP-VAE
29.69
15.18
33.48
22.18
VAE
29.69
15.18
33,48
22.18
CC--VE
27.11
18.88
31.92
21.91
CP-VAE
proto
26.14
17.43
31.13
19.16
AE
26.56
17.98
31.61
21.45
GP-VAE & CP-VAE
31.60
19.37
32.84
22.39
VAE
27.11
18.88
31.92
21.91
Table 4: Ablating our major components.
Table 5:
Analyzing different PL schemes.
Module Ablation. We ablate the major components of our framework on FS-ScanNet split-1 with K 3,5 5. From Tab. E
that each component contributes to improving the performance and our full pipeline attains the highest ratings for all settings. We can observe that individual usage of CP-VAE works less effectively than individual usage of GP-VAE The reason is that incorporate GP-VAE in the scene-level reconstruction, which is prior to the object-level reconstruction by using CP-VAE. If we omit GP-VAF, the features from the scene encoder are not sufficiently representative to support the sequential clustering, thus it could hurt prototype learning in CP-VAE.
Prototype Learning Schemes. Based on the Prototypical VoteNet 10), denoted as Proto in Tab. 5 optimize the prototype learning scheme to AE and VAE. AE introduces reconstruction task with an encoder-decoder architecture, while the latent features to optimize prototypes are directly Iearned from the encoder instead of sampled from the distributions. AE can be viewed as poborbbbaii prototypical learning. VAE is built upon AE by further constructing a probabilistic latent space to enable us to sample diverse features. can be viewed as probabilistic prototypical learning. As Tab reports, we conduct independent quantitative comparisons for GP-VAE and CP-VAE on FS-ScanNet split-1 with K- 3.5. The results of AE over Proto show that incorporating a reconstruction task helps preserve the geometric information within the latent embeddings. VAE further consistently outperforms AE on all the metrics for both GP-VAE and CP-VAE demonstrating that the probabilistic modeling can significantly boost feature learning for FS3D.
Prototype Quality. Thanks to the reconstruction tasks, we can visualize our prototypes as point clouds. Fig. Fleft) shows some the geometric-informatii prototypes learned by the GP-V more results can be found in the Appendix. From them we can see the diversity and distinctiveness of the prototypes, each exhibiting a certain geometric pattern. In Fig. rigght), we illustrate our class-specific prototypes learned by the VA-VAE. These results show certain class-level abstractions of the general object appearance.
8

---

*
Sofa
Chair
Sink
Table
Picture
Cabinet
Bed
Refrigerator
Figure 4: Left: reconstructed examples of learned geometric- informative prototypes, 49 points for each. Right: reconstructed examples of learned class-specific prototypes, 256 points for each.
Base Chair
Base Table & Desk
Novel Sink
Novel Bed
Protalype- Compased Scenes
Figure 5: Illustration gcometric-informatiee prototypes compositing objects and scenes. Each color represents unique prototype learned from VAE. These prototypes show strong inter-class physics consistency and intra-class semantic correspondences.
Prototype Effeett. After generating the prototypes with a well-designed prototype learning scheme, our final question is: do these prototypes help representation learning?
For geometric-ing -nnnttrrciiioommatieee prototypes, illustrate how these pro- totypes help to construct an object or a whole scene in Fig. where coametric-infommatiin prototype. We
Num. of Geo-Proto
AP2s
AP 6o
each color represents observe that each object, regardless of its base/novel category label, composed of multiple prototypes with semantic correspondence patterns. For example, the red prototype indicates an open semicir- cle shape, appearing not only the base chair class (consistently at the seat region) but also in the novel sink class (consistently at the center region). Results demonstrate that these prototypes are robust geometric patterns that help enhance the generalization ability for both base and novel classes, so they can provide transferable and consistent 3D hints for scene-level feature encoding. Please see Tab. for quantitative and Appendix for qualitative ablations on varying numbers of prototypes, denoting Ngeo in Sec. 3.3
Ngco 16
22.19
6.48
Ngen
28.59
14.52
Naco 64
31.78
18.51
Noco 128
32.84
22,39
No
32.06
20.30
Table 6: Ablation for N geo-
For class-specific prototypes, in Fig. 6 we use t-SNE visualization to show the high-level object embeddings {w; ) of five common categories. From left right: without class-specific prototypes, CP-AE (defined in Tab. 55. and CP- CP-VAE. Compared with the vanilla feature space, where the features are disordered and chaotic, incorporating class prototypes leads to a more compact latent space, such that each prototype can be more separated and the sampled features are better organized by clustering.
Pluggable P-VAE, Besides the VoteNet baseline, we extend multiple point cloud detection frameworks with our P-VAE. Please refer to Appendix for quantitative results that demonstrate the effectiveness of our VAE as a plug-in module for var- ious networks to better adapt FS3D.
Figure 6: Visualize object-level embeddings with t-SNE.

---

Conclusions
To tackle few-shot 3D point cloud object detection (FS3D), we design a specific VAE for prototype learning, named Prototypical VAE (P-VAB). P-VAE constructs a probabilistic latent space that allows us to sample diverse and distinctive features, such that we can learn more representative prototypes even if the novel samples are limited. To adopt P-VAE for FS3D, we propose the Geometric- informative VAE and the Class-specifie CP- VAE based on our common observations in the real world. Experimental results demonstrate the effectiveness of PAE, and further prototype analysis shows its potential value in other prototype-biie based few-shot 3D perception tasks.
10

---

References
[1] Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1135, 2017.
[2] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 30, 2017.
[3] Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philiip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In IEEE Conference Computer Vision and Puttern Recognition, pages 1199-1208, 2018.
[4] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. arXiv preprint arXiv: 1707.3307,,,111, 2017.
[5] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29, 2016.
[6] Serean Ö Arik and Tomas Pfister. Protoattend: Attention-based prototypical learning. The Journal of Machine Learning Research, 21(1):8691-877 2020.
[7] Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966 2020.
[8] Shuai Lin, Chen Liu, Pan Zhou, Zi- Yuan Hu, Shuojia Wang, Ruihui Zhao, Yefeng Zheng, Liang Lin, Eric Xing, and Xiaodan Liang. Prototypical graph contrastive learning. IEEE Trans Neural Networks and learning systems, 2022.
[9] Junhua Wang and Yongping Zhai. Prototypical siamese networks for few-shot learning. In International Conference on Electronics Information and Emergency Communication, pages 178-181, 2020.
[10] Shizhen Zhao and Xiaojuan Qi. Prototypical votenet for few-shot 3d point cloud object detection. arXiv preprint arXiv:2210.05553, 2022.
[11] Na Zhao, Tat-Seng Chua, and Gim Hee Lee. Few-shot 3D point cloud semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 8873-8882, 2021.
[12] Aming Wu, Yahong Han, Linchao Zhu, and Yi Yang. Universal-prototyp enhancing for few- shot object detection. In IEEE International Conference on Computer Vision, pages 9567-9576. 2021.
[13] Gong Cheng, Bowei Yan, Peizhen Shi, Ke Li, Xiwen Yao, Lei Guo, and Junwei Han. Prototype- enn for few-shot object detection in remote sensing images. IEEE Transactions Geoscience and Remote Sensing, 10:1-10, 2021.
[14] Huifang Li, Yidong Li, Yuanzhouhan Cao, Yushan Han, Yi Jin, and Yunchao Wei. Weakly supervised object detection with class prototypical network. IEEE Transactions on Multimedia, 2022.
[15] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic seg- mentation: A prototype view. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2582-2593, 2022.
[16] Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, Yong Wang, and Fang Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 12414-12422 2021.
[17] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In IEEE International Conference on Computer Vision, pages 9197-9206, 2019.

---

[18] Boyu Yang, Chang Liu, Bohao Li. Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. European Conference on Computer Vision, pages 763-778, 2020.
[19] Yongfei Liu, Xiangyi Zhang. Songyang Zhang, and Xuming He. Part-aware prototype network for few-shot semantic segmentation. In European Conference on Computer Vision, pages 142-158, 2020.
[20] Hui Zhang and Henghui Ding. Prototypical matching and open set rejection for zero-shot semantic segmentation. In IEEE International Conference on Computer Vision, pages 6974- 6983, 2021.
[21] Zhengkai Jiang, Yuxi Li, Ceyuan Yang, Peng Gao, Yabiao Wang, Ying Tai, and Chengjie Wang. Prototypical contrast adaptation for domain adaptive semantic segmentation. In European Conference on Computer Vision, pages 36- 56, 2022.
[22] Dan Andrei Ganea, Bas Boom, and Ronald Poppe. Incremental few-shot instance segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1185-1194, 2021.
[23] Jinlu Liu, Liang Song, and Yongqiang Qin. Prototype rectification for few-shot learning. In European Conference on Computer Vision, pages 741-756, 2020.
[24] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.00 14, 2013.
[25] Jrg Bornschein, A. Mnih, D. Zoran, and D. J. Rezende. Variational memory addressing in generative models. In Advances in Neural Information Processing Systems, volume 30. 2017.
[26] Sajad Norouzi, David Fleet, and Mohammad Norouzi. Exemplar vae: Linking generative models, nearest neighbor retrieval, and data augmentation. In Advances in Neural Information Processing Systems, volume 33, pages 8753-8764, 2020.
[27] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.0634) 2015.
[28] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling, Improved variational inference with inverse autoregressive flow. In Advances Neural Infor- mation Processing Systems, volume 29, 2016.
[29] Bin Dai and David Wipf. Diagnosing and enhancing vae models. 193.::1903.05789, 2019.
arXiv preprint
[30] Andrea Asperti. Variance loss in variational autoencoders. In Machine Learning, Optimization, and Data Science, pages 297-308, 2020.
[31] Serena Yeung, Anitha Kannan, Yann Dauphin, and Li Fei-Fei. Tackling over-pruning in variational autoencoders. arXiv preprint
2017.
[32] Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. arXiv preprint arXiv:1901.000 00941004166 2019.
[33] Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv: 022622 2017.
[34] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. arXiv preprint ar
2017.
[35] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
[36] Tejas Anvekar, Ramesh Ashok Tabib, Dikshit Hegde, and Uma Mudengudi. Vg-vae: A venatus geometry point-cloud variational auto-encoder. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2978-298S, 2022.
12

---

[37] Zhizhong Han, Xiyang Wang, Yu-Shen Liu, and Matthias Zwicker. Multi-angh point cloud- vae: Unsupervised feature learning for 3d point clouds from multiple angles by joint self- reconstruction and half-to-half prediction. In IEFE International Conference Computer Vision, pages 10441-10450, 2019.
[38] Shidi Li, Miaomiao Liu, and Christian Walder. Editvae: Unsupervised parts-aware controllable 3d point cloud shape generation. AAAI Conference on Artificial Intelligence, volume 36, pages 1396-1394, 2022.
[39] Pulak Purkait, Christopher Zach, and lan Reid. Sg-vae: Scene grammar variational autoencoder to generate new indoor scenes. In European Conference Computer Vision, pages 155-171, 2020.
[40] Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimenez Rezende. Nerf-vae: A geometry aware 3d scene generative model. In International Conference on Machine Learning, pages 5742-5752 2021.
[41] Xin Wang, Thomas E Huang, Trevor Darrell, Joseph Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. arXiv preprint MoKiv:2003.09955/ 2020.
[42] Jiaxi Wu, Songtao Liu, Di Huang, and Yunhong Wang. Multi-scale positive sample refinement for few-shot object detection. In European Conference on Computer Vision, pages 456-472, 2020.
[43] Weilin Zhang and Yu-Xiong Wang. Hallucination improves few-shot object detection. In IEEE Conference on Computer Vision and Pattern Recognition, pages
2021.
[44] Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang. Fsce: Few-shot object detection via contrastive proposal encoding. In IEEE Conference on Computer Vision and Pattern Recognition, pages 7352-7362 2021.
[45] Bingyi Kang. Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few- shot object detection via feature reweighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8420-8429, 2019.
[46] Guangxing Han, Shiyuan Huang, Jiawei Ma, Yicheng He, and Shih-Fu Chang. Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment. In AAAI Conference Artificial Intelligence, volume 36, pages 780-789, 2022.
[47] Qi Fan, Wei Zhuo, Chi-Keung Tang. and Yu-Wing Wi-ning Tai. Few-shot object detection with attention- rpn and multi-relation detector. In IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 4013-4022, 2020.
[48] Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin. Meta R-CNN: Towards general solver for instance-level low-shot learning. In IEEE International Conference Computer Vision, pages 9577-9586, 2019.
[49] Guangxing Han, Yicheng He, Shiyuan Huang, Jiawei Ma, and Shih Chang. Query adap- tive few-shot object detection with heterogeneous graph convolutional networks. In IEEE International Conference on Computer Vision, pages 3263-3272, 2021.
[50] Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially-aware few-shot transfer. In Advances Neural Information Prucessing Systems, volume 33, pages 2 21981-21993, 2020.
[51] Tung-I Chen, Yueh-Cheng Liu, Hung-Ting Su, Yu-Cheng Chang, Yu-Hsiang Lin, Jia-Fong Yeh, Wen-Chin Chen, and Winston Hsu. Dual-awareness attention for few-shot object detection. IFFE Transactions on Multimedia, 2021.
[52] Ding-Jie Chen, He- Yen Hsieh, and Tyng-Luh Liu. Adaptive image transformer for one-shot object detection. IEEE Conference on Computer Vision and Pattern Recognition, pages 12247-12256, 2021.
13

---

[53] Guangxing Han, Jiawei Ma. Shiyuan Huang, Long Chen, and Shih-Fu Chang. Few-shot object detection with fully cross- transforanf.rr.r. In JEEE Conference on Computer Vision and Pattern Recognition, pages 5321-5330, 2022.
[54] Zhaoxin Fan, Hongyan Liu, Jun He, Qi Sun, and Xiaoyong Du A graph-based one-shot learning method for point cloud recognition. In Computer Graphics Forum, volume 39, pages 313-323, 2020.
[55] Ali Cheraghian, Shafin Rahman, Dylan Campbell, and Lars Petersson. Transductive zero-shot learning for 3D point cloud classification. In IEEE Winter Conference on Applications of Computer Vision, pages 923-933, 2020.
[56] Charles Ruizhongtai Qi. Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In Advances in Neural Information Processing Systems, volume 30, 2017.
[57] Yaoqing Yang. Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In IEEE Conference Computer Vision and Pattern Recognition, pages 206-215, 2018.
[58] Charles R. Qi, Or Litany, Kaiming Hc, and Leonidas J. Guibas. Deep hough voting for 3d object detection in point clouds. In IEEE International Conference on Computer Vision, 2019.
[59] Jiawei Liu, Xingping Dong, Sanyuan Zhao, and Jianbing Shen. Generalized few-shot 3d object detection of lidar point cloud for autonomous driving. arXiv preprint arXiv:2302.03914 2023.
[60] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European Conference on Computer Vision, pages 574-591, 2020.
[61] Ryosuke Yamada, Hirokatsu Kataoka, Naoya Chiba, Yukiyasu Domae, and Tetsuya Ogata. Point cloud pre-training with natural 3d structures. In IEEE Conference on Computer Vision and Pattern Recognition, pages 21283 21283-21293, 2022.
[62] Shuaihang Yuan, Xiang Li, Hao Huang, and Yi Fang. Meta-det3d: Learn to learn few-shot 3d object detection. In Asian Conference on Computer Vision, pages 1761-1776, 2022.
[63] Xiongwei Wu, Doyen Sahoo, and Steven Hoi. Meta-renn: Meta learning for few-shot object detection. In ACM International Conference on Multimedia, pages 1679-1687, 2020.
[64] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Puttern Recognition, pages 255, 2009.
[65] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In Advances in Neural Information Processing Systems, 2017.
[66] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pages 652-660, 2017.
[67] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rX1v:1711.05101 2017.
[68] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Rinhly-annotated 3d reconstructions of indoor scenes. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
14


Reviews:
[{"review_id": "XXNbN5Evna", "reviewer": "Reviewer_dL8b", "summary": "The paper introduces Prototypical Variational Autoencoder (P-VAE) for Few-Shot 3D Point Cloud Object Detection. It tackles the preservation of geometric information and data imbalance through learning distribution parameters. The authors propose two extensions, GP-VAE and CP-VAE, focusing on geometry and class specificity. Experiments demonstrate improved performance over state-of-the-art methods.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The P-VAE is innovative, focusing on learning distribution parameters instead of features, which is useful for few-shot learning.\n\n2. The encoder-decoder architecture effectively preserves geometric information, critical for 3D detection.\n\n3. The paper includes a robust experimental evaluation, showing performance gains and insights into prototype contributions.", "weaknesses": "1. The paper does not include experiments to assess the effectiveness and generalization of P-VAE across different domains on few shot learning, despite P-VAE being one of the main contributions.\n\n2. The observed improvements in the main experiments are relatively marginal, particularly in regard to the evaluation metric AP25.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time and reviewing our paper. Overall, most of the review comments request clarifications and minor revision on the paper. We will carefully revise the paper accordingly.  Also, we will release code upon the publication of this work.\n\n**Q1: Effectiveness and generalization of P-VAE across different datasets.**\n\nA1: Thanks for your insightful suggestion. Cross-dataset validation is indeed critical to address the distribution gap between point cloud data, which can be commonly observed in the real world. Inspired by your suggestion, we conduct a series of experiments across the FS-ScanNet and FS-SUNRGBD datasets. We compare the AP results on not only VoteNet [1] (the vanilla detector) and Prototypical VoteNet [2] (the few-shot learning detector) but also an implemented transferable P-VAE+. Since Cross-domain Few-shot learning (CDFS) for point cloud object detection is under-explored, we adapt a 2D CDFS method [3], which utilizes high-order self-supervision to regularize the latent space, the training scheme follows [4]. Please see Table A-C below for the details.\n\n- Part 1 Train on FS-ScanNet Split-1 then test on FS-SUNRGBD (5-shot).\n| |AP25|AP50|\n|:---:|:---:|:---:|\n|VoteNet[1] + FineTune|20.66|1.64|\n|Prototypical VoteNet[2]|21.95|1.77|\n|P-VAE|23.49|**1.91**|\n|P-VAE+|**23.52**|1.87|\n\nTable A: Train on base classes in FS-ScanNet split-1 (Cabinet, Bed, Chair, Sofa, Table, Door, Picture, Counter, Desk, Curtain, Refrigerator, ShowerCurtain, and Sink), then test on novel classes in FS-SUNRGBD (Toilet and Night Stand).\n\n- Part 2 Train on FS-ScanNet Split-2 then test on FS-SUNRGBD (5-shot).\n| |AP25|AP50|\n|:---:|:---:|:---:|\n|VoteNet[1] + FineTune|24.92|13.08|\n|Prototypical VoteNet[2]|26.25|18.90|\n|P-VAE|28.67|19.30|\n|P-VAE+|**29.02**|**19.76**|\n\nTable B: Train on split-2 base classes in FS-ScanNet (Cabinet, Chair, Sofa, Toilet, Door, Picture, Counter, Desk, Curtain, Refrigerator, ShowerCurtain, and Sink), then test on novel classes in FS-SUNRGBD (Bed, Table, and Night Stand).\n\n- Part 3 Train on FS-SUNRGBD then test on FS-ScanNet (5-shot).\n| |AP25|AP50|\n|:---:|:---:|:---:|\n|VoteNet[1] + FT|23.18|2.98|\n|Prototypical VoteNet[2]|25.93|3.54|\n|P-VAE|26.17|3.73|\n|P-VAE+|**26.73**|**3.76**|\n\nTable C: Train on base classes in FS-SUNRGBD (Sofa, Chair, Desk, Dresser, Bookshelf, and Bathtub), then test on novel classes in FS-ScanNet (Bed, Table, Toilet, and Garbagebin).\n\nWe will survey other CDFS approaches and exploit them on point cloud object detection. In the revised version, we will provide additional cross-dataset results to support further studies on 3D CDFS.\n\n**Q2: Explanation for the marginal improvements, especially on AP25.**\n\nA2: Thank you for your revision on the experiment section. For both benchmarks, our method achieves consistently top on the quantitative results, as shown in Tables 1 and 2 in the paper.\n(1) As for those comparable ones, in fact P-VAE gains significant improvements on the common-in-real-world categories. Please see Table D. These objects frequently appear in daily life, thus their precisions are critical for practical usages. Please also see the qualitative results in the linked Figure A https://drive.google.com/file/d/1_yfH-zu4PIa-vFGEr9AZ-0FF4bcGggRO/view?usp=sharing (top: FS-ScanNet and bottom: FS-SUNRGBD), where P-VAE predicts comparable or less boxes, but with superior accuracy (similar xyz location, but better lwh dimensions).\n(2) This result of Figure A is in accord with your observation, that our improvements on AP50 are stronger than AP25. AP50 is a stricter metric. In some cases, [2] predicts a coarse box with obvious deviation, though it can be counted for AP25, it is actually a false positive for AP50.\n(3) As for your suggestion on AP25, a possible solution for producing more predicted candidates is to augment novel samples with PointMix [5] then use the post-augmentation data for self-ensembling training[6]. The implementation is still on-going so we will provide the results in the revised version.\n|               |                    |                    |                           |                           |\n|:-------------:|:------------------:|:------------------:|:-------------------------:|:-------------------------:|\n| Category      | P-VAE              | P-VAE              | Prototypical VoteNet\\[2\\] | Prototypical VoteNet\\[2\\] |\n|               | AP25               | AP50               | AP25                      | AP50                      |\n| Bathtub       | **42.99 (+9.97)**  | **35.75 (+13.71)** | 33.02                     | 22.04                     |\n| Toilet        | **45.51 (+6.61)**  | **19.31 (+3.80)**  | 38.80                     | 16.23                     |\n| ShowerCurtain | **29.14 (+13.80)** | **4.98 (+1.26)**   | 15.34                     | 3.72                      |\n| Sofa          | **14.27 (+2.31)**  | **9.81 (+3.35)**   | 11.96                     | 6.46                      |\n\nTable D: Results on common-in-real-world classes on FS-ScaNet Split-1(1-shot).\n\n[1] Qi, Charles R., et al. \"Deep Hough voting for 3D Object Detection in Point Clouds.\" CVPR 2019.\n\n[2] Zhao, Shizhen, and Xiaojuan Qi. \"Prototypical VoteNet for Few-Shot 3D Point Cloud Object Detection.\" NeurIPS 2022.\n\n[3] Yuan, Wang, et al. \"Task-level Self-Supervision for Cross-Domain Few-Shot Learning.\" AAAI 2022.\n\n[4] Oh, Jaehoon, et al. \"Understanding Cross-Domain Few-Shot Learning Based on Domain Similarity and Few-Shot Difficulty.\" NeurIPS 2022.\n\n[5] Chen, Yunlu, et al. \"PointMixup: Augmentation for Point Clouds.\" ECCV 2020.\n\n[6] Zhao, Na, Tat-Seng Chua, and Gim Hee Lee. \"SESS: Self-Ensembling Semi-Supervised 3D Object Detection.\" CVPR 2020.\n\n"}, {"review_id": "nlTUGMvXC3", "reviewer": "Reviewer_WEFx", "summary": " The paper proposes an approach to enhance Few-Shot 3D Point Cloud Detection (FS3D) through prototype learning with VAEs. The authors leverage VAEs to learn prototypes represented by GMM-like distributions. Two VAEs are specifically designed to preserve geometric information and refine instance features. The effectiveness of the approach is validated through various experiments.\n", "questions": " 1.The paper does not explain the reason why prototypes learned by previous methods lose substantial 3D information and become less geometric-informative.\n\n 2.The motivation for using VAEs is not fully stated.\n\n 3.Are there any empirical results to demonstrate the underrepresentation / overfitting problem of prototypes?\n\n 4.Why can GMM-like distribution handle the overfitting problem of the latent space?", "limitations": " 1.The authors have addressed the limitations that data-rooted problem will lead the representative prototypes to distort and perturb the latent space negatively.\n\n 2.Additionally, the proposed approach depends on the number of prototypes. The sensitivity raises concerns about the practical applicability of the proposed approach.\n", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": " 1.The paper introduces an approach utilizing VAEs to learn geometric and class-specific prototypes with improved performance on FS3D tasks.\n\n 2.The effectiveness of the modules and prototypes is demonstrated through various experiments.\n\n 3.The writing is easy to follow.", "weaknesses": " 1.The authors do not provide a comprehensive and detailed discussion of their motivations for the proposed approach. This should include a clear identification of the core issues, a thorough evaluation for\u0010 the limitations of previous works in addressing this problem, and a convincing explanation of how incorporating VAEs can effectively tackle these limitations. It would greatly enhance the paper's credibility if qualitative and quantitative results were presented to support their motivations and demonstrate the superiority of the proposed method.\n\n 2.The sensitivity of the number of geometric-informative prototypes as observed in Table 6 raises concerns about the practical applicability of the proposed approach. It would be beneficial if the authors discussed the implications of this sensitivity and provided insights on how to determine an optimal or adaptive number of Geo-proto for real-world applications.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read our submission and for the valuable comments. We address the reviewer’s comments below. \n\n**Q1: Why are the prototypes learnt by previous methods less geometric-informative? More detailed discussions about the limitations of previous works.**\n\nA1: Thanks for your in-depth comments. The typical FS3D methods related to geometric prototypes are [1][2]. Since [2] do not explicitly learn the prototypes, we will mainly discuss [1] and briefly mention [2]. \n\n[1] learns geometric prototypes in the encoder embedding space, this high-dimensional disordered space (where they group features and use average each group as one prototype) is directly connected to the down-stream task. The framework is supervised by only a task loss, which is specified to fit the target GT (i.e., 3D boxes). Without extra regularization, the deep network loses control of the latent space. The linked Figure A {https://drive.google.com/file/d/1SVDLQj6F0wXsO-kO0-HuYg-0ERVaCkNW/view?usp=sharing} illustrates how geometric prototypes can construct a whole scenario (please see Supplementary B.2 for implementation details). Each prototype is painted with a unique color. Figure A demonstrates that the latent space of [1] fails to preserve meaningful 3D information. The geometric prototypes are not well-organized and show high randomness. In contrast, our prototypes retain reasonable physics structures and semantic consistency. \n\nThe prototypes in [2] are not learnable features but 3D point clouds. These low-level prototypes are generated from random points through affine transformations, then clustered into categories by paired similarity. Please see Figure B https://drive.google.com/file/d/1xTdfyuL8mtXrlGED2-lsaQWBUqFDFB-x/view?usp=sharing , such prototypes can be very unrealistic, based on our common sense for real-world objects.  \n\n**Q2: The motivation of using VAE.**\n\nA2: The core of FS3D is understanding 3D scenes with the least data requirement. We detail the motivation from four aspects.\n\n(i) For Geometric-informative Prototypes (GP), we use the set of a fixed number of GP to describe the entire 3D world, thus we expect each GP can have different shape variants given different contexts (e.g., the 'stick' GP can be short in a chair but long in a desk). VAE allows us to sample diverse features based on the prototypes (Gaussian centers), thus can provide a wide range of variants.\n\n(ii) For Class-specific Prototypes (CP), the base-class prototypes can be superior to the novel-class ones because of data imbalance, shown in Figure C of A3. We therefore use VAE for alignment. The parameterized latent space can help augment data through covering various features around a prototype center.\n\n(iii) Particularly for the novel classes CP, since we access very limited samples, we cannot fully understand all the shape variants of a category, as shown in Figure D https://drive.google.com/file/d/1k0bP29YyCBvvErqhcCCSZQ5S3q-bz-NB/view?usp=sharing. The distribution space of VAE retains higher feature transferability, such that we can learn class-specific prototypes instead of instance-specific ones. \n\n(iv) VAE is a generative model, its reconstruction task can be a regularization scheme to preserve 3D information of the latent space, as we discussed in A1. \n\n**Q3: Results that can explain the underrepresentation/overfitting of prototypes.**\n\nA3: Please see Figure C https://drive.google.com/file/d/13xzTPOpgCTA_FqKbGgA6C0AWD6YuhpEJ/view?usp=sharing for comparison results. Due to the data imbalance problem, for the baseline[1], we can observe that the quality of novel-class prototypes is significantly worse than that of base classes, whereas P-VAE can generate equally convincing prototypes without much bias. \nAs an extension for A2 (iii), we conduct the following experiments. Given the novel class ‘Table’, we split the training samples into two sets w.r.t. their shapes: Standard Tables (i.e., the four-leg tables with a rectangular tabletop, as we can commonly find in the real world), and; Strange Tables (i.e., ones with a round or irregular-shaped top, supported by a steady base instead of four sticks). The networks are trained on one type and tested on another. The quantitative results are in Table A. [1] fits the given instance and cannot generalize for all tables, whereas P-VAE learns the generalized category information thus can better transfer to other table-like shapes. \n|                           |                                 |                                 |                                 |                                 |\n|:-------------------------:|:-------------------------------:|:-------------------------------:|:-------------------------------:|:-------------------------------:|\n|                           | Train on Table1, Test on Table2 | Train on Table1, Test on Table2 | Train on Table2, Test on Table1 | Train on Table2, Test on Table1 |\n|                           | AP25                            | AP50                            | AP25                            | AP50                            |\n| Prototypical VoteNet\\[1\\] | 18.68                           | 7.01                            | 16.92                           | 6.72                            |\n| P-VAE                     | **23.05**                       | **10.44**                       | **21.97**                       | **9.41**                        |\n\nTable A: Using Standard-shaped table (Table1) and Strange-shaped table (Table2) for training/testing on FS-ScanNet Split-2 (5-shot).\n\n**Q4: Why can the GMM-like distribution handle the overfitting problem?** & **Q5: More discussion on the number of prototypes (related to Table 6 in the paper). How sensitive is it? How to determine an optimal number?**\n\nA4 & A5 & Reference: Due to space limitation, we link the response here https://drive.google.com/file/d/1FVx8bJ6zxUTpsBfvExtRuwnaF4MvbOhH/view?usp=sharing. We appreciate your patient reading.\n\n"}, {"review_id": "NJgXltJ8E1", "reviewer": "Reviewer_cCbd", "summary": "This paper studies a challenge task called FS3D. They first presents that the previous work on FS3D lacks fine-level supervision, as the intermediate features are simply averaged to update the prototypes, which are then used to augment features for sequential detection. In order to solve this problem, they leverage Prototypical Variational Autoencoder to conduct regularization on both local and global levels. ", "questions": "1. Please see the weakness\n2. Can the proposed method be used to other few-shot learning tasks in point cloud, such as segmentation ?  \nMy main concern is that why does author choose Prototypical Variational Autoencoder as the reconstruction model. Does it have outstanding property for FS3D ?  It is the key for the acceptance of this paper. \nif authors could address my concerns, I am willing to increase the score. ", "limitations": "yes ", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow since the motivation is clear.\n2. The experimental improvement is significant. \n", "weaknesses": "1. The related works on FSL for 3D Point Cloud is insufficient. \n2. why does Prototypical Variational Autoencoder work in this task ? Have you tried other models for reconstruction ? \n\n \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our submission and for the insightful comments. We address the reviewer’s comments below.\n\n**Q1: Survey on 3D FSL.**\n\nA1: Thank you for pointing this out. Our work focuses on FS3D, we believe a more careful review on FSL for point cloud can lead to high-level insights. We categorize recent works into four families according to the FSL strategies. [10,14,16] incorporate self-supervision or semi-supervision. [6,7,8,9,13] regularize and align the latent space with the concept ‘prototype’. High-dimensional prototypes are used to cluster the embeddings while low-level prototypes represent basic geometric structures. [1,4,7,8,9,12,17] aim to evaluate the difference/similarity between base and novel pairs, such that the features can be separated more discriminatively. [2,3,5,11,15] enrich the limited 3D samples with data augmentation and multi-modality fusion. We will give a more detailed and comprehensive overview in the revised version. \nHere we link the reference [1-17] https://drive.google.com/file/d/1WyLnRVP53627-qZ861wJpZgq241Slefi/view?usp=sharing.\n\n**Q2: Why does P-VAE work? Why it is a better reconstruction model in comparison with other reconstruction approches?**\n\nA2: Thank you for your careful revision. Parallel to the detection task, we add a reconstruction task for preserving 3D information in the latent space. This is a widely-used self-supervised learning scheme for 3D FSL. Compared with the original reconstruction model AE, P-VAE learns a multi-center distribution that each centers at a prototype, enabling us to sample from the probabilistic space. This design imitates the real 3D environments (a prototype can have lots of variants), and also tackles the data imbalance problem (although the novel samples are limited, we can augment features by sampling). See Table A for the quantitative results of P-VAE superior to AE. In fact, the key is to obtain a parameterized latent space in the reconstruction model. Inspired by your suggestion, we study reconstruction models[18,19]. In Table B, ‘w/’ and ‘w/o’ mean whether we use variational probabilistic learning or not. We modify a CP-VoteNet as our baseline. We can find that the extra reconstruction task gains only marginal improvement (regardless of the approach), whereas the incorporation of a parameterized feature space does matter. \nFor more discussion on how P-VAE helps FS3D, due to space limit please refer to this link https://drive.google.com/file/d/1EBdjNVKS194HKfJtCOCje4UPQ-fKj1ry/view?usp=sharing. We appreciate your patient reading.\n|                                 |           |           |\n|:-------------------------------:|:---------:|:---------:|\n|                                 | AP25      | AP50      |\n| Prototypical VoteNet + AE | 14.97     | 9.32      |\n| P-VAE                           | **16.00** | **10.22** |\n\nTable A: Performance of using P-VAE over [13] + AE on FS-ScanNet Split-1 (1-shot).\n\n|                        |           |           |\n|:----------------------:|:---------:|:---------:|\n|                        | AP25      | AP50      |\n| Baseline               | 13.47     | 8.33      |\n| \\+ GAN\\[18\\] w/o       | 14.13     | 8.92      |\n| \\+ GAN\\[18\\] w/        | 14.96     | 9.51      |\n| \\+ Diffusion\\[19\\] w/o | 14.20     | 9.05      |\n| \\+ Diffusion\\[19\\] w/   | 15.43     | 9.68      |\n| P-VAE                  | **16.00** | **10.22** |\n\nTable B: Reconstruction models on FS-ScanNet Split-1 (1-shot).\n\nFor GAN, we pass the voted instance-level features through an MLP to predict distribution parameters, then feed the sampled features into a generator[20] to reconstruct the object, while the discriminator produces per-point scores to distinguish our prediction and GT. For diffusion model, we follow the similar scheme to incorporate probabilistic modeling, and the shape latent is replaced by each voted feature. \n\n[18] Goodfellow, Ian, et al. \"Generative Adversarial Nets.\" NIPS 2014.\n\n[19] Luo, Shitong, and Wei Hu. \"Diffusion Probabilistic Models for 3D Point Cloud Generation.\" CVPR 2021.\n\n[20] Li, Chun-Liang, et al. \"Point Cloud GAN.\" arXiv preprint arXiv:1810.05795 (2018).\n\n**Q3: Does P-VAE work for other FSL 3D tasks? e.g., Segmentation.**\n\nA3: Yes. P-VAE is a general prototype learning scheme that can be plugged into many point cloud networks. Please refer to Supplementary C.5, where we can easily adapt P-VAE for various architectures. \n[21] and [22] are grouping-based instance segmentation methods. We can deploy GP-VAE at the pre-grouping stage and CP-VAE at the post-grouping stage. [23] is a transformer-based network, we then deploy GP-VAE prior to the transformer decoder, and CP-VAE between the decoder and the mask module. Considering the computational cost, P-VAE is not preferred in the shallow layers.\nWe provide some preliminary results of [21] + plug-in P-VAE, comparing with a recent segmenter[24] on ScanNetv2. Please see Table C and Figure A https://drive.google.com/file/d/1fKREOe6VjEZmwRqWmLVxxRzFLpU8Rugs/view?usp=sharing. We also observe that our recall, especially for objects with larger sizes, is lower than [24]. The coarse implementation has not been thoroughly checked so we will finalize the results in the revised version.\n|                       |          |          |\n|:---------------------:|:--------:|:--------:|\n|                       | mAP      | AP50     |\n| Geodesic-Former\\[24\\] | **10.6** | **19.8** |\n| \\[21\\] + P-VAE                 | 8.04     | 15.35    |\n\nTable C: Both methods are trained on Fold0 and tested on Fold1(1-shot).\n\n[21] Jiang, Li, et al. \"Pointgroup: Dual-set point grouping for 3d instance segmentation.\" CVPR 2020.\n\n[22] Vu, Thang, et al. \"Softgroup for 3d instance segmentation on point clouds.\" CVPR 2022.\n\n[23] Schult, Jonas, et al. \"Mask3D: Mask Transformer for 3D Semantic Instance Segmentation.\" ICRA 2023.\n\n[24] Ngo, Tuan, and Khoi Nguyen. \"Geodesic-Former: A Geodesic-Guided Few-Shot 3D Point Cloud Instance Segmenter.\" ECCV 2022.\n\n\n"}, {"review_id": "wLvhjltVEw", "reviewer": "Reviewer_B5ky", "summary": "This paper proposes a novel approach for few-shot 3D object detection by combining prototype learning and variational autoencoders. To address the weak geometry regularization and data imbalance issues of the existing methods, it proposes a novel VAE specifically designed for prototype learning named Prototypical VAE (P-VAE). Moreover, it extends P-VAE with geometric-informative prototypes and class-specific prototypes to enhance the object detection performance in the few-shot regime. Experimental results on several benchmarks validate the effectiveness of the proposed method.", "questions": "Please refer to the weaknesses.", "limitations": "The limitations of the proposed method and potential solutions are discussed in the final section of the paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposes two prototypical VAE based on geometric-informative and class-specific prototypes, which helps solve the weak geometry regularization and data imbalance issues encountered by the existing few-show 3D object detection methods based on prototype learning. This combination of prototype learning and VAE is novel and effective.\n\n2. Experimental results showed strong support for the effectiveness of the proposed method in the few-shot regime. In particular, it shows consistent improvement over baseline methods on various settings of different benchmarks. \n\n3. The paper is well written, with easy-to-follow equations and proper notations.", "weaknesses": "1. In order to detect objects from the scene point clouds, a number of instances should be predicted. However, it is not clear how to obtain the number of the predicted instances $N_{ins}$ from $N_z$ features $z'_i$.\n\n2. There is no ablation study of the two feature calibration described in Eq-6 and Eq-9, how much do they contribute to the final performance improvement? \n\n3. What are the object detection heads in Figure-2? How are they trained exactly?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our submission carefully and for the insightful suggestions. We address the reviewer’s comments below. Also, we will release code upon the publication of this work.\n\n**Q1: How to obtain the number of instances $N_{ins}$?**\n \nA1: Given the per-point features $\\\\{z'_i\\\\} _{i=1}^{N_z}$, we use MLP to predict both feature offsets ($D$-dimension) and point offsets (3-dimension). We add the point offsets to the original point to get the shifted points, then adopt Furthest Point Sampling (FPS) to obtain $N_{ins}$ votes. $N_{ins}$ is produced by the network on-the-fly, depending on the input scenario. The above procedure is a clustering module following VoteNet[1], here we link pseudocode    https://drive.google.com/file/d/1f_nBjNV_eWK90lP1Xya6ByVUszC97yNw/view?usp=sharing for your reference.\n\n**Q2: Ablation studies for feature calibration.** \n\nA2: Thank you for pointing this out. In GP- and CP-VAE, we use a cross-attention module (Eq. 6 and Eq. 9) to enrich the features with the prototype information. Our studies for calibration are inadequate in the submitted version. Here we add comparisons with three calibration approaches: concatenation[2], perturbed attention[3] and adaptive attention[4]. See Table A, we find that adaptive attention can achieve better performance than base attention. The reason may be, the soft attention with meta-reweighting strategy can localize and highlight the region of interest in query samples. We will modify our method carefully and thanks again for your suggestion. \n|                          |           |           |           |           |\n|:------------------------:|:---------:|:---------:|:---------:|:---------:|\n|                          | 3-shot    | 3-shot    | 5-shot    | 5-shot    |\n|                          | AP25      | AP50      | AP25      | AP50      |\n| P-VAE                    | 31.60     | 19.37     | 32.84     | 22.39     |\n| Concatenation\\[2\\]       | 23.44     | 11.07     | 29.18     | 17.92     |\n| Perturbed Attention\\[3\\] | 29.31     | 10.09     | 31.86     | 19.23     |\n| Adaptive Attention\\[4\\]  | **31.89** | **20.04** | **33.01** | **22.78** |\n\nTable A: Ablations studies for feature calibration strategies on FS-ScanNet Split-1.\n\n**Q3: Detail the detection heads. Besides, how to train the heads?**\n\nA3: Thank you for pointing this out. Referring to the 'Object Detection Heads' in Figure 2 of the paper, the detection includes three steps: \n\n(i) Normalization. Given $N_{ins}$ clusters, each cluster is denoted as $C = \\\\{c_i\\\\}_{i=1}^n$, where $n$ is the number of features assigned to $C$ through FPS (discussed in A1). $c_i$ is composed of two attributes: the $D$-dimension features $w'_i$  (see Figure 2 in the paper) and the 3-dimension shifted point coordination $p_i$. We calculate the clustering center $\\\\bar p=\\\\frac{1}{n} \\\\sum_{p=1}^n {p_i}$ then locally normalize $p_i$ to $p'_i=(p_i - \\\\bar p)/r$, where $r$ is the maximum distance to the center. \n\n(ii) We concatenate $w'_i$ with $p'_i$ for the ($D$+3)-dimension $c'_i$, then use a 1-layer MLP on each $c'_i$, resulting in $n$ features. We collect them and use MaxPooling to get a single ($D$+3)-dimension feature $c$. \n\n(iii) We use a 1-layer MLP on $c$ to obtain the final vector, which contains an objectness score, bounding box information (i.e., xyzlwh) and classification scores.\n\nSimilar to VoteNet, we use the standard 3D detection loss to supervise the head. We assign predictions within 0.3m to a GT as 'positive', and assign predictions far from any GT more than 0.6m as 'negative'. We use Cross-Entropy loss to supervise the pos/neg objectness. For each positive candidate, we take the nearest GT as its paired GT, which is used to supervise the box location and dimension with Smooth-L1 loss. We also use Cross-Entropy loss to supervise the classification score of each positive candidate.\n\nWe will add this part in the Implementation Details section of the revised version for clarification.\n\n[1] Qi, Charles R., et al. \"Deep Hough Voting for 3D Object Detection in Point Clouds.\" ICCV 2019.\n\n[2] He, Shuting, et al. \"Prototype Adaption and Projection for Few-and Zero-Shot 3D Point Cloud Semantic Segmentation.\" TIP 2023.\n\n[3] Lu, Yu, et al. \"Attention Calibration for Transformer in Neural Machine Translation.\" ACL-IJCNLP 2021.\n\n[4] Jiang, Zihang, et al. \"Few-Shot Classification via Adaptive Attention.\" arXiv preprint arXiv:2008.02465 (2020).\n\n\n\n"}]

Author rebuttal:

Meta-review: The paper proposes an approach to enhance Few-Shot 3D Point Cloud Detection (FS3D) through prototype learning with VAEs. The reviewers note the novelty of the approach and its good empirical performance. They raised concerns regarding the motivation of the method and its sensitivity in the number of prototypes. The rebuttal submitted by the authors addressed these concerns.
The AC read the paper and found the presentation and command of English and accuracy of expression below the bar of acceptance. Yet, the result of the paper showing that reconstruction tasks assist few-shot discriminative tasks is an interesting one, that may be useful for the community, outweighing the low presentation quality of the paper.
Decision: Accept (poster)