Title: Prototypical Variational Autoencoder for 3D Few-shot Object Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3D Point Cloud Object Detection

Few Shot Learning

Computer Vision

Geometric Prototype
Abstract: Few-Shot 3D Point Cloud Object Detection (FS3D) is a challenging task, aiming to detect 3D objects of novel classes using only limited annotated samples for training. Considering that the detection performance highly relies on the quality of the latent features, we design a VAE-based prototype learning scheme, named prototypical VAE (P-VAE), to learn a probabilistic latent space for enhancing the diversity and distinctiveness of the sampled features. The network encodes a multi-center GMM-like posterior, in which each distribution centers at a prototype. For regularization, P-VAE incorporates a reconstruction task to preserve geometric information. To adopt P-VAE for the detection framework, we formulate Geometric-informative Prototypical VAE (GP-VAE) to handle varying geometric components and Class-specific Prototypical VAE (CP-VAE) to handle varying object categories. In the first stage, we harness GP-VAE to aid feature extraction from the input scene. In the second stage, we cluster the geometric-informative features into per-instance features and use CP-VAE to refine each instance feature with category-level guidance. Experimental results show the top performance of our approach over the state of the arts on two FS3D benchmarks. Quantitative ablations and qualitative prototype analysis further demonstrate that our probabilistic modeling can significantly boost prototype learning for FS3D.

Prototypical Variational Autoencoder for Few-shot 3D Point Cloud Object Detection

Abstract

Few-Shot 3D Point Cloud Object Detection (FS3D) is a challenging task, aiming to detect 3D objects of novel classes using only limited annotated samples for training. Considering that the detection performance highly relies on the quality of the latent features, we design a VAE-based prototype learning scheme, named prototypical VAE (P-VAE), to learn a probabilistic latent space for enhancing the diversity and distinctiveness of the sampled features. The network encodes a multi-center GMMlike posterior, in which each distribution centers at a prototype. For regularization, P-VAE incorporates a reconstruction task to preserve geometric information. To adopt P-VAE for the 3D object detection framework, we formulate Geometricinformative Prototypical VAE (GP-VAE) to handle varying geometric components and Class-specific Prototypical V AE (CP-V AE) to handle varying object categories. In the first stage, we harness GP-VAE to aid feature extraction from the input scene. In the second stage, we cluster the geometric-informative features into per-instance features and use CP-V AE to refine each instance feature with categorylevel guidance. Experimental results show the top performance of our approach over the state of the arts on two FS3D benchmarks. Quantitative ablations and qualitative prototype analysis further demonstrate that our probabilistic modeling can significantly boost prototype learning for FS3D.

1 Introduction

3D point cloud object detection is a fundamental scene understanding task, aiming at recognizing and localizing objects in point-based scenes. Recent deep-learning methods are mostly fully supervised, requiring extensive labeled data to support the model training. Such a data-hungry setting is, however,

Reconstruction

pro1

Encoder

pr02

Decoder

Input

GMMDistributionSpace

Geometric-informative

Prototypes

Class-specific

DiverseSampledFeatures

Objects

Prototypes

impractical in real applications, considering the heavy and costly loads to densely annotate the data. Inspired by humans' capability of recognizing novel objects by learning from limited data, Few-shot Learning (FSL) [1, 2, 3, 4, 5] has been proposed to enable neural networks to handle novel data by training on large labeled data of base classes and small labeled data of novel classes.

Prototype learning is an effective approach to achieve FSL [2, 6, 7, 8, 9, 10, 11]. It has been applied successfully to various 2D vision tasks, e.g., object detection [12, 13, 14], semantic segmentation [15, 16, 17, 18, 19, 20], and instance segmentation [21, 22]. In prototype learning, some principles are first defined to separate the prototypes, e.g., by category [23, 17, 13, 14, 16, 21], by partial similarity [19, 11], or by intrinsic characteristics [12, 15, 17, 18, 20]. These prototypes can be viewed as clustering centers, such that we can group the latent features based on our pre-defined principles. Importantly, the prototypes provide transferable knowledge that allows us to generalize a model for handling novel classes, thereby facilitating representation learning for downstream tasks.

In this work, we exploit prototype learning for Few-Shot 3D Point Cloud Object Detection, named FS3D. We aim to predict accurate 3D bounding boxes on objects of novel classes, given only a small amount of labeled novel-class exemplars. To address FS3D with prototype learning, Zhao et al. [10] propose a pioneering work. Yet, their learning scheme lacks fine-level supervision, as the intermediate features are simply averaged to update the prototypes, which are then used to augment features for sequential detection. There are two fundamental issues. First, without strong regulation on prototype learning, the prototypes can lose substantial 3D information and become less geometric-informative. Second, due to the data imbalance, prototypes of novel classes are clearly underrepresented than those of the base classes, so the latent space can easily overfit the base ones.

To address the above issues, we propose a variational autoencoder approach particularly designed for prototype learning, named Prototypical V AE (abbr. P-V AE). As Fig. 1(a) illustrates, instead of directly learning the features, our P-V AE learns the distribution parameters, by which we can construct a Gaussian Mixture Model(GMM)-based posterior for sampling features from the probabilistic latent space. For issue 1, we propose to perform reconstruction tasks with an encoder-decoder architecture in our P-VAE framework to better preserve the geometric information in the intermediate features. For issue 2, our P-V AE allows us to sample diverse and varied features controlled by a multi-center distribution. Hence, even with limited novel samples, our approach can still gather richer information to generate good-quality representative prototypes for the novel classes.

To enhance the object detection task with the proposed P-V AE, we formulate two crucial extensions: Geometric-informative Prototypical VAE (GP-VAE) and Class-specific Prototypical VAE (CP-VAE). As Fig. 1 shows, we can view real-world 3D objects from two aspects. In a broader sense, all the objects, regardless of their class labels, can be reasoned by a set of local geometric components; see Fig. 1(b). For GP-VAE, each component is a geometric-informative prototype, such that a given point cloud scene can be constructed by combining these prototypes. Narrowing the scope to each object, objects of the same category have similar shapes, while objects of different categories look more different; see Fig. 1(c). Hence, we propose CP-VAE to obtain class-specific prototypes, such that each prototype can serve as a strong shape prior to guiding the object-level feature learning. We incorporate the two modules in our object detection pipeline, leveraging GP-VAE to account for scene-level feature extraction and CP-V AE for object-level feature refinement; see Fig. 2.

We conduct extensive experiments on two FS3D benchmarks, FS-ScanNet and FS-SUNRGB [10]. The results in Sec. 4.2 show that our method significantly outperforms the SOTA approaches in various few-shot settings. More analysis on prototypes in Sec. 4.3 demonstrate the effectiveness of leveraging VAE for prototypical learning. To summarize, our contributions are listed as follows:

We propose the Prototypical VAE (P-VAE) to learn multiple distributions centering at prototypes, such that it can construct a probabilistic latent space that allows us to effectively sample diverse features with higher controllability and flexibility.

To harness P-VAE for object detection, we extend it for geometric-informative and classspecific prototypical V AEs (i.e., GP-V AE & CP-VAE) based on our observations.

Results demonstrate the superiority of our new approach for FS3D and further studies analyze the prototypes and explain the effectiveness in detail; see Sec. 4 and Appendix.

2 Related Works

Variational Autoencoder (VAE) is first introduced by Kingma et al. [24] to learn a generative model for constructing a distribution-based latent space. It provides a probabilistic representation learning scheme with theoretical support. Bornschein et al. [25] assume that features follow a multicenter prior distribution, which encourages unsupervised deep clustering in the embedding space. Norouzi et al. [26] select exemplars from the training data and use them to model the embedding space with GMM. To strengthen the robustness, [27, 28, 29] balance between reconstruction authenticity and generation diversity; [30, 31, 32] aims at preventing V AE models from collapse; [33, 34, 35] explore usages of different regularizations. VAE is widely adopted in 3D generation tasks, e.g., point cloud generation [36, 37, 38] and scene reconstruction [39, 40]. However, it has not been fully explored for 3D perception tasks, especially for the challenging FS3D task.

Few-shot 2D Image Object Detection (FS2D) methods can be classified into two categories. The first family aims to address the data imbalance problem, for example, by fine-tuning with balanced data [41], augmenting objects by different scales [42], augmenting novel samples with generative networks [43], learning reasonable and unbiased latent space [44], etc. The second family focuses on learning support (base)-novel (query) correlations. Kang et al. [45] reweight the query features given the support features; Han et al. [46] fuse the support and query features by spatial alignment; Fan et al. [47] and Yan et al. [48] exploit deep support information to guide the region proposal network and the detection heads; some other works incorporate support information learned from the backbone to guide the feature extraction of the query images, using transformers [49, 50, 51, 52, 53] or graph convolution networks [49]. An intuitive solution is to adopt these FS2D strategies to tackle FS3D, as some of our comparison methods in Sec. 4.2. However, their detection results (see Tab. 1 and 2) are far worse than our P-VAE. The reason is that 3D point clouds are intrinsically more disordered and complex than pixel-grid 2D images, thus naturally lacking abundant data with full annotations.

FSL for 3D Point Cloud is under-explored, as we discussed before. As a pioneer work, Fan et al. [54] tackle the one-shot point cloud classification, exploring base-novel sample relations with graph convolution. Cheraghian et al. [55] introduce a transductive zero-shot learning method with a triplet loss to obtain a better classifier. Zhao et al. [11] incorporate FSL for 3D semantic segmentation with a transformer for multi-prototype transductive inference. To our best knowledge, Prototypical VoteNet [10] is the first and only work on FS3D that harnesses prototype learning. However, their experimental results, refer to the indoor scenes in Fig. 3, are not satisfactory. First, it lacks a strong regulation for prototype learning, like our reconstruction task, to preserve latent geometric information. Second, the quality of their novel-class prototypes is impaired due to data imbalance, whereas we propose a new VAE-based approach to sample features with significant diversity.

3 Method

In this section, we first define the FS3D task and give a modular overview of our pipeline in Sec. 3.1. Then, we present the formulation of our novel Prototypical VAE (P-VAE) to learn a probabilisticdistribution-based feature space in Sec. 3.2. Building upon the P-V AE, we develop (i) Geometricinformative Prototypical V AE (GP-VAE) (Sec 3.3) and (ii) Class-specific Prototypical V AE (CP-V AE) (Sec 3.4), which are customized for learning prototypes of varying geometric components and varying object categories, respectively.

Sample

Scene-levelReconstruction

Ngeo-center Prior p(Z)

ge01

Z1

ge02

Scene

Scene

Encoder

★

Decoder

ZNz

Input point cloud

Nz-centerPosterior q(Z|X)

Reconstructed point cloud

Calibration

Sample

Distribution

{z}

Ncls-centerPrior p(W)

cls1

W1

Object-levelReconstruction

approximation

clS2

Object

Object

GP-VAE

Encoder

★

Decoder

ClSNcls

WNins.

Reconstructed 3D objects

CP-VAE

Nins-centerPosteriorq(W|z)

Calibration

Cluster Nfeatures to Nins

{w3

instance-levelfeatures

ObjectDetectionHeads

Detectionresults

3.1 Problem Definition and Framework Overview

In the FS3D task, the full class set C is split into a base set C base and a novel set C novel , where C base ∩ C novel = ∅ . Each base class in C base has plenty of annotated objects, while the amount of labeled samples is very limited for each novel class in C novel . The FS3D annotation for an instance of classi is ( pos, dim, c i ) , denoting the box center point, 3D box dimensions, and the object class label. Following the FSL definition, the K -shot setting means only K annotated objects can be used for training in each novel class. Our goal is to enable a model pre-trained on sufficiently-annotated C base samples to generalize over new classes using only K labeled objects per class in C novel .

Fig. 2 shows the overall pipeline. Built upon the detection framework, we perform two reconstruction tasks during the training to regulate the prototype learning and model the probabilistic latent space:

(i) Scene-level reconstruction, in which we incorporate our proposed GP-VAE to learn geometric prototypes { geo } with a variational encoder-decoder architecture. From the learnt distribution, we sample features { z } for decoding and further refine { z } to { z ′ } with the geometric prototypes for down-stream object detection (see Sec. 3.3); and

(ii) Object-level reconstruction, in which we adopt a voting-based object encoder to cluster the scene-level features { z ′ } into object-level features and incorporate the proposed CP-VAE to learn class-specific prototypes { cls } as distribution centers. Similar to (i), we sample features { w } for decoding and output calibrated features { w ′ } (see Sec. 3.4).

In the end, we use multiple feed-forward network heads, which are fed with { w ′ } , to generate instance-level 3D box positions and dimensions as the detection results. During the inference, we can skip the reconstruction decoders and adopt the pre-trained prototypes for forward data flow.

3.2 Prototypical Variational Autoencoder

Our proposed Prototypical VAE (P-VAE) is built upon a standard VAE as its backbone to learn high-level prototypes. Referring to Fig. 1(a), the overall architecture consists of an encoder and a decoder. X is the variable that represents our data and the encoded latent space follows a certain prior distribution p ( Z ) . The encoder f parameterized by θ f takes the point cloud x ∈ R N x × 3 as the input and generates parameters µ ∈ R N z × d and σ ∈ R N z × d , where N z is the number of parameters { µ i , σ i } and d is the latent space dimension. We adopt them to obtain a set of predicted posteriors q θ f ( Z i | X ) ∼ N ( µ i , σ i ) , where i = 1 ...N z .

Based on these N z normal distributions, we can sample N z new features { z i | z i ∈ R d } N z i =1 . Then, we adopt a decoder g , which is parameterized as θ g , taking each z i to reconstruct a point cloud with N p points p i ∈ R N p × 3 . Finally, we combine these reconstruction results to form our point cloud prediction p . Given the ground-truth point cloud ˆ p , the optimization goal for P-V AE is

where CD denotes the Chamfer Distance that supervises the reconstruction task and KL denotes the Kullback-Leibler Divergence to regularize the posteriors q θ f ( Z i | X ) , i = 1 ...N z to stay close to the latent space prior p ( Z ) . By far, we can learn this distribution space in a standard V AE term.

Importantly, we focus on obtaining prototypes based on latent distributions. Denoting { pro i | pro i ∈ R d } N pro i =1 as N pro prototypes, our goal is to use them as clustering centers to group the latent embeddings. Hence, we define p ( Z ) following an approximate GMM, which is composed of N pro independent normal distributions that each centers at pro i and has variance I :

Compared with the standard GMM, our approximate GMM is designed with two strategies. First, we adopt a fixed weight 1 A ( ∫ R d p ( Z = z ) dz = 1 ) instead of distinctive learned weights for these normal distributions, such that each prototype can be equally accessed. This design is for imitating a real-world situation, where all the geometric components share the same probability to be observed among common geometric structures. Second, we adopt a hard regularization 1 in Eq. (2), such that each sampled feature z i can be directly assigned to its nearest prototype. In this way, our intermediate features are more organized, since they are discriminatively grouped by the prototypes. By introducing pro , we modify the posterior distribution q θ f ( Z i = z ∥ X ) = A exp( -1 2 ∥ z -µ i ∥ 2 2 ) 1 {∥ z -pro j ∥ 2 2 ≤ ∥ z -pro k ∥ 2 2 } , where A is the normalization term and pro j is the closest prototype to µ i .

With Eq. (2), we can optimize the KL term of Eq. (1) and rewrite Eq. (1) as

where B i = { µ k |∥ µ k -pro i ∥ 2 ≤ ∥ µ k -pro j ∥ 2 , j = 1 ...N pro } , indicating the set of features that are assigned to prototype pro i by least Euclidean distance. Please see Appendix for detailed derivations.

To find the optimal prototypes { pro i } N pro i =1 is to solve the 'Term Pro' in Eq. (3), which can be viewed as a clustering task to seek N pro clustering centers, such that the mean distance between all pairs of the sampled features to their closest prototypes is minimized. We adopt the K-means algorithm updated with Exponential Moving Average (EMA) to enhance the training efficiency. Overall, at the t th training iteration, pro i is estimated as

where α is the smoothness hyperparameter and all the prototypes are initialized with the Gaussian random initialization before the training. After we obtain the new prototypes in an iteration, we take them back to Eq. (3) to get our loss for training the network:

where the first term is the reconstruction loss and the last term is the clustering loss for encouraging each sampled feature to stay close to its associated nearest prototype.

3.3 Scene Reconstruction with Geometric Prototypical VAE (GP-VAE)

The detection framework starts by extracting features from the input scene. Here, we incorporate GP-VAE to enhance the feature quality. We reconstruct the whole point cloud scene with GP-VAE, and obtain geometric-informative prototypes denoted as { geo i } N geo i =1 . These prototypes represent basic geometric components of common real-world 3D objects.

Please refer to Fig. 2 for GP-V AE. We use the PointNet++ [56] as our encoder f scene . It takes a raw point cloud x ∈ R N x × 3 as input and outputs parameters { µ i , σ i } N z i =1 to construct the posterior normal distributions N ( µ i , σ i ) , i = 1 ...N z . We sample { z i } N z i =1 from the learned distributions and build a decoder g scene to reconstruct the original point cloud scene. For g scene , we adopt the architecture

of FoldingNet [57], which transforms a plane into a 3D object surface. Specifically, we use furthest point sampling (FPS) to obtain N p points on a 0 . 4 m × 0 . 4 m plane B ∈ R N p × 2 . For each z i , we repeat the feature N p times then concatenate them with plane B , followed by two 2-layer MLPs to generate p i ∈ R N p × 3 . We use Eq. (4) to obtain the geometric prototypes { geo i } N geo i =1 . We combine the N z predicted point clouds to construct the scene-level result p ∈ R N z N p × 3 , then use Eq. (5) to supervise the scene-level reconstruction, where the ground truth is the input point cloud x .

Instead of directly using the sampled features { z i } N z i =1 to cluster the per-object features, we calibrate { z i } N z i =1 with the geometric prototypes via a cross attention layer. This calibration enriches the information diversity of each feature by not only its nearest prototype but also other geometric prototypes. For brevity, we use geo ∈ R N geo × d as the united representation of the prototypes:

where Q , K , and V are three MLPs to generate the query, key, and value in the attention framework. In this way, we can obtain robust embedding features { z ′ i } N z i =1 .

3.4 Object Reconstruction with Class-specific Prototypical VAE

Given the features { z ′ i } N z i =1 generated in Sec. 3.3, we use CP-VAE to reconstruct each object and obtain class-specific prototypes as shown in Fig. 1(c). We build a voting-based [58] encoder f object . For detection purpose, we need per-object features { ˆ z i } N ins i =1 , where N ins is the number of the predicted instances. Therefore, we use a 1-layer MLP to predict the feature offsets and point offsets from { z ′ i } N z i =1 , then adopt FPS on the shifted points to obtain N ins votes. For each vote, we collect a maximum number of 16 shifted features within 0.3m from the vote, then use max pooling on these features to generate the associated object-level feature ˆ z i . Another built MLP is then fed with { ˆ z i } N ins i =1 to output parameters { µ i , σ i } N ins i =1 , which can be used to construct the posterior truncated normal distributions N ( µ i , σ i ) where i = 1 ...N ins . We sample { w i } N ins i =1 and build a FoldingNet decoder g object (Sec. 3.3) for object-level reconstruction. For each w i , g object outputs an unique 3D object p i .

According to the sampling and grouping mechanisms in feature extraction, we reversely map each w i to a set of corresponding points in x (we will detail this operation in the Appendix), which constructs our reconstruction target ˆ p i . Therefore, we modify the reconstruction term of Eq. (5) to

where B i = { µ k |∥ µ k -cls i ∥ 2 ≤ ∥ µ k -cls j ∥ 2 , j = 1 ...N cls } . Importantly, we further modify the prototype update formula in Eq. (4) to customize for class-specific prototype learning. As discussed before, we can reversely map each µ i to a set of input points. During the training, we use this point set to find the nearest ground-truth instance by the least center Euclidean distance, then assign the corresponding category label to µ i . We collect features of the same category to update the class-specific prototypes accordingly. Denoting l ( · ) as the operation to get the category index for each feature, to obtain precise and convincing class-specific prototypes, Eq. (4) is rewritten as

Similar to Sec. 3.3, we refine the sample features { w i } N ins i =1 with the prototypes by using a crossattention augmentation, where we adopt cls ∈ R N cls × d as the united representation of the prototypes:

4 Experiment

4.1 Experiment Setup

Weconduct experiments on two benchmarks FS-ScanNet and FS-SUNRGBD [10]. The FS-ScanNet dataset consists of 18 object categories and 1,513 point cloud scenes in total. This benchmark has two

Prototypical

Votenet

Ours

different base/novel splits, each including 6 novel classes and 12 base classes. The FS-SUNRGBD dataset contains 5,000 point cloud scenes, covering 10 object categories. It has only one base/novel split, including 4 novel classes and 6 base classes. Following the standard 3D object detection protocol, comparison results on both datasets are quantitatively evaluated on mean Average Precision (mAP) under IoU thresholds 0.25 and 0.5, denoted as AP 25 and AP 50 . More details on the splits of both datasets, the network architecture, and the training scheme can be found in the Appendix.

4.2 Comparison Results

Comparison Approaches. We quantitatively compare our method with VoteNet [58] baseline and five recent methods that utilize different FSL techniques to tackle FS3D. For the baseline, we train the VoteNet detector and then directly infer its performance on the novel test data without FSL. The other five recent methods are (i) Generalized FS3D [59], an incremental fine-tuning method with a sample adaptive balance loss built upon TFA [41]; (ii) PointContrast-VoteNet [60], which obtains generic and transferable features by self-supervised pre-training; (iii) Fractal-VoteNet [61], which adopts a pre-training similar to (ii), whereas it learns features from fractal geometry 3D patterns (this concept can be viewed as a simplified geometric prototype without any regulation principle); (iv) Meta-Det3D [62], which uses a meta detector [63] to output class-specific re-weighted vectors to refine RoI features (this is similar to our CP-V AE but their class vectors are negatively affected by data imbalance); (v) Prototypical VoteNet [10], the most recent SOTA approach that groups the latent features to obtain prototypes.

FS-ScanNet Benchmark. Tab. 1 summarizes our quantitative comparison results on FS-ScanNet . Despite their achievements in FS2D, the improvements of fine-tuning and pre-training [59, 60, 61] over the baseline are marginal for FS3D. They rely on a robust feature space pre-trained with largescale datasets (ImageNet1K [64] for 2D), which are hard to obtain and annotate for the 3D scenes.

As we discussed before, [61] and [62] use the concept of prototypes but their learning schemes are less in-depth. Our proposed P-VAE consistently outperforms all the above methods, thanks to the proposed distribution-based prototype learning scheme adopted in GP-VAE and CP-VAE. Fig. 3 shows convincing qualitative results compared with SOTA [10], our predictions (green) better overlap with the ground truths (red), while [10] may produce more false positives and box errors.

FS-SUNRGBD Benchmark. Tab. 2 summarizes our quantitative comparison results on FSSUNGRBD . Compared with FS-ScanNet , FS-SUNGRBD is more challenging with higher clutter levels and more severe object diversity. Under such cases, our method still consistently surpasses SOTA [10] in all the few-shot settings, especially on the more strict AP 50 metric.

Comparison on Base Classes. We further test only on base classes to see if our prototype learning scheme for novel samples may impair the features learned on the base samples. Please see Tab. 3. VoteNet (Baseline) is specifically designed for base classes. Compared with it, Prototypical VoteNet (Proto) suffers from

consistent precision drops, which means the extracted knowledge from the novel samples is less incompatible with one of the base classes, thus the base feature space is hurt by those chaotic signals. Our P-VAE is able to better preserve the base feature structures and improves the origin detection performance on all the benchmarks. This observation demonstrates that our novel and base features are better aligned and can be simultaneously enhanced by our distribution-based prototype learning.

4.3 Ablation Studies and Detailed Analysis

Module Ablation. We ablate the major components of our framework on FS-ScanNet split-1 with K = 3 , 5 . From Tab. 4, we can see that each component contributes to improving the performance and our full pipeline attains the highest ratings for all settings. We can observe that individual usage of CP-VAE works less effectively than individual usage of GP-V AE. The reason is that we incorporate GP-VAE in the scene-level reconstruction, which is prior to the object-level reconstruction by using CP-VAE. If we omit GP-VAE, the features from the scene encoder are not sufficiently representative to support the sequential clustering, thus it could hurt prototype learning in CP-V AE.

Prototype Learning Schemes. Based on the Prototypical VoteNet [10], denoted as Proto in Tab. 5, we optimize the prototype learning scheme to AE and VAE. AE introduces a reconstruction task with an encoder-decoder architecture, while the latent features to optimize prototypes are directly learned from the encoder instead of sampled from the distributions. AE can be viewed as non-probabilistic prototypical learning. VAE is built upon AE by further constructing a probabilistic latent space to enable us to sample diverse features. It can be viewed as probabilistic prototypical learning. As Tab. 5 reports, we conduct independent quantitative comparisons for GP-VAE and CP-VAE on FS-ScanNet split-1 with K = 3 , 5 . The results of AE over Proto show that incorporating a reconstruction task helps preserve the geometric information within the latent embeddings. VAE further consistently outperforms AE on all the metrics for both GP-V AE and CP-VAE, demonstrating that the probabilistic modeling can significantly boost feature learning for FS3D.

Prototype Quality. Thanks to the reconstruction tasks, we can visualize our prototypes as point clouds. Fig. 4(left) shows some of the geometric-informative prototypes learned by the GP-VAE; more results can be found in the Appendix. From them we can see the diversity and distinctiveness of the prototypes, each exhibiting a certain geometric pattern. In Fig. 4(right), we illustrate our class-specific prototypes learned by the CP-V AE. These results show certain class-level abstractions of the general object appearance.

Sofa

Chair

Sink

Table

Picture

Cabinet

Bed

Refrigerator

BaseChair

BaseTable&Desk

NovelSink

NovelBed

Prototype-ComposedScenes

Prototype Effects. After generating the prototypes with a well-designed prototype learning scheme, our final question is: do these prototypes help representation learning?

For geometric-informative prototypes, we illustrate how these prototypes help to construct an object or a whole scene in Fig. 5, where each color represents a geometric-informative prototype. We can observe that each object, regardless of its base/novel category label, is composed of multiple prototypes with semantic correspondence patterns. For example, the red prototype indicates an open semicircle shape, appearing not only in the base chair class (consistently at the seat region) but also in the novel sink class (consistently at

the center region). Results demonstrate that these prototypes are robust geometric patterns that help enhance the generalization ability for both base and novel classes, so they can provide transferable and consistent 3D hints for scene-level feature encoding. Please see Tab. 6 for quantitative and Appendix for qualitative ablations on varying numbers of prototypes, denoting N geo in Sec. 3.3.

For class-specific prototypes, in Fig. 6, we use t-SNE visualization to show the high-level object embeddings { w i } N ins i =1 of five common categories. From left to right: without class-specific prototypes, CP-AE (defined in Tab. 5), and CP-VAE. Compared with the vanilla feature space, where the features are disordered and chaotic, incorporating class prototypes leads to a more compact latent space, such that each prototype can be more separated and the sampled features are better organized by clustering.

Pluggable P-VAE. Besides the VoteNet baseline, we extend multiple point cloud detection frameworks with our P-VAE. Please refer to Appendix for quantitative results that demonstrate the effectiveness of our P-VAE as a plug-in module for various networks to better adapt FS3D.

bed

bed

bed

chair

chair

chair

table

table

table

refrigerator

refrigerator

refrigerator

cabinet

cabinet

cabinet

5 Conclusions

To tackle few-shot 3D point cloud object detection (FS3D), we design a specific V AE for prototype learning, named Prototypical VAE (P-VAE). P-VAE constructs a probabilistic latent space that allows us to sample diverse and distinctive features, such that we can learn more representative prototypes even if the novel samples are limited. To adopt P-VAE for FS3D, we propose the Geometricinformative GP-VAE and the Class-specific CP-VAE based on our common observations in the real world. Experimental results demonstrate the effectiveness of P-V AE, and further prototype analysis shows its potential value in other prototype-based few-shot 3D perception tasks.

References

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning , pages 1126-1135, 2017.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems , 30, 2017.

Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition , pages 1199-1208, 2018.

Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. arXiv preprint arXiv:1707.03141 , 2017.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems , 29, 2016.

Sercan Ö Arik and Tomas Pfister. Protoattend: Attention-based prototypical learning. The Journal of Machine Learning Research , 21(1):8691-8725, 2020.

Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966 , 2020.

Shuai Lin, Chen Liu, Pan Zhou, Zi-Yuan Hu, Shuojia Wang, Ruihui Zhao, Yefeng Zheng, Liang Lin, Eric Xing, and Xiaodan Liang. Prototypical graph contrastive learning. IEEE Trans Neural Networks and learning systems , 2022.

Junhua Wang and Yongping Zhai. Prototypical siamese networks for few-shot learning. In International Conference on Electronics Information and Emergency Communication , pages 178-181, 2020.

Shizhen Zhao and Xiaojuan Qi. Prototypical votenet for few-shot 3d point cloud object detection. arXiv preprint arXiv:2210.05593 , 2022.

Na Zhao, Tat-Seng Chua, and Gim Hee Lee. Few-shot 3D point cloud semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition , pages 8873-8882, 2021.

Aming Wu, Yahong Han, Linchao Zhu, and Yi Yang. Universal-prototype enhancing for fewshot object detection. In IEEE International Conference on Computer Vision , pages 9567-9576, 2021.

Gong Cheng, Bowei Yan, Peizhen Shi, Ke Li, Xiwen Yao, Lei Guo, and Junwei Han. Prototypecnn for few-shot object detection in remote sensing images. IEEE Transactions on Geoscience and Remote Sensing , 60:1-10, 2021.

Huifang Li, Yidong Li, Yuanzhouhan Cao, Yushan Han, Yi Jin, and Yunchao Wei. Weakly supervised object detection with class prototypical network. IEEE Transactions on Multimedia , 2022.

Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmentation: A prototype view. In IEEE Conference on Computer Vision and Pattern Recognition , pages 2582-2593, 2022.

Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, Yong Wang, and Fang Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition , pages 12414-12424, 2021.

Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In IEEE International Conference on Computer Vision , pages 9197-9206, 2019.

Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. In European Conference on Computer Vision , pages 763-778, 2020.

Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He. Part-aware prototype network for few-shot semantic segmentation. In European Conference on Computer Vision , pages 142-158, 2020.

Hui Zhang and Henghui Ding. Prototypical matching and open set rejection for zero-shot semantic segmentation. In IEEE International Conference on Computer Vision , pages 69746983, 2021.

Zhengkai Jiang, Yuxi Li, Ceyuan Yang, Peng Gao, Yabiao Wang, Ying Tai, and Chengjie Wang. Prototypical contrast adaptation for domain adaptive semantic segmentation. In European Conference on Computer Vision , pages 36-54, 2022.

Dan Andrei Ganea, Bas Boom, and Ronald Poppe. Incremental few-shot instance segmentation. In IEEE Conference on Computer Vision and Pattern Recognition , pages 1185-1194, 2021.

Jinlu Liu, Liang Song, and Yongqiang Qin. Prototype rectification for few-shot learning. In European Conference on Computer Vision , pages 741-756, 2020.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Jrg Bornschein, A. Mnih, D. Zoran, and D. J. Rezende. Variational memory addressing in generative models. In Advances in Neural Information Processing Systems , volume 30, 2017.

Sajad Norouzi, David J Fleet, and Mohammad Norouzi. Exemplar vae: Linking generative models, nearest neighbor retrieval, and data augmentation. In Advances in Neural Information Processing Systems , volume 33, pages 8753-8764, 2020.

Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349 , 2015.

Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems , volume 29, 2016.

Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv:1903.05789 , 2019.

Andrea Asperti. Variance loss in variational autoencoders. In Machine Learning, Optimization, and Data Science , pages 297-308, 2020.

Serena Yeung, Anitha Kannan, Yann Dauphin, and Li Fei-Fei. Tackling over-pruning in variational autoencoders. arXiv preprint arXiv:1706.03643 , 2017.

Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. arXiv preprint arXiv:1901.03416 , 2019.

Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262 , 2017.

Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein autoencoders. arXiv preprint arXiv:1711.01558 , 2017.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems , volume 30, 2017.

Tejas Anvekar, Ramesh Ashok Tabib, Dikshit Hegde, and Uma Mudengudi. Vg-vae: A venatus geometry point-cloud variational auto-encoder. In IEEE Conference on Computer Vision and Pattern Recognition , pages 2978-2985, 2022.

Zhizhong Han, Xiyang Wang, Yu-Shen Liu, and Matthias Zwicker. Multi-angle point cloudvae: Unsupervised feature learning for 3d point clouds from multiple angles by joint selfreconstruction and half-to-half prediction. In IEEE International Conference on Computer Vision , pages 10441-10450, 2019.

Shidi Li, Miaomiao Liu, and Christian Walder. Editvae: Unsupervised parts-aware controllable 3d point cloud shape generation. In AAAI Conference on Artificial Intelligence , volume 36, pages 1386-1394, 2022.

Pulak Purkait, Christopher Zach, and Ian Reid. Sg-vae: Scene grammar variational autoencoder to generate new indoor scenes. In European Conference on Computer Vision , pages 155-171, 2020.

Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimenez Rezende. Nerf-vae: A geometry aware 3d scene generative model. In International Conference on Machine Learning , pages 5742-5752, 2021.

Xin Wang, Thomas E Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. arXiv preprint arXiv:2003.06957 , 2020.

Jiaxi Wu, Songtao Liu, Di Huang, and Yunhong Wang. Multi-scale positive sample refinement for few-shot object detection. In European Conference on Computer Vision , pages 456-472, 2020.

Weilin Zhang and Yu-Xiong Wang. Hallucination improves few-shot object detection. In IEEE Conference on Computer Vision and Pattern Recognition , pages 13008-13017, 2021.

Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang. Fsce: Few-shot object detection via contrastive proposal encoding. In IEEE Conference on Computer Vision and Pattern Recognition , pages 7352-7362, 2021.

Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Fewshot object detection via feature reweighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8420-8429, 2019.

Guangxing Han, Shiyuan Huang, Jiawei Ma, Yicheng He, and Shih-Fu Chang. Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment. In AAAI Conference on Artificial Intelligence , volume 36, pages 780-789, 2022.

Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu-Wing Tai. Few-shot object detection with attentionrpn and multi-relation detector. In IEEE Conference on Computer Vision and Pattern Recognition , pages 4013-4022, 2020.

Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin. Meta R-CNN: Towards general solver for instance-level low-shot learning. In IEEE International Conference on Computer Vision , pages 9577-9586, 2019.

Guangxing Han, Yicheng He, Shiyuan Huang, Jiawei Ma, and Shih-Fu Chang. Query adaptive few-shot object detection with heterogeneous graph convolutional networks. In IEEE International Conference on Computer Vision , pages 3263-3272, 2021.

Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially-aware few-shot transfer. In Advances in Neural Information Processing Systems , volume 33, pages 21981-21993, 2020.

Tung-I Chen, Yueh-Cheng Liu, Hung-Ting Su, Yu-Cheng Chang, Yu-Hsiang Lin, Jia-Fong Yeh, Wen-Chin Chen, and Winston Hsu. Dual-awareness attention for few-shot object detection. IEEE Transactions on Multimedia , 2021.

Ding-Jie Chen, He-Yen Hsieh, and Tyng-Luh Liu. Adaptive image transformer for one-shot object detection. In IEEE Conference on Computer Vision and Pattern Recognition , pages 12247-12256, 2021.

Guangxing Han, Jiawei Ma, Shiyuan Huang, Long Chen, and Shih-Fu Chang. Few-shot object detection with fully cross-transformer. In IEEE Conference on Computer Vision and Pattern Recognition , pages 5321-5330, 2022.

Zhaoxin Fan, Hongyan Liu, Jun He, Qi Sun, and Xiaoyong Du. A graph-based one-shot learning method for point cloud recognition. In Computer Graphics Forum , volume 39, pages 313-323, 2020.

Ali Cheraghian, Shafin Rahman, Dylan Campbell, and Lars Petersson. Transductive zero-shot learning for 3D point cloud classification. In IEEE Winter Conference on Applications of Computer Vision , pages 923-933, 2020.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems , volume 30, 2017.

Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In IEEE Conference on Computer Vision and Pattern Recognition , pages 206-215, 2018.

Charles R. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas. Deep hough voting for 3d object detection in point clouds. In IEEE International Conference on Computer Vision , 2019.

Jiawei Liu, Xingping Dong, Sanyuan Zhao, and Jianbing Shen. Generalized few-shot 3d object detection of lidar point cloud for autonomous driving. arXiv preprint arXiv:2302.03914 , 2023.

Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European Conference on Computer Vision , pages 574-591, 2020.

Ryosuke Yamada, Hirokatsu Kataoka, Naoya Chiba, Yukiyasu Domae, and Tetsuya Ogata. Point cloud pre-training with natural 3d structures. In IEEE Conference on Computer Vision and Pattern Recognition , pages 21283-21293, 2022.

Shuaihang Yuan, Xiang Li, Hao Huang, and Yi Fang. Meta-det3d: Learn to learn few-shot 3d object detection. In Asian Conference on Computer Vision , pages 1761-1776, 2022.

Xiongwei Wu, Doyen Sahoo, and Steven Hoi. Meta-rcnn: Meta learning for few-shot object detection. In ACM International Conference on Multimedia , pages 1679-1687, 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems , 2017.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 652-660, 2017.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In IEEE Conference on Computer Vision and Pattern Recognition , 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach for few-shot 3D object detection by combining prototype learning and variational autoencoders. To address the weak geometry regularization and data imbalance issues of the existing methods, it proposes a novel VAE specifically designed for prototype learning named Prototypical VAE (P-VAE). Moreover, it extends P-VAE with geometric-informative prototypes and class-specific prototypes to enhance the object detection performance in the few-shot regime. Experimental results on several benchmarks validate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper proposes two prototypical VAE based on geometric-informative and class-specific prototypes, which helps solve the weak geometry regularization and data imbalance issues encountered by the existing few-show 3D object detection methods based on prototype learning. This combination of prototype learning and VAE is novel and effective.\n\n2. Experimental results showed strong support for the effectiveness of the proposed method in the few-shot regime. In particular, it shows consistent improvement over baseline methods on various settings of different benchmarks. \n\n3. The paper is well written, with easy-to-follow equations and proper notations.\n\nWEAKNESSES:\n1. In order to detect objects from the scene point clouds, a number of instances should be predicted. However, it is not clear how to obtain the number of the predicted instances $N_{ins}$ from $N_z$ features $z'_i$.\n\n2. There is no ablation study of the two feature calibration described in Eq-6 and Eq-9, how much do they contribute to the final performance improvement? \n\n3. What are the object detection heads in Figure-2? How are they trained exactly?\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nThe limitations of the proposed method and potential solutions are discussed in the final section of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-24"}

comments: SUMMARY:
This paper studies a challenge task called FS3D. They first presents that the previous work on FS3D lacks fine-level supervision, as the intermediate features are simply averaged to update the prototypes, which are then used to augment features for sequential detection. In order to solve this problem, they leverage Prototypical Variational Autoencoder to conduct regularization on both local and global levels.

STRENGTHS:
1. The paper is well written and easy to follow since the motivation is clear.
2. The experimental improvement is significant.

WEAKNESSES:
1. The related works on FSL for 3D Point Cloud is insufficient. 
2. why does Prototypical Variational Autoencoder work in this task ? Have you tried other models for reconstruction ?

QUESTIONS:
1. Please see the weakness
2. Can the proposed method be used to other few-shot learning tasks in point cloud, such as segmentation ?  
My main concern is that why does author choose Prototypical Variational Autoencoder as the reconstruction model. Does it have outstanding property for FS3D ?  It is the key for the acceptance of this paper. 
if authors could address my concerns, I am willing to increase the score.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach to enhance Few-Shot 3D Point Cloud Detection (FS3D) through prototype learning with VAEs. The authors leverage VAEs to learn prototypes represented by GMM-like distributions. Two VAEs are specifically designed to preserve geometric information and refine instance features. The effectiveness of the approach is validated through various experiments.\n\nSTRENGTHS:\n1.The paper introduces an approach utilizing VAEs to learn geometric and class-specific prototypes with improved performance on FS3D tasks.\n\n 2.The effectiveness of the modules and prototypes is demonstrated through various experiments.\n\n 3.The writing is easy to follow.\n\nWEAKNESSES:\n1.The authors do not provide a comprehensive and detailed discussion of their motivations for the proposed approach. This should include a clear identification of the core issues, a thorough evaluation for\u0010 the limitations of previous works in addressing this problem, and a convincing explanation of how incorporating VAEs can effectively tackle these limitations. It would greatly enhance the paper's credibility if qualitative and quantitative results were presented to support their motivations and demonstrate the superiority of the proposed method.\n\n 2.The sensitivity of the number of geometric-informative prototypes as observed in Table 6 raises concerns about the practical applicability of the proposed approach. It would be beneficial if the authors discussed the implications of this sensitivity and provided insights on how to determine an optimal or adaptive number of Geo-proto for real-world applications.\n\nQUESTIONS:\n1.The paper does not explain the reason why prototypes learned by previous methods lose substantial 3D information and become less geometric-informative.\n\n 2.The motivation for using VAEs is not fully stated.\n\n 3.Are there any empirical results to demonstrate the underrepresentation / overfitting problem of prototypes?\n\n 4.Why can GMM-like distribution handle the overfitting problem of the latent space?\n\nLIMITATIONS:\n1.The authors have addressed the limitations that data-rooted problem will lead the representative prototypes to distort and perturb the latent space negatively.\n\n 2.Additionally, the proposed approach depends on the number of prototypes. The sensitivity raises concerns about the practical applicability of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper introduces Prototypical Variational Autoencoder (P-VAE) for Few-Shot 3D Point Cloud Object Detection. It tackles the preservation of geometric information and data imbalance through learning distribution parameters. The authors propose two extensions, GP-VAE and CP-VAE, focusing on geometry and class specificity. Experiments demonstrate improved performance over state-of-the-art methods.

STRENGTHS:
1. The P-VAE is innovative, focusing on learning distribution parameters instead of features, which is useful for few-shot learning.

2. The encoder-decoder architecture effectively preserves geometric information, critical for 3D detection.

3. The paper includes a robust experimental evaluation, showing performance gains and insights into prototype contributions.

WEAKNESSES:
1. The paper does not include experiments to assess the effectiveness and generalization of P-VAE across different domains on few shot learning, despite P-VAE being one of the main contributions.

2. The observed improvements in the main experiments are relatively marginal, particularly in regard to the evaluation metric AP25.

QUESTIONS:
None

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06