SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} = \frac {1}{N _ {i n s}} \sum_ {i = 1} ^ {N _ {i n s}} \mathrm {C D} \left(p _ {i}, \hat {p} _ {i}\right) + \frac {1}{2 N _ {c l s}} \left(\sum_ {i = 1} ^ {N _ {c l s}} \left[ \| \sigma_ {i} \| _ {2} ^ {2} - \| \log \sigma_ {i} ^ {2} \| _ {1} \right] + \sum_ {i = 1} ^ {N _ {c l s}} \sum_ {\mu_ {j} \in B _ {i}} \| c l s _ {i} - \mu_ {j} \| _ {2} ^ {2}\right), \tag {7}
$$
where  $B_{i} = \{\mu_{k}||\mu_{k} - cls_{i}\|_{2}\leq ||\mu_{k} - cls_{j}\|_{2}, j = 1\dots N_{cls}\}$ . Importantly, we further modify the prototype update formula in Eq. (4) to customize for class-specific prototype learning. As discussed before, we can reversely map each  $\mu_{i}$  to a set of input points. During the training, we use this point set to find the nearest ground-truth instance by the least center Euclidean distance, then assign the corresponding category label to  $\mu_{i}$ . We collect features of the same category to update the class-specific prototypes accordingly. Denoting  $l(\cdot)$  as the operation to get the category index for each feature, to obtain precise and convincing class-specific prototypes, Eq. (4) is rewritten as
$$
c l s _ {i} ^ {t + 1} \leftarrow \alpha c l s _ {i} ^ {t} + (1 - \alpha) \frac {\sum \mu_ {j} \in B _ {i}}{| B _ {i} |}, \quad B _ {i} = \left\{\mu_ {j} \mid l \left(\mu_ {j}\right) = i \right\}. \tag {8}
$$
Similar to Sec. 3.3, we refine the sample features  $\{w_{i}\}_{i = 1}^{N_{ins}}$  with the prototypes by using a cross-attention augmentation, where we adopt  $cls\in \mathbb{R}^{N_{cls}\times d}$  as the united representation of the prototypes:
$$
w _ {i} ^ {\prime} = \operatorname {S o f t m a x} \left(\frac {\mathbf {Q} \left(w _ {i}\right) \mathbf {K} ^ {\top} (c l s)}{\sqrt {d}}\right) \mathbf {V} (c l s). \tag {9}
$$
We conduct experiments on two benchmarks FS-ScanNet and FS-SUNRGBD [10]. The FS-ScanNet dataset consists of 18 object categories and 1,513 point cloud scenes in total. This benchmark has two
Prototypical  
Votenet
Figure 3: Example detection results by our method and Prototypical VoteNet [10] (SOTA) on FS-ScanNet split-1 with  $K = 5$ . The red bounding boxes are the ground truths, while the green bounding boxes are the predictions.
different base/novel splits, each including 6 novel classes and 12 base classes. The FS-SUNRGBD dataset contains 5,000 point cloud scenes, covering 10 object categories. It has only one base/novel split, including 4 novel classes and 6 base classes. Following the standard 3D object detection protocol, comparison results on both datasets are quantitatively evaluated on mean Average Precision (mAP) under IoU thresholds 0.25 and 0.5, denoted as  $\mathrm{AP}_{25}$  and  $\mathrm{AP}_{50}$ . More details on the splits of both datasets, the network architecture, and the training scheme can be found in the Appendix.
Method	split-1	split-2
1-shot	3-shot	5-shot	1-shot	3-shot	5-shot
\(AP_{25}\)	\(AP_{50}\)	\(AP_{25}\)	\(AP_{50}\)	\(AP_{25}\)	\(AP_{50}\)	\(AP_{25}\)	\(AP_{50}\)	\(AP_{25}\)	\(AP_{50}\)	\(AP_{25}\)	\(AP_{50}\)
Beseline [58]	11.72	8.02	21.13	9.57	28.63	15.69	8.79	1.71	18.19	5.52	22.68	11.64
Generalized FS3D [59]	12.03	8.19	24.90	10.26	29.29	16.67	9.19	1.87	19.41	6.80	25.18	12.74
PointContrast-VoteNet [60]	12.59	8.52	20.12	11.16	25.83	15.49	9.55	1.97	18.44	5.23	20.06	10.19
Fractal-VoteNet [61]	11.81	7.57	21.38	10.11	24.66	14.73	9.16	1.68	15.65	4.88	20.35	10.26
Meta-Det3D [62]	10.28	4.03	23.42	10.64	25.65	13.88	5.21	1.32	15.44	4.37	22.13	7.09
Prototypical VoteNet [10]	15.34	8.25	31.25	16.01	32.25	19.52	11.01	2.21	21.14	8.39	28.52	12.35
Ours	16.00	10.22	31.60	19.37	32.84	22.39	12.66	4.15	21.27	10.09	31.70	14.43
Comparison Approaches. We quantitatively compare our method with VoteNet [58] baseline and five recent methods that utilize different FSL techniques to tackle FS3D. For the baseline, we train the VoteNet detector and then directly infer its performance on the novel test data without FSL. The other five recent methods are (i) Generalized FS3D [59], an incremental fine-tuning method with a sample adaptive balance loss built upon TFA [41]; (ii) PointContrast-VoteNet [60], which obtains generic and transferable features by self-supervised pre-training; (iii) Fractal-VoteNet [61], which adopts a pre-training similar to (ii), whereas it learns features from fractal geometry 3D patterns (this concept can be viewed as a simplified geometric prototype without any regulation principle); (iv) Meta-Det3D [62], which uses a meta detector [63] to output class-specific re-weighted vectors to refine RoI features (this is similar to our CP-VAE but their class vectors are negatively affected by data imbalance); (v) Prototypical VoteNet [10], the most recent SOTA approach that groups the latent features to obtain prototypes.
Table 1: Comparing the FS3D performance of our method against existing works on FS-ScanNet.
Method	1-shot	2-shot	3-shot	4-shot	5-shot
\(AP_{25}\)	\(AP_{50}\)	\(AP_{25}\)	\(AP_{50}\)	\(AP_{25}\)	\(AP_{50}\)	\(AP_{25}\)	\(AP_{50}\)	\(AP_{25}\)	\(AP_{50}\)
Baseline [58]	5.46	0.22	6.52	0.77	13.73	2.20	20.47	4.50	22.99	5.90
Generalized FS3D [59]	6.81	1.58	12.21	2.02	17.52	4.69	22.12	5.97	22.84	6.76
PointContrast-VoteNet [60]	7.03	1.17	8.16	2.33	20.32	4.19	21.13	4.49	21.03	6.71
Fractal-VoteNet [61]	7.54	1.39	9.16	3.01	21.08	4.25	21.23	5.68	22.01	6.77
Meta-Det3D [62]	6.77	0.73	8.29	1.21	15.37	2.99	19.60	4.67	24.22	5.68
Prototypical VoteNet [10]	12.39	1.52	14.54	3.05	21.51	6.13	24.78	7.17	29.95	8.16
Ours	14.36	2.42	22.28	4.30	27.70	8.73	31.55	13.84	33.21	13.98
Table 2: Comparing the FS3D performance of our method against existing works on FS-SUNRGBD.
FS-ScanNet Benchmark. Tab. 1 summarizes our quantitative comparison results on FS-ScanNet. Despite their achievements in FS2D, the improvements of fine-tuning and pre-training [59, 60, 61] over the baseline are marginal for FS3D. They rely on a robust feature space pre-trained with large-scale datasets (ImageNet1K [64] for 2D), which are hard to obtain and annotate for the 3D scenes.
As we discussed before, [61] and [62] use the concept of prototypes but their learning schemes are less in-depth. Our proposed P-VAE consistently outperforms all the above methods, thanks to the proposed distribution-based prototype learning scheme adopted in GP-VAE and CP-VAE. Fig. 3 shows convincing qualitative results compared with SOTA [10], our predictions (green) better overlap with the ground truths (red), while [10] may produce more false positives and box errors.
FS-SUNRGBD Benchmark. Tab. 2 summarizes our quantitative comparison results on FS-SUNGRBD. Compared with FS-ScanNet, FS-SUNGRBD is more challenging with higher clutter levels and more severe object diversity. Under such cases, our method still consistently surpasses SOTA [10] in all the few-shot settings, especially on the more strict  $\mathrm{AP}_{50}$  metric.
Comparison on Base Classes. We further test only on base classes to see if our prototype learning scheme for novel samples may impair the features learned on the base samples. Please see Tab. 3. VoteNet (Baseline) is specifically designed for base classes. Compared with it, Prototypical VoteNet (Proto) suffers from
Method	FS-ScanNet-1	FS-ScanNet-2	FS-SUNRGBD
AP25	AP50	AP25	AP50	AP25	AP50
Baseline [58]	57.96	32.60	54.63	35.76	47.77	26.78
Proto [10]	53.83	28.20	51.22	32.41	46.07	25.26
Ours	58.16	32.69	56.05	38.77	47.92	27.75
consistent precision drops, which means the extracted knowledge from the novel samples is less incompatible with one of the base classes, thus the base feature space is hurt by those chaotic signals. Our P-VAE is able to better preserve the base feature structures and improves the origin detection performance on all the benchmarks. This observation demonstrates that our novel and base features are better aligned and can be simultaneously enhanced by our distribution-based prototype learning.
4.3 Ablation Studies and Detailed Analysis
Table 3: Comparing base-class results on three datasets.
Method	3-shot	5-shot
\( AP_{25} \)	\( AP_{50} \)	\( AP_{25} \)	\( AP_{50} \)
Baseline	21.13	9.57	28.63	15.69
+ GP-VAE	29.69	15.18	33.48	22.18
+ CP-VAE	27.11	18.88	31.92	21.91
+ GP-VAE & CP-VAE	31.60	19.37	32.84	22.39
Table 4: Ablating our major components.
Module	PL scheme	3-shot	5-shot
AP25	AP50	AP25	AP50
GP-VAE	proto	25.71	13.57	29.04	17.17
AE	29.57	14.48	33.14	19.16
VAE	29.69	15.18	33.48	22.18
CP-VAE	proto	26.14	17.43	31.13	19.16
AE	26.56	17.98	31.61	21.45
VAE	27.11	18.88	31.92	21.91
Table 5: Analyzing different PL schemes.
Module Ablation. We ablate the major components of our framework on FS-ScanNet split-1 with  $K = 3,5$ . From Tab. 4, we can see that each component contributes to improving the performance and our full pipeline attains the highest ratings for all settings. We can observe that individual usage of CP-VAE works less effectively than individual usage of GP-VAE. The reason is that we incorporate GP-VAE in the scene-level reconstruction, which is prior to the object-level reconstruction by using CP-VAE. If we omit GP-VAE, the features from the scene encoder are not sufficiently representative to support the sequential clustering, thus it could hurt prototype learning in CP-VAE.
Prototype Learning Schemes. Based on the Prototypical VoteNet [10], denoted as Proto in Tab. 5, we optimize the prototype learning scheme to AE and VAE. AE introduces a reconstruction task with an encoder-decoder architecture, while the latent features to optimize prototypes are directly learned from the encoder instead of sampled from the distributions. AE can be viewed as non-probabilistic prototypical learning. VAE is built upon AE by further constructing a probabilistic latent space to enable us to sample diverse features. It can be viewed as probabilistic prototypical learning. As Tab. 5 reports, we conduct independent quantitative comparisons for GP-VAE and CP-VAE on FS-ScanNet split-1 with  $K = 3, 5$ . The results of AE over Proto show that incorporating a reconstruction task helps preserve the geometric information within the latent embeddings. VAE further consistently outperforms AE on all the metrics for both GP-VAE and CP-VAE, demonstrating that the probabilistic modeling can significantly boost feature learning for FS3D.
Prototype Quality. Thanks to the reconstruction tasks, we can visualize our prototypes as point clouds. Fig. 4(left) shows some of the geometric-informative prototypes learned by the GP-VAE; more results can be found in the Appendix. From them we can see the diversity and distinctiveness of the prototypes, each exhibiting a certain geometric pattern. In Fig. 4(right), we illustrate our class-specific prototypes learned by the CP-VAE. These results show certain class-level abstractions of the general object appearance.
Figure 4: Left: reconstructed examples of learned geometric-informative prototypes, 49 points for each. Right: reconstructed examples of learned class-specific prototypes, 256 points for each.
Figure 5: Illustration of geometric-informative prototypes compositing objects and scenes. Each color represents a unique prototype learned from GP-VAE. These prototypes show strong inter-class physics consistency and intra-class semantic correspondences.
Prototype Effects. After generating the prototypes with a well-designed prototype learning scheme, our final question is: do these prototypes help representation learning?
For geometric-informative prototypes, we illustrate how these prototypes help to construct an object or a whole scene in Fig. 5, where each color represents a geometric-informative prototype. We can observe that each object, regardless of its base/novel category label, is composed of multiple prototypes with semantic correspondence patterns. For example, the red prototype indicates an open semicircle shape, appearing not only in the base chair class (consistently at the seat region) but also in the novel sink class (consistently at
the center region). Results demonstrate that these prototypes are robust geometric patterns that help enhance the generalization ability for both base and novel classes, so they can provide transferable and consistent 3D hints for scene-level feature encoding. Please see Tab. 6 for quantitative and Appendix for qualitative ablations on varying numbers of prototypes, denoting  $N_{geo}$  in Sec. 3.3.
For class-specific prototypes, in Fig. 6, we use t-SNE visualization to show the high-level object embeddings  $\{w_{i}\}_{i = 1}^{N_{ins}}$  of five common categories. From left to right: without class-specific prototypes, CP-AE (defined in Tab. 5), and CP-VAE. Compared with the vanilla feature space, where the features are disordered and chaotic, incorporating class prototypes leads to a more compact latent space, such that each prototype can be more separated and the sampled features are better organized by clustering.
Pluggable P-VAE. Besides the VoteNet baseline, we extend multiple point cloud detection frameworks with our P-VAE. Please refer to Appendix for quantitative results that demonstrate the effectiveness of our P-VAE as a plug-in module for various networks to better adapt FS3D.
Num. of Geo-Proto	AP25	AP50
Ngeo = 16	22.19	6.48
Ngeo = 32	28.59	14.52
Ngeo = 64	31.78	18.51
Ngeo = 128	32.84	22.39
Ngeo = 256	32.06	20.30
Table 6: Ablation for  ${N}_{geo}$  .
Figure 6: Visualize object-level embeddings with t-SNE.
To tackle few-shot 3D point cloud object detection (FS3D), we design a specific VAE for prototype learning, named Prototypical VAE (P-VAE). P-VAE constructs a probabilistic latent space that allows us to sample diverse and distinctive features, such that we can learn more representative prototypes even if the novel samples are limited. To adopt P-VAE for FS3D, we propose the Geometric-informative GP-VAE and the Class-specific CP-VAE based on our common observations in the real world. Experimental results demonstrate the effectiveness of P-VAE, and further prototype analysis shows its potential value in other prototype-based few-shot 3D perception tasks.