SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Visualization of predictions obtained by our Point-In-Context and their corresponding targets in different tasks, such as reconstruction, denoising, registration, and part segmentation. For part segmentation, we visualize the generated target together with the mapping back, both adding category-specific colors for better comparisons.
4.1 Baseline Methods and Training Details
To evaluate the performance of the proposed framework, we evaluate the two variants PIC-Sep and PIC-Cat on the dataset mentioned in Sec. 3.2, both equipped with the proposed Joint Sampling module. We compare them with the following related methods:
Point-BERT [44] is a masked auto-encoder. Like the settings in PIC-Cat, we concatenate 2 pairs of input-target point cloud tokens as a token sequence and input it to Point-BERT, which takes a token sequence as the input and converts them into discrete point tokens from a pre-trained dVAE [40] vocabulary of size 8192.
Task-specific Models. We selected three representative methods: PointNet [27], DGCNN [40], PCT [12], and ACT [9], and individually train them on four different tasks mentioned in Sec. 3.2. Additionally, we also designed task-specific heads for each task.
Multitask Models. For a fair comparison, we develop a multitask model based on PointNet [27], DGCNN [40], and PCT [12], respectively, which are capable of multitask learning. These models feature a shared backbone network and task-specific heads designed to address the needs of different tasks. This design allows simultaneous learning of all four tasks.
Point-MAE [26] is a masked auto-encoder. Unlike Point-BERT, Point-MAE directly rebuilds points in each local area. ACT [9], I2P-MAE [47], and ReCon [29] are recent SOTA methods that involve other modalities like image and text knowledge in the pre-training stage and enhance the performance on different tasks after fine-tuning the models. Similar to multitask models, we use a pre-trained encoder and combine it with different task heads for simultaneous training on the four tasks.
Copy Example is a baseline that utilizes the target point cloud of the prompt as its prediction.
We report extensive experimental results of various models on the dataset we proposed in Tab. 1. From where, we found that our PIC-Cat and PIC-Sep exhibit impressive results and are capable of adapting to different tasks after only one training, achieving state-of-the-art results in all four tasks amount multitask models. Besides, we visualize the in-context 3D inference results of PIC-Sep in
(a) Registration results on ModelNet40
(b) The generalization capacity
Figure 5: (a) Generalized to out-of-distribution data. We evaluate the registration task on the ModelNet40 [41] dataset, which is not present in the training set. The Chamfer Distance (x1000) is used as the evaluation metric. (b) Generalized to new tasks. Our model is able to rotate at any angle according to the example pairs and show the completion ability to the broken point cloud, which is not present in the training set. (c) Limitation of our model. PIC cannot reconstruct the detailed parts of complex point clouds very well.
Fig. 4, where our model can generate the corresponding predictions given the provided prompts among all four tasks, including reconstruction, denoising, registration, and part segmentation.
(c) Limitation
Comparison to Task-specific Models. Task-specific models including PointNet [27], DGCNN [40], and PCT [12] outperform PIC-Cat and PIC-Sep in most indicators on all tasks. Specifically, in the part segmentation task, PIC-Cat achieves better results than PointNet and DGCNN, though they are well-designed for segmentation. It needs to be clarified that the performance of our model is directly related to the choice of prompts. When the quality of the prompt is better, PIC-Sep and PIC-Cat can achieve much better results, and this will be demonstrated in the following section.
Generalization. Usually, in-context Learning has a certain degree of generalization ability, thus allowing the model to quickly adapt to different tasks. This also applies to our models. First, we test the proposed methods on out-of-distribution point clouds by conducting a registration evaluation on the ModelNet40 [41] dataset. As shown in Fig. 5 (a), our models, PIC-Sep and PIC-Cat, both exhibit superior performance on open-class tasks compared to supervised learning models specifically trained for the registration task. We compare it with models trained on the single task, such as PointNet [27], DGCNN [40], and PCT [12]. These models suffer obvious performance drops when transferred to the new dataset. The more difficulty (higher level of rotating), the more drops.
Furthermore, we validate their generalization abilities on unseen tasks, such as registration and local completion. As shown in Fig. 5 (b), our proposed models work well on both tasks, validating their abilities to transfer learned knowledge. As a comparison, task-specific models are unable to infer the rotated point cloud unless trained with clear supervision. Besides, recovering a local hole is also an impossible task for them, even if they are trained in the reconstruction task.
Effectiveness of Joint Sampling. We investigate whether the JS module has a valid effect on four tasks. As shown in Tab. 2, both PIC-Sep and PIC-Cat face a sharp decline in performance on the four tasks when the JS module is absent, and they even fail to achieve the most basic goal of reconstructing masked tokens. These findings validate our intuition that maintaining the consistency of the input and target token sequence positions is an indispensable design. That is to say, the JS module is a simple yet effective way to compensate for the missing positional information.
Table 2: Effectiveness of JS Module.
Model	JS	Den. CD↓	Part Seg. mIOU↑
PIC-Cat	X	29.3	17.03
✓	5.3	78.95
PIC-Sep	X	36.3	23.72
✓	7.6	74.95
Ablation on Point Sampling. We study how the sampling method used by PIC-Sep in JS modules affects the performance of the model. We use the two most common sampling methods: Farthest Point Sampling (FPS) [27], and Random Sampling (RS) [14]. As shown in Tab. 3 (a), for the tasks of reconstruction, denoising, and registration, FPS produced better results than RS. Especially for
Table 3: Ablation study on our Point-In-Context. Gray: default setting
(a) Different sampling strategies.
#	Sample method	Rec.	Den.	Reg.	Part Seg.
CD↓	CD↓	CD↓	mIOU↑
1	RS	42.6	11.6	12.1	77.24
2	FPS	4.7	7.6	10.3	74.95
(b) Different loss functions.
#	Loss function	Rec.	Den.	Reg.	Part Seg.
CD↓	CD↓	CD↓	mIOU↑
1	l1	5.0	8.1	11.1	72.35
2	l2	4.7	7.6	10.3	74.95
3	l1+l2	5.3	7.9	13.3	70.46
(c) Prompt position.
#	order	Rec.	Den.	Reg.	Part Seg.
CD↓	CD↓	CD↓	mIOU↑
1	behind	4.8	8.2	8.0	74.04
2	before	4.7	7.6	10.3	74.95
point cloud reconstruction, where FPS can collect more key points than RS, and these key points can describe the original outline of the entire point cloud. However, in the task of part segmentation, the results of RS exceed those of FPS.
Loss Function. We conduct an exploration to determine which loss function is most suitable for our Point-In-Context model. During training, we experiment with using  $\ell_1$ ,  $\ell_2$ , and a combination of  $\ell_1$  and  $\ell_2$  as the loss functions for our PIC-Sep. As Tab. 3 (b) shows,  $\ell_2$  achieves the best result on all four tasks mentioned above.
Prompt Engineering. We investigate the influence of the layout of the prompt and the query on the experimental results. For PIC-Sep, we set up two layout options: one with the prompt before the query, and the other with the prompt after the query. As shown in Tab. 3 (c), the performance between the two designs has negligible differences. We simply choose the "before" option to align with 2D in-context learning works.
Mask Ratio. We conduct ablation experiments on the mask ratio at a wide range (20%-70%). As shown in Tab 4, training our Point-In-Context with a lower mask ratio weakens its performance across various tasks, especially on the mask ratio 20%. Meanwhile, the best results in the four tasks are distributed at different mask ratios, but considering all downstream tasks as a whole, the model can achieve the highest performance when the mask ratio is 70%. Different from language data, we also find keeping sparsity in training is necessary for mask point modeling for in-context learning. We find similar results as in MAE [13], a higher mask ratio is required to make sure that the model can learn hidden features well.
Prompt Selection. We explore the impact of the prompt selection on the model's prediction results. For the random selection method, during testing, we randomly select a pair of input-target point clouds from the training set that performs the same task as the query point cloud, serving as a task prompt. For the class-aware selection method, based on the previous one, we further select point clouds belonging to the same category as the query, such as airplanes, tables, chairs, etc. Additionally, we further delve into selecting two alternative prompts. To select pairs of examples that are paired with the query point cloud, we consider two factors: the Chamfer Distance (CD) [11] between the prompt and the query point cloud and the feature similarity between them (features are extracted from pre-trained PointNet [27]), which are respectively denoted
Table 4: Ablation study on mask ratio.
#	Mask Ratio	Rec. CD↓	Den. CD↓	Reg. CD↓	Part Seg. mIOU↑
1	0.2	27.8	33.5	68.2	43.84
2	0.3	5.2	7.3	14.8	56.72
3	0.4	5.0	7.4	12.3	60.25
4	0.5	4.0	7.5	11.5	64.68
5	0.6	4.9	7.8	9.4	70.17
6	0.7	4.7	7.6	10.3	74.95
Table 5: Prompt selection methods.
Model	Selection method	Rec. CD↓	Den. CD↓	Reg. CD↓
PIC-Cat	Random	4.3	5.3	14.1
Class-aware	4.3	5.3	10.5
Fea-aware	4.3	5.3	10.7
CD-aware	4.3	5.3	9.6
PIC-Sep	Random	4.7	7.6	10.3
Class-aware	4.6	7.4	5.1
Fea-aware	4.9	7.6	5.8
CD-aware	4.4	7.1	4.1
as CD-aware and Fea-aware. As depicted in Fig. 5, the CD-aware method demonstrates the best performance and even outperforms the task-specific models on the registration task. Note that we report the random method in the main results (Fig. 1), which means our model has a higher ceiling. This provides us a great opportunity to improve downstream task results by selecting higher-quality prompts, which will be the direction of future work.
Figure 6: Visualization of comparison results between PIC and multitask models.
Comparison Results between PIC and Multitask Models. We conduct a comparison of visualization results between our Point-In-Context models and multitask models on three tasks, including reconstruction, denoising, and registration. It is important to note that the multitask models in this comparison do not utilize the pre-trained backbone. As shown in Fig. 6, compared with other multitask models, our PIC-Sep and PIC-Cat output results are more satisfactory.
**Limitation.** Our experimental results show that our model can adapt to multiple downstream tasks after a single training with the assistance of prompt support. It performs well on the four proposed tasks, indicating its outstanding generalization ability. Nonetheless, our study has inherent limitations. Our model performs conditional generation for all tasks and presents unchallenged performance for concise point clouds. But for point clouds with complex contours, our model struggles and cannot reconstruct the detailed parts of complex point clouds very well, As shown in Fig. 5 (c).
Board Impact. Our work is the first to explore in-context learning for 3D point cloud understanding and is a very relevant but underexplored problem, including task definition, benchmark, and baseline models. Besides, we hope the setup of in-context learning in 3D and the curation of the in-context learning dataset is helpful to the community.
We propose Point-In-Context (PIC), the first framework adopting the in-context learning paradigm for 3D point cloud understanding. Specifically, we set up an extensive dataset of point cloud pairs with four fundamental tasks to achieve in-context ability. We propose effective designs that facilitate the training and solve the inherited information leakage problem. PIC shows its excellent learning capacity, achieves comparable results with single-task models, and outperforms multitask models on all four tasks. Besides, it shows good generalization ability to out-of-distribution samples and unseen tasks and has great potential via selecting higher-quality prompts. We hope it paves the way for further exploration of in-context learning in the 3D modalities.
Acknowledgements: This work was supported by the National Natural Science Foundation of China (No. 62203476), and the Natural Science Foundation of Shenzhen (No. JCYJ20230807120801002). This study is also supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contributions from the industry partner(s). It is also supported by Singapore MOE AcRF Tier 2 (MOE-T2EP20120-0001). It is also supported by the interdisciplinary doctoral grants (iDoc 2021-360) from the Personalized Health and Related Technologies (PHRT) of the ETH domain.
Figure 7: Overall scheme of our Point-In-Context-Cat. Top: During training, each sample comprises two pairs of input and target point clouds that tackle the same task. Unlike PIC-Sep, PIC-Cat concatenates the input and target to form a new point cloud. Bottom: In-context inference on multitask. Our Point-In-Context could infer results on various downstream point cloud tasks.
Overview. The supplementary material includes sections as follows:
- Section A: Pipeline of Point-In-Context-Cat, including training and inference stages.
- Section B: More results about multitask models trained using a pre-trained backbone with multitask heads.
- Section C: More visualization results.
A More Details of PIC
Pipeline of PIC-Cat. During the training phase, our approach involves selecting a pair of query point clouds and a pair of prompt point clouds from the training dataset. These point clouds are then grouped using the Joint Sampling module. Following this, we perform encoding and tokenization on each point cloud and concatenate them to create a new point cloud. A masking operation is applied to the entire point cloud to conduct the MPM task. We set the mask ratio as  $60\%$  for our specific approach, PIC-Cat. During the in-context inference stage, we only mask the last quarter of the tokens, which corresponds to the desired output. This approach allows our PIC-Cat model to reconstruct the masked tokens, leveraging its training experience. It is important to note that the task on the query point cloud is determined by the prompt in the example pair.
Comparison of Model Parameters, GFLOPs, and Test Speed. We compare the parameters and GFLOPs of each model in the main results of the main text in Tab. 6. Our model achieves a favorable balance between structural complexity and task performance, making it a compelling choice. Note that the parameters and GFLOPs of task-specific models are computed, including four individual models for four different tasks. Besides, we report the speed of models by samples/second tested on one NVIDIA RTX 3080 Ti GPU. Our PIC-Cat presents a high inference speed (953 samples/second), which is second only to DGCNN [40].
Table 6: The comparison of parameters, GFLOPs, and test speed.
Task-specific models	multi-task models	In-context learning models
PointNet [27]	DGCNN [40]	PCT [12]	PointNet [27]	DGCNN [40]	PCT [12]	Point-MAE [26]	Point-BERT [44]	PIC-Cat	PIC-Sep
Params(M)	8.9	7.9	13.0	6.0	7.6	10.8	27.0	52.6	29.0	28.9
FLOPs(G)	1.9	3.1	6.3	1.9	10.2	6.4	11.8	12.0	12.1	8.4
Test speed	694	1500	694	844	1185	717	742	190	953	291
Table 7: Results of multitask models composed of a multitask head and a pre-train backbone trained on ShapeNet [7] for classification. For reconstruction, denoising, and registration, we report Chamfer Distance  $\ell_2$  loss (x1000). For part segmentation, we report mIOU.
Models	Acc.(%)	Reconstruction CD ↓	Denoising CD ↓	Registration CD ↓	Part Seg.
L1	L2	L3	L4	L5	Avg.	L1	L2	L3	L4	L5	Avg.	L1	L2	L3	L4	L5	Avg.
multitask models: share backbone + multi-task heads
PointNet [27]	88.7	47.0	45.8	45.4	45.4	45.8	45.9	22.9	23.2	26.3	28.3	30.0	26.1	35.5	34.8	37.1	37.2	38.6	36.6	10.13
DGCNN [40]	89.4	46.7	47.2	48.1	48.6	48.5	47.8	8.2	8.3	8.4	8.8	9.2	8.6	14.2	15.8	18.2	21.8	23.5	18.7	21.35
PCT [12]	89.5	64.7	60.8	59.2	60.1	59.7	61.0	14.5	12.2	12.4	12.0	11.8	12.6	22.6	25.2	28.3	31.1	33.2	28.1	15.43
Figure 8: Additional visualization results of PIC-Sep. The output of our model is marked in red. Note that the results of part segmentation have been processed by adding XYZ coordinates.
B More Results of Multitask Models
Pre-trained Backbone + Multitask Heads. For multitask models, we utilize a pre-trained backbone feature extraction network that is trained on the ShapeNet [7] dataset for classification tasks. This pre-trained backbone network is equipped with multiple task-specific heads to perform multitask learning on our benchmark, allowing for the simultaneous handling of various tasks. As shown in Tab. 7, while these supervised models perform well when trained on individual tasks, they exhibit poor performance on multitask benchmarks.
C More Visualization
More Visualization of PIC-Sep. We visualize more examples in Fig. 8, including reconstruction, denoising, registration, and part segmentation.