Explore In-Context Learning for 3D Point Cloud Understanding
Authors: Zhongbin Fang
Xiangtai Li
Xia Li
Joachim M. Buhmann
Chen Change Loy
Mengyuan Liu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: In-context learning
Point cloud
Prompt tuning
TL;DR: We present a novel framework for in-context learning applied to 3D point cloud understanding, a learning paradigm that has demonstrated significant potential in various domains such as large language model inference and multi-task image processing.

Abstract: With the rise of large-scale models trained on broad data, in-context learning has become a new learning paradigm that has demonstrated significant potential in natural language processing and computer vision tasks. Meanwhile, in-context learning is still largely unexplored in the 3D point cloud domain. Although masked modeling has been successfully applied for in-context learning in 2D vision, directly extending it to 3D point clouds remains a formidable challenge. In the case of point clouds, the tokens themselves are the point cloud positions (coordinates) that are masked during inference. Moreover, position embedding in previous works may inadvertently introduce information leakage. To address these challenges, we introduce a novel framework, named Point-In-Context, designed especially for in-context learning in 3D point clouds, where both inputs and outputs are modeled as coordinates for each task. Additionally, we propose the Joint Sampling module, carefully designed to work in tandem with the general point sampling operator, effectively resolving the aforementioned technical issues. We conduct extensive experiments to validate the versatility and adaptability of our proposed methods in handling a wide range of tasks. Furthermore, with a more effective prompt selection strategy, our framework surpasses the results of individually trained models.

Explore In-Context Learning for 3D Point Cloud Understanding
Zhongbin Fang', Xiangtai Li?, Xia Li Joachim M. Buhmann Chen Change Loy²
Mengyuan Liu
Sun Yat-sen University "S-Lab, Nanyang Technological University
"Department Computer Science, ETH Zurich
Key Laboratory of Machine Perception, Shenzhen Graduate School, Peking University fangzhb6emai12 sysu. edu. en, xiangtai. -icntu edu.sg, sg, xia.li@inf ethz.
ccloyEntu. edu. sg. 1iumengyuan@pku. edw.eeuuueddeccom edu. cn http:p///gggaab..cc/////aaaaaaap com/ffagglaossi/ooinii Iontter
(a) Language understanding
(b) image inpainting
Segmentation
Je vais dinclé
I'm serry
Translation
glace
WIZZA
love ice cream
GPTY
Delicous food!
Positive
Sentiment
MAR
Inpainting
Turrible dishas!
Negative
(c) 3D point cloud modeling
Registration
Context
Danoising
RReconstructin
Figure 1: (a) In-context learning in NLP [6]. with different text prompts for corresponding tasks: translation and sentiment analysis. (b) In-context learning in 2D vision [4], with 2D visual prompts for different tasks: segmentation and inpainting. (c) Our proposed in in-context learning for 3D point clouds, with 3D visual prompts for different tasks: reconstruction, denoising, registration, etc.
Abstract
With the rise of large-scale models trained broad data, in-context learning has become a new learning paradigm that has demonstrated significant potential in natural language processing and computer vision tasks. Meanwhile, in-context learning is still largely unexplored in the 3D point cloud domain. Although masked modeling has been successfully applied for in-context learning in vision, directly extending it to 3D point clouds remains formidable challenge. In the case of point clouds, the tokens themselves are point cloud positions (coordinates) that masked during inference. Morcover, position embedding in previous works may inadvertently introduce information leakage. To address these challenges, we introduce novel framework. named Point-In-Context designed especially for in in-context learning in 3D point clouds, where both inputs and outputs are modeled coordinates for each task. Additionally, carefully designed to work in tandem with the general point sampling operator, effectively resolving the aforementioned technical issues. We conduct extensive experiments to validate the versatility and adaptability of our proposed methods handling wide range of tasks.
propose the Joint Sampling module,
Corresponding author is Mengyuan Liu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
In recent years, large-scale models [1. 32, 15, 5] with enormous parameters pre-trained on broad data have emerged in computer vision and natural language processing. Capable of handling diverse tasks simultaneously, these models can be adapted for new tasks when prompted. Text-to-image generation models such as DALL-E [34] and language models like GPT [33] are examples of this development, representing significant progress toward general intelligence. However, training these models is resource- assource-intenive, intensive, which makes full fine-tuning [18, 21, 20] or even parameter-effiiient tuning techniques [50, 17, 25] such as prompt tuning impractical for many users.
In-context learning, originating from natural language processing (NI.P) [33, 35, 6]. holds potential as the mainstream approach for efficient model adaptation and generalization. Unlike other methods that necessitate model parameter updates for new tasks, in-context learning incorporates dormann-specitie input-output pairs, known as in-context examples or prompts, into a test example. In NLP, the prompt can be a machine translation pair or sentiment analysis, as shown in Fig. 1 (a). This allows the model to produce optimal results without requiring any parameter updates for previous tasks. Several works [4, 38, 39, 48] explore in-context learning in computer vision. The visual prompt [4] is the first to adopt pre-trained neural network for filling missing patches in grid-like images, as shown in Fig. (bb. Other studies investigate the effect of visual prompts or generalization to more vision tasks. The above methods adopt Mask Image Modeling (MIM) architecture [13, 3] for in-context task transfer. Inspired by MIM, Masked Point Modeling (MPM) is proposed recently and is widely used in different point cloud tasks [44]. To our knowledge, no work has explored in-context learning for 3D point cloud understanding using the MPM framework.
Our work is the first to explore in-context learning for 3D point cloud understanding, as shown in
Fig. 1(c). Given the absence of previous works, we propose a benchmark based on the ShapeNet [7]
and ShapeNetPart [42], encompassing four different tasks: point cloud reconstruction, denoising. registration, and part segmentation. Mcanwhile, we benchmark several representative baselines [27, 40, 26, 12], including individual models for each task and the models equipped with shared backbone with multiple task heads. Then, to tackle the benchmark mentioned above, we present the Pnint-In- Context (PIC) to explore in-context learning for 3D point clouds.
A straightforward extension of the 2D MIM architecture to point clouds for in-context learning may encounter two primary obstacles. First, using the conventional position embedding approach could potentially lead to information leakage, attributed to the use of invisible center points', Second, unlike ID word embeddings or 2D images, 3D point cloud data, which are inherently unordered [27], present the risk of having their positional information become disarrayed when partitioned into a patch sequence. Consequently, it is indispensable to devise new sampling and grouping strategy for 3D point cloud data. In response to this issue. we propose a simple yet effective solution, termed joint sampling. This technique involves recording the indices of sampled center points and using the K-nearest neighbor strategy to sample both the input and target concurrently. In addition, leveraging the mask point transformer architecture, we explore two distinct baseline methodologies for PIC, which encompass separating inputs and targets akin to the Painter strategy [38] and concatenating inputs and targets in a manner analogous to the MAE approach [13] for reconstruction. Contrary to the focus of few-shot learning on single specific task, our objective is to explore the in-context ability of the generative model, facilitating the execution of tasks commensurate with the 3D prompt.
Our main contributions are as follows. 1) We introduce simple yet effective general framework for 3D visual prompting. Given two pairs of point clouds, we demonstrate that multiple 3D point cloud tasks can be treated as task-aware prompting, as seen in 2D image and NLP tasks. 2) We create a new benehmark, including four different point cloud tasks for 3D in-context learning, and evaluate several representative baselines. 3) Our comprehensive study addresses various 3D prompt examples, sampling methods, and masking strategies. Moreover, we show that the choice of in-context examples greatly impacts performance for 3D in-context learning.
2 Related Work
3D Point Cloud Classification. Deep neural networks have been proposed for the 3D point cloud classification task. Both PointNet [27] and its improved versions [28, 30] are pioneers of point-based
Previous MPM methods [44, 26] embed position with the masked center points during pre-training.

---

methods in 3D point cloud analysis, which adopt multi-layer perceptron (MLP) to handle point clouds directly. Several graph-based methods [40, 22, 16] exploit geometric properties and propose different dynamic kernels. In particular, DGCNN [40] designs EdgeConv, which dynamically computes each layer output. Recently, several works [49] adopt pure waasformer-basee architecture to model the
global context. Point Transformer [49, 12, 43] applies the vectorized self-attention mechanism to construct a point Transformer layer for 3D point cloud learning. Meanwhile, PointMLP [24] directly applies pure residual MLP network to the 3D point cloud analysis. Recently, several works have explored the vision language models [46] and joint 2D and 3D training [31, 47] via transformer architectures. However, these approaches are only designed for single task, so they cannot be used directly for in-context learning.
Masked Image Modeling (MIM) For 2D and 3D Vision. The GPT and BERT series [8] have greatly enhanced natural language processing performance through masked modeling and fine- tnning downstream tasks. BEiT [3] is the first to propose matching image patches with discrete tokens via d- VAE [34] and pre-train standard vision transformer [37, 10] using masked image modeling [8]. MAE [13] then directly reconstructs the raw pixel values of masked tokens and achieves high efficiency with a high mask ratio. For 3D point cloud pre-training using MIM, several works [44, 26, 45, 47, 9, 29, 19] aim to improve feature representation. Point-BERT [44] adopts BERT-like architecture, while Point-MAE [26] transfers the MAE-like framework for pre- vre-training. These methods use standard transformer networks to process 3D point clouds and achieve competitive performance on various downstream tasks. Our approach follows a similar point MIM pipeline but explores the in-context ability of point transformers and MIM. which has not been investigated previously.
In-Context Learning. In-context learning [33, 35, 6] is a new learning paradigm in large language models like GPT-3. This paradigm enables an autoregressive language model to perform inference on unseen tasks by conditioning the input on specific input-output pairs, known "context." This powerful paradigm allows users to customize a model's output to fit their downstream datasets without modifying the often inaccessible internal model parameters. Recent research in natural language processing has demonstrated the efficacy of in-context learning across a range of language tasks, including machine translation, sentiment analysis, and
Recently, several works [4, 38, 39, 48, 36, 2] explore in-context learning in computer vision. The visual prompt [4] the first pure vision model, which was pre-trained to fill missing patches in images that are made of academic figures and infographics. Then, Painter [38] generalizes visual in-context learning as learning to paint the image via different task prompts. The following works explore the effect of task prompts. Recently, SegGPT [39] extends Painter by learning a generalized one-shot segmentation. In contrast, our method explores the effect of 3D prompts on in-context learning in the point cloud and proposes new baselines for benchmarking 3D in-context learning.
Method
In this section, we first introduce the task settings of in-context learning in the 3D point cloud, which is motivated by 2D visual in-context learning. Then, we elaborate on the dataset construction and task definitions. Next, based on the MPM framework, we point out the information leakage issue and propose a joint sampling strategy. Finally, we build two distinct baselines using the MPM training methodology.
3.1 Modeling In-Context Learning in 3D Point Cloud
In-context Learning in 2D. During training, the 2D in-context learning models [4] take two pairs as inputs, including reference pair (or task prompt pair), R. (T,, T,t and query inputs, Q" {T, Th), ) where R is the task prompt containing one image I; and one target T.K. Q} represent current input image Iy. Here, k represents the task index while and j indicate different example indexes. When i = j. we term the prompt as an ideal prompt. During training, both Visual Prompt [4] and Painter [38] combine two pairs of images that perform the same task into grid-like image and randomly mask portions, following MAE [13]. During the inference stage, only example pairs and query image are provided. They are combined into a grid-like image with a quarter mask to mask the I and a pre-trained model is used to restore the missing parts.
3

---

Information Leokagee
Input
Tanget
Pou Embosdinng
Consistent
KNN
Trursformm
Mask
Enbbed
Adding
Viaile center
Misked (ivisiblu) center
Embbedding
(s) Masked Modeling pre-training presious
Input
Gather
FPS
Idx
0
KNN
Input
Torget
Consistant
Separate input target
KNN
Gather
(b) Our proposed Johnt sampling module
(d) Concatemate input and target
Figure 2: (a) The pre-training pipeline used in previous works. When performing Masked Point Modeling (MPM), these works [44, 26, 45] use the center position of the target patches for position embedding, which results in information leakage. (b) Joint Sampling module. When selecting the center points for the input and target point clouds, the same indexes are used, which are sampled from the input point cloud. (d) Different combination forms of input and target point clouds, which respectively denote the input form of our two baselines, PIC-Sep and PIC-Cat.
In-context Learning in 3D Point Cloud, Motivated by 2D in-context learning, we design a similar procedure for 3D in-context learning. During training, each input sample contains two pairs of point clouds that perform the same task as in 2D-context learning. Each pair consists of an input point cloud and its corresponding output point cloud for the given task. Similar to PointMAE [26], we adopt the farthest point sampling and K-nearest neighbor (KNN) techniques to convert the point clouds into sentence-like like data format. These point patches are subsequently encoded into tokens. During the inference, the input point cloud is combination of example input and query point cloud, while the target point cloud consists of an example target along with masked tokens, as shown in Fig. 1(c). Based on different R, given input Pft, the model outputs a corresponding target Ty..
3.2 Dataset and Tasks Definition
ShapeNet In-Context Datasets. Since there is no previous benchmark for 3D in-context learning, in order to establish the first benchmark for 3D in-context learning, we carefully curate datasets and define task specifications. Firstly, we obtain samples from publicly available datasets, such as ShapeNet [7]. ShapeNetPart [42] and transform them into the "input-target" format as stated in 1. Additionally, augment the sample size for the part segmentation task, we conduct several random operations, including point cloud perturbation, rotation, and scaling, on the ShapeNetPart. Consequently, we construct an extensive dataset encompassing all four types of tasks (mentioned below), comprising 217,454 samples. Each sample comprises an input point cloud and its correspond- ing target for a specific task. Then, we standardize the inputs and outputs to solely contain only the point cloud's XYZ coordinates. For reconstruction and denoising tasks, our aim is to create a clean and aligned point cloud; whereas for the registration task, our aim is to create a clean, registered point cloud. Additionally, the output of the part segmentation task is several point clusters, representing different parts of the object.
Reconstruction. The objective of this task is to reconstruct a complete dense point cloud using only sparse set of points. To evaluate the model's 's reconstruction capability, we establish five levels for input point clouds, which contain 512, 256, 128, 64, and 32 points respectively.
Denoising. In this task, the input consists of a point cloud with Gaussian noise. The objective is to remove the noise surrounding the point cloud, resulting in a clear and distinct object shape. To evaluate the model's 'ss performance across different noise levels, we establish five noise levels ranging from 100 to 500 noisy points.
Registration. The objective of this task is to restore rotated point cloud to its original orientation. It assumed that during both training and inference, the query point cloud and prompt are synchronized in terms of the rotation angle. To avoid coupling with the outputs of the denoising and reconstruction

---

Pars-abs Paint Coudh
Input
Target
Transforme roforreet
Embed
do
Modul
Concet
mask
Promppt Ingist
Prompt Target
Query Imput
Query Target
Masked Token
Trans for meer
Query
Roconsrrucion
Denoking
Registation
Part Segmantion
Training pipeline of the Masked Point Modeling (MPM) framework. During training, each sample comprises pairs of input and target point clouds that tackle the same task. These pairs are fed into the transformer model to perform the masked point reconstruction task, which follows random masking process. Bottom: In-context inference on multitask. Our Point-In-Contex could infer results on various downstream point cloud tasks, including reconstruction, denoising, registration, and part segmentation.
Figure 3: Overall scheme of our Point-In-Context. Top:
tasks, we set the registration task's output as both an upright and an upside-down point cloud. To provide comprehensive evaluation, we define five levels for the registration task. As the level increases, the range of optional rotation angles also increases.
Part Segmentation. 2-6 components. Conventionally, a C'-dimensional one-hot code is assigned every point to classify them, where P is the total count of categories. However, for in-context learning, we need to keep the input and output in the same space, which only contains XYZ coordinates, making a regression task. To achieve this, we convert the P component labels to P discrete points containing XYZ coordinates that are uniformly distributed within cube. Thus, the output of this task is clusters of points.
The goal of this task is to segment an object into several components, typically
3.3 Point-In-Context Models
MPM For 3D In-Context Learning. Following previous works [4, 38], we adopt the masked point modeling framework for point clouds and propose Point-In-Context (PIC), where we treat the points MPM as image tokens MIM, allowing us to leverage transformer for processing both types of data. As shown in Fig 3, during training, we aim to reconstruct the masked points, aided by the input and visible target 3D points. During the inference stage, given query point clouds as input, depending on the task prompted by the example pair, it can generate the corresponding target, such as reconstruction, denoising, registration, or part segmentation outputs of the query point cloud.
Information Leakage. Though MPM [26, 45] exists as base framework for us, simply adapting it to point clouds is not feasible. As shown in Fig. 2(a), previous pre-training pipelines embed positional information with the center point coordinates of all patches, even for those that are masked out (invisible). Since the patches masked out the target are invisible in our setting, such an operation will cause information leakage, which does not satisfy the requirements. Furthermore, we find that sine-cosine encoding sequence will significantly reduce the model performance compared to learned embedding, and even lead to the collapse of the training. The reason is that valuable position information is missing, making it impossible for the model to locate the patches that need to be reconstructed during the processing. Unlike 2D images, 3D point cloud patch sequences have fixed position (unordered property [27]). so we need to align the patch sequences generated from the input and target point clouds.
Joint Sampling (JS) Module. To handle the above issues, we collect N central points from each input point cloud and retrieve their indexes, which we then use to obtain the central points of every patch both the input and target point clouds. The process is shown Fig. 2(b). The key of our IS module is the consistency between center point indices of corresponding patches in both target and input point clouds. In other words, the order of the input token sequence and the target token sequence are well well-aligned. Such design compensates for the missing positional embedding of the
5

---

Table 1: Comparison of task-specific. multitask, and in-context learning models on four 3D point cloud tasks. For reconstruction, denoising. and registration, we report Chamfer Distance [11] loss (x1000). For part segmentation, we report mIOU.
Reconsiruction
CD 4
Denoising CD 4
Registration CD
Seg.
Models
Venues
L1
L2
L3
L4
L5 Avg.
LI
L2
L3
14
L5
Avg.
LI
L2
L3
LA
L3
Avg.
mIOUt
Task-specific models (trained separately)
PointNet 127]
CVPR'17
3.7
3.7
3.8
3.9
4.1
3.9
4.1
4.0
4.1
4.0
4.2
4.1
5.3
59
6.9
7.7
8.5
6.9
77.45
DGONN [40]
TOG 19
3.9
3.9
4.0
4.1
4.3
4.0
4.7
4.5
4.6
45
4.7
4.6
62
6.7
73
74
7.7
7.1
76.12
[12)
2.4
2.5
2.6
3.0
2.6
23
79.46
ACT [9]
ICLR'21
2.4
2.5
2.3
2.5
2.8
2.5
2.2
2.3
22
2.3
2.5
2.3
5.1
5.6
5.9
6.0
7.0
5.9
81.24
Multiusk models: share backhone
multi-rak heads
PointNet [27]
CVPR'17
87.2
86.6
87.3
90.8 92.2 88.8
17.8
22.0
25.6
30.4
33.2
25.8
25.4
22.6
24.9
25.7
26.9
25.1
15.33
DGCNN [40]
TOG'
38.8
36.6
37.5
37.9
42.9 37.7
6.5
6.3
6.5
6.4
7.1
6.5
12.5
14.9
17.9
19.7
20.7
17.1
16.95
[12]
CVM'21
34.7
44.1
49.9
50.0
52.3
10.5
16.71
Point-MAF [26]
2CCV122
5.5
5.5
6.1
64
6.4
6.0
5.6
54
5.6
5.5
5.8
5.6
114
12.8
14.8
16.0
16.9
14.5
5.42
ACT
ICLR'23
7.4
6.6
6.5
6.6
7.0
6.8
7.3
6.8
7.0
6.8
7.2
7.0
12.2
14.4
19.4
25.5
29.0
20.1
12.08
[AP
CVPR'23
16.7
20.1
22.60
ReCon [29]
ICMI/23
12.4
12.1
12.4
12.5
13.1
12.5
20.4
24.5 27.2
29.2
32.5
26.9
14.7
16.3
19.2
21.5
22.5
18.8
7.71
In In-ormtext learning madels
Copy
155
153
152
156
155
1:4
149
155 157 155 155
154
155
157
156
148
154
154
24.18
Point-BERT [44] CVPR'22
288
285
292
286
308
292
292
293
298
296
299
296
291
295
294
295
298
294
0.65
PIC-Cat
4.9
5.5
5.3
Our PIC-Sep
4.7
4.3
4.3
4.4
5.7
4.7
6.3
7.2 7. 8.2
8.6
7.6
8.6
9.2
10.2
11.3
12.4
10.3
74.95
target while avoiding information leakage. Therefore, it facilitates the model to learn the inherent association between input and target and streamlines the learning process. Subsequently, all point clouds search for neighborhoods containing M points based on the center points corresponding to each patch.
Doint-In-Contee Model Architecture. We use a standard transformer with an encoder-decoder structure the backbone of our Point-In-Context, and simple 1 1 convolutional layer the task head for point cloud reconstruction. Inspired by Painter [38] and MAE [13], we explore two different baselines for PIC and name them PIC-Sep and PlC-Cat. For PIC-Sep. take the input and masked target point clouds parallel to the transformer and then merge their features after several blocks, using simple average for the fusion operation. For PIC-Cat, we concatenate the input and target to form new point cloud. Then we mask it globally and feed it to the transformer for prediction. We denote the prompt pair as R (P,, T,* and query inputs, O3 = (P3 TT,3. then PIC-Sep and PIC-Cat can be formalized as:
pSep_ Transformer((44 || P,1, (IT (751,
pCa Transformerr(T, || Th I, || TT), M),
(2)
where || is the concatenate operation, and M is the masked token to replace the invisible token. These two input forms are shown in Fig. 2(c)(d).
Loss Function. The model is trained to reconstruct the masked point patches. To this end, we use the la Chamfer Distance as the training loss. Specifically, we calculate the Chamfer Distance between each predicted patch P and its corresponding ground truth G.
L(P,G) = Σ min |p
gll2 +
Σ
min llp PEP
(3)
PEp gEG
BEG
4 Experiments
Implementation Details. We sample 1024 points of each point cloud and divide it into N 64 point patches, each with M = 32 neighborhood points. We set the mask ratio as 0.7. For PIC-Sep. we merge the feature of input and target at the third block. We randomly select a prompt pair that performs the same task with the query point cloud from the training set. We use an AdamW optimizer [23] and cosine learning rate decay, with the initial learning rate as 0.001 and a weight decay of 0.05. All models are trained for 300 epochs.

---

Input
Prediction
Ground truth
Input
Prediction
Ground truth
Input
Prediction
Ground truth
Denosing
Registration
Segmentation
Figure 4: Visualization of predictions obtained by our Point-In-Context and their corresponding targets in different tasks, such as reconstruction, denoising, registration, and part segmentation. For part segmentation, we visualize the generated target together with the mapping back, both adding category-speciie colors for better comparisons.
4.1 Baseline Methods and Training Details
To evaluate the performance of the proposed framework, evaluate the variants PIC-Sep and PIC-Cat on the dataset mentioned in Sec. 3.2, both equipped with the proposed Joint Sampling module. We compare them with the following related methods:
Point-BERT [44] is a masked auto-encoder. Like the settings in PIC-Cat, we concatenate 2 pairs of input-target point cloud tokens token sequence and input it to Point-BEs BERT, which takes token sequence as the input and converts them into discrete point tokens from a pre-trained dVAE [40] vocabulary of size 8192.
Task-specific Models. We selected three representative methods: PointNet [27], DGCNN [40], PCT [12], and ACT [9], and individually train them on four different tasks mentioned in Sec. 3.2. Additionally, we also designed task-specific heads for each task.
Multitask Models. For fair comparison, we develop a multitask model based on PointNet [27], DGCNN [40], and PCT [12], respectively, which are capable multitask learning. These models feature a shared backbone network and task-specific heads designed to address the needs of different tasks. This design allows simultaneous learning of all four tasks.
Point-MAE [26] is a masked auto-encoder. Unlike Point-BERT, Point-MAE directly rebuilds points in each local area. ACT [9], 12P-MAE [47], and ReCon [29] are recent SOTA methods that involve other modalities like image and text knowledge in the pre-training stage and enhance the performance on different tasks after fine-tuning the models. Similar to multitask models,
pre-trained encoder and combine it with different task heads for simultaneous training on the four tasks.
Copy Example is baseline that utilizes the target point cloud of the prompt its prediction.
4.2 Main Results
report extensive experimental results various models on the dataset we proposed in Tab. From where, we found that our PIC-Cat and PIC-Sep exhibit impressive results and are capable of adapting to different tasks after only one training, achieving state-of-the-a results all four tasks amount multitask models. Besides, visualize the in-context 3D inference results of PIC-Sep in

---

Registration
Completion
Reconstrution
-Q- PointNet
CD (41000
30
20
PlC
Levell Level2 Levela
Leveld
Levels
target
Pred
(a) Registration results on ModelNet40
(b) The generalization capacity
(c) Limitation
Figure 5: (a) Generalized to out-of-distribution data. We evaluate the registration task on the ModelNet40 [41] dataset, which is not present in the training set. The Chamfer Distance (x1000) is used as the evaluation metric. (b) Generalized to new tasks. Our model is able to rotate at any angle according the example pairs and show the completion ability the broken point cloud, which is not present in the training set. (c) Limitation of our model. PIC cannot reconstruct the detailed parts of complex point clouds very well.
Fig. 4, where our model can generate the corresponding predictions given the provided prompts among all four tasks, including reconstruction, denoising, registration, and part segmentation.
Comparison Task-specific Models. Task-speeific models including PointNet [27]. DGCNN [40]. and PCT [12] outperform PIC-Cat and PIC-Sep in most indicators on all tasks. Specifically, in the part segmentation task, PIC-Cat achieves better results than PointNet and DGCNN, though they are well-designed for segmentation. It needs to be clarified that the performance of our model is directly related to the choice of prompts. When the quality of the prompt is better, PIC-Sep and PIC-Cat can achieve much better results, and this will be demonstrated in the following section.
Generalization. Usually, in-context Learning has certain degree of generalization ability, thus allowing the model to quickly adapt to different tasks. This also applies to our models. First, we test the proposed methods on -iittr-ddittiuttionn point clouds by conducting registration evaluation on the ModelNet40 [41] dataset. As shown in Fig. 5 (a), our models, PIC-Sep and PIC-Cat, both exhibit superior performance on open-class tasks compared supervised learning models specifically trained for the registration task. We compare it with models trained on the single task, such as PointNet [27], DGCNN [40], and PCT [12]. These models suffer obvious performance drops when transferred to the new dataset. The more difficulty (higher level of rotating), the more drops.
Furthermore, we validate their generalization abilities on unseen tasks, such registration and local completion. As shown in Fig. 5 (b), our proposed models work well on both tasks, validating their abilities transfer learned knowledge. As a comparison, task-specific models unable infer the rotated point cloud unless trained with clear supervision. Besides, recovering local hole is also an impossible task for them, even they are trained in the reconstruction task.
4.3 Analysis
Effectiveness of Joint Sampling. We investigate whether the JS module has valid effect on four tasks. As shown Tab. 2, both PIC-Sep and PIC-Cat face sharp decline in performance on the four tasks when the JS module is absent, and they even fail to achieve the most basic goal of reconstructing masked tokens. These findings validate our intuition that maintaining the consistency of the input and target token sequence positions is an indispensable design. That is to say, the JS module is simple yet effective way to compensate for the missing positional information.
Table 2: Effectiveness of JS Module.
Den. CDJ
Part Seg. mIOUf
Model
JS
X
29.3
17.03
PlC-Cat
5.3
78.95
X
36.3
23.72
PIC-Sep
7.6
74.95
blation on Point Sampling. We study how the sampling method used by PIC-Sep JS modules affects the performance of the model. We use the two most common sampling methods: Farthest Point Sampling (FPS) [27], and Random Sampling (RS) [14]. shown in Tab. (a), for the tasks of reconstruction, denoising, and registration, FPS produced better results than RS. Especially for

---

Table 3: Ablation study on our Point- Context. Gray: default setting
(a) Different sampling strategies.
(h) Different loss functions.
(c) Prompt position.
Sample
Rec.
Den.
Reg.
Part
Reg.
Seg.
Rec.
Den.
Part
function
CD)
CD)
CD}
miDI
method
CD1
CDL
CD.
mlQUf
CD,
CDJ
CD2
miOUt
5.0
8.1
11.1
72.38
RS
42.6
11.6
12.1
77.24
1
bohind
4.8
8.2
8.0
74.04
10.3
FPS
4.7
7.6
10.3
74.95
4
533
79
13.3
70:46
2
before
4.7
7.6
103
74.95
point cloud reconstruction, where FPS can collect more key points than RS, and these key points can describe the original outline of the entire point cloud. However, in the task of part segmentation, the results of RS exceed those of FPS.
Loss Function. We conduct exploration determine which loss function is most suitable for Point-In-Contex model. During training, we experiment with using &,, fg, and a combination of & and as the loss functions for PIC-Sep. As Tab. 3 (b) shows, &2 achieves the best result all four tasks mentioned above.
Prompt Engineering. We investigate the influence of the layout of the prompt and the query the experimental results. For PIC-Sep, we set up two layout options: one with the prompt before the query, and the other with the prompt after the query. As shown in Tab. 3 (c), the performance between the two designs has neglectable differences. We simply choose the "before" option to align with 2D in-context learning works.
Mask Ratio. We conduct ablation experiments on the
Table 4: Ablation study on mask ratio.
mask ratio at wide range (20%-70%). As shown in
Tab 4, training our Point-In-Context with a lower mask ratio weakens its performance across various tasks, espe- cially on the mask ratio 20%. Meanwhile, the best results in the four tasks are distributed at different mask ratios, but considering all downstream tasks as a whole, the model achieve the highest performance when the mask tio is 70%. Different from language data, we also find keeping sparsity in training is necessary for mask point modeling for in-context learning. We find similar results in MAE [13], higher mask ratio is required to make sure that the model can learn hidden features well.
Mask Ratio
Rec.
Den.
Reg.
Part Seg.
CDJ
CDL
CDL
m10Uf
0.2
27.8
33,5
68.2
43.84
2
0.3
5.2
7.3
14.8
56.72
0.4
5.0
12.3
60.25
0.5
11.5
64.68
0.7
10.3
74.95
Prompt Selection. We explore the impact of the prompt selection on the model's prediction results. For the ran- dom selection method, during testing, we randomly select a pair of input-target point clouds from the training set that performs the same task as the query point cloud, serving as a task prompt. For the class-aware selection method, based on the previous one, further select point clouds belonging to the same category as the query, such as air- Additionally,
Table 5: Prompt selection methods.
Selection
Rec.
Den.
Reg.
Model
method
CDJ
CDJ
CDJ
Random
4.3
5.3
14.1
Class-aware
4.3
5,3
10.5
PtC-Cat
Fea-aware
4.3
10.7
planes, tables, chairs, into selecting two alternative prompts. To select pairs of examples that are paired with the query point cloud, we consider two factors: the Chamfer Distance (CD) [11] between the prompt and the query point cloud and the fea- ture similarity between them (features are extracted from pre- pre-trained PointNet [27]), which are respectively denoted as CD-aware and Fea-aware. As depicted in Fig. 5, the CD-aware method demonstrates the best performance and even outperforms the task-specific models on the registration task. Note that we report the random method in the main results (Fig. 1), which means our model has higher ceiling. This provides us a great opportunity to improve downstream task results by selecting higher-quality prompts, which will be the direction of future work.
further delve
CD-aware
4.3
9.6
Random
4.7
7.6
10.3
Class-aware
4.6
7.4
5.1
PIC-Sep
aware
CD-aware
4.4
4.1

---

In-context learning models
Multitask models
Sep
PIC
PointMAE
PointNet
DGCNN
Figure 6: Visualization of comparison results between PIC and multitask models.
Comparison Results between PIC and Multitask Models. We conduct comparison of visual- ization results between our Coint-ln-Contes models and multitask models on three tasks, including reconstruction, denoising, and registration.
is important to note that the multitask models in this comparison do not utilize the pre-trained backbone. As shown in Fig. 6, compared with other multitask models, our PIC-Sep and PIC-Cat output results are more satisfactory.
Limitation. Our experimental results show that our model can adapt to multiple downstream tasks after a single training with the assistance of prompt support. performs well on the four proposed tasks, indicating its outstanding generalization ability. Nonetheless, our study has inherent limitations. Our model performs conditional generation for all tasks and presents unchallenged performance for concise point clouds. But for point clouds with complex contours, our model struggles and cannot reconstruct the detailed parts of complex point clouds very well, As shown in Fig. (c).
Board Impact. Our work is the first to explore in-context learning for 3D) point cloud understanding and is very relevant but underexplored problem, including task definition, benchmark. and baseline models. Besides, we hope the setup of in-context learning in 3D and the curation of the in-context learning dataset helpful the community.
Conclusion
We propose Point-In-Conted (PIC), the first framework adopting the in-context learning paradigm for 3D point cloud understanding. Specifically, we set up an extensive dataset of point cloud pairs with four fundamental tasks to achieve in-context ability. We propose effective designs that facilitate the training and solve the inherited information leakage problem. PIC shows its excellent learning capacity, achieves comparable results with single-task models, and outperforms multitask models on all four tasks, Besides, shows good generalization ability to out-of-distribution samples and unseen tasks and has great potential via selecting higher-quality prompts. We hope it paves the way for further exploration of in-context learning in the 3D modalities.
Acknowledgements: This work was supported by the National Natural Science Foundation of China (No. 62203476), and the Natural Science Foundation of Shenzhen (No. 00X1200308071200800000 This study is also supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contributions from the industry partner(s). It is also supported by Singapore MOE AcRF Tier 2 (MOE-T2EPH)) MOF-12EP20120-00011 It is also supported by the interdisciplinary doctoral grants (iDoe 2021-360) from the Personalized Health and Related Technologies (PHRT) of the ETH domain.
10

---

References
[1]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022. 2
[2] Ivana Balažević, David Steiner, Nikhil Parthasarathy, Relja Arandjelović, and Olivier J. Hénaff. Towards in-context scene understanding. arXiv:2306.0166) 2023.
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2022. 2.3
[4] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. In NeurlPS, 2022. 1, 2, 3,5
[5] Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv:2108 07258, 2021. 2
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are fow-shot leamers.
In NeurlPS, 2020. 1, 3
[7] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository.
arXiv: www.:1512.0000 03012,000222222 2015. 2, 4, 14
[8] Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In NAACL, 2019.
[9] Rumpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? In ICLR, 2023. 3, 6,
[10] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021. 3
[11] Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3d object reconstruction from a single image. In CVPR, 2017. 6.9
[12] Meng-Hay Hao Guo, Jun-Xiong Cai,, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. In CVM, 2021. 2, 3, 6, 7, 8, 13, 14
[13] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 6,
[14] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang. Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In CVPR, 2020. [15] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Hun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision language representation learning with noisy text supervision. In ICMI., 2021. 2
[16] Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, and Jiaya Jia. Hierarchical point-edge
interaction network for point cloud semantic segmentation. ICCV, 2019. 3
[17] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation.
arXiv:2101.0 100.00099, 2021.
[18] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks, 11.00221 2021. 2
[19] Yang Liu, Chen Chen, Can Wang, Xulin King, and Mengyuan Liu. Regress before construct: Regress autoencoder for point cloud self- superviid learning. In ACM MM, 2023. 3
[20] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, 2022. 2
[21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In /CCV, 2021. 2
[22] Zhe Liu, Shunbo Zhou, Chuanzhe Suo, Peng Yin, Wen Chen, Hesheng Wang. Haoang Li, and Yun-Hui
Liu. Lpd-net: 3d point cloud learning for large-scale place recognition and environment analysis. In ICCV, 2019. 3
[23]
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization adam. In ICLR, 2018.
[24] Xu Ma. Can Qin, Haoxuan You. Haoxi Ran, and Yua Fu. Rethinking network design and local geometry in point cloud: A simple residual mip framework. In ICLR, 2022.
[25] Junting Pan, Ziyi Lin, Xiatian 7hu, Jing Shao, and Hongsheng Li. Parameter-effiieen image-to-video transfer learning. arXiv:2206. 13559, 2022.
[26] Yatian Pang, Wenxiao Wang, Francis EH Tay. Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In ECCV, 2022. 2, 4, 6, 7, 13
11

---

[27] Charles R Qi. Hao Su. Kaichun Mo. and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In CVPR, 2017. 2, 5, 6, 7,7,8 13, 14
[28] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature
learning point sets in a metric space. In NeurlPS, 2017. 2
[29] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge. Xiangyu Zhang, Kaisheng Ma, and Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In JCML, 2023. 3, 6,
[30] Guocheng Qian, Yuchen Li, Houwen Peng. Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Neur!PS,
2022. 2
[31] Guocheng Qian, Xingdi Zhang, Abdullah Hamdi, and Bernard Ghanem. Improving standard transformer models for 3d point cloud understanding with image pretraining. arXiv:2208. 12259, 2022. 3
[32] Alec Radford, Jong Wook Kim. Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021. 2
[33] Alec Radford, Jong Wook Kim. Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021. 2,
[34] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021. 2,
[35] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-contexi learning.
arXiv:2/ 08633, 2021. 2,3
[36] Yanpeng Sun, Qiang Chen, Jian Wang. Jingdong Wang, and Zechao Li. Exploring effective factors for improving visual in-context learning. arXiv:2304.04748 2023. 3
[37] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017. 3
[38] Xinlong Wang. Wen Wang. Yue Cao. Chunhua Shen, and Tiejun Huang. Images speak in images:
generalist painter for in-context visual learning. In CVPR, 2023, 2, 5, 6
[39] Xinlong Wang. Xiaosong Zhang, Yue Cao. Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt:
Segmenting everything context. In ICCV, 2023. 3.
[40] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon.
Dynamic graph can for learning on point clouds. In TOG, 2019. 2, 3, 6, 7, 8. 13, 14
[41] Zhirong Wu. Shuran Song. Aditya Khosla, Fisher Yu,, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.
3d shapenets: A deep representation for volumetric shapes. In CVPR, 2015.
[42] Li Yi. Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections.
In TOG, 2016. 2,4
[43] Xumin Yu. Yongming Rao, Ziyi Wang. Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud
completion with geometry-aware transformers. ICCV, 2021. 3
[44] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu, Point-bert: Pre-training 3d
point cloud transformers with masked point modeling. In CVPR, 2022. 2, 3.4, 6, 7, 13
[45] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training, In NeurIPS, 2022.
3. 4,
[46] Renrui Zhang. Ziyu Guo, Wei Zhang. Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and
Hongsheng Li. Pointelip: Point cloud understanding by clip. In CVPR, 2022. 3
[47] Renrui Zhang. Liuhui Wang, Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d representations from 2d pre-trained models via image-to- to- point masked autoencoders. In CVPR, 2023. 3, 6, 7
[48] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? 2XXiv2330.136777 2023. 2.3
[49] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip H.S. Torr, and Vladlen Koltun. Point transformer. In ICCV,
2021. 3
[50] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In IJV, 2022. 2
12

---

Supplementary Material
Paks Point deeds
Input
Target
Embred
pairs
Mast
Ti
120
101
Medule
Transformer
Outpun
Prompe
Nosbed Taken
Modile,, Embediing, Nosking
Rieerpe
Query
Transformer
Figure 7: Overall scheme of our Doint-InCContext-S5 comprises two pairs of input and target point clouds that tackle the same task. Unlike PIC-Sep, PIC-Cat concatenates the input and target to form new point cloud. Bottom: In-context inference on multitask. Our Point-In-Contex could infer results on various downstream point cloud tasks.
Top: During training, each sample
Overview. The supplementary material includes sections follows:
Section A: Pipeline of Point-In-Conter In-Context-Cat, including training and inference stages.
Section B: More results about multitask models trained using a pre-trained backbone with multitask heads.
Section C: More visualization results.
A More Details of PIC
Pipeline of PIC-Cat. During the training phase, our approach involves selecting a pair of query point clouds and a pair of prompt point clouds from the training dataset. These point clouds are then grouped using the Joint Sampling module. Following this, we perform encoding and tokenization on each point cloud and concatenate them to create new point cloud. A masking operation is applied to the entire point cloud to conduct the MPM task. We set the mask ratio as 60% for our specific approach, PIC-Cat. During the in-context inference stage, we only mask the last quarter of the tokens, which corresponds to the desired output. This approach allows our PlC-Cat model to reconstruct the masked tokens, leveraging its training experience. important to note that the task on the query point cloud is determined by the prompt in the example pair.
Comparison of Model Parameters, GFLOPs, and Test Speed. We compare the parameters and GFLOPs of each model the main results the main text in Tab. 6. Our model achieves a favorable balance between structural complexity and task performance, making that the parameters and GFLOPs of task-specific models are computed, including four individual models for four different tasks. Besides, report the speed of models by samples/second tested on one NVIDIA RTX 3080 Ti GPU, Our PlC-Cat presents a high inference speed (953 samples/second), which is second only DGCNN [40].
compelling choice. Note
Table 6: The comparison of parameters, GFLOPs, and test speed.
Task-spetie models
multi-task models
In-context learning models
PointNer [27)
DGCNN [40] PCT PCT [12] PointNet (27) DGCNN [60] PCT [2]] Poin: MAE [26] Point BERT [44] PIC-Car PIC-Sep
Parama(M)
8.9
52.6
29.0
FLOPs(G)
12.1
Test speed
694
953
13

---

Table Results of multitask models composed of a multitask head and a pre-train backbone trained on ShapeNet [7] for classification. For reconstruction, denoising, and registration, we report Chamfer Distance l loss (x1000). For part segmentation, we report mlOU.
Reconstruction CD
Demoising CD +
Registration CD 4
Part Seg.
Models
Acc.9%)
1.1
12
13
14
1.5 Avg.
1.2
1,3 14 1.5 Avg.
1.1
1.2
1.3
14
1.5
Avg.
mIOU1
maltitask
multi-task
PointNes [27]
88.7
47.0
45.8 45.4 45.4 45.8 45.9
22,9
23.2
26.3 28.3
30.0 26.
35.5
34.8
37.1
37.2
38.6 36.6
10.13
DGONN
46.7
48.1
48.6
47.8
8.3
9.2
14.2
18.2
21.8
23.5
21.35
PCT [12]
89.5
64.7
60.8
59.2 60.1 59.7 61.0
14.5 12.2 12.4 12.0
11.8 12.6
22.6
25.2
28.3
31.1
33.2 28.1
15.43
Reconstruction
Denoising
Registration
Part Segmentation
Figure 8: Additional visualization results of PIC-Sep. The output of model is marked in red. Note that the results of part segmentation have been processed by adding XYZ coordinates.
B More Results of Multitask Models
Pre-trained Backhone Multitask Heads. For multitask models, utilize a pre-trained backbone feature extraction network that is trained on the ShapeNet [7] dataset for classification tasks. This pre-trained backbone network is equipped with multiple task-specific heads perform multitask learning on our benchmark, allowing for the simultaneous handling of various tasks. As shown in Tab. 7, while these supervised models perform well when trained on individual tasks, they exhibit poor performance on multitask benchmarks.
C More Visualization
More Visualization PIC-Sep. We visualize more examples Fig. 8, including reconstruction. denoising, registration, and part segmentation.
14


Reviews:
[{"review_id": "EhzKYkBKo7", "reviewer": "Reviewer_Uvps", "summary": "Inspired by in-context learning in NLP and 2D vision tasks, this paper aims to explore the in-context learning in the 3D point cloud. The authors present Point-In-Context, which is a 3D mask point modeling framework. Meanwhile, to handle the data leakage issues, the authors also present a simple solution, named joint sampling. Then, extensive experiments are carried out in modified ModelNet40 dataset. The performance of two different baselines is good to other specific baselines.", "questions": "Please refer to the *Weaknesses.", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed point in-context is the first work that explores the in-context ability in the point cloud understanding, which is novel and interesting to me. The authors show the effectiveness on four different tasks, including rotations, registration, denoising and part segmentation. \n\n2. The overall writing and motivation is good, clear, and easy to follow. \n\n3. The proposed joint sampling can effectively solve the data leakage problems. \n\n4. The experiments results are good. The ablation studies are extensive. The authors re-benchmark several representative works, including SOTA single model and multi-task models. The analysis on visual point example is good and convincing. \n\n5. This work may bridge the connection between 3D point cloud and 2D image in-context learning. \n", "weaknesses": "1. Are there any other solutions to replace joint sampling to handle the data leakage problems?\n\n2. The proposed approach is verified effective on simple scene on ModalNet40. The ability to extend to large scale scene including in-door point segmentation is unknown. \n\n3. What are the results of two combined prompts: denoising and part segmentation jointly?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1: Are there any other solutions to replace joint sampling to handle the data leakage problems?**\n\n**A1**: \n\n&emsp; When we adopt the original MPM pre-training framework for our task, we find that model utilizes center point coordinates that should have been masked before position embedding. This inadvertent inclusion results in unintended information leakage.\n\n&emsp; This problem is **not explored in previous work and is an open question**. To address this predicament, we introduce the **Joint Sampling module** designed to align the positional attributes of patches/tokens between input and target point clouds. This alignment serves to compensate for the consequential loss of vital positional data, thereby enabling the model to understand the intricate mapping correlation inherent in the input-target pairs.\n\n&emsp; During our exploration, we tested various positional embeddings, learnable matrices, and sampling methods, but none met our expectations. In the end, our **simple yet highly effective** Joitn Sampling module provided the perfect solution to this challenge. In future work, we will consider **enhancing** the JS module's structure by incorporating a learnable matrix. This will strengthen the linkage between the input point clouds and the target point clouds, thereby advancing overall performance and efficacy.\n\n>**Q2: The ability to extend to large-scale scenes including in-door point segmentation is unknown.**\n\n**A2**:\n\n&emsp; Segmenting scene-level point clouds presents a formidable challenge. The 3D point cloud datasets comprise an extensive number of points and contain intricate object composition, such as ScanNet [Dai et al., 2017], Matterport3D [Chang et al., 2017], and S3DIS [Armeni et al., 2016].\n\n&emsp; To solve these problems, more fine-grained design and more computing resources are required. However, we think that **this is feasible**. Referring to PointNet and others' works, we can divide the 3D scene into blocks using a non-overlapping sliding window of 1m x 1m on the xy plane. By processing these blocks individually and subsequently merging their results, we can derive the ultimate segmentation result of a whole 3D scene. Naturally, if an ample supply of computational resources is available, a comprehensive analysis of the entire scene can yield a deeper understanding of both the 3D scenes and their constituent objects.\n\n&emsp; This is a work worth exploring in the future: applying in-context learning to 3D scene-level point cloud segmentation to achieve the unification of semantic segmentation, instance segmentation, and panoptic segmentation. We would like to explore more possibilities of in-context learning in 3D point clouds, not only involving object-level point clouds.\n\n[Dai et al., 2017] Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR.\n\n[Chang et al., 2017] Matterport3d: Learning from rgb-d data in indoor environments. In 3DV.\n\n[Armeni et al., 2016] 3d semantic parsing of large-scale indoor spaces. In CVPR.\n\n>**Q3: What are the results of two combined prompts: denoising and part segmentation jointly?**\n\n**A3**:\n\nCaption: performance (mIOU) of PIC-Sep and PIC-Cat of part segmentation. The given prompts contain various levels of noise points ranging from 100 to 500 noisy points (1024 points per sample).\n\n| Models  | Original | Level=1 | Level=2 | Level=3 | Level=4 | Level=5 | Average |\n|:-------:|:--------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n| PIC-Sep |  74.95   |  75.13  |  75.02  |  74.94  |  74.84  |  74.68  |  74.92  |\n| PIC-Cat |  78.95   |  77.73  |  77.33  |  77.15  |  77.04  |  76.95  |  77.24  |\n\n&emsp; We use pre-trained PIC-Sep and PIC-Cat in Table 1 of the main text to test the combined task, when different levels of noises are added to the prompts during the testing. As the above table shows, **our PIC-Sep and PIC-Cat can naturally generalize to noised prompt without re-training**. For PIC-Sep, adding noise has little effect on its performance. For PIC-Cat, its mIOU drops about 1.2-1.7 in various levels of noise points. PIC shows great robustness. We **visualize** the part segmentation results of PIC-Sep and present the figure in **the PDF of the global rebuttal**.\n\n&emsp; Besides, we also notice that **the choice of prompt impacts the performance**. When we select the prompt for PIC according to the minimum Chamfer Distance between the query point cloud and the prompt (CD-aware), we find that the performance (mIOU) increases, as shown in the following table.\n\nCaption: The higher-quality prompt can improve the performance of PIC.\n\n| Models   | Random | CD-aware |\n|:---------:|:------:|:--------:|\n| PIC-Cat  |  78.95  |   **80.49** $\\uparrow$ |\n| PIC-Sep  |  74.95  |   **78.46**$\\uparrow$  |"}, {"review_id": "Jd1F2HXSzJ", "reviewer": "Reviewer_GuqQ", "summary": "This work is conducted toward in-context learning for 3D point cloud data. Similar to 2D in-context learning, the authors first define and construct the in-context learning 3D dataset covering reconstruction, denoising, registration, and part segmentation tasks. To avoid information leaking during masked point modeling, a joint sampling module that samples correspondingly from inputs and outputs is proposed. Extensive experiments have been conducted, and interesting results have been obtained.", "questions": "Besides questions and concerns listed before, I have following questions:\n- Is all tasks jointly trained or one model per task?\n- What if the 3D in-context learning involves other modalities? For example, images or languages?\n- In ACT [Dong et al., 2023], I notice that the authors report the reconstruction CD-$\\ell_2$ on ShapeNet as 2.110, which is significantly lower (better) than the best reconstruction results in this paper (4.3). Can authors explain this?\n\nI am looking forward to the author's response, and I am happy to raise my score if my concerns are solved.", "limitations": "The authors have discussed the limitations, and I think it is somewhat okay for this work.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This works targets in-context learning for 3D point clouds, which is a very relevant but underexplored problem.\n- The setup of in-context learning in 3D and curation of the in-context learning dataset is helpful to the community.\n- Extensive experiments have been conducted, where results demonstrate the superiority of the methodology design, which also indicates that in-context learning in 3D is non-trivial.\n- Code and dataset are promised to be released. Good.", "weaknesses": "- For 2D or NLP, in-context learning is generally free-form, targeted at various tasks. However, for 3D, due to a seriously lacking of data, it only shows very limited applications. It seems not practical in real-world deployments.\n- The novelty and technical contribution of the method part is somewhat limited. The joint sampling module is relatively straightforward. The model and loss function is the same as in previous works. However, the contribution of setting up the 3D in-context learning baseline is good.\n- Missing citations or comparison: For 3D MIM methods, important recent cross-modal representation/prompt learning methods should be discussed or compared [Dong et al., 2023; Qi et al., 2023]; For the loss function, Chamfer Distance should be cited [Fan et al., 2017]; For in-context learning, some works are missing [Sun et al., 2023; Balažević et al., 2023].\n- The compared methods Point-BERT and Point-MAE are a bit old and are not SOTA. I wonder what about the comparison to other cross-modal 3D MIM methods like ACT [Dong et al., 2023], I2P-MAE [Zhang et al., 2023], and ReCon [Qi et al., 2023]? I think it is important to conduct solid comparisons to more advanced methods.\n- Minor suggestion: There are not many formulations but the current formulation is not neat. Please improve the presentation quality including the formulations. For example, pure texts should not be italic in equations. Besides, the writing is overall okay but could be improved.\n\n[Dong et al., 2023] Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning? In ICLR.\n\n[Qi et al., 2023] Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining. In ICML.\n\n[Fan et al., 2017] A point set generation network for 3d object reconstruction from a single image. In CVPR.\n\n[Sun et al., 2023] Exploring Effective Factors for Improving Visual In-Context Learning. arXiv preprint.\n\n[Balažević et al., 2023] Towards In-context Scene Understanding. arXiv preprint.\n\n[Zhang et al., 2023] Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders. In CVPR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Due to word limit, we omit the questions, answer order is consistent with the above questions.\n\n**A1**:\n\n&emsp; Seriously lacking data is a **common problem for 3D tasks**. With the advent of 3D sensors such as LiDAR and Kinect, 3D point clouds have gained increasing popularity and are widely used in robotics, autonomous driving, object detection, etc.\n\n&emsp; First, our work is **the first to explore the application of in-context learning in 3D point clouds and goes one step further than previous methods**. \n\n&emsp; Secondly, our PIC can **easily be generalized to related datasets (Figure 5(a) of the main text) and tasks (A3 of the reviewer kyHc)**. We strive to tap the multi-task potential of the model as much as possible under the condition of fewer data (can see **A1 of the reviewer kyHc**) and provide a new idea and direction for future work of 3D point clouds. Such capability is urgently needed for a deep learning model in the condition of a seriously lacking 3D point cloud data.\n\n&emsp; In conclusion, the proposed method **is good at solving the lacking data problem**.\n\n**A2**:\n\n&emsp; Our work is **the first to explore in-context learning for 3D point cloud understanding**. We unify the output space of the four tasks into the 3D coordinate space, and apply the MPM framework for model training, realizing one training, one model, and processing multiple tasks.\n\n&emsp; It is **a challenge to train multiple tasks jointly**. Previous methods cannot achieve an ideal balance among multiple tasks, resulting in poor multi-task performance, as shown in Table 1 of the main text. Our PIC shows a great ability for dealing with multiple tasks jointly with an MPM-style training scheme.\n\n&emsp; However, it is infeasible to simply adopt the MPM framework into our work due to **information leakage**. That is the model knows the coordinates of the center point of the patch to be reconstructed in advance, which **is not allowed**. It is an open problem that is never explored.\n\n&emsp; Therefore, in order to make up for the lack of position information, we propose a simple but effective method, the **Joint Sampling module**, so that each patch/token of the input point cloud and the target point cloud is well-aligned. **During our exploration**, we tested various positional embeddings, learnable matrices, and sampling methods, but none met our expectations.\n\n&emsp; As for the model structure, we improve some detailed architecture for our different forms of inputs. But what we want to express most is that **we are the first work to introduce in-context learning into the 3D point clouds, laying the foundation for subsequent research**.\n\n**A3**:\n\n&emsp; Thank you for your valuable suggestions. We will cite these articles in the final version and add the comparison of ACT, I2P-MAE, and ReCon to the main experimental results.\n\n**A4**:\n\n&emsp; Thank you for your constructive suggestions, which make our work more complete and solid. We conduct experiments on other cross-modal methods like ACT, I2P-MAE, and ReCon. In the implementation, we use a pre-trained encoder and combine it with different task heads for simultaneous training on the four tasks. As shown in the table of **the PDF of the global rebuttal**, our PICs outperform them on four tasks. Such results demonstrate that our PICs show an excellent ability to deal with multi-task. **We will include these results in the final version**.\n\n**A5**:\n\n&emsp; Thanks for your valuable advice. Except for the typo you have mentioned, we will carefully check all formulas and polish the expression of the article in the final version to improve its readability.\n\n**A6**:\n\n&emsp; Our Point-In-Context is trained on all tasks jointly. As for compared methods, the multi-task models are trained on all tasks jointly, too. The task-specific models are trained on each task individually. The joint training signifies the proposed method that is capable of learning various contexts and bringing mutual benefits.\n\n**A7**:\n\n&emsp; It's an exciting idea, and we consider it feasible to involve the other two modalities. Our work is the first to explore in-context learning for 3D point cloud modality. Similar to the ICL in language and 2D images, our model is based on transformer architecture, providing us a chance to involve the other two modalities. \n\n&emsp; The main challenge is that there are not many data sets with three modalities and data alignment, and how to **align the features** of the three modalities to the same feature space is a semi-open problem. There are already several works exploring the multimodal understanding of 3D scenes (PLA [1] and Open Scene [2], etc.), and future work will focus on exploring the in-context learning in 3D point cloud scene segmentation. Please see **A2 of reviewer Uvps** for more details.\n\n[1] PLA: Language-Driven Open-Vocabulary 3D Scene Understanding. CVPR2023.\n\n[2] OpenScene: 3D Scene Understanding with Open Vocabularies. CVPR2023.\n\n**A8**:\n\n&emsp; The reconstruction result of 2.110 is an ablation experiment in the ACT paper, and we did not find more details about this experiment in the paper. Therefore, we speculate that there are two reasons:\n\n&emsp; **1. The settings of the experiments are different.** We establish five levels for input point clouds, which contain 512, 256, 128, 64, and 32 points respectively for our reconstruction task. The difficulty of our reconstruction task cannot be measured and compared with the reconstruction task in the ACT directly.\n\n&emsp; **2. About the pre-processing of experimental data.** In order to unify the data between different tasks and different datasets, we normalize the data, resulting in the point clouds of the testing set in our reconstruction dataset being **larger** than the original data (see **table R2 in PDF of the global rebuttal**). So our results may be seen as higher than the results of ACT intuitively. However, our results cannot be compared with the result of the ACT paper directly."}, {"review_id": "MmRffLF0k4", "reviewer": "Reviewer_kyHc", "summary": "This paper introduces a novel framework, named Point-In-Context, designed explicitly for in-context learning 3D point clouds. The authors conduct extensive experiments to validate the versatility and adaptability of the proposed methods in handling a wide range of tasks.", "questions": "1. Is the Transformer backbone trained from scratch or it is initialized from something, whether this backbone has been pre-trained on some 3D data before, for example, pointbert, pointmae .etc? Is there any leakage between train dataset and test dataset.\n2. Have you tried only train the model for part of the tasks not all of them, and try to see if there is any emergent ability; for example, only train it for reconstruction and registration, do it have some zero-shot emergent ability for denoising and part-segmentation?", "limitations": "refer to the weakness part", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper explores an interesting topic -- how to do in-context learning for 3D understanding; and it demonstrates a reasonable way to achieve it, addresses some issues like the \"position information leakage\", and shows some positive results.", "weaknesses": "The reviewer is concerned about how well this framework can generalize, its general usefulness, and its scope, since to some extent the performance may depend on the \"prompt\". In reality, it might be hard to choose a proper prompt for it and the performance may not be stable, and it kind of involves some extra tuning effort compared to using the model which is directly trained for that task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1: In reality, it might be hard to choose a proper prompt for it and the performance may not be stable, and it kind of involves some extra tuning effort compared to using the model which is directly trained for that task.**\n\n**A1**:\n\n&emsp; The prompt can indeed affect the performance of our PIC, but we consider it to be **the icing on the cake for us**. Moreover, our PIC shows **great generalization ability** which is needed in actual applications.\n\n&emsp; Usually, the prompt plays a very important role in In-context Learning. High-quality prompts can improve the performance of the model. In Table 1 of the main text, the main results of this paper, the prompts we used are randomly selected under the same task in the training set. It can be seen that **the performance brought by these random prompts is already remarkable**, surpassing all multi-task models on the four tasks. However, we have a higher ceiling. When it is possible, **a good prompt shall greatly improve the performance** (see Table 3(a) of the main text and Table 5(a) of the appendix). \n\n&emsp; Besides, we demonstrate that **our PIC has great generalization ability**. In actual scenarios, the 3D domain shows seriously lacking data (as the reviewer GuqQ said), and the data to be tested in actual applications do not necessarily have corresponding training data, so the generalization of the model is more needed. As we mentioned in Figure 5(a) of the main text, when our model generalizes to other datasets (ModelNet40), it can outperform the single-task training model under the same training environment, which shows that **our model can perform well under the limitation of training data in practical applications**. We also demonstrate that our PIC can generalize to other related tasks (see the following A3). \n\n&emsp; In conclusion, we regard **the possibility of choosing a better prompt as an advantage of the proposed method, providing the chance of achieving more reliable performance**. we do not need a “proper” prompt to maintain our performance, the random prompt under the same task is satisfied. Moreover, **our generalization ability can make it more practical in reality compared to task-specific models**.\n\n>**Q2: Is the Transformer backbone trained from scratch or it is initialized from something, whether this backbone has been pre-trained on some 3D data before? Is there any leakage between the training dataset and the testing dataset?**\n\n**A2**:\n\n&emsp; Our model is trained from scratch, **no additional data is used for pre-training**, and **no pre-trained model parameters are used**, all parameters are only updated on our training set. In addition, our training set and test set are divided according to the ShapeNet [Chang et al., 2015] and ShapeNetPart [Yi et al., 2016] datasets **without any leakage**. Our code is reproducible, which is provided in the supplementary material, and the dataset will be made public in the future.\n\n[Chang et al., 2015] Shapenet: An information-rich 3d model repository. In arXiv.\n\n[Yi et al., 2016] A scalable active framework for region annotation in 3d shape collections. In TOG.\n\n>**Q3: Have you tried only training the model for part of the tasks not all of them, and try to see if there is any emergent ability; for example, only train it for reconstruction and registration, does it have some zero-shot emergent ability for denoising and part-segmentation?**\n\n**A3**:\n\nCaption: The performance of PIC-Sep and PIC-Cat on unseen tasks. **Bold represents tasks included in the training set**. For instance, the first row of PIC-Cat denotes that it is trained on reconstruction and registration, and is tested on denoising and part segmentation.\n\n\n| Models        | Reconstruction CD$\\downarrow$ | Denoising CD$\\downarrow$ | Registration CD$\\downarrow$ | Part Seg. mIOU$\\uparrow$ |\n|-------------- |:--------------:|:---------:|:------------:|:---------:|\n| PIC-Cat (Original)  |      **4.3**       |    **5.3**    |     **14.1**     |   **78.95**   |\n| PIC-Cat (new)        |     **3.9**       |   19.5    |     **3.9**      |    2.98   |\n| PIC-Cat (new)         |      **4.1**       |   21.0    |     **18.8**     |   **79.51**   |\n|  -  |  -  |  -  |  -  |  -  |\n| PIC-Sep (Original)  |      **4.7**       |    **7.6**    |     **10.3**     |   **74.95**   |\n| PIC-Sep (new)        |      **2.4**       |   25.3    |     **3.9**      |    3.41   |\n| PIC-Sep (new)        |      **6.7**       |   22.9    |     **11.0**     |   **80.46**   |\n\n&emsp; Thanks for your advice. We conduct experiments that train our PIC for part of the tasks and test its emergent ability for the remaining tasks. As the above table shows, we train our PIC for **reconstruction & registration & part segmentation**, and evaluate it on the **denoising** task, which is an unseen task in the training set. Despite the formidable challenge, the PIC demonstrates **great emergent capability on denoising task**.\n\n&emsp; For training on reconstruction & registration and testing on denoising & part segmentation, we don't think it makes sense. Because **the output space of part segmentation is kind of different from the other three, it is hard for PIC to adapt to such a polar task**. Despite it, we conducted the experiment. As we expected, we find that PIC shows excellent generalization capability on the denoising task but performs poor results on the part segmentation task due to the extremely different output space form.\n\n&emsp; It is worth noting that our PIC trained on our dataset **can also generalize to the other dataset (ModelNet40)**, and its performance surpasses the task-specific model trained on our dataset, as shown in **Figure 5(a) of the main text**."}, {"review_id": "mRj7JBnFQ6", "reviewer": "Reviewer_uxmh", "summary": "This paper proposes an in-text learning method for multi-task 3D shape analysis. It handles several tasks such as denoising, part segmentation, reconstruction and registration with a single pretrained masked point model (MPM).  The authors also claim previous MPM methods introduce information leakage during pretraining. To solve this, this paper proposes a JS module, which facilitates the model to learn the inherent association between input and target and streamlines the learning process.", "questions": "see above", "limitations": "see weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tIn-context learning for point clouds is a new and interesting topic.\n2.\tThe unified modeling of different 3D shape analysis tasks makes sense.\n3.\tThe performance of the proposed method is satisfactory.", "weaknesses": "1.\tThe writing of this paper is not clear. (3.1) The authors should add more description on the task definition, such as which kind of model should be pretrained, is it fixed during inference, etc.  (3.3) It is claimed that previous methods will bring information leakage. However, the explanation is confusing. The authors should clearly show this in Figure 2.  Moreover, as a core technical contribution of this paper, JS module should be clearly demonstrated. It is hard for me to understand why using the same FPS Idx.\n2.\tThe JS Module seems to be straight-forward (and confusing). As a simple technique, more insight should be explained.\n\nI will consider to improve my rating if my concern can be properly solved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1**:\n>\n>(1) **Task definition**\n>\n>(2) **Information leakage**\n>\n>(3) **JS module**\n\n**A1**:\n\n&emsp; We are sorry that you were confused about our paper and thank you very much for your suggestions on our paper. We provide more detailed explanations:\n\n&emsp; **1. More details of the task definition in Section 3.1.**\n\n&emsp; **During training**, motivated by 2D in-context learning [Bar et al., 2022; Wang et al., 2022], we provide two pairs of point clouds as input to the model and train the model with an MPM-style framework, which is to reconstruct the masked parts of the target point clouds belonging to two pairs. **During testing**, we provide the prompt pair and the query pair that is comprised of a query point cloud and a mask as input to the model. Then the model will reconstruct the mask like what to do in training, which is the corresponding output to the query point cloud.\n\n&emsp; Besides, Our Point-In-Context is trained from scratch, with **no pre-training nor additional dataset**. As for compared methods, the multi-task models are trained on all tasks jointly, too. But they are comprised of a pre-trained encoder and multi-task heads. The task-specific models are trained on each task individually. \n\n&emsp; Finally, **the model is fixed during inference** with no more learning.\n\n&emsp; **2. The elaboration on the information leakage problem mentioned in Section 3.3.**\n\n&emsp; Previous methods based on the MPM pre-training framework will first be pre-trained on the ShapeNet dataset with Masked Point Modeling. The pre-trained encoder is then combined with task-specific heads to perform fine-tuning on downstream tasks, such as classification, few-shot learning, or segmentation. \n\n&emsp; During pre-training, the model will embed the position of **all sampled local center points** to the extracted point cloud features, even if **they belong to the center points of the patches to be reconstructed (have been masked out, invisible)**. This is possible under their pre-training scheme, but it will cause information leakage under our setting, which **is not allowed**. In other words, our model shouldn't see the masked center points, because we just train one time for multiple tasks and no more fine-tuning. \n\n&emsp; Based on your confusion, we **improve Figure 2(a)** of the main text and put it in **the PDF of the global rebuttal**. Please see the improved figure of the demonstration of information leakage for more details.\n\n&emsp; **3. More explanations and insights for JS modules.**\n\n&emsp; In order to solve the above problems, we replace the position embedding that leads to information leakage with a sin-cos fixed encoding, that is, do not use the local center point for position embedding. However, we found that the performance of the model dropped drastically, and even failed to converge, as shown in the following table.\n\nCaption: The performance (mIOU) of PIC-Sep and PIC-Cat on denoising and part segmentation.\n\n| Model         | Denoising CD$\\downarrow$ | Part Seg. mIOU$\\uparrow$ |\n|--------------|:---------:|:---------:|\n| PIC-Cat w/o JS |   29.3    |   17.03   |\n| PIC-Cat w/ JS  |   **5.3**     |   **78.95**   |\n|-|-|-|\n| PIC-Sep w/o JS |   36.3    |   23.72   |\n| PIC-Sep w/ JS  |   **7.6**     |   **74.95**   |\n\n&emsp; Unlike images, each point of a point cloud has no clear order structure (**unordered property**), so the position information is seriously missing between the input point cloud and the target point cloud, resulting in the model being unable to learn the input point cloud and the mapping relationship between the target point cloud. Such the unordered property mentioned above is **an unexplored challenge** for introducing in-context learning to 3D point clouds.\n\n&emsp; Therefore, in order to make up for the lack of position information, we **align the indices of input sample pairs for each task during training**. During the **generation of our dataset**, the input point cloud of all tasks is generated from the target point cloud, and different operations are performed for different tasks. Taking the reconstruction task as an example, a sparse input point cloud is obtained by discarding a certain percentage of points in the target point cloud. We zero the XYZ coordinates of the discarded points, so the shape of the input point cloud remains consistent with the target point cloud. **This is why we can make the patch/token sequences of both the input point cloud and the target point cloud well-aligned by using the same FPS index**.\n\n&emsp; For the denoising task, we add Gaussian noise to the points at different levels, so the shape of the input point cloud and the target point cloud are the same and aligned. For the registration task, we multiply the target point cloud with a rotation matrix to get the input point cloud. For the part segmentation task, we convert each point's part label to a point in XYZ space, and points representing the same part are clustered together. All in all, the input and target point clouds are consistent in shape as well as indices of each point for all tasks.\n\n&emsp; Then, we propose a **simple but effective** method, the **Joint Sampling module**, so that each patch/token of the input point cloud and the target point cloud is well-aligned. \n\n&emsp; **During inference**, for the query point cloud, its target is masked out, so **no alignment is required**, we only need to provide the aligned task prompt pair which is randomly sampled in the training set. During our exploration, we tested various positional embeddings, learnable matrices, and sampling methods, but none met our expectations.\n\n[Bar et al., 2022] Visual prompting via image inpainting. In NeurIPS.\n\n[Wang et al., 2022] Images speak in images: A generalist painter for in-context visual learning. In CVPR.\n\n>**Q2: The JS Module seems to be straight-forward (and confusing). As a simple technique, more insight should be explained.**\n\n**A2**: \n\n&emsp; Please see the above explanations."}]

Author rebuttal:
&emsp; We would like to thank the four reviewers for their suggestions, which make our paper more solid.

&emsp; We are grateful that the reviewers acknowledge our work. Here list some excerpts:

&emsp; **1.** As the reviewer **Uvps** and **GuqQ** said, Our work is the first to explore in-context learning for 3D point cloud understanding and is a very relevant but underexplored problem. Besides, the setup of in-context learning in 3D and the curation of the in-context learning dataset is helpful to the community.

&emsp; **2.** As the reviewer **uxmh** said, we unify the output space of the four tasks into the 3D coordinate space, and apply the MPM framework for model training, realizing one training, one model, and processing multiple tasks.

&emsp; **3.** As the reviewers **kyHc**, and **GuqQ** said, we design our Point-In-Context for in-context learning in 3D point clouds and address the "information leakage" problem via the Joint Sampling module. Such indicates that in-context learning in 3D is non-trivial.

&emsp; As suggested by four reviews, we conduct some additional experiments on our Point-In-Context and demonstrate more inspirable abilities.

&emsp; **1.** For the response of the reviewer **Uvps**, we use pre-trained PIC-Sep and PIC-Cat in Table 1 of the main text to test the combined task, when different levels of noises are added to the prompts during the testing part segmentation task. The results show our PIC-Sep and PIC-Cat **can naturally generalize to noised prompt without re-training**.

&emsp; **2.** For the response of the reviewer **GuqQ**, we conduct experiments on **other cross-modal SOTA MPM methods** like ACT, I2P-MAE, and ReCon. Such experiment results will be included in the final version.

&emsp; **3.** For the response of the reviewer **kyHc**, we conduct experiments that train our PIC for part of the tasks and test its emergent ability for the remaining tasks. Despite the formidable challenge, the PIC demonstrates **great emergent capability** on the other unseen task in the training.

&emsp; **4.** For the response of the reviewer **uxmh**, we explain more details about **task definitions, information leakage, and insights into our technological** methods.

&emsp; In conclusion, our work is **the first to explore in-context learning for 3D point cloud understanding**. We **unify the output space of the four tasks** into the 3D coordinate space, and apply the MPM framework for model training, realizing **one training, one model, and processing multiple tasks**. Extensive experiment results demonstrate the **superiority of the methodology design**, which also indicates that **in-context learning in 3D is non-trivial**. Besides, our Point-In-Context shows **great generalization capability**.

&emsp; We include two figures and two tables in **the PDF** and cite them in some rebuttal.

- Figure R1: Visualization of PIC-Sep on the part segmentation. The given prompts contain various levels of noise points ranging from 100 to 500 noisy points (1024 points per sample).

- Figure R2: The improved demonstration of information leakage in the previous MPM works.

- Table R1: The additional results of the cross-modal methods: ACT, I2P-MAE, and ReCon.

- Table R2: The numerical statistical results of target point clouds of the ShapeNet testing set and our reconstruction testing set.

Meta-review: The paper delves into in-context learning for 3D point clouds, an important yet often overlooked issue. Reviewers unanimously commend the basic idea and concur on the method's efficacy towards its objectives. The authors have furnished a rebuttal addressing the reviewers' concerns. Consequently, the AC endorses the acceptance of the submission.
Decision: Accept (spotlight)