Multi-View Representation is What You Need for Point-Cloud Pre-Training
Authors: Siming Yan
Chen Song
Youkang Kong
Qixing Huang
Venue: ICLR 2024 poster (2024)
Keywords: point cloud
point cloud pre-training
multi-view representation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: This paper introduces a novel approach to pre-training 3D point cloud models using multi-view representation.

Abstract: A promising direction for pre-training 3D point clouds is to leverage the massive amount of data in 2D, whereas the domain gap between 2D and 3D creates a fundamental challenge. This paper proposes a novel approach to point-cloud pre-training that learns 3D representations by leveraging pre-trained 2D networks. Different from the popular practice of predicting 2D features first and then obtaining 3D features through dimensionality lifting, our approach directly uses a 3D network for feature extraction. We train the 3D feature extraction network with the help of the novel 2D knowledge transfer loss, which enforces the 2D projections of the 3D feature to be consistent with the output of pre-trained 2D networks. To prevent the feature from discarding 3D signals, we introduce the multi-view consistency loss that additionally encourages the projected 2D feature representations to capture pixel-wise correspondences across different views. Such correspondences induce 3D geometry and effectively retain 3D features in the projected 2D features. Experimental results demonstrate that our pre-trained model can be successfully transferred to various downstream tasks, including 3D shape classification, part segmentation, 3D object detection, and semantic segmentation, achieving state-of-the-art performance.

Published as a conference paper at ICLR 2024
MULTI- VIEW REPRESENTATION IS WHAT YOU NEED FOR POINT-CLOUD PRE- TRAINING
Siming Yan', Chen Song', Youkang Kongi, Qixing Huang
The University of Texas at Austin, MMicoosoft Research Asia
{s tsimings song, huangqx]@cs utexas. edu, (kykdqs)@gma com
ABSTRACT
A promising direction for pre-training 3D point clouds is to leverage the mas- sive amount of data in 2D, whereas the domain gap between 2DD and 3D creates a fundamental challenge. This paper proposes a novel approach to point-cloud pre-training that learns 3D representations by leveraging pre-trained 2D networks. Different from the popular practice of predicting 2D features first and then ob- taining 3D features through dimensionality lifting, our approach directly uses a 3D network for feature extraction. We train the 3D feature extraction network with the help of the novel 2D knowledge transfer loss, which enforces the 2D projections of the 3D feature to be consistent with the output of pre-trained 2D networks. To prevent the feature from discarding 3D signals, we introduce the multi-view consistency loss that additionally encourages the projected 2D feature representations to capture pixel-wise correspondences across different views. Such correspondences induce 3D geometry and effectively retain 3D features in the projected 2D features. Experimental results demonstrate that our pre-trained model can be successfully transferred to various downstream tasks, including 3D shape classification, part segmentation, 3D object detection, and semantic segmentation, achieving ofate-of-the-ar performance.
INTRODUCTION
The rapid development of commercial data acquisition devices (Daneshmand et al. 2018) and point- based deep learning networks (Qi al. 2017a b) Choy et al.] (2099) has led growing research interest in models that can directly process 3D point clouds without voxelization. Remarkable success
has been achieved in various applications. including but not limited to object detection (Misra et al.
2021
Liu et al. 202 Wang et al. 2022a). segmentation (Qian et al., 2022 Tang et al.) 2022 Zhao
ct al.
2021
Yan al. 2021), ) and tracking (Qi al. 2020 Zheng et al. 20 al. (021))
Despite the significant advances in 3D point cloud processing, acquiring task-specific 3D annotations a highly expensive and severely limited process due to the geometric complexity. The shortage of
data annotations highlights the need for adapting pre-training paradigms. Instead of training the deep network from randomly initialized weights, prior work suggests that pre-training the network on a relevant but different pre-task and later fine-tuning the weights using task-specific labels often leads to superior performance, In natural language processing (Devlin et al. (008)) and 2D vision (He et al. Radford et al. 2013; Doersch et al. 2015 He et al. [0 et al. 2021 2019), pre-trained models are the backbones of many exciting applications, such as real-time chatbots (Touvron et al.
2022
2023 OpenAl 2023) and graphic designers (Meng al. 2021 Wang et al.) (022b). However, pre-
training on point clouds has yet to demonstrate a universal performance improvement. From-scratch
training remains a common practice in 3D vision.
Initial attempts towards 3D) point-cloud pre-training primarily leverage contrastive learning (Chopra al. 2005). especially when the point clouds are collected from indoor scenes (Xie et al. 2020 Rao et al.) 2021 Liu et al.] 2021a Zhang et al. 2021 Chen et al. 2022).. However, the broad application of contrastive learning- based pre- pre-training techniques is impeded by the requirement of large batch sizes and the necessity to carefully define positive and negative pairs. In contrast to natural language processing and 2D vision, pre-training on 3D point clouds presents two unique challenges.

---

Published as a conference paper at ICLR 2024
First, the data is extremely scarce, even without annotations. Public 3D datasets are orders of magni- tude smaller than 2D image datasets. Second, the lack of data annotations necessitates 3D pre- pre-training meth- ods to adhere the self-supervised learning paradigm. Without strong supervision, pre-task design be- comes particularly crucial in effec- tive knowledge acquisition.
$3D15-45
Seg-lit
53D15
13.75
62.5
75.75
67.5
67.5
Det-SUN RGG-D
90.0
-05.0
NAA
73.0
74.0
06
57
50,20
Point-MZAE
PointContrast
SceneContext
Figure 2 illustrates our novel ap- proach to 3D pre-training, which is designed to address the aforemen- tioned challenges, Our key idea is to leverage pre-trained 2D networks in the image domain to enhance the performance of 3D representation learning while bridging the 2D-3D
FartSeg ShopeWetPart
FeaR's
12P MAE
ULIP
Ours
Cls-Scanübjectti
Figure 1: Our model (blue) achieves state-of-the-art perfor- mance across a broad range of tasks at both the scene and shape levels. The distance to the origin indicates the task re- sult.
domain gap. We begin by applying a 3D-based network to an input point cloud with M points, which generates 3D feature volume represented by an .M x C-dimensional embedding, where C? is the length of the per-point feature vectors. To compensate for data scarcity, we align the 3D feature volume with features predicted by a 2D encoder trained on hundreds of millions of images. To this end, we project the 3D feature volume into pixel embeddings and obtain H W x C' image-like feature maps in 2D, We then use pre-trained image encoders to extract hierarchical features from corresponding RGB images and train a similarly structured encoder on projected 2D maps to enforce feature alignment. To ensure geometrical meaningfulness, we project the same 3D feature volume into multiple camera views and perform hierarchical feature alignment in each individual view. Our pipeline is generic enough support all major 2D mre-trained models (e.g., CLIP (Radford et al. 2021), SAM (Kirillov et al. 2023), DINOv2 (Oquab et al., 2023)). In our experiments, DINOv2 exhibits the best performance.
One issue with the approach described above is that the 3D feature learning may overfit to the pre-trained 2D networks. Consequently, it could potentially discard 3D features that are critical for 3D recognition but not well-captured during image-based 2D pre-training. To address this issue, we introduce an auxiliary pre-task predict 2D multi-view pixel-wise correspondences from pairs of projected 2D feature maps. As such correspondences induce 3D depth information, the learned feature representations are forced capture 3D signals.
Our approach is fundamentally different from existing work based on multi-view representations (Su al. 20155] Kalogerakis et al., 2017: Kanezaki et al.] 2018) Wei et al.) 22200; Kundu et al. 2020)] Hamdi et al. 2021; 2023), which either aggregate image-based analysis results or lift 2D feature maps into 3D for understanding. Our approach employs a 3D network for feature extraction trained with the additional help of pre-trained 2D models. While multi-view based 3D understanding approaches (Su et al. 2015 Kalogerakis et al. 2017) tremendously benefit from the rich experience in 2D vision and have long dominated the design of shape classification networks when 3D data is limited, recent 3D-based understanding networks (Riegler et al.] 2017] Qi et al., 2017a by Wang et al.) 2017) are shown to outperform their multi-view counterparts. The superior performance is the result of enhanced architecture design and the clear advantage of directly learning 3D patterns for geometry understanding. In our setting, the 3D feature extraction network provides implicit regularization on the type of features extracted from pre-trained 2D models. We seek to combine the advantage of multi-view representations and the geometric capacity of 3D networks, while leveraging the rich knowledge embedded in pre-trained 2D networks.
In summary, we make the following contributions:
We formulate point-cloud pre-training as learning multi-view consistent 3D feature volume.
To compensate for data scarcity, we leverage pre-trained 2D image- based models to supervise 3D pre-training through perspective projection and hierarchical feature-hased 2D knowledge transfer.
2

---

Published as a conference paper at ICLR 2024
To prevent overfitting to pre-trained 2D networks, we develop an auxiliary pre-task where the goal is to predict the multi-view pixel-wise correspondences from the 2D pixel embeddings.
We conduct extensive experiments demonstrate the effectiveness of our approach across wide range of downstream tasks, including shape classification, part segmentation, 3D detection, and semantic segmentation, achieving consistent improvement over baselines (Figure
2 RELATED WORK
Point-cloud pre-training The success of point-based deep neural networks demonstrates the potential for machine learning models to directly perceive point clouds. Point-based pre-training refers to the practice of pre-training the point-based prediction network on one or more pre-tasks before fine-tuning the weights on downstream tasks. The expectation is that the knowledge about pre- tasks will be transferred downstream tasks, and the network can achieve better performance than random parameter initialization. We refer readers to (Xiao et al. 2023) for comprehensive survey of the field, which we highlight one important observation, stating that point-based pre-training "still lags far behind as compared with its counterparts", and training from-scratch "is still the prevalent approach". The challenges and opportunities call for innovations in point-based pre- pre-traine ree-training methods.
Shape-level pre-training. Self-reconssruutionn is popular pre-task at the shape level, where the network encodes the given point cloud as representation vectors capable of being decoded back to the original input data (Yang al. 2018 (Sauder & Sievers 2019). To increase the pre-task difficulty, it common for self-reconstrruction based methods to randomly remove certain percentage of the points from the input (Wang al. [222]] Pang et al. [022 Zhang et al.) 2022a Yu al. 2022, Yan
et al. 2023b a). In addition to self-reconstruction. (Rao et al. (020) develop a multi-task approach
that unifies contrastive learning, normal estimation, and selftreconttruuttion into the same pipeline.
There are also attempts to connect 3D pre-training to 2D (Atham et al. 2022: Xu et al.
2021 Dong
al., 2022; Zhang et al. (022b), or to explore the inverse relationship (Hou et al. 2023). For example, (Dong et al. 2022) utilize pre-trained 2D transformers as cross-modal teachers. (Zhang jet al. 2022b) leverage self self-supervised pre-training and masked autoencoding to obtain high-quality 3D features from 2D pre-trained models. However, these methods focus on synthetic objects, leading to significant performance degradation in downstream tasks using real data from indoor scenes.
Scene-level pre-training. Existing scene-level pre-training methods focus exploiting the contrastive learning paradigm. As the first pre-training method with demonstrated success at the scene level, PointContrast (Xie et al. (2020) defines the contrastive loss using pairs of 3D points across multiple views. RandomRoom (Rao et al. 202I) exploits synthetic datasets and defines the contrastive loss using pairs of CAD objects instead. Experimentally, RandomRoom additionally confirms that synthetic shape-level pre-training is beneficial to real-world scene-level downstream tasks. DepthContrast (Zhang et al. (021)) simplifies PointContrast by proposing effective pre-training method requiring only single-view depth scans. 4DContrast (Chen et al. (022) further includes temporal dynamics in the contrastive formulation, where the pre-training data comprises sequences of synthetic scenes with objects in different locations. In contrast to these approaches, we focus on utilizing pre-trained 2D networks boost the performance of 3D pre-training without leveraging contrastive learning.
3 METHOD
We present our pre-training approach pipeline, denoted as MVNet in the following sections. As illustrated in Figure 2 given a pair of RGB-D scans, we first project the 2D pixels into 3D point clouds using camera parameters. We then extract the point-cloud feature volumes using the feature encoding network (Section Subsequently, we project the feature volume onto two different views to generate 2D feature maps (Section 3.2 We design the 2D knowledge transfer module to learn from large-scale 2D pre-traines models (Section 3.3). Finally, we utilize the multi-view consistency module to ensure the agreement of different view features (Section 3.4h, promoting that the 3D feature encoding network extracts 3D-wware features from pre-trained 2D image models. Finally, the weights of the feature encoding network are transferred to downstream tasks for fine-tuning.
3

---

Published as a conference paper at ICLR 2024
212 Knowlodge Transfer
u.
Feature Encodiig
Multi-view Consistency
Input P
Feature Volume Fp
Know
Transfer
(ss D2) RGB-D Seans
Fg Feature Maps
RGB Images
Figure 2: Approach overview. We used complete and semi-transparen point clouds to represent the input P and the feature volume Fp for better visualization. We encourage readers to frequently reference to this figure while reading Section 3
3.1 FEATURE ENCODING NETWORK
Let (I,, Di) and (Iz, D2) two RGB-D scans from the same scene, where I and In denote the RGB images, and D, and D2 represent the depth maps. Using camera parameters, we project the RGB-D scans into a colored point cloud P E RMx6 with M points, with the first three channels representing coordinates and the remaining three channels representing RGB values, The feature encoding network of MVNet takes / as input and outputs a dense feature field Fp E IR MCC where each point associated with a feature vector of dimension C.
The network architecture adopts the design of Sparse Residual U-Net (SR-UNet) (Choy et al. 2019). Our network includes 21 convolution layers for the encoder and 13 convolution layers for the decoder, where the encoder and the decoder are connected with extensive skip connections. Specifically, the input point cloud P is first voxelized into M' voxels, yielding grid-based representation V E TR MRA' x6 In our implementation, we use the standard Cartesian voxelization with the grid length set to 0.05 m. The output of the network is a set of C-dimensional voxel features, Fy E RM x jointly generated by the encoder and the decoder. Due to space constraints, refer interested readers the supplementary material for implementation details.
Next, interpolate the per-voxel features Fy obtain C-dimensional feature vector for each of M points in the input point cloud. This achieved via the classical KNN algorithm based on point-to-point distances. As shown in the middle of Figure 2 the resulting Fp is our point-cloud feature volume, awaiting further analysis and processing.
Importantly, unlike existing work that utilizes multi-view representations from 2D feature extraction networks, MVNet exploits a 3D feature extraction network. As a result, MVNet has the intrinsic ability to capture valuable 3D patterns for 3D understanding.
3.2 POINT-CLOUD FEATURE VOLUME PROJECTION
The key idea of MVNet to learn 3D feature representations through 2D projections, enabling us to leverage large-scale pre-trained models 2D. To this end, we proceed project the feature volume Fp back onto the two input views, generating 2D feature maps, Pi and P2, both with dimensions H W x C. Here, X W is the spatial resolution of the input.
The projection operation uses the one-to-one mapping between each pixel in each input view and the corresponding 3D point in the concatenated 3D point cloud P. We refer readers to the supplementary material for the implementation details and the conversion formula between 2D and 3D coordinates.
3.3
2D KNOWLEDGE TRANSFER MODULE
The first feature learning module of MVNet, Fed. aims transfer knowledge from large- scale 2D pre-trained models. To this end, we consider each input RGB image 1, and the corresponding projected 2D feature map F, Our goal is to train F, using a pre- trained 2D model fp that takes I; as input. Our implementation uses ViT-B (Dosovit skiy et al. 2020 as the network architecture. More details are provided Section
4

---

Published as a conference paper at ICLR 2024
1
F;
The technical challenge here lies in the seman- tic differences between F; and I, As illustrated in Figure 3 introduce ft that takes F; as input. fk shares the same net- work architecture Ip except the first layer, which is expanded channel-wise to accommodate the C- dimensional input. Let N be the number of layers in fü and fp. We define the training loss as
additional network
fp
fk
Layer
Layer
N
Layer
Layer
L2d Σ 1=3
B pre-trained
-B3
Layer
Layer
L2d
where B denotes the output feature map of the jth block in the pre-trained 21) model fp- and B3D denotes the output feature map of the jth block in fu- The objective of the 2D knowledge transfer process to minimize Lia, thereby enabling Ss to learn the hierarchical features through knowl- edge transfer from the pre-trate pre-trained 2D model. We add the loss starting from the third layer, which experimentally leads to the best performance.
Layer
Layer
Figure 3: 2D knowledge transfer module. fp is completely frozen during training. For simplicity, we represent the C-dimensional feature map F; using semi-transparent image.
During training, the weights of the 2D pre-trained model fp unchanged, whereas f& and the feature encoding network are jointly optimized. In our implementation, we select ViT-B (Dosovitskiy al., 2020) as the network backbone for both 2D neural network fa and fp- We take the pre-trained weights of DINOv2 (Oquab et al. 2023) for fp- DINOv2 has been demonstrated to generate features with strong transferability. We refer readers Section 4.3 for a detailed analysis of these design choices through extensive ablation studies.
3.4 MULTI-VIEW CONSISTENCY MODULE
The goal of the previous module is to leverage pre-traie pre-trained 2D networks. However, pre-trained 2D networks only contain feature representations that are primarily suitable for 2D tasks. A complete reliance on the 2D knowledge transfer module encourages the 3D feature encoding net- work to discard geometric cues that are important for 3D recognition. To address this issue, we introduce a novel multi-view consistency module F.. where the goal is to use the projected features, Fj and F2, to predict dense correspondences between the two corresponding input images. Prior research has demonstrated that dense pixel- wise correspondences between calibrated images allows the faithful recovery of 3D geometry (Zhou et al. 2016). Therefore, enforcing ground-truth correspondences to be recoverable from F, and F2 prevents the learned feature volume from discarding 3D features.
Fi
F2
Attention-based Decoder
C
Figure 4: Illustration of multi-view con- sistency module. @ denotes side-by-side concatenation.
Our dense correspondence module adopts cross attention-based transformer decoder DD As depicted in Figure 4 we concatenate the views' feature maps side by side, forming a feature map F... The context feature map F. is fed into a cross attention- based transformer decoder D, along with the query point from the first view 1, Finally, we process the output of the transformer decoder with fully connected layer to obtain our estimate for the corresponding point, x', in the second view Iz:
Fm F1, F2) D(r, Fi + F2)
Following (Jiang et al. 202 we design the loss for correspondence error and cycle consistency:
L. -بي - 2'1 || Fm (m'
(3)
where n8 denotes the ground truth corresponding point of in the second view Jz.
5

---

Published as a conference paper at ICLR 2024
S3DIS Area 5
S3DIS 6. Fold
ScanNet
Method
Backhome
OA
mAcc
mioU
QA
mAcc
mIoU
mfoU
Jigsaw Sauder & Sievers
2019)
DGCNN
82.8
52.1
84.4
56.6
Wang et
DGCNN
55.4
58.5
STRL
Fruang v 8.
202
DGCNN
82.6
51.8
84.2
57.1
MM-3DScenn
2023
PrintTrans
78.0
71.9
72.8
PointContrast
XT al
A1221
SR-IINet
77.0
70.9
74.1
DopthContrast
Zhmng
2021 2021
SR-UNet
70.6
71.2
SceneContext
SR-UNer
72.2
MVNet
SR UNet
91.7(91.6) 79.5(79.3)
73.8
91.8(91.7) 86.2(86.2
78.3 (78.1)
75.6(75.2)
Table I: Comparison of semantic segmentation results with pre-training methods. We show comprehen- sive evaluation across all the benchmarks. The average result of 3 runs is given in parentheses.
Combing and B the overall loss function is defined as:
L L2d + XLm.
(4)
where X = 0.5 is the trade-off parameter.
4 EXPERIMENTAL ANALYSIS
4.1 PRE-TRAINING SETTING
Data preparation We choose ScanNet DDii al. 2017) the pre-training dataset, which contains approximately 2.5M RGB-D scans from 1,513 indoor scenes. Following (Qi et al. 2201)), we down- sample 190K RGB-D scans from 1,200 video sequences in the training set. For each scan, build the scan pair by taking it as the first scan (71, D1) and select five other frames with an overlap ratio between (0.4, 0.8] the second scan (I2, D2) candidates. We also evaluate our model at the shape level. We pre-trained our model on Objaverse (Deitke et al., 2023), which contains approximately 800K real-world 3D objects. We also pre-trained our model on ShapeNet (Chang et al. 2013) for fair comparison with previous approaches. For each 3D object, we use Blender (Kent) (2015) to render 6 images with their camera angles spaced equally by 60 degrees. At each training step, we randomly choose two consecutive images for pre-training.
Network training Our pre-training model is implemented using PyTorch, employing the AdamW optimizer (Loshchilov Hutter 2017) with a weight decay of 10-4, The learning rate is set to 10 The model is trained for 200 epochs on eight 32GB Nvidia V100 GPUs, with 64 as the batch size.
4.2 DOWNSTREAM TASKS
The goal of pre-training is to learn features that can be transferred effectively to various downstream tasks. In the following experiments, we adopt the supervised fine-tuning strategy, popular way to evaluate the transferability of pre-trained features. Specifically, we initialize the network with pre-trained weights described in Section and fine-tune the weights for each downstream task.
Semantic segmentation We use the original training and validation splits of ScanNet (Dai et al. 2017) and report the mean Intersection over Union (IoU) on the validation split. We also evaluate on S3DIS (Armeni et al., 2017) that has six large areas, where one area is chosen as the validation set, and the remaining areas are utilized for training. We report the Overall Aceuracy (OA), mean Accuracy (mAcc), and mean IoU for Area-5 and the 6-fold cross- cross-validane validation results.
We adopt SR-UNet (Choy et al. 2019) as the semantic segmentation architecture and add head layer at the end of the feature encoding network for semantic class prediction. Although it is intuitive to transfer both the encoder and decoder pre-trained weights in segmentation fine-tuning, we observe that only transferring the encoder pre-trained weights results in better performance. We fine-tune our model using the SGD+momentum optimizer with a batch size of 48, 10,000 iterations, an initial learning rate of 0.01, and a polynomial- bassdd learning rate scheduler with power factor of 0.9. The same data augmentation techniques in (Chen et al. 2022) are used.
As demonstrated in Table 1 both datasets, achieve the best performance compared to other pre- pre-trainn training methods, with +1.5 val mloU improvement on ScanNet compared to the second-best method. On the S3DIS dataset, report +1.6 mloU improvement the Area 5 validation set.

---

Published as a conference paper at ICLR 2024
Method
Detection
ScanNet
SUN RGB-D
Architecture
mAP@0.25
mAP@0.5
mAP@0.25
mAP@0.5
STRL (Huang et al.
2021)
VoteNet
59.5
38.4
58.2
35.0
Point-BEKI [u al. 2022
3DETR
61.0
38.3
RandomRoonis (Kao
VoteNet
61.3
36.2
59.2
35.4
MaskPoint (Liu
20,2
3DETR
63.4
40.6
PointContrast (Xie el al. (2220)
VoteNet
59.2
38.0
57.5
34.8
DepthContrast IZhang et
12021
VoteNet
62.1
39.1
60.4
35.4
MM-3DScene (Xi cl 202.1
VoteNet
63.1
60.6
37.3
SceneContext (Hou ct al
20221
VoteNet
39.3
36.4
4DContrast (Chen et 2022
VoteNet
40.0
38.2
MVNet
VoteNet
64.0(63.4) 4)
41.5(41.0)
62.0(61.7)
39.6(39.2)
MVNet
CAGroup3D
76.0(75.7)
62.4(61.8)
67.6(67.2)
51.7(51.2)
Table 2: Comparison of 3D object detection results with pre-training methods. We show mean Average Precision (mAP) across all semanntc classes with 3D IoU thresholds of 0.25 and 0.5. The average result of 3 runs is given in parentheses.
3D object detection ScanNet (Dai et al. 2017) contains instance labels for 18 object categories, with 1,201 scans for training and 312 for validation. SUN RGB-D (Song et al. 2013) comprises of 5K RGB-D images annotated with amodal, 3D-oriented bounding boxes for objects from 37 categories.
We employ the encoder of the SR-UNet (Choy et al. 2019) as the detection encoder and transfer the pre-trained weights for network initialization. For the decoder design of 3D detection, we consider VoteNet (Qi et al. (019)) and CAGroup3D (Wang et al.) 2022a). VoteNet is a classical 3D detection approach commonly used by existing pre-training methods (Xie et al.) 2020 Zhang et al. [021] Rao et al. 2021 Huang et al. 2021). However, VoteNet's performance lags behind the state-of- tre-art. Therefore, we also evaluate our approach on CAGroup3D, which offers a stronger decoder architecture and delivers state-of-the-ar 3D detection performance. During training, for the VoteNet setting, we fine-tune our model with the Adam optimizer (the batch size and the initial learning rate is 0.001). The learning rate is decreased by factor of 10 after 80 and 120 epochs. For the CAGroup3D setting, we decrease the learning rate at 80 and 110 epochs.
As shown in Table on ScanNet, our approach (MVNet) improves VoteNet by +4.5/+3.1 points on mAP@0.25 and m5p@0.5, outperforming other pre-training approaches. When replacing VoteNet with CAGroup3D, MVNet further elevates the mAP@0 00t.000.5//P...... to 76.0/62.4. On the SUN RGB-D dataset, VVNet also exhibits consistent improvements. Specifically, MVNet achieves +3.8/44.6 points mAP@0.25 and mAP@0.5 under the VoteNet backbone and 67.6/51.7 points on mAP@0.25 and mAP@@05 under the CAGroup3D backbone.
3D shape part segmentation In addition scene-level downstream tasks, also evaluate MVNet at the shape level. Our experiments on 3D shape part segmentation utilizes the ShapeNetPart (Yi al. 2016), which comprises 16,880 models across 16 shape categories. We adopt the standard evaluation protocol as described by (Qi et al. 20 2017b), sampling 2,048 points from each shape. We report the evaluation metrics in terms of per-class mean Intersection over Union (cls. mloU) and instance-level mean Intersection over Union (ins. mloU).
To make a fair comparison, in accordance with Yuu al. 2022 we employ a standard transformer network with depth of 12 layers, a feature dimension of 384, and 6 attention heads, denoted as MVNet-B. The experimental results are summarized in Table With the same network backbone, our approach demonstrates consistent improvements. Additionally, more results involving pre-training approaches with different network backbones can found the supplementary material.
3D shape classification We also conduct experiments on the real-world ScanObjectNN dataset UUy et al. 2019), which contains approximately 15,000 scanned objects distributed across classes. The dataset is partitioned into three evaluation splits: OBJ-BG, OBJ-ONLY, and PB-T50-RS, where PB- T50-RS is considered the most challenging one. We employ the same transformer-be classification network (MVNet-B) described by (Yu et al. 2022). The experimental outcomes, presented in Table reveal consistent performance gains across all three test sets.
Model scalability The scalability of model is crucial factor when evaluating the effectiveness of pre-trate training approach. Unfortunately, many previous methods overlook this aspect. The commonly used transformer architecture has limited capacity due the small number of parameters. To further
7

---

Published as a conference paper at ICLR 2024
ScanObjectNN
ShapeNetPart
Method
Dataset
OBJ-BG
OBJ-ONLY
PB-T50-RS
ins. mloU
cls. mloU
Transformer
et 2022
79.9
80.6
77.2
85.1
83.4
PointBERT Yu al. 2022
ShapeNet
87.4
88.1
83.1
85.6
84,1
MaskDiscr
Liu et 2022
ShapeNet
89.3
84.3
MaskSurfel (Zhang et 2022c
ShapeNet
91.2
89.2
85.7
86.1
84.4
ULIP (Xue al. 2025
ShapeNet
91.3
89.4
86.4
PointMAE [Pang et 2022
ShapeNet
90.0
88.3
85.2
86.
MVNettBB
ShapeNet
89.7
86.7
86.1
MVNettBB
Objaverse 91.5
90.1
87.8
86.2
84.8
MVNet--L.
Ohjaverse 95.2
94.2
91.0
86.8
85.2
Table Performance comparison with other pre-training approaches on shape-level downstream tasks.
All the methods and MVNet-B B use the same transformer backbone architecture. represents from scratch results and all other methods represent the fine-tuning results using pretrained weights.
substantiate the efficacy of our proposed approach, we improve the network's s feature dimension to 768 and double the number of attention heads 12. As evidenced by the results in Table our scaled model, MVNet-L, attains significant improvements across all evaluated settings.
4.3 ABLATION STUDY
We proceed to present ablation study to justify our design choices. Due to the space constraint, focus on the semantic segmentation results using the ScanNet dataset.
Choice of 2D pre-trained models As of the most popular backbone architectures, ViT (Dosovitskiy et al. 2020) has variants: ViT-S and ViT-B. VIT-S (Small) contains 12 transformer blocks, each of which has 6 heads and 384 hidden dimensions. ViT-B (Base) contains 12 transformer blocks, each of which has 12 heads and 768 hidden dimensions. We also compare four different pre- training methods. As shown in Figure 5 'VIT-X' 'CLIP-
76
74
X', 'SAM-X', and 'DINOv2-X' represent pre- pra-ingin the model by ImageNet- K classification (Deng et al. 2009). the model with the CLIP approach (Radford et al. 202T)
72
VIT-s
VIT.
CLI--S
CLIP.B
SAMBB
DINOv2.5
DINOvZ.B
the model with the SAM approach (Kirillov et al. 2023),
and the model with DINOv2 (Oquab al. 2023). respec- tively. We observe that all the models show improvements compared with training from scratch (72.2 mloU). Larger
Figure 5: Comparison of different 2D pre- trained models.
models (X-B) show consistent improvements compared with smaller models (X-S). DINOv2-B yields the best performance in terms of semantic segmentation accuracy. This demonstrates the advantage of leveraging DINOv2's strong transferability properties in our method.
Loss design Table demonstrates the performance of our method with different loss configurations. We investigate the effectiveness of each loss component: Cea (2D knowledge transfer loss) and L. (multi-view consistency loss). The table shows that using either loss independently leads to an improvement performance. However, combining both losses yields the best results, validating the importance of incorporating both 2D knowledge transfer and multi-view consistency in our approach.
Masking ratio During pre-training, our approach involved masking the input point cloud guided by the corresponding RGB image. Following the conventional practice in masked autoencoder- baseed pre-training (He et al. 2022), we divided the image into regular non-overlapping patches. Then sample subset of patches and mask the remaining ones. The remaining part is projected onto the point cloud, forming the input for our method. We employ this masking operation as a data augmentation technique, which assists in the development of more robust features during pre-training. As evidenced in Table 4b an appropriate masking operation (i.e., 30%) during pre-training could enhance performance in downstream tasks.
View overlap ratio Table 4c we examine the impact of the view overlap ratio on the performance of our method. The view overlap ratio refers the percentage of shared field of view between the
8

---

Published as a conference paper at ICLR 2024
(a) Loss design
(b) Masking ratio
(c) View overlap
(d) View number
Lza
Cm
ScanNet
Ratio (%)
ScanNet
Overlap (%)
ScanNet
Num
L2a
L.
72.2
10
74.8
20
74.7
74.5
72.9
74.7
30
75.6
40
75.1
74.6
73.1
73.6
60
74.2
60
75.3
74.6
73.3
75.6
80
73.5
80
75.4
74.7
73.6
Table 4: Ablation studies of our design choices. We report the best val mloU result on ScanNet. Please refer to Section 4.3 for a detailed analysis.
two views used for multi-view consistency. important note that this experiment, we only use one candidate second view. Compared to the results without the multi-view consistency loss (74.7 mloU), overlaps between 40% and 80% all result improvements. However, when the overlap too small, specifically, at 20%, fail to observe any improvement. We believe this is because when the overlap is insufficient, multi-view correspondences become too sparse, obtaining very limited 3D signals. As result, our default setting selects views with overlaps in the range of [40%, 80%] as candidate second views, which ensures a balance between shared and unshared information for robust model performance.
Number of views In Table 4d we investigate the impact of using different numbers of candidate second views. We experiment with view, 2 views, 4 views, and views. Led denotes only adding 2D knowledge transfer loss, while &. represents only adding multi-view consistency loss. We find that incorporating more views leads improved performance for both modules. However, it shows minor performance variations for 2D knowledge transfer module. Our interpretation is that since the loss applied on each view independently, the different number candidate views exert minimal influence. Note that we did not experiment with more views we observed that when the number of views exceeds 5, some candidates have very small overlap with the first scan, which, as proven in the previous paragraph, impede the learning of the multi-view consistency module.
2D knowledge transfer loss We proceed to investigate optimal locations to enforce the 2D knowledge transfer loss. We explore the impact of incorporating the loss starting from different layers of fx. As shown in Figure 6 adding the feature-based loss starting from the third layer yields the best performance in terms of semantic segmentation val mIoU on ScanNet. When adding the loss on the first and sec- ond layers, the model's 's performance is negatively impacted. We believe this is because the inputs of the 2D tre-trained model and the 2D neural network differ. Forcefully adding the loss on the early layers may not be beneficial. Further- more, the performance drops when the loss is added starting from the fourth layer and beyond. This demonstrates that only regularizing the last few layers weakens the knowledge transferred from the pre-trained networks.
75
74
73
72
4
6
10
12
starting layer
Figure 6: Comparison of 2D knowledge
transfer loss on different starting layers.
5 CONCLUSION AND LIMITATION
this paper, we present novel approach to 3D point-cloud pre-training. We leverage pre-trained networks to supervise 3D pre-training through perspective projection and hierarchical feature-based 2D knowledge transfer. We also introduce a novel pre-task k to enhance the geometric structure through correspondence prediction. Extensive experiments demonstrate the effectiveness of approach.
Potential avenues for future work include exploring additional pre-training data, such as NYUv2 (Cou- prie al., 2013), and investigating the effect of increasing the size of the feature encoding network. Furthermore, we plan to extend our approach to outdoor settings. MVNet opens up new avenues for 3D poont-cloud pre- pre-training and provides promising direction for future research.
Acknowledgement. We would like to acknowledge the gifts from Google, Adobe, Wormpex AI. and support from NSF ITS-2047677, 1934999,,32, CCF-2019844, and IARPA WRIVA program.

---

Published as a conference paper at ICLR 2024
REFERENCES
Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thi- lakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised mrossl contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9902-9912, 2022.
I. Armeni, A. Sax, A. R. Zamir, and S. Savarese. Joint 2D-3D-Semantic Data for Indoor Scene Understanding. ArXiv e-prints, February 2017.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint aoXtv:/512..3312,,20 2015. 2015.
Yujin Chen, Matthias Nießner, and Angela Dai. 4dcontrast: Contrastive learning with dynamic correspondences for 3d scene understanding. In Computer Vision- 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pp. 543-560. Springer, 2022.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR'05) volume 1, pp. 539-546. IEFE, 2005.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3075-3084, 2019.
Camille Couprie, Clément Farabet, Laurent Najman, and Yann LeCun. Indoor semantic segmentation using depth information. arXiv preprint arXiv:1301.3577 2013.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias NießBne. Scannet: Richly-annotater 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Morteza Daneshmand, Ahmed Helmi, Egils Avots, Fatemeh Noroozi, Fatih Alisinanoglu, Hasan Sait Arslan, Jelena Gorbova, Rain Eric Haamer, Cagri Ozcinar, and Gholamreza Anbarjafari. 3d scanning: A comprehensive survey. arXiv preprint 1001. 2018.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142 11142-1115,, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1422-1430, 2015.
Runpei Dong. Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? arXiv preprint arXiv:2212.08320 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
10

---

Published as a conference paper at ICLR 2024
Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. MVTN: multi view transformation network for 3d shape recognition. In 2021 IEEE/CVE International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pp. 1-11. IEEE, 2021. doi: 10. 1109/CCC48992.202 2020449822220200000777 URL htttp::///// 1t10///...1/////////222221 00007
Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Voint cloud: Multi-view point cloud representation for 3d understanding. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL htttps://openreviie net net./pdffii--IIGGffpi.c..
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15587-15597, 2021.
Ji Hou, Xiaoliang Dai, Zijian He, Angela Dai, and Matthias Nießner. Mask3d: Pre-training 2d vision transformers by learning masked 3d priors. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 13510-13519, 2023.
Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6535-6545, 2021.
Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. Cotr: Correspon- dence transformer for matching across images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6207-6217, 2021.
Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 3d shape segmentation with projective convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 6630-6639 IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017. 702
URL URL http::ttp////t..... /tetps://do.... httpp:://o...og////oo
1109/CVPR.
Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi Nishida. Rotationnet: Joint object categoriza- tion and pose estimation using multiviews from unsupervised viewpoints. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18- 22, 2018, pp. 5010-5019. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CV/RR2001 2018 00526..
thecvf. http://c//coocctccccc/ceo 0b ject CVPR 20tp:/__011_ppprr..._ html
201 8 httptt///////tmmmmme /aaamm//KKaezaaaa www.aaaeaaaaaottttttineeeeeeeeee www.tttttteeeetttooonttnnnnennet
Brian R Kent. 3D scientific visualization with blender. 2015.
Alexander Kirillov. Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.00 02644,, 2023.
Abbijit Kundu, Xiaoqi Yin, Alireza Fathi, David A. Ross, Brian Brewington, Thomas A. Funkhouser, and Caroline Pantofaru. Virtual multi-view fusion for 3d semantic segmentation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.). Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIV, volume 12369 of Lecture Notes in Computer Science, pp. 518-535. Springer, 2020. doi: 10.1007/ 31. URL https: /ooi org 1007/ 97 3 030 58586-C
3

---

Published as a conference paper at ICLR 2024
Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrimination for self- self-supervised learning point clouds. 2022.
Yueh-Cheng Liu, Yu-Kai Huang, Hung- Yueh Chiang, Hung- Su, Zhe-Y Yu Liu, Chin-Tang Chen, Ching- Yu Tseng, and Winston H Hsu. Learning from 2d: Contrastive pixel-to-poine knowledge transfer for 3d pretraining. arXiv preprint arXiv:2104.0465 2021a.
Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2949-2958, 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 10X0501711.05101, 2017.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2906-2917, 2021.
OpenAL. Gpt-4 technical report, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 2023.
Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In Computer Asion-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part Il, pp. 604-621. Springer, 2022.
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEF conference computer vision and pattern recognition, pp. 652-660, 2017a
Charles Qi, Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1277-9286, 2019.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017b.
Haozhe Qi, Chen Feng, Zhiguo Cao, Feng Zhao, and Yang Xiao. P2b:
Point-to-box network for 3d object tracking in point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6329-6338, 2020.
Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. Advances in Neural Information Processing Systems, 35:23192 22204, 2022.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 151 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- 8748-8763. PMLR. 2021.
Yongming Rao, Jiwen Lu, and Jie Zhou. Global-local bidirectional reasoning for unsupervised representation learning of 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5376-5385, 2020.
12

---

Published as a conference paper at ICLR 2024
Yongming Rao, Benlin Liu, Yi Wei, Jiwen Lu, Cho-Jui Hsich, and Jie Zhou. Randomrooms: unsupervised pre-training from synthetic shapes and randomized layouts for 3d object detection. In Proceedings of the IEEE/CVP International Conference Computer Vision, pp. 3283-3292, 2021.
Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pp. 6620 6629. IEEE Computer Society, 2017. doi: N0.1100/CVPR.2017..000
hRttss:: org/ 11..0.1100/V/E. 2017. 701
Jonathan Sauder and Bjarne Sievers. Self-super deep learning on point clouds by reconstructing space. arXiv preprint arXiv:1901.0833: 2019.
Jiayao Shan, Sifan Zhou, Zheng Fang, and Yubo Cui. Ptt: Point-track-rransfrmmee module for 3d single object tracking in point clouds. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. -31--316 IEEE, 2021.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 567-576, 2015.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller Multi-view con- volutional neural networks for 3d shape recognition. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 945-953, IEEE Computer Society, 2015, doi: 10.1 111100///CV22015.1144 URL https: /ttp:::///ooooooo doi wtw.:/oo....oooooo
2CCV.2015.112
Liyao Tang, Yibing Zhan, Zhe Chen, Baosheng Yu, and Dacheng Tao. Contrastive boundary learning for point cloud segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8489-8499, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302,13977 2023.
Mikacla Angelina Uy, Buang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. pp. 1588-1597. IEEE, 2019.
Haiyang Wang, Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, Liwei Wang, et al. Cagroup3d: Class-aware grouping for 3d object detection on point clouds. Advances in Neural Information Processing Systems, 35:29975-29998 2022a.
Hanchen Wang. Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9782-9792, 2021.
Peng-Shuai Wang, Yang Liu, XiaX Guo, Chun-Y Yu Sun, and Xin Tong. O-CNN: octree- based convolutional neural networks for 3d shape analysis. ACM Trans. Graph., 36(4):72:1-72111 2017. doi: 10. 145/3072959 3073368. URL TtRL:bttss://eed /tttpp:://ol...r/....11111111 114//10.1145/3072333 /055//00255522
3073660
Peng-Shuai Wang, Yu-Qi Yang, Qian-Fang Zou, Zhirong Wu, Yang Liu, and Xin Tong. Unsupervised 3d learning for shape analysis via multiresolution instance discrimination. ACM Trans. Graphic, 2020.
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. arXiv preprint arXiv:2205.122: 12952, 2022b.
13

---

Published as a conference paper at ICLR 2024
Xin Wei, Ruixuan Yu, and Jian Sun. View-gen: View-based graph convolutional network for 3d shape analysis. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle WA, USA, Jume 13-19, 2020, pp. 1847-1856. Computer Vision Foundation / IERE, 2020. doi: 10. WWW.CC/VPP220002220000000
URL htttps::/opeeacceess thecvf. www.commm///..oo comm/conentt/.mm CVPR 2020 22200/ttmlll.m httm//tmmmmmmmmmmmmmmmmmm / View- GCN Vi Vi-w-Bl Ba sed__radddpdheeeeeee
Convolutional Network for 3D Shape Anwlysinnnasaa.aa CVPR 2020 paper.
Aoran Xiao, Jiaxing Huang. Dayan Guan, Xiaoqin Zhang, Shijian Lu, and Ling Shao. Unsupervised point cloud representation learning with deep neural networks: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2023.
Saining Xie, Jiatao Gu, Demi Guo, Charles Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European Conference Computer Vision, pp. 574-591. Springer, 2020.
Chenfeng Xu, Shijia Yang, Tomer Galanti, Bichen Wu, Xiangyu Yue, Bohan Zhai, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Image2point: 3d point-cloud understanding with 2d image pretrained models. arXiv preprint orXiv:2106.044 04000 2021.
Mingye Xu, Mutian Xu, Tong He, Wanli Ouyang, Yali Wang, Xiaoguang Han, and Yu Qiao. Mm- 3dscene: 3d scene understanding by customizing masked modeling with pnoemattvv--reseevve reconstruction and self-distilled consistency. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3380-4390, 2023.
Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1 1179-1 1189, 2023.
Siming Yan, Zhenpei Yang, Chongyang Ma, Haibin Huang, Etienne Vouga, and Qixing Huang. Hpnet: Deep primitive segmentation using hybrid representations. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2753-2762, 2021.
Siming Yan, Yuqi Yang. Yuxiao Guo, Hao Pan, Peng- Wang, Xin Tong, Yang Liu, and Qixing Huang. 3d feature prediction for masked-auttoencoder--beee point cloud pretraining, arXiv preprint arXiv:2304.066% 06910, 2023a.
Siming Yan, Zhenpei Yang, Haoxiang Li, Chen Song, Li Guan, Hao Kang, Gang Hua, and Qix- ing Huang. Implicit autoencoder for point-cloud self- sulf-super representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14530-14542, 2023b.
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 206-215, 2018.
Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. scalable active framework for region annotation in 3d shape collections. ACM Trans. Graph., 35(6), 2016.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre- training 3d point cloud transformers with masked point modeling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19313-19322. 2022.
Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. arXiv preprint 2rXiv:2255.44444 14401, 2022a.
Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders. arXiv preprint 2rXiv:2212.06785, 2022b.
Yabin Zhang, Jiehong Lin, Chenhang He, Yongwei Chen, Kui Jia, and Lei Zhang. Masked surfel prediction for self-supervised point cloud learning. trXiv:2207.03111 2022c.
14

---

Published as a conference paper at ICLR 2024
Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. arXiv preprint arXiv:2101.06691, 2021.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16259-16268, 2021.
Chaoda Zheng, Xu Yan, Jiantao Gao, Weibing Zhao, Wei Zhang, Zhen Li, and Shuguang Cui. Box-aware feature enhancement for single object tracking on point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13199-13208, 2021.
Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei A. Efros. Learning dense correspondence via 3d 3d-guided cycle consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Chengxu Zhuang, Siming Yan, Aran Nayebi, and Daniel Yamins. Self-supervised neural network models of higher visual cortex development. In 2019 Conference on Cognitive Computational Neuroscience, pp. 566-569. CCN, 2019.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, (18(3):2014199111 2021.
15


Reviews:
[{"review_id": "mzVv4n6QaZ", "reviewer": "Reviewer_RdJt", "summary": "This paper proposes a new learning framework for point cloud pre-training. The core motivation is to learn a view-consistent 3D feature representation. To enhance the pre-training effectiveness, the authors introduce pre-trained 2D image learning networks for knowledge transfer and also develop an auxiliary task of building pixel-space correspondences. Experiments on various downstream tasks demonstrate the effectiveness of the proposed pre-training method.", "questions": "For different experiments, the authors are also suggested to report their results of training from scratch, such that readers can better observe the relative gains after pre-training the backbone using the proposed method.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "(1) The paper is well-written and easy to follow. The working mechanism and motivation of each module is clearly explained. \n\n(2) The proposed learning pipeline and pretext task are technically sound. \n\n(3) The resulting supervised fine-tuning performances are encouraging.", "weaknesses": "(1) In practice, we may adopt various backbone models for point cloud learning. This requires a generic pre-training scheme that can be used to enhance the feature extraction capability of various point cloud backbones. However, in the manuscript, only the SR-UNet backbone is involved. Hence, the authors should explore more types of backbones for pre-training to validate the proposed method. Note that it is not convincing enough to only give a few results in Table 2 of the supplementary material without detailed explanations and more comprehensive experiments. \n\n(2) Furthermore, the experimental comparison with previous point cloud pre-training approaches is problematic because the adopted backbone models differ. To evaluate the effectiveness of the proposed pre-training framework, the authors should make sure that the performance gains are not from the stronger backbone. \n\n(3) For scene-level experiments, the authors perform pre-training on ScanNet and fine-tune on ScanNet, S3DIS, and SUN RGB-D. In fact, the pre-training dataset and the fine-tuning datasets are all indoor room data, with small domain gaps. Therefore, the actual transferability, which is known to be a key consideration factor for a pre-training method, is questionable. Some cross-domain verifications are needed, such as fine-tuning on some outdoor datasets.\n\n (4) For object-level experiments, the pre-training dataset Objaverse (with 800K objects) is much larger than ShapeNet/ModelNet used in previous pre-training methods. Therefore, the performance comparison is unfair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CbKwpOagvh", "reviewer": "Reviewer_GBMb", "summary": "This paper introduces a 3D representation learning approach that involves projecting 3D dense features into 2D images. This method employs multi-level feature maps for cross-modal knowledge distillation and employs coordinate mapping as a pretext task for achieving multi-view consistency learning. Extensive experiments demonstrate the effectiveness of both modules in various downstream tasks, including 3D shape classification, part segmentation, 3D object detection, and semantic segmentation.", "questions": "1. Why not directly use the transformation matrix of two views as training targets within the multi-view consistency module?\n\n2. Is voting strategy employed in the classification of object point clouds? \n\n3. I am curious to understand why SAM yields inferior performance as a 2D encoder compared to DINOv2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method exhibits a high degree of applicability, yielding favorable results across indoor scene-level datasets such as ScanNet and S3DIS, as well as at the object level dataset such as ScanObjectNN.\n\n2. The approach of projecting 3D features to 2D and learning consistency through point mapping aligns with physical intuition.\n\n3. The paper is well-written and ablation experiments are well-designed.", "weaknesses": "1. Given that the model can generate dense features containing rich semantics, I strongly recommend incorporating zero-shot experiments, including zero-shot classification experiments on datasets such as ModelNet and LVIS, as well as zero-shot semantic segmentation experiments on ScanNet and S3DIS.\n\n2. The classification experiments at the object level lack references and comparisons to recent works, such as ULIP-2 [Liu et al., 2023] and ReCon [Qi et al., 2023]. Furthermore, MVNet is trained on Objaverse and includes multiple versions with different parameters, making the comparisons in Table 3 unfair in terms of both the training dataset and model parameters.\n\n3. Some papers are repeatedly cited in the bib, such as: \"Charles R. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas. Deep hough voting for 3d object detection in point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019a.\"\n\n4. The result of ScanObjectNN in Figure 1 is different from the result in Table 3.\n\n[Qi et al., 2023] Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining. In ICML.\n\n[Liu et al., 2023] ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding. arXiv preprint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FtmE3IogJT", "reviewer": "Reviewer_JZVU", "summary": "This paper proposes to leverage the pre-trained 2D **networks** instead of data for 3D point cloud pretraining. Specifically, a 3D network is first applied to extract 3D feature volumes that are then projected into image-level embeddings with the help of depth information. Next, the key idea is to make the projected image features close to their counterparts that are pretrained using powerful 2D vision foundation models such as Dinov2, CLIP, SAM. Additionally, to realize multi-view consistency, another module is adapted to maintain the point correspondences across different views, yielding the whole proposed method called MVNet. For verifying the effectiveness of MVNet, experiments are conducted on various benchmarks and tasks.", "questions": "This work shows great potential to serve as a new pretraining method for 3D point cloud processing. Hope the code and the pretrained models could be public upon the acceptance of this paper.\n\nThe major concerns are detailed in the weaknesses part. The missing discussions, experimental results and qualitative results are expected to be provided during the rebuttal.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-organized, well-written, and easy to follow.\n2.\tThe key idea to leverage the pre-trained 2D networks and learn the 3D-2D correspondence at the feature level is interesting and effective. Such pretext task design provides a new direction for future research on 3D pretraining.\n3.\tThe proposed method achieves **superior results** on various benchmarks and tasks.\n4.\tThe supplementary material provides more experimental results and states the broader impacts. Additionally, the qualitative results of multi-view consistency predictions further prove the effectiveness of the proposed method.\n5.\tSome possible limitations and future potentials are also discussed.", "weaknesses": "1.\tThe **masking ratio** is *only discussed* in the ablation study, which is very confusing. Some more details about this masking operation should be discussed in the method section. For example, why do you use this operation? Why can it benefit the proposed method?\n2.\tThere are several **important recent related works** [1, 2] that should be discussed in Related Works and compared in the tables in the main paper.\n3.\tThe proposed method introduces two modules, thus the **#params and FLOPs** during pretraining (all the parts that need to be trained online in Fig. 2 should be counted) need to be provided and compared with other methods to show the efficiency of the proposed method.\n4.\tThe proposed method requires the **pre-processing** on RGB-D scans to generate point clouds. This operation may bring extra processing time and disk usage. How long will it take? How much extra storage will this need? This should be at least discussed in the supplementary material.\nAlso, this may be a limitation of the proposed method, which should be mentioned in the limitation section.\n5.\tSome **qualitative/visualization results** of different tasks such as 3D segmentation, and detection should be provided in the supplementary material.\n\n**Refs**:     \n[1] Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors. CVPR 2023.    \n[2] MM-3DScene: 3D Scene Understanding by Customizing Masked Modeling with Informative-Preserved Reconstruction and Self-Distilled Consistency. CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MyhYUZ9De5", "reviewer": "Reviewer_1Vvn", "summary": "The paper proposes a pre-training method for 3D point clouds that leverages 2D pre-trained foundation models. The method performs multiview consistency pre-training with a distillation loss from 2D pre-trained model. The experiments show the effectiveness of this method on several tasks on point cloud data.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The major strength of the paper is the effectiveness and simplicity of the method. Although I think the paper does not have a big technical contribution, since knowledge distillation from 2D foundation models is becoming a common approach, I like the paper since it is simpler than other concurrent works [1] and shows significant improvements. \n\n[1] Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Models, Chen and Li 2023", "weaknesses": "Although I like the concept, I think the evaluation should be improved. In particular, I am concerned about the weight initialization scheme used. The paper states that only the weights of the encoder are re-used for initialization since it leads to better performance. However, all previous approaches have used the pre-trained weights from the encoder and decoder. Therefore, it is difficult to assess if this pre-training setup improves over previous work. Since this work uses the same backbone as previous pre-training works and these methods provide pre-trained weights, it would be easy and necessary to provide a comparison of fine-tuned models for previous works where only the encoder is reused. Moreover, numbers for a fine-tuned model with the proposed method where the decoder weights are also reused should be provided. With this experiment, we will be able to assess if this is only necessary for the proposed pre-training method or if all pre-training strategies suffer from it. More importantly, we will be able to assess if the described improvement on the paper really comes from the pre-training strategy or from the fine-tuning setup used. I would rate the paper as marginally above acceptance and I would change my rating after rebuttal when these numbers are available.\n\nMoreover, it is not clear the masking ratio ablation experiment. There is no description in the methods section of any masking applied during pre-training, so I think it will be necessary to clarify how this masking is applied during pre-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)