Let Images Give You More: Point Cloud Cross-Modal Training for Shape Analysis
Authors: Xu Yan
Heshen Zhan
Chaoda Zheng
Jiantao Gao
Ruimao Zhang
Shuguang Cui
Zhen Li
Venue: NeurIPS 2022 Accept (2022)

Abstract: Although recent point cloud analysis achieves impressive progress, the paradigm of representation learning from single modality gradually meets its bottleneck. In this work, we take a step towards more discriminative 3D point cloud representation using 2D images, which inherently contain richer appearance information, e.g., texture, color, and shade. Specifically, this paper introduces a simple but effective point cloud cross-modality training (PointCMT) strategy, which utilizes view-images, i.e., rendered or projected 2D images of the 3D object, to boost point cloud classification. In practice, to effectively acquire auxiliary knowledge from view-images, we develop a teacher-student framework and formulate the cross-modal learning as a knowledge distillation problem. Through novel feature and classifier enhancement criteria, PointCMT eliminates the distribution discrepancy between different modalities and avoid potential negative transfer effectively. Note that PointCMT efficiently improves the point-only representation without any architecture modification. Sufficient experiments verify significant gains on various datasets based on several backbones, i.e., equipped with PointCMT, PointNet++ and PointMLP achieve state-of-the-art performance on two benchmarks, i.e., 94.4% and 86.7% accuracy on ModelNet40 and ScanObjectNN, respectively.

Let Images Give You More: Point Cloud Cross-Modal Training for Shape Analysis
Xu Yan 1.2t, Heshen Zhan 1,2f, Chaoda Zheng Jiantao Gao", Ruimao Zhang", Shuguang Cui2.1.5, Zhen Li FNii, CUHK-Shenzhen, SSE, CUHK-Shenzhen SDD, CUHK Shennhen,,,,, 'USV, Shanghai University, *Penchhngg Lab fxuyan 1 @1 flink. heshenzhan@link cuhh edu cn
Abstract
Although recent point cloud analysis achieves impressive progress, the paradigm representation learning from a single modality gradually meets its bottleneck. this work, we take step towards more discriminative 3D point cloud representation by fully taking advantages images which inherently contain richer appearance information, e.g., texture, color, and shade. Specifically, this paper introduces a simple but effective point cloud cross-modality training (PointCMT) strategy, which utilizes view- hiew-images, Le.,, rendered or projected 2D images of the 3D object, to boost point cloud analysis. In practice, to effectively acquire auxiliary knowledge from view images, we develop teacher-student framework and formulate the cross- modal learning as knowledge distillation problem. PointCMT eliminates the distribution discrepancy between different modalities through novel feature and classifier enhancement criteria and avoids potential negative transfer effectively. Note that PointCMT effectively improves the point-only representation without architecture modification. Sufficient experiments verify sigmificant gains on various datasets using appealing backbones, i.e., equipped with PointCMT, PointNet++ and PointMLP achieve state-of-the-as performance on two benchmarks, i.e., 94.4% and 86.7% % accuracy on ModelNet40 and ScanObjectNN, respectively. Code will be made available at https: // thub. com homm/Zhannieshen/oonnne PoinnCCTT.
1 Introduction
As the fundamental 3D representation, point clouds have attracted increasing attention for various applications, e.g., self-driving [2, 33, 34], robotics perception [7, 13, 6), elc. Generally, point cloud consists of sparse and unordered points in the 3D space, which is significantly different from a 21) image with a dense and regular pixel array. Prior studies treat the understanding of 2D images and 3D point clouds as two separate problems, and both have their own merits and drawbacks. Concretely, rich color and fine-grained texture easily obtained in 2D images, but they are ambiguous in depth and shape sensing. Previous works extract features on images through convolution neural networks (CNN). In contrast, point clouds are superior in providing spatial and geometric information but only preserve sparse and textureless features. Several prior studies process features on unstructured point clouds through local aggregation operators [34, 43]. It natural raise a question: Could we use the rich information hidden in 2D images to boost 3D point cloud shape analysis?
To address the above issue, one straightforward way to leverage the benefits of both images and point clouds, i.e., fusing information from two complementary representations with task-specific design [58, 24, 10, 32, 42]. However, utilizing additional image representation requires designing
'CCrresponding author: Zhen Li. Equal first authorship.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Rendered from CAD
Perspective Projection
Trained with point clouds only
94.5
Truined with PoinCMT
Images
94
93.5
Cross-Modal Training
93
Point Clouds
"Airplane"
92.5
Point Cloud Analysis Models
92
"Sofa"
DGCNN
RS-CNN
PointNet*+
(a)
(b)
Figure 1: (a) Our proposed general Cross- Training (Poin(CMT) strategy. It introduces priors from images into point cloud shape analysis models only in the training stage without any baseline model modification. (b) Classification accuracy (%) on ModelNer40 with or without training with our proposed PoinCCMT strategy. Noticeable improvements can be observed.
a multi-modal network, which takes the extra image inputs in both training and inference phases. Moreover, the exploiting extra astra-images is usually computation-intensii and paired-images are usually unavailable during inference. Thus, multi-modal learning meets its bottleneck in many aspects.
This paper tries to the barrier of cross-modal learning between images and point clouds. Inspired by knowledge distillation (KD) that achieves knowledge transfer from a teacher model to student one, formulate the cross-modal learning representations learned by images and point clouds. However, previous KD approaches usually assume that the training data used by the teacher and student are from the same distribution [17]. Thus, since sparse and disordered point clouds represent visual information different from images, naive feature alignment between two representations appeals cause limited gains or negative transfer for the cross moosl scenario. To this end, we design a novel framework for cross-modal KD and propose the point cloud cross-modal training strategy, i.e., PointCMT in Figure (a), which distills features derived from images into the point cloud representation. Specifically, multiple view images for each 3D object can be generated through either rendering the CAD model or conducting perspective projection on the point cloud from different viewpoints. These free auxiliary images are fed into the image network obtain the global representations for the object. Besides, feature and classifier enhancements are conducted between the point cloud and image features, in which the newly proposed criteria effectively avoid negative transfer between different modalities, i.e., directly applying [17] hampers the performance on ModelNet40. After training, the model gains higher performance, only taking point clouds as input without architecture modification.
KD problem, conducting alignment between sample
Compared with multi-modal approaches, our solution has the following preferable properties: 1) Gen- erality: It can be integrated with arbitrary point cloud analysis models without structural modification. 2) Effectively: It significantly boosts the performance upon several baseline approaches, e.g., Point- Net++ [34] achieves of-te-of-the-arr 94.4% from 93.4% overall accuracy on ModeINet40, as shown in Figure/ (b). 3) Efficiency: Our PointCMT only utilizes auxiliary image data in the training stage. After training, the enhanced 3D model infers without image inputs. 4) Flexibility: The extensive experiments illustrate that PointCMT performs superior even without colorized and dedicated ren- dered images, í.e., can also greatly improve the performance when using images directly projected by sparse and textureless point clouds. Thus, it provides an alternative solution to enhance the point cloud shape analysis when the additional rendered images are not accessible.
In summary, our contributions are: 1) This paper formulates cross-modal learning on point cloud analysis as knowledge distillation problem, where we utilize the merits of texture and coler-aware 2D images to acquire more discriminative point cloud representation. 2) We propose point cloud cross-modal training, i.e., PointCMT, strategy with corresponding criteria to boost point cloud models during the training stage. 3) Extensive experiments on several datasets verify the effectiveness of our approach, where PointCMT greatly boosts several baseline models even on the state-of-the- e.g. PointNet++ [34] trained with PointCMT gains 1.0% and 4.4% accuracy improvements on ModelNet40 [48] and ScanOhje,,NN, respectively. Even based upon PointMI.P [31], it increases its accuracy by 1% to 86.7 % ScanObjectNN dataset.

---

2
Related Works
3D Shape Recognition Based on Point Clouds. These stream methods directly process raw point clouds input (also called point-based methods). They are pioneered by PointNet [33], which approximates permutation-invariann set function using a per-point Multi-Layer Perceptron (MLP) followed by max-pooling layer. Later on, point-based methods aim at designing local aggregation operators for local feature extraction. Specifically, they generally sample multiple sub-points from the original point cloud, and then aggregate neighboring features of each sub sub-point through local aggregation operators, in which point-wise MLPs [34, 31 adaptive weight [44, 46, 27] and pseudo grid based methods [39, 20] are proposed. More recently, there are some attempts to utilize non-local operator [54], or Transformer [60, 14] to mine the long distance dependency. This paper also follows the paradigm of point-based methods conduct point cloud shape analysis.
3D Shape Recognition Based on Images. Since point clouds are irregular and unordered, some works consider projecting the 3D shapes into multiple images from different viewpoints (also called view-based methods) and then leverage the well-developed 2D CNNs to process 3D data. One seminal work of multi-view learning is MVCNN [38]. It extracts per-view features with shared CNN in parallel, then aggregates via a view-level max-pooling layer. Most follow-up works propose more effective modules aggregate the view-level features. For instance, some of them enhance the aggregated feature by considering similarity among views [1 1, 56] while others focus on the viewpoint relation [45, 22]. The above methods usually utilize ad-hoc rendered images for each 3DD shape, including shade and texture for the surface mesh. Therefore, they generally achieve higher performance than point-based methods using sparse point clouds as input. Recently, [12] proposes a simple but effective method (SimpleVien through directly projecting sparse point clouds onto image planes, achieving comparable performance with point-based methods. Inspired by view-based methods, this paper takes advantage of extracted image features from the view-based method, which are utilized as prior knowledge to boost point cloud shape analysis.
Knowledge Distillation. Knowledge distillation (KD) aims at compressing a large network (teacher) to compact and tiny one (student), and boost the performance of the student at the same time. The concept was first shown by Hinton et al. [17], which trains a student by using the softened logits of teacher as targets. Over the past few years, several subsequent approaches [23, 1, 5, 40, 9, 61] use different criteria to align the sample representations between the teacher and student. However, almost all the existing works assume that the training data used by the teacher and student networks are from the same distribution. Our experiment illustrates that new biases and negative transfer will be introduced in the distillation process if cross- cross-modal data from different distribution (e.g., features extracted from unordered point cloud and regular grid image) is utilized directly on previous KDs.
Cross-Modal Knowledge Transfer. Cross- modal knowledge transfer in computer vision is relatively emerging field that aims to utilize additional modalities at the training stage and enhance the model's performance the target modality the inference. Recently, there are 3D-to-2D knowledge transfer approaches, which adopt geometric aware 3D features from point clouds to enhance the performance of 2D tasks through contrastive manner [18] or feature alignment [30]. Later on, approaches attempt to transfer priors in images to enhance 3D point cloud-related tasks, and some are designed for specific tasks. Concretely, [28, 26] propose the images-to-point contrastive pre-training, [50] inflats 2D convolution kernels to the 3D ones and [57, 59, 53] independently apply modal training for visual grounding, captioning and semantic segmentation. Inspired but different from the above, we are the first to conduct image-to-poing knowledge distillation for point cloud analysis.
3 Methodlogy
3.1 Problem Statement
Let P E RNN3 and y E R be the point cloud and ground-truth label of the 3D object. Its corresponding view-image counterparts can be denoted as I E VX where N, V and (H, W) are the number of points, number of view-images and image size, respectively. View images can be gained by rendering the 3D CAD model [38] perspective projecting the raw point cloud [12]. We denote T and S as the image and point cloud analysis networks, respectively, and regard them as the teacher and student in traditional knowledge distillation (KD). For these networks, we split each of them into two parts: (i) Encoders (i.e., feature extractors Enc (mg())() and Enc)**(.)), the output of
3

---

Feltures
Seporsiedd by: Conoaed Trush
View imaggss
Pre-trained Image Encoder
Cross-Modal Puint Generator
Luo
Image Features
3D Clasifter
Fouser
Cheattrr
(b) Training of CMPG
Point Cloud
Point Cloud
Encoder
dH
y
Fixed Cross-Modal
Point Generator
Let 4 Spenntaa by Ground Trust
you Point Clouds Reconstruction
Cross-modal Fectures
Training Inforaee
(a) PoinCCMT
(c) Feature Enhuncement
Figure 2: (a) The architecture of PointCMT. Multiple view images are gained through rendering the 3D CAD model perspective projecting the raw point cloud, and the pre-trained image network distills the knowledge to the point cloud analysis network via two matching processes. The first is feature enhancement, which aligns features through pre- pre-trained cross- moosl point generator through the process in (b). The second one is classifier enhancement, which aligns the output distribution of the point classifier by taking cross- modal features as inputs. (b) Training process of cross-modal point generator (CMPG). (c) Illustration of feature enhancement.
which at the last layer are global feature representations Frimg E TRD and Fats E RDD (ii) Classifiers, which project the feature representation Fimg and Fente into class logits through Cl (Is'msFimmg and Cls (FFtt), where Cls(-) denotes the classifier.
Since we formulate the cross modsl learning as a KD problem, its goal in the learning process is to distill the priors knowledge from the image into point cloud features, obtaining an image-enhanced ideal feature FKD. During the knowledge distillation, we parameterize the teacher and student networks with 0r and 0g, and denote the knowledge of the teacher network as Krr. From the Bayesian perspective, a neural network can be viewed as a probability model, e.g., P(y ([yl,,0 0s) for point cloud analysis model an example: given an input point cloud P, the network assigns an output probability with the parameters 0g. Therefore, if we want the student network guided by image knowledge on the input sample, our goal can be further reformulated as maximizing the probability P(F* PIP, .P.P 0s, Kr), which can also be used to measure the ability of student network extracting the feature with cross-modal information. To find the lower bound of the above probability, we define the discrepancy g between theoretically discriminative features Fiig and F! as
g P(Fimo [I, y; 0r) P(F [P, yi 0g, KT),
(I)
where Fimg and F are ideal features in specific modalities. In Lemma 1, we give the lower bound of cross modal learning as KD problem. The proof is provided in the supplementary material.
Lemma 1: By the definition above, P(FKD (P, yi 0s, KT) is bounded below by P(FKD II, yi 0T)
X g. where X is
= = PFFmm I. y: 0r) P(FKD (r
In Lemma 1, measures the compatibility of knowledge distillation of the image networks, and can be viewed as constant when the architectures are determined. P(FK y; 0r)) is also a constant when the parameters Or and the architecture of the point cloud analysis model are fixed, e.g., using pre-trained image network. Therefore, the Lemma ensures that during the knowledge distillation, one can maximize P(JKD (P, y; 0s, (0s,,T,, KT) through minimizing g. which gives a theoretical guarantee for the KD problem.
If we adopt previous KD studies in cross-modal scenario, they minimize g through making student directly approximate the teacher's features [23, 1, 5] or predict logits [17]. However, in common KD problem, the teacher and student are generally trained on the same dataset with an identical distribution [17]. Moreover, the teacher generally achieves better performance than the student. In contrast, the image and point cloud analysis models tend to learn different feature representations and logits distribution which are generally complementary. Direct alignment may cause a limited gain or even negative transfer. Moreover, previous KD approaches treat encoders and classifiers as a whole architecture since the teacher and student networks generally have the same components. However, the point cloud convolution encoder significantly different from the 2D CNN, but they have the same classifier design.

---

Algorithm 1: Process in point cloud cross-modal training (Poin(CMT)
Data:
The point clouds (PtR=1, corresponding ground truth labels { and view-images set {Z, JM where M is a numher of training data samples.
Stage I: Training image encoder and image classifier:
Taking view-images set {I, J= imput, image
encoder produce the image features (F which are then fed into image classifier and obtain prediction logits [Cls""u( mm (Fimm)) (CFss"(( } 2=1 supervised by ground-truth labels {
Stage II: Training cross-modal point generator: As shown in Figure 2 (b). the generator takes features
Mf Ji=1 1 as input, reconstructs point cloud (P!"")) MM and supervised by point clouds (P:)a= (F J
Stage III: Image priors assisted training: The point encoder takes point clouds {Pt) i= as input and generate point features (Frte) Ji-1. The point classifier gains prediction logits {Cl,
through taking features (JT
Ji=1- The feature enhancement align cross-modal features (Fi"M) Jizl and
{FFP's hi h through Feature Enhancement, and Classifier Enhancement enhance the point classifier via matching [Cls"*> (FFP)) )=== and (C16")((Fmm))
To solve the cross-modal KD problem, we propose PointCMT, which effectively solves the cross- modal learning problem. The workflow of PoinCCMT demonstrated in Figure 2 (a) and can be summarized as Algorithm 1. Specifically, there are three stages exist in PointCMT. In Stage I (Section 3.2), we train the image encoder and classifier using view-images and ground-trute labels. In Stage II (Section 3.3), we train the cross moosl point generator (CMPG) through image features, and we independently align features and logits of two modalities in stage III (Section 3.4), where the encoder and classifier of point cloud analysis network are both enhanced.
3.2 Learning Image Priors
For each 3D object, we use multiple view-images (i.e., rendered color images or projected images from raw point cloud) as additional data, and the whole process can be described as:
grmng A(CNN
Inspired by view-based learning approaches [38]. V view-images from I flow into shared- weights image feature extractor CNN(-) to obtain a series of vectors. By aggregating the view-level vectors via aggregation function J(.). we obtain global feature representation Fing from all images, which integrates shape information from multiple views. Finally, an image classifier maps the above global feature to gain prediction logits through Cls (Fimm), which is supervised by the ground truth label y through cross-entropy loss LCE.
3.3 Cross-Modal Point Generator
cross-modal point generator (CMPG) can be seen as a nonlinear transformation RD + N33 that maps the global feature representation Fiin acquired from images into the Euclidean space. Thus, can avoid potential negative transfer effectively by directly aligning cross-modas features from different distributions. In order to better learn image priors in the point cloud analysis network, we pre-train the CMPG through the image feature Finng E RDD, and fix it in the distillation stage. Figure 2 (b) illustrates the pre-training stage of CMPG. It takes the image feature input and reconstruct point cloud piing Nx3 which is supervised by the original point cloud P through Earth Mover's ' distance (EMD) [37]:
LEMD ping) min Σ |lp x(p)l.
(4)
+
PEP
where |P|= = |pimg| and P + ping is a bijection, i.e., for each point E P, ((-) finds sole point correspondence in Dimo, After pre-training, CMPG reconstructs point cloud through an image-relative feature representation.
3.4 Image Priors Assisted Training
During the training stage, three objectiveness should be optimized:
5

---

Classification Loss. In our PointCMT. arbitrary point cloud analysis models can be assembled. Generally, it should be designed through point encoder and a classifier, where the point encoder takes point cloud as input, generates the point cloud feature representation F and feeds it into the classifier Cls
o (.) to obtain class logits. Finally, the cross-entropy loss LCE is used as criteria.
Feature Enhancement Loss. Unlike previous KD methods that directly align features of teacher and student, we first transform the cross-modal features into Euclidean space. As shown in Figure 2 (c), the pre-trained CMPG independently transform Fing and FFpss obtaining two point clouds Pimg and Puta, respectively. After that, EMD loss is conducted on the two-point clouds as an objectiveness:
Feature = LEMD(PAts ,iim)) = min Σ (lp-((p)ll
(5)
PEPpte
where | |PDimo| and Φ -ppt= - Ping is a bijection. Compared with traditional T, loss, the EMD distance natural for solving an assignment problem for permutation-inval invvaaaatioonnniaaaantn point sets. For all but a zero-measure subset of point set pairs, the optimal bijection & is unique and invariant under the infinitesimal movement of the points. Thus, EMD differentiable almost everywhere.
Classifier Enhancement Loss. In addition to supervising the point feature extractor through the above Feature Enhancement, further propose constraints conducted on the point classifier (as Figure 2 (a)). Specifically. the image feature generated by the teacher network is fed into the point classifier. in which the gradient only proppagaes to the point classifier. Besides, classifier enhancement is proposed to enable the point classifier to handle the image feature during the distillation. It aligns outputs logits the point classifier by using image and point features as inputs. This constraint is modified based on the distillation loss in Hinton ef al. [17] as in Equation (6), but in this case, two sets of logits come from the same classifier.
Liimtnn DKL (Cls'mg (Fimm)))
|[C (Frts)
(6)
where DEL (-1l)) KL. divergence. In contrast, our proposed classifier enhancement more suitable for the cross-modal scenario where great discrepancy exist between image and point cloud features. Concretely, the loss for classifier enhancement can be written as
LClassifier
D PCls (FIm9)|[CIs)
(Jrlx)). [|CIs=*((==)))
Final Loss. The final loss is a weighted sum of the above three losses L = LCE + aL C.Isssiffr where 30 and 0.3 are the weights adjust the ratios of each loss, respectively.
4
Experiment
4.1
Shape Classification on ModelNet40
Raw Point Cloud
Rendered from CAD
Projection
We firstly evaluate PointCMT syn- thetic dataset ModelNet40 [48], which is a large-scale 3D CAD model dataset.
ModelNNe40
Dataset Description and Processing, Mod- elNet40 is composed of 9, 3,843 train models and 2,468 test models 40 classes. For the input of the 3D network, we use point clouds provided by the official dataset, which the same as PointNet [33]. For the input of the
Raw Point Cloud
Projection w/ Colors
Projection
ScunObjec/NN
image network, we use 20 rendered view images from CAD models utilized in Rota- tionNet [22]. These images have a resolu- tion of 224 224. Since they consider both the mesh surfaces and illumination, they can provide more information to the 3D network. Selected samples of point clouds and corresponding multi-view images are shown in Figure 3.
Figure Different view-image generation strategies used in our experiment.
Implementation. For image network, we use ResNer-18 [16] pre-trained on ImageNet [8] as the feature extractor. Following MVCNN [38], we obtain the global shape feature by applying view- wise max-pooling to the view-level features. Finally, fully-connected layer is used to output the

---

Table I: Classification results on ModelNet40 dataset. With only Ik points, PointNet++ trained with PointCMT achieves state-of-the-an results on both class mean accuracy (mAce) and overall accuracy (OA) metrics. Here, 'pnt" and 'nor' denote points and normal vectors, respectively. The speed (samples/second) tested on one Tesla V100 GPU and four cores AMD EPYC 7351 @2 60GGZ CPUs, where denotes the results from the original paper. For PointNet++, we train it with protocol of RS-CNN [27] as mentioned in [12]. The best and second best are marked bold and underline.
Method
Input
#Points
mAcc(56)
OA(%)
Speed
Param.
PointNet [33]
pnt
1k
86.0
89.2
3.47M
PointNet++ [34]
pnt, nor
5k
91.9
1.47M
PointCNN [25]
pnt
1k
88.0
92.5
PointConv [47]
pnt, nor
1k
92.5
801
18.6M
KPConv [39]
pnt
7k
92.9
10T
15.2M
PointASNL [54]
pnt, nor
1k
93.2
PosPool [29]
pnt
5k
93.2
Point Transformer [60]
pnt
Ik
93.7
GBNet [36]
pnt
1k
91.0
93.8
112
8.4M
GDANet [51]
pnt
1k
93.8
14T
0.9M
SimpleView [12]
pnt
1k
93.9
2208
1.64M
CurveNet [49]
pnt
1k
94.2
15T
2.0M
PointML.P [31]
pnt
1k
91.4
94.5
139
12.6M
DGCNN [43] (bascline)
pnt
Ik
90.2
92.9
518
1.68M
RS-CNN [27] (baseline)
pnt 1k
89.3
92.9
2174
1.17M
PointNct++ [34] (baseline)
pnt 1k
90.1
93.4*
300
1.62M
DGCNN w/ PointCMT
pnt 1k
90.8 (40.6)
93.5 (+0.6)
518
1.8MM
RS RS-CNN w/ PointCMT
pnt
1k
90.1 (+0.8)
93.8 (+0.9)
2174
1.17M
PointNet++ w/ PointCMT
pnt
1k
91.2 (+1.1)
94.4 (+1.0)
300
1.62M
classification logits. During the training process of the above network, we use SGD as our optimizer with a learning rate of 0.01. The batch size is set to 128 for 50 epochs. After that, we fix the
image network and train the CMPG with Adam and 32 batch size for 50 epochs. In practice, CMPG
consists of three-layer MLP. For the point cloud analysis models, DGCNN [43] and RS-CNN [27]
are independently trained with the training strategies provided their official codes. PointNet++ [34] trained with strategy of RS-CNN [27] as [12] for better performance.
Comparison with State-of-the-arts. The classification results on ModelNet40 are shown Table 1,
where the overall accuracy (OA) and class
accuracy (mAcc) compared. The upper part of
the table illustrates the results of current state-of-te of-te-of-the-art methods, in which we use PointNet++ [34], baselines. We do not PointMLP [31] as baseline since it cannot robustly reproduce the highest results on ModelNer40, where the issue is mentioned in their open-sourced codes. For models trained from scratch, PointMLP [31] achieves the highest accuracy. As shown in the lower part of the table, after training with PointCMT, the performance of all baselines is greatly boosted, i.e., 1.0% improvement upon PointNet++ and 0.9% for RS-CNN and 0.6% for DGCNN. We also compare our method to several open-sourced methods and report the parameters and testing speed. As shown in the last columns of the table, though PointMLP gains 0.1 higher overall accuracy, its network consists of about 7.7x parameters and only achieves 46% speed of PointNet++. In contrast, PoinCCMT performs well on light-weighter models, which shows its great potential for real-time applications, e.g., scene parsing in autonomous driving.
RS-CNI [27] and DGCNN [43]
4.2 Shape Classification on ScanObjeetNN
Though ModelNet40 is the widely used benchmark for point cloud analysis, may not meet the realistic requirement due to its synthetic nature. To this end, we also conduct experiments on the ScanObjectNN benchmark [41], which real-world dataset.
Dataset Description and Processing. ScanObjectNN collects 2.902 objects from real-world indoor scenes ScanNet [7] and SceneNN [19], categorizing into 15 categories. Several variants are provided in the dataset, where the most challenging one is PB 23_5500___ RR, i.e., introducing perturbation objects 1,,416 and 2,882 data for training and test) via random translation, shift, rotation and scaling. Due to background, noise, and occlusions, this benchmark poses significant challenges to existing point cloud

---

Table 2: Classification on ScanObjectNN, We examine all methods on original objects and more challenging variant (PB_T50 RS). We train PointNet++ and PointMLP with protocol of [12] for fair comparison. The best and second best are marked in bold and underline. We train and test for four runs and report mean std results.
OBJ ONLY
PB T50 RS
Method
mAcc( (%)
OA(5)
mAco((5)
OA(%)
3DmFV [3]
73.8
58.1
63.0
PointNet [333]
79.2
63.4
68.2
SpiderCNN [52]
79.5
69.8
73.7
PointNet++ [34]
84.3
75,4
77.9
DGCNN [43]
86.2
73.6
78.1
PointCNN [25]
85.5
75.1
78.5
DRNet [35]
78.0
80.3
GBNet [36]
77.8
80.5
SimpleView [12]
86.2
89.0
80.8
PRANet [4]
79.1
82.1
MVTN [15]
82.8
PointNet++ [34] (baselinc)
85,4+0.2
87.4+0.1
75.5
79.2+00.2
PoinMMLP [3]] (baseline)
89.110..3
92.2-10.3
83.910.5
85.4
PointNet++ w/ PointCMT
89.0+00.3 (43.7)
91.6::0.2 (44.3)
79.9=0.3 (+4.4)
83.1=0.2 (+3.9)
PointMLP w/ PointCMT
91.8++0.2 (+2.6)
93.2++0. (+1.0)
84.4=0.4 (+0.4)
86.4=:0.3 (+1.0)
analysis methods. Furthermore, since the PB 1B_T5O___ RS dataset only preserves the spatial coordinates (XYZ) for each object while the other information, such as RGB, is discarded, we also compared original 2,902 objects OOB_), which includes additional RGB information. On both above datasets, we only use depth images through conducting perspective projection on raw point cloud as additional inputs, shown in the last column in Figure 3. Section 4.6 will discuss more results using projection with additional color information.
Implementation. All view- newwimmaess in ScanObjectNN are gained by the projection of raw point clouds, and we follow the structure of [12] only generate six images for PB _500 RS and OBJ OBJ_OBO_O_OO ONLY. We train the image network from scratch with batch size 32 and SGD optimizer for longer epochs of 1,000. The training strategy of CMPG the same as ModelNet40. For both point cloud models trained from scratch and with PointCMT, we use SGD optimizer for 1,000 epochs with batch size 32.
Comparison with State-of-the-arts. The results are shown in Table 2, where PointNet++ [34] and current state-of-the-art PointMLP [31] are chosen as our baselines. PointCMT significantly improves the performance on both class mean accuracy (mAcc) and the overall accuracy (OA), even on state-of-the-ar methods. Specifically, although background, noise, and occlusions exist on PB 18_SOO_ RS dataset, PointCMT still improves the overall accuracy of PointNet++ by 3.9%. Moreover, PointCMT also achieves state-of-fe saate-of-the-all the-hehttttt results on OBJ JBLL dataset. Note that there is no auxiliary information provided in images, and all view images are generated through the perspective projection of points coordinates. Nevertheless, PointCMT still dramatically increases the mAce of PointMLP from 89.4% to 92.0% (+2.6%).
4.3 Data Efficient Learning
We evaluate our approach under limited data scenarios in Table 3. Here, we only sample a small amount of training data in each category on ModelNet40, and evaluate the entire testing data. Our
Table 3: Data efficient lcarning on ModelNet40. We train PointNet++ [34] with small amount of training data and train with PointCMT.
Data percentage
Train from scratch
w/ PointCMT
PointCMT shows an even more signifi- cant gap when using small subset of the training data, again compared to Point- Net++ which trained from scratch. Es- pecially when facing only 2% and 10% of the training data, we achieve about 1.9% and 2.8%
2%
73.3
75.2 (+1.9)
5%
82.1
83.5 (+1.4)
10%
87.9 (+2.8)
20%
88.4
89.3 (+0.9)
improvements, respectively. This result illustrates that PointCMT provides more vital guidance for point cloud models in the low data regime.
8

---

Table 4: Ablation study on ModelNet40 and ScanObjetNN datasets. Overall accuracy (%) as metrics.
Model
FE
CE
ModelNet40
OBJ_ONLY
PB_T50_RG
X
X
93.4
87.5
79.4
/
x
93.8 (+0.4)
89.2 (+1.7)
82.5 (43.1)
PointNet++
X
94.0 (+0.6)
91.3 (+3.8)
82.3 (+2.9)
94.4 (+1.0)
91.8 (+4.3)
83.3 (+3.9)
4.4 Ablation study
The ablation results on three datasets summarized in Table 4, in which use PointNet++ as baselines. We first test the effectiveness feature enhancement (FE) and classifier enhancement (CE) in PointCMT. The results demonstrate that only using FE already significantly boosts the performance on both datasets, i.e., increases the overall accuracy by 0.4% and 3.1% on ModelNet40 and ScanObjetNN. Only using classifier enhancement (CE) improves the accuracy by around 0.6% and 2.9%. Finally, it is surprising that when we use both FE and CE during the training phase, achieves the best result of 94.4% and 83.3%, respectively.
4.5 Comparison with Knowledge Distillation Methods
To further verify the effectiveness of our proposed method compared with typi- cal teach-student architecture and other knowledge distillation manners, com- pare PointCMT with typical approaches of knowledge transfer in Table 5. Among all the methods, Hinton et al. [17] is pioneer study for knowledge distilla- tion, while Huang et al. [21] and Yang al. [55] recent works. As shown in the table, directly aligning features
Table 5: Comparison with knowledge distillation meth- ods. We compare overall accuracy (OA, 9) gained by PointNet++ on ModelNet40 and ScanObjectNN.
Method
ModelNet40
PB PB_T50, RS
Baseline
93,4
79.4
Hinton al. [17]
93.1 (-0.3)
81.8 (+2.4)
Huang et al. [21]
93.6 (+0.2)
82.0 (+2.6)
Yang al. [55]
93.9 (+0.5)
81.1 (+1.7)
PoinICMT (ours)
94.4 (+1.0)
83.3 (43.9)
modalities will cause negative transfer ModelNet40. This phenomenon does not appear on ScanObjectNN, since view images projected via point clouds may have smaller gap than rendered images of CAD models. Nevertheless, other KD techniques only achieve marginal improvement compared with PointCMT.
[17] between
4.6 Different View-image Generation
Table 6: Results through different view-image genera- tion. We compare overall accuracy (OA, %) on Model- Net40 and ScanObjectNN OBI_ONLY datasets.
In this section, compare results with different view-image generation strate- gies. As shown in Figure 3, multiple view-image types can be applied in our framework, and we compare the results in Table 6. As illustrated in the table, images rendered from the CAD model improve more compared with only using projection since the former provides additional shade and texture information. In contrast, we find out that using additional colors the OBJ ONLY dataset cannot boost performance. The reason is that OBJ ONLY dataset only contains 2,902 objects, and image networks are easier to overfit when using the color information.
View-images
ModelNer40
OBJ ONLY
Rendered from CAD
94.4
Projection
94.0
91.8
Projection w/ color
90.7
Conclusion
In this work, we propose a point cloud cross-modal training strategy named PointCMT. By exploit- ing some sophisticated architecture and reasonable criteria function, our PointCMT can boost the performance significantly for point cloud analysis methods on several benchmarks, outperforming previous methods by a large margin. We believe that our work can be applied to broader range of other scenarios in the future, such as 3D semantic segmentation and object detection. Meanwhile, our method provides an alternative solution to the comprehension of 3D scenes with severe texture details missing. It can improve performance through image priors and knowledge transfer.

---

Acknowledgment
This work was supported part by NSFC- Youth 61902335, by the Basic Research Project No. CTQB-KCZYZ-2021066 of Hetao Shenzhen HK S&T Cooperation Zone, by the National Key R&D by Shenzhen Outstanding Talents Training Fund, by Guangdong Research Project No. 20172T07X 152 and No. 2019CX0IX 104, by the Guangdong Provincial Key Laboratory of Future Networks Intelligence (Grant No. 2022B 12 2212010001), by the NSFC 6193 102200111999 11928899220 2046, by NSFC- Youth 62106154, by zelixir biotechnology company Fund, by Tencent Open Fund, and by ITSO at CUHKSZ.
Program China with grant D..2018YFF180005
References
[1] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Vari- ational information distillation for knowledge transfer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9163-9171,
[2] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickiti:: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE International Conference on Computer Vision, pages 9297-9307, 2019.
[3] Yizhak Ben-Shabat, Michael Lindenbaum, and Anath Fischer. 3dmfv: Three-dimensional point cloud classification in real-time using convolutional neural networks. IEEE Robotics and Automation Letters, 3(4):3145-3153) 2018.
[4] Silin Cheng, Xiwu Chen, Xinwei He, Zhe Liu, and Xiang Bai. Pra-net: Point relation-aware network for 3d point cloud analysis. IEEE Transactions on Image Processing. 30:4436 44448 2021.
[5] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4794-4802, 2019.
[6] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 3075-3084, 2019.
[7] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5828-5839, 2017.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lece, 2009.
[9] Xiang Deng and Zhongfei Zhang. Comprehensive knowledge distillation with causal interven- tion. Advances in Neural Information Processing Systems, 34, 2021.
[10] Khaled EI Madawi, Hazem Rashed, Ahmad El Sallab, Omar Nasr, Hanan Kamel, and Senthil Yogamani. Rgb and lidar fusion based 3d semantic segmentation for autonomous driving. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 7-12. IEEE, 2019.
[11] Yifan Feng, Zizhao Zhang, Xibin Zhao Rongrong Ji, and Yue Gao. Gvenn: Group-view convolutional neural networks for 3d shape recognition. In Proc. IEEE Conf. Comput. Vision Pattern Recognition, June 2018.
[12] Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with simple and effective baseline. In International Conference Machine Learning, pages 3809-3820. PMLR, 2021.
[13] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 9224-9232, 2018
10

---

[14] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):187-199, 2021.
[15] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages -11,2021,
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Pruceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NeurlPS Workshops, 2014.
[18] Ji Hou, Saining Xie, Benjamin Graham, Angela Dai, and Matthias Nießner. Pri3d: Can 3d priors help 2d representation learning? arXiv preprint arXiv:2104 11225, 2021.
[19] Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Yu, and Sii-Kit Yeung. Scenenn: A scene meshes dataset with annotations. In 2016 Fourth International Conference 3D Vision (3DV), pages 92-101. IEEE, 2016.
[20] Binh-Son Hua, Minh-Khol Tran, and Sai-Kit Yeung. Pointwise convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 984-993, 2018.
[21] Zhen Huang, Xu Shen, Jun Xing. Tongliang Liu, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-Shenn Hua. Revisiting knowledge distillation: An inheritance and exploration framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pages 3579-3558 2021.
[22] Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi Nishida. Rotationnet: Joint object cate- gorization and pose estimation using multiviews from unsupervised viewpoints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5010-5019, 2018.
[23] Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. Advances neural information processing systems, 31, 2018.
[24] Georg Krispel, Michael Opitz, Georg Waltner, Horst Possegger, and Horst Bischof. Fuseseg: Lidar point cloud segmentation fusing multi-modal data. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1874-1883, 2020.
[25] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x-transformed points. Advances neural information processing systems, 31, 2018.
[26] Zhenyu Li, Zehui Chen, Ang Li, Liangji Fang, Qinhong Jiang, Xianming Liu, Junjun Jiang, Bolei Zhou, and Hang Zhao. Simipu: Simple 2d image and 3d point cloud unsupervised pre-training for spatial-aware visual representations. AAAI 2021, 2021.
[27] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. delation-shape convolutional neural network for point cloud analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8895-8904, 2019.
[28] Yueh-Cheng Liu. Yu-Kai Huang, Hung- Yuch Chiang, Hung- Ting Su, Zhe-Yo Liu, Chin-Tam Chen, Ching-Yo Yu Tseng, and Winston H Hsu. Learning from 2d: Pixel-to-point knowledge transfer for 3d pretraining, arXiv preprint arXiv:2 04687, 2021.
[29] Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look local aggregation operators in point cloud analysis. In European Conference on Computer Vision, pages 222-342. Springer, 2020.
[30] Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu. 3d-to-2d distillation for indoor scene parsing. CVPR, 2021.
11

---

[31] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: simple residual MLP framework. In International Conference on Learning Representations, 2022.
[32] Gregory P Meyer, Jake Charland, Darshan Hegde, Ankit Laddha, and Carlos Gonzalz Sensor fusion for joint 3d object detection and semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[33] Charles R Qi, Hao Su, Kaichun Mo, and LeonidasJ Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[34] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In Advances in neural information processing systems, pages 5099-5108, 2017.
[35] Shi Qiu, Saced Anwar, and Nick Barnes. Dense-resolution network for point cloud classification and segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3813-388 3822, 2021.
[36] Shi Qiu, Saeed Anwar, and Nick Barnes. Geometric back-projectio network for point cloud classification. IEEE Transactions on Multimedia, 2021.
[37] Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as a metric for image retrieval. International journal of computer vision, 40(2):99-121, 2000.
[38] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learnd-Milllr.. Multi-view convolutional neural networks for 3d shape recognition. In Proc. ICCV, 2015.
[39] Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, Francois Goulette, and Leonidas J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
[40] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint 19I0v 2019.
[41] Mikaela Angelina Uy, Quang-Hics Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEF/CVF international conference on computer vision, pages 1588-1597, 2019.
[42] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages (604-4612 2020.
[43] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. 2019.
[44] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions Graphics (TOG), 38(5):1-12, 2019.
[45] Xin Wei, Ruixuan Yu, and Jian Sun. View-gen: View-based graph convolutional network for 3d shape analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1850-1859, 2020.
[46] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9621-96300 2019.
[47] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9621-9630 2019.
12

---

[48] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[49] Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. Walk in the cloud: Learning curves for point clouds shape analysis. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 915-924, 2021.
[50] Chenfeng Xu, Shijia Yang. Bohan Zhai, Bichen Wu, Xiangyu Yue, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Image2point: 3d point-cloud understanding with pretrained 2d convnets. arXiv preprini (0EI::2006.018880 2021.
[51] Mutian Xu, Junhao Zhang, Zhipeng Zhou, Mingye Xu, Xiaojuan Qi, and Yu Qiao. Learning geometry disentteddd representation for complementary understanding of 3d object point cloud. arXiv preprint arXiv:2012. 100::10112.0001, 2021.
[52] Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning on point sets with parameterized convolutional filters. In Proceedings of the European Conference Computer Vision (ECCV), pages 87-102, 2018.
[53] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shuguang Cui, and Zhen Li. 2dpass: 2d priors assisted semaantic segmentation on lidar point clouds. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
2022.
[54] Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui. Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5589-5598, 2020.
[55] Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Trimiropoulos. Knowledge distillation via softmax regression representation learning. In International Conference Learning Representations, 2020.
[56] Ze Yang and Liwei Wang. Learning relationships for multi-view 3d object recognition. In Proc. IEEE Int. Conf. Comput. Vision, pages 7505-7514, 2019.
[57] Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat 2d semantics assisted training for 3d visual grounding. 2021.
[58] Haoxuan You, Yifan Feng, Rongrong Ji, and Yue Gao. Pvnet: A joint convolutional network of point cloud and multi-vier for 3d shape recognition. In Proc. ACM Int. Conf. Multimedia, MM 18, pages 13 1310-133 1318, New York, NY, USA, 2018. ACM.
[59] Zhihao Yuan, Xu Yan, Yinghong Liao, Yao Guo, Guanbin Li, Zhen Li, and Shuguang Cui. X-trans2cap: Cross-modee knowledge transfer using transformer for 3d dense captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[60] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Tomr and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CF International Conference on Computer Vision, pages 1 16259- 16268, 2021.
[61] Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2102. 00650, 2021.
13

---

Checklist
Did you include the license the code and datasets? [No] The code and the data are The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "deziy14nKk3", "reviewer": "Reviewer_vTJ8", "summary": "This paper proposes a cross-modal training scheme called PointCMT which utilizes both 3D point cloud and synthetically rendered or view projected 2D image to boost point cloud classification performance. The cross-modal training is formulated as a knowledge distillation problem and can be easily combined with existing 3D point cloud based algorithms such as PointNet++ and PointMLP. Experiment shows that PointNet++ with PointCMT can achieve 1.0% and 4.4% accuracy improvements on ModelNet40 and ScanObjectNN benchmark dataset, respectively.", "questions": "1. How about the generalization capability of cross-modal training? for example, the performance of training on ModelNet40 and test on ScanObjectNN or vice versa, as similar experiment has been done in [14]", "limitations": "There is no potential negative societal impact found", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The idea of using view projected 2D image for 3D point cloud classification has been presented in SimpleView[14], but the scheme of cross-modal training using both 3D point cloud and view projected 2D image is novel and interesting.\n2. A cross-modal point generator is proposed to solve the cross-modal knowledge distillation problem when the feature distribution of 3D point cloud and 2D image is different and complementary\n3. The analysis and proof on the discrepancy between the discriminative image and point cloud features is good.\n\nWeakness:\n1. There are some concerns on the experiment: \n    a) In table 1, it seems that even equipped with PointCMT, The PointNet++/RS-CNN/DGCNN seems no better than SimpleView [14] which achieves 93.9% overall accuracy with speed of 2208 samples/second, does this mean that cross-modal information fusion (3D point cloud + 2D view image) may not help on the ModelNet40 dataset and only 2D view image is enough? \n    b) In table 6, the performance of \"with projection\" achieves 91.8% accuracy on OBJ_ONLY, which is about +4.3% improvement according to Table 2, but the performance of \"with Project with color\" only achieves 90.7% accuracy, dropped by 1.1%, does this mean the use of point color information hurts performance of cross-modal training? \n2. Since the cross-model training may increase the training time, it would be good to include both training cost and performance gain in the comparison experiment\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cIITVr40Kcg", "reviewer": "Reviewer_J8fY", "summary": "This paper shows a 2D-to-3D distillation framework PointCMT that improves 3D classification through cross-modal training. Despite the performance improvement in two classification datasets,  the reviewer has concerns about the effectiveness of PointCMT: (1) the theoretical argument of this approach; (2) some important ablation studies are missed; (3) more serious applications like segmentation should be considered.", "questions": "#### Majors:\n\n1. PointCMT might not be theoretically sound. Feature loss (Eqn. 5) will be zero if global features of image and point cloud are equal, since the same CMPT is used for both inputs. Classifier loss (Eqn. 7) will be zero if global features of image and point cloud are equal, since the same classifier (Cls^{pts}) is used. Therefore, the whole regularization term in PointCMT is just to make two global features identical, which means the PointCMT can be replaced by a simple regularization term that makes two features closer to each other.\n2. Table 5 needs more clarification. It is not clear which component is ablated. Take row 1 (baseline) and row 2 (Hinton) for example, is it only the logits distillation part be ablated? In other words, are you using eqn. 6 as a replacement of eqn. 7. Or, is the feature distillation ablated such that you are studying the different possibilities of Eqn. 5? Or, both?\n3. An ablation study on with/without CMPG should be added if Table 5 is not for it.\n4. An ablation study on using Eqn. 6 instead of Eqn. 7 should be added if Table 5 is not for it.\n5. An ablation study on values of tradeoff parameters (alpha and beta) in L217 Eqn. should be added. 30 and 0.3 are not common values. How much variance can the parameters cause to the performance? Is that possible that you outperform other distillation techniques simply because the parameters are not good for them in Table 5.\n6. Overclaim. There is already a work in 2d-to-3d knowledge distillation. [1]\n7. Application is limited to classification only. Classification is the most trivial task in point cloud understanding and the most unimportant task. The classification benchmark datasets in this work are rather small-scale and networks are easily saturated in performance. More serious applications such as segmentation should be considered.\n\n[1]  Yueh-Cheng Liu, etc. Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining\n\n#### minors:\n\n1. Revise CMPG description. For L181, you should revise to something like: the cross-modal point generator (CMPG) is used to map the global feature representation acquired from images/points into the Euclidean space. Note that CMPG will be used to transform both images and point clouds after pretraining. Since it is the first time you define CMPG, the function of CMPG should be made clear.\n2. Better illustration for Fig. 2. Fig.2 (a) dash lines from global feature to feature Enhancement can be illustrated in other colors, \\eg blue, since it requires the CMPG module and thus should be highlighted differently for easier understanding. In (b), image features should be changed to global features from images or points.\n3. Highlight training cost instead of inference speed. It is weird to highlight the inference speed in L257, since PointCMT only influences training speed and has no effect on inference speed. The inference speed is totally dependent on the point cloud network used in PointCMT. The author should highlight training costs instead.\n4. The authors should explicitly highlight that for all experiments, they trained baseline with and without PointCMT using the same optimization techniques, data augmentation, and evaluation techniques. For example, SimpleView found these tricks matter a lot in performance.\n5. Any clue why PointCMT improves PointNet++ more than other networks?", "limitations": "The authors did not include limitations and potential negative societal impacts. Limitations, e.g. only for classification, can be added. ", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "PointCMT shows cross-modal training that improves performance. Check questions for the weakness. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UwzBykJRjbM", "reviewer": "Reviewer_R3qG", "summary": "The authors present a method to enhance point cloud classification models by exploiting training on images. Features extracted from images may be able to capture complementary details to what is typically learned by training on point clouds alone. A distillation procedure is presented so that such complementary features can be integrated in a model that only processes a point cloud at test time.", "questions": "- why have chosen the EMD instead of the Chamfer distance as a metric for your loss function? \n- it seems you reference the published results for PointMLP. Have you been able to reproduce them? Based on experience and online discussions (https://github.com/ma-xu/pointMLP-pytorch/issues/1), it seems they are quite tricky to reproduce", "limitations": "I would like to see an extended discussion about the single-modal setting, in which only point cloud data are used at testing time (although multiple modalities may be available for training) considered in this paper against a fully multi-modal setting.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the method is sound and generally well explained. A few mistakes in the use of English are present and could be fixed (for example page 2 line 55 \"Effectively\" --> \"Effectiveness\").\n\nStrenghts:\n- sensible approach to improve feature learning by leveraging complementary features that are more easily extracted from images\n- the distillation approach allows to still work in a single-modal setting at test time\n- good results showing a benefit from the proposed technique on state of the art architectures\n- good ablations to validate each proposed component (CPMG/Feature enhancement)\n\nWeaknesses:\n- distillation allows to train the point cloud encoder to extract those features that are easily extracted from images but generally escape classifiers trained on point clouds, despite those features being present in the input point cloud. However, because at test time only the point cloud is used, it is not possible to exploit truly complementary features that an image might carry and that are not present in the point cloud data. This would only be possible in a truly multi-modal setting, also at test time\n- the training of CPMG requires paired data with the image and point cloud of the same object. This is somewhat constrasting with the motivation of the work in page 1 line 34 highlighting the potential difficulty in having paired multimodal data  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EpuV85Z7bd_", "reviewer": "Reviewer_N54s", "summary": "The paper proposes a knowledge distillation strategy to improve point cloud classification. It uses images of point clouds to train an image classification network. The representation from this network can be distilled into any existing point cloud network. The distillation requires multi-stage training including -- training the image classifier, training point generator from images and training the point-cloud network assisted by the image classifier. Experiments show the effectiveness of this scheme. ", "questions": "Refer to the weakness section for questions. Overall, I am (weakly) positive about the paper. I will update the score based on the rebuttal.", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper achieves impressive performance on both the ModelNet40 and ScanObjectNN datasets.\n\n- Gains due to knowledge distillation seem to consistent across various networks (Table 1, Table 2)\n\nWeakness:\n\n- The method archives performance improvements; however, the training pipeline seems to become more complicated with many training stages. This could be a potential limitation. It would be useful if the paper discusses this and potential ways to mitigate it.\n\n- It would be nice if the paper could provide some mean +- std measures. This could be done by running the same experiments multiple times (with random initialization) and reporting the mean and variance. This is particularly important as point-based benchmark methods can have significant variations across runs.\n\n- What is performance of the teacher networks on the two datasets? Also, is the teacher network potentially better because of the additional image to point cloud task? I asking this because as shown in the paper 3D projections could be used and these 3D projections can be created from the point cloud, which is available at test time. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept