Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models
Authors: Chenfeng Xu
Shijia Yang
Bohan Zhai
Bichen Wu
Xiangyu Yue
Wei Zhan
Peter Vajda
Kurt Keutzer
Masayoshi Tomizuka
Venue: ICLR 2022 Submitted (2022)
Keywords: Computer vision
Point-cloud
Cross-modality.

Abstract: 3D point-clouds and 2D images are different visual representations of the physical world. While human vision can understand both representations, computer vision models designed for 2D image and 3D point-cloud understanding are quite different.
Our paper explores the potential for transferring between these two representations by empirically investigating the feasibility of the transfer, the benefits of the transfer, and shedding light on why the transfer works.
We discovered that we can indeed use the same architecture and pretrained weights of a neural net model to understand both images and point-clouds. Specifically, we can transfer the pretrained image model to a point-cloud model by \textit{inflating} 2D convolutional filters to 3D and then \textbf{f}inetuning the \textbf{i}mage-\textbf{p}retrained models (FIP). 
We discover that, surprisingly, models with minimal finetuning efforts --- only on input, output, and optionally batch normalization layers, can achieve competitive performance on 3D point-cloud classification, beating a wide range of point-cloud models that adopt task-specific architectures and use a variety of tricks. When finetuning the whole model, the performance further improves significantly. Meanwhile, we also find that FIP improves data efficiency, achieving up to 10.0 points top-1 accuracy gain on few-shot classification. It also speeds up the training of point-cloud models by up to 11.1x to reach a target accuracy.

Under review as a conference paper at ICLR 2022
IMAGE2POINT: 3D POINT-CLOUD UNDERSTANDING WITH 2D IMAGE PRETRAINED MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
3D point-clouds and 2D images are different visual representations of the physical world. While human vision can understand both representations, computer vision models designed for 2D image and 3D point-cloud understanding are quite different. Our paper explores the potential of transferring 2D) model architectures and weights to understand 3D point-clouds, by empirically investigating the feasibility of the transfer, the benefits of the transfer, and shedding light on why the transfer works. We discover that we can indeed use the same architecture and pretrained weights of neural net model to understand both images and point-clouds. Specifically, we transfer the image-pretraiin convolutional filters to 3D convolutional filters and finetuning the inflated image- pretrained models (FIP). We find that models with minimal finetuning efforts only on input, output, and optionally, batch normalization layers - can achieve competitive performance on 3D point-cloud classification, beating a wide range of point-cloud models that adopt task task-specific architectures and use variety of tricks. When finetuning the whole model, the performance gets further improved. Meanwhile, FIP improves data efficiency, reaching up to 10.0 points top- accuracy gain on few-shot classification. also speeds up training of point-cloud models by up to 1 11.1x for target accuracy.
model point-cloud model by inflating 2D
INTRODUCTION
Point-cloud is an important visual representation for 3D computer vision. It is widely used in applications such as autonomous driving (Behley et al. 2019 Caesar et al. 2020 Yue al. 2008). robotics (Armeni et al.] 2017 Pomerleau et al. 2015 Xu et al. 2021). augmented and virtual reality (Sketchup) 2021 Wu et al. 2015 Shi et al. (015)), etc. A point-cloud represents visual information in highly different way from 2D image. A point-cloud consists of a set of unordered points lying on the object's 's surface, with each point encoding its spatial x, y. Z coordinates and potentially other features. In contrast, 2D image organizes visual features dense 2D pixel array. Due to the representation differences, 2D image and 3D point-cloud understanding are treated as separate problems. Image models and point-cloud models designed have different architectures and trained on different types of data. Few research efforts have tried to directly transfer models from images point-clouds or vice versa.
Intuitively, both 3D point-clouds and 2D images are visual representations of the physical world. Their lowelevel representations are drastically different, but they can represent the same underlying visual concept. Furthermore, human vision has no problem understanding both representations. However, can computer vision models trained on one modality understand the other?
Somewhat remarkably. the answer to the question above is: Yes. 2D image models trained on image datasets can be transferred to understand 3D) point-clouds with minimal efforts. As illustrated in Figure [ we transfer 2D ConvNet to a 3D ConvNet whose input is a 3D voxel representation converted from a point-cloud. Based on a pretrained 2D ConvNet, we inflate its 2D convolutional filters to 3D by copying the filter weights along a third dimension. We add linear input and output layers to the network; and on a target point-cloud dataset, we only finetune the input/output layers, and optionally, the normalization layers, while keeping the pretrained model weights untouched. We term such partially-innetuned finetuned -aaeede---ppeeeee pretr-ine models as FIP-I0 (finetuning only input and output layers) or FIP-10+ BN (finetuning input, output, and BN layers). FIP-IO+BN achieve competitive

---

Under review as a conference paper at ICLR 2022
Pretrained 2D ConvNet
Input Layer
Norm
Conv2D
Classifier
Dog
Inflation
Copy
Inflation
Copy
Input Layer
Classifier
Chair
Sparse Conv3D
Sparse Conv3D
Norm
Norm
Input Layer
Decoder
Input Layer
3D Sparse ConvNet
Decoder
Figure 1: We investigate the feasibility of pretrained 2D ConvNets transferring to 3D sparse ConvNets. With filter inflation and finetuning only the input, output layer (classifier for classification task and decoder for semantic segmentation task), and optionally, normalization layers, 3D Sparse ConvNets are capable of dealing with point-cloud classification, indoor, and driving scene segmentation.
performance up to 90.8% top-1 accuracy on the ModelNet 3D Warehouse dataset, on top a ResNet50, outperforming previous point-cloud models that adopt task task-specific model architectures and tricks.
Most point-cloud models except projection-basse based models are only trained from scratch. Based on our discovery, we further investigate fintt-fnndi-iiaaeeee We observe that FIP-ALL brings significant improvement top of ResNet series. Besides applying FIP-ALL to voxel-based method, we also find that generalizes other popular methods, such as point- based method (PointNet++ (Qi et al. 2017)) and projection-baseee method (Simple View (Goyal et al., a)) as well as current popular vision transformers (VIT (Dosovitskiy et al. 2020)). Specifically, FIP-ALL largely outperforms the training-from-sernee
models (termed as FIP-ALL).
by 0.88, 0.50, 3.50, 4.18 points top-1 accuracy on top of PointNet++, Simple View. ViT-B-16, and ViT-L-16, respectively. In addition to the performance gain, FIP-ALL exhibits superior data efficiency with up to 10.0 points improvement in few-shot classification on the ModelNet 3D Warehouse dataset. Comparing with momning-from-scraaee
FIP-ALL also dramatically speeds up the training by using 11.1 times fewer epochs to reach a target validation accuracy.
In order understand why the image pretraining can be utilized benefit point-cloud understanding. we conduct experiment studying the network dissection (Bau et al. 2017). text-shape representation transferring (Geirhos al. 2018). and distribution distance.
RELATED WORK
2.1 POINT-CLOUD PROCESSING MODEL
3D convolution-based method is one of the mainstream point-cloud processing approaches which efficiently process point-clouds based on voxelization. In this approach, voxelization is used to rasterize point-clouds into regular grids (called voxels) so that conventional 3D convolutions can be
applied. Sparse convolution is proposed to apply on the non-empty voxels (Liu et al.) 2015 Choy
jet al. 2019 Tang et al. 2020, Zhou et al. 2020 Yan et al. 2018 Feng al. 202 largely improving the efficiency of 3D convolutions.
Projection-based method attempts to project a 3D point-cloud to 2D plane and uses 2D convolution to extract features (Wang et al. 2018 Wu et al.) 2018 2019 Xu et al.
2020 Su et al. 2015 Lawin
al. 2017 Boulch et al. 20177) Specifically, vird-eye-view projection
Yang et al. 20188] Lang et al.
2019) and spherical projection (Wu et al. 2018 2019 Xu et al. 2020
Milioto et al. 2019 make
great progress in outdoor point-cloud tasks.
Point-based method directly processes the point-cloud data. The most classic methods, PointNet (Qi et al.) 2016) and PointNet++ (Qi et al. 2017). consume points by customized feature aggregation. Many works further develop advanced local-feature aggregation operators that mimic the convolution
2

---

Under review as a conference paper at ICLR 2022
to structure data (uu al. 2021 Li al.
2018b; 2019).
Hua et al.
2018 Liu al. 2019
2020
Wang et al.
2017 Li et al. 2018a Komarichev et al.
2.2
PRETRAINING IN 2D AND 3D VISION
Prefraining in 2D vision has shown effectiveness under supervised (Dosovitskiy al., 2020 Gir-
shick et al. 2014), self-supervised (Jing & Tian, 2020 Goyal et al.) 202 and unsupervised
contrastive approach (He et al.
2020; Bachman et al. [019]] Chen et al. 2020a Caron al. 2020
Chen et al. 2020c, Hjelm et al. 2018). After pretraining on large amount of data, a 2D model re-
quires much fewer computational resources and data for finetuning to reach competitive performance
on downstream tasks (Kataoka et al. 2020 Caron et al. 2019 Chen et al. 2020b, [ennaf 2220))
Prefraining in 3D vision has been studied similarly as pretraining in 2D vision: both self-supervised and contrastive pretraining (Xie et al., 2020) show promising results. Due to the lack of large, annotated point-cloud datasets, pretraining in 3D vision motivated to achieve data efficiency (Xu & Lee 2020). Recent works (Hou et al. 2020; Zhang et al., (2021b) consider pretraining methods with data efficiency in mind, for example, Contrastive Scene Contents which makes use both point-level correspondences and spatial contexts.
2.3 CROSS- TRANSFER LEARNING
Cross-modal transfer learning attempts to take advantage of data from different modalities (Dai & Nießner 2018 Liu al., 2021b). For example, [iu al. (2021a) proposed pixel-to-point knowledge transfer (PPKT) from 2D to 3D which uses aligned RGB and RGB-D images during pretraining. Our work does not rely on joint image-point-clout pretraining. Instead, we directly transfer an image-pretrained model to point-cloud with the simplest pretraining -iietiinngg scheme.
Some of the previous works for video and medical images (Carreira Zisserman 2017] [haa et al.) 2018) have adopted the method of simply extending pretrained 2D convolutional filter along time or depth direction for transferring to 3D models. Between language and image modality, transfer learning with minimal finetuning also shows competitive performance (Lu et al. 2021).
3 CONVERTING A 2D CONVNET TO A 3D CONVNET
In this paper, we primarily focus on the 3D sparse-convolution based method to process point-clouds because can be extended to all point-cloud tasks. As discussed in [.T] we consider a set of points where each point is represented by its 3D coordinates and additional features such as intensity and RGB. We then voxelize/quantize these points into voxels according to their 3D space coordinates, following Choy et al. (2019). A voxel's feature is inherited from the point that lies in the voxel. If there are multiple points in a voxel, then we average all points' feature and assign the mean to the voxel. If there is point in the voxel, then simply set the voxel's 's feature to O. When using sparse convolution, we skip the computation on empty voxels.
Given a pretrained 2D) ConvNet, convert it 3D ConvNet that takes 3D voxels input. The key element of this procedure is to convert 2D convolution filters to 3D, i.e. constructing 3D filters with the weights directly inherited from 2D filters. 2D convolutional filter can be represented with a 4D tensor of shape [M, NN, K, K], representing output dimension, input dimension, and two spatial kernel sizes, respectively. A 3D convolutional filter has an extra dimension, and its shape is [M, N, K, K, K). To better illustrate, we ignore the output and input dimensions and only consider a spatial slice of the 2D filter with shape [K, K]. The simplest way to convert this 2D filter to 3D is to copy the 2D filter and repeat it by K times along third dimension. This operation is the same as the inflation technique used by (Carreira & Zisserman 2017) to initialize video model with a pretrained 2D ConvNet.
Besides convolution, other operations such as downsampling, BN, nonlinear activation can be easily migrated to 3D. Our 3D model inherits the architecture of the original 2D ConvNet, but we also add linear layer as the input layer and an output layer depending on the target task. For classification, we global average pooling layer followed by one fully connected layer to get the final prediction. For semantic segmentation, the output layer is a U-Net style decoder (Ronneberger et al. 2015). The architecture of the input/output layers is described in more detail in Appendix A.9
3

---

Under review as a conference paper at ICLR 2022
(a) Trainable Parameters vs. Accuracy
(b) Pretrained Dataset
23
8512
50.7
50.7
50.2
50.8
50.315
HIND-EN (99.00)
RIP-DwaN (50.44) HP 10-iN (88.7)
RP-ID-aH (18.75)
Reshecte
883
83.35
85
Top Accuracy
77.5
00
FIP-
365.5x
Rediectd iragehutiK
Reclerd0 on IragshistIK
11h-0 (73.24)
Reafec50 tragsPretZiK
Resterti52 Inagoteect
Derpitano
Rolettet
Poimtorte
Diccan
Ruste Branting
KDMG
Restento 10
63
Restestio rot 10
FP-40 (o4.G)
0.001
0.002
0.003
0.004
Trainable parameters/Total /Total parameters
Train from scratch
Thy--imageke pretests
ImageRettk pretzxin
ImageMer2EK pretroin
precrale
FroctalDe
Figure 2: a) the left figure shows the trainable parameters ratio w.zf top-1 accuracy on ModelNet 3D Warehouse dataset. b) the right figure shows the performance of FIP-IO and FIP-IO+BN on top of ResNet50 pretrained on different datasets.
4 EMPIRICAL EVALUATION
To explore the image to point-cloud transfer, we study three settings: 1) partially-ineuune--iaa---- pretrained model, only finetuning input and output layers (FIP-IO), 2) finetuning input, output, and batch normalization layers (FIP-IO+BN), and 3) finetuning the whole pretrained network (FIP-ALL). Under the three settings, we extensively explore the feasibility of transferring the image-pretrainee model for point-cloud understanding and its benefits. The entire empirical evaluation is organized as four questions: 1) Can we transfer pretrained-image models to recognize point-clouds? (Section 4.
2) Can maag-preerrining benefit the performance of point-cloud recognition? (Section 4.2) 3) Can
amage-pretraingg model improve the data efficiency on point-cloud recognition? (Section 4.3 4) Can
image-pretraine model accelerate training point-cloud models? (Section 4.4
Datasets. We benchmark the transferred models on ModelNet 3D Warehouse classification (Wu ett al., 2015), S3DIS indoor segmentation (Armeni et al. 2017), and SemanticKITTI outdoor segmen- lation (Behley et al. that consists of clont with 40 categories. CAD models in this benchmark come from 3D Warchouse (Sketchup 2021). In this benchmark, we only utilize x. y, coordinates features. S3DIS is a dataset collected from real-world indoor scenes and includes 3D scans of Matterport Scanners from areas. It provides point-wise annotations for indoor objects like chair, table, and bookshelf, etc. SemanticKITTI dataset from KITTI Vision Odometry (Geiger et al., 2012) a driving scene dataset. It provides dense point-wise annotations for the complete 360 degrees field-of-view of the deployed automotive lidar, which is currently one of the most challenging datasets.
2019) tasks. ModelNet 3D Warehouse is a CAD model classification dataset
ResNet (He al., 2016a) series is used mostly throughout our experiments. Depending the experiments, ResNets are pretrained on Tiny- InnelmmaeNNet ImageNet-IK, IK, ImageNet-21K (Deng et al. 2009). and Fractal database (FractalDB) (Kataoka al., 22220). Our pretrained models are directly downloaded from various sources, with detailed links provided in Section A.1 To study the benefits of using pretrained image models, we also utilize PointNet++ (Qi et al. (017). VIT (Dosovitskiy al. 2020), and Simple View (Goyal et al. (22 as baselines.
4.1
CAN WE TRANSFER PRETRAINED- MODELS TO RECOGNIZE POINT-CLOUDS?
To evaluate the feasibility of transferring pretrained 2D image models to 3D point-cloud tasks, we conduct experiments on top of the ResNet series since there are abundant open-source pretrained ResNet available. In particular, we convert 2D ConvNets into 3D ConvNets using the procedure described in Section 3 We hypothesize that, if pretrained 2D image model is capable of understand- ing point-clouds directly, we can see a non-trivial performance by only finetuning input and output layers of the transferred model. Further, BN parameters as well, the transferred model can achieve better performance, even surpassing Praning-from-screee performance.
gradually relax the frozen parameters, finetuning
4

---

Under review as a conference paper at ICLR 2022
Table 1: ModelNet 3D Warchouse classification results (top-1 accuracy %) of fully-finttuncd-maage pretrained models (FIP-ALL) based on different pretrained models. We include 2021 SOTAs, such as
RSMix (Lee et al., (021), Point Transformer (Point-Trans) (Zhao al. 202T), DRNet (Qiu et al. 2021 and PointCutMix (Zhang et al. 2021a for comparison.
Method
ResNet18
ResNet50
ResNet152
ResNet101x2
From Scratch
90.39 90.52 (+0.13)
90.32
90.28 91.09 (+0.81)
90.03 90.52 (+0.49)
FIP-ALL on ImageNet1K
90.92 (+0.60)
FIP-ALL on ImageNe121K
91.05 (+0.73)
Method
(SSS)Net++(SSG)
ViT-R-16
ViT-I-116
SimpleVite
From Seratch
90.34
84.27
83,48
93.3
FIP-ALL on ImageNetIK
91.22 (+0.88)
93.8 (+0.50)
FIP-ALL on ImageNe121K
87.77 (43,50)
87.66 (+4.18)
Method
RSMix
Point-Trame
DRNet
PointCutMix
From Scratch
93.5
93.7
93.1
93.4
Table 2: Indoor scene and outdoor scene segmentation results (mloU %) of fully-finetuned ielll--iinttned-imaae -imae---------------- pretrained Model (FIP-ALL). In this table, all
models are pretrained ImageNet1K K.
S3DIS (mloU %)
SemanticKITTI (mloU %)
Method
PointNet++(SSG)
ResNet18
HRNecV2-W48
ResNet18
From Scratch
52.45
55.09
44.12
64.75
FIP-ALL on ImageNetI
55.01 (+2.56)
56.62 (+1.53)
47.53 (+3.41)
65.57 (+0.82)
We conduct two groups of experiments with FIP-IO and FIP-IO+BN, with the results shown in Figure The first is to evaluate the performance as the trainable parameters gradually increase, As shown in Figure (a), training
more than 0.3 % (345.5x fewer) of the whole parameters, the image pretraining even beats the training-froom Seeting-ffromeee from-scratch (100 % trainable parameters). Specifically, 5esNet152 FIP- IO-BN with ImageNetIK pretraining improves training- from--cratel by 0.16 points, and ResNet50 FIP-IO+BN with ImageNet21K pretraining improves 0.48 points. Meanwhile, FIP-IO also reaches non-trivial performance. ResNet50 FIP-IO pretrained on ImageNet1 K achieves 81.20 % top-1 accuracy, only 9.12 points worse than maining-from-scren with approximately 0.1 % trainable parameters.
Furthermore, to investigate the effect of different datasets, as shown in the right figure of Figure 2 we inflate ResNet50 pretrained from different image datasets, including Tiae-ImageNet, ImageNetI ImageNet21K: IK, FractalDB IK, and FractalDB 10K, then evaluate on the ModelNet 3D Warehouse.
We discover that, even if we only finetune the input and output layers while keeping the image- pretrained weights frozen, the FIP-IO pretrained from ImageNet KK, FractalDBIK, and FractalDB 10K achieves competitive performance. Specifically, ResNet50 FIP-IO with ImageNet1 K pretraining outperforms 3D ShapeNet (Wu et al. (015)) and DeepPano (Shi et al. 2015). which were the state-of- the-arts in 2015, by 4.2 and 3.6 points respectively top-1 accuracy on ModelNet 3D Warehouse. More importantly. with ImageNet21K pretrained model, ResNet50 FIP-IO+BN surpasses training- from- serats by 0.48 points, even beating a variety of well-known methods including PointNet (Qi
al. 2016), MVCNN (Su et al. 2015). DGCNN (Wang et al. 2019), KDNet (Klokov & Lempitsky 2017
etc.
Notably, we find out the answer to "Can transfer pretrained-image models to recognize point- clouds?": Yes. The pretrained 2D image models can be directly used for recognizing point-clouds. is also noteworthy that the pretraining dataset is not restricted to natural but also synthetic images like those in FractalDB D K/10K.
4.2 CAN
BENEFIT POINT-CLOUD RECOGNITION?
From the previous subsection, we find unexpectedly that pneee-prtrained model can be directly used for point-cloud understanding. In this subsection, we investigate whether image-pretrained model
5

---

Under review as a conference paper at ICLR 2022
Table 3: Comparison with PointContrast (Xie et al. 2020) the ModelNet 3D Warchouse. PointCon- trast provides two different pretrained models with using PointInfoNCE loss and Hardest Contrastive loss, respectively.
From scratch
PointInfoNCE
Hardest Contrastive
ImageNet1K pretrain (Ours)
89.95
90.24 (+0.29)
90.15 (+0.20)
90.88 (+0.93)
helpful to improve the performance on point-cloud tasks. We use different baselines, including roxelization-base method (simply ResNet), point-based method (PointNet+++ (Qi et al. (017])) projection baaodddd---dddd-- method (SimpleViens View (Goyal et al., 2021a)), and current popular transformer-bas method (ViT-B-16 and ViT-L-16 (Dosovitskiy et al. 20200), and fully finetune them three point- cloud datasets: classification on ModelNet 3D Warehouse, indoor scene segmentation on S3DIS, and outdoor scene segmentation SemanticKITTI, shown in Table Tad Table
For PointNet+++, we use ImageNetIK to pretrain: we break each image into pixels and regard it as a point-cloud. For ViT, directly use the open-source pretrained model and finetune it on ModelNet 3D Warehouse. All the implementation details are illustrated in Appendix A.
Table presents performance on ModelNet 3D Warehouse dataset. We observe that FIP-ALL improves all baselines steadily and significantly. Besides, pretraining brings more improvements to deeper models. For example, ResNet18 can only be improved by 0.13% top-1 accuracy, but pretraining on ImageNetIK K leads to 0.81 points top-1 accuracy improvement on top of ResNet 52, Moreover, larger pretrained datasets also lead better performance. Specifically, ResNet50 FIP-ALL from ImageNet21K K can reach 91.05% top-1 acc, with 0.73 points improvement over training from- scratch. Such FIP-ALL significantly outperforms series well-known methods such as (Qi et al.. 2016 2017 Klokov & Lempitsky 2017 Wang et al.) 2019; Su et al. 2015 Li et al.) 2018a).
We also explore FIP-ALL on different architectures, as shown in the second group of Table In particular, FIP-ALL top of PointNet++, ViT-B-16, ViT-L-16 and SimpleView with image dataset pretraining improve the fimiing-from-scracch by 0.88, 3.50, 4818, 0.50 points, respectively. Especially for the current superior baseline in image recognition, ViT-B-16 and ViT-L-16, the improved performance is quite significant, revealing the huge potential of using image- mnage-pretrained models for point cloud recognition.
For the challenging indoor and outdoor scene segmentation, using ImageNetIK pretrained models (FIP-ALL on ImageNetIK) also improve the training- from-scratch consistently, shown in Table PointNet++ (resp. ResNet18 8) pretrained on ImageNet outperforms the vraning-from-serann 2.56 points (resp. 1.53 points) mIoU on S3DIS dataset. For SemanticKITTI, we utilize the commonly used projection-based method with 2D ConvNet HRNet. With ImageNetIK pretraining, we observe 3.41 points mloU improvement, large margin in such challenging task. Since HRNetV2-Wa --4 has rich pretrained models, we finetune Cityscapes pretrained HRNetV2-WW -W-4 and observe this enhances more (5.25% mIoU improvement over training from scratch). Even for the ResNet18 with a high from-scratch performance of 64.75% mloU, the ImageNet1 K pretraining can also bring 0.82 points mloU improvement.
by
Finally, we compare the performance gain with the well-known point-cloud self-supervised method PointContrast (Xie et al. 2020), as presented in Table We use the same model architecture and finetuning recipe, and the only difference is the pretraining weights. We can observe that image- pretraining on ImageNetIK K significantly boosts the training-froor from-scratch by 0.93 points, surpassing the PointContrast by at least 0.64 points.
Therefore, the answer to "Can saage-pretraining benefit point-cloud recognition" is: Yes. Image- pretraining can indeed improve point-cloud recognition, which can generalize wide range of backbones and benefit more challenging tasks.
4.3 CAN PAAGE-PRETRAINED MODELS IMPROVE DATA EFFICIENCY ON POINT-CLOUD RECOGNITION?
Data efficiency is extremely important in plont understanding due to the huge labor of collecting and annotating point-cloud data. In this subsection, investigate whether the pretraind

---

Under review as a conference paper at ICLR 2022
Table 4: Few-shot experiments top of different ResNets the ModelNet 3D Warchouse dataset.
ResNet18
ResNet50 (from scratch/FIP-ALL)
ResNet152
Few-shot
10-shot
72.2::0.873.22066 (+1.0)
71.7:0.774.1 (+2.4)
69.821. (+4.1)
5-shot 11771166/66.1111 (+2.9)
(+3.6)
59, (+7.1)
1-shot
26.8:4.4/36.-06 (+10.0)
28. 1..100.//4.1100. (+6.0)
23.3=4.3/33.2 (+9.9)
Volidation /top 1 acturacy) Epoch
Validation (top accuracy) Epoch
Validation (top accuracy)
28 opoch
11 spoch
19 epach
epach
56 opoch
A5
1th spoch: $1.05
Redtet 18 trale from scratch
spach:
Restert 50 train from scratch
s0
1th opech: 79.34
-Resiie from
ReiNett 11 FIPALL or imagetietic
RosNet
Reshve 152 RP All on Imagehietik
11h spoch: 70.22
1th spech: AB
11h spach:
11
21
Epoch
Epoch
Epoch
Figure 3: The curves of validation accuracy w.r. training epoch. We compare the results be- tween training-fromd fcoa-scratch and FIP-ALL on the ImageNetIK, on top of ResNet18, ResNet50, and ResNet152, respectively.
model can help to improve the data efficiency by conducting few-shot setting experiments, including 1-shot, 5-shot, and 10-shot. We conduct 3 trials for each setting and report the results as mean + std.
In detail, for each class (ModelNet 3D Warehouse involves 40 classes), we randomly choose a few point-clouds as training data, and still evaluate on the whole test set. We compare the results between training-frone from-scratch and FIP-ALL pretrained on the ImageNet1 K dataset. The experimental results are shown in Table 4 We observe that FIP-ALL dramatically surpasses training-fromm from-scratch on the low data regime (1-shot): pretraining on ImageNet K brings 10.0, 6.0, and 9.9 points top-1 accuracy improvement for ResNet18, ResNet50, and ResNet152, respectively. For 5-shot and 10-shot settings, using ImageNet1K pretraining can still consistently improve the performance. However, also observe that as the amount of training data increases, the performance gain becomes saturated.
Therefore, answer to "Can amage-pretraing model improve the data efficiency point-cloud recognition?" is: Yes. amage-pretrained model recognition, especially low data regime. When the training data increases, it can still improve the performance, but the gain becomes marginal.
can improve the data efficiency on point-cloud
4.4 CAN IMAGE-PRETRAINED MODELS ACCELERATE POINT-CLOUD TRAINING?
We also investigate whether pretrinde model can help point-cloud task train faster. The results are shown in Figure 3
We discover that, after training only one epoch on ModelNet 3D Warehouse dataset, FIP-ALL on ImageNetl K achieves very impressive performance, yet the performance of training-frommsseracch is still at a low level. For example, after the first epoch, ResNet50 (resp. ResNet152) with training from scratch only achieve 28.48% (resp. 13.94%) top-1 accuracy while ResNet50 (resp. ResNet152) with ImageNetiK pretraining reaches 80. 11% (resp. 79.34%) top-1 accuracy. Moreover, to reach 90% top-1 accuracy, non-trivial performance, FIP-ALL significantly accelerates the training by 2.14x (28 vs. 60 epoch). 11.1x (11 vs. 122 epoch), 2.95x (19 vs. 56 epoch) over training-froo foogg-ffr--ttcchhhe on top of ResNet18, ResNet50, and 55s respectively.
Therefore, our answer to "Can mmage-pretraince model accelerate point-cloud training?" is still: Yes. The maage-pretrained model can significantly accelerate the training speed of point-cloud tasks.
7

---

Under review as a conference paper at ICLR 2022
Table 5: Texture- shape representation transferring experiment ModelNet 3D Warchouse.
Method
Pretrained dataset
top-1 accuracy
ResNe150 FIP-IO
ImageNetIK
81.20
ResNet50 FIP-IO
Stylized-Imand ImageNet
83.52
BagNet17 FIP-IO
ImageNet1 ImageNet1
57.53
BagNet33 FIP-IO
68.40
Unit 157
Unit 161
Unit 166
Brodin
Modeinet
Unit 190
Unit 322
Unit 508
Broden
Modeinet
Figure 4: Network dissection of FIP-IO+BN. The visualization displays what the units are attending to on both image dataset and ModelNet 3D Warehouse dataset.
5 DISCUSSION
In this section, we attempt to shed light on why transferring image- peeteeprettained models for point-cloud understanding works. Inspired by recent related works (Geirhos al.) 2018 Brendel & Bethge 2019 He et al.) |015))2066b]]] [auu et al. 2017), we explore this from the aspects of the network dissection, texture-shape representation transferring, and feature distribution distance.
5.1
WHAT DO MAAGE-PRETRAINGE MODELS TRANSFER TO POINT-CLOUD MODEL.?
Does the image-pretrained model transfer the visual concepts? Inspired by the network dissec- tion for 2D Broden dataset (Bau et al. 2017), we also attempt to explore what the transferred units are focus on in the ModelNet 3D Warehouse dataset. We present the visualization when the whole dataset
FIP-IO+BN
pretrained ImageNet K. The visualization shows the most activated passes through each unit of the last model stage, as displayed in Figure 4 More visualization can be found in Appendix 7 From the visualization alone, we do not get obvious cues of what visual concepts are transferred between the two modalities. For example, unit 161 strongly activating to computer screens in the Broden, yet attends cars and shelves in the ModelNet 3D Warehouse. Surprisingly, we find that the pretrained units are prone to cluster similar objects. In fact, such clustering ability is an important cue performing well on classification tasks (Hartigan & Wong 1979: Caron et al. 1)2T).
Do pretrained-image models transfer shape or texture representation? Recent work (Geirhos al. 2018) proposed that models learn texture and shape from ImageNet. We follow this direction to explore what cettrinne--maaee mode transfers. For the experiment, we take two image-pretrained models with either more shape texture representation and compare their FIP-IO performance.
In order to force the pretrained models to acquire more shape representation, Geirhos et al. (2018) stylizes the images in ImageNet into artwork style, such that the models trained those images are confused by variant textures, hence having stronger shape representation. We directly take the pretrained ResNet50 stylized ImageNet the stronger shape representation model.
8

---

Under review as a conference paper at ICLR 2022
To get a stronger texture representation model, we are inspired by BagNet (Brendel & Bethge 2019). By controlling the receptive field, BagNet breaks the shape in an image and focuses more on the texture information. Our experimental result is shown in Table []] BagNet17 means the size of attended patches is 17 x 17, and NetNet33 means the size of attended patches 33 x 33. Note that after inflating the BagNet17 and BagNet33, both architectures Besides, both BagNet and ResNet pretrained on Byylized-ImageNeeel K perform worse than the original ResNet50 on ImageNet (Geirhoo al., 2018 Brendel & Bethge 2019).
the same as inflated ResNet50.
We can observe that the ResNet50 FIP-IO on Stylized-ImageNer (with stronger shape representation) outperforms the baseline ResNet50 FIP-IO on ImageNet1 K over 2.32 points top-1 accuracy, while both inflated BagNets perform dramatically worse than the baseline. This shows that the shape representations are transferred.
To make clear what parts of the image are critical for the transfer, we further conduct experiments using different image processing methods for pretraining. The results are reported in Appendix A.7
5.2 WHY DOES FINETUNING BATCH NORMALIZATION HELP THE TRANSFERRING?
Our experiment in Figure 2 shows that finetuning BN, in addition to the input and output layer, can greatly improve the transfer performance. It is interesting why such a small part of the network, in terms of parameter size and FLOPs, can have a big impact.
We calculate the wasst-wws distance (FWD) (Rubner al.. 2000 measurement of Gaussian distribution distance, between the 2D image and 3D point-cloud feature distributions. For estimating the 2D feature distribution, pass the whole ImageNetl K dataset into the pretrained-image models, collect the pre-activation features after each convolution layer, then sample 15,000 data points from the element-wiss distribution of collected features (we assume the pre-activation features present Gaussian distribution). For 3D feature distribution, we use FIP-IO, FIP-IO+BN, and FIP-ALL to conduct the same operation on the ModelNet 3D Warehouse dataset and also collect 15,000 data points. We then calculate the FWD between the element-wise feature distribution of image- preeeepret model and each of the FIP-IO, FIP-IO+BN, FIP-ALL model.
We observe that the layer-wise average FWD) between FIP-IO (point-cloud features) and image model (image features) is 2.1 x 10 yet after finetuning batch normalization layers, the distance dramatically reduced. Particularly, average FWD between FIP-IO+BN and image model is 0.27, and FWD between FIP-ALL and image model is only 0.093, Complete FWD for each layer is shown in This suggests that batch normalization plays critical role in transforming the
Appendix Table []] point-cloud representation to be closer to the image representation.
6 CONCLUSION
In this work, we use finetuned wwaeeedd-imae-pepe immageprettranne models (FIP) to explore the feasibility of transferring maage-peetrained models for point-cloud understanding and the benefits of using image-pretraiing models on point-cloud tasks. We surprisingly discover that, with simply inflating a 2D pretrained ConvNet and minimal finetuning input, output, and optionally, batch normalization layer (FIP-IO or FIP-IO+BN), FIP can achieve very competitive performance on 3D point-cloud classification, beating wide range of point-cloud models that adopt variety of tricks. Moreover, we find that when finetuning all the parameters of the pretrained models (FIP-ALL), the performance can be significantly improved on point-cloud classification, indoor and outdoor scene segmentation. Fully finetuned models generalizes to most of the popular point-cloud methods. We also find that FIP-ALL can improve the data efficiency on few-shot learning and accelerate the training speed by a large margin. Additionally, we shed light on why amage-pretrained models can be used for point-cloud understanding from three aspects: network dissection, texture-shape representation transferring, and feature distribution distance. Compared with previous works that seek improvements from designing architectures and pretraining only on plint-cloud modality, our work is not limited by the architecture design and the small-scale point-cloud dataset. We believe that image pretraining is one of the solutions to the bottleneck of point-cloud understanding and hope this direction can inspire the research community.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXivv:702.000 105, 2017.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint urXiv:9006. 00900, 2019.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
J. Behley, M. Garbade, A. Milioto, J. Quenzel, Behnke, C. Stachniss, and Gall. SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences. In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV), 2019.
Alexandre Boulch, Bertrand Saux, and Nicolas Audebert. Unstructured point cloud semantic labeling using deep segmentation networks. 3DOR, 2:7, 2017.
Wieland Brendel and Matthias Bethge. Approximating cnns with sas-of-local-feature models works surprisingly well on imagenet. International Conference on Learning Representations, 2019. URL fMWMAQYY
htttp:://ppenrev//
net
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Osear Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVT conference on computer vision and pattern recognition, pp. 1621-11631, 2020.
John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):6796698, 1986.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2959-2968 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 098882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-super vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR. 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.0423 04297, 2020c.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski the convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3075-3084 2019.
Angela Dai and Matthias Nießner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. Proceedings of the European Conference on Computer Vision (ECCV), pp. 452-468, 2018.
10

---

Under review as a conference paper at ICLR 2022
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A surge-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Di Feng, Yiyang Zhou, Chenfeng Xu, Masayoshi Tomizuka, and Wei Zhan. A simple and ef- ficient multi-task network for 3d object detection and road understanding. arXiv preprint arXiv:2103.04056, 040060.0.006.... 2021.
A. Geiger, P. Lenz, and R. Urtasun. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In Proc. the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 3354-3361, 2012.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trainee cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: /8/ 1. /2231, 2018
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587,1 2014.
Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with simple and effective baseline. arXiv preprint arXiv:2106.05:04, 2021a.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang. Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2103.01988, 2021b.
John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series c (applied statistics), 28(1):100-10) 1979.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 635-645. Springer, 2016b.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9733, 2020.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning, pp. 4182-4192. PMLR, 2020.
R Devon Hjelm, Alex Fedorov, Samuel
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 00080000660.... 2018.
Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. arXiv preprint arXiv:2012.09165, 2020.
Binh-Son Son Hua, Minh-Khin Tran, and Sai-Kit Yeung. Pointwise convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 984-993, 2018.
11

---

Under review as a conference paper at ICLR 2022
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh. Pre-training without natural images. In Proceedings of the Asian Conference Computer Vision, 2020.
Roman Klokov and Victor Lempitsky. Escape from cells: Deep dd-neeworks for the recognition of 3d point cloud models. In Proceedings of the IEEE International Conference on Computer Vision, pp. 863-872, 2017.
Artem Komarichev, Zichun Zhong, and Jing Hua. A-cnn: Annularly convolutional neural networks on point clouds. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7421-7430, 2019.
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12697-12705, 2019.
Felix Järemo Lawin, Martin Danelljan, Patrik Tosteberg, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. Deep projective 3d semantic segmentation. In International Conference on Computer Analysis of Images and Patterns, pp. 95-107. Springer, 2017.
Dogyoon Lee, Jaeha Lee, Junhyeop Lee, Hyeongmin Lee, Minhyeok Lee, Sungmin Woo, and Sangyoun Lee. Regularization strategy for point cloud via rigidly mixed sample. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15900-15909, 2021.
Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9397-94466 2018a.
Yangyan Li, Rui Bu, Mingehao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolu- tion on x-transformed points. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 828-838, 2018b.
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolu- tional neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 806-814, 2015.
Yongcheng Liu, Bin Fan, Gaofeng Meng. Jiwen Lu, Shiming Xiang, and Chunhong Pan. Densepoint: Learning densely contextual representation for efficient point cloud processing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5239-5248, 2019.
Yueh-Cheng Liu, Yu-Kai Huang, Hung- Yueh Chiang, Hung-Ting Su, Zhe- Yu Liu, Chin-Tang Chen, Ching Yu Tseng, and Winston H Hsu. Learning from 2d: Pixel-to-point knowledge transfer for 3d pretraining. arXiv preprint arXiv:2104.044 04687, 202
Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look at local aggregation operators in point cloud analysis. In European Conference on Computer Vision, pp. 326-342 Springer, 2020.
Zhengzhe Liu, Xiaojuan Qi, and Chi- thin Fu. 3d-to-2d distillation for indoor parsing. arXiv preprint arXiv:2104.0224 022443, 2021b.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers universal computation engines. arXiv preprint arXiv:2103.05247 2021.
Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. Rangenet++: Fast and accurate lidar semantic segmentation. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4213-4220. IEEE, 2019.
François Pomerleau, Francis Colas, and Roland Siegwart. A review of point cloud registration algorithms for mobile robotics. Foundations and Trends in Robotics, 4(1):1-104, 2015.
12

---

Under review as a conference paper at ICLR 2022
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation, 2016. U cite arxiv:1612.00599
Charles R Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning point sets in metric space. arXiv preprint drXiv:1706.024133 2017.
Shi Qiu, Saeed Anwar, and Nick Barnes. Dense-resolution network for point cloud classification and segmentation. In Proceedings of the IEEF/CVF Winter Conference Applications Computer Vision, pp. 3813 3813-3822, 2021.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21K pretraining for the masses, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as a metric for image retrieval. International journal of computer vision, 40(2):99- 2000.
Hongming Shan, Yi Zhang, Qingsong Yang, Uwe Kruger, Mannudeep K Kalra, Ling Sun, Wenxiang Cong, and Ge Wang. 3-d convolutional encoder-decoder network for low-dose et via transfer learning from a 2-d trained network. IEEE transactions on medical imaging, 3766):1522-1534, 2018.
Baoguang Shi. Song Bai, Zhichao Zhou, and Xiang Bai. Deeppano: Deep panoramic representation for 3-d shape recognition. IEEE Signal Processing Letters, 22(12):3339-233 2015. doi: 10. 1109/LLSP.215..2888800
Sketchup. 3d modeling online freel3d warehouse models. httpp::/3Bwarriiussesskkett com
2021.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned- Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pp. 0055--53,22 2015.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. CVPR, 2019.
Kamrul Hasan Talukder and Koichi Harada. Haar wavelet based approach for image compression and quality assessment compressed image. arXiv preprint arXiv: 1010. 4084, 2010.
* Tang. Zhijian* Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. European Conference on Computer Vision, 2020.
Carlo Tomasi and Roberto Manduchi. Bilateral filtering for gray and color images. In Sixth international conference computer vision (IEEE Cat. No. 98CH36271), pp. 839-846. IEEE, 1998.
Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. O-enn: Octree-based convolutional neural networks for 3d shape analysis. ACM Transactions on Graphics (TOG), 36 (4):1-11, 2017.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning point clouds. Acm Transactions On Graphics (tog). 38(5):1112, 2019.
Zining Wang, Wei Zhan, and Masayoshi Tomizuka. Fusing bird's eye view lidar point cloud and front view camera image for 3d object detection. In 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1-6. IEEE, 2018
13

---

Under review as a conference paper at ICLR 2022
Bichen Wu, Alvin Wan, Xiangyu Yue, and Kurt Keutzer. Squeezeseg: Convolutional neural nets with recurrent erf for real-time road-object segmentation from 3d lidar point cloud. In ICRA, 2018.
Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer. Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from lidar point cloud. In ICRA, 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: deep representation for volumetric shapes. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2015.
Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European Conference on Computer Vision, pp. 574 591. Springer, 2020.
Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeczesegy3: Spatially-adaptive convolution for efficient point-cloud segmentation. In European Conference on Computer Vision, pp. 1-19. Springer, 2020.
Chenfeng Xu, Bohan Zhai, Bichen Wu, Tian Li, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. You only group once: Efficient point-cloud processing with token representation and relation inference module. arXiv preprint (XXi::2103309975 2021.
Xun Xu and Gim Hee Lee. Weakly supervised semantic point cloud segmentation: Towards 10x fewer labels. In Proceedings of the IÉEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13706-13715, 2020.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18 (10):3337, 2018
Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 7652-7660, 2018.
Xiangyu Yue, Bichen Wu, Sanjit A Seshia, Kurt Keutzer, and Alberto L Sangiovanni- A lidar point cloud generator: from virtual world autonomous driving. Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval, pp. 458-464, 2018.
Jinlai Zhang, Lyujie Chen, Bo Ouyang, Binbin Liu, Jihong Zhu, Yujing Chen, Yanmei Meng, and Danfeng Wu. Pointcutmix: Regularization strategy for point cloud classification. arXiv preprint arXiv:2101.01461, 2021a
Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. arXiv preprint arXiv:2 101.02691, 2021b.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16259-16268. 2021.
Hui Zhou, Xinge Zhu, Xiao Song, Yuexin Ma, Zhe Wang, Hongsheng Li, and Dahua Lin. Cylin- der3d: An effective 3d framework for driving-scene lidar semantic segmentation. arXiv preprint arXiv:2008.00 01550, 2020.
14

---

The pretrained ResNet50 on Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 IMPLEMENTATION DETAIL.
Our experiments are conducted on ModelNet 3D Warehouse, S3DIS, and SemanticKITTI datasets. For the ModelNet 3D Warehouse dataset, we train all models on the train set and evaluate on the validation set. For the S3DIS, we train all models on area 1, 2, 3, 4, 6 and evaluate on area 5. For the SemanticKITTI dataset, we train all models on splits 00- 10 except 08 which used for evaluation. For each of the dataset, all ResNet series models use the same training scheme, and all experiments are implemented with PyTorch.
When training on the ModelNer 3D Warehouse dataset, coordinates of point-clouds are randomly scaled, translated, and jittered. We use SGD optimizer with momentum 0.9, weight-decay 10-4, and initial learning rate 0.1 with cosine learning rate scheduler. Each mini batch is set to 32, and models are trained for 300 epochs. For both training and inference phase, we only utilize x. y. z coordinates without other features and set voxel size to 0.05. The experiments for ModelNet 3D Warehouse all conducted on a Titan RTX GPU,
When training on the S3DIS dataset, we concatenate all subparts of an indoor scene train and validate on. Along x, y directions, scenes are applied horizontal flip randomly. RGB features are randomly jittered, translated, and auto contrasted. Finally, normalize and clip point-clouds. We set voxel size to 0.05, use SGD optimizer with momentum 0.9, weight-decay 10-4, and initialize learning rate 0.1 with polynomial learning rate scheduler. Each mini batch is set 3, and models are trained for 400 epochs on 2 Titan RTX GPUs.
When training on the SemanticKITTI dataset, coordinates of each point-cloud are randomly scaled and rotated. We use SGD optimizer with momentum 0.9, weight-decar 10-4, and initial learning rate 0.24 with cosine warmup learning rate scheduler. Each mini batch set to 2, and models are trained for 15 epochs on 4 Titan RTX GPUs. For both training and inference phase, we utilize x, y, coordinates as well as intensity feature and set voxel size to 0.05.
Most of our pretrained models come from open-sources, 5 so do not need to take time and computational resources for pretraining. We use torchsparse to produce sparse 3D convolutions.
Details of Section 4.1 In this section, we take the ResNet architecture, inflate the pretrained models of different image datasets, and add linear input and output layers as shown in Section A.9 The ResNet50 pretrained
ImageNetIK is directly taken from PyTorch. We use the same training recipe provided by PyTorch to train the ResNet50 on ImageNet21 K comes from Ridnik al. 2021
Details of Section 4.2 Section and 4.4 In this section, the ResNet pretrained models are taken from the same sources illustrated above.
For PointNet++ pretraining on ImageNetIK, we utilize the PointNet++ SSG version (Qi ct al. 2017). We break the image into pixels and regard the group of pixels as a point-cloud with coordinates of y positions in the original image and appending = = 1 to all pixels. Then, we set center sampling number 1024 and 256 for first and second stage, and the radius set into and 64, respectively. For each center point, we query 64 neighbouring points. The training recipe is also provided by PyTorch.
For ViT models, we directly take the pretrained weights from Dosovitskiy et al. (2020). To apply it on ModelNet 3D Warehouse, we sample 256 centers and group 64 nearby points, regarding these as "point-cloud patches". Then, we use linear embedding to project the point-cloud patches into sequence, and ViT processes them same as image patches. Except the linear embedding and the final
15

---

directly use the Under review as a conference paper at ICLR 2022
Table 6: ResNet50 results (evaluated on ModelNet 3D Warchouse) of fintuning the mean and variance in batch normalization layers on different datasets. 10 indicates finctuning input and output layer, 10ms indicates updating input, output, mean and variance, IOmsWb indicates finetuning input, output and the whole BN.
Layers
Image-t
ImageNetIK
ImageNet21K
FractaIDBIK
FractalDB10K
67.666
81.199
73.744
83.347
80.105
10ms
83.793
82.942
84.076
72.326
79.66
I0msWb
89.992
89.87
90.721
89.263
89.344
From scratch
90.316
90.316
90.316
90.316
90.316
Table ResNet18, 50, 152 results (evaluated on ModelNet 3D Warehouse) of finetuing the mean and variance in batch normalization layers.
Layers
ResNet18
ResNet50
ResNet152
10
71.029
81.199
64.627
10mv
81.888
82.942
82.658
IOmvWb
88.574
89.87
90.438
From Scratch
90.397
90.316
90.276
output classifier, all the models are kept same as the origin version. For the experiments on S3DIS and SemanticKITTI, the architecture detail of ResNet18 shown in A.5 listing 2
For Simple model, all the experiment settings are the same as Goyal et al. (2021a). The only difference is whether to the pretrained ResNet18. For PRRNeV2--W45 ImageNetI and Cityscape pretrained models from (Sun et al. 2019).
We conduct three trials the few-shot experiments. For each trial, we change the random seed but keep all the other settings the same. To plot the training speed curve, we directly use the training log without any other changes, such as smoothing.
A.2 FINETUNING THE MEAN AND VARIANCE OF BATCH NORMALIZATION.
For the first group of experiments, ResNet50 FIP either has IO or IO+BN finetuned. In addition these two experimental settings, we also investigate fintuning input, output layers, and mean, variance of normalization layers, while fixing the convolution layer weights, normalization layer weights, and bias. The full experiment results with this extra setting is reported in Table and 7 We can observe that compared with only finetuning input and output layers, updating mean and variance can also largely improve the performance of point-cloud recognition. As suggested in Section 5.2 in the main paper, updating mean and variance is to push the FIP models to generate point-cloud representation that is similar to image representation.
A.3 ABLATION STUDY OF INFLATING TOWARDS DIFFERENT DIRECTIONS.
We conduct experiments of inflating filters along different directions, the illustration figure is shown Figure 5 and the results are shown Table g We find that the performance is different when using different inflation methods. In particular, with ResNet50 pretrained on ImageNetlK, inflating along x axis and the axis leads to better performance compared with inflating along z axis for both FIP-IO and FIP-IO+BN. More importantly, the minimally finetuned FIP-IO+BN with inflating along the x and y axis even surpasses the maining-from-scrateh
A.4 ABLATION STUDY OF LOADING DIFFERENT STAGES OF IMAGE-PRETRAINER MODEL.
We investigate the effect of loading different subset of stages. The results are shown in Table 9 In detail, we load the pretrained weights partially while keeping the other weights randomly initialized.
16

---

Under review as a conference paper at ICLR 2022
2D Filter
3D Filter
X axis
Y axis
Z axis
Figure 5: Visualization of filter inflation along different axis.
Table 8: ModelNet 3D Warehouse results (Top-1 accuracy) of partially finetuning ResNet50 pretrained on ImageNet1K K with inflation along the x, y, axis.
Method
x axis
y axis
axis
FIP-IO
82.17 90.44
81.73 90.84
81.20
FIP-IO+BN
89.87
From Scratch
90.32
Table 9: ModelNet 3D Warehouse results (Top-1 I accuracy) of finetuning ResNet50 pretrained on I with only subset of stages loaded.
Loaded Stages
2
1 223
2 3
2
3 4
FIP-ALL
89.91
90.36
90.64
90.92
91.09
90.19
89.99
From Scratch
90.32
We observe that excluding the weights of the first stage achieves the best performance, bringing 0.77 points improvement.
A.5 AN INTUITION OF WHY THE IMAGE-PRETRAINED WEIGHTS CAN BE USED FOR POINT-CLOUD RECOGNITION.
In this subsection, we try to give an intuition of why the weights pretrained on images can be used for point-cloud tasks from the aspect of local feature projection. Let's only look at the one time convolution on a local point-cloud feature Xlocat E RCi XCoee In fact, a filter inflation can be represented as linear projection, given by W24 @ T W3a where W2a E RC x Cout xX? is pretrained 2D weight, Wad RCsn x is the transformed 3D weight from W24, and T E RK* x R the projection matrix. Therefore, for a local 3D feature Xtocel E RC x Cous R3 the convolution can be formed as W3d @ Xiocal = W X T @ Xlocal = Was @ "Xtocat-24 where Xlocal-2d E RCinxCoo K2 is the 2D feature projected from Xtocat- Essentially, applying the inflated filters on 3D point-clouds equal to the local projection of 3D point-clouds to 2D then applying the pretrained 2D filters. A visual explanation in Figure
3D Convolution
Linear Projection
2D Convolution Projected Voxels
Local Voxels
3D Filter
Local Voxels
Default T
2D Filter
Projected Vosels
2D Filter
Figure 6: Visualization of filter inflation explained with linear projection matrix.
17

---

Under review as a conference paper at ICLR 2022
Table 10: ModelNet 3D Warchouse results (Top- accuracy) of finetuning ResNet18 pretrained on different
Method
Original image
Canny edge detection
Bilateral filters
Haar transformation
FIP-1O
78.89
78.85 (-0.04)
78.97 (+0.08)
78.49 (-0.40)
Method
Shuffle patch 56
Shuffle patch 28
Shuffle patch 14
Shuffle patch 7
FIP-IO
78.44(-0.45)
78.00 (-0.89)
76.21 (-2.68)
74.60 (-4.29)
A.6
MORE VISUALIZATION NETWORK DISSECTION.
We present visualization results based on the technique of network dissection, as shown in We can observe that for most cases, although there is no obvious visual concept transferred, the pretrained filters are prone cluster similar objects on ModelNet 3D Warehouse dataset.
Figure 7
A.7 USING DIFFERENT IMAGE PROCESSING METHODS FOR PRETRAINING.
To further investigate the image-to-pone toe-ttoooot-oooudd transfer in a more rigorous manner, we design a new set of experiments and report them in Appendix A.7 in the paper revision. The basic idea is that perform a transformation (such as low-pass filter) on the original image to destroy certain properties (high frequency components). Then, train ResNet18 point-cloud recognition. This way, we can better attribute which parts the images are useful when transferred to point-cloud recognition. We perform the following transformations. Implementation code is taken from external sources R
the transformed images, then transfer to
1) Canny edge detection (Canny discarding the low-frequency such as the surface of objects;
1986 transforms the images into high-frequency edge maps,
2) Bilateral filters (Tomasi & Manduchi] 1998 preserve the low- www-freuency properties of images while filtering out the high-freeuency noise. Note that the high-frequency edges are still preserved and even sharpened;
3) Haar wavelet image compression Talukkeer & Harada 2010). The wavelet transform divides the information of an image into approximation and detailed sub-signals. If the detail small, then would be thrown;
4) Patch shuffling divides an image into patches with different patch sizes, shuffle the patches, and rearrange them into new image. This transformation preserves the statistics of the images, while destroying the original order of the pixels. "Shuffle patch 56" means the patch size 56 56.
The results are reported in the Table TO Specifically, the edge images from canny edge detection are dramatically different from original images while keeping minimal performance drop. The bilateral filters discard the high-rreuency noise, sharpening the high- gigh-ffeeuency edge lead to the best performance, even outperforming the result of finetuning from the original image pretraining. As details of images are thrown (haar transformation), the performance slightly drops. On the other hand, shuffling the patches results in worse performance as the patch size gradually becomes small, which indicates that the order of the pixels are important to the transfer.
Therefore, we may infer that the structured (ordered) edges are important to the transfer.
A.8 DETAILS OF FIRST- WASSERSTEIN DISTANCE ON RESNET18.
We list all the results in each layer of ResNet18, 8, as shown Table TI We can observe that for each layer, the first- fast-wwaserstein distance is largely reduced when finetuning more parameters.
18

---

Under review as a conference paper at ICLR 2022
Table 11: First-wasserstenn distance between image model features and different 3D model features of all 16 layers ResNet18
Training-from: scratch
Image Model
FIP/I
FIP/1O-BN
FIP/ALL
Average
10a
0.27
0.093
0.15
Layer
1.9
0.2
0.094
0.12
Layer 2
1.1
0.064
0.11
0.048
Layer
1.3
0.24
0.068
0.095
Layer 4
2.6
0.14
0.051
0.077
Layer 5
1.1
0.23
0.039
0.082
Layer
0.21
0.025
0.063
Layer 7
5.4
0.29
0.05
0.15
Layer
13
0.26
0.051
0.097
Layer 9
0.28
0.04
0.095
Layer
0.2
0.077
0.13
Layer
0.27
0.032
0.13
Layer
10a
0.25
0.076
0.094
Layer
1.9 10²
0.29
0.028
0.11
Layer
7.7 x 10a
0.16
0.075
0.041
Layer 15
9.5 x 10²
0.29
0.081
0.088
Layer 16
1.2 x 10
0.89
0.59
A.9 DETAILS OF USED ARCHITECTURES.
Class 3DRes cls (nn. :oddlee)
def _init_ (self, res_block):
super () the conventional ResNet. Resteettt
res_bleck means the residual block as sane
self input__layerrrr Seqquenntiall
sparse cinpad dim, @yyerl_Iding k=3, s=1), sparse_bn (laye__bnlayerr__dddi
self. layerl la
layerl_Idim,
layeri_odimi
felf.layer2 inflated_resnet_layee22 (res layer2_Idim,
1ayer2_odim)
self, layer3 - inflated_resnet_laye (res_blocc 1ayer3_Idim,
Layer3 Odim)
11
self. layerd resset
resnet Layer4 beocklockk layerd Idim,
1ayer4_ Odin)
12
self.output_ output_layer nn. Sequential alobal_average_poolll nn. Linear (layer4_odim, class_num), bn (clas_s__s_nmm
15
16
def forward x)
19
X self.
(x)
20
x self. calerlayerl (x)
self. (x)
self. fel (x)
self. (x)
19

---

Under review as a conference paper at ICLR 2022
self. output (x)
Listing Pseudo code of inflated ResNet with linear input and output for classification
Class 3DRes seg (nn. Module)
def isit_ res block)
super ()__innt____))_
res_bleck means the residual block as sane
the conventional ResNet. Restet.
self. self.input_layer- .ts.uuueaeeesmmseeueeee.eeee n...equentiail
sparse_conv3d(inert dim, layer1_ k=3, s=1),
sparse_bn(Layer__ddim)
sparse_ReiU(True),
aparsa__conv38@ayyyy1lldal Idim, layer1 k=3, s=1),
sparse__on([ayrr1Idddm)))
10
sparse_Rel((Irue),
11
gparse_conv3@[1ayer1[oogg Iddi,,, layerl_Idim, k-3, s-2),
(parsa_bn((aayrr_[ddm) sparse_Rel.
14
15
seyer
self. layerl = inflated seslated_resne__ine leyeee (res_blockk layer1 Idim,
layerl Odim)
self. layer2 inflarddreennt_llyyeerree...dd layer2_ldim,
(ayer2_odim))
17
self.layer3
self. layer4 = arrlateeureemee_llayyrrtet_e___d layer3_Idim,
layer3_Odimg
(res (res_block, layer4_Ide
layer4_Odim)
19
20
self.upl = sparse cetersedddcovvveeee (layer4_odial 1ayer4_0dim, k=2, s=2),
self.decodel = Sel.f.eeqentiell
res res_block
1ayer3_odir), res_block (layer3_0dim, layer3_odim))
24
25
splf.up2 = sparse (Layer3 Odim, Sayeer__ddin k=2, s=2) self. decode2 = self Self.Sequential
block (layer3 Odim+layer2 _dimm, layer2_Odir) res_block (layer2_odim, layer2_odim))
28
29
30
self. up3 sparse hatprs___ddcoontaaaaa_ooooo
laycr2_odim, k-2, s=2)
self.decoder = self Sell. res res_block block (layer2_0dimtlayerr res_block (layeri_Odim, 1ayerl_0dim)
layerl_odim),
33
34
35
self up4 = sparse deersseyecccnn selt.Sequentiall
Layeri_Odim, k=2, s=2)
37
res block (layerl Laa Odim, layerl_Odim)
38
res_block playerl_odine 1ayer1_0dim)]
39
self. ouye nn.Sequential(
Lnnein (layerl_Odiie class_num))
42
43
def forward x)
44
xi self. solf.input_layer (x)
x1 self. self.layeri (x_1)
16
x2 self.layers (x1)
47
x3 1ayer (x2)
48
x4 se1 fayerr (x3)
x3 self decoder1 (cat (x3, self (x4)
31
x2 self decoder2 (cat (x2, self up2 11)
52
xl
(cat (x1, self. up3 (x)
53
xi_ self.decodere (cat (x_i, self. (x1_])
return self. ouypeulllyerreeeteea (xi_)
Listing Pseudo code of inflated ResNet for segmentation
20

---

Under review as a conference paper at ICLR 2022
Unit 16
Unit 17
Unit 38
Broden
Modelinet
Unit
Broden
Modainet
Unit 35
Unit 42
102
Broden
Modelnat
Unit 227
Unit 394
Unit 420
Broden
Modeinet
Unit 169
Unit
Unit 198
Broden
Modelnat
Unit 427
Unit 438
Unit 487
Brodin
Modeinet
Unit 80
Unit
Unit 109
Broden
Modelnat
Brodan
Modeinet
Figure More network dissection results.
21


Reviews:
[{"review_id": "z1SBMPf8Gw_", "reviewer": "Reviewer_KdMv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper proposes a model to connect 2D and 3D representations in transfer learning.\n- Detailed experiments show that the proposed model improves the performance and data efficiency compared to training from scratch.\n\nWeaknesses\n- Will larger dataset leads to better performance? A lot of comparisons between results from datasets of different size are missing in Table 1. \n- The paper shows that using 2D pretrained features improve performance. However, results of state-of-the-art task specific methods should also be listed. In addition, it would be better to compare with different self-supervised learning method.\nFor example, for shape classification on ModelNet40:\nFoldingnet: Point cloud auto-encoder via deep grid deformation. CVPR 2018\nPointcontrast: Unsupervised pretraining for 3d point cloud understanding. ECCV 2020\n-  Explanation why this kind of inflating 2D filter to 3D is reasonable. There is a huge domain gap between 2D dataset and 3D dataset. It's still unclear why this kind of transfer is reasonable. It would be better to add theoretical analysis besides experiments.", "summary_of_the_paper": "The paper proposes to transfer the 2D image features for 3D point cloud understanding tasks. Specifically, the proposed method first inflates the 2D convolution filters from a pretrained model to 3D and then optimizes only input, output and optionally batch normalization layer.  Detailed experiments show that the proposed finetuned-image-pretrained models can improve the performance and data efficiency.\n", "summary_of_the_review": "Although the experiments in the paper show that the proposed model can improve the performance and data efficiency compared to training from scratch, the overall experiments results are not strong enough. In addition, it's still unclear why this kind of 2D to 3D transfer is reasonable. I am inclined to reject.\n\nUpdate: After reading the feedbacks from the author, I raise my score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uglJT9PfIGh", "reviewer": "Reviewer_2iHv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: The idea of transfer learning between image and point cloud data seems to be novel and surprising.\n\nCons: \n1. The idea of inflating model weight from 2D to 3D is not new[1].  \n2. The performance on point cloud segmentation on SemanticKitti dataset, while is reasonably well, still has gap to the state-of-the-art(This one is minor)\n3. The effectiveness of pretrained weight would be more convincing if the author could do further ablation study to initialize new model with expanding subsets of layers and exploring the effect on model performance. It will indicate which part of the pertained model is really useful in this transferred learning setting.\n\n=====\nUpdate after rebuttal: After reading the author's response and the new experimental results amended by the author, I raise my score to 6.\n\n[1] Carreira, Joao, and Andrew Zisserman. \"Quo vadis, action recognition? a new model and the kinetics dataset.\" proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.", "summary_of_the_paper": "This paper described an experiment of transfer learning between image and point cloud data. Despite of apparent dissimilarity between the two domain, the author was able to use structure and weights from pertained models in image domain to get reasonable performance in point cloud tasks like classification and segmentation. The author also made comparison of using pertained weights versus training from scratch, the result shows training pertained weights improves data efficiency. The proposed method is also proven to be useful when the training data is limited by training under few-shot learning setting. \n", "summary_of_the_review": "Overall the paper has interesting experimental result on transfer learning between image and point cloud data. The authors did a good job explaining the idea, concepts, procedures and experiments, showing the effectiveness of the transferred model on performance. data efficiency and usefulness on few-shot setting. The experiment itself could be further extended though.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qSxlT6S-p13", "reviewer": "Reviewer_DFK1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- This paper investigates a new problem of transferring 2D pre-training to 3D.\n- The presentation is clear and easy to follow.\n- The experiments are extensive. The performance gain using the 2D pre-training is notable and non-trivial, showing the proposed weight transferring technique has the ability to convert useful learned information on 2D to 3D.\n\nWeakness:\n\n- It's good that the authors showed some visualizations to help understand the proposed framework. However, as the authors pointed out, the visualization cannot show why the transfer works or what information can be transferred.\n- The last sentence on page 8 is questionable: the conclusion of \"shape representations are better transferred from image to point-cloud\" cannot be inferred from overall dataset performances. Also, what does \"shape representations\" mean? \n", "summary_of_the_paper": "This paper proposed a pipeline for transferring convolutional network weights that are pre-trained on 2D images to 3D convolution networks. The proposed approach is to inflate the 2D kernels to 3D kernels similar to video models. The experiments are conducted on 2D image datasets such as ImageNet, and 3D datasets such as ModelNet.", "summary_of_the_review": "This paper proposed a solution to a new problem of transferring 2D weights to 3D. The solution works well on many dataset pairs. Though some claims made in the paper are questionable, this paper deserves to be published at the conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "H5DHmJb_F9c", "reviewer": "Reviewer_aajt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths: \n1. Idea is good and helpful for the 3D point cloud field.\n2. The experiments are designed to test different advantage of this idea. The results support the arguments. \n\nWeaknesses / questions: \n1. Since such idea has already used in other 2D-3D domain, the novelty is limited. Such paper needs extra emphasis on discussing why it works and ablation studies. \n  - In discussion section, the author provides some experiments, but the result seems quite random (figure 4). The \"shape representation\" are better transferred seems make sense, but why? The author tried to explain the finding but seems not quite convincing.\n  - When designing inflation, is there a difference on which axis to inflate. x, y or z axis. This is different to video since 3 axis is symmetric. \n2. The biggest concern is the parameter sizes and network structure.\n  - In 4.1, the author did not compare the parameter size. I doubt the fairness of the comparison with the baseline since it is possible that the performance gain is fully from the increasing parameters. \n  - In 4.2, there is a possibility that the ResNet structure is not good structure to train on point cloud making it quite bad on scratch training. For pointnet++, the training method is quite strange. \n3. The writing for this paper is interesting... not formal enough. The author used 10 \"surprising(ly)\" in this paper...", "summary_of_the_paper": "This paper introduced \"inflation\" (2D CNN kernel to 3D one by repeating on the 3rd axis) to transform a 2D image pre-train backbone into a 3D version so that 3D point cloud task can be benefit from 2D image pretraining. Detailed experiments are performed to address this idea. Results show minimal finetuning efforts can achieve competitive performance on 3D tasks. The authors are kind of \"surprised\" by these results.", "summary_of_the_review": "This is an interesting idea. This paper provided detailed experiments to test different advantage of this idea. However, the discussion on why it work seems not quite convincing. Also, part of the experiments missed some details. Some conclusions may not correct.\nWe expect the author provided more convincing discussion and more comprehensive comparison detail.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1zCA49f-mw7", "reviewer": "Reviewer_SBBb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe paper is simple yet very interesting, which simple inflate the 2D convolution kernels to 3D kernels to learn the representation of the 3D point clouds. 2D images and 3D point cloud are of great differences. Thus, it seems to impractical to simply performed the inflation and make it work. However, the authors proposed one simple idea and demonstrate that the idea does work.  I like the simple idea and would like to recommend the acceptance.\n\n\nWeakness:\n1. Some detailed information is missing. Therefore, I am not able to follow the detailed process. In Table 1, the authors list the performances of pointnet++ and the performances. I am curious how to performance the inflation in point net++\n2. The authors perform the experiments on the classification and segmeantion. What about the performances on detection task.\n\n\n\n\n=========\nPost-rebuttal\n\nI have read the authors rebuttal as well as the other fellow reviewers' comments and the corresponding discussions. \n\nI agree with the other fellow reviewers' comment, 2D inflating to 3D has been studied in video action recognition. However, to me, inflating 2D CNN of image for 3D pointcloud is new. Therefore, I do think the paper is of novelty. \nAlso I agree with the other reviewers' comments that the discussion of why the proposed method works does not been clearly addressed. \n\n\nConsidering this, I am lowering my rating from 8 to 6.\n", "summary_of_the_paper": "The paper proposed to perform the 3D point cloud understanding based on the pretrained 2D image models. Simple inflation technical, same as the work in I3D, are used to convert the 2D convnet to 3D convnet.\n\nExtensive experiments performed on the point cloud classification and segmentation tasks demonstrate the effectiveness of the work.\n", "summary_of_the_review": "It seems to the reviewer that this is the first paper to use 2D image pertained models for 3D point cloud understanding. Although the idea is simple, the work is interesting and inspiring. Therefore, I am recommending the acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to transfer the image-pretrained model to a point cloud model by inflating 2D convolutional filters to 3D convolutional filters and finetuning the inflated image-pretrained model, so that 3D point cloud tasks can benefit from 2D image pretraining. Extensive experiments are conducted to validate the effectiveness of the proposed method. Even though the performance gain using the 2D pre-training is notable, the novelty of the paper is limited since inflating 2D model to 3D video action recognition has been studied, and theoretical understanding of the proposed model is lacking. During the rebuttal period, the authors addressed most of the reviewers’ concerns by conducting additional experiments. Even though the performance is compelling, all reviewers agree that the novelty for the paper is limited and the discussion on why this method work is not convincing. Meanwhile, one reviewer points out that some claims made by authors are not well supported. Besides, one reviewer points out that the paper might have a broader impact in a computer vision conference but only provide a limited contribution to the ICLR community. After an internal discussion with reviewers. the AC agrees with the reviewers on their judgments and recommends rejecting the paper because of the limited novelty of the paper.
Decision: Reject