Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?
Authors: Runpei Dong
Zekun Qi
Linfeng Zhang
Junbo Zhang
Jianjian Sun
Zheng Ge
Li Yi
Kaisheng Ma
Venue: ICLR 2023 poster (2023)
Keywords: Representation Learning
Cross-Modal Learning
3D Point Clouds

Abstract: The success of deep learning heavily relies on large-scale data with comprehensive labels, which is more expensive and time-consuming to fetch in 3D compared to 2D images or natural languages. This promotes the potential of utilizing models pretrained with data more than 3D as teachers for cross-modal knowledge transferring. In this paper, we revisit masked modeling in a unified fashion of knowledge distillation, and we show that foundational Transformers pretrained with 2D images or natural languages can help self-supervised 3D representation learning through training Autoencoders as Cross-Modal Teachers (ACT). The pretrained Transformers are transferred as cross-modal 3D teachers using discrete variational autoencoding self-supervision, during which the Transformers are frozen with prompt tuning for better knowledge inheritance. The latent features encoded by the 3D teachers are used as the target of masked point modeling, wherein the dark knowledge is distilled to the 3D Transformer students as foundational geometry understanding. Our ACT pretrained 3D learner achieves state-of-the-art generalization capacity across various downstream benchmarks, e.g., 88.21% overall accuracy on ScanObjectNN. Codes have been released at https://github.com/RunpeiDong/ACT.

Published as a conference paper at ICLR 2023
AUTOENCODERS AS CROSS-MODAL TEACHERS: CAN PRETRAINED 2D IMAGE TRANSFORMERS HELP 3D REPRESENTATION LEARNING?
Runpei Dong! Zekun Qi-
Linfeng Zhang Junbo Zhang²
Jianjian Sun
Zheng Ge
Li Yia: Kaisheng Ma
Xi' an Jiaotong University Tsinghua University MEGVVI Technology
4 Shanghai Artificial Intelligence Laboratory $ Shanghai Qi Zhi Institute
ABSTRACT
The success of deep learning heavily relies large-scale data with comprehensive labels, which is more expensive and time-consuming to fetch in 3D compared to 2D images natural languages. This promotes the potential of utilizing models pre- trained with data more than 3D as teachers for cross- rross-modal knowledge transferring. In this paper, we revisit masked modeling in a unified fashion of knowledge distil- lation, and we show that foundational Transformers pretrained with 2D images or natural languages can help self supf-supervieed 3D representation learning through train- ing Autoencoders as Cross-Modal Teachers (ACT). The pretrained Transformers are transferred as cross-modal 3D teachers using discrete variational autoencoding self- self-supervision, during which the Transformers are frozen with prompt tuning for better knowledge inheritance. The latent features encoded by the 3D teachers are used as the target of masked point modeling, wherein the dark knowledge is dis- tilled to the 3D Transformer students as foundational geometry understanding. Our ACT pretrained 3D learner achieves state-of-the-art generalization capacity across various downstream benchmarks, e.g., 88.2 18.21% overall accuracy on ScanObjectNN. Codes have been released at https: / //gth/tttttt//// com com//RunpeiDonn//ACT.
INTRODUCTION
In recent years, AI systems powered by data-driven deep learning have been deployed in various areas (LeCun et al., 2015; He et al., 2016; Vaswani et 2017). The advancements in computing hard- ware have largely facilitated machine intelligence developments, which also encourages an emerging paradigm of transferring models trained on broad data, i.e., foundational models (Bommasani et al., 2021). Great success has been witnessed in natural language processing (NLP) (Devlin et al., 2019; Radford et al., 2018; 2019; Brown et al., 2020; Radford et al., 2021), where the models are designed to learn generic representations through self-supervised knowledge probing on data of extreme size. Since the rapid development of Transformer (Vaswani al., 2017) in vision (Dosovitskiy et al., 2021; Liu et al., 2021b), various efforts have been made to spread this trend from NI.P towards foundational 2D visual understanding (Bao et al., 2022; He al., 2022b; Wang al., 2022a).
Mcanwhile, compared to 2D vision and NLP, this course towards foundational visual computing is significantly lagging in the 3D community. We ask: What makes 3D representation learning more challenging than 2D vision or NLP? We offer some analytical answers from the following three perspectives:
Table 1: Data pattern comparison.
Format
Scale
Semantics
Language
Broad
Dense & Structured
RGB Pixel
Large
Sparse & Unstructured
Coordinates
Moderate
Sparse & Unstructured
Architecture disunity. Pioneering architectures like PointNet (Qi et al., 2017abb) can only en- code 3D coordinates and it is not applicable for masked denoising autoencoding (DAE) (Vincent et al., 2008; 2010; Devlin al., 2019) which is proved successful in NLP and 2D vision (He et al., 2022b). Transformers (Vaswani et al., 2017) has now closed this architectural gap, which enables unified representation across all modality formats (Wang et al., 2022a) and brings a great potential of extending DAE for 3D (Yu et al., 2022; Pang et al., 2022).
CCoresponding authors.
Work partially done during the internship of Runpei Dong (runpei dong@gmai com) at MEGVIT.

---

Published as a conference paper at ICLR 2023
ii. Data desert. In comparison to images and free-form languages, it is more difficult collect and label 3D (Chang et al., 2015) or 4D (Liu et al., 2022b) data, which generally requires more expensive and labor-intensive efforts. In addition, 3D data seriously lacking considering the scale of data'. This motivates the usage of cross-modal knowledge transfer. Recent works either jointly train with other modalities for more effective contrast (Afham et al., 2022) or directly fine-tune 2D Transformers pretrained on image data (Wang et al., 2022b).
ii.. Pattern difference. Table I shows the data pattern comparison of languages, 2D images and 3D point clouds. It is observed that: (i) 3D point cloud is usually unstructured containing sparse semantics unlike the language. This leads the discrete identification learning for stRT-style tokenizer (Devlin et al., 2019) on point clouds more difficult (Yu et al., 2022) (see Sec. 6.1). (ii) 2D images regularly distributed on grids, while 3D point clouds irregularly sampled from the object surface. This structural difference leads to the difficulty of constructing contrastive targets both for single-modality augmentations (Hou al., 2021) and for cross- modal correspondence (Li et al., 2022). (iii) How to design a better representation with enriched semantics becomes the de-facto principal for sulf-super 3D understanding.
Motivated by the analysis above, we propose to train Autoencoders as Cross-Modal Teachers (ACT). Our ACT utilizes foundational Transformers pretrained with 2D images or natural languages as cross- modal teachers, carrying profound knowledge and powerful representation capacity. In this way, the data desert issue in 3D is alleviated. Transformer is employed as the generic 3D learner, which closes the architectural gap toward masked modeling representation learning. By simply tuning pretrained Transformers as autoencoders on 3D data in self-supervised fashion, the Transformers can consume and encode 3D point clouds into representations with rich semantics. In order to preserve and inherit the pretrained foundational knowledge, prompt tuning (Jia et al., 2022) is used during this procedure. As result, our ACT makes the pretrained Transformers spontaneously cross-modke teachers that provide semantically enriched masked modeling targets for 3D point clouds.
Since the pretrained Transformers are tuned as 3D autoencoders, no image, language data, or 3D downstream annotations are required during this cross-modal Transformer transfer. Besides, as the tuned Transformers are only used as the teacher for 3D Transformer student learning, our method does not introduce additional computing or storage costs during downstream feature transferring. Extensive experiments on various tasks have been conducted, which show the superior generalization performance of our ACT pretrained 3D Transformers. For example, average accuracy improvement of +1 11... .99% is achieved on ScanObjectNN dataset.
To the best of our knowledge, this paper firstly shows that a pretrained foundational Transformer can help 3D representation learning without accessing any 2D, language data, or 3D downstream annotations. ACT is a self-supervised framework that can be generalized to other modalities and tasks, expect this could spur more exploration of such ACT-style representation learning.
2 RELATED WORKS
Self-Supervised Representation Learning for 3D Geometric Processing is currently arousing significant interest the community. Classical methods are built upon reconstruction-base geometry understanding pre-tasks, egg. point cloud part reordering (Sauder & Sievers, 2019), orientation estimation (Poursaeed et al., 2020), local and global reconstruction (Rao et al., 2020), flow consis- tency (Mittal al., 2020), deformation (Achituve et al., 2021). and occlusion (Wang et al., 2021). Concurrently, Xic et al. (2020) propose PointContrast to Iearn discriminative view consistency be- tween augmented point clouds. Following this direction, various works have been proposed (Zhang et al., 2021; Hou et al., 2021; Chen et al., 2022). Recently, many works have proposed to apply DAE pretraining of point cloud Transformers, and remarkable success has been achieved. Yu et al. (2022) pioneers this direction by extending the idea of BERT-style pretraining (Devlin et al., 2019; Bao et al., 2022), combined with a global contrastive objective (He et al., 2020). Liu al. (2022a) propose to add some noisy points and classify whether the masked tokens are real or fake for each masked position, which shares a similar pattern with Selfie (Trinh et al., 2019) that classifies whether masked image patches are real or fake. Pang et al. (2022) proposes exploring MAE on point clouds by masked modeling of 3DD point cloud coordinates. We follow this DAE-style representation learning paradigm, but different from previous methods, our work seeks to use latent features encoded by the 3D autoencoder with pretrained foundational Transformers as masked modeling targets.
For example, the in-house JFT-300M dataset from Google covers over one billion labels for 300M images, and the Common Crawl dataset (Raffel al., 2020) for NL.P consists of nearly one trillion words.
2

---

Published as a conference paper at ICLR 2023
Cross-Modal 3D Representation Learning aims at leveraging more moddlltt-inneeeen learning signals besides 3D point clouds, e.g., 2D images are known to have rich contextual and textural knowledge, while free-form languages are of dense semanties. Mainstream methods are developed upon contrastive learning of global feature matching. For instance, Jing et al. (2021) propose a discriminative Center loss for feature alignment of point clouds, mesh, and images. Afham al. (2022) propose an intra- and inter-modal contrastive learning framework among augmented point clouds and the corresponding rendered 2D images. By utilizing the geometry prior information for dense association, another line of work is proposed to explore fine-grained local feature matching. Liu et al. (2021a) propose contrastive knowledge distillation method to align fine-grained 2D and 3D features. Li et al. (2022) propose a simple contrastive learning framework for inter- and intra- modal dense feature contrast, with the Hungarian algorithm used for better correspondence. Recently, great progress has been made by directly using pretrained 2D image encoders via supervised fine-tuning. Image2Point (Xu et al., 2022) proposes to transfer pretrained weights by convolutional layer inflating. P2P (Wang et al., 2022b) proposes to project 3D point clouds to 2D images imput to the image backbone through a learnable coloring module. Our work also explores whether pretrained foundational models could help 3D learning. However, method (1) does not use the pretrained 2D or language models as the backbone model for inference, (2) explores using pretrained foundational models from other modalities during self-supervised pretraining without downstream 3D annotations, and (3) does not need the paired point-image or point-language data. Besides 2D images, some works are proposed to utilize natural languages for contrastive 3D representation leanring (Rozenberszki et al., 2022), zero-shot learning (Zhang et al., 2022c), and scene understanding (Zhang et al., 2023).
3 PRELIMINARIES
3.1 3D POINT CLOUD REPRESENTATIONS WITH TRANSFORMERS
Different from images that lie on regular grids, point clouds are known to be irregular and less structured. Many efforts have been devoted to deep learning architecture design for point cloud data (Qi et al., 201 b Wang et al., 2019), which exploits permutation and translation invariance of a point set for feature learning. Instead of purely relying on such specialized backbones, we leverage the Transformer backbone (Vaswani et al., 2017), which is casier to be unified with other modalities such as image and language and to facilitate cross-modee knowledge transfer. We feed Transformers with local geometry patch embeddings computed using specialized point networks like Qi et al. (2017a) to output more effective geometric representations.
Local Geometry Patch Embedding Suppose we have point cloud P {p:li = 1,2,..., ,N R.N x3 with N coordinates encoded in a (z,, y, 2) Cartesian space, we follow Yu et al. (2022) to first sample N, seed points using farthest point sampling (FPS). The point cloud is then grouped into N. neighborhoods N = {Nii 1, 2....... N.} E R N, K x3 with group centroids from the seed point set Ps. Each neighborhood contains K points generated by searching the K-nearest neighbor of the corresponding seed point. The local geometry feature x; around each seed point p: E P, is computed by max- max-pooling per-point features within the neighborhood:
MAX (oo (4i,)), EN,
X
where (-)()) is a point feature extractor with parameters 8, e.g., per-point MLP in (Qi et al., 2017a;b), Ex, is the feature of j-th neighbour point Pi.j in the neighborhood Mg. We will use the set of neighborhood features as token features feed the following Transformer blocks.
Transformer Point Feature Encoding Standard Transformer block (Vaswani et al., 2017) is used as the encoder to further transform local patch embeddings X = {x.|i = 1,2, N.) E R.N.CC with C' being the embedding size. Following Yu et al. (2022). we a two-layer MLP dip with learnable parameters p as the positional embedding, which is applied to every block for stable training.
Epes
[ [CLS], (P.)).
pos
E [CLS]
RC
(2)
pos
ho = E [CLS]; X1; X2; XN.] + Epos:
(CLS)
E CRC,
Epuc E (Nx+
(3)
h': = MSA + Epos)) + he-1,
f= 1
(4)
h MLP(LN(h')) + h'z,
{ 1
(5)
where MSA denotes altermating layers of multi-head self-attention, LN denotes Layernomm and MLP is two layers with GELU as non- lineariia,..y. E [ctss] is a learnable global representation embedding with E E(cLo) its learnalle positional embedding (Dosovitskiy et al., 2021).
3

---

Published as a conference paper at ICLR 2023
Languages
Masked Point Modeling with Toahhr Feature Distillation
Oute cut with biue eyea astring on the pollow
Reconstruction
V 0 O 0
distill
Self-Supervision
Studons Prodictions
3D Autoencoder Decoder
Decoder
pretrain
Embedding
Musk Takara
Lune as crose moddl maached Tt Aertoenoooooo
Transformers
Transformer
Transfer
Pretranid Transformer Frozznn & Prompti
Point Cloud Transformer
[Standard Transformrr
Repre Repres-mmann ereeerttton Learnes
(uved for freeuum www.utt-ttee toiisteriig wwwferrriigg
(VLT/BERT/CLIP/->
Vith Trikee
O
O O
0 O
Teacher T Encader
pretrain
Frompn
Local Goomerr Embreednng
Embedding
Embedding
FPS Grouping
FPS & Grouping
Images from ImageNet
Point Cloud Inputs
Point Cloud Inputs
(a) Pretroining Transformers Lorge-scale Data
(b) 3D Autoencoding with Pretrained Transformer
(c) Masked Modeling
Cross Cross-Modal Modal Knowledge Distillation
Figure 1: Overview of of our ACT framework (Sec. 3-4). (a) ACT utilizes the Transformers pretrained on large-scale data, e.g., ViT (Dosovitskiy et al., 2021) pretrained with 2D images or BERT (Devlin al., 2019) pretrained with languages. (b) Stage I of ACT (Sec. 4.1), the pretrained Transformers are tuned by self-supervised 3D autoencoding with prompts (Jia et al., 2022). (c) Stage of ACT (Sec. 4.2), the 3D autoencoder encoder used cross-modal teacher that encodes latent features as masked point modeling targets for 3D Transformer student representation learning.
3.2 KNOWLEDGE DISTILLATION: A UNIFIED VIEW OF MASKED MODELING
Masked signal modeling be viewed as an extension of the classical denoising autoencoders (DAE) with masked corruption (He et al., 2022b), which has been recently explored for language models (Devlin al., 2019) and vision (Bao et al., 2022). Formally, given sequence of N, tokens T tt = 1, 2, ..,,,... Neh. e.g., the token embeddings of an RGB image or point cloud data. The objective is to train the a student encoder fs to predict/reconstruct the output from teacher encoder fr. where the teacher could be discrete variational autoencoder (dVAE) (Bao et al., 2022) or simply identity mapping (He et al., 2022b). this fashion, the student learns the dark knowledge within data under the guidance of the teacher. In order to corrupt the input data, a set of masks E {0. 10.1) are generated for each position, indicating whether the token is masked or not. A learnable corruption embedding e [M] is used to replace the masked position, with which the corrupted representation ZM = 1(M) O e [M]] + 1(1 M) 3 T is input to encoder (Devlin et al., 2019) or decoder (He et al., 2022b) Here, @ denotes the Hadamard product,
M = {myli 1,2,.. ..... N₂)
and 1 is the indicator function. With a distance function Lo(:,) -) defined in some metric space D and hs, hy as the decoders, the objective is minimize:
N m4 Co o fs(z hy ST(T)).
(6)
The decoders h vary with the modeling targets, e.g., it is non-linear projection with softmax for BERT (Devlin et al., 2019; Bao et al., 2022) where the metric function becomes Cross-Entropy. Eqn. (6) can be viewed as unified formulation for masked modeling. is thus natural to consider how to build a knowledgeable teacher in masked 3D modeling. And our idea is to leverage cross-modal teachers from 2D or language foundation models.
For MAE, the encoder only receives visible tokens, and the T for caleulating M should fs (It, Iym, m. E MI), where the corrupted representation MM is fed into the decoder for masked modeling distillation.
4

---

Published as a conference paper at ICLR 2023
4 ACT: AUTOENCODERS AS CROSS-MODAL TEACHERS
Our goal to facilitate 3D representation learning through a pretrained 2D image or language Transformer, which carries dark knowledge absorbed from massive data. However, 3D point clouds are known to have different structures (Li et al., 2022; Afham et al., 2022) from 2D images or languages, which makes the association of fine-grained knowledge difficult. We address this issue by using a two-stage training procedure. An overview of our ACT framework is illustrated in Figure 1.
Stage I. We tune the pretrained 2D or language Transformers as 3D autoencoders, where it learns to understand 3D geometry through self-supervised prompt tuning (Sec. 4.1).
Stage II. We use the pretrained 3D autoencoder
modal teacher, which is used to distill the latent features to the 3D point cloud Transformer student through masked modeling (Sec. 4.2).
4.1 3D AUTOENCODING WITH PRETRAINED FOUNDATIONAL TRANSFORMER
Transformers, recently the dominant architecture in various areas, can model sequential data of any modality in a unified fashion (Vaswani et al., 2017). Therefore, we could directly use the pretrained Transformer blocks by feeding the sequential tokens with 3D positional embeddings of the input point clouds, as described in Sec. 3.1. A lightweight DGCNN is used following Yu et al. (2022), where o in Eqn. (1) represents the edge convolution layer (Wang al., 2019).
Cross-Modar Embedding with Prompts The point cloud P is first encoded by the DGCNN-style patch embedding network g producing set of token embeddings: = g gP)) (P). Then we prompt the token embeddings and feed them into D layers of pretrained and frozen Transformer blocks, e.g., 2D Transformer 9²th 1g 2, .... D). Here use gi to denote the b-th layer of the 2D Transformer. We use learnable prompt embeddings E,P) {e( E I N, 1 S k s m). which applied to each layer of the Transformer (Jia et al., 2022). Specifically, the b-th layer 9² of the Transformer transforms the hidden representations h from the (e - 1)-th layer to h, as below:
[, E [P] 9²"h 2rn ([by-1; E,
1= D
(7)
With this sarameter-efficieee prompt tuning strategy, we are able to tune the pretrained foundational Transformer while preserving as much pretrained knowledge as possible (He et al., 2022a).
Point Cloud Autoencoding Another DGCNN network g is used to extract local geometric features from foundational Transformer-embedeee hidden representations hę. After this, we leverage FoldingNet (Yang et al., 2018) to reconstruct the input point cloud. We train the above 3D autoencoder as a discrete variational autoencoder (dVAE) (Kingma & Welling, 2014; Ramesh et al., 2021; Bao et al., 2022) for log-likelihood P(pi(p:) maximization, where (pi.P.) E D denotes the original and reconstructed point clouds respectively. The overall optimization is to maximize the evidence lower bound (ELBO), which holds when / 1 (Ramesh et al., 2021):
Σ In Pn N
Σ (P..At)ED
(BerQs(E)))) In (pol())) - 3Lxx. Px
(PD.Pi)ED
(8)
where (1) (zlp)) denotes the discrete 3D dVAE tokenizer; (2) (p.pp))) is the dVAE decoder given discrete point tokens; (3) Po(zlp) reconstructs the input point clouds in autoencoding way.
4.2 MASKED POINT MODELING AS CROSS-MODAL. KNOWLEDGE DISTILLATION
By simply training the 3D autoencoder, the strong representation of the pretrained Transformer is translated into the 3D feature space, making the autoencoder spontancously cross- cross-modal teacher. We motivate our method with a similar formulation to Eqn. (6). We use the pretrained point cloud 3D Transformer Fs = hg o fs as the student. The masked point modeling as cross-modal knowledge distillation minimizes a negative cosine similarity Loos t)
encoder introduced in Sec. 4.1 the teacher Fy hy og o 2D g and
between the encoded teacher and b
student features:
No mi Leoe ),
(9)
5

---

Published as a conference paper at ICLR 2023
Table 2: Classification results on ScanObjectNN. Ours": results trained with no data augmentation. Ours?: results trained with simple point cloud rotation. DA: data augmentation is used during fine- tuning training. The overall accuracy, i.e., OA (%) is reported.
Method
#Params(M)
DA
OBJBC
OBJ ONLY
PB T50 RS
Supervised Learning Only
PointNet (Qi ct al., 2017a)
3.5
73.3
79.2
68.0
SpiderCNN (Xu et al., 2018)
77.1
79.5
73.7
PoimNet++ (Qi et al., 2017b)
1.5
82.3
84.3
77.9
DGCNN (Wang al., 2019)
1.8
V
82.8
86.2
78.1
PoinCCNN (Li al., 2018)
0.6
V
86.1
85.5
78.5
BGA-DGCNN (Uy al., 2019a)
1.8
79.7
BGA-PN++ (Uy et al., 2019a)
1.5
V
80.2
DRNet (Qiu et al., 2021)
80.3
GBNet (Qiu et al.. 2022)
8.8
80.5
Simple (Goyal et al., 2021)
80.5+
PRANet (Cheng et al., 2021)
2.3
81.0
MVTN (Hamdi et al., 2021)
82.8
PointMLP (Ma ct al., 2022)
13.2
85.4+10.3
with Self Superviied Representation Learning (FULL)
Transformer (Vaswani et al., 2017)
22.1
V
79.86
80.55
77.24
OcCo (Wang al., 2021)
22.1
4
84.85
85.54
78.79
Point-BERT (Yu al., 2022)
22.1
v 87.43
88.12
83.07
MaskPoint (Liu et al., 2022a)
22.1
v 89.30
88.10
84.30
Point-MAE (Pang et al., 2022)
22.1
V 90.02
88.29
85.18
ACT (Ours*)
22.1
X 91.22
89.16
85.81
ACT (Ours?)
22.1
93.29
91.91
88.21
Point-MAE (Pang et al., 2022)
22.1
V
89.31-10.41
0.36
84.35
ACT (Ours*)
22.1
90.06:10.56
89.02:10.22
0.227
ACT (Ours?)
22.1
92.48-10.59
91.5770.37
87.8810.36
with Self-Supervised Representation Learning (MI.P-LINEAR)
Point-MAE (Pang et al., 2022)
22.1
82.58+0.588
83.52+-0.41
73.08+0..0
ACT (Ours!)
22.1
82.71+0045
84.34+-0.29
74.17+0.05
ACT (Ours?)
22.1
85.20+0.83
85.84+00.15
76.31+0.26
with Supe-Siud Representation Learning (MLP-3)
Point-MAE (Pang et al., 2022)
22.1
84.2911.555
85.2410.67
77.34
ACT (Ours²)
22.1
85.67:0..29
86.79.0.30
78.29
ACT (Ours?)
22.1
87.14+0.22
88.90=10.40
81.52+0.
EXPERIMENTS
5.1 TRANSFER LEARNING DOWNSTREAM TASKS
Transfer Protocol We three variants of transfer learning protocols for classification tasks:
(a) FULL: Fine-tuning pretrained models by updating all backbone and classification heads.
(b) MLP-LINEAR: The classification head is single-layer linear MLP, and we only update this head parameters during fine-tuning
(c) MLP-3: The classification head is three-layer non-linear MLP (which is the same as the one used FULL), and we only update this head parameters during fine-tuning.
3D Real-world Dataset Classification We first show the evaluation of 3D shape recognition on the challenging real-world dataset ScanObjectNN (Uy et al., 2019b). The results are shown in Table 2, where it is observed that: (i) Comparing to Transformer from scratch baseline under FULL tuning protocol, our ACT gains significant improvement of +10.4% accuracy averaged the three variant ScanObjectNN benchmarks. Further, with simple point cloud rotation, ACT achieves an average improvement of +1 +19%; (ii) In comparison to methods explicitly designed with 3D geometry understanding purpose, our ACT achieves consistently better results. (iii) Compared to other self-supervised learning (SSL.) methods, our ACT achieves the best generalization across all

---

Published as a conference paper at ICLR 2023
Table 3: Classification results the ModelNet40 dataset. The overall accuracy, i.e. OA (%) is
reported. [ST]; standard Transformer architecture.
Method
[ST] #Point OA (%)
Method
[ST]
PPoit
OA (%)
Supervised Only
Self Supervised Representation Learnnng (FULL)
PointNet eQi 2017a)
89.2
Transformer (Vaswani al., 2017)
Ik
91,4
PointNet++ (Qi et al. 2017b)
lk P 90.7
Transformer (Vaswani al., 2017)
P
91.2
PointNet++ (Q) et al., 2017b)
Sk P+N
91.9
OcCo (Wang al., 2021)
Ik
92.1
PoinCCNN (Li et 2018)
lk
92.5
OcCo (Wang ot al., 2021)
4k
92.2
PoiniConv (Wu et al., 2019)
Ik P+N
92.5
Point-BERT (Yu et, 2023)
Ik
93.2
KPConv (Thomas al., 2019)
Ik P
92.9
Point-MAE (Pang ct al., 2022)
IkP
93.8
DGCNN (Wang et 2019)
Ik P
92.9
ACT (Ours)
IKP 93.7
RS-CNN (Liu et al., 2019b)
Ik P
92.9
Point-MAE (Pang ct al., 2022)
IkP
93.1240.25
DensePoint (Liu et al.. 2019s)
Ik P
93.2
ACT (Ours)
1kP
93.50.10.08
PointASNL (Yan et al., 2020)
Ik P
92.9
with Self-Superiised Representation Learning (MLP-LINEAR)
PosPool (Lii et al., 2020)
P 93.2
DRNet (Qiu et al., 2021)
P
93.1
Point-MAB (Pang et al., 2022)
Ik
91.2244.26
ACT (Ours)
1kP
91.36:10.17
Point Trans. (Engel et al., 2020)
92.8
PCT (Guo et al. 2021)
1k P
93.2
with Self-Supervised Representation Learning (MLP-3)
PointIransformer (Zhao ot al., 2021)
Ik 93.7
Point-MAE (Pang et al., 2022)
Ik
92.33:00.09
NPCT (Guo et al., 2021)
lk P
91.0
ACT (Ours)
92.69.10.18
transferring protocols on ScanObjectNN. Besides, ACT succeeds in reaching the trate-of-the-at (SOTA) performance among methods using pure 3D Transformer architecture on ScanObjectNN, e,g., ACT outperforms Point-MAE by +3.0% accuracy on the most challenging PB_T50_RR JSS benchmark.
3D Scene Segmentation Semantic segmen-
Table 4: Semantic segmentation results on the S3DIS Area 5. The mean accuracy and mean IoU across all categories, i.., mAcc (%) and mloU (%) are reported. zyz: point cloud coordinates are used. zyz+rgh: both coordinates and RGB color are used.
tation large-scale 3D scenes is challeng- ing, demonstrating the understanding of con- textual semantics and local geometric relation- ships. In Table 4, we report the results on S3DIS dataset (Armeni al., 2016). It can be seen that: (i) ACT significantly improves the from scratch baseline by +2.5% and +1.2% mAce and mloU, respectively. (ii) ACT outperforms the SSL counterpart Point-MAE by +1.2% and +0.4% mAce and mloU, showing superior transferring capacity on the large-scene dataset. (iii) With only geometric inputs zyz, ACT can achieve comparable or better performance to architec- tures with the meticulous design using zyz+rgh data, including 3D-specific Transformer archi- tecture (Guo et al., 2021).
Methods
Input
mAce (%)
mloU (%)
PointNet
wyz+rgb
49.0
41.1
PointNct++
zys+rgb
67.1
53.5
PointCNN
zys+rgb
63.9
57.3
PCT
wyz+rgb
67.7
61.3
Transformer
xyz
68.6
60.0
Point-MAE
zyz
69.9
60.8
ACT (Ours)
ay²
71.1
61.2
3D Synthetic Dataset Classification We show
Table 5: Few-shot classification on ModelNet40, overall accuracy (%) is reported.
the evaluation of 3D shape classification on syn- thetic dataset ModelNet40 (Wu et al., 2015). To demonstrate the data-efficiency property of ACT given limited training examples, we first follow Sharma & Kaul (2020) to evaluate few- shot learning. From Table 5, we see: (i) ACT brings significant improvements of +9.0%, +4.7%, +8.7%, +6.2% respectively for the four settings over from scratch FULL. transferring baseline. (ii) Our ACT consistently achieves the best performance compared to other SSL methods. Then, show results the full dataset in Table 3, where we observe that our ACT achieves +2.5% accuracy improvement compared to the from scratch baseline under FULL protocol, and the results are comparable or better to other self-supervised learning meth- ods across all transferring protocols.
5-way
10-way
Method
10-shot
20-shot
10-shot
20-shot
DGCNN
31.6
40.8 4.6 19.9 4: 2.1 16.9
OcCh
90.6 2.8
92.5 + 1.9 82.9 + 1.3 86.5 2.2
with Self Supervised Representution Learning (FULL)
Transformer 87.8 5.2 93.3 4,3 84.6 5.5 89.4
6,3
OcCo
94.0
95.9
894 5.1
92.4
4.6 4.6
Point-BERT
94.6 3.1
96.3
91.0
3,4
92.7
5.1
Point-MAE
96.3 3 2,5
97.8
92.6
4.1 95.0 3.0
(Ours)
96.8
1.4
Self Supervised Represemation Learning (MLE-LINEAR)
Miint-MAE ACT (Ours)
91.1 5.6 91.7 1 4.0
83.5 1 6.1 89.7 4.1
91.8 4.7
93.1
: 4.2
84,5 : 6.4 90.7 4,3
with Self Supervised Representation Learning (MLP-3)
Point-MAE 95.0 96.7 + 2.4 90.6 4 4.7 93.8 = 5.0
ACT (Ours) 95.9 97.7 + 1.8
4 5.0 94.7

---

Published as a conference paper at ICLR 2023
Table 6: Ablation study the depth of the pretraining decoder.
Figure 2: Ablation study of masking ratio and cross- modal Transformer teacher choice.
(a) Masking Strategy
(b) Teacher Choice
85.5
85.5
Dec. Depth
OA (%)t
83.69
84.5
84.5
85.11
85.33
83.5
83.5
rendom
block
84.98
0.6
VTT.S
VIT-B
D
DuTT-B
ResHLE
Miver
BERT-B
CLIP-B
Masking Ratio
Table 7: Ablation study on different training strategies of the dVAR tokenizer. The F-Score, Chamfer distance using L1-norm and 1.2-norm, Le., CD-& and CD-42 reported.
Methods
Num. of Prompt
Prompt Type
Freeze
F-Scoret
CD-4 4
CD-L2 4
Point-BERT dVAE
N/A
N/A
N/A
0.166
25.933
2.697
DeïT-B dVAE
N/A
0.175
24.589
2.380
DeiT-B dVAE
N/A
V
0.180
24.090
2.274
DeîT-B dVAE
32
shallow
V
0.188
23.769
2.196
DeïT-B dVAE
32
deep
V
0.189
23.873
2.173
DeīT-B dVAE
deep
0.189
23.229
2.127
ViT-B dVAE
64
deep
0.193
23.524
2.110
5.2
ABLATION STUDY
Decoder Depth Table 6 shows the average fine-tuning accuracy on ScanObjectNN using ACT with different depths decoders. can be seen that the performance is not sensitive the decoder depth, and we find that decoder with 2 blokcs achieves the highest results. Note that when decoder depth is we adopt a masked modeling architecture similar to BERT (Devlin et al., 2019), where there is no decoder, and the encoder all tokens, including masked ones. We find that this leads to inferior result, consistent with the observation in 2D that data of low semantics requires a non-trivial decoder for modeling purpose (He al., 2022b).
Masking Strategy and Teacher Choice Figure 2(a) shows the average fine-tuning on ScanobjectNN with different masking strategies. It can be observed that higher masking ratio using random masking yields better results, while block masking has an appetite for lower masking ratios. Note that when the masking ratio is zero, we use vanilla knowledge distillation for all tokens, and leads to inferior performance. Figure 2(b) shows average fine-tuning accuracy on ScanObjectNN using ACT with different teacher Transformers including Vision Transformers (Dosovitskiy et al., 2021; Touvron et al., 2021b), alI-MLP architectures (Tolstikhin et al., 2021; Touvron et al., 2021a). language model (Devlin et al., 2019) and vision-language model (Radford et al., 2021). is observed that a larger teacher consistently yields better performance. Moreover, surprisingly, our ACT with language model BERT-B (i.e., BERT base) as the cross- modal teacher can achieve an average accuracy of 85, 12..2. 54%
(up to 85. 85.88%) demonstrating that ACT can generalize any modality.
3D Autoencoder Training Table shows the reconstruction results of different training config- urations for the 3D autoencoder with pretrained 2D image Transformer. It is observed that: (i) Our 3D dVAE model with pretrained image Transformer achieves significantly better reconstruction results than BERT. It demonstrates that the pretrained 2D image Transformers have strong representation capacity for 3D. (ii) Prompt tuning or freezing the model leads to better results than full tuning, and we argue that it is because some pretrained 2D knowledge is forgotten, and prompt tuning effectively addresses this issue. Reconstruction visualizations can be found in Appendix D.
DISCUSSIONS
6.1 IS A STRONGER TOKENIZER ALL YOU NEED?
In order to understand the necessity of the pretrained 2D image Transformer in the 3D dVAE model, we have conducted experiments with different dVAE teachers and masked modeling configurations. From Table 8, image Transformers, by distilling the latent feature instead discrete tokens, we can achieve +0.62% improvement. Our analysis agrees that discrete token identification is more challenging learn for
that: (i) When using the Point-BERT dVAE model without pretrained 2D
8

---

Published as a conference paper at ICLR 2023
Table 8: Study the effect of pretrained image Transformer-basee 3D Autoencoder.
Table 9: Study of applying method auxil-
lary knowledge distillation during pretraining.
Teacher
Target
OA (%)
Method
KD
OA (%))
Point-BERT
Point-BERT
83.07
Point-MAE
X
85.18
Point-BERT
Ours
83.69
Our Impl.
86.05
Ours
Point-BER -
82.51
Our Impl.
84.35+00.1
Ours
Ours
85.81
Our Impl.
84.96:10.58
Table 10: Study of different positional embeddings for 2D image transformer in dVAE model. (a)
N/A: no positional embedding is used. (b) 2D/2: positional embedding with only 2D zy plane coordinates. (c) 3D: positional embedding with all 3D æyz coordinates. The F-Score, Chamfer distance using Ll-norm and L2-norm, i.e., CD-&, and CD-62, and OA on ScanObjectNN are reported.
Methods
pos embed
F-Scoret CD-
CD-f2
OA (%)f
ViT-B dVAE
N/A
0.166
25,918
2.698
84.21::0.45
VIT-B dVAE
2D/2
0.184
24.135
2.259
85.10=:.445
VIT-B dVAB
3D
0.193
23.524
2.110
85.33+0.27
3D data. (ii) When using Point-BERT discrete token the masked modeling target, by applying dVAE model with pretrained 2D image Transformers, we get the worst performance. It demonstrates that the discrete tokens are not suitable for the semantically sparse point cloud data, matter how strong the tokenizer is. (iii) When using our ACT, the performance is significantly improved. It demonstrates that the 3D dVAE with pretrained 2D image Transformer can encode features with rich semantics, which is better suited for masked point modeling.
6.2 CAN ACT BE USED AS AN AUXILIARY KNOWLEDGE DISTILLATION METHOD?
Since our ACT uses encoded features as masked modeling targets, it brings another potential to apply our method as auxiliary feature distillation. Table 9 shows the results of training Point-MAE with ACT auxiliary deep supervision of the intermediate features, where the ACT encoded latent features are distilled to the encoder feature of Point-MAE. We can observe that ACT can improve Point-MAE significantly by +0.87% of accuracy on ScanObjectNN, demonstrating that ACT is scalable and effective as a knowledge distillation method.
6.3 How DOES THE 2D VISION TRANSFORMER UNDERSTAND 3D POINT CLOUDS?
To better understand how the 2D image Transformers understand 3D inputs through the autoencoder training, we study the effect of positional embedding used by ViT-B in our ACT dVAE model. From Table 10, we can observe that: (i) Without any positional embedding, the pretrained VIT still learss transferable 3D features (84.21
accuracy). We argue that it is because the positional geometry information is already contained in the input 3D coordinates and the pretrained 2D Transformer can process 3D data purely by geometry features without explicit positional hints. (ii) When using positional embedding with only 2D zy plane coordinates, accuracy is improved significantly by +0.89%. We argue that 2D positional embedding is learned to fit the frozen image Transformer, enabling the image Transformer to encode 3D inputs into pretrained 2D feature space with high semantics. (iii) With all 3D coordinates used for positional embedding, the 2D image Transformer succeeds in leveraging the additional coordinate information for better feature encoding.
7 CONCLUSIONS
This paper presents self- self-supervised learning framework ACT that performs masked modeling as feature distillation from pretrained foundational Transformers to 3D Transformer students. ACT first transfers the pretrained foundational Transformers cross-modal 3D teachers via self-supervised 3D autoencoding. The semantic-enriched latent feature from the tuned 3D autoencoder then used as masked modeling targets for the 3D Transformer students' representation learning, which shows remarkable generalization performance over various downstream 3D tasks. As a general SSL framework, believe ACT could be easily extended to other modalities than 3D data. A great potential is shown to transfer cross-modee knowledge in this self-supervised fashion, which may largely facilitate the development of foundational modeling in this data-driven deep learning era.

---

Published as a conference paper at ICLR 2023
REFERENCES
Idan Achituve, Haggai Maron, and Gal Chechik. Self-supervised learning for domain adaptation on point clouds. In IEEE Winter Conf. Appl. Comput. Vis. (WACV), pp. 123-133, IEEE, 2021.
Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thi- lakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2022.
Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese, 3d semantic parsing of large-scale indoor spaces. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 1534-1543, 2016.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In Int. Conf. Learn. Represent. (ICLR). OpenRevie 2022.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi. and et al. On the opportunities and risks of foundation models. CoRR, abs/2108.0m 000258, 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2020.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. (KDD), pp. 535-541. ACM, 2006.
Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. CoRR,
2015.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proc. Int. Conf. Mach. Learn. (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 1691-1703. PMLR, 2020a.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proc. Int. Conf. Mach. Learn. (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 15750-15758, 2021.
Yujin Chen, Matthias Nießner, and Angela Dai. 4dcontrast: Contrastive learning with dynamic correspondences for 3d scene understanding. In Eur. Conf. Compul. Vis. (ECCV), 2022.
Silin Cheng, Xiwu Chen, Xinwei He, Zhe Liu, and Xiang Bai. Pra-net: Point relation- -warr network for 3d point cloud analysis. IEEE Trans. Image Process. 30:44664444444 2021.
Ching Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debi- ased contrastive learning. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2020.
10

---

Published as a conference paper at ICLR 2023
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias 3d reconstructions of indoor scenes. In IEEE/CVE Conf.
NieBner. Scannet:
Comput. Vis. Pattern Recog. (CVPR), pp. 5828-52885 (2288-589,440 2017.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In /EEE/CVF
Conf. Comput. Vis. Pattern Recog, (CVPR), pp. 886-893. IEEE Computer Society, 2005.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 248-255, IEEE Computer Society, 2009.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171-4186. Association for Computational Linguistics, 2019.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin A. Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI), 38(9):1734 1747, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In Int. Conf. Learn. Represent. (ICLR). OpenReview.net, 2021.
Nico Engel, Vasileios Belagiannis, and Klaus Dietmayer. Point transformer. CoRR, abs/201 0ats/20110009311 1.00931, 2020.
Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with simple and effective baseline. In Proc. Int. Conf. Mach. Learn. (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 3809-3820. PMLR, 2021.
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukeuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self-supervised learning. In Adv. Neural Inform. Process. Syst. (NeurlPS), 2020.
Meng- Hao Guo, Junxiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R. Martin, and Shi-Min Hu. PCT: point cloud transformer. Comput. Vis. Media, 1(2):187-199 2021.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 1742, 2006.
Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. MVTN: multi-view transformation network for 3d shape recognition. In Int. Conf. Comput. Vis. (ICCV), pp. 1-11. IEFF, 2021.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-effccient transfer learning. Int. Conf. Learn. Represent. (ICLR). OpenReview.net, 2022a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. JEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 9726-9735. Computer Vision Foundation / IEEE, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autocncoders are scalable vision learners. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2022b.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In Ady. Neural Inform. Process. Syst. (NeurIPS), volume abs/1503.025.. 2015.

---

Published as a conference paper at ICLR 2023
R. Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In Int. Conf. Learn. Represent. (ICLR). OpenReview.net, 2019.
Karan Grewal, Philip Bachman, Adam
Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 15587-15597. Computer Vision Foundation / IEFE, 2021.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Eur. Conf. Comput. Vis. (ECCV), volume 9908 of Lecture Notes Computer Science, pp. 646-661. Springer, 2016.
Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu, Spatio-temporal self-supervised representation learning for 3d point clouds. In Int. Conf. Comput. Vis. (ICCV), pp. 6515-6525. IEEE, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Eur. Conf. Comput. Vis. (ECCV), 2022.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. In EMNLP, volume EMNLP 2020 of Findings of ACL, pp. 4163-4174, Association for Computational Linguistics, 2020.
Longlong Jing, Elahe Vahdani, Jiaxing Tan, and Yingli Tian. Cross-modal center loss for 3d cross-modal retrieval. In IEEE/CVV Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 3142-3151. Computer Vision Foundation IEEE, 2021.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Int. Conf. Learn. Represent. (ICLR), 2014.
Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. Deep learning. Nat., 521(7553):436 444, 2015.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x-transformed points. In Adv. Neural Inform. Process. Syst. (NeurIPS), pp. 828-838, 2018.
Zhenyu Li, Zehui Chen, Ang Li, Liangji Fang, Qinhong Jiang, Xianming Liu, Junjun Jiang, Bolei Zhou, and Hang Zhao. Simipu: Simple 2d image and 3d point cloud unsupervised pre-training for aware visual representations. In AAAI Conf. Artif. Intell. (AAAI), pp. 1500-1508, 2022.
Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrimination for self- learning point clouds. In Eur. Conf. Comput. Vis. (ECCV), 2022a.
Yongcheng Liu, Bin Fan, Gaofeng Meng, Jiwen Lu, Shiming Xiang, and Chunhong Pan. Densepoint: Learning densely contextual representation for efficient point cloud processing. Int. Conf. Comput. Vis. (ICCV), pp. 5238-5247. IEEE, 2019a.
Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In /EEE/CVF Conf. Comput. Vis. Pattern Recog, (CVPR), pp. 8895-8904. Computer Vision Foundation IEEE, 2019b.
Yueh-Cheng Liu, Yu-Kai Huang, Hung- Yueh Chiang. Hung-Ting Su, Zhe- Yu Liu, Chin-Tang Chen, Ching-Yu Yu Tseng, and Winston H Hsu. Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining. CoRR, fass2204.04466 2021a.
Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang. Zhoujie Fu, He Wang. and Li Yi. HO14D: A 4d egocentric dataset for letegory-level human-object interaction. In JEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR). 2022b.
Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look at local aggregation operators in point cloud analysis. In Eur. Conf. Comput. Vis. (ECCV), volume 12368 Lecture Notes in Computer Science, pp. 326-342 Springer, 2020.
12

---

Published as a conference paper at ICLR 2023
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Int. Conf. Comput. Vis. (ICCV), pp. 9992-10002. IEEE, 2021b.
Hya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Int. Conf. Learn. Represent. (ICLR). OpenReview.net, 2019.
Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: simple residual MLP framework. In Int. Conf. Learn. Represent. (ICLR). OpenReview.net, 2022.
Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Int. Conf. Comput. Vis. (ICCV), pp. 2886-2897. IEEE, 2021.
Himangi Mittal, Brian Okorn, and David Held. Just go with the flow: Self-supervised scene flow estimation. In IEEE/CVE Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 11174-11182, 2020.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Eur. Conf. Comput. Vis. (ECCV), volume 9910 of Lecture Notes in Computer Science, pp. 19-84. Springer, 2016.
Yatian Pang, Wenxiao Wang. Francis E. H. Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In Eur. Conf. Comput. Vis. (ECCV), 2022.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 20-222536--- 2544. IEEE Computer Society, 2016.
Pavlin G. Poličar, Martin Stražar, and Blaž Zupan. opentsne: modular python library for t-sne dimensionality reduction and embedding. bioRxiv, 2019. URL https: hetss::/gittub.ooo com/ pavlin-policae www.caaroppensate.....
Omid Poursaeed, Tianxing Jiang, Han Qiao, Nayun Xu, and Vladimir G. Kim. Self-supervised learning of point clouds via orientation estimation. In Int. Conf. 3D Vis. (3DV), pp. 1018-1028.
IEEE, 2020.
Charles R. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas. Deep hough voting for 3d object detection in point clouds. In Int. Conf. Comput. Vis. (ICCV), pp. 2276-9285. IEEE, 2019.
Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In /EEF/CVF Conf. Comput. Vis. Puttern Recog.
(CVPR), pp. 77-85. IEEE Computer Society, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Adv. Neural Inform. Process. Syst. (NIPS), pp. 5099-5108, 2017b.
Shi Qiu, Saeed Anwar, and Nick Barnes. Dense-resolution network for point cloud classification and segmentation. In IEEE Winter Conf. Appl. Comput. Vis. (WACV), pp. 3812-3821 IEEE, 2021.
Shi Qiu, Saeed Anwar, and Nick Barnes. Geometric back-projection network for point cloud classification. JEEE Trans. Multimedia (TMM), 14:1943-1955, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Mach. Learn. (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 8748-8788778 8763. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2023
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. (JMLR), 21:140:1-140:67 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proc. Int. Conf. Mach. Learn. (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 2021.
Yongming Rao, Jiwen Lu, and Jie Zhou. Global-local bidirectional reasoning for unsupervised representation learning of 3d point clouds. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 5375-5384. Computer Vision Foundation / IEEE, 2020.
Yongming Rao, Benlin Liu, Yi Wei, Jiwen Lu, Cho-Jui Hsich, and Jie Zhou. Randomrooms: Unsupervised pre-training from synthetic shapes and randomized layouts for 3d object detection. In Int. Conf. Comput. Vis. (ICCV), pp. 5263-3272 IEEE, 2021.
Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser-Nam Lim, and Jiwen Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. CoRR, tbs/2207.11288. 2022.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Conf. Learn. Represent. (ICLR), 2015.
Dávid Rozenberszki, Or Litany, and Angela Dai. Language-grounded indoor 3d semantic segmenta- tion in the wild. In Eur. Conf. Comput. Vis. (ECCV), 2022.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108, 2019.
Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. In Adv. Neural Inform. Process. Syst. (NeurIPS), pp. 12942-12952, 2019.
Charu Sharma and Manohar Kaul. Self-supervised few-shot learning on point clouds. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2020.
Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang. and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning. CoRR, abs/2206.01204, 2022.
Lyne Tchapmi, Christopher Choy, Iro Armeni, Jun Young Gwak, and Silvio Savarese. Segeloud: Semantic segmentation of 3d point clouds. In Int. Conf. 3D Vis. (3DV), pp. 537-547. IEEE, 2017.
Hugues Thomas, Charles R. Qi, Eeaannel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Int. Conf. Comput. Vis. (ICCV), pp. 4419. IEEE, 2019.
Yunjie Tian, Lingxi Xie, Jiemin Fang, Mengnan Shi, Junran Peng. Xiaopeng Zhang, Jianbin Jiao, Qi Tian, and Qixiang Ye. Beyond masking: Demystifying token-based pre-training for vision transformers. CoRR, abs/2203.1433 2022.
Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucie, and Alexey Dosovitskiy. Mip-mixer: An all-mip architecture for vision. In Adv. Neural Inform. Process. Syst. (NeurIPS), pp. 24261-24272, 2021.
Hugo Touvron, Piotr Bojanowski. Mathilde Caron, Matthieu Cord, Alaacldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training. CORR, abs/2105.03404 2021a.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Proc. Int. Conf. Mach. Learn. (ICML), pp. 10347-10357. PMLR, 2021b.
Trieu H. Trinh, Minh-Thang Thang Luong, and Quoc V. Le. Selfie: Self-supervised pretraining for image embedding. CoRR, abs/1 htss/190.0000 906990994000 2019.
14

---

Published as a conference paper at ICLR 2023
Mikacla Angelina Uy, Quang- Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Int. Conf. Comput. Vis. (ICCV), pp. 1588-15977 IEEE, 2019a.
Mikaela Angelina Uy Quang- Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In /EEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp- 1588-1597, 2019b.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/ $07.03748, 2018.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. J. Mach. Learn. Res. (JMLR), 9(11), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Adv. Neural Inform. Process. Syst. (NIPS), pp. 5998-6008, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proc. Int. Conf. Mach. Learn. (ICML), volume 307 of ACM International Conference Proceeding Series, pp. 103, ACM, 2008.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie. Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. J. Mach. Learn. Res. (JMLR), 11:3371-3408, 2010.
Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt Kusner. Unsupervised point cloud pre-training via occlusion completion. In Int. Conf. Comput. Vis. (ICCV), pp. 9782-9792, 2021.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision- vision-language tasks. CoRR, abs/2208. 10442, 2022a.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph CNN for learning on point clouds. ACM Trans. Graph., (:::)):44::114444
2019.
Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, and Jiwen Lu. P2P: tuning pre-trained image models for point cloud analysis with point-to-pixel prompting. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022b.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. IEEE/CVF Conf. Comput. Vis. Puttern Recog, (CVPR), 2022.
Wenxuan Wu, Zhongang Qi, and Fuxin Li. Pointconv: Deep convolutional networks on 3d point clouds. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In IEEE/CVF Conf. Comput. Vis. Puttern Recog, (CVPR), pp. 1912-1920, 2015.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE/CVF Conf. Comput. Vis. Puttern Recog, (CVPR), pp. 3733-3742 Computer Vision Foundation IEEE Computer Society. 2018.
Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas J. Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding, In Eur. Conf. Comput. Vis. (ECCV), volume 12348 of Lecture Notes in Computer Science, pp. 574-591. Springer, 2020.
Chenfeng Xu, Shijia Yang. Bohan Zhai, Bichen Wu, Xiangyu Yue, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Image2point: 3d point-cloud understanding with pretrained 2d convnets. In Eur. Conf. Comput. Vis. (ECCV), 2022.
15

---

Published as a conference paper at ICLR 2023
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning on point sets with parameterized convolutional filters. In Eur. Conf. Comput. Vis. (ECCV), 2018.
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui. Pointasnl: processing using nonlocal neural networks with adaptive sampling. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 5588-5597 5597. Computer Vision Foundation / IEEE, 2020.
Robust point clouds
Jihan Yang, Shaoshuai Shi, Runyu Ding, Zhe Wang, and Xiaojuan Qi. Towards efficient 3d object detection with knowledge distillation. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022.
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In IEEE/CVE Conf. Comput. Vis. Pattern Recog. (CVPR), 2018.
Mang Ye, Xu Zhang, Pong C. Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In JEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 0210-6219, Computer Vision Foundation IEEE, 2019.
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. CoRR, abs/2205.096 1.0.6,,,,66,,... 2022.
Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Trans. Graph., 35(6): 2016.
Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-awart transformers. In Int. Conf. Comput. Vis. (ICCV), 2021.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training
3d point cloud transformers with masked point modeling. In IEEE/CVF Conf. Comput. Vis. Pattern
Recog. (CVPR), 2022.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In Int. Conf. Learn. Represent. (ICLR). OpenReview.net, 2017.
Junbo Zhang, Guofan Fan, Guanghan Wang. Zhengyuan Su, Kaisheng Ma, and Li Yi. Language- assisted 3d feature learning for semantic scene understanding. In AAAI Conf. Artif. Intell. (AAAI), 2023.
Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In Int. Conf. Learn. Represent. (ICLR), 2021.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Int. Conf. Compur. Vis. (ICCV), pp. 3712-3721. IEEE, 2019.
Linfeng Zhang, Xin Chen, Junbo Zhang. Runpei Dong. and Kaisheng Ma. Contrastive deep supervi- sion. In Eur. Conf. Comput. Vis. (ECCV), pp. 1-19. Springer, 2022a.
Linfeng Zhang, Runpei Dong, Hung-Shuo Tai, and Kaisheng Ma. Pointdistiller: Structured knowledge distillation towards efficient and compact 3d detection. CoRR, abs/2205.1 10098, 2022b.
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by CLIP. In IEEF/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2022c.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In Eur Conf. Comput. Vis. (ECCV), volume 9907 of Lecture Notes in Computer Science, pp. 649-666. Springer, 2016.
Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In Int. Conf. Comput. Vis. (ICCV), pp. 10232-10243. IEEE, 2021.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip H. S. Torr, and Vladlen Koltun. Point transformer. In Int. Conf. Comput. Vis. (ICCV), pp. 16239-16248. IEEE, 2021.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan L. Yuille, and Tao Kong. ibot: Image BERT pre-training with online tokenizer. In Int. Conf. Learn. Represent. (ICLR), 2022.
16

---

Published as a conference paper at ICLR 2023
A ADDITIONAL RELATED WORKS
Self-Supervised Representation Learning has achieved remarkable success in both natural lan- guage processing (Devlin et al., 2019; Brown et al., 2020) and 2D visual understanding (Noroozi & Favaro, 2016; Dosovitskiy et al., 2016; Pathak et al., 2016; Yc et al., 2019). One prominent strand of research follows the contrastive objective via construct, then contrast for learning constructed invariance and consistency (Hadsell et al., 2006; Wu et al., 2018; van den Oord et al., 2018; Hjelm et al., 2019; Chuang et al., 2020; Grill et al., 2020; Chen et al., 2020b; He et al., 2020; Chen & He, 2021; Zhang al., 2022a). Another paradigm lies in training denoising autoencoders (DAE) (Vincent et al., 2008; 2010) via corrupt, then reconstruct (predict) data signals in a -efessupe fashion. With rapid development of Transformers in vision (Vaswani et al., 2017: Dosovitskiy al., 2021: Liu et al., 2021b), abundant works have been proposed to generalize DAE to masked modeling of RGB pixel (Zhang et al., 2016; Chen et al., 2020a; He et al., 2022b), pretrained DALL-E token (Ramesh et al., 2021; Bao et al., 2022). online teacher token feature (Zhou et al., 2022), and HOG feature (Dalal & Triggs, 2005; Wei al., 2022). Recently, the exploration of combining the merits of these two paradigms has been proposed by several works (Tian al., 2022; Yi et al., 2022; Tao et al., 2022).
Knowledge Distillation generally requires training of the student model to mimic the knowledge- able teacher, in which the dark knowledge transferred. This technique was first proposed by Bucila et al. (2006) for model compression purposes, which is further extended by Hinton et al. (2015) for deep neural networks. Afterwards, it becomes a most utilized technique for model compression in 2D vision (Romero et al., 2015; Zagoruyko & Komodakis, 2017: Zhang & Ma, 2021). natural language processing (Sanh et al., 2019; Jiao al., 2020) and 3D vision (Zhang et al., 2022b; Yang et al., 2022). Recently, this technique has been extended for efficient visual representation learning through self-distillation (Zhang et al., 2019) of distillation token (Touvron et al., 202 or momentum tokenizer feature (Zhou et al., 2022).
B IMPLEMENTATION DETAILS
B. SELF-SUPERVISED PRETRAINING SETUP
Data We use ShapeNetCore from ShapeNet (Chang et al., 2015) as the pretraining dataset. ShapeNet is collection of clean 3D CAD object models with rich annotations consisting of ~51K unique 3D models from 55 common object categories. We sample 1,024 points per 3D model sample using farthest point sampling (FPS), which is further divided into 64 groups of 32 points as local geometry patches using KNN. Standard data augmentations are adopted during pretraining the 3D autoencoder and 3D point cloud Transformer, i.e., random scaling and translation.
3D Autoencoder Following Yu al. (2022), we use lightweight DGCNN (Wang et al., 2019) as the local geometry patch embedding module, which takes the KNN groups as input and models the local geometry relationship through dynamic graph message passing. The encoded geometry patch embedding is then fed into a pretrained 2D image Transformer, e.g., ViT (Dosovitskiy et al., 2021) or DeiT (Touvron et al., 2021b). Note that without specific descriptions, the results in the paper use ViT-B pretrained on ImageNet (Deng et al., 2009) as the 2D image Transformer. Besides, only the Transformer blocks and layer normalization are used while other layers like original 2D patch embedding are dropped. The decoder is several DGCNN layers to further model 2D-embedded 3D features, followed by the FoldingNet (Yang et al., 2018) for autoencoder reconstruction. As pointed out by Ramesh et al. (2021). the weight of the KL divergence loss (i.e., 8 in Eqn. (8)) during training must be small, we also set the KL divergence loss to 0 in the first 10K steps which is gradually increased 0.1 in the following 100K steps. We use AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate 5e-4. The cosine learning rate scheduler is adopted with 60K warming-up steps. Following Chen et al. (2020a). The Gumbel-softmal temperature decayed from 0.0625 in 100K steps. The batch size is set to 64, and the overall training includes ~ 150K steps.
The training of the 3D autoencoder is supervised by the reconstruction objective and the variational distribution loss. Following Yu et al. (2021), we use coarse- and fine-grained predictions with the
17

---

Published as a conference paper at ICLR 2023
ground-truth point cloud. The sstsle Chamfer Distance is used as the reconstruction objective:
Lc (P.
1 PP|
min lp gl PEP SEG
Σ
Eiin
min l1g
p/|-
(10)
9
SEg PEP
where denotes the predicted point clouds and G denotes the ground-truth point clouds. Follow- ing Ramesh et al. (2021), we use a uniform prior for the discrete variational autoencoder (dVAE) training, where the KL-diivrgence is adopted for distribution alignment. Hence, the overall objective function is:
LIVAR Len-n, (Poones 9) + Lcn (Pone 9) + BLE..
(11)
Masked Point Modeling For masked point modeling, the autoencoder encoder as the backbone model is a standard Transformer architecture (Vaswani et al., 2017) with a lightweight PointNet (Qi al., 2017a) patch embedding module, and the decoder is also Transformer architecture. The encoder Transformer has 12 blocks with an embedding dimension set to 384, while the decoder Transformer has only blocks with the same embedding dimension. The multi-head attention in the Transformer has 6 heads, and the MLP ratio is set to 4. Stochastic depth (Huang et al., 2016) with rate 0.1 is applied to all Transformer blocks. The AdamW optimizer is adopted with a cosine learning rate of 1c-3 and weight decay of 5c-2. The model is pretrained for 300 epochs with batch size of 128.
B.2 TRANSFER LEARNING SETUP
ModelNet40 ModelNet40 (Wu et al., 2015), as one of the most classical datasets, is used for the evaluation of object classification clean 3D CAD models. There are 122 meshed 3D CAD models covering 40 categories. For benchmarking purposes, we use the standard data split of 9,843/2,468 respectively for training and validation, following Qi al. (2017b). The classification head is a three-layer MLP with a dropout of 0.5, and the hidden layer dimension is set to 384, the same the Transformer backbone. AdamW optimizer with 0.05 weight decay is used. Cosine learning rate scheduler is used with a 5e-4 learning rate, warming up 10 epochs. The batch size is 32, and the total training is 300 epochs. Standard random scaling and translation augmentations are used and note that we use voting- based evaluation strategy (Liu et al., 2019b) for fair comparison.
ScanObjeetNN ScanObjectNN dataset (Uy et al., 2019b) is collection of 3D object point clouds from the challenging real-world indoor scene ScanNet dataset (Dai et al., 2017), which includes 15K objects from 15 categories. We use three variants ScanObjectNN following Uy et al. (2019b), iee. OBJ_BG, OBJ_ JNLLY and PB_ PB_T50_RP TS._RS. The optimization and other training settings (e.g. training epochs) are the same with ModelNet40. For data augmentations, we report results trained with no data augmentations and simple point cloud rotation used by Wang al. (2022b). Note that voting strategy is adopted during testing. and if without a specific description, we report overall accuracy (OA) the most challenging PB T50 RS benchmark.
ShapeNetPart ShapeNetPart dataset (Yi et al., 2016) is a popular point-level synthetic object part segmentation benchmark, which covers 17K objects from 16 object categories with 50 fine-grained part categories. We use Adam optimizer with le-5 weight decay. Cosine Learning rate 2e-5 with 10 epochs warming up is used. Standard random scaling and translation are used as data augmentation strategy. The batch size is set to 16, and we train models for 300 epochs.
S3DIS S3DIS dataset (Armeni et al., 2016) provides densely annotated semantic labels for point clouds. It is consisted of six large-scale indoor areas from three different buildings, covering a total of 273 million points from 13 categories. Following Tchapmi et al. (2017), we advocate using Area 5 for evaluation purposes for better and fair generalization performance benchmarking. We use AdamW W optimizer with le-5 weight decay, with a cosine learning rate of 2e-5 warming up to 10 epochs. The batch size is 32, and the total training involves 60 epochs.
ScanNetV2 ScanNerV2 (Dai et al., 2017) is a large-scale dataset that collects ~2.5M RGB-D scans from 1,513 indoor scenes with comprehensive annotations. Following Liu et al. (2022a), we construct a ScanNet- subset containing ~ ~15K frames with a sampling rate of 100 from the raw dataset for 300 epochs ACT pretraining. We use 3DETR (Misra et al., 2021) with the same training recipe for 3D object detection downstream transferring. Note that only the encoder is pretrained and transferred, which has 3 layers with an embedding dimension of 384, and the decoder has 8 layers.
18

---

Published as a conference paper at ICLR 2023
C ADDITIONAL EXPERIMENTS
3D Object Detection We evaluate the representation capability of ACT with downstream 3D object detection on large-scale scene dataset ScanNetV2 with 3DETR (Misra et al., 2021). From Table 11, observed that (i) ACT significantly improves by +1.7% AP25 and +4.2% APso to the from scratch baseline. (ii) In comparison to other SSL methods, ACT outperforms MaskPoint by a clear margin.
Table 11: 3D object detection on the ScanNetV2 dataset. The detection performance using mean Average Precision (mAP) at two different IoU thresholds of 0.50 and 0.25, i.e., AP50 and AP25 are reported. ry2: point cloud coordinates are used.
Method
SSL
Input
AP50 50
AP 25
VoteNet (Qi et al., 2019)
X
ryz
33.5
58.6
PointContrast (Xie et al., 2020)
ryz
38.0
59.2
STRL (Huang et al., 2021)
ryz 38.4
59.5
RandomRooms (Rao et al., 2021)
ryz 36.2
61.3
DepthContrast (Zhang et al., 2021)
TYZ
61.3
3DETR (Misra al., 2021)
Tyz 37.9
62.1
Point-BERT (Yu et al., 2022)
xyz 38.3
61.0
MaskPoint (Liu et al., 2022a)
xyz 40.6
63.4
ACT (Ours)
ryz
42.1
63.8
Comparison to Supervised Cross-Modal 3D Representation Learning Methods Table 12 shows the comparison of our method to the cross-modes moddll 3D representation learning method P2P (Wang extra image data by supervised fine-tuning of the pretrained image
al., 2022b) that also models. From the results, it is observed that our ACT achieves 88.21 % OA on PB_T50 _S with only 22. pure 3D Transformer, while P2P achieves 87.4% with 42. 7..77M/195... large-scale image models (i.e., ResNets101 (He et al., 2016) and HorNet (Rao et al., 2022)).
Table 12: Comparison to supervised cross-modal 3D representation learning method on ScanOb- jectNN. Overall accuracy, i.e., OA (%) is reported.
Method
Backbone
#Params (M)
OA (%)f
P2P (Wang al., 2022b)
ResNet101
42.7
87.4
P2P (Wang et al., 2022b)
HorNet
195.8
89.3
ACT (Ours)
Transformer
22.1
88.2
3D Part Segmentation ShapeNetPart (Yi et al., 2016) is used to evaluate the learning capacity toward knowledge of detailed shape semantics within 3D objects. Table 13 shows the detailed loU results of every category, from which we see: (i) ACT significantly improves the from scratch baseline by 1.2% and 1.0% Cls. mloU and Ins. mloU, respectively, (ii) ACT outperforms the other methods, achieving up to 12 top or second IoU performances over the total 16 categories.
Table 13: Part segmentation results on the ShapeNetPart dataset. The mean IoU across all categories, i.., Cls. mloU, the mean IoU across all instances, i.e., Ins. mloU (%), and IoU (%) for each category are reported. The best results are bolded and the second best results are underlined.
Mothod
Cls. mloli
mlol]
scro
bag
cap
car
chair
acrp- hone
guitar
knife
lamp
laptop
motor- bike
mug
pistol
rocket
skate-
board
table
PaintNer
91.5
PointNec++
$1.85
85.1
82.4
79.0
87.7
77.3
90.8
71.8
91.0
85.9
83.7
95.3
71.6
94.1
81.3
58.7
76.4
82.6
DGCNN
Transformer
83.42
85.1
82.9
85.4
87.7
78.8
90.5
80.8
91.1
87.7
3
95.6
73.9
94.9
83.5
61.2
74.9
80.6
OcCo
83.42
85.1
83.3
88.3
91.9
87.6
84.7
95.4
75.5
63.1
80.8
Point- BERT
84.11
85.6
84.3
84.8
85.0
79.8
91.0
81.7
91.6
87.9
85.2
95.6
75.6
94.7
84.3
63.4
76.3
81.5
Point-MAE
84.19
86.1
84.3
0
88.3
80.5
91.3
78.5
92.1
87.4
86.1
96.1
75.2
94.6
84.7
63.5
77.1
82.4
(Ours)
85.2
19

---

Published as a conference paper at ICLR 2023
D VISUALIZATION
Reconstruction Results Figure 3 compares the reconstruction results from our 2D image Transformer based 3D dVAE and Point-BERT 3D dVAE model. The results show that our 3D autoencoder can reconstruct high-quality details of the objects. For some relatively simple objects like the rectangular table in the second row, both method and Point-BEi point sets with relatively complicated details, such as the thin shelf and armchair in the third row, our method can still reconstruct the object with detailed local geometric information. These qualitative observations are consistent with quantitative results in Table 7.
reconstruct them well. However, for
Point-BERT
Ours
Ground Truth
Point-BERT
Ours
Ground Truth
MATLL
Figure 3: Reconstruction results of synthetic objects from ShapeNet test set.
t-SNE Figure shows the t-SNE (Van der Maaten Hinton, 2008; Poličar al., 2019) feature manifold visualization of models after pretraining on ShapeNet and fine-tuning on the ModelNet40 and ScanObjectNN PB PB_T50_ RS dataset. is observed that: (i) After pretraining on ShapeNet, the model can already yield discriminative features ModelNet due relatively minor domain gap. (ii) After fine-tuning the downstream datasets, discriminative features are obtained on both ModelNet40 and the challenging ScanObjectNN datasets. (iii) The feature distribution extracted by ACT on ScanObjectNN looks less discriminative. We argue that two reasons cause it: (i) the large domain gap between the synthetic ShapeNet and real-world ScanObjectNN datasets, and (ii) no contrastive loss for instance discrimination (e.g., MoCo (He et al., 2020) loss used by Point-BERT (Yu et al., 2022)) is used by ACT. Interestingly, this yields better generalization performance on ScanObjeetNN (88.21% 11% OA of ACT versus 83.07% of
ShapeNet-
(2) ModelNestO (Pretrained)
(b) ModelNeestO (Finetune))
(c) ScanObjeaNN (Pretrained)
(d) ScanOhjeetNN (f (Finetuned)
-20
-20
-40
-50
-40
-20
22
40
-40
-23
42
60
400
-40
-22
40
60
23
-40
-20
20
Figure 4: 1-SNE (Van der Maaten & Hinton, 2008) feature manifold visualization on ModelNet40 and ScanObjectNN PB 1B_T5O__R_ RS datasets. Feature vectors extracted by ACT models after ShapeNet pretraining and downstream fine-tuning are visualized in (a), (c), and (b), (d), respectively.
20


Reviews:
[{"review_id": "zUgWt8AcqiV", "reviewer": "Reviewer_8fVd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an interesting strategy for transferring the knowledge from a model pretrained on 2D images to the 3D point cloud data. In detail, the developed strategy consists of two stages, one fine-tuning the pretrained models with new modules designed for 3D data and one learning a new 3D model via distilling knowledge from the model obtained in the first stage. The method is evaluated on three datasets, including ModelNet40, ScanObjectNN, and S3DIS, covering classification and segmentation. \n\nThe main contribution of this paper is the novel knowledge transfer strategy across different modalities.", "summary_of_the_review": "This paper is well-written, and the proposed method is novel.", "strength_and_weaknesses": "== strength ==\n\n- this paper is well-written and easy to follow. this paper is organized clearly. code is also provided for reproductivity.\n\n- the experimental results on different tasks and datasets are convincing, showing the effectivenesses of the proposed method.\n\n- the motivation is reasonable. the analysis for model pretraining on 3d and 2d is insightful including architecture, data, and pattern. based on the observation, the devised training strategy is also reasonable and efficient.\n\n=== weaknesses ===\n\n- I wonder how the model after the first stage performs.\n\n- feature visualization is expected. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "wjHZ51-3su", "reviewer": "Reviewer_8SEp", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a cross-modal feature distillation framework for 3D representation learning. Specifically, the proposed framework performs masked modeling as feature distillation from pre-trained 2D image Transformer to 3D Transformer students. Experiments show that the proposed framework has satisfactory generalization performance. ", "summary_of_the_review": "This paper may inspire future work in transferring cross-modal knowledge in a self-supervised fashion.", "strength_and_weaknesses": "### Strength\n* The paper is well-written and very enjoyable to read. \n* The proposed approach is novel and technically sound. This is the first work showing that a pre-trained 2D vision transformer can\nhelp 3D representation learning without accessing any 2D data.\n* The experiment setups and analysis are comprehensive. Moreover, the paper offers informative insights with several interesting discussions.\n* The improvements are significant. The proposed framework ACT consistently performs well against the other baselines by a large margin.\n### Weaknesses\nI found no major weaknesses.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "dWlnWW-kQD", "reviewer": "Reviewer_FxNn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to leverage 2D pre-trained transformer models to help 3D understanding. The whole process follows two steps:\n- The first is constructing a teacher model using the 2D / language knowledge via self-supervised prompt tuning. \n- The second is pre-training a student 3D network via masked point modeling with the features from the teacher model as the guidance.\n\n", "summary_of_the_review": "In general, I think this is an interesting paper to combine multi-modal knowledge. Some questions are listed in the weakness part. ", "strength_and_weaknesses": "# Strength:\n- The paper leverages the knowledge from the image/language domains to improve the performance in 3D tasks, mitigating the lack of data in 3D.\n- Instead of directly using 2D networks in 3D tasks, the paper constructs a two-step process to distill knowledge from 2D networks to 3D.\n- Clear performance gain is observed in ScanObjectNN. Different settings (Full, MLP-Linear, MLP-3) are tested. \n\n# Weaknesses: \n- The performance on scene dataset S3DIS is far from state-of-the-art, though showing improvement over the baseline model. Is it possible to replace the backbone network with some stronger 3D networks to see if there is any improvement? \n- It is not clear what causes the performance diversity with different teacher choices. Why do the vision models and language models perform better than language-vision models?\n- In table 7, the DeiT-B dVAE without prompt has already achieved better performance than Point-BERT, especially the 'Freeze' one. In this case, what bridges the gap between 2D and 3D modalities? Is it related to the pre and post embedding network? Also, is it possible to change the g_pre and g_post to some more complicated structures to improve the performance?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "76kUUodUXo", "reviewer": "Reviewer_Dx8R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1) The paper proposes ACT to utilize 2D pre-trained models as teachers for cross-modality knowledge transfer by masked point modeling. This can alleviate the data desert problem in 3D by using rich semantics learned from 2D images. \n2) ACT consists of three steps: 1. Adapt the pre-trained 2D/language models into 3D teachers via DAE and prompting on ShapeNet. 2. Regard this transferred model as distillation teachers to pre-train a Point-MAE. 3. Fine-tune the pre-trained Point-MAE on specific downstream tasks, e.g., classification, segmentation.\n3) ACT achieves favorable performance on various downstream benchmarks. The related discussion solves part of the confusions.", "summary_of_the_review": "The reviewer expects authors to respond to above questions in the weakness, which largely matter the contribution of this paper.", "strength_and_weaknesses": "Strength:\n1) The problem ACT aims to solve is practical and reasonable: insufficient 3D data can not afford large-scale pre-training like 2D and languages. The motivation is also clearly illustrated in the introduction that pre-trained models from other modalities with cross-modal teaching might be helpful.\n2) Writing and figures are presented with high qualities. It is easy to follow and understand the proposed multi-step method.\n3) The idea to construct cross-modal teachers by intermediate antoencoding and prompting is interesting.\n\nWeakness:\n1) The first concern is the too much complicated pipeline. As summarized above, for a downstream task, the pre-training requires two steps of pre-training (actually three steps if 2D's is included), which is time/space-consuming. The reviewer is suspicious about utilization values of ACT. The author could give some pre-training efficiency comparison with existing 3D self-supervised methods.\n2) It is unclear whether the 2D models indeed helps 3D understanding. Although the authors try to explain it in the discussion section, it still remains black-box. \n\n>**(a)** As shown in Figure 2 (b), using BERT-B pre-trained by languages as the teacher seems to surpass all of the vision models except ViT-B. If this results indicate that a good pre-trained transformer as initialization is already working for ACT, no matter it is a 2D model or not? \n\n>**(b)** In Table 8, the paper only utilizes Point-BERT to replace 2D teachers. Considering Point-BERT itself underperforms the newer PointMLP, MaskPoint and Point-MAE, using Point-BERT for ablation is not convincing to verify the effectiveness of ACT's 2D teachers. What happens if using Point-MAE here? Will the simple 3D teacher be better than 2D? \n\n>**(c)** The reviewer fails to understand the analysis about Table 10, e.g. why different positional encodings reveal 2D model understands 3D?\n\n3) The performance of ACT is not strong enough. \n\n>**(a)** Why ACT performs a little worse than Point-MAE on ModelNet40 (93.7 vs 93.8)? Given that ACT completely inherits network architectures of Point-MAE, if this means the simple 3D MIM in Point-MAE is already better than ACT's complicated two-step pre-training on synthetic data? \n\n>**(b)** On ShapeNetPart, ACT can hardly bring enhancement over Point-MAE (86.14 vs 86.1).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster