Adversarially Robust 3D Point Cloud Recognition Using Self-Supervisions
Authors: Jiachen Sun
Yulong Cao
Christopher Choy
Zhiding Yu
Anima Anandkumar
Zhuoqing Mao
Chaowei Xiao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Adversarial Training
Point Cloud Recognition
Self-supervised Learning

Abstract: 3D point cloud data is increasingly used in safety-critical applications such as autonomous driving. Thus, the robustness of 3D deep learning models against adversarial attacks becomes a major consideration. In this paper, we systematically study the impact of various self-supervised learning proxy tasks on different architectures and threat models for 3D point clouds with adversarial training. Specifically, we study MLP-based (PointNet), convolution-based (DGCNN), and transformer-based (PCT) 3D architectures. Through extensive experimentation, we demonstrate that appropriate applications of self-supervision can significantly enhance the robustness in 3D point cloud recognition, achieving considerable improvements compared to the standard adversarial training baseline. Our analysis reveals that local feature learning is desirable for adversarial robustness in point clouds since it limits the adversarial propagation between the point-level input perturbations and the model's final output. This insight also explains the success of DGCNN and the jigsaw proxy task in achieving stronger 3D adversarial robustness.

Adversarially Robust 3D Point Cloud Recognition Using Self-Supervisions
Jiachen Sun " Yulong Cao ' Christopher Choy?, ? Zhiding Yu 2, Anima Anandkumar 2.3
Z. Morley Mao I and Chaowei Xiao
University of Michigan, NVIDIA, Caltech, ASU
Abstract
3D point cloud data is increasingly used in safety-critical applications such autonomous driving. Thus, the robustness of 3D deep learning models against adversarial attacks becomes major consideration. In this paper, we systemati- cally study the impact various self-supervised learning proxy tasks on different architectures and threat models for 3D point clouds with adversarial training. Specifically, we study MLP-based (PointNet), convolution based (DGCNN), and ransformer-basee (PCT) 3D architectures. Through extensive experimentation, we demonstrate that appropriate applications of self-supervision can significantly enhance the robustness in 3D point cloud recognition, achieving considerable improvements compared to the standard adversarial training baseline. Our analysis reveals that local feature learning is desirable for adversarial robustness in point clouds since it limits the adversarial propagation between the point-level input perturbations and the model's final output. This insight also explains the success of DGCNN and the jigsaw proxy task in achieving stronger 3D adversarial robustness.
Introduction
Point cloud data is one of the most broadly used representations in 3D computer vision. It is a versatile data format available from various sensors and computer-aided design (CAD) models. Given such advantages, many deep learning-based 3D perception systems have been proposed [1-6] and achieved great success in safety-critical applications (e.g., autonomous driving) [7-9]. Although deep learning [5, 10] on point clouds has exhibited high performance, they are particularly vulnerable to adversarial attacks [1 [1]-13]. Because the wide applications safety-critical fields, is imperative to study the adversarial robustness of point cloud recognition models.
However, there are at present two obstacles on the path reaching robust point cloud recognition:
Architectural Diversity. Deep 3D point cloud recognition is an emerging field. Many 3D architec- tures with various feature aggregation methods have been proposed. We broadly categorize them into three families based on how networks aggregate the geometric information: multi-layer-peree perceptron (MLP)-based networks [0], convolutional networks [3.4, 6, 14-16], and transformer sosssebberbbeddddd net- works [17, 18]. These networks have been studied mostly using clean accuracy as their main metric, but an in-depth study of their robustness is lacking.
Vulnerability to Adaptive Attacks. A few defenses against 3D attacks have been recently pro- posed [19-21]. Some of the methods, however, merely obfuscate attackers by limiting the malicious agents from accessing the defense systems and true gradients [22]. They have been shown vulnerable to adaptive attackers who have the full knowledge of the defenses and can approximate the gradi- ents [21]. Similar to the 2D domain [23], adversarial training (AT) [21] provides more longstanding robustness in 3D even against adaptive attacks [22]. Nevertheless, further robustness improvements on adversarial training are highly desired for practical usage and deployments.
Coorespondencee to jiachensOumich ch. edu
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

PanthetosChiknn Bactbore
MLP Clanatipaion
Zoom
Kusolaaiio Logat
SD Josaw
Brandhing Layer
Pooking
e:
Autoenccder
Auoillary Hoad
Pindral Ouipal
SShike Ano X Drop
Adverand Pre-aaining Adverssrrill Fine-trinn = Adverand Juir Traning (b) Illustration of Adversarial Pre-training for Fine-tuning
(a) Illustration of Different Threats and Self- Supervised Learning Tasks.
(APF) and Adversarial Joint Training (AJT).
Figure I: Overview of Our Analysis in 3D Point Cloud Classification.
Self-supervised learning (SSL) has been incorporated into adversarial training in 2D image perception models lately. It has shown great potential to enhance adversarial robustness without requiring any additional data or labels [24, 25]. Given such achievements, a natural question emerges: can we mimic the application of SSL to improve adversarial robustness in 3D point cloud recognition? Such a label-free strategy preferred due to the cost and difficulty
point cloud data annotation [26].
Summary of Our Contributions:
In this paper, we present a systematic analysis of the adversarial robustness 3D point cloud recogmition using self-supervisions on three representative architectures: a multi-layer-- layer-perceptron (MIP) network (PointNet) [5], a convolutional network (DGCNN) [15], and a transformer-based network (PCT) [17]. Specifically, we use two strategies to integrate self-supervised learning and adversarial training, including (1) adversarial pre-training for fine-tuning (APF), which uses the SSL tasks only for pre-training, and (2) adversarial joint training (AJT), which jointly trains the SSL task with the recognition task, as shown in Figure 1. To further study the importance self-supervised tasks for adversarial robustness, we select three representative SSL proxy tasks, including 3D rotation prediction [27], 3D jigsaw [28], and autoencoding [29]. Our key observations are as follows:
We show that pre-training on SSL tasks improves adversarial robustness of the ine-ttnnd models. Unlike the 2D domain, where both APF and AJT have enhanced the robustness, our study finds that only APF consistently achieves robustness improvements 3D. AJT does not always help since the distributional gap between data for SSL and recognition tasks will distract each other in AJT. Evaluation results of various unforeseen attacks further confirm such improvements by APF:
We find that the convolutional network, iee, DGCNN, is more robust than the other architectures in point cloud recognition tasks. Moreover, 3D jigsaw SSL task, which predicts the permutation of 3D point cloud patches, helps achieve stronger robustness than the others. Both convolutional architecture and jigsaw SSL task enforce the model to learn better local semantics. Intuitively, robust local features help limit the propagation of adversarial effect from point-level input perturbations to the model's final output.
We demonstrate that fine-tuned models from different pre-training tasks have different vulnerabili- ties, and adversarial examples generated by attacking them do not transfer well among each other. Thus, further leverage two simple yet powerful ensemble methods boost the adversarial robustness by substantial margin. Our best ensemble models, for instance, achieve robust accuracy of 53.5% ++1..6)) 69.4% (+7.4%) and 57.9% (+8.8%) with PointNet, DGCNN, and PCT on the representative dataset, ModelNet40 [30].
2 Analysis Methodology
In this section, we detail our adversarial robustness analysis methodology. We first introduce the principal 3D point cloud recognition architectures and the threat models used in study. We then introduce two ways to generalize and improve AT using 3D point cloud SSL proxies.
2.1 3D Point Cloud Recognition Models and Threats
We introduce the adopted model designs and the formulations of threats to 3D point clouds below.
network PointNet [5], a convolutional network Dynamic Graph CNN (DGCNN) [15], and a transformer-based network Point Cloud Transformer (PCT) [17] as our primary backbone architectures, denoted as Ma... Specifically, PointNet directly aggregates learned features from each point to form global embedding for final
Model Variants. We use shared centtilayeerperceptroo------

---

recognition. By doing so, the point-level features are independent of each other before global pooling. PointNet embraces high efficiency that has been widely utilized as base operation in complex tasks [8,31]. DGCNN instead builds a graph based on k-nearest neighbors (kNN) and uses a variant of continuous convolution (EdgeConv) on edges to enable local feature learning. The combination of local and global representation learning helps DGCNN achieve a higher clean accuracy. PCT extends the Transformer [32], the dominant framework in natural language processing, for point cloud recognition. The core of PCT is the full attention mechanism, which establishes much more flexible scheme, where each point has the potential to affect every other point in the point cloud. The classification head He, parameterized with 9n for these backbones is an MLP and the part segmentation head He. is a set of 1 x convolutions. We use Fe, parameterized with 9 (0, := [0m: 0%. represent the overall model architecture, consisting of the stacked backbone . and recognition head H. where F = H o M. Given the input point cloud æ; the model F aims to predict the corresponding label y. where y = Faz)). More details of the architectures are in the supplements.
Threats. There are mainly three types of threats against point cloud perception models, which could be abstracted as point shifting (PS), point dropping (PD), and point adding (PA) attacks (Figure (()).. We formally define these threats within Ep projected gradient descent (PGD) style attacks. First, we assume PS adversary is able to shift all existing points within a lp norm ball:
x xz+11 (x + sign(V ((x y; F))); Zo = U(-c,e)
(1)
where v, the adversarial example in the s-th iteration, II the projection function to project the adversarial example to the pre-defined perturbation space S, a is the attack step size, and U( U(-e, e) represents a uniform distribution from -6 c. Second, we allow PD adversary to discard the top
salient points [33] in each iteration until total drop of N points:
1s+1 x. arg topkæEæ, saliency(x,, y: F); kxs<N
(2)
where we follow [33] to implement saliency() () to calculate the importance scores of the input points w.r.t. to the prediction accuracy and/ / denotes the dropping operation. We detail saliency () in the supplements due to space limits. Third, PA adversary is capable of adding new points bounded by an l norm ball to the original point cloud. PA randomly initializes N new points from the original point coordinates and only perturbs the added points. We follow the same setting as PS to formulate the perturbation:
Es+1 = 5( + sign Vn, (( vs), y: F))); xo sample(@ N) + U(-e,) (3)
where Tori is the original point cloud and sample() initializes N new points denoted as Zn. We believe the chosen threats mostly cover the existing attack surfaces on point cloud data. It is also beneficial to study how different threats would affect the adversarial robustness in point cloud recognition under our adversarial training strategies which will be introduced next.
2.2 Adversarial Training with Self-Supervisions
We first introduce the chosen 3D self-supervised learning methods, followed by two strategies to incorporate these pretext tasks in adversarial training.
3D Self-Supervised Learning. The primary goal of self-supervised learning (SSL) is to learn effective feature representations with unlabeled data. Given a pretext task Pt. the pre-training process is still conducted in a supervised manner with self- self-generated data x' and label y from pristine data æ, where (x', y²) (x)). Therefore, a target loss function C, y²; FF,)) will be minimized during the optimization, where Or consists of the shared backbone parameters @,, and customized branch parameters 0. (i.e., 0, == i fz)). We utilize the following 3D SSL tasks in our study (Figure (a))..
3D Rotation [27]: Similar to the rotation task in 2D vision [34], the data and label are generated by rotating the original point clouds to pre-defined angles 1) in the 3D space. Therefore, the problem to correctly predict 3D rotation angles w.E. the input point cloud. The objective function can be formulated as cross-entropy (CE) loss: Lrotation CE(x rotation .y rooaation Frotation).
3D Jigsaw [28]: Different from the jigsaw task in 2D vision [35] which is defined as classification problem, 3D jigsaw solicits a segmentation model. A point cloud is evenly divided to 133 small cubes and shuffled to different positions. Points inside each small cube are assigned to label signaling its original position. The problem, thus, is to correctly predict the original cube position of each point. Similarly, its objective is to minimize a CE loss: Ljigeaw CE(x (ig y yigaaw, F.iggaaw)
3

---

Autoencoder [29,36]: An autoencoder utilizes an encoder 2 = E(x) to learn a compact representa- tion and a decoder (()())) to reconstruct the point cloud. We utilize different backbones as the encoder F.()) and NoldingNet [29] as the decoder (.)) due to its satisfactory performance. We use three different positional encodings: plane, 3D sphere, and 3D gaussian in our experiments, we use the Chamfer distance [37] as the reconstruction loss: Le Chamfer( D (F(E Exx")) x"e; ; Fue).
The detailed description of the pretext tasks can be found in the supplements.
Adversarial Pre-training for Fine-tuning (APF). As introduced in 11, adversarial training (AT) [23, 38, 39] has been demonstrated to be one of the most longstanding and practical defenses. We thus enable AT in both pre-training and fine-tuning stages:
arg min 0
E(m.y) max C(x + 0, 0.
(4)
where L Lyr for loss functions in pre-training (t) and fine-tuning (f) stages, is the adversarial perturbations, and represents its manipulation space (i.e. foo in our study). AT essentially solves min-max problem. In the inner loop, the optimizer tries find adversarial examples that maximize the target loss, and the outer loop updates the network parameters to correctly recognize the generated adversarial examples. In contrast, standard training (ST) is simply to optimize arg ming ) [L y, 0)].
In the pre-training stage of APF, we leverage both standard and adversarial training to get the pre- trained backbones Me... and MM... Given a pre-trained backbone parameterized by 6n. in the second stage, we adversarially fine-tune all 0 == [0m 0n. for the recognition task, as illustrated in Figure 1(b). The network branches at the penultimate vector for the rotation task and the first global feature [21] for the jigsaw and autoencoder tasks since they use the segmentation head.
Adversarial Joint Training (AJT). Besides pre-training for fine-tuning, joint training another way to apply SSL. The objective function is formulated as:
arg min O,.
E (a,y)~D
max (f( + +., + Lo 0,) EE8
(5)
where À is a hyperparameter to balance the SSL and recognition tasks. Two tasks share the same backbone 6... with two different branches, parameterized by 0, and o.. respectively. We also enable dual batch normalization [40] in AJT for æ and x' since they should belong to different underlying distributions. We use two model-agnostic tasks, i.e., 3D rotation and jigsaw in AJT.
Similarly, in our AJT analysis, all L, and Ly can be formulated as CE loss. We empirically set and leverage the same branching point with APF. The whole network is trained to predict the supervised task with the original head and the SSL task with the auxiliary head (Figure (bb)..
3 Experiments and Results
In this section, we present our experimental setups and results. We first introduce the adopted datasets and adversarial settings. We leverage PS the primary threat in our study since we find it is the most powerful adversary and the other two adversaries easier to detect. Next, we detail our extensive evaluation of two fundamental point cloud recognition tasks: classification and part segmentation.
3.1 Evaluation Setups
Datasets. We leverage four datasets (D): ModelNet40 [30] (40 classes), ModelNet10 [30] (10
classes), ScanObjectNN [41] (15 classes), and ShapeNetPart [42] throughout our experiments. Specifically, the first three datasets are utilized for the classification task and the last one is for the part segmentation task. ModelNet and ShapeNetPart are captured from CAD models, and ScanObjectNN is scanned and extracted from real-worke indoor scenes. For each point cloud, we randomly sample 1024 points and normalize them to an edge-length-2 2 cube ([-1, 1) for experimentation. We follow the default split of training and test sets in [5] and [43]. For SSL, we randomly sample y from the pre-defined label sets and further generate x based on y in each iteration. Specifically, we choose 7 6, 18 and k = 3,4 4 for rotation and jigsaw tasks, followed by the suggestion of [27] and [28].
Adversary. As introduced in 1, for PS, we exploit 7-step and 200-step foo PGD attacks [23] targeting the cross-entropy loss for adversarial training and testing, respectively. We follow Sun et al. [21] to empirically set the perturbation boundary 0.05 (||a]|oc S 0.05) since perturbed

---

point clouds with e = 0.05 are at the edge of correct human predictions of objects. Numerically, e 0.05 out of the range [-1,1] is also similar to the commonly used 25; in 2D adversarial training [23]. Note that, different from discrete RGB values in 2D images, point cloud's features (coordinates) are continuous. We thus utilize PGD step size a = 0.01 and a = 0.005 the training and testing phases, respectively. We allow a stealthy perturbation of N = 100 points for PD and PA since we believe a larger number of modification would be easy to detect [13,33], We drop the most A: = 14 and 5 salient points every iteration in training and testing phases for PD, respectively. The behind rationale is that a severer attacker is preferred in the testing phase to evaluate the true robust accuracy. The attack setups in PA are the same as those in PS since they utilize similar 2o PGD perturbation methods. We exploit PS as the primary threat for the classification and part segmentation tasks, and leverage the other two threats further demonstrate our robustness improvements in the classification task.
Original Point Cloud Chair
Point Shifting Bookshe
Sailent Point Gropping
Point Addlnng Lamp
Tollet
Figure 2: Adversarial Examples of Different Threats.
3.2 Point Cloud Classification
In this section, we first introduce the setups of our point cloud classification analysis. Next, we introduce the detailed study of APF and AJT. We further evaluate our robust models with unforeseen attacks to demonstrate that our robustness improvements are non-trivial. We find that pre-trained models from different SSL tasks preserve different vulnerabilities; hence we use simple yet powerful ensemble methods to boost the robustness. Lastly, we evaluate our methods in different threats to demonstrate their generality.
Training Details. We generally follow data augmentation methods (e.g., jitter and translation) in DGCNN [15] in our study. All pre-trained and fine-tume tnned models in APF are trained using Adam [44]. We use batch sizes of 32 for PointNet and DGCNN, and 128 for PCT. The initial learning rate is set to 0.001 for PointNet and DGCNN, and 5 x 10 for PCT. Both pre-training and fine-tuning take 250 epochs, where 10x decay happens at the 100-th, 150-th, and 200-th epoch. We leverage the same training setups in AJT. All experiments are done on 1 to 4 NVIDIA V100 GPUs [45].
Model Adaptation to AT. First, PointNet [5] leverages exponential learning rate decay, and DGCNN and PCT utilize cosine annealing learning rate decay. Through preliminary experiments, we find that piecewise decay of the learning rate im- proves the default baseline by a noticeable margin. Second, two T-Nets in PointNet [5] will also make AT unstable since they holistically modify the point cloud and features. We thus remove them in our experiments. Third, PCT by default leverage farthest point sampling (FPS) to sample and group local points, which will cause AT unstable since the perturbed point cloud samples different anchor points in each iteration. We replace FPS with EdgeConv each point. It is worth noting that such a change will not modify the application of self-attention in PCT. As presented in Table 12, the modifications will both improve the standard and adversarial training baseline, and we introduce the details in the supplements.
Table I: Summarized Results (%) on Mod- elNet40 by Model Adaptation.
PointNer
DGCNN
PCT
CA
RA
CA
RA
CA
RA
Defiult ST
$8.0
0.0
91.3
2.9
91.9
0.0
Selected ST
88.7
0.3
91.5
5.2
92.1
2,3
Defanlt AT Selected
87.1
33.6
90.6
58.
88.4
44.2
3.2.1 Self-Supervised Pre-training Helps Adversarial Fine-tuning
We systematically evaluate all configurations in APF under PS attack. As introduced earlier, we use standard and adversarial training to get the pre- traited models. From Table 2, we can make several interesting observations. First, we find that our APF strategy generally enhances the adversarial robustness. The best-fine-tuned models achieve 14.2%, 5.4%, and 2.2% robustness improvements in PointNet, DGCNN, and PCT on ModelNet40, respectively. The enhancements on the real-world dataset, ScanObjectNN, i.e., 1.8%, 10.4%, and 6.9% in PointNet, DGCNN, and PCT, are also significant, which demonstrate the generality of APF. Second, we find that DGCNN outstands to be the most robust architecture, consistently achieving 155% stronger robustness than the other two models on both ModelNet40 and ScanObjectNN, Lastly, jigsaw- APF offers more robustness improvements than the other two methods while maintaining slightly higher clean accuracy (CA).
'The 1-st and 2-nd highest accuracy among tuned models in each column are noted, and we use the same mark throughout this paper. CA and RA denote clean and robust accuracy, respectively.
5

---

Table 2: Evaluation Results (%) of Adversarial Pre-training for Fine-tuning and Task Ensembles.
ModeINesO
ScanObjeeNNN
ModelNetto
PointNet
DGCNN
PCT
PoinNeet
DGCNN
PCT
PointNet
DOCNN
PCT
Protest Task
Parameters
|
CA
RA
CA
RA
RA
RA
CA
RA
CA
RA
RA
CA
CA
AT Baseline
NAA
|
87.2
[37,99]
90.6
62.00
89.7
49.1
09.9
2377
74.4
30.91
72.4
(0.5
96.6
79.7
98.1
96.3
97.4
800
3D Kotation
1 6
18
57.2 87.2
48.3
08.0
91.4
91.1
63.6
64.1
90.2
90.2
50.7
49.5
69.5
69.1
24.5
25.0
73.8
75.7
32.9
32.2
72.5
72.6
20.1
20.5
97.1
96.8
79.3
79.0
98.5
97.7
$1.9
85.3
97.8
97.2
50.4
80.3
Advenaniaa
? 6
87.6
42.1
90.8
61.8
90.4
50.8
09.6
25.3
75.0
36.8
71.6
28.7
97.0
79.9
97.7
87.5
98.0
K2.2
3D Rotation
18
57.4
45.7
90.9
02.9
90.4
30.1
09.3
24.5
75.0
36.3
73.1
26.9
97.0
79.7
98.0
58.1
97.4
85.7
3L ligsaw
87.6
50.1
90.0
67.4
51.1
50.2
70.2
79:0
76.2
33.8
72.4
73.8
23.2
24.6
98.0
98.0
Advemariia
58.2
52.1
89.6
65.6
89.5
51.3
69.0
24.8
77.5
41:3
72.5
26.3
97.0
B00.6
95.5
90.5
97.4
83.5
3D ligsow
89.9
$99.6
51.0
40.6
73.1
27.4
97.01
98.0
89.1
97.55
83.9
sphere plane
57.4
50.0
59.9
62.8
90.2
50.7
09.9
25.1
76.1
36.0
71.3
24.1
97.0
S0L.5
98.2
96.3
97.1
90.1
Ausoenceder
$7.1
48.8
90.1
62.2
90.2
50.2
69.1 4
25.5
76.2
35.6
71.1
22.6
96.5
901.5
97.8
$7.6
97.0
90.1
glussian
$7.4
48.9
90.8
63.3
89.7
50.3
69.7
23.8
75.6
35.8
71.3
24.8
96.8
$0.5
97.8
56.4
97.1
80.1
Advensriia
sphere
87.1
49.7
90.0
62:2
90.3
30.0
70.4
25.0
75.2
36.2
72.6
22.2
96.7
80.4
97.5
87.3
97.5
Astoenceder
gaussion
plane
56.9
57.1
46.5
48.5
90.7
89.7
62.7
61.8
90.2
89.7
30.0
30.5
68.8
09.2
24.0
25.0
74.7
75.6
36.
38.0
72.6
73.3
21.5
23.4
97.0
97.0
9016
50..2
97.5
98.0
55.4
56.1
97.4
97.7
$2.5
83.2
Max husecable
$8.5
53.5
91.4
69.4
90.5
55.4
70.8
27.3
79.1
42.6
74.00
28.9
97.2
82.5
98.5
91.0
98.1
85.2
Mean Ensemblle
91.4
57.9
70.8
20.9
79.0
74.2
98.6
98.0
84.9
Specifically, jigsaw-based APF, on average, further boosts DGCNN's robust accuracy (RA) by 2.8%. 2.2%, and 2.7% on three datasets, respectively.
Insights. Different from 2D images that possess both texture and shape information, 3D point clouds naturally bias towards shape. In 2D image space, it is widely recognized that local and global features correspond to the texture and shape information, respectively [46]. Recent studies have demonstrated that appreciation of global/shape features can help improve model robustness on image classification [47]. However, we find some distinctions in point cloud recognition. As mentioned above, PointNet with only global feature learning will be easily affected by the perturbed points (Table 2). Due to the sparsity of point clouds, the local feature actually represents the smoothness of the object's surface. Thus, learning robust local features is critical for correctly recognizing a perturbed point cloud, as it limits the adversarial effect propagation to the model output. We also include a preliminary study on contrastive pre-training [48] and find that it can be viewed as a global feature learning scheme as well. Detailed results can be found in the supplements.
As summarized above, DGCNN achieves the strongest robustness under AT, attributed to the hier- archical usage of EdgeConv [15]. EdgeConv dynamically aggregates local features by exploiting ANN. Such an aggregation method has the ability to calibrate the adversarial effect in the local feature learning stage. Although transformer-based architectures have gained tremendous visibility recently [49], we find that PCT does not have major robustness improvement compared to PointNet. Self-attention increases the capacity of the model architecture, but it also enlarges the receptive field of the model [50]. In PCT, each point can influence every other point's feature, which will potentially increase the model's s fragility [51].
Moreover, we also find that grssawbbase. APF is the most effective method to improve adversarial robustness, aligning well with our above insights. Jigsaw SSL makes the model learn to reassemble the randomly displaced local point clusters, where the model is enforced to learn the displaced local features. Meanwhile, to correctly reconstruct the point cloud, jigsaw SSL also requires the model to capture the global and holistic semantics. Nevertheless, rotation and autoencoder- methods focus more on global feature learning. Therefore, we believe jigsaw-based APF a perfect candidate to strengthen the association between local and global features in point cloud learning, hence improving the adversarial robustness under APF.
pre-training
3.2.2 Adversarial Joint Training Does not Always Improve Robustness
We further analyze the implication of SSL tasks in AJT. As presented in Table 3, AJT can still enhance the robustness in PointNet and DGCNN, For instance, AJT improves their RA by 1.8% and 8.0% on ScanObjectNN, respectively. However, AJT overall cannot outperform APF in point cloud recognition. Especially, we find AJT even degrades the RA of PCT compared to the standard AT.
Insights We find this also to be related to the natural characteristic of point cloud data. Although SSL. can help models learn strong priors and context information, it is still a separate learning task. Rotated and disassembled images still preserve similar local features to the original images since the

---

Table 3: Evaluation Results (%) of Adversarial Joint Training.
ModeINas40
ScanOhjearNN
ModelNertO
PointNer
DGUNN
Poimter
DGUNN
PCT
PointNet
DGCNN
PCT
Protest Tisk
Parameters
CA
RA
CA
RA
CA
RA
CA
RA
CA
RA
CA
RA
CA
RA
CA
RA
CA
RA
Basetine
N/A
87.7
37.9
90.6
66.0
89.7
49.1
69.9
23.7
74.4
30.9
72.4
20.5
96.6
79.7
98.1
97.4
80.0
Rotation
1 18
86.5
8688
62.0
60.7
44.3
67.8
76.2
74.2
37.8
87.2
20.3
97.0
96.6
97.9
98.1
75.6
73.8
3D Tigsow
mm 3
8776
87.2
42.5
46.7
91.0
91.1
61.7
62.3
9012
89.8
43.1
40.9
69.4
70.0
24.6
25.5
75,9
77.1
38.4
38.9
72.1
73.7
20.5
20.7
96.8
96.8
79.8
77.9
98.0
98.4
88.6
$7.9
97.7
97.1
76.8
78.0
sphere plane ganssion
87.5
44.4
90.9
60.1
89.6
49.2
08.9
24:2
75.5
36.5
72.5
20.5
96.7
19.88
98.2
50.3
97.5
80.3
Autoencoder
67.4
42.1
90.7
61.9
89.3
48.7
68.5
23.0
75.6
94-7
72.8
20.6
96.7
70.7
98.1
86-2
97.4
70.9
86.9
43.9
90.9
61.9
88.9
49.2
68.7
24.4
76.1
35.1
72.1
20.5
96.6
79.7
98.3
96.9
97.5
80.0
RGB values do not change, so that the auxiliary optimization in AJT will not distract AT but help models learn robust global features [24]. However, point cloud models take point coordinates zyz as input. Rotated and disassembled point clouds have significant variations in their coordinates' numeric values. Although we apply dual batch normalization [52] migrate the feature heterogeneous problems, such discrimination will consequently distract model learning in AJT, and thus hurt the RA performance. The usage of self-attention in PCT will further expand this impact since it introduces global receptive field [51]. We find that the results of AJT using autoencoders are more stable than the other two tasks. We believe is because the input for the autoencoder is the same as the recognition task so that the distributional gap is small. However, it is still worse than the pre- pre-training scheme.
3.2.3 Robustness against Unforeseen Attacks and Noises
We have so far shown that APF generally improves the robustness of point cloud classification. In this section, we leverage unforeseen attacks to further demonstrate that the enhancements are non-trivial. We select the fine-tuned models with the highest RA in Table 2, and all results here averaged from five runs using different random seeds.
We leverage unforeseen attacks: Auto Attack (AA) [53], Momentum Iterative Method (MIM) [54], and PGD with both cross-entropy and margin loss (i.e., Lmargin Z(x), max Z(x);; where (xx) is the logit output of the target classifier). We elaborate the details of AA (i.e., A-PGD) and MIM in the supplements. All attacks use 200 steps to find the potential adversarial examples. As shown in Figure 3(a) and 3(b), the best fine-tuned models consistently achieve higher RA than models from standard AT with 2.6% 15.3% and 1.0% 10.1% improvements on ModelNet40 and ScanObjectNN, respectively. We find that DGCNN still outperforms the other two models achieving the highest RA even against the strongest Auto Attacks [53].
Moreover, we exploit the transfer attack to confirm the effectiveness of APF. We fine-tune five models using the same (best) setting but different random seeds for transfer-based attacks and test the transferability of generated adversarial examples among different models. As Figure 3(d) and 3()) show, adversarial examples cannot transfer well and there is ~ 100% gap in RA compared to attacks on target models, which further validates the real robustness improvements.
We also find that Gaussian and uniform noises are ineffective in breaking model robustness, and the RA is therefore close to the CA. As shown in Figure 3(d) and 3(e), the fine-tuned models still outperform standard AT baseline by 0. 1% 2.2% and 1.0% - 5.7% on ModelNet40 and ScanObjectNN, respectively, in RA. More evaluation results of our adversarial training strategy are elaborated in the supplements.
3.2.4 Attack Transferability and Task Ensemble
Since we leverage three pre- text tasks for self-supervised pre- training, we also test the trans- ferability of adversarial exam- ples generated by models pre- trained on different SSL tasks. As shown in Figure 4, different fine-tuned models preserve dif- ferent vulnerabilities, and there at least 100 gain of RA when transferring attacks mod-
0.483
0.687
0.967
0.641
0.735
0.778
0.507
0.734
0.724
3.9
0.735
0.323
0.729
0.528
0.67U
0.6993
0.504
0.099
0.659
0.678
0,500
0,750
0.707
0.635
0,780
0.781
0.506
Terges
Target
(a) PointNet.
(b) DGCNN.
(c) PCT.
Figure 4: Robust Accuracy on Transfer Attacks among Fine-tuned Models from Different SSL Tasks on Ne1400
7

---

(a) ModelNet40.
(b) ScanObjectNN,
(c) ModelNet10.
(d) ModelNet40.
(e) ScanObjectNN
ModelNet10.
Figure 3: Evaluation Results of Unforescen Attacks. Plots in the first row present the evaluation results of unforeseen adversarial attacks, where "-M" denotes the attacks on margin loss, Plots in the second row present the evaluation results of noise and transfer attacks.
els fine-tuned from different tasks on ModelNet4O. We thus combine the models and fur- ther test the ensemble model's robustness. We use two ensemble methods, which are tak- ing the max and mean value of the stacked logits from individual models i.e., Lensemble = max/ htt///ean(LLLaa fotationn L jigsow L Lauutoeeoderr)) where L denotes the i-th value in the logit L. By doing so, form "wider" model where the gradients can still propagate smoothly back to the input. Therefore, we also exploit the 200-step PGD attack to test their robustness. As presented in the last two rows of Table 2, both ensemble models can further boost the RA by a significant margin while maintaining similar clean accuracy. For example, our best ensemble models achieve robust accuracy of 53.5% (+15.6%), 69.4% (+7.4%) and 57.9% (+8.8%) with PointNet, DGCNN, and PCT on ModelNet40 respectively. It is worth noting that our fine-tuning strategy does not enforce a diversity regulation, unlike [25], which further demonstrates that pre-training on different pretext tasks indeed brings distinct context information into pre- rrettrined models.
3.2.5 Robustness against Other Point Cloud Threats
As mentioned before, we also consider two other threats that specifically target point cloud recognition. As we have demonstrated that jigsaw-based APF reaches the best robustness enhancement under point shifting (PS), replace PS attack in AT with salient point dropping (PD) and point adding (PA) to test the effectiveness of jigsaw-base based APF strategy.
Table 4: Evaluation (%) of PD on ScanObjeetNN.
Table 5: Evaluation (%) of PA on ScanObjectNN.
PointNer
DGCNN
PCT
PointNer
DGCNN
PCT
Pretext
Parameters
CA
Parameiers
CA
RA
ST Baseline
N/A
68.8
48.5
79.3
60.9
74.2
$6.1
ST Buseline
N/A
69.8
57.6
79.3
38.0
74.2
39.6
AT Baseline
N/A
70.9
64.2
76.2
77.1
73.3
73.7
AT Raseline
NIA
69.0
59.4
77.4
63:5
72.1
56.8
3D Tigsaw
71.3
81.2
81.4
76.8
76.9
Tigaan
70.4
70.2
59.9
66.3
66.6
75.7
Advenarial
71.1
64.7
78.7
79.5
75.4
76.2
Adversatial
70.4
02.5
77.5
67.8
75.2
64.4
3D Jiguaw
3D ligsaw
We find that based APF can still strengthen the robustness under these two threats on three datasets. Especially, as shown in Table 4 and 5, DGCNN still achieves the highest RA, which on average outperforms PointNet and PCT by 14.8% and 2.3% in PD-AT, and 6.6% and 4.9% in PA-AT, respectively, which is consistent with our previous findings. Moreover, as PD and PA tend to be much weaker adversaries than the PS attack, the enhancements mainly appear in the challenging real-workh dataset, ScanObjectNN, where jigsaw-based based APF boosts the RA of DGCNN by 3.6% and 4.7% under PD and PA. The key insights still hold in ModelNet datasets under these two threats, and we present the detailed results in the supplements due to space constraints.
8

---

3.3 Point Cloud Part Segmentation
In addition to object classification, we conduct the first study on analyzing the robustness of the point cloud part segmentation on ShapeNetPart [42] with 17 classes of 3D objects. Apart from semantic segmentation that all samples from a dataset share the same set of labels, part segmentation task assigns unique label set for each class of objects. For instance, the class of chair has the label set: [seat, back, arm, leg) and the airplane has [wing, body, tail, engine), ). as illustrated in Figure 5. We use the same backbone with a segmentation head for this task and mean nnterseciion-veer-uiion (mloU) [43] as the evaluation metric. Especially since each class possesses its own label set, the models also take the category (one-hot vector) as input. As we find APF is an effective and efficient strategy, we use the PS attack with the same AT and APF setups as the classification task to train the models ($ 3.1).
Table 6: Evaluation Results (%) of the Part Segmentation Task.
back
PoiniNei
DOCNN
PCT
Protect
Parsmeters
C-mlot
R-mtoU
C-mioU
R-mioU
C-mioU
R-mloU
amm
ST Baseline
NA
83.2
32,5
84.2
42.0
84.2
54.6
AT Reseline
N/A
79.2
62.7
82.9
60.5
82.6
67.8
sext
79.1
63.0
82.8
69.3
82.8
67.3
3D Rotizion
18
79.3
63.0
82.9
00.5
82.9
67.5
Adversarial
1
79.3
62.9
83.9
69.2
82.2
67.
6-0.06
log
3D Rotition
17 18
79.1
62.6
82.9
69.0
81.0
65.7
3D ligsaw
A
79.4
79.4
63.1
63.0
82-7
87.9
20.1
70.1
87.4
82.8
68.4
08.1
cannot modify the strong positional pattem of part segmertation.
Figure 5: Illustration of the Point Cloud Part Segmentation Using "Chair".
Adversurial
A
79:7
62.5
$2.8
09.7
$1.1
65.8
3D Jigsaw
79.9
62.7
62.7
70.3
66.7
Table 6² summarizes the quantitative results. To our surprise, we find that part segmentation is more robust task than classification. The robust mIoU is relatively high even in the ST baseline. Moreover, DGCNN still achieves the highest clean and robust mIoU in part segmentation, further confirming its supremacy in point cloud learning. Furthermore, jiggaaa jigsaw- based APF can also promote the robustness of part segmentation by 0.4%, 0.8%, and 0.6% in PointNet, DGCNN, and PCT, respectively.
Insights. We attribute the natural robustness of part segmentation to the intrinsic feature of point clouds. Since all point clouds in ShapeNetPart are sampled from CAD models, they posed to face the same direction. Thus, models may memorize the 3D space corresponding to the specific label during learning. Taking the class of chair an example, the points whose E [-1,0] most likely belong to the "leg" label. However, bbsed adversary with = 0.05 might not be able to perturb this strong pattern (Figure 5). We also envision that the robustness of part segmentation may help and transfer to other complex learning tasks if appropriately used. Moreover, DGCNN with jigsaw-based APF still delivers the best performance, which further generalizes our previous findings on the importance of balancing local and global features in point cloud learning.
4 Related Work
In this section, review a few topics related to the adversarial robustness of 3D perception: 3D deep learning, self-supervised learning, and adversarial attacks and defenses.
4.1 Deep Learning on 3D Point Clouds
In image-based perception, there has been stellar progress on architectures and learning algorithms for convolutional neural networks. However, in 3D perception, there has been no consensus on the type of models for 3D data partially because there is no standard data format for 3D perception. Yet, point clouds are the most commonly used data format because raw data from both 3D scanners and triangular meshes can easily be converted into a point cloud. Some of the earlier 3D networks use dense voxel grids for perception [1, 21,2,55,66 56), which discretize a point cloud to voxel grids for classification, segmentation, and object detection. Later, to overcome cubic memory complexity, various types of data structures have been proposed: octrees [3, 4], surfaces [57], graphs 588], point sets [5, 14, 17], and sparse tensors [6, 16). In this work, we focus on 3D point cloud perception with PointNet [5], DGCNN [15], and PCT [17] our evaluation backbones since they are widely used and achieve state-of-ther the-art results for point cloud recognition [59].
CC-mloU and R-mloU denote clean and robust mean interseettion-vve--uiiee respectively.

---

4.2 Self-Supervised Learning Approaches
Self-supervised unsupervised learning (SSL) techniques have shown effective in learning powerful representation without manual efforts, where the labels are generated from the data itself [60-62]. The embeddings pre-trained through self-supervision could be utilized for fine-tuning multiple downstream supervised tasks with better generalization and calibration [63, 64]. Because of the impressive success of SSL in 2D computer vision [34], recent studies have also proposed several self-supervisions for 3D point clouds including autoencoders [29], generative adversarial networks (GANs) [36]. 3D rotation [27], and 3D jigsaw [28]. Those primitives hold promises to improve the clean accuracy of point cloud learning. This work systematically explores whether and how SSL could be leveraged to improve the adversarial robustness in point cloud recognition.
4.3
Adversarial Attacks and Defenses
Despite the accomplishments that DNNs have achieved, adversarial attacks [65] are becoming the major obstacle in real-world deep learning deployments, especially in safety-critical areas [11, 12, 66- 69]. Numerous attacks have been widely studied for various tasks in the 2D [70-76] and 3D [13, 77- 79] domains. To address this problem, many defense methods have been proposed to enhance the robustness against ad adversarial attacks in the 2D [80-88] and 3D domains [19-21]. However, most of them including adding randomization [20,89,90 90], model distillation [83], adversarial detection [84], and input transformation [19,80 80-82, 91] have been compromised by adaptive attacks 121, 92]. Certified methods are recently applied to 3D, but they focus on threat models with modified number of points [93] and isometric transformations [94]. which are inapplicable to &o norm. Adversarial training (AT) [23, 38, 39, 95,,, on the other hand, provides one of the most longstanding and practical defenses. Various methods have been proposed to improve AT in the 2D domain [52, 96, 97]. Jeddi et al. establish fast adversarial fine-tuning scheme [98]. Hendrycks et al. [24] have shown to jointly train the adversarial loss and SSL loss to improve the robustness. Chen et al. [25] have applied adversarial pre- pre-training and fine-tuning strategy to improve the robustness. In this work, we explore to apply and improve AT in 3D point cloud recognition.
5 Conclusion
In this work, we systematically explore the impact of self- sepervueee learning (SSL) on the adversarial robustness 3D point cloud recognition. We find tangible robustness improvements by the adversarial pre- pre-training for fine-tuning strategy. We also experimentally show that robust local features are critical to achieving robustness in 3D, explaining the success of DGCNN and the jigsaw proxy task. Our results shed light for future research on designing more robust models and SSL schemes for 3D point clouds. By providing empirical evaluations on the robustness, our study also motivates future studies in developing theoretical guarantees on the robustness in the 3D domain.
Acknowledgements
We appreciate our area chairs and anonymous reviewers for their insightful comments. We thank Qingzhao Zhang for proofreading our manuscript. Jiachen Sun thanks Zhao Su for her considerate care and help during COVID-19. This project is partially supported by NSF grants CMMI-2038215 and CNS-193004|
References
[1] Dominic Zeng Wang and Ingmar Posner. Voting for voting in online point cloud object detection. In Robotics:
Science and Systems, volume 1, pages 10-15607. Rome, Italy, 2015.
[2] Daniel Maturana and Sebastian Scherer. Voxnet: 3d convolutional neural network for real-time object recognition. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 922-928. IEEE, 2015.
[3] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations high resolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[4] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun- Yu Sun, and Xin Tong. O-enn: Octrec- based con- volutional neural networks for 3d shape analysis. ACM Transactions Graphics (TOG), 36(4):1-11, 2017.
[5] Charles R Qi, Hao Su. Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
10

---

[6] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 3075-3084, 2019.
[7] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Center-based 3d object detection and tracking. CVPR, 2021.
[8] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-779, 2019.
[9] Shaoshuai Shi, Chaoxu Guo, Li Jiang. Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-remm: Point-voxel feature set abstraction for 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529-1055 2020.
[10] Charles R Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnct++: Deep hierarchical feature learning on point sets metric space. arXiv preprint 1706:02413 2017.
[11] Yulong Cao, Chaowei Xiao, Benjamin Cyr, Yimeng Zhou, Won Park, Sara Rampazzi, Qi Alfred Chen, Kevin Fu, and Z Morley Mao. Adversarial sensor attack on lidar based perception in autonomous driving. In Proceedings of the 2019 ACM SIGSAC conference computer and communications security, pages 1267-22811 2019.
[12] Jiachen Sun, Yulong Cao, Qi Alfred Chen, and Z. Morley Mao. Towards robust lidar-based perception in autonomous driving: General black-box adversarial sensor attack and countermeasures. In 29th USENIX Security Symposium (USENIX Security 20), pages 877-894, USENIX Association, August 2020.
[13] Chong Xiang, Charles R Qi, and Bo Li. Generating 3d adversarial point clouds. In Proceedings of the IEEFJCVF Conference on Computer Vision and Pottern Recognition, pages 9136-9144, 2019.
[14] P. Hermosilla, T. Ritschel, P-P Vazquez, A. Vinacua, and T. Ropinski. Monte carlo convolution for learning on non-uniformly sampled point clouds. ACM Transactions on Gruphics (Proceedings of SIGGRAPH Asia 2018), 37(6), 2018.
[15] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[16] Benjamin Graham and Laurens van der Maaten. Submanifold sparse convolutional networks. arXiv preprint XrXi::706.000 703,01307, 2017.
[17] Meng-Hao Guo, Jun-Xiong Cai,, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. arXiv preprint arXiv:2012. 2.6688, 2020.
[18] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. arXiv preprint 20l.09900909166 2020.
[19] Hang Zhou, Kejiang Chen, Weiming Zhang, Han Fang. Wenbo Zhou, and Nenghai Yu. Dup-net: De- noiser and upsampler network for 3d adversarial point clouds defense. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1961-1970, 2019.
[20] Xiaoyi Dong, Dongdong Chen, Hang Zhou, Gang Hua, Weiming Zhang, and Nenghai Yu. Self-robust 3d point recognition via gather-vector guidance. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11513-11521. IEEE, 2020.
[21] Jiachen Sun, Karl Koenig, Yulong Cao, Qi Alfred Chen, and Z. Morley Mao. On the adversarial robustness of point cloud classification, 2020.
[22] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pages
283. PMLR, 2018.
[23] Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 06083, 2017.
[24] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning improve model robustness and uncertainty. arXiv preprint 1.XXvv1906.11344 2019.
[25] Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 699-708, 2020.
[26] Charles R Qi, Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, and Dragomir Anguelov. Offboard 3d object detection from point cloud sequences. arXiv preprint arXiv:2103.0555 2021. [27] Omid Poursaeed, Tianxing Jiang, Quintessa Qiao, Nayun Xu, and Vladimir G Kim. Self-supervised learning of point clouds via orientation estimation. arXiv preprint arXiv:2008.0 00305, 2020.
[28] Jonathan Sauder and Bjarne Sievers. Self supervieed deep learning on point clouds by reconstructing space. arXiv preprint arXiv: 00090008836.00000
11

---

[29] Yaoqing Yang. Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 206-215, 2018.
[30] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912- 1920, 2015.
[31] Alex H Lang- Sourabh Vora, Holger Caesar. Lubing Zhou, Jiong Yang. and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12697-12705, 2019.
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, arXiv preprint arXiv:1706. 170.:1770..3362.. 2017.
[33] Tianhang Zheng, Changyou Chen, Junsong Yuan. Bo Li, and Kui Ren. Pointeloud saliency maps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019.
[34] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, arXiv preprint
2018.
[35] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European conference on computer vision, pages 69 84. Springer, 2016.
[36] Panos Achlioptas, Olga Diamanti. Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pages 40-49.
PMLR, 2018.
[37] Gunilla Borgefors. Distance transformations in arbitrary dimensions. Computer vision, graphics, and image processing, 27(3) 334,, 1984.
[38] Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001 03994, 2020.
[39] Ali Shafahi, Mahyar Najibi. Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer. Larry Davis. Gavin Taylor, and Tom Goldstein. Adversarial training for free! arXiv preprint 1rXiv:1904.12865 2019.
[40] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoe V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 819 9288 2020.
[41] Mikaela Angelina Uy. Quang- Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai- Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1588-1597 2019.
[42] Li Yi. Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35(6):1-2,, 2016.
[43] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-ric 3d model repository.
arXiv preprint arXiv: 15/2.03012, 2015.
[44] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv: 1xXiv:4412.666 1411.6980,. 2014.
[45] NVIDIA V100 TENSOR CORE GPU, https://www..viieee www.e.coooooeeaeeeeeee en- conn-s/dataaa. htt///ttaaacetttevoo/..o. 2020.
[46] Anh Bui, Trung Le, He Zhao, Paul Montague, Olivier Vel, Tamas Abraham, and Dinh Phung. Improving adversarial robustness by enforcing local and global compactness. arXiv preprint arXiv:2007.05122 2020.
[47] Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Somesh Jha, and Xi Wu. Concise explanations of neural networks using adversarial training. arXiv, pages arXiv-1810, 2018
[48] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European Conference on Computer Vision, pages 574 591. Springer, 2020.
[49] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. //929, 2020.
[50] Han Zhang, lan Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. International conference on machine learning, pages 7354-73646 PMLR, 2019.
[51] Chong Xiang, Arjun Nitin Bhagoji, Vikash Sehwag, and Prateek Mittal. Patchguard: Provable defense against adversarial patches using masks small receptive fields. arXiv preprint arXiv:2005 1088,, 2020.
12

---

[52] Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. In International Conference on Learning Representations, 2020.
[53] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning, pages 2206 2216. PMLRR 2020.
[54] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Discovering adversarial examples with momentum. arXiv preprint arXiv:1710.06088 2017.
[55]
Shuran Song and Jianxiong Xiao. Deep Sliding Shapes for amodal 3D object detection in RGB-D images. In CVPR, 2016.
[56] Lyne P. Tchapmi, Christopher B. Choy, Iro Armeni, Jun Young Gwak, and Silvio Savarese. Segcloud: Semantic segmentation of 3d point clouds. In International Conference on 3D Vision (3DV), 2017.
[57] Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian- Yi Zhou. Tangent convolutions for dense prediction in 3d. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3887-3896, 2018
[58] Hugues Thomas, Charles R Qi, Jcan-EEmaauul Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6411-6420, 2019. www.ootwwooooooo.eooooooe com/sota/
[59] 3D Point Cloud Classification Benchmark on ModelNet40. 3d- p--ppoonttlloo cloud- www.sslllaaaiiiooo on- modemmete0,, 2021.
[60] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 1920-1929, 2019.
[61] Honglak Lee. Roger Grosse, Rajesh Ranganath, and Andrew Y Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th annual international conference on machine learning, pages 609-616, 2009.
[62] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pages 1422-1430. 2015.
[63] Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre- pre- training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 201-208. JMI. Workshop and Conference Proceedings, 2010.
[64] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarian representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
[65] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
[66] Kevin Eykholt, Ivan Eytimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical world attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1634, 2018.
[67] Yulong Cao, Chaowei Xiao, Dawei Yang, Jing Fang, Ruigang Yang, Mingyan Liu, and Bo Li. Adversarial objects against lidar- based autonomous driving systems. arXiv preprint arXiv:1907.00 1907005418, 2019.
[68] Haonan Qiu, Chaowei Xiao, Lei Yang, Xinchen Yan, Honglak Lee, and Bo Li. Semanticadv: Generating adversarial examples via autributeccondition image editing. In European Conference on Computer Vision, pages 19 37. Springer, 2020.
[69] Yulong Cao, Ningfei Wang, Chaowei Xiao, Dawei Yang, Jin Fang, Ruigang Yang, Qi Alfred Chen, Mingyan Liu, and Bo Li. Invisible for both camera and lidar: Security of multi-sensor fusion based perception in autonomous driving under physical-world attacks. In 2021 IEEE Symposiun on Security and Privacy (SP), pages 176-194. IEEE, 2021.
[70] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pages 97. IEEE, 2017.
[71] Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. arXiv preprint arXiv: 0026000000,0......00000000000110 2018
[72] Chenglin Yang, Adam Kortylewski, Cihang Xie, Yinzhi Cao, and Alan Yuille. Patchattack: A black- texture- based attack with reinforcement learning. In European Conference on Computer Vision, pages 681-698. Springer, 2020.
13

---

[73] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In International Conference on Computer Vision. IEEE, 2017.
[74] Lifeng Huang. Chengying Gao, Yuyin Zhou, Cihang Xie, Alan Yuille, Changqing Zou, and Ning Liu. Universal physical camouflage attacks object detectors, 2019.
[75] Lifeng Huang. Chengying Gao. Yuyin Zhou, Cihang Xie, Alan L Yuille, Changqing Zou, and Ning Liu. Universal physical camouflage attacks on object detectors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 720-729, 2020.
[76] Chaowei Xiao, Jun- Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. arXiv preprint arXiv: 1001: 026112 2018.
[77] Yuxin Wen, Jiehong Lin, Chen, and Kui Jia. Geometry aware generation adversarial and cooperative point clouds. 2019.
[78] Abdullah Hamdi, Sara Rojas, Ali Thabet, and Bernard Ghanem. Advpe: Transferable adversarial per- turbations on 3d point clouds. In European Conference on Computer Vision. pages 241-257. Springer. 2020.
[79] Chaowei Xiao, Dawei Yang, Bo Li, Jia Deng, and Mingyan Liu. Meshadv: Adversarial meshes for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6898 6907, 2019.
[80] Yuzhe Yang, Guo Zhang, Dina Katabi. and Zhi Xu. Me-net: Towards effective adversarial robustness with matrix estimation, arXiv preprint urXiv:1905. 11971,, 2019.
[81] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv:1704. 01 155, 2017.
[82] Mitali Bafna, Jack Murtagh, and Nikhil Vyas. Thwarting adversarial examples: An L_O-robustsparse fourier transform. arXiv preprint 111.:1811..000..3. 2018.
[83] Nicolas Papernot, Patrick McDaniel, Xi Wu. Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on xecurity and privacy (SP), pages 582-597. IEEE, 2016.
[84] Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pages 135-147, 2017.
[85] Huan Zhang. Hongge Chen, Chaowei Xiao, Sven Gowal. Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards stable and efficient training of verifiably robust neural networks. arXiv preprint arXiv:1906.0633 2019.
[86] Chaowei Xiao, Ruizhi Deng. Bo Li. Fisher Yu, Mingyan Liu, and Dawn Song. Characterizing adversarial examples based on spatial consistency information for semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 117-234,222 2018.
[87] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li. Duane S Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on observations. 2020.
[88] Chaowei Xiao. Ruizhi Deng, Bo Li, Taesung Lee, Benjamin Edwards, Jinfeng Yi, Dawn Song, Mingyan Liu, and Ian Molloy. Advit: Adversarial frames identifier based on temporal consistency in videos. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 3968-3977, 2019.
[89] Daniel Liu, Ronald Yu, and Hao Su. Extending adversarial attacks and defenses to deep 3d point cloud classifiers. In 2019 IEEE International Conference on Image Processing (ICIP), pages 2279-2283 IEEE, 2019.
[90] Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton. Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. Stochastic activation pruning for robust adversarial defense. arXiv preprint arxiv: 1033vv 803,01442 2018.
[91] Nicolas Papernot and Patrick McDaniel. Extending defensive distillation. arXiv preprint arXiv: 1705.55544, 2017.
[92] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. arXiv preprint arXiv:2002. 2020.
[93] Honghin Liu, Jinyuan Jia, and Neil Zhenqiang Gong. Pointguard: Provably robust 3d point cloud classification. arXiv preprint (0X.::2103.03000 2021.
[94] Tobias Lorenz, Anian Ruoss, Mislav Balunović, Gagandeep Singh, and Martin Vechev. Robustness certification for point cloud models. arXiv preprint arXiv:2103. 2021.
[95] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1412:6442 2014.
14

---

[96] Cihang Xie. Mingxing Tan, Boqing Gong, Alan Yuille, and Quoe V Le. Smooth adversarial training. arXiv preprini arXiv:2006.14553 14536, 2020.
[97] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael Jordan. Theo- retically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pages 7472 7482. PMLR, 2019.
[98] Ahmadreza Jeddi. Mohammad Javad Shafiee, and Alexander Wong. simple fine-tuning is all you need: Towards robust deep learning via adversarial fine-tuning. arXiv preprint arXi::00/2. 2020.
15


Reviews:
[{"review_id": "ziRV4SALWd3", "reviewer": "Reviewer_tgUE", "summary": "The paper provides a deep dive into the adversarial robustness of a 3D point cloud recognition models. Specifically, the paper considers PointNet, DynamicGraphCNN, and PointCloudTransformer based 3D point cloud recognition architectures for the analysis. There are comprehensive experiments followed with detailed discussions shedding some light into the behavior of the respective model under adversarial perturbations. Furthermore, the paper also demonstrate few ways to create a adversarially robust model and provides an analyses if existing approaches proposed to counter adversarial attacks for images translate well to the 3D task.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n+ The paper is very well structured and nicely written. The motivation is clearly stated and leads right into the contributions of the paper. The paper does a good job in introducing the problem and discussing the status of existing art and why there needs to be more exploration of the topic of interest. \n\n\n+ The paper addresses an important outstanding problem in the field. There are many works in the literature trying to come-up with novel ways to attack a 3D point-cloud recognition models. However, there is very little research done in providing a defense mechanism that can counter those attack to provide a robust 3D point-cloud recognition models. The robustness of these models is an important and interesting aspect of 3D point-cloud recognition and requires more exploration as rightly pointed out in the paper.\n\n\n+ The experiments provided by the paper are extensive and covers a lot of aspects related to robustness of 3D point-cloud recognition models. The experiment analysis includes many popular and recent 3D point-cloud recognition architectures such as PointNet, DynamicGraphCNN, and PointCloudTransformer. Furthermore, each model is trained with multiple different strategies to provide a comprehensive comparison. The performance is evaluated on multiple 3D point-cloud recognition benchmark datasets.\n\n\n+ Since, this is one of the first work attempting to develop strategies that can learn adversarially robust 3D point-cloud recognition model, the experimental analysis provided in paper can prove useful to the community. This extensive analysis enables the others to be able to easily built on top of this work and compare with the benchmark established in this work. Moreover, the interesting insights provided in the paper could prove very useful in driving the future research in this domain.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\n- The paper shows clear usability of having pretext-task based self-supervision in improving model robustness against adversarial perturbations. However, in recent self-supervised learning literature pretext-task based approach are rarely used. Most of the methods utilize contrastive learning based approach for self-supervised training. Given their popularity, it would be a crucial contribution to the community to study usability of contrastive learning based self-supervision in improving adversarial robustness of a 3D point-cloud recognition models. Furthermore, there are existing work [1] studying contrastive learning based self-supervision for 3D point-cloud data, it makes sense to include these methods in to experimental study.\n\n\n- The paper follows a PGD-style attack model to fool the 3D point-cloud based recognition models. However, there are many recent adversarial perturbation methods designed specifically for the 3D point-cloud data, whereas the paper does not include these attack models. In my opinion, having these attack model is of utmost important as they are created specifically considering 3D point-cloud data in mind, unlike a straight-forward extension of PGD-style attack which was created with 2D images in mind.\n\n\n- The novelty of proposed robustness strategy is limited. Many other works have previously explored this approach [2] and the ability of self-supervision in helping adversarial robustness of the model.\n\n\n- The paper only considers adversarial training approach for baseline experiments. Whereas there have been many other methods proposed over the years that has been shown to work better than adversarial training such as denoising layers [3], input pre-processing [4] etc. To clearly show the effectiveness of the proposed self-supervised training approach, comparison with these baseline is neccessary to evaluate how well pretext-task based self-supervision works for improving adversarial robustness of the model.\n\n\n[1] Xie, Saining, et al. \"Pointcontrast: Unsupervised pre-training for 3d point cloud understanding.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\n[2] Hendrycks, Dan, et al. \"Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty.\" Advances in Neural Information Processing Systems 32 (2019): 15663-15674.\n\n[3] Xie, Cihang, et al. \"Feature denoising for improving adversarial robustness.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[4] Samangouei, Pouya, Maya Kabkab, and Rama Chellappa. \"Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models.\" International Conference on Learning Representations. 2018.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "m5nXGme3CrI", "reviewer": "Reviewer_6Xbb", "summary": "\nThis paper profoundly analyzes the robustness of 3D models for point clouds. The analyses are insightful. Some of the observations are indeed interesting, which differs from previous observations. Still, I have some concerns about this paper.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n(Positive) Previous works show local features in CNN could hurt robustness. This paper offers an opposite observation. The analysis in this paper reveals that local feature learning is desirable for adversarial robustness since it limits the adversarial propagation between the point-level input perturbations and the model's final output. \n\nIs this difference because point data is different from image data? In Line 221, the authors have provided insightful comments for this. I really like these comments. The authors find that DGCNN outstands to be the most robust architecture, consistently achieving ∼15% stronger robustness than the other two models on both ModelNet40 and ScanObjectNN.\n\n(Negative) Although the authors have provided good explanations, I still have some concerns. Is the transformer not correctly used? For example, for transformer, is an adjacent matrix used to restrict the attention maps? Besides, is MLP not correctly applied to point cloud? Can we use appreciation of global/shape features can help improve model robustness for point cloud similar to that in the 2D domain?\n\n(Positive) The authors understand that a label-free strategy is preferred due to the cost and difficulty of 3D point cloud data annotation. This idea is clear and promising.\n\n(Neutral) The authors show that pre-training on SSL tasks improves the adversarial robustness of the fine-tuned models. But it is unclear why this can improve the robustness. Could the authors provide some theoretical analyses?\n \n(Neutral) Similarly, the authors find that only APF consistently achieves robustness improvements in 3D. AJT does not always help since the distributional gap between data for SSL and recognition tasks will distract each other in AJT. It would be better if the authors could provide some theoretical analyses.\n\n(Positive) The authors have an insightful observation that adversarial examples generated by attacking them do not transfer well among each other. I am delighted that the authors use this property to improve the robustness point could neural networks.\n\n(Negative) The architecture description part lacks details. For example, for transformer, is an adjacent matrix used to restrict the attention maps? What is the overall Point cloud transformer like? What is the MLP like? What is the GCN like?\n\n(Negative) Line 87: what is the \\circ \"F = M ◦ H\"?\n\n(Negative) This paper lacks too many critical details. A rewriting is necessary. For example, the definitions of \\Pi, saliency() are crucial; please give a full definition.\nLine 93: \\Pi is the projection function to project the adversarial example to the pre-defined perturbation space S \nLine 97:  where saliency() calculates the importance scores of the input points and / denotes the dropping operation\n\n(Negative) I am very familiar with self-supervised learning. In the 2D image, currently, we usually use contrastive learning or metric learning (e.g., MoCos, BYOL, and SwAV) or masking (BEiT); In NLP, we typically use masking (e.g., BERT). 3D Rotation, 3D Jigsaw, and Autoencoder are outdated methods for self-supervised learning methods. Even for 2D adversarial training with self-supervised learning, many state-of-the-art methods use contrastive learning rather than the outdated technique. It is strange that the author does not adopt the new techniques. Could the authors provide analysis/results on these new techniques? \n\nMoCo: Improved Baselines with Momentum Contrastive Learning;\nMomentum Contrast for Unsupervised Visual Representation Learning\nBYOL: Bootstrap your own latent: A new approach to self-supervised learning\nBEiT: BERT Pre-Training of Image Transformers\nBERT: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding\n\n(Negative) Why the second term \\mathcal{L}_t in Eqn. (5) are not adversarially trained? If \\mathcal{L}_t is not adversarially trained, I do not think this is a proper Adversarial Joint Training (AJT). Hence, all the claimed observation about Adversarial Joint Training (AJT) does not convince me. One important baseline, i.e., the true Adversarial Joint Training (AJT), in which both terms are adversarially trained, should be included.\n\n(Negative) Since the AJT is not a true AJT, I am not convinced by the following analysis.\n Line 252: it is still a separate learning task. Rotated and disassembled images still preserve similar local features to the original images since the RGB values do not change, so that the auxiliary optimization in AJT will not distract AT, but help models learn robust global features [24]. \n\n(Negative) This paper lacks sufficient ablation studies to provide component analysis for the proposed method.\n\n(Positive) I am happy to see the cross-task examination of the adversarial examples and am happy to see the use of cross-task ensembling to improve the robustness. The authors found that different fine-tuned models preserve different vulnerabilities, and there is at least ∼10% gain of RA when transferring attacks on models fine-tuned from different tasks on ModelNet40. They thus combine the models and further test the ensemble model's robustness.\n\n(Negative) As far as I am concern, not only different fine-tuned models preserve different vulnerabilities, but different architectures also preserve different vulnerabilities. Please provide cross-architecture examination and proved cross-architecture ensembling.\n\n(Positive) I am happy to see part segmentation is a more robust task than classification. I am so glad to see the author's insightful comments on this observation.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nPlease refer to and respond to the \"negative\" items in the above main review.\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "\nNo.\n\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "SXqk1f6341g", "reviewer": "Reviewer_wEE7", "summary": "This paper presents a self-supervision based method for adversarially robust 3D point cloud recognition, in which three types of self-supervised tasks are dealt with to help learn 3D point features: 3D rotation, 3D jigsaw and autoencoder. Empirical results demonstrate the effectiveness of the proposed method. Besides, the paper is well-written and easy to follow. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. It is great to see a thoughtful experimental setup: 4 datasets, 3 point-cloud threats, 3 network backbone architectures, 3 pretraining tasks and 2 supervised tasks. But it seems that generation network based adversarial attacks (e.g. LG-GAN [3]) and optimization based ones (e.g. 3D Carlini-Wagner attack) are not verified. I need to see more evaluation performances to justify the significance of self-supervised learning on this task. \n\n2. Novelty seems a little bit limited. The core idea is related to researches on 2D domain [1][2], and this paper is all about applying this key idea to 3D point recognition. Beyond that, I don’t see any insights of either new technical emergence or discoveries. \n\n3. I am confused about adversarial pretraining. During adversarial pretraining, only self-supervised tasks are implemented, without classification. How to implement adversarial attacks in this stage? My understanding is that adding adversarial disturbance during this stage to impact on the self-supervised loss.\n\n4. Two strategies are proposed to integrate self-supervised learning and adversarial learning: adversarial pre-training for fine-tuning (APF) and adversarial joint training (AJT). In Table 3, only two SSL tasks are shown for AJT, yet there’s a lack of autoencoder task for APF in Table 2. It’s clearly better to have a comprehensive comparison between two strategies. \n\n5. Detailed annotation should be provided: RA means robust accuracy. \n\n6. Spelling error: l117 The I seobjective\n\n[1] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. https://arxiv.org/pdf/1906.12340.pdf\n[2] Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning. https://arxiv.org/pdf/2003.12862.pdf\n[3] Hang Zhou, Dongdong Chen, Jing Liao, Kejiang Chen, Xiaoyi Dong, Kunlin Liu, Weiming Zhang, Gang Hua, Nenghai Yu. LG-GAN: Label Guided Adversarial Network for Flexible Targeted Attack of Point Cloud-based Deep Networks. https://arxiv.org/pdf/2011.00566.pdf\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are not clear to me. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "A622l2KAQ30", "reviewer": "Reviewer_Vmom", "summary": "The paper studies how self-supervised learning and adversarial training may interact to affect the accuracy and adversarial robustness of models that process point clouds. The study includes several architectures, threat models, self-supervised learning objectives, and tasks on which performance is measured. The experimental setup of the paper is comprehensive and reasonable; further, its findings are interesting and are reported in a sensible manner.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper's main object of study is important for both the 3D vision community and the adversarial robustness community. I think both fields can gather insights from the experiments presented in this paper.\n+ The experimental framework presented in the paper is comprehensive, considering architectures, threats, learning objectives and tasks, making it a rather complete benchmark.\n+ The experimental results themselves are presented in a sensibly-interpreted manner, which I think many readers can value. (I am making reference to the \"insight\" paragraphs reported in the paper)\n+ The paper's overall writing is of good quality, sustaining a reasonable narrative that incrementally introduces the notions that the reader requires for understanding the results.\n+ The paper shares spirit with previous works, I think in particular with [25], but provides a significant original contribution in itself.\n\n\nWeaknesses that affect my decision: (mainly my concerns are of experimental nature, as I find some results somewhat confusing or unexpected. I think this is most likely an understanding issue from my side, but, if it is not, and these are actually implementation issues that invalidate the experimental findings of the paper, I would be forced to lower my rating substantially)\n+ For the \"max\" type of ensemble (as reported in L297), have the authors considered the fact that, while gradients can backpropagate smoothly to the model whose logit was the max, perhaps the max-selection itself is introducing noise into the gradient? Could this be a source of noise for the results then?\n+ I understand the paper applies dual batch normalization to the inputs (and perhaps also in intermediate layers of the network). What impact, exactly, does this have in terms of spatial coordinates, which have a clear correlation between them, as given by the captured object's actual shape? Could this dual batch normalization be, in itself, introducing noise into the overall experimental framework.\n+ I am unsure regarding how to interpret the results of Table 2: how is it possible that in many cases \"Adversarial $x$\" has *lower* robust accuracy than just \"$x$\" (for $x \\in {3D rotation, Jigsaw, Autoencoder}$)? Is this an implementation issue? Or is it that this notation is making reference to a different concept than the one I say? It would be rather expected that regular accuracy drops a bit (as happens in some entries in the table), but I do not understand why robust accuracy would not improve. Could the authors please elaborate on this phenomenon? Also regarding this phenomenon, is it also related to the fact that Table 3 does not report rows \"Adversarial $x$\"?\n+ Tables 4, 5 and 6 report that robust accuracy of standardly-trained models (\"ST Baseline\") is rather large (definitely well-above 0). This stands in stark contrast with the results presented in Table 1 (and the literature on adversarial attacks in general). Can the authors confirm these results? I find them (suspiciously) strange.\n+ In L230 the paper hypothesizes that the robustness properties of DGCNN can be attributed to the usage of EdgeConv. Has this claim been verified experimentally? I do not see such results in the paper.\n+ L125 reports that different backbones were utilized for the encoder, but I did not see experiments on this.\n+ L155 reports that the add/drop adversaries are easier to detect than the shift adversary. Are there experiments for this \"detection\" or on what grounds is this statement based? (A similar statement is done in L177)\n\n\n\nWeaknesses that do not affect my decision but should be addressed if possible:\n+ L19: deep learning [...] they\n+ Figure 1(b) could benefit from displaying the symbols for $\\mathcal{M}_{\\theta_m}$, $\\mathcal{H}_{\\theta_h}$ and $\\mathcal{F}_{\\theta_f}$ within the figure itself.\n+ L64: state under which attack this robust accuracy is being computed\n+ L94: I assume the pre-defined perturbation space $\\mathcal{S}$ is implicitly related to the $p$ in the $\\ell_p$ norm ball, but perhaps it would be helpful to explicitly say so.\n+ L100: perturb\n+ L117: I seobjective\n+ L145: recongnition\n+ L147: $x_t$ vs. $x^t$ (same in L165)\n+ L163: it\n+ L177: what does \"k=14,5\" mean? 14 for training and 5 for testing? If so, what is the rationale behind this decision?\n+ L177: most k [...] salient\n+ L192: epoch\n+ Table 1: report what \"CA\" and \"RA\" stand for (same for Table 2).\n+ In Table 2's caption, report under which attack these results are obtained (same comment for Table 3). I understand they are against shifting, but it would not hurt to explicitly state this.\n+ Legends and labels are kind of uncomfortable to read in Figure 3.\n+ Figure 3 caption: \"rest plots\"\n+ L273: attacks\n+ L279: We\n+ L326: inter-over...\n+ L339: a\n+ L356: pint\n+ L356: preception", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After some discussion, four reviews, all of them pretty thorough, recommend accepting this paper.
I see nothing that would cause me to want to contradict this unanimous decision, and so I am recommending acceptance.
Decision: Accept (Poster)