Isometric 3D Adversarial Examples in the Physical World
Authors: Yibo Miao
Yinpeng Dong
Jun Zhu
Xiao-Shan Gao
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D adversarial examples
physical attacks
isometry

Abstract: Recently, several attempts have demonstrated that 3D deep learning models are as vulnerable to adversarial example attacks as 2D models. However, these methods are still far from stealthy and suffer from severe performance degradation in the physical world. Although 3D data is highly structured, it is difficult to bound the perturbations with simple metrics in the Euclidean space. In this paper, we propose a novel $\epsilon$-isometric ($\epsilon$-ISO) attack method to generate natural and robust 3D adversarial examples in the physical world by considering the geometric properties of 3D objects and the invariance to physical transformations. For naturalness, we constrain the adversarial example and the original one to be $\epsilon$-isometric by adopting the Gaussian curvature as the surrogate metric under a theoretical analysis. For robustness under physical transformations, we propose a maxima over transformation (MaxOT) method to actively search for the most difficult transformations rather than random ones to make the generated adversarial example more robust in the physical world. Extensive experiments on typical point cloud recognition models validate that our approach can improve the attack success rate and naturalness of the generated 3D adversarial examples than the state-of-the-art attack methods.

Isometric 3D Adversarial Examples in the Physical World
Yibo Miao 1.3,, Yinpeng Dong2.3t, Jun 2,,453 Xiao-Shan Gao
1 KLMM, UCAS, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China 2 Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua-Bon Joint MI. Center, THBI Lab, BNRist Center, Tsinghua University, Beijing 100084, China
RealAI Peng Cheng Laboratory Pazhou Laboratory (Huangpu), Guangzhou, China yibomiao21@163.cmmm com, (dongyinpeng, @tszs) edu. en, xgao@mmro, iss ac.cn
Abstract
3D deep learning models are shown to be as vulnerable to adversarial examples as 2D models. However, existing attack methods are still far from stealthy and suffer from severe performance degradation in the physical world. Although 3D data is highly structured, is difficult to bound the perturbations with simple metrics in the Euclidean space. In this paper, we propose novel c-isometric (c- ISO) attack to generate natural and robust 3D adversarial examples in the physical world by considering the geometric properties of 3D objects and the invariance physical transformations. For naturalness, we constrain the adversarial example be e-isometric to the original one by adopting the Gaussian curvature a surrogate metric guaranteed by theoretical analysis. For invariance to physical transformations, we propose a maxima over transformation (MaxOT) method that actively searches for the most harmful transformations rather than random ones make the generated adversarial example more robust in the physical world. Experiments on typical point cloud recognition models validate that our approach can significantly improve the attack success rate and naturalness of the generated 3D adversarial examples than the state-of-the art attack methods.
1 Introduction
Deep neural networks (DNNs) have achieved unprecedented performance on numerous tasks, in- cluding 2D image classification [33, 25, 27] and 3D point cloud recognition [48, 49, 68). However, DNNs are vulnerable to adversarial examples [59, 20] inputs crafted by adding imperceptible perturbations original examples that can cause misclassification of the victim model. Adversarial examples are prevalent in various domains beyond images, including texts [28], speeches [84] and 3D objects [73]. As deep 3D point cloud recognition has been adopted in safety-critical applications, such as autonomous driving [5, 85], robotics [65, 94], medical image processing [60], it has garnered increasing attention studying the adversarial robustness of 3D point cloud recognition models [6].
However, the existing adversarial attacks on point cloud recognition models are still far from stealthy and suffer from drastic performance degeneration in the physical world. There is usually trade-off between the stealthiness and the real-world attacking performance, making it challenging to achieve the best of both worlds. Early methods [79, 73, 42] adopt gradient-based attacks add, remove, and modify points, but they are limited to digital-world attacks. The KNN attack [63] and the G attack [69] constrain the smoothness of the adversarial point clouds and reconstruct adversarial
"This work was done when Yibo Miao was intern at RealAI, Inc; Corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Original
KNN
Good!
e-15O
Figure 1: An illustration of adversarial objects crafted by KNN attack [63], CeoA* attack [69], Mesh Attack [88] and our e-ISO attack against the PointNet model: KNN attack and Geo. attack can produce unnatural adversarial objects (and often low success rates); Mesh Attack can generate lot of distortions: while c-ISO attack improves the naturalness of the 3D adversarial sample and ensures the consistency between the intrinsic geometric properties of the adversarial and original 3D objects [8].
meshes from the point clouds that can be 3D-printed in the physical world. Although these works demonstrate successful physical attacks, point cloud reconstruction introduces large noises and errors, resulting in low attack success rates and unnaturalness of the adversarial objects in the physical world. Mesh Attack [88] is recently proposed to perturb the mesh representation of 3D objects, which improves the success rate but often creates large distortions that can be easily detected by humans as anomalies, as illustrated in Fig. 1. Overall, it is difficult to achieve both the naturalness and effectiveness of 3D adversarial attacks in the physical world, which we think largely due to the lack of an appropriate metric to characterize the naturalness of 3D data.
To address these issues, we propose an e-isometric (e-ISO) attack method generate natural and robust 3D adversarial examples in the physical world against point cloud recogmition models. The e-ISO attack improves the naturalness of the 3D adversarial example by constraining it to be e- isometric (see Definition 1) to the original one, which guarantees the consistency between the intrinsic geometric properties of two 3D objects [8]. We theoretically demonstrate that Gaussian curvature (see Definition 2) can be used to provide a sufficient condition to ensure that two surfaces are e-isometric. Due to the computable and differentiable nature of Gaussian curvature, we adopt as new regularization loss to practically generate natural 3D adversarial examples. To improve the robustness of 3D adversarial examples under physical transformations, we further propose maxima over transformation (MaxOT) method that actively searches for the most harmful transformations rather than random ones [i] for optimization. Armed with Bayesian optimization that provides better initialization of the transformations, MaxOT is able to find set of diverse worst-case transformations, leading to improved performance the 3D adversarial examples in the physical world.
We conduct extensive experiments to evaluate the performance of our method on attacking typical point cloud recognition models [48, 49, 68]. Results demonstrate that, in comparison with the alter- native state-of-thee-rt attack methods [63, 69, 88], e-ISO attack achieves higher success rates, while making the generated adversarial examples more natural and robust under physical transformations. A physical-worke world experiment is conducted by 3D-printing the adversarial meshes and re- re-scanning the objects for evaluation, which also validates the effectiveness of our method.
Related work
Deep learning on 3D point clouds. Deep 3D point cloud recognition [48, 21, 74, 76, 81, 50, 61] has emerged in recent years with various applications in many fields, such as 3D object classifica- tion [55, 37, 83, 86), 3D scene segmentation [22, 67, 77, 26], and 3D object detection in autonomous driving [97, 78]. One of the pioneering works is PointNet [48], which directly applies a multilayer perceptron to learn point features and aggregates them in an efficient way using a max-pool module. PointNet++ [49] and a large number of later works [13, 43, 80] are built on PointNet to further capture fine-grate finennriieed local structure information from the neighborhood of each point. Recently, some works have focused on designing special convolutions on 3D domains [2, 39, 44, 62] or developing graph neural networks [18, 52, 53, 68] to improve point cloud recognition.
3D adversarial attacks. Following the previous studies on adversarial machine learning the image domain [59, 34, 4, 10, 19, 82, 11, 12], many works [73, 3, 41] apply adversarial attacks to the

---

3D point cloud domain. Xiang et al. [73] proposed point generation attacks by adding a limited number of synthetic points to the point cloud. Recently, more studies [70, 93] use gradient-based attack methods to identify key points from the point cloud for deletion. More point perturbation attacks [24, 45, 91, 9] learn to perturb the xyz coordinates of each point through a C&W W framework [4] based on metrics defined in the Euclidean space. Zhao et al. [92] attack by the isometric transformations in the Euclidean space such as rotation. It is worth noting that we consider isometric mappings between surfaces, which is essentially different from [92]. Later works [35, 87] further apply iterative gradient methods to achieve more advanced adversarial perturbations. Besides, other works consider generative models [96], 3D data attacks [64, 57], adversarial robustness [89, 58], attacks against LIDAR [36, 30], autonomous driving [56, 90], backdoor attacks [38, 751, etc., in the 3D domain. However, the existing attacks on 3D point cloud recognition
still far from stealthy and the only
three methods that consider the physical-world attacks [63, 69, 88) are not very effective. In this paper, surpass the performance of previous methods by proposing novel e-isometric (c-ISO) attack method to generate natural and robust 3D adversarial examples in the physical world.
3 Methodology
We now formally present E-ISO attack. We first present the general problem formulation, and then describe how e-ISO attack enhances the imperceptibility and robustness of the generated 3D adversarial samples, respectively.
3.1 Problem formulation
To generate 3D adversarial objects in the physical world, it is more straightforward perturb the mesh representation of 3D objects rather than point clouds [88] since the reconstruction process can incur large errors [63, 69). A mesh M FV. is approximate shape representation of its underlying surface, where V {u, 1= is the set of n. vertices of zyz coordinates, and F := {2r is the set of ny triangle faces represented by the indices of vertices. We let denote a differentiable sampling process such that P := ((M) E X' is the point cloud obtained by randomly sampling the mesh surface, where X' is the set of all point clouds. We let y E y denote the corresponding ground-truth label of M as well as P.
In this paper, we focus on the challenging targeted attacks against deep 3D point cloud classification models [48, 49, 68]. Given a point cloud classifier f: X' + y, the goal of the attack is to generate an adversarial mesh Mads = (Vady: F) for the original one .M with vertex perturbations such that the sampled point cloud Pade := S(Madr) will be misclassified by f as the target class (f v)- In general, the perturbation should be small to make the adversarial mesh Made inconspicuous under human inspection. Thus, the optimization problem of generating the adversarial mesh can be generally formulated as
(oMaad))"")) min L3 (S (Madv), -²") + 3- -RR.RR MaddoyM.. Madn
where Ly is the loss that facilitates the miselassification of Pado y², R. is the regularization term that minimizes a perceptibility distance between Made and M, and B is a balancing hyperparameter between these two losses. In this paper, we try develop stealthy and robust attack method by proposing a new regularization loss R based on Gaussian curvature with theoretical guarantees to remain the naturalness as well as a new attacking loss L, enhance the robustness of the generated 3D adversarial objects under physical transformations, R and &3 will be introduced in the following.
3.2
eOO attack
Most of the existing 3D adversarial attacks only consider the constraints R defined in the Euclidean space [15, 81, 23]. The generated adversarial examples have noticeable point outliers that cause spikes appear on the object's ' surface, thus losing the naturalness. Moreover, the outliess are more easily removed and defended against. The main reason is that the existing methods do not consider the geometric properties of the 3D objects. In differential geometry, isometric mapping guarantees the consistency of the intrinsic geometric features of two objects [8]. Therefore, we propose constraint loss R based on e-isometric mapping to restrict the naturalness of 3D adversarial objects. We first give the definition of e-isometric below.
3

---

(a) Original
(b) KNN & GeoA*
(c) Mesh Atlack
(d) E-ISO
Figure 2: An illustration of e-isometric attack. (a): Original mesh. (b) and (c): Adversarial meshes generated by KNN & GeoA* attack and Mesh Attack, respectively. They consider only the constraints defined in the Euclidean space, and the curve lengths (shown as black curves) the generated adversarial examples differ significantly from those of the original samples, which not satisfy e-isometric and lose naturalness. (d): Adversarial mesh generated by e-ISO. We consider the geometric features of 3D objects and constrain the 3D adversarial example to be e-isometric to the original one, such that the curve lengths of the generated adversarial samples vary little and have naturalness.
Definition 1. Let S and denote two surfaces of IR3.. A diffeomorphism 9 S is called an e-isometric mapping if there exists constant n such that it takes any local curve C in curve Ö = CC)) in S satisfying (1 ne)s(C) < s(C)
ne)s(C) where s(-) is the length. The
surfaces S and S are then said be e-isometric.
As shown in Fig. 2, Fig. 2(a) is the original mesh, Fig. 2(b) is the adversarial mesh generated by KNN attack and Geo.9 attack, and Fig. 2(c) is the adversarial mesh generated by Mesh Attack. These three methods only consider the constraints defined the Euclidean space, and the curve lengths of the generated adversarial samples differ greatly from those of the original samples, which not e-isometric and lose naturalness. Fig. 2(d) the adversarial mesh generated by our proposed e-ISO attack. We consider the geometric features of 3D objects to generate natural adversarial examples by constraining them to be e-isometric to the original examples (i.e., the curve length of the resulting adversarial examples varies very little). However, it is intractable directly optimize the adversarial mesh to be e-isometric as the original one. Therefore, we give the definition of the Gaussian curvature of the surface from [8].
Definition 2. Let be a surface of R parameterized by r r= - [x(u, v), y(u, v), (u, U) where (u, v) E R. We let Tu. denote the partial derivatives of w.r.t. and v. Tuur Tuvs Fov denote the second partial derivatives of r, and A, ( denote the outer product and inner product, respectively. The parametrization thus defines unit normal vectors
[rArc] Arr of the surface S. We
E F
denote the eigenvalues of the coefficient matrix of the Weingarten map
M MNNI F
as 12 and kzz where E (uu, F (re: Te) and G (Te, re) are coefficients of the first fundamental and L = (Tuus n), M = (Tav, n) and N = (Tee, T2) are coefficients of the second
fundamental form. The Gaussian curvalure is defined K kik2 EN-M"
EG-FT.
Remark 1. The Gaussian curvature intrinsically measures the bending degree of the surface reflected
by the Gaussian mapping. Let the area element of the surface S be d.A (r" A Po; n)) dudv and the
area element under the Gaussian mapping S E TR3 S2 r(u, v) + nuu,v)
be dA' = (nz A nu, n) dudv.
From n. A n2 Kr, Ard (proof in Appendix D), obtain
Area (g(D) lim LACD) dA'
So KdA
lim D-P
lim D-P
K(P).
(2)
Area (D) D-P Jn dA
dD dA
Eq. (2) illustrates that the geometric meaning of Gaussian curvature is the ratio of the area of the domain at the point P on the surface and the area of the domain the corresponding point under the Gaussian mapping, i.c., the bending degree of the surface reflected by the Gaussian mapping.
Based Definitions 1 and 2, have the following theorem.
Theorem 1 (proof in Appendix A). Let S and Š denote two surfaces TR3; 40: : - 5 denote diffeomorphism that takes point v in S to point o' = =v) in S: and K(-) be the Gaussian curvature of the points. If (v) K(v')l < for any point v, then the surfaces S and S are e-isometric.

---

Theorem indicates that to make two surfaces e-isometric, one can constrain their Gaussian curvatures. Since the Gaussian curvature is computable and differentiable w.r.t. vertices, we adopt it constrain the naturalness of 3D adversarial meshes as
RGauss (Made M) Thy VEV.e'=p())VV.ds
[KK(V') (
where (7()) is the corresponding mapping between vertices in V and Vade- We follow the Gauss-
Bonnet formula [7] calculate the Gaussian curvature of the vertices as
2T EIEN(w) 02(v) A(v)
K(u)
(4)
where A() is the area of the vertex neighborhood, i.e. the area of the polygonal region joined by the consecutive midpoints of triangles incident the vertex of interest, N(e) is the set of faces containing v, and (v) is the interior angle of the face at vertex v. Note that the value of EieN(e) 0z(v) for a plane is 2T and the Gaussian curvature is O. The more curved the surface, the smaller the value of Die((,) (v) and the larger the Gaussian curvature.
In addition, we prevent the generated adversarial meshes from self-intersecting by using the Laplace loss [16], denoted as Lap (Made): which represents the distance between a vertex and its nearest neighbor's s center of mass, and the edge length loss [66], denoted as Redge (Made), which represents the smoothness of the surface. Thus, the overall regularization term can be expressed as:
R (Mados M) = X1 RGause (Madu: M) + A2 RLap (Madu) + X Redge (Madv): (5)
where Å1 A2 and λ are balancing hyperparameters.
3.3 Improving the robustness under physical transformations
Besides concerning the naturalness of 3D adversarial examples, we further enhance their robustness under physical transformations, such as 3D rotations, affine projections, cutouts, etc. A common method is the expectation over transformation (EOT) algorithm [1]. which optimizes the adversarial example over the distribution of different transformations. However, it is still challenging maintain the attacking performance under various physical transformations. As shown in the experiments, after using the EOT algorithm, there still some transformations that the generated adversarial examples are not robust to, leading to a reduction of the attack success rate.
To address this issue, our key insight is to consider the worst-case transformations rather than their ex- pectation, since if the adversarial examples are resistant to the most harmful physical transformations, they can also resist much weaker transformations, inspired by adversarial training [46]. Therefore, we propose a maxima over transformation (MaxOT) algorithm to actively search for physical transformations that maximize the misclassification loss. The loss function , is thus formulated as:
(S( ). y²) max E Cce (t(S(Madx))) T'CT
(6)
where T contains all possible transformations, T is a subset of transformations in T. and Lee the cross-entropy loss. Note that in Eq. (6) we consider a subset of transformations T" rather than single one because the loss landscape w.r.t. transformations is largely non-convex and contains many local maxima [14]. Thus we aim to find set of diverse worst-case transformations. By integrating Eq. (6) into Eq. (1), it forms minimax optimization problem, where the inner maximization aims to find physical transformations that maximize the cross-entropy loss, while the outer minimization aims to optimize the adversarial example with the worst-case transformations.
3.3.1 Bayesian optimization
To solve problem (6), we search for the worst-case transformations one by one. Given an initialized transformation, we perform gradient- based optimization to update the transformation parameters (e.g., angles for rotations). However, randomly selecting initialized transformations is ineffective since the random initialization may drop into regions of weak transformations, which limits the exploration of the space of all transformations. To address this issue, we propose to adopt the Bayesian optimization [17, 54] to better break the dilemma between exploration and exploitation to find more appropriate initialized transformations.
5

---

Original
Goouf
Mesh
--IOO
Original
Figure 3: Adversarial objects obtained by KNNN attack, GeoA attack, Mesh Attack, and e-ISO attack against PointNet model. The KNN and GeoA attack produce adversarial samples with dense irregular deformations. The Mesh Attack produces distortions and self-intersecting surfaces (i.e., black surfaces). None of them guarantees naturalness, while c-ISO attack is more natural.
Bayesian optimization is an efficient method for solving global optimization problems consisting of two key components: surrogate model, such as Gaussian process (GP) [51] Bayesian neural network (BNN) [32], which models the unknown objective; and an acquisition function a(-), which maximized by balancing exploitation and exploration recommend the next query location. We choose GP as a surrogate model, which provides a Bayesian posterior distribution to describe the unknown function fee (t(S(Madv)) )), y²). We use the expected improvement (EI) [29, 47] acquisition function, which measures the expected improvement of each point with respect to the current best value. Then, the unknown function is sampled by maximizing the acquisition function better explore the space of all transformations by selecting the next query point in the region where the prediction is high and the model is very uncertain.
As shown in the overall algorithm (in Appendix B), we update the Bayesian posterior distribution of the objective f using the observations obtained from the previous iterations the gradient descent process. Then, we maximize the EI acquisition function agı (t; D,) to find the initial transformation. From the initialization, we optimize the transformation parameters by gradient- based method to solve problem (6). This process is repeated for the number of transformations in the MaxOT algorithm.
Experiments
4.1 Experimental setup
Dataset. We use the ModelNet40 [71] dataset in our experiments. This dataset contains 12,311 CAD models with 40 common object semantic categories in the real world. We use the official split [48, 49], where 9,843 examples are used for training and the remaining 2,468 examples are used for testing. We follow [88] and get closed manifolds. For adversarial attacks, we follow [69] and take all the instances of the 40 categories that are well classified in the ModelNet40 testing set.
Victim models. Following [69, 88, 40], we select three commonly used point cloud classification networks the victim models, i.e., PointNet [48] PointNet++ [49], and DGCNN [68].
Evaluation metrics. To quantitatively evaluate the effectiveness of our proposed e-ISO attack, we measure the attack success rate (ASR). Besides, to measure the imperceptibility of different attack methods, we use the Chamfer distance D. [15] and Gaussian curvature distance D, (i.c., Eq. (3)) as evaluation metrics. We also report the attack success rates on several existing defenses [79, 95, 72] and under the physical world to further validate the superiority of our c-IOO attack.
Implementation details. We use Adam [31] to optimize the objective of our proposed e-IOO attack. We use a fixed learning schedule of 250 iterations, where the learning rate and momentum are respectively set as 0.01 and 0.9. We assign the weighting parameters X = 1.0, X2 = 0.2 and As = 0.8. The balancing parameter (3 is initialized as 1,500 and automatically adjusted by conducting 10 runs of binary search following [4]. We uniformly sample 1.024 points from the adversarial mesh.

---

Table Quantitative results KNN attack, GeoAA attack, Mesh Auack, and our proposed c-ISO attack against different models. Our proposed e-ISO attack outperforms all existing methods in terms of attack success rate (ASR) and imperceptibility. We adopt the Chamfer distance De [15] and Gaussian curvature distance D, as the evaluation metries.
PointNet
PointNet++
DGCNN De
Model
ASR
Dz
Dg
ASR
De
Dg
ASR
Dg
KNN
14.78%
0.0034
0.0096
6.24%
0.0027
0.0122
4.17%
0.0036
0.0125
Geo
19.65%
0.0066
0.0037
11.20%
0.0192
0.0031
8.24%
0.0172
0.0042
Mesh Attack
88.10%
0.0054
0.0048
94.79%
0.0045
0.0046
66. 06.79%
0.0051
0.0055
e-ISO
98.45%
0.0031
0.0004
99.58% %
0,0040
0.0004
84.16% %
0.0032
0.0009
90
Attack Success Rate (%g
80
78.47%
Attack Success Rate (%)
Attak Sucoess Rate (%)
70.72%
75
77.90%
23
78.48%
69.34%
70
65
15
60
64
55
50
50
100
150
200 250 300 350 400 500
50 100
150
200
300 350 400 450 500 Steps
50 100 1.50 200 250 300 360 400 450 500 Iseration Steps
Soeps
Figure 4: The attack success rate w.rt number of iterations curves of EOT and MaxOT against PointNet, PointNet+++, and DGCNN. Our proposed MaxOT algorithm can lead to higher attack success rate with small number of iterations than that of EOT with large number of iterations.
4.2 Pseudo physical attack
In this section, compare proposed c-ISO attack with existing methods [63, 69, 88] that generate adversarial objects under the setting of white-box targeted attack, including the KNN attack [63], GeoA3 attack [69], and Mesh Attack [88]. In Table 1, compare different methods under the measures of attack success rate (ASR), Chamfer distance D. [15] and Gaussian curvature distance Dg. Our proposed e-ISO attack outperforms all existing methods in terms of attack success rate and is almost optimal in terms of Chamfer distance D. and Gaussian curvature distance Dz, the evaluation metrics for measuring imperceptibility. These comparisons confirm the effectiveness of our proposed regularization term based on Gaussian curvature, to simultancously achieve the strongest adversarial attack and the most natural perturbations.
Fig. 3 shows the adversarial objects generated by different methods. The KNN attack and G attack produce dense irregular deformations the mesh surface, losing smoothness and regularity. The Mesh Attack creates severe deformations and spurs, which are easily perceived by humans visually. In addition, the severe irregularities and distortions generated by the Mesh Attack can lead to self-intersecting phenomena in the mesh of 3D objects, i.e., the black surfaces in Fig. 3. These self-intersecting surfaces can prohibit 3D-printing the adversarial objects in the physical world. By considering geometric features of the object instead of using metrics in the Euclidean space, our e-IOO attack produces adversarial samples without dense deformations, bursts, distortions, and self-intersection phenomena, while are stealthy and natural.
4.3 Robustness under physical transformation
To further verify the superior robustness of our proposed MaxOT algorithm under transformations in the physical world, we compare MaxOT to EOT. In Table 2, we compare the different methods under rotations to calculate the attack success rate, Chamfer distance D. and Gaussian curvature distance Dz. Our proposed MaxOT algorithm outperforms the EOT algorithm in terms of attack success rate. This comparison confirms our insight that the resistance to the most harmful physical transformations is better than random ones.

---

Table 2: Quantitative results of attacking different models using EOT and MaxOT. Our proposed MaxOT
algorithm outperforms the EOT algorithm in terms of attack success rate.
PointNct
PointNet++
DGCNN
Model
ASR
De
Dg
ASR
D.
Dg
ASR
De
Dg
EOT
76.20%
0.0074
0.0009
74.28%
0.0094
0.0007
65.72%
0.0068
0.0041
MaxOT
82.50%
0.0074
0.0009
84.14%
0.0094
0.0006
72.40%
0.0067
0.0039
Table 3: The attack success rate of various methods against PointNet under different defense methods. Our proposed e-ISO attack performs better under stronger defenses, and the attack success rate under DUP-Net and IF-Defence is higher than all other attacks.
Table 4: The success rate
of attacking the PointNet model in the physical world using the EOT algorithm and our proposed MaxOT al- gorithm.
no defence
SRS
SOR
DUP-Net
DF-Defence
KNN
32.15%
24.34%
11.25%
G
40.87%
3.34%
33.22%
11.97%
1.04%
Pointeet
Mosh Attack
93.39%
87.38%
89.93%
78.28%
49.06%
EOT
72.30%
e-ISO
99.1%
79.85%
83.42%
78.55%
60.51%
MaxOT
80.12
In Fig. 4, we give the success rate w.r.t. iterations curves for EOT and MaxOT of different victim models. Our proposed MaxOT algorithm is always ahead of the EOT algorithm at different number of iterations. Moreover, we note that our proposed MaxOT algorithm can lead to higher attack success rate with small number of iterations than that of EOT with large number of iterations. Therefore, our proposed MaxOT algorithm is more efficient.
4.4 Performance under defense
We further verify the effectiveness of our e-IOO attack under the existing defense methods. To evaluate the adversarial robustness of various attacks, we use the following defense methods: Simple Random Sampling (SRS) [79], Statistical Outlier Removal (SOR) [95], DUP-Net defense [95], and IF-Defense [72]. We give the attack success rate of various attacks under each defense method in Table 3. Under simple defenses such as SRS and SOR, the attack success rate of Mesh Attack is higher than our e-ISO attack.
However, under more advanced and effective defenses such as IF-Defence, the attack success rate of our e-ISO attack is higher than all other attacks. This is because KNN attack and GeoA attack generate adversarial objects through dense local deformation, generating outliers that are easily detected by the defense methods. The Mesh Attack generates adversarial objects through large anomalous deformation, resulting in fewer local outliers that have advantages under simple statistical defenses such as SRS and SOR, but cannot effectively attack the IF-Defence with better defense performance. Our c-IOO attack produces adversarial samples without local outliers or anomalous deformations, with better attack performance, especially under IF-Defence.
4.5 Physical attack
We randomly select 100 instances in Table 2, 50 of which are generated by the MaxOT algorithm and the other 50 are the corresponding instances generated by the EOT algorithm. The selected meshes printed by the Stratsys J850 Prime 3D printer and scanned by the EinScan-SE 3D scanner. The attack success rate in the physical world is shown in Table 4, where our algorithm almost completely maintains the effect of the pseudo physical world attack. None of the previous algorithms could maintain the pscudo physical world attack well in the physical world because the dense deformation and self-intersection phenomena would interfere with the scanning and prevent the scanner from correctly scanning the surface of the adversarial samples generated by their algorithms. In contrast, the surface of the adversarial sample produced by our c-ISO attack smooth and natural and can be scanned correctly by the scanner, preserving the adversarial effect. In addition, the improvement on the attack success rate of our proposed MaxOT algorithm against the EOT algorithm also reflected in the physical world. Our proposed MaxOT algorithm is necessary to improve the robustness of the adversarial sample under the transformation of the physical world. The adversarial meshes, 3D printed physical meshes, and scanned point clouds are shown in Fig. 5.
8

---

Advenarial Mesh
Physical Mesh
Seanned Point Cloud
Prodiction
bed - stool
chair
table piano
bookshelf
curtain
bent vase
table - bench
Figure S: Visualization of the e-ISO attack in the physical world. The adversarial mesh is randomly selected from Table 2. After 3D-printing and scanning. the point cloud obtained from the scanning used attack PointNet. A green check mark indicates successful attack. The adversarial samples we generated are natural and stealthy, and because of their smoothness and naturalness, can be scanned correctly by the 3D-scanner, maintaining the adversarial effect in the physical world.
100
0.016
PotntNet PointNete+
dissance
Atnek Succes Rate (9)
0.012
DOCNN
80
Chamter disnnce
4.5
0.01
0.000
3.5
60
0.004
0.002
40
1.5
10 20
0
0.1
10
20
Ag
Figure 6: The effects the penalty parameter A of the Gaussian curvature consistency regularization term. When Aı is large, the attack success rate decreases rapidly. When Aı is small, the naturalness of the adversarial objects is worse. The default Å1 gives the best result for balancing the attack success rate and stealthiness.
4.6 Ablation studies
To investigate the effect the penalty parameter X of the Gaussian curvature consistency regulariza- tion term on the attack success rate and imperceptibilityy metrics, we adjust the value of X to perform quantitative and qualitative experiments. The plots in Fig. show that when A₁ tuned high, the attack success rate decreases rapidly. When A₁ is adjusted lower, the values of the evaluation metrics increase and deteriorate rapidly. The qualitative visualization results Appendix C demonstrate the irregularity of the 3D objects A is turned down. Our default value of À gives the best result for balancing the attack success rate and stealthiness. To investigate the effectiveness of Bayesian optimization find the initial transformations, the initial transformation in the MaxOT algorithm. As shown in Appendix C, the improvement of random initialization in the MaxOT algorithm significant. This
test the attack success rate for randomly selecting
using Bayesian optimization confirms the importance of Bayesian optimization methods in our approach. More ablation studies can be found in Appendix C.
Conclusion
In this paper, propose novel e-isometric (s-ISO) attack method to generate natural and robust 3D adversarial examples in the physical world. We improve the naturalness of 3D adversarial examples by constraining it to be e-isometric to the original one. Through theoretical analysis, adopt

---

the Gaussian curvature as the surrogate metric. We further propose maxima over transformation (MaxOT) algorithm armed with Bayesian optimization that actively searches for the most harmful transformations rather than random ones to make the generated adversarial example more robust in the physical world. Extensive Experiments on typical point cloud recognition models validate the effectiveness of our approach in terms of attack success rate and naturalness compared to the state-of-the-ar methods. A potential negative societal impact of our work in that the malicious adversaries may adopt our method to generate 3D adversarial objects in the physical world, which can cause severe security/safety consequences for real-world applications. Thus is imperative to develop more robust 3D recognition models, which we leave to future work.
Acknowledgement
Yibo Miao and Xiao-Shia Shan Gao were supported by NKRDP grant No. N0.2018YFA0704700 and NSFC grant No. N.....288800 12288201. Yinpeng Dong and Jun Zhu were supported by the National Key Research and Development Program of China (2000AAA0106000, Projects (Nos. 62061 12061136000 136001, 62076145, 62076147, U19B2034, U181 U1811461, 1461, U19A2081, 61972224), Beijing NSF Project (No. JQ19016), BNRist (BNR2022RC01006) Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. Y. Dong was also supported by the China National Postdoctoral Program for Innovative Talents and Shuimu Tsinghua Scholar Program. J. Zhu was also supported by the XPlorer Prize.
0220AAA1043044 2020AAA0106302), NSFC
References
[1] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International conference on machine learning, pages 284-293. PMLR, 2018
[2] Matan Atzmon, Haggai Maron, and Yaron Lipman. Point convolutional neural networks by extension operators. arXiv preprint arXiv:1803. 2018.
[3] Yulong Cao. Chaowei Xiao, Dawei Yang. Jing Fang, Ruigang Yang, Mingyan Liu, and Bo Li. Adversarial objects against lidar-based autonomous driving systems. arXiv preprint arXiv: 1xXv:/907.00418 2019.
[4] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pages 39-57. IEEE, 2017.
[5] Xiaozhi Chen. Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1915- 2017.
[6] Wenda Chu, Linyi Li, and Bo Li. Tpe: Transformation-specffic smoothing for point cloud models. arXiv preprint arXiv:2201.12333 2022.
[7] David Sohen-Steinee and Marne Morvan. Restricted delaunay triangulations and normal cycle. In Proceedings of the nineteenth annual symposium on Computational geometry, pages 312 3211, 2003.
[8] Manfredo P Do Carmo. Differential geometry of curves and surfaces: revised and updated second edition. Courier Dover Publications, 2016.
[9] Xiaoyi Dong. Dongdong Chen, Hang Zhou, Gang Hua, Weiming Zhang, and Nenghai Yu. Self-robust 3d point recognition via vether- guidance. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1511-115211 IEFF, 2020.
[10] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[11] Yinpeng Dong, Zhijic Deng, Tianyu Pang, Jun Zhu, and Hang Su. Adversarial distributional training for robust deep learning. Advances in Neural Information Processing Systems, 33:8270 8283, 2020.
[12] Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang, Zhijie Deng, Hang Su, and Jun Zhu. Exploring memorization in adversarial training. arXiv preprint arXiv:2106001600 2021.
[13] Yueqi Duan, Yu Zheng. Jiwen Lu, Jie Zhou, and Qi Tian, Structural relational reasoning of point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 949 958, 2019.
10

---

[14] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. In International Conference Machine Learning, pages 1802-1811, 2019.
[15] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 613,,11,, 2017.
[16] David A Field. Laplacian smoothing and delaunay triangulations. Communications in applied numerical methods, 4(6):709-1) 1988.
[17] Peter I Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv: 1,07.0281,
[18] Xiang Gao, Wei Hu, and Guo-Jun Qi. Graphter: Unsupervised learning of graph transformation equivariant representations via auto-cncoding node-wise transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7163-7172, 2020.
[19] Justin Gilmer. Ryan P Adams, lan Goodfellow, David Andersen, and George Dahl. Motivating the rules of the game for adversarial example research. arXiv preprint arXiv: 2018.
[20] lan Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and hamesssing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
[21] Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Deng. Revisiting point cloud shape classification with a simple and effective baseline. In International Conference on Machine Learning, pages 1809-3820. PMLR, 2021.
[22] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and puttern recugnition, pages 9224-9233 2018.
[23] Groueix, Fisher, VG Kim, BC Russell, and M Aubry. Atlasnet: A papier-máché approach to learning 3d surface generation, arxiv 2018. arXiv preprint arXiv: 1802.05384, 1802.
[24] Abdullah Hamdi, Sara Rojas, Ali Thabet, and Bernard Ghanem. Advpe: Transferable adversarial per- turbations on 3d point clouds. In European Conference on Computer Vision, pages 241..57. Springer, 2020.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedingy of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[26] Qingyong Hu, Bo Yang. Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Learning semantic segmentation large-scale point clouds with random sampling. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[27] Gao Huang, Zhuang Liu. Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4700777 4708, 2017.
[28] Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 801 8-8025, 2020.
[29] Donald Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-495 1998.
[30] Jaeyeon Kim, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Minimal adversarial examples deep learning on 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7797-7806, 2021.
[31] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6981) 2014.
[32] Igor Kononenko. Bayesian neural networks. Biological Cybermetics, 61(5):361 370, 1989.
[33] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2012.
[34] Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In International Conference Learning Representations (ICLR) Workshops, 2017.
11

---

[35] Itai Lang. Uriel Kotlicki, and Shai Avidan. Geometric adversarial attacks and defenses on 3d point clouds. In 2021 International Conference on 3D Vision (3DV), pages (196-1205. IEEE, 2021.
[36] Kibok Lee, Zhuoyuan Chen. Xinchen Yan, Raquel Urtasun, and Ersin Yumer. Shapeadv: Generating shape-aware adversarial 3d point clouds. arXiv preprint urXiv:2005.11626 2020.
[37] Huan Lei. Naveed Akhtar, and Ajmal Mian. Spherical kernel for efficient graph convolution on 3d point clouds. IEEE transactions pattern analysis and machine intelligence, 43(10):3664 3680, 2020.
[38] Xinke Li, Zhirui Chen, Yue Zhao, Zekun Tong. Yabang Zhao, Andrew Lim, and Joey Tianyi Zhou, Pointba: Towards backdoor attacks in 3d point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16492-16501, 2021.
[39] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on X- transformed points. Advances in neural information processing systems, 31, 2018.
[40] Daizong Liu and Wei Hu. Imperceptible transfer attack and defense 3d point cloud classification. arXiv preprint arXiv:2111. 10990, 2021.
[41] Daizong Liu, Wei Hu, and Xin Li. Point cloud attacks in graph spectral domain: When 3d geometry meets graph signal processing. arXiv preprint arXiv:2207. 13326, 2022.
[42] Daniel Liu, Ronald Yu, and Hao Su. Extending adversarial attacks and defenses to deep 3d point cloud classifiers. In 2019 IEEE International Conference on Image Processing (ICIP), pages 2279-2283. IEEE, 2019.
[43] Yongcheng Liu, Bin Fan, Gaofeng Meng, Jiwen Lu, Shiming Xiang, and Chunhong Pan. Densepoint: Learning densely contextual representation for efficient point cloud processing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5239-5248, 2019.
[44] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1995-8904, 2019.
[45] Chengeheng Ma. Weiliang Meng, Baoyuan Wu, Shibiao Xu, and Xiaopeng Zhang. Efficient joint gradient based attack against sor defense for 3d point cloud classification. In Proceedings of the 28th ACM International Conference Multimedia, pages 1819-1822 2020.
[46] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference Learning Representations (ICLR), 2018
[47] Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of bayesian methods for sceking the extremum. Towards global optimization,, 2(117-129):2, 1978.
[48] Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 652-660) 2017.
[49] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hicrarchical feature learning on point sets a metric space. Advances in neural information processing systems, 30, 2017.
[50] Yongming Rao, Jiwen La,, and Jie Zhou. Global-local bidirectional reasoning for unsupervised representa- tion learning of 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5376-5385, 2020.
[51] Carl Edward Rasmussen. Gaussian processes in machine learning. Summer school on machine learning. pages 63-71. Springer, 2003.
[52] Yiru Shen, Chen Feng, Yaoqing Yang, and Dong Tian. Mining point cloud local structures by kernel correlation and graph pooling. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4548 4557, 2018
[53] Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In Proceedings of the IEEE conference on computer vision and puttern recognition. pages -702, 2017.
[54] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
12

---

[55] Hang Su, Subhransu Maji. Evangelos Kalogerakis, and Erik Learned-Milles Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference compuler vision, pages 945-953, 2015.
[56] Jiachen Sun, Yulong Cao, Qi Alfred Chen, and 7. Morley Mao. Towards robust (LiDAR-hased) perception in autonomous driving: General black-box adversarial sensor attack and countermeasures. In 29th USENIX Security Symposium (USENIX Security 20), pages 877-894, 2020.
[57] Jiachen Sun, Karl Koenig, Yulong Cao, Qi Alfred Chen, and Z Morley Mao. On adversarial robustness of 3d point cloud classification under adaptive attacks. arXiv preprint arXi::201 1.19922, 2020.
[58] Jiachen Sun, Yulong Cao, Christopher B Choy. Zhiding Yu, Anima Anandkumar, Zhuoqing Morley Mao, and Chaowei Xiao. Adversarially robust 3d point cloud recognition using self-supervisiios. Advances in Neural Information Processing Systems, 14:41549815115 2021.
[59] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 131.6193,,,, 2013.
[60] Abdel Aziz Taha and Allan Hanbury. Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC medical imaging, 15(1):1-28, 2015.
[61] Gusi Te, Wei Hu, Amin Zheng, and Zongming Guo. Rgenn: Regularized graph enn for point cloud segmentation. In Proceedings of the 26th ACM international conference on Multimedia, pages 746-754. 2018.
[62] Hugues Thomas, Charles R Qi, Jean-EEmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6411-6420, 2019.
[63] Tzungyu Tsai, Kaichen Yang, Tsung-Yi Yi Ho, and Yier Jin, Robust adversarial objects against deep learning models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 954-962, 2020.
[64] James Tu, Mengye Ren, Sivabalan Manivasagam, Ming Liang, Bin Yang. Richard Du, Frank Cheng, and Raquel Urtasun. Physically realizable adversarial examples for lidar object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13716-13725, 2020.
[65] Jacob Varley, Chad DeChant, Adam Richardson, Joaquín Ruales, and Peter Allen. Shape completion enabled robotic grasping. In 2017 IEEE/RSJ international conference on intelligent robuts and systems (IROS), pages 2442-2447. IEEE, 2017.
[66] Nanyang Wang. Yinda Zhang. Zhuwen Li. Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgh images. In Proceedings of the European conference on computer vision (ECCV), pages 52-67, 2018
[67] Weiyue Wang, Ronald Yu, Qiangui Huang, and UIrich Neumann. Sgpn: Similarity group proposal network for 3d point cloud instance segmentation. In Proceedingy of the IEEE conference on computer vision and pattern recognition, pages 2569 2578, 2018
[68] Yue Wang, Yongbin Sun, Ziwci Liu, Sanjay E. Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. Acm Transactions On Graphics (tog), 38 2019.
[69] Yuxin Wen, Jiehong Lin, Ke Chen, CL Philip Chen, and Kui Jia. Geometry-aware generation of adversarial point clouds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[70] Matthew Wieker and Marta Kwiatkowska. Robustness of 3d deep learning in adversarial setting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11767- 11775, 2019.
[71] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 19200, 2015.
[72] Ziyi Wu, Yueqi Duan, He Wang. Qingnan Fan, and Leonidas J Guibas. If-defense: 3d adversarial point cloud defense via implicit function based restoration. arXiv preprint arXiv:2010.05272, 2020.
[73] Chong Xiang, Charles R Qi, and Bo Li. Generating 3d adversarial point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 9136 9144, 2019.
13

---

[74] Tiange Xiang. Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. Walk in the cloud: Learning curves for point clouds shape analysis. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 915-924, 2021.
[75] Zhen Xiang. David Miller, Siheng Chen, Li, and George Kesidis. A backdoor attack against 3d point cloud classifiers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7597-7607, 2021.
[76] Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. Paconv; Position adaptive convolution with dynamic kernel assembling on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 3173-3182, 2021.
[77] Qiangeng Xu, Xudong Sun, Cho- Wu, Panqu Wang, and Urrich Neumann. Grid-gen for fast and scalable point cloud learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5661-5670, 2020.
[78] Bo Yang. Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, and Niki Trigoni. Learning object bounding boxes for 3d instance segmentation point clouds. Advances neural information processing systems, 32, 2019.
[79] Jiancheng Yang, Qiang Zhang, Rongyao Fang, Bingbing Ni, Jinxian Liu, and Qi Tian. Adversarial attack and defense on point sets. arXiv preprint 1rXiv:1902.108990 2019.
[80] Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point clouds with self-attention and gumbel subset sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 3323-3332, 2019.
[81] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 206-215, 2018.
[82] Lijia Yu, Yihan Wang, and Xiao-Shan Gao. Adversarial parameter attack on deep neural networks. arXiv preprint arXi::2203. 10502, 2022.
[83] Tan Yu, Jingjing Meng, and Junsong Yuan. Multi-view harmonized bilinear network for 3d object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 186-194, 2018.
[84] Xuejing Yuan, Yuxuan Chen, Yue Zhao, Yunhui Long, Xiaokang Liu, Kai Chen, Shengzhi Zhang, Heqing Huang, Xiaofeng Wang, and Carl A Gunter. /CommanderSong]: A systematic approach for practical adversarial voice recognition. In 27th USENIX security symposium (USENIX security 18), pages 49-64, 2018
[85] Xiangyu Yue, Bichen Wu, Sanjit Seshia, Kurt Keutzer, and Alberto I. Sangiovanni-Vincettlli A lidar point cloud generator: from a virtual world to autonomous driving. Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval, pages 458 464, 2018.
[86] Manzil Zaheor, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[87] Maciej Zamorski, Maciej Zigha, Piotr Klukowski, Rafat Nowak, Karol Kurach, Wojeiech Stokowiee, and Tomasz Trzciáski. Adversarial autoencoders for compact representations of 3d point clouds. Computer Vision and Image Understanding, 193:102921, 2020.
[88] Jinlai Zhang, Lyujic Chen, Binbin Liu, Bo Ouyang, Qizhi Xic, Jihong Zhu, Weiming Li, and Yanmei Meng. 3d adversarial attacks beyond point cloud. arXiv preprint arXiv:2104. 12146, 2021.
[89] Jinlai Zhang, Lyujie Chen, Bo Ouyang, Binbin Liu, Jihong Zhu, Yujin Chen, Yanmei Meng, and Danfeng Wu. Pointcutmix: Regularization strategy for point cloud classification. Neurocomputing, 505:58-67, 2022.
190] Jinlai Zhang, Weiming Li, Shuang Liang, Hao Wang, and Jihong Zhu. Adversarial samples for deep monocular 6d object pose estimation. arXiv preprint arXiv:2203.00. 00302, 2022.
[91] Yu Zhang, Gongbo Liang, Tawfiq Salem, and Nathan Jacobs. Defense- petense-pointet: Protecting pointnet against adversarial attacks. In 2019 IEEE International Conference on Big Data (Big Data), pages 5654 5660. IEEE, 2019.
14

---

[92] Yue Zhao, Yuwei Wu, Caihua Chen, and Andrew Lim. On isometry robustness of deep 3d point cloud models under adversarial attacks. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 1201-1210, 2020.
[93] Tianhang Zheng. Changyou Chen, Junsong Yuan. Bo Li, and Kui Ren. Pointcloud saliency maps. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 1998,1606, 2019.
[94] Boxuan Zhong, He Huang, and Edgar Lobaton. Reliable vision-based grasping target recognition for upper limb prostheses. IEEE Transactions on Cybernetics, 2020.
[95] Hang Zhou, Kejiang Chen, Weiming Zhang, Han Fang. Wenbo Zhou, and Nenghai Yu. Dup-net: De- noiser and upsampler network adversarial point clouds defense. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1961-1970, 2019.
[96] Hang Zhou, Dongdong Chen, Jing Liao, Kejiang Chen, Xiaoyi Dong. Kunlin Liu, Weiming Zhang, Gang Hua. and Nenghai Yu. Lg-gan: Label guided adversarial network for flexible targeted attack of point cloud based deep networks, In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10356-10365, 2020.
[97] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong. Wei Li, Yuexin Ma, Hongsheng Li, Ruigang Yang. and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar-based perception. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We do not develop more robust
3D models this work, which we leave future work.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [Yes] The code is included the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4.1.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "z5WH6sG7m0t", "reviewer": "Reviewer_mBRv", "summary": "The paper proposes a white-box adversarial attack on 3D point cloud recognition models such that the corresponding attacks are robust, natural, and reproducible in the physical world. For naturalness, they propose a novel framework based on maintaining $\\epsilon$-isometry between the original and adversarial point cloud model. To boost robustness and ensure reproducibility, they develop a new framework combining maxima over transformation with bayesian optimization. Experimental evaluation against existing attacks on state-of-the-art models shows that the proposed approach achieves higher attack success rate while remaining imperceptible. The authors further show reproducibility in the real world.", "questions": "* In Section 3.1, $\\mathcal{X}$ is undefined. I assume it's the set of all meshes or point clouds? Line 101, how is $\\Delta$ measured? Is the sampling function $\\mathcal{S}$ fixed for a given mesh $\\mathcal{M}$? Can you provide a mathematical definition of the length function \"s\" used in Definition 1? How do you get the equality on line 145? \n\n* Line 132, \"Therefore, we introduce the definition of the Gaussian curvature [11] of the surface\", is it your definition or taken from literature? Please be precise with the wording.\n\n* In Definition 1, what is the range of values of n? For arbitrarily n, it looks like the difference in the lengths of curves on $S$ and $\\hat{S}$ can be quite large?\n\n* In Definition 2, can you explain how the parameters $(u,v)$ relate to the meshes?\n\n*  Is Theorem 1 your contribution or adapted from prior work?\n\n* Can you compare the runtime of the different baselines and your method?\n\n* \"The Mesh Attack generates adversarial objects through large anomalous deformation, resulting in fewer local outliers that have advantages under simple statistical defenses such as SRS and SOR, but cannot effectively attack the DUP-Net defense and IF-Defence with better defense performance. \" While there is a significant improvement over the mesh attack under the IF-defence, improvement under the DUP-Net defense is marginal (<0.5%), so it would be misleading to claim that mesh attack is ineffective and your attack is effective in this case.\n\n* Can your MaxOT + Bayesian optimization provide any formal guarantees on the robustness of your examples (e.g., as in https://arxiv.org/pdf/2007.12133.pdf)? Further, does your robustness enable you to be more effective against certified defenses based on randomized smoothing (e.g., https://arxiv.org/abs/2103.03046)?\n\n* Why did you not analyze the effects of the hyperparameters $\\lambda_2$ and $\\lambda_3$?\n\n* How do $\\mathcal{D}_c$ and $\\mathcal{D}_g$ relate to $\\epsilon$ in your $\\epsilon$-ISO attacks?  \n\n* I observed you only consider rotations when considering robustness in experiments. Any particular reason for that? Why not consider the composition of translation, rotation, other transformations, etc.?", "limitations": "The authors have adequately addressed the limitations and potential negative impacts of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper considers a novel setting where the author's goal is to generate robust and natural adversarial examples for NNs working on  3-D point clouds that can also be realized in the physical world. \n\n+ The technical contributions are novel and tailored for creating physical, natural, and robust adversarial examples in the 3-D point cloud setting. \n\n+ Strong experimental results showing the value of the contributions including reproducibility in the real world.\n\nWeaknesses:\n- The technical details in Section 3.1 are not rigorous enough which makes the corresponding text hard to follow. Further, while the high-level idea in 3.3.1 is clear, the text is too dense. I suggest the authors improve their writing at these places for making their technical contributions more accessible. \n\n-  While input-specific white-box attacks show the non-robustness of NNs, they are usually not realistic. The authors do not mention cases where an attacker can gain white-box access to the point cloud model and how it knows what inputs will be processed by the model in advance and whether it can create physical examples fast enough before the input is processed by the model. For example, in the case of a self-driving car, the model will continuously process new inputs. Is your attack feasible in such settings? Therefore I am not sure if the proposed method reveals any vulnerability that can be exploited by a real-world adversary.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MZoLgOuSGlI", "reviewer": "Reviewer_WZCq", "summary": "This paper presents $\\epsilon$-ISO  attack method to generate natural and robust 3D adversarial examples in the physical world. The authors present a new metric or definition on the naturalness of the 3D adversarial example. They also provide theoretical proof to demonstrate that Gaussian curvature could be used as a proxy for $\\epsilon$-isometric. MaxOT is proposed as a method combined with Bayesian to optimize the 3D adversarial examples.   ", "questions": "See the first point in the weakness section.", "limitations": "The authors should discuss the potential societal impact of the proposed attack.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper contains both formal and empirical results to demonstrate the effectiveness of the proposed method.\n2. The organization of the paper is good, and the content is rich.\n3. The authors leverage extensive evaluation to show that the proposed attack is effective and realistic in the real world. The experiments with 3D printed mesh are convincing. \n\nWeakness:\n1. Although the authors successfully demonstrate the physical attack on 3D point cloud recognition task, it is still questionable how these attacks will have real-world impacts. Real-world application of point clouds mostly include object detection and segmentation. Delving into the recognition task and showing naturalness seem to be off the correct direction in this area.\n2. I like the organization of this paper, but the content is too crowded and sometimes it is hard to follow every detail.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M-9aYjY2I1Y", "reviewer": "Reviewer_FQV5", "summary": "This paper proposes a novel $\\epsilon$-ISO attack algorithm to generate natural and robust 3D adversarial samples. It is motivated by the fact that the distance metric on the Euclidean space does not consider geometric features and cannot accurately measure the similarity between 3D objects. The proposed method constrains the adversarial sample to be $\\epsilon$-isometric to the original sample to ensure the geometric similarity and adopts the Gaussian curvature as the regularization. The theoretical analysis is provided to uncover the connection between isometry and Gaussian curvature. The authors further propose MaxOT to improve the robustness of 3D adversarial samples in the physical world. MaxOT is reasonable by actively searching for the most harmful transformations during optimization. The experiments validate the naturalness and effectiveness of the proposed 3D attack.", "questions": "### Other comments:\nI was inspired by the authors’ discussion of metrics on Non-euclidean space. As analyzed by the authors, the distance metric on Euclidean space is not a perfect representation of the similarity between 3D objects because it does not take geometric features into account. While reading, I kept thinking if the Fisher information metric on statistical manifold mapped from topological manifold [1] would be an interesting consideration.\n\n[1] F. Li et al, \"Toward a unified framework for point set registration,\" ICRA 2021\n\n\nOverall, I think the strengths of this paper overweigh its weaknesses. The contributions of this paper could be significant for the venue. I’d like to see my concerns addressed by the authors and consider increasing the rating.\n\n### Question:\n- In fig4, the title of the y-axis should be the attack success rate.\n- Line 16 of the appendix: The authors should further clarify the relationship between Gaussian curvature and the first fundamental form in the proof in the appendix to ensure that it can be clearly understood.\n- Line 142: \"bending degree\" should be \"the bending degree\".\n- Line 148: \"bending degree\" should be \"the bending degree\".\n- Line 149: \"Definition\" should be \"Definitions\".\n- Line 262: \"higher\" should be \"a higher\".\n- table 3: \"DUP-Net\" should be \"DUP-Net defense\".\n- Line 281: \"selecte\" should be \"select\".\n- In fig6(a), the title of the y-axis should be the attack success rate.", "limitations": "/na", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- First, the paper is easy to read. The algorithm and experiments are well organized and explained. The experiments are thorough to prove the effectiveness.\n- Overall, I find the proposed approach novel. To the best of my knowledge, previous 3D adversarial attacks only considered the distance metric on Euclidean space, i.e. Hausdorff distance, Chamfer distance, etc., whereas this is the first work to consider the geometric properties on Non-euclidean space of 3D objects. This paper introduces $\\epsilon$-isometric mapping to ensure the geometric similarity between the adversarial sample and the original one. It further connects $\\epsilon$-isometry to the Gaussian curvature, which can be computed and optimized in generating 3D adversarial samples. Furthermore, the MaxOT algorithm is reasonable to improve the robustness of 3D adversarial samples.\n- The theoretical analysis makes this paper more solid and convincing.\n- The paper has conducted thorough evaluation comparisons between the proposed attack and other attacks. The performance improvement is obvious. The 3D adversarial objects look more natural according to the visualization. The effectiveness measured by the attack success rate is also validated.\n\nWeaknesses:\n- Table 2 and Figure 4 demonstrate that the proposed MaxOT outperforms EOT in terms of attack success rate and efficiency in the physical world. Is it due to the 3D objects generated by MaxOT are more perceptible?\n- To further verify the effectiveness of the proposed method, the authors are encouraged to conduct experiments of black box attacks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept