Geometry-aware Generation of Adversarial and Cooperative Point Clouds

1 INTRODUCTION . The existence of adversarial examples shows the vulnerability of machine learning models , and triggers a great amount of research attention paid to either attacking and defense studies for safetycritical issues , or robustness analysis on machine learning models themselves . In existing literature , adversarial examples generally satisfy two key properties : visually imperceptible by humans and capable to mislead machine learning models . For 2D adversarial images ( Szegedy et al. , 2013 ; Carlini & Wagner , 2017 ; Madry et al. , 2017 ) , their imperceptible nature indicates less sensitivity by humans to the texture changes of the original images , which can be viewed as noises with small magnitudes anchored on RGB values of pixels . Different from texture changes in adversarial images , adversarial point clouds as approximate shape representations of object surface demand imperceptible shape deformations . Heuristic manners of existing several works are taken to allow shape deformations in arbitrary directions for adversarial attack effects . Consequently , they produce adversarial examples that contain obvious point outliers ( Xiang et al. , 2019 ; Liu et al. , 2019a ) , which can thus be easily defended by simple outlier removal method such as SOR ( Zhou et al. , 2018 ) . In this work , we aim to adversarially perturbing benign point clouds with awareness of the underlying geometric properties , which can be generally described as some sort of approximate geometric smoothness , e.g. , point-wise normals ( Wang et al. , 2018 ) , curvatures ( Bae , 2006 ) , or geodesics for pairs of points ( He et al. , 2019 ) , in view of the nature of object surface as 2-manifold embedded in the 3D space . To this end , we propose a novel geometry-aware loss in combination with the attack one . Our proposed geometry-aware loss is composed of three terms : the Chamfer Distance ( CD ) and Hausdorff Distance ( HD ) terms prevent geometric/topological changes of global shape via constraining point-wise perturbations , while the term encouraging consistency of local curvatures between the adversarial and benign point clouds achieves smooth local surface deformations . Extensive experiments on the ModelNet40 dataset ( Wu et al. , 2015 ) can verify the imperceptibility and effectiveness of geometry-aware adversaries . Our success in generating less noticeable point clouds inspires us to think of an opposite direction : is it possible to perturb raw point clouds in a cooperative , geometry-aware manner such that the resulting imperceptible perturbations of point clouds can improve classification performance with existing machine learning models ? To the best of our knowledge , very few existing works are pursuing this direction , at least in the domain of 3D shape analysis . Technically , the objective of cooperative generation on point clouds is achieved by learning point perturbations in favor of either improved classification confidence or corrected class predictions . In view of their geometryaware nature , our cooperative point clouds could be practically meaningful in terms of guiding design of certain objects such that they can be better perceived by in-built machine learning systems in physical world . Experiments on the ModelNet40 again verify our motivation on cooperative generation of point clouds . 2 RELATED WORKS . A number of adversarial attack algorithms have been proposed on 3D semantic analysis ( Xiao et al. , 2019 ; Liu et al. , 2019a ; Xiang et al. , 2019 ; Yang et al. , 2019 ; Zheng et al. , 2018 ; Wicker & Kwiatkowska , 2019 ; Liu et al. , 2019b ) . Beyond ( Xiao et al. , 2019 ) manipulating both texture and shape on the mesh level to attack image classifiers and detectors , most of adversarial algorithms on point clouds attack 3D classifiers via points attachment ( Xiang et al. , 2019 ; Yang et al. , 2019 ) , point detachment ( Yang et al. , 2019 ; Zheng et al. , 2018 ; Wicker & Kwiatkowska , 2019 ) or point-wise perturbation ( Xiang et al. , 2019 ; Liu et al. , 2019a ; Yang et al. , 2019 ; Liu et al. , 2019b ) . Existing methods fail to exploit geometric properties during adversarial example generation , and thus have evident outliers and can be easily defended by SOR defense ( Zhou et al. , 2018 ) , which encourage our adversaries ( see Figure 2 and Table 1 ) . A number of deep models have been proposed for point-based surface reconstruction such as Point Set Generation ( PSG ) ( Fan et al. , 2017 ) , the AtlasNet ( Groueix et al. , 2018b ) , and Deep Cascade Generation ( DCG ) ( Wang et al. , 2019a ) from 2D images . These methods exploiting local mesh structure into point clouds generation have inspired our geometry-aware concept , but they are directly regressing from the latent vector encoded from images rather than point-wise deformations in the proposed method . Data argumentation on point clouds such as jittering , rotation and random scale firstly introduced in ( Qi et al. , 2017a ) shares similar concept as our cooperative deformations , both of which generate new data to boost classification . However , the key differences between our cooperative learning and data argumentation lie in two folds : 1 ) optimization based vs. simple physical processing ; and 2 ) deformations on the underlying shape vs. point density changes . Simply put , the proposed cooperative deformation can be a promising learning paradigm , while data argumentation is an effective pre-processing step to mitigate sparse sample distributions . 3 METHODOLOGY . Our problem setting assumes the availability of a collection of point clouds in the input space X , which are approximate shape representations of object surfaces of certain categories . Each point cloud P ∈ X contains an orderless set of n points { pi } ni=1 , with the corresponding label y ∈ Y of object category , where any p denotes the coordinates ( x , y , z ) in the 3D Euclidean space . In this work , we focus on machine learning models of 3D point set classification ( Qi et al. , 2017a ; b ; Wang et al. , 2019b ) , which learn a classifier f : X → Y and expect f ( P ) = y for any input P with the true label y. P is a discrete approximation of an object surface that satisfies the general properties of smoothness and fairness ( Botsch et al. , 2010 ) , which concern with the continuity and variation of ( partial ) derivatives of a parametric surface function ; or in a more intuitive way , the properties concern with the curvatures of local surface patches . Our objective in this work is to obtain from P a deformed point cloud P ′ via learning to perturb individual points { pi } ni=1 of P . Since the deformation is expected to be imperceptible by humans , we argue that P ′ should have the aforementioned surface properties with similar degrees of smoothness/fairness to local surface patches of P , while being globally close to P as well , measured by certain distance metrics of point sets ; otherwise , humans would notice either the global , possibly topological changes of part configuration of the object surface , or those of local surface details . We present shortly in section 3.1 our technical solutions to the above objective of geometry-aware point perturbations , and discuss how such solutions can be used to generate either adversarial or cooperative point clouds in sections 3.2 and 3.3 respectively . 3.1 GEOMETRY-AWARE POINT PERTURBATIONS . In 2D image domain , pixel-wise lp-norms are usually used to constrain the noise magnitudes of adversarial examples ( Carlini & Wagner , 2017 ; Madry et al. , 2017 ) . Due to the sharply different data nature of point clouds , we consider the following learning criteria to perturb points { pi } ni=1 of P in a geometry-aware manner . These criteria are to constrain the deformation magnitudes of the resulting P ′ , while taking into account prevention of point outliers and smooth regularization of local point neighborhoods . Combined use of these criteria leads to point cloud deformations that are less noticeable by humans , as verified in the comparative experiments in section 4 . Chamfer Distance Given two point sets P and P ′ , the Chamfer distance computes CChamfer ( P ′ , P ) = 1 n ∑ p′∈P′ min p∈P ‖p′ − p‖22 + 1 n ∑ p∈P min p′∈P′ ‖p− p′‖22 , ( 1 ) which shows that Chamfer distance is symmetric w.r.t . P andP ′ . Although the Chamfer distance ( 1 ) is not a strict distance function , since the triangle inequality does not hold , it is popularly used in the recent literature of learning based 3D shape generation ( Yang et al. , 2018 ; Groueix et al. , 2018a ; Fan et al. , 2017 ) . It measures the distance between the two point sets by averaging over the individual deviation of any p ∈ P from P ′ and that of any p′ ∈ P ′ from P . However , Chamfer distance is less effective in prevention of outlier points in P ′ , since a small portion of outliers in P ′ increases the distance ( 1 ) negligibly — one can intuitively think of outliers of a point cloud as those away from the object surface , represented by the point cloud , with relatively large distances . This shortcoming of Chamfer distance motivates us to additionally use the Hausdorff distance as introduced below . Hausdorff Distance In this work , we use a non-symmetric Hausdorff distance between the two point sets P and P ′ that computes CHausdorff ( P ′ , P ) = max p′∈P′ min p∈P ‖p′ − p‖22 , ( 2 ) since only the deformation of P ′ is concerned . As ( 2 ) indicates , the Hausdorff distance finds the largest one among the smallest distances of individual p′ ∈ P ′ from P . It is thus sensitive to generation of outliers in the resulting P ′ . Distances computed by the functions ( 1 ) and ( 2 ) rely on those between the individual points { p′i } ni=1 and { pi } ni=1 , which does not involve geometries of local surface patches associated with these individual points ; consequently , the resulting P ′ could be close to P when measured by ( 1 ) and/or ( 2 ) , but changes of geometric details at certain local patches could be clearly visible , causing failure to achieve the objective of imperceptible deformation . Consistency of Local Curvatures Our way of achieving geometry-aware imperceptibility is to constrain the point cloud deformation such that local patches of the surface approximated by the resulting P ′ have curvatures whose magnitudes are similar to those of the corresponding patches of the surface approximated by P . Since computations in this work are conducted on the discrete surface approximation of point clouds , we propose discrete notions that approximately characterize curvatures of local surface patches . More specifically , for any point p′ ∈ P ′ , we find its closest point p ∈ P by p = arg minp∈P ‖p′ − p‖2 . There exist local point neighborhoods N ′p′ ⊂ P ′ and Np ⊂ P respectively associated with p′ and p , which are obtained in this work by searching k nearest neighbors , suggesting |N ′p′ | = |Np| = k. To capture the local geometry of Np , we rely on the following discrete notion κ ( p ; P ) = 1 k ∑ q∈Np |〈 ( q − p ) /‖q − p‖2 , np〉| , ( 3 ) where np denotes the unit normal vector of the surface at p. The term ( 3 ) intuitively measures the averaged angles between the normal vector and the vector defined by pointing p towards each q of its neighboring points . Indeed , since the normal vector np is orthogonal to the tangent plane of the surface at p , each inner product in ( 3 ) characterizes how the normals vary directionally in the local neighborhood Np , thus approximately measuring the local , directional curvature , and an average of |Np| inner products in ( 3 ) approximately measures the local , mean curvature . Note that unit normal vector np in ( 3 ) can be computed from Np via eigen-decomposition of the set N ( p ) ( Hoppe et al. , 1992 ) . We compute κ′ ( p′ ; P ′ ) in the same way as ( 3 ) , with a subtle difference that instead of computing n′p′ from N ′p′ , we directly use np , i.e. , the unit normal vector of the point in P that is closest to p′ , as a surrogate of n′p′ , since normal vectors of P can be pre-computed and efficiently retrieved during the deformation learning . Given κ′ ( p′ ; P ′ , P ) and κ ( p ; P ) , we use the following criterion to encourage the consistency of local geometries between any p′ ∈ P ′ and its closest point p ∈ P CCurvature ( P ′ , P ) = 1 n ∑ p′∈P′ ‖κ′ ( p′ ; P ′ , P ) − κ ( p ; P ) ‖22 s.t . p = arg minp∈P ‖p ′ − p‖2 . ( 4 ) where we write κ′ ( p′ ; P ′ , P ) since the normal vector involved in its computation is from the corresponding one of P . Note that terms similar to ( 3 ) are also used in ( Wang et al. , 2018 ; Tang et al. , 2019 ) for single-view surface reconstruction . Our use of the term ( 3 ) in ( 4 ) is to encourage the consistency of local surface geometries between P ′ and P , rather than to directly minimize ( 3 ) as in ( Wang et al. , 2018 ; Tang et al. , 2019 ) . The Combined Geometry-aware Objective We use the following combined objective to learn the deformed P ′ by either adversarial or cooperative perturbations of individual points of P CGeometry ( P ′ , P ) = CChamfer ( P ′ , P ) + α · CHausdorff ( P ′ , P ) + β · CCurvature ( P ′ , P ) , ( 5 ) where α and β are the weighting parameters .

Summary: This paper proposes a novel loss function to account for imperceptible, geometry-aware deformations of point clouds. The loss is used in two cases: generating adversarial point clouds to attack representative models of point set classifiers, and generating cooperative point clouds to improve classification confidence or accuracy. The combined geometry-aware objective is well-introduced, which mainly contains Chamfer distance term, Hausdorff distance term and local curvatures consistency term. The authors apply the geometry-aware objective to generate adversarial point clouds by adopting the framework of C&W attack. For generating cooperative point clouds, the authors introduced a training procedure to reduce the overfitting of the deformed point clouds. Most of the experiments are well-conducted, and demonstrate the effectiveness of the proposed loss function.