Title: Unlearnable 3D Point Clouds: Class-wise Transformation Is All You Need
Venue: NeurIPS 2024
Accepted: yes
Keywords: Unlearnable examples

3D point clouds

deep neural networks
Abstract: Traditional unlearnable strategies have been proposed to prevent unauthorized users from training on the 2D image data. With more 3D point cloud data containing sensitivity information, unauthorized usage of this new type data has also become a serious concern. To address this, we propose the first integral unlearnable framework for 3D point clouds including two processes: (i) we propose an unlearnable data protection scheme, involving a class-wise setting established by a category-adaptive allocation strategy and multi-transformations assigned to samples; (ii) we propose a data restoration scheme that utilizes class-wise inverse matrix transformation, thus enabling authorized-only training for unlearnable data. This restoration process is a practical issue overlooked in most existing unlearnable literature, i.e., even authorized users struggle to gain knowledge from 3D unlearnable data. Both theoretical and empirical results (including 6 datasets, 16 models, and 2 tasks) demonstrate the effectiveness of our proposed unlearnable framework. Our code is available at https://github.com/CGCL-codes/UnlearnablePC.

Unlearnable 3D Point Clouds: Class-wise Transformation Is All You Need

Abstract

Traditional unlearnable strategies have been proposed to prevent unauthorized users from training on the 2D image data. With more 3D point cloud data containing sensitivity information, unauthorized usage of this new type data has also become a serious concern. To address this, we propose the first integral unlearnable framework for 3D point clouds including two processes: (i) we propose an unlearnable data protection scheme, involving a class-wise setting established by a categoryadaptive allocation strategy and multi-transformations assigned to samples; (ii) we propose a data restoration scheme that utilizes class-wise inverse matrix transformation, thus enabling authorized-only training for unlearnable data. This restoration process is a practical issue overlooked in most existing unlearnable literature, i.e. , even authorized users struggle to gain knowledge from 3D unlearnable data. Both theoretical and empirical results (including 6 datasets, 16 models, and 2 tasks) demonstrate the effectiveness of our proposed unlearnable framework. Our code is available at https://github.com/CGCL-codes/UnlearnablePC .

1 Introduction

Recently, 3D point cloud deep learning has been making remarkable strides in various domains, e.g. , self-driving [6] and virtual reality [1, 46]. Specifically, numerous 3D sensors scan the surrounding environment and synthesize massive 3D point cloud data containing sensitive information such as pedestrian and vehicles [32] to the cloud server for deep learning analysis [12, 23]. However, the raw point cloud data can be exploited for point cloud unauthorized deep learning if a data breach occurs, posing a significant privacy threat. Fortunately, the privacy protection approaches for preventing unauthorized training have been extensively studied in the 2D image domain [9, 19, 28, 29, 39, 42]. They apply elaborate perturbations on images such that trained networks over them exhibit extremely low generalization, thus failing to learn knowledge from the protected data, known as "making data unlearnable". Nonetheless, the stark disparity between 2D images and 3D point clouds poses significant challenges for drawing lessons from existing 2D solutions.

Specifically, migrating 2D unlearnable schemes to 3D suffers from following challenges: (i) Incompatibility with 3D data. Numerous model-agnostic 2D image unlearnable schemes operate

Original sample

Rotation

Scaling

Shear

Reflection

Translation

Transformations

None

Rotation

∗

Scaling

∗

Shear

Twisting

Tapering

Reflection

∗

Translation

∗

Illustrations of 3D

point cloud samples

Descriptions

Clean sample

Changing the

angles

Changing the

size

Stretching or

compressing

Twisting the

shape

Narrowing or

shortening

Creating a

mirror object

Changing the

position

Is it reversible?

Quantity of possible

transformations

Infinite

Infinite

Infinite

Infinite

Infinite

3

Infinite

Dimension of the

multiplicative

transformation matrix

ℝ 𝟑𝟑

×

𝟑𝟑

ℝ 𝟑𝟑

×

𝟑𝟑

ℝ 𝟑𝟑

×

𝟑𝟑

ℝ 𝟑𝟑

×

𝟑𝟑

ℝ 𝟑𝟑

×

𝟑𝟑

ℝ 𝟑𝟑

×

𝟑𝟑

ℝ 𝟒𝟒

×

𝟒𝟒

in the pixel space, such as convolutional operations [39, 42], making them fail to be directly transferred to the 3D point space. (ii) Poor visual quality. Migrating model-dependent 2D unlearnable methods [5, 9, 19, 28] to 3D point clouds requires perturbing substantial points, leading to irregular three-dimensional shifts which may significantly degrade visual quality. Hence, these challenges spur us to start directly from the characteristics of point clouds for proposing 3D unlearnable solutions.

Recent works observe that 3D transformations can alter test-time results of models [7, 17, 44]. To explore this, we conduct an in-depth investigation into the properties of seven 3D transformations as shown in Fig. 1 and reveal the mechanisms by which transformations employed in a certain pattern serve as unlearnable schemes (Sec. 3.2). In light of this, we propose the first unlearnable approach in 3D point clouds via multi class-wise transformation (UMT), transforming samples to various forms for privacy protection. Concretely, we newly propose a category-adaptive allocation strategy by leveraging uniform distribution sampling and category constraints to establish a class-wise setting, thereby multiplying multi-transformations to samples based on categories. To theoretically analyze UMT, we define a binary classification setup similar to that used in [20, 33, 39]. Meanwhile, we employ a Gaussian Mixture Model (GMM) [38] to model the clean training set and use the Bayesian optimal decision boundary to model the point cloud classifier. Theoretically, we prove that there exists a UMT training set follows a GMM distribution and the classification accuracy of UMT dataset is lower than that of the clean dataset in a Bayesian classifier.

Moreover, an incompatible issue in existing unlearnable works [9, 19, 28, 29, 39, 43] is identified [54], i.e. , these approaches prevent unauthorized learning to protected data, but they also impede authorized users from effectively learning from unlearnable data. To address this, we propose a data restoration scheme that applies class-wise inverse transformations, determined by a lightweight message received from the protector. Our proposed unlearnable framework including UMT approach and data restoration scheme is depicted in Fig. 3.

Extensive experiments on 6 benchmark datasets (including synthetic and real-world datasets) using 16 point cloud models across CNN, MLP, Graph-based Network, and Transformer on two tasks (classification and semantic segmentation), verified the effectiveness of our proposed unlearnable scheme. We summarize our main contributions as follows:

The First Integral 3D Unlearnable Framework. To the best of our knowledge, we propose the first integral unlearnable 3D point cloud framework, utilizing class-wise multitransformation as its unlearnable mechanism (effectively safeguarding point cloud data against unauthorized exploitation) and proposing a novel data restoration approach that leverages class-wise reversible 3D transformation matrices (addressing an incompatible issue in most existing unlearnable works, where even authorized users cannot effectively learn knowledge from unlearnable data).

Theoretical Analysis. We theoretically indicate the existence of an unlearnable situation that the classification accuracy of the UMT dataset is lower than that of the clean dataset under the decision boundary of the Bayes classifier in Gaussian Mixture Model.

Experimental Evaluation. Extensive experiments on 3 synthetic datasets and 3 real-world datasets using 16 widely used point cloud model architectures on classification and semantic segmentation tasks verify the superiority of our proposed schemes.

2 Preliminaries

Notation. Considering the raw point cloud data ( X , Y ) ∈ X × Y sampled from a clean distribution D for training a point cloud network, the user's goal is to obtain a model F : X → Y by minimizing the loss function ( e.g. , cross-entropy loss) L ( F ( X ) , Y ) . Let T ∈ T be a 3D transformation matrix that does not seriously damage the visual quality of point clouds. Note that X ∈ R 3 × p , T ∈ R 3 × 3 , and p represents the number of points. In theoretical analysis, following [20, 33], we simplify a training dataset D k to a Gaussian Mixture Model (GMM) [38] N ( yµ, I ) , where y ∈ {± 1 } denotes the class labels, µ ∈ R d denotes the mean value, and I ∈ R d × d denotes the identity matrix. Thus the Bayes optimal decision boundary for classifying D k is defined by P ( x ) ≡ µ T x = 0 . The accuracy of the decision boundary P on D k is equal to ϕ ( || µ || 2 ) , where ϕ denotes the Cumulative Distribution Function (CDF) of the standard normal distribution.

Data protector G p . G p aims to protect the knowledge from the clean training set (with size of n ) D c = { X i , Y i } n i =1 ∼ D by compromising the unauthorized models who train on the unlearnable point cloud data { T i ( X i ) , Y i } n i =1 , resulting in extremely poor generalization on the clean test distribution D t ⊆ D . This objective can be formalized as:

where G p assumes that training samples are all transformed into unlearnable ones while maintaining normal visual effects, in line with previous unlearnable works [19, 28, 39, 42]. By the way, solving Eq. (1) directly is infeasible for neural networks because it necessitates unrolling the entire training procedure within the inner objective and performing backpropagation through it to execute a single step of gradient descent on the outer objective [8].

Authorized user G a . G a aims to apply another transformation T ′ ∈ T on the unlearnable sample, making the protected data learnable. This is formally defined as:

where G a assumes that, without access to any clean training samples, T ′ can be constructed by utilizing a lightweight message M received from data protectors.

3 Our Proposed Unlearnable Schemes

3.1 Key Intuition

Several recent works [7, 13, 44] reveal employing 3D transformations can mislead the model's classification results. Such a phenomenon implies that there might be some defects in point cloud classifiers when processing transformed samples, leading us to infer that 3D transformations are probable candidates for data protection against unauthorized training. If the transformed point cloud data are used to train unauthorized DNNs, only simple linear features inherent in 3D transformations (at which transformations may act as shortcuts [14]) are captured by the DNNs, successfully protecting point cloud data privacy.

⋅⋅⋅

DNN

Shortcut

⋅⋅⋅

Applying class-wise

transformations

Establishing a new mapping

Original

samples

Transfor-

mations

'bed'

'chair'

'toilet'

⋅⋅⋅

⋅⋅⋅

truth label

(b)

(a)

3.2 Exploring the Mechanism

We summarize existing 3D transformations in Fig. 1 and formally define them in Appendix A. To seek clarity on the application and selection of transformations, we explore three aspects: (i) execution mechanism , (ii) exclusion mechanism , and (iii) working mechanism as follows.

(i) Which execution mechanism successfully satisfy Eq. (1)? The extensively employed execution patterns in 2D unlearnable approaches are sample-wise [9, 19] and class-wise [19, 39] settings. We further complement the dataset-wise setting (using universal transformation) and implement the above execution mechanisms for training a PointNet classifier [35] on the transformed ModelNet10 [50], obtaining test accuracy results in Fig. 2 (a). We discover that model achieves considerably low test accuracy under the class-wise setting, satisfying Eq. (1). Sample-wise and dataset-wise settings do not obviously compromise model performance, which cannot serve as promising unlearnable routes. Moreover, we note that sample-wise transformation is often considered as a data augmentation scheme to improve generalization, which contradicts our aim of using class-wise transformation to lower model generalization.

(ii) Which transformations need to be excluded? Not all transformations are suitable candidates. We exclude three transformations, tapering, reflection, and translation. (1) The tapering matrix may cause point cloud samples to become a planar projection when ηz defined in Eq. (18) equals to -1, rendering the tapered samples meaningless; (2) The reflection matrix has only three distinct transformation matrices, rendering it incapable of assigning class-wise transformations when the number of categories exceeds three; (3) The translation transformation is a straightforward and simple additive transformation that is too easily defeated by point cloud data pre-processing approaches.

(iii) Why does class-wise transformation work? We conduct experiments using class-wise transformations (see Tab. 6), indicating that the model training on the class-wise transformed training set achieves a relatively high accuracy on the class-wise transformed test set (using the same transformation process as the training set). Besides, if we permute the class-wise transformation for the test set, we obtain a significant low accuracy on the test set. Therefore, we conclude that the reason why class-wise transformation works is that the model learns the mapping between class-wise transformations and corresponding category labels as shown in Fig. 2 (b), which results in the model being unable to predict the corresponding labels on a clean test set lacking transformations. This analytical process yields conclusions that are in agreement with prior research [39, 44].

3.3 Our Design for UMT

3.3.1 Category-Adaptive Allocation Strategy

We assign transformation parameters based on categories to realize class-wise setting. For rotation transformation R∈ R 3 × 3 , we refer to α and β as slight angles imposed on the x and y axes, γ as the primary angle for z axis. We generate random angles for A N times in three directions:

where U denotes uniform distribution, N denotes the number of categories, r s is a small range that controls α and β , while r p is a large range that controls γ . A N is computed in such a way to ensure that the number of combinations of three angles is greater than or equal to N . Concretely, in the rotation operation, each of the three directions has A N distinct angles, which means that the final rotation matrix has A 3 possible combinations. To satisfy the class-wise setup, A 3 must be at least

N N N , requiring A N to be no less than ⌈ 3 √ N ⌉ . Finally, we randomly select N combinations of angles for the allocation. The scaling transformation S ∈ R 3 × 3 resizes the position of each point in the 3D point cloud sample by a certain scaling factor λ , which is sampled N times from a uniform distribution U :

where b l and b u represent the lower bound and upper bound of the scaling factor, respectively. For shear H ∈ R 3 × 3 defined in Eq. (13), twisting W ∈ R 3 × 3 defined in Eq. (16), the process of generating parameters within ranges ( ω l , ω u ) and ( h l , h u ) is consistent to scaling. The range of these parameters ensures the visual effect of the sample.

Our unlearnable scheme: UMT

Airplane

(

i

-th class)

Class-wise rotation

transformation

𝓡𝓡

Raw point cloud data

Unlearnable

point cloud data

X

Z

𝛾𝛾

𝑖𝑖

𝛼𝛼

𝑖𝑖

𝛽𝛽

𝑖𝑖

�

Y

O

�

X

Z

Y

�

�

�

�

�

𝜆𝜆

𝑖𝑖

O

Category-Adaptive Allocation Strategy

Number of

categories

N

Uniform distribution

U

𝑘𝑘

,

𝑟𝑟

𝑠𝑠

,

𝑟𝑟

𝑝𝑝

,

𝑏𝑏

𝑙𝑙

,

𝑏𝑏

𝑢𝑢

,

…  ,

𝑤𝑤

𝑙𝑙

,

𝑤𝑤

𝑢𝑢

Parameter lists

ℒℛ

,

ℒ𝑆𝑆

, … ,

ℒℋ

…

Assigning class-wise

parameters

Input raw 3D point

cloud data (w/o

any protection)

…

Employing Class-wise

Transformations

Data Restoration Scheme

Data protector

Authorized

user

Unauthorized

user

Secure

transport

Training

Reverse transformation

Training

Legally accessed

Illegally  accessed

(Lightweight)

Class-wise scaling

transformation

𝑺𝑺

Class-wise shear

transformation

𝑯𝑯

�

Z

Y

�

𝜘𝜘

𝑖𝑖

O

Matrix

multiplication

X

Property 1. Since rotation matrices R α , R β , and R γ around three directions are all orthogonal matrices, R is also an orthogonal matrix, which can be defined as:

where we can determine the orthogonality by matrix multiplication through the definitions of Eq. (11). Since R = R α R β R γ and RR T = R α R β R γ R T γ R T β R T α = I , so R is also an orthogonal matrix.

Property 2. All four transformation matrices we employ, R , S , W , and H , and the multiplicative combinations of any these matrices are all invertible matrices, which can be formally defined as:

where the inverse matrices of R , S , W , and H are given in Appendix A. This property allows the authorized users to normally train on the protected data due to that multiplying a matrix by its inverse results in the identity matrix, leading us to propose a data restoration scheme in Sec. 3.4.

3.3.2 Employing Class-wise Transformations

Assuming the point cloud training set D c is defined as { ( X c 1 , Y i ) , ( X c 2 , Y i ) , ..., ( X cn i , Y i ) } N i =1 , where n 1 , n 2 , ..., n N represent the number of samples in the 1st, 2nd, ..., N -th category, respectively. We formally define the spectrum of transformations as k to indicate the number of transformations involved. Thus the ultimate unlearnable transformation matrix T k is defined as:

̸

̸

Once we employ the proposed category-adaptive allocation strategy to T k , the unlearnable point cloud dataset D u is constructed as:

Our proposed UMT scheme is described in Algorithm 1. We enumerate possible transformations in Eq. (7) to obtain the unlearnability in Tab. 7 and select one type of class-wise transformation for each k for more comprehensive experiments in Tab. 1. To facilitate the theoretical study of UMT 2 , we opt for RS as the transformation matrix T , which achieves the best unlearnable effect as suggested in Tabs. 1 and 7. Thus, in the GMM scenario, the class-wise transformation matrix is defined as T y = R y S y = λ y R y ∈ R d × d , where λ y ∈ R is the scaling factor.

Lemma 3. The unlearnable dataset D u generated using UMT on D c can also be represented using a GMM, i.e., D u ∼ N ( y T y µ, λ 2 y I ) .

Proof: See Appendix D.1. Lemma 3 demonstrates that the unlearnable dataset D u can also be represented as a GMM, which is derived from Property 1.

Proof: See Appendix D.2. Lemma 4 reveals that the Bayesian decision boundary for classifying D u is a quadratic surface based on the GMM expression of D u .

Lemma 5. Let z ∼ N (0 , I ) , Z = z ⊤ z + b ⊤ z + c , where b = B A , c = C A , and ∥·∥ 2 denote 2-norm of vectors. For any t ≥ 0 and γ ∈ R , we employ Chernoff bound to have:

Proof: See Appendix D.3. Lemma 5 enables us to establish an upper bound on the accuracy of the unlearnable decision boundary P u applied to the clean dataset D c , denoted as τ D c ( P u ) , as presented in Theorem 6 below.

Theorem 6. For any constant t 1 and t 2 satisfying 0 ≤ t 1 < 1 2 and 0 ≤ t 2 < 1 2 , the accuracy of the unlearnable decision boundary P u on D c can be upper-bounded as:

̸

Furthermore, if µ ⊤ µ + b ⊤ µ + c + d < 0 and -µ ⊤ µ + b ⊤ µ -c -d < 0 , we have τ D c ( P u ) < 1 . Moreover, for any µ = 0 , ∃ matrix T i such that τ D c ( P u ) < τ D c ( P ) , where P is the Bayes optimal decision boundary for classifying D c .

Proof: See Appendix D.4. The unlearnable effect takes place when τ D c ( P u ) < τ D c ( P ) . To achieve this, we elaborately choose T y , which is formalized as µ ⊤ λ -2 -1 T ⊤ -1 + λ -2 1 T ⊤ 1 λ -2 -1 -λ -2 1 µ ≪ 0 . Therefore, Theorem 6 theoretically explains why UMT is effective in generating unlearnable point cloud data.

3.4 Data Restoration Scheme

To ensure that authorized users can achieve better generalization after training on unlearnable data, i.e. , satisfying Eq. (2), we exploit the inverse properties of 3D transformations, presented in Property 2, to calculate the inverse matrix of T k as:

̸

̸

In particular, we note that R -1 = R T , S -1 = 1 λ I . Afterwards, the authorized user receives a lightweight message M containing class-wise parameters from the data protector through a secure channel, thereby assigning M to the inverse transformation matrix in Eq. (9) for multiplying the unlearnable samples. Our proposed integral unlearnable process is illustrated in Fig. 3.

4 Experiments

4.1 Experimental Details

Datasets and Models. Three synthetic 3D point cloud datasets, ModelNet40 [50], ModelNet10 [50], ShapeNetPart [4], and three real-world datasets including autonomous driving dataset KITTI [32] and indoor datasets ScanObjectNN [41], S3DIS [2] are used. We choose 16 widely used 3D point cloud models PointNet [35], PointNet++ [36], DGCNN [45], PointCNN [25], PCT [16], PointConv [48], CurveNet [51], SimpleView [15], 3DGCN [26], LGR-Net [59], RIConv [57], RIConv++ [58], PointMLP [31], PointNN [56], PointTransformerV3 [49], and SegNN [62] for evaluation of classification and semantic segmentation tasks.

Experimental Setup. The training process involves Adam optimizer [22], CosineAnnealingLR scheduler [30], initial learning rate of 0.001, weight decay of 0.0001. We empirically set r s , r p , b l , b u , ω l , ω u , h l , and h u to 15 ◦ , 120 ◦ , 0.6, 0.8, 0 ◦ , 20 ◦ , 0, and 0.4 respectively. The main results of different

Datasets

Schemes

PointNet

PointNet++ DGCNN

PointCNN

PCT

PointConv

CurveNet

SimpleView

RIConv++

3DGCN

PointNN

PointMLP

AVG

ModelNet10

Clean

89.85±0.54

92.11±1.57

91.81±0.84

87.99±1.58

90.18±2.23

91.26±1.12

91.88±1.51

89.52±0.41

87.90±1.64

88.51±5.39

83.22±0.73

91.07±0.19

89.61±0.42

UMT(k=1)

40.64±10.69

28.73±2.00

27.41±6.71

32.82±2.70

31.58±4.27

33.64±6.63

39.78±6.80

45.31±11.85

86.72±1.46

28.01±8.72

34.47±0.29

32.85±2.35

38.50±2.01

UMT(k=2)

21.18±0.93

26.36±1.71

18.84±6.14

21.97±4.04

19.72±4.52

20.84±5.96

25.04±2.15

22.75±2.43

16.53±3.63

32.79±3.23

31.94±2.11

25.41±3.96

23.61±1.06

UMT(k=3)

22.84±1.71

23.81±8.43

29.16±4.30

27.03±9.97

29.43±9.11

18.49±7.39

25.51±9.57

32.12±12.58

19.94±1.13

36.19±13.38

30.54±1.24

32.21±5.28

27.27±6.20

UMT(k=4)

18.83±2.40

20.56±14.61

15.92±1.51

20.52±6.06

20.29±2.14

21.66±2.24

23.46±12.58

24.12±6.20

26.41±2.47

34.28±17.20

25.59±0.25

26.65±11.36

23.19±5.98

ModelNet40

Clean

86.18±0.07

90.55±0.73

89.51±0.86

81.89±5.35

87.11±1.39

88.90±0.89

87.82±0.23

85.25±0.31

84.59±1.07

86.81±1.69

74.81±0.16

87.31±0.91

85.89±0.40

UMT(k=1)

28.62±1.80

20.69±0.87

28.60±1.65

21.92±1.86

29.06±0.38

24.99±2.63

33.29±3.92

26.89±1.48

75.86±8.43

26.30±1.92

28.57±0.39

29.28±1.72

31.17±0.20

UMT(k=2)

8.30±0.87

18.71±1.65

11.60±1.97

10.85±2.61

9.21±2.31

12.99±0.99

12.71±3.70

12.05±3.88

10.48±2.01

26.17±0.35

25.05±0.06

10.90±4.74

14.09±0.78

UMT(k=3)

7.48±1.66

17.62±3.10

10.41±5.48

9.43±3.09

9.91±4.19

11.28±4.93

11.23±3.07

11.21±4.09

10.92±0.48

25.31±1.29

24.53±0.10

8.94±3.43

13.19±2.41

UMT(k=4)

7.83±2.25

15.72±4.12

10.86±2.57

8.60±2.28

9.68±1.58

10.52±2.09

10.91±3.00

14.32±0.92

18.09±3.83

17.16±5.47

24.31±0.28

12.38±1.51

13.36±0.43

ShapeNetPart

Clean

98.21±0.03

98.50±0.12

98.06±0.65

97.54±0.22

96.38±0.84

98.23±0.23

98.42±0.08

98.26±0.21

96.70±0.64

96.18±1.90

94.66±0.11

98.39±0.12

97.46±0.22

UMT(k=1)

63.85±12.71

50.30±21.71

64.71±13.44

51.95±14.13

54.39±3.36

29.34±0.14

71.38±9.34

56.22±6.48

96.93±0.09

37.90±9.67

58.44±0.61

47.32±8.26

56.89±3.00

UMT(k=2)

18.41±6.45

45.61±4.64

25.99±2.73

28.97±3.73

37.72±16.26

25.60±8.59

26.49±8.58

38.38±5.96

5.05±3.63

32.21±13.01

49.82±1.56

34.19±4.43

30.70±1.43

UMT(k=3)

32.50±10.03

39.64±12.52

37.29±11.20

46.77±19.59

43.52±22.24

31.28±4.67

37.27±10.30

41.51±2.66

6.01±6.11

47.84±7.05

50.85±0.07

47.80±21.37

38.52±6.17

UMT(k=4)

23.72±15.04

23.30±4.43

36.18±10.00

34.52±16.15

45.07±18.99

29.48±11.09

29.79±4.79

40.04±9.97

32.22±1.38

33.06±28.16

51.91±0.03

40.93±21.81

35.02±11.23

KITTI

Clean

98.04±2.23

99.49±0.50

99.33±0.34

99.23±0.25

98.93±0.54

98.04±1.68

99.10±1.05

99.38±0.33

99.64±0.09

99.67±0.24

99.49±0.50

95.72±5.46

98.84±0.87

UMT(k=1)

36.23±30.18

62.90±20.51

38.93±33.08

57.80±13.99

58.26±21.66

39.99±6.82

33.33±10.75

56.71±30.96

98.53±1.19

68.34±10.46

70.34±0.98

47.20±20.79

55.71±5.54

UMT(k=2)

31.24±7.11

51.07±20.29

27.90±10.33

49.69±11.52

51.47±17.36

25.95±11.38

20.55±4.93

51.77±8.53

99.80±0.09

70.42±27.97

47.31±10.58

39.90±10.00

47.26±5.95

UMT(k=3)

19.13±6.93

53.73±12.44

31.26±21.68

56.08±17.90

54.45±11.77

42.54±21.46

27.38±19.20

32.62±11.76

98.48±1.61

69.51±8.27

70.14±1.30

58.44±6.61

51.15±4.79

UMT(k=4)

26.84±16.26

61.79±6.65

29.89±15.72

54.27±11.99

64.21±10.76

57.29±12.05

37.70±11.60

54.63±13.07

99.34±0.23

72.33±3.47

70.49±0.76

56.25±12.33

57.09±5.97

ScanObjectNN

Clean

65.20±1.49

77.38±2.60

72.66±2.56

66.96±6.42

50.75±15.16

75.42±2.06

70.92±1.06

51.93±2.89

66.34±1.21

73.84±2.75

58.17±0.30

73.32±1.39

66.91±0.72

UMT(k=1)

56.55±0.11

63.01±2.38

57.93±3.11

51.97±11.99

47.05±7.12

56.71±3.61

65.49±3.15

38.96±2.08

62.33±10.68

52.89±1.13

54.56±0.17

62.96±4.19

55.87±0.95

UMT(k=2)

14.55±1.81

49.41±3.44

20.96±3.99

13.61±4.97

15.10±2.05

20.78±4.74

21.35±2.59

20.73±9.64

10.76±1.76

52.37±5.62

48.25±0.20

16.32±4.78

25.35±0.49

UMT(k=3)

10.92±3.87

41.96±3.84

14.62±5.56

10.62±2.35

22.02±7.83

19.96±3.13

21.43±8.67

11.79±3.96

11.69±1.41

56.32±0.83

48.94±0.70

23.05±16.32

24.44±2.46

UMT(k=4)

5.77±2.10

34.65±8.31

12.16±4.84

9.87±0.95

17.86±11.63

12.79±7.46

15.28±4.86

20.09±6.26

31.83±2.20

46.06±14.07

45.27±0.17

8.76±3.18

21.70±3.61

k on unlearnability is shown in Tab. 1. Specifically, k = 1 uses R , k = 2 uses RS , k = 3 uses RSW , and k = 4 uses RSWH . More results for different combinations of class-wise transformations are provided in Tab. 7. The table values covered by gray denote the best unlearnable effect.

Evaluation Metrics. For classification, we report the test accuracy (in %) derived from the classification accuracy on clean test set, which aligns with the metrics used in the 2D unlearnable schemes [19, 28, 39]. For semantic segmentation, we use eval accuracy and mean Intersection over Union (mIoU) as evaluation metrics (in %), where eval accuracy is the ratio of the number of points classified correctly to the total number of points in the point cloud, mIoU calculates the IoU for each class between the ground truth and the predicted segmentation [61], and then takes the average of these ratios. The lower these metrics are, the better the effect of the unlearnable scheme.

4.2 Evaluation of Proposed Unlearnable Schemes

Effectiveness. As shown in Tab. 1, UMT results in a significant decrease in test accuracy compared to clean baseline, indicating the unlearnable effectiveness of UMT. Moreover, all values of k achieve good unlearnable results. The average performance of RS is the best, which is employed as the default in the remaining experiments. We note that rigid transformations are easily defeated by invariance networks [10, 26, 58]. Therefore, in practical settings, we will include non-rigid trans-

PointNet

PointNet++

DGCNN

PointCNN

20

40

60

80

100

Test accuracy (%)

ModelNet10

Clean dataset

Restoration dataset

UMT dataset

PointNet

PointNet++

DGCNN

PointCNN

20

40

60

80

100

Test accuracy (%)

ShapeNetPart

Clean dataset

Restoration dataset

UMT dataset

formations, using combined transformations to enhance the robustness of the UMT scheme.

Robustness. (i) Data augmentations. Similar to [18, 24], we employ data augmentations like random scaling , random jitter , random rotation , Statistic Outlier Removal (SOR) [60], and Simple Random Sampling (SRS) [53] against UMT. Tab. 2 suggests UMT is robust to random data augmentations. SOR detects and removes outliers or noisy points but is ineffective in countering UMT because UMT does not introduce irregular perturbations or add outliers. SRS randomly selects a small subset from the entire set of points with equal probability. UMT is robust to SRS as it alters the coordinates of all points. Even if an arbitrary subset is chosen, all points within the subset have already undergone the unlearnable transformations. (ii) Adaptive attack. We assume the unauthorized user gains

knowledge about the G p 's use of RS . Thus we propose random rotation & scaling as an adaptive scheme. In Tab. 2, the adaptive scheme exhibits a higher accuracy than other schemes, confirming its effectiveness. Nonetheless, it remains 28.67% lower than clean baseline, revealing the robustness of UMT against adaptive attack. More results of adaptive attacks are provided in Appendix C.3.

Visual Effect. We visualize UMT samples in Figs. 7 to 10, indicating that the unlearnable point cloud samples still retain their normal feature structure with visual rationality.

Evaluation of Semantic Segmentation. We evaluate UMT using common metrics for point cloud semantic segmentation tasks in Tab. 3. As can be seen, the performance of semantic segmentation of data protected by UMT significantly decreases. The underlying reason is that the DNNs learn the features of class-wise transformations and establish a new mapping, which leads to the inability of test samples without transformations to be correctly segmented by the segmentation model.

Evaluation of Data Restoration. We multiply UMT samples by the transformation matrix in Eq. (9). The data becomes learnable after the restoration process, with test accuracy reaching a level comparable to the clean baseline as shown in Fig. 4. This strongly validates the effectiveness of the proposed data restoration scheme.

30

rp=120°,b=0.6,by=0.8

30

r;=15°, b/=0.6 , by=0.8

40

r=15°,rp=120°,b=0.8

30

rs=15°,rp=120°, b=0.6

35

25

25

25

Test accuracy (%)

201

Test accuracy (%)

20

Test accuracy (%)

20

15

15

20

151

10

AVG

10

AVG

AVG

AVG

PointNet

PointNet

10

PointNet

PointNet

DGCNN

5

DGCNN

5

DGCNN

5

DGCNN

PointCNN

PointCNN

PointCNN

PointCNN

0

5

10

15

20

0

120

150

180

210

0

0.55

0.60

0.65

0.70

0.70

0.75

0.80

0.85

rs (°)

rp (°)

Ig

bu

4.3 Ablation Study and Hyper-Parameter Sensitivity Analysis

Ablation on Rotation Module. As shown in Tab. 4, the average accuracy increases by 15.18% and 21.65%, respectively, when only using S . This suggests the importance of class-wise rotation module. The high test accuracy demonstrated by 3DGCN [26] can be attributed to its scaling invariance, which endows it with robustness against scaling transformation.

Ablation on Scaling Module. As also shown in Tab. 4, the average accuracy increases by 18.80% and 16.88% when only using the rotation module, respectively. The high test accuracy achieved on RIConv [57] and LGR-Net [59] is due to the fact that both networks are rotation-invariant, thus providing resistance against rotation transformations. These ablation results furthermore emphasize the importance of incorporating more non-rigid transformations.

Hyper-Parameter Analysis. We analyze four hyperparameters r s , r p , b l , and b u in Fig. 5. The influence of r s and r p on the accuracy remains relatively small, exhibiting their best unlearnable effect when set to 15 ◦ and 120 ◦ , respectively. We attribute this to the crucial role played by the class-wise setting, while it is not highly sensitive to the size of specific values. The unlearnable effect is the best when b l and b u are set to 0.6 and 0.8, respectively. Similarly, the variations in b l and b u do not significantly alter the effect due to the class-wise setting.

(b)

With clean training data

With UMT

training data

Low

𝓛𝓛

𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕

𝜽𝜽

𝟎𝟎

𝜽𝜽

𝒕𝒕-𝟏𝟏

𝜽𝜽

𝒕𝒕

𝜽𝜽

𝒕𝒕+𝟏𝟏

𝜽𝜽

𝒄𝒄

∗

𝜽𝜽

𝒖𝒖

∗

(a)

model weight

With clean training data

With UMT

training data

Low

𝓛𝓛

𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕

𝜽𝜽

𝟎𝟎

𝜽𝜽

𝒕𝒕-𝟏𝟏

𝜽𝜽

𝒕𝒕

𝜽𝜽

𝒕𝒕+𝟏𝟏

𝜽𝜽

𝒄𝒄

∗

𝜽𝜽

𝒖𝒖

∗

Low

𝓛𝓛

𝒖𝒖

(b)

model weight

4.4 Insightful Analysis Into UMT

We formalize L ( D ) = E ( X , Y ) ∼D [ L c ( F ( X ; θ ) , Y )] , where L c is the cross-entropy loss, X , Y is the point cloud data sampled from dataset D . We define D tr , D u , and D c as the training set, unlearnable test set ( i.e. , test set transformed by UMT), and clean test set, respectively. Thus we have the training loss L train = L ( D tr ) , unlearnable test loss L u = L ( D u ) , and clean test loss L c = L ( D c ) .

The models trained on both clean and UMT training sets exhibit low L train as shown in Fig. 6 ( yellow ellipses ), indicating the models converge well during training. Furthermore, when tested on D c as shown in Fig. 6 (a), the clean model (trained with clean training set) achieves a low L c ( blue ellipse ), while the UMT model (trained with UMT training set) exhibits a high L c ( red ellipse ), also supporting the unlearnable effectiveness of UMT. Fig. 6 (b) reveals that the clean model and UMT model both exhibit low L u ( green ellipse ), suggesting that they can classify UMT samples. But the mechanisms underlying the two cases of low L u differ. The clean one is due to that the semantics of samples can be remained by UMT and thus normally classified. The UMT one is that the UMT model learns the mapping between transformations and labels, thereby correctly predicting the samples containing the same transformations. We conclude that the clean model effectively classifies both clean and UMT samples, while the UMT model successfully classifies UMT samples (the UMT process is the same for both training and test samples) but fails to classify clean samples.

5 Related Work

5.1 2D Unlearnable Schemes

The development of 2D unlearnable schemes [19, 29, 37, 39, 42, 55] has been booming. Specifically, model-dependent methods are initially proposed in abundance [9, 19, 29]. Afterwards, numerous model-agnostic methods that significantly improve the generation efficiency have surfaced [39, 42, 47]. However, due to the structural disparities between 3D point cloud data and 2D images, applying unlearnable methods directly from 2D to 3D reveals significant challenges.

5.2 Protecting 3D Point Cloud Data Privacy

Some works proposed an encryption scheme based on chaotic mapping [21] or optical chaotic encryption [27], and a 3D object reconstruction technique was introduced [34], both achieving privacy protection for individual 3D point cloud data. Nevertheless, no privacy-preserving solution has been proposed specifically for the scenario of unauthorized DNN learning on abundant raw 3D point cloud data. It is worth mentioning that both parallel works [44, 63] study availability poisoning attacks against 3D point cloud networks, which largely reduce model accuracy, and both have the potential to be applied as unlearnable schemes. However, the feature collision error-minimization poisoning scheme proposed by Zhu et al. [63] overlooks the problem of effective training for authorized users, which limits its practical use in real-world applications. The rotation-based poisoning approach proposed by Wang et al. [44] is easily defeated by rotation-invariant networks [57, 58, 59], as revealed in Tabs. 1 and 4.

6 Conclusion, Limitation, and Broader Impacts

In this research, we propose the first integral unlearnable framework in 3D point clouds, which utilizes class-wise multi transformations, preventing unauthorized deep learning while allowing authorized training. Extensive experiments on synthetic and real-world datasets and theoretical evidence verify the superiority of our framework. The transformations include rotation, scaling, twisting, and shear, which are all common 3D transformation operations. If unauthorized users design a network that is invariant to all these transformations, they could potentially defeat our proposed UMT. So far, only networks invariant to rigid transformations like rotation and scaling have been proposed, while networks invariant to non-rigid transformations like twisting and shear, have not yet been introduced. Therefore, our research also contributes to the design of more transformation-invariant networks.

Our research calls for the design of more robust point cloud networks, which helps improve the robustness and security of 3D point cloud processing systems. On the other hand, if our proposed UMT scheme is maliciously exploited, it may have negative impacts on society, such as causing a sharp decline in the performance of models trained on it, affecting the security and reliability of technologies based on 3D point cloud networks. More transformation-invariant 3D point cloud networks need to be proposed in the future to avoid potential negative impacts.

References

Evangelos Alexiou, Nanyang Yang, and Touradj Ebrahimi. Pointxr: A toolbox for visualization and subjective evaluation of point clouds in virtual reality. In Proceedings of the 12th International Conference on Quality of Multimedia Experience (QoMEX'20) , pages 1-6, 2020.

1

Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3D semantic parsing of large-scale indoor spaces. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16) , pages 1534-1543, 2016. 6, 18

Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. SemanticKITTI: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV'19) , pages 9297-9307, 2019. 19

Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012 , 2015. 6, 18, 21, 22

Sizhe Chen, Geng Yuan, Xinwen Cheng, Yifan Gong, Minghai Qin, Yanzhi Wang, and Xiaolin Huang. Self-ensemble protection: Training checkpoints are good data protectors. In Proceedings of the 11th International Conference on Learning Representations (ICLR'23) , 2023. 2

Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3D object detection network for autonomous driving. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'17) , pages 1907-1915, 2017. 1

Wenda Chu, Linyi Li, and Bo Li. Tpc: Transformation-specific smoothing for point cloud models. In Proceedings of the 39th International Conference on Machine Learning (ICML'22) , pages 4035-4056, 2022. 2, 3, 15, 16

Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojciech Czaja, and Tom Goldstein. Adversarial examples make strong poisons. In Proceedings of the 35th Neural Information Processing Systems (NeurIPS'21) , pages 30339-30351, 2021. 3

Shaopeng Fu, Fengxiang He, Yang Liu, Li Shen, and Dacheng Tao. Robust unlearnable examples: Protecting data against adversarial learning. In Proceedings of the 10th International Conference on Learning Representations (ICLR'22) , 2022. 1, 2, 4, 9

Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE (3)-transformers: 3D roto-translation equivariant attention networks. In Proceedings of the 34th Neural Information Processing Systems (NeurIPS'20) , volume 33, pages 1970-1981, 2020. 7

Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE(3)-transformers: 3D roto-translation equivariant attention networks. In Proceedings of the 34th Neural Information Processing Systems (NeurIPS'20) , volume 33, pages 1970-1981, 2020. 20

Cong Gao, Geng Wang, Weisong Shi, Zhongmin Wang, and Yanping Chen. Autonomous driving security: State of the art and challenges. IEEE Internet of Things Journal , 9(10):7572-7595, 2021. 1

Kuofeng Gao, Jiawang Bai, Baoyuan Wu, Mengxi Ya, and Shu-Tao Xia. Imperceptible and robust backdoor attack in 3D point cloud. IEEE Transactions on Information Forensics and Security (TIFS'23) , 19:1267-1282, 2023. 3

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , pages 665-673, 2020. 3

Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with a simple and effective baseline. In Proceedings of the 38th International Conference on Machine Learning (ICML'21) , pages 3809-3820. PMLR, 2021. 6

Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R. Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational Visual Media , 7:187-199, 2021. 6, 17

Shengshan Hu, Wei Liu, Minghui Li, Yechao Zhang, Xiaogeng Liu, Xianlong Wang, Leo Yu Zhang, and Junhui Hou. Pointcrt: Detecting backdoor in 3D point cloud via corruption robustness. In Proceedings of the 31st ACM International Conference on Multimedia (MM'23) , pages 666-675, 2023. 2, 17

Shengshan Hu, Junwei Zhang, Wei Liu, Junhui Hou, Minghui Li, Leo Yu Zhang, Hai Jin, and Lichao Sun. Pointca: Evaluating the robustness of 3D point cloud completion models against adversarial examples. In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI'23) , pages 872-880, 2023. 7

Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. In Proceedings of the 9th International Conference on Learning Representations (ICLR'21) , 2021. 1, 2, 3, 4, 7, 9

Adel Javanmard and Mahdi Soltanolkotabi. Precise statistical analysis of classification accuracies for adversarial training. The Annals of Statistics , 50(4):2127-2156, 2022. 2, 3

Xin Jin, Zhaoxing Wu, Chenggen Song, Chunwei Zhang, and Xiaodong Li. 3D point cloud encryption through chaotic mapping. In Proceedings of the 17th Pacific Rim Conference on Multimedia (PCM'16) , pages 119-129, 2016. 9

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 6, 17

Peiliang Li, Siqi Liu, and Shaojie Shen. Multi-sensor 3D object box refinement for autonomous driving. arXiv preprint arXiv:1909.04942 , 2019. 1

Xinke Li, Zhirui Chen, Yue Zhao, Zekun Tong, Yabang Zhao, Andrew Lim, and Joey Tianyi Zhou. Pointba: Towards backdoor attacks in 3D point cloud. In Proceedings of the 18th IEEE/CVF International Conference on Computer Vision (ICCV'21) , pages 16492-16501, 2021. 7

Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on x-transformed points. In Proceedings of the 32nd Neural Information Processing Systems (NeurIPS'18) , pages 828-838, 2018. 6

Zhi-Hao Lin, Sheng-Yu Huang, and Yu-Chiang Frank Wang. Convolution in the cloud: Learning deformable kernels in 3D graph convolution networks for point cloud analysis. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'20) , pages 1800-1809, 2020. 6, 7, 8, 19

Bocheng Liu, Yongxiang Liu, Yiyuan Xie, Xiao Jiang, Yichen Ye, Tingting Song, Junxiong Chai, Meng Liu, Manying Feng, and Haodong Yuan. Privacy protection for 3D point cloud classification based on an optical chaotic encryption scheme. Optics Express , 31(5):8820-8843, 2023. 9

Shuang Liu, Yihan Wang, and Xiao-Shan Gao. Game-theoretic unlearnable example generator. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI'24) , 2024. 1, 2, 3, 7

Yixin Liu, Kaidi Xu, Xun Chen, and Lichao Sun. Stable unlearnable example: Enhancing the robustness of unlearnable examples via stable error-minimizing noise. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI'24) , pages 3783-3791, 2024. 1, 2, 9

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016. 6, 17

Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework. arXiv preprint arXiv:2202.07123 ,

2022. 6

Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'15) , pages 3061-3070, 2015. 1, 6

Yifei Min, Lin Chen, and Amin Karbasi. The curious case of adversarially robust models: More data can help, double descend, or hurt generalization. In Uncertainty in Artificial Intelligence , pages 129-139. PMLR, 2021. 2, 3

Arpit Nama, Amaya Dharmasiri, Kanchana Thilakarathna, Albert Zomaya, and Jaybie Agullo de Guzman. User configurable 3D object regeneration for spatial privacy. arXiv preprint arXiv:2108.08273 , 2021. 9

Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'17) , pages 652-660, 2017. 4, 6

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Proceedings of the 31st Neural Information Processing Systems (NeurIPS'17) , pages 5099-5108, 2017. 6, 8

Jie Ren, Han Xu, Yuxuan Wan, Xingjun Ma, Lichao Sun, and Jiliang Tang. Transferable unlearnable examples. In Proceedings of the 11th International Conference on Learning Representations (ICLR'23) , 2023. 9

Douglas A. Reynolds. Gaussian mixture models. Encyclopedia of Biometrics , 741(659-663), 2009. 2, 3

Vinu Sankar Sadasivan, Mahdi Soltanolkotabi, and Soheil Feizi. CUDA: Convolution-based unlearnable datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'23) , pages 3862-3871, 2023. 1, 2, 3, 4, 7, 9

StubbornAtom. Distributions of quadratic form of a normal random variable. Cross Validated, 2020. https://stats.stackexchange.com/q/478682. 24

Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the 17th IEEE/CVF International Conference on Computer Vision (ICCV'19) , 2019. 6, 18, 21

Xianlong Wang, Shengshan Hu, Minghui Li, Zhifei Yu, Ziqi Zhou, and Leo Yu Zhang. Corrupting convolution-based unlearnable datasets with pixel-based image transformations. arXiv preprint arXiv:2311.18403 , 2023. 1, 2, 3, 9

Xianlong Wang, Shengshan Hu, Yechao Zhang, Ziqi Zhou, Leo Yu Zhang, Peng Xu, Wei Wan, and Hai Jin. ECLIPSE: Expunging clean-label indiscriminate poisons via sparse diffusion purification. In Proceedings of the 29th European Symposium on Research in Computer Security (ESORICS'24) , pages 146-166, 2024. 2

Xianlong Wang, Minghui Li, Peng Xu, Wei Liu, Leo Yu Zhang, Shengshan Hu, and Yanjun Zhang. PointAPA: Towards availability poisoning attacks in 3D point clouds. In Proceedings of the 29th European Symposium on Research in Computer Security (ESORICS'24) , pages 125-145, 2024. 2, 3, 4, 9

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions On Graphics (TOG'19) , pages 1-12, 2019. 6

Florian Wirth, Jannik Quehl, Jeffrey Ota, and Christoph Stiller. Pointatme: efficient 3D point cloud labeling in virtual reality. In Proceedings of the 2019 IEEE Intelligent Vehicles Symposium (IV'19) , pages 1693-1698, 2019. 1

Shutong Wu, Sizhe Chen, Cihang Xie, and Xiaolin Huang. One-pixel shortcut: on the learning preference of deep neural networks. In Proceedings of the 11th International Conference on Learning Representations (ICLR'23) , 2023. 9

Wenxuan Wu, Zhongang Qi, and Fuxin Li. Pointconv: Deep convolutional networks on 3D point clouds. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'19) , pages 9621-9630, 2019. 6

Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster, stronger. arXiv preprint arXiv:2312.10035 , 2023. 6, 8

Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3D shapenets: A deep representation for volumetric shapes. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'15) , pages 1912-1920, 2015. 4, 6, 18, 21, 23

Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. Walk in the cloud: Learning curves for point clouds shape analysis. In Proceedings of the 18th IEEE/CVF International Conference on Computer Vision (ICCV'21) , pages 915-924, 2021. 6

Zhen Xiang, David J. Miller, Siheng Chen, Xi Li, and George Kesidis. A backdoor attack against 3D point cloud classifiers. In Proceedings of the 18th IEEE/CVF International Conference on Computer Vision (ICCV'21) , pages 7597-7607, 2021. 17

Jiancheng Yang, Qiang Zhang, Rongyao Fang, Bingbing Ni, Jinxian Liu, and Qi Tian. Adversarial attack and defense on point sets. arXiv preprint arXiv:1902.10899 , 2019. 7

Jingwen Ye and Xinchao Wang. Ungeneralizable examples. In Proceedings of the 2024 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'24) , 2024. 2

Jiaming Zhang, Xingjun Ma, Qi Yi, Jitao Sang, Yugang Jiang, Yaowei Wang, and Changsheng Xu. Unlearnable clusters: Towards label-agnostic unlearnable examples. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'23) , 2023. 9

Renrui Zhang, Liuhui Wang, Yali Wang, Peng Gao, Hongsheng Li, and Jianbo Shi. Parameter is not all you need: Starting from non-parametric networks for 3D point cloud analysis. arXiv preprint arXiv:2303.08134 , 2023. 6

Zhiyuan Zhang, Binh-Son Hua, David W. Rosen, and Sai-Kit Yeung. Rotation invariant convolutions for 3D point clouds deep learning. In Proceedings of the 2019 International Conference on 3D Vision (3DV'19) , pages 204-213. IEEE, 2019. 6, 8, 9, 19

Zhiyuan Zhang, Binh-Son Hua, and Sai-Kit Yeung. Riconv++: Effective rotation invariant convolutions for 3D point clouds deep learning. International Journal of Computer Vision (IJCV'22) , 130(5):1228-1243, 2022. 6, 7, 9, 19

Chen Zhao, Jiaqi Yang, Xin Xiong, Angfan Zhu, Zhiguo Cao, and Xin Li. Rotation invariant point cloud classification: Where local geometry meets global topology. arXiv preprint arXiv:1911.00195 , 2019. 6, 8, 9, 19

Hang Zhou, Kejiang Chen, Weiming Zhang, Han Fang, Wenbo Zhou, and Nenghai Yu. Dup-net: Denoiser and upsampler network for 3D adversarial point clouds defense. In Proceedings of the 17th IEEE/CVF International Conference on Computer Vision (ICCV'19) , pages 1961-1970, 2019. 7

Ziqi Zhou, Yufei Song, Minghui Li, Shengshan Hu, Xianlong Wang, Leo Yu Zhang, Dezhong Yao, and Hai Jin. Darksam: Fooling segment anything model to segment nothing. arXiv preprint arXiv:2409.17874 , 2024. 7

Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Jiaming Liu, Han Xiao, Chaoyou Fu, Hao Dong, and Peng Gao. No time to train: Empowering non-parametric networks for few-shot 3D scene segmentation. In Proceedings of the 2024 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'24) , 2024. 6, 8

Yifan Zhu, Yibo Miao, Yinpeng Dong, and Xiao-Shan Gao. Toward availability attacks in 3D point clouds. arXiv preprint arXiv:2407.11011 , 2024. 9

Appendix: Unlearnable 3D Point Clouds: Class-wise Transformation Is All You Need

A Definitions of 3D Transformations

Existing 3D transformations are summarized in Fig. 1 and formally defined in this section. The 3D transformations are mathematical operations applied to three-dimensional objects to change their position, orientation, and scale in space, which are often represented using transformation matrices. We formally define the transformed point cloud sample with transformation matrix T ∈ R 3 × 3 as:

where X ∈ R 3 × p is the clean point cloud sample, X t ∈ R 3 × p is the transformed point cloud sample.

A.1 Rotation Transformation

The rotation transformation that alters the orientation and angle of 3D point clouds is controlled by three angles α , β , and γ . The rotation matrices in three directions can be formally defined as:

Thus we have T = R α R β R γ while employing rotation transformation. Besides, we have R -1 α = R T α , R -1 β = R T β , and R -1 γ = R T γ .

A.2 Scaling Transformation

The scaling matrix S can be represented as:

where the scaling factor λ is used to perform a proportional scaling of the coordinates of each point in the point cloud.

A.3 Shear Transformation

In the three-dimensional space, shearing 3 is represented by different matrices, and the specific form depends on the type of shear being performed. Specifically, the shear transformation matrix H xy (employed in UMT) of shifting x and y by the other coordinate z , and its corresponding inverse matrix can be expressed as:

The shear transformation matrix H xz of shifting x and z by the other coordinate y , and its corresponding inverse matrix can be expressed as:

The shear transformation matrix H yz of shifting y and z by the other coordinate x , and its corresponding inverse matrix can be expressed as:

Algorithm 1 Our proposed UMT scheme

A.4 Twisting Transformation

The 3D twisting transformation [7] involves a rotational deformation applied to an object in threedimensional space, creating a twisted or spiraled effect. Unlike simple rotations around fixed axes, a twisting transformation introduces a variable rotation that may change based on the spatial coordinates of the object. For instance, considering a twisting transformation along the z -axis, where the rotation angle is a function related to the z -coordinate, it can be expressed as:

where θ is the parameter of the twisting transformation, and z is the z -coordinate of the object. The inverse matrix of W z ( θ, z ) is:

A.5 Tapering Transformation

The tapering transformation [7] is a linear transformation used to alter the shape of an object, causing it to gradually become pointed or shortened. In three-dimensional space, tapering transformation can adjust the dimensions of an object along one or more axes, creating a tapering effect. The specific matrix representation of tapering transformation depends on the chosen axis and the design of the transformation. Generally, tapering transformation can be represented by a matrix that is multiplied by the coordinates of the object to achieve the shape adjustment, which is defined as:

where z is the z -coordinate of the object. Considering that ηz could indeed equal to -1, in such a case, the 3D point cloud samples would be projected onto the z -plane, losing their practical significance and the tapering matrix is also irreversible.

A.6 Reflection Transformation

Reflection transformation is a linear transformation that inverts an object along a certain plane. This plane is commonly referred to as a reflection plane or mirror. For reflection transformations in threedimensional space, we can represent them through a matrix. Regarding the reflection transformation matrices of the xy plane, yz plane, and xz plane, we have:

A.7 Translation Transformation

The 3D translation transformation 4 refers to the process of moving an object in three-dimensional space. This transformation involves moving the object along the x , y , and z axes, respectively, smoothly transitioning it from one position to another, which is defined as:

where t x , t y , and t z represents the translation along the x , y , and z axes, respectively. This 4 × 4 matrix is a homogeneous coordinate matrix that describes the translation in three-dimensional space. Additionally, the translation matrix also can be represented as a additive matrix to original point x i ∈ R 3 × 3 , which can be defined as:

B Supplementary Experimental Settings

B.1 Experimental Platform

Our experiments are conducted on a server running a 64-bit Ubuntu 20.04.1 system with an Intel Xeon Silver 4210R CPU @ 2.40GHz processor, 125GB memory, and four Nvidia GeForce RTX 3090 GPUs, each with 24GB memory. The experiments are performed using the Python language, version 3.8.19, and PyTorch library version 1.12.1.

B.2 Hyper-Parameter Settings

The model training process on the unlearnable dataset and the clean dataset remains consistent, using the Adam optimizer [22], CosineAnnealingLR scheduler [30], initial learning rate of 0.001, weight decay of 0.0001, batch size of 16 (due to insufficient GPU memory, the batch size is set to 8 when training 3DGCN on ModelNet40 dataset), and training for 80 epochs. Due to the longer training process required by PCT [16], the training epochs for PCT in Tab. 1 on ModelNet10, ModelNet40, and ScanObjectNN datasets are all set to 240.

B.3 Settings of Exploring Experiments

We initially investigate which approach yields the best unlearnable effect among sample-wise , datasetwise , and class-wise settings in Fig. 2 (a). Specific experimental settings are as follows:

In the dataset-wise setting, the same parameter values are applied to the entire dataset. Specifically, we have α = β = γ = 10 ◦ in the rotation transformation, the scaling factor λ is set to 0.8, both shearing factors s and t are set to 0.2. The angle θ in twisting is 25 ◦ . The tapering angle η is set to 25 ◦ , and the parameters in translation transformation t x , t y , and t z are set to 0.15.

In the sample-wise setting, each sample has its independent set of parameters, meaning the parameter values for each sample are randomly generated within a certain range. In the rotation transformation, α , β , γ are uniformly sampled from the range of 0 ◦ to 20 ◦ . The scaling factor λ is uniformly sampled from 0.6 to 0.8, shearing factors s and t are uniformly sampled from the range of 0 to 0.4. Both the twist angle θ and tapering angle η are sampled from 0 ◦ to 50 ◦ , and the parameters in translation transformation t x , t y , and t z are sampled from 0 to 0.3.

In the class-wise setting, the parameters for transformations are associated with the point cloud's class. The selection of parameters for each class is also obtained by random sampling within a fixed range. The chosen ranges are generally consistent with those described for the sample-wise setting above. However, a difference lies in the consideration of slight angle range and primary angle range in the case of rotation transformations, where these ranges are 20 ◦ and 120 ◦ , respectively.

The specific results of test accuracy under different transformation modes, including sample-wise (random), class-wise, and dataset-wise (universal), are provided in Tab. 5. It can be seen that under the class-wise setting, the final unlearnable effect is the best.

B.4 Benchmark Datasets

Dataset Introduction. The ModelNet40 dataset is a point cloud dataset containing 40 categories, comprising 9843 training and 2468 test point cloud data. ModelNet10 is a subset of ModelNet40 dataset with 10 categories. ShapeNetPart that includes 16 categories is a subset of ShapeNet, comprising 12137 training and 2874 test point cloud samples. ScanObjectNN is a real-world point cloud dataset with 15 categories, comprising 2309 training samples and 581 test samples. Similar to [17, 52], we split KITTI object clouds into class 'vehicle' and 'human' containing 1000 training

data and 662 test data. The input point cloud objects for the models encompass 256 points for the KITTI dataset and 1024 points for other datasets. The Stanford 3D Indoor Scene Dataset (S3DIS) dataset contains 6 large-scale indoor areas with 271 rooms. Each point in the scene point cloud is annotated with one of the 13 semantic categories.

Dataset Licenses. The dataset license information is listed as:

ModelNet10, ModelNet40 [50]: All CAD models are downloaded from the Internet and the original authors hold the copyright of the CAD models. The label of the data is obtained by us via Amazon Mechanical Turk service and it is provided freely. This dataset is provided for the convenience of academic research only. Link is https://modelnet. cs.princeton.edu/ .

ShapeNetPart [4]: We use the ShapeNet database (the "Database") at Princeton University and Stanford University. Link is https://www.shapenet.org/ .

ScanObjectNN [41]: The license is MIT License: Copyright (c) 2019 Vision & Graphics Group, HKUST. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. Link is https://hkust-vgd.github.io/ scanobjectnn/ .

S3DIS [2]: The copyright is from Stanford University, Patent Pending, copyright 2016. Link is http://buildingparser.stanford.edu/dataset.html .

B.5 Settings of Robustness Experiments

The scaling factor in random scaling augmentation is set to a minimum of 0.8 and a maximum of 1.25. In the random rotation operation, the three directional rotation angles are identical and uniformly sampled from [0, 2 π ). The perturbations in the random jitter are sampled from a normal distribution with a standard deviation of 0.05, and the perturbation magnitude is constrained within 0.1. The parameters k and α in SOR are set to 2 and 1.1, respectively. The number of dropped points in SRS is 500.

C Supplementary Experimental Results

C.1 Results for Execution Mechanism

We present the specific experimental results of Fig. 2 (a) in Tab. 5. It can be observed that under the class-wise setting, the average test accuracy is the lowest, while the unlearnable condition yields the best performance. Furthermore, we conduct investigations into training the PointNet classifier with class-wise transformed training sets (ModelNet10 dataset), analyzing accuracy results across different test sets, as shown in Tab. 6. It is noteworthy that the accuracy on class-wise test sets (using consistent transformation parameters with class-wise transformed training set) is consistently above 90%, indicating that the model has learned the mapping between transformations and labels. This leads to correct classification on test samples with the same transformations. However, when the class-wise test set undergoes permutation, the accuracy drops to a level similar to that of the clean test set. This clearly demonstrates that the model can correctly classify samples only when they have corresponding transformations, and samples without transformations or with mismatched transformations cannot be correctly classified. This further validates that the model learns a one-to-one mapping between class transformations and labels.

C.2 Results of Diverse Combinations of Transformations

We investigate combinations of four transformations, i.e. , rotation, scaling, shear, and twisting, and create unlearnable datasets using the class-wise setting. The test accuracy results across five point cloud models are presented in Tab. 7. It can be observed that the combination of rotation and scaling,

two types of rigid transformations, achieves the most effective unlearnable effect. The transformation parameters used in this section are consistent with Appendix B.3.

To ensure the reliability of the experimental results, we combine two different transformations and obtain the results from three runs using random seeds with 23, 1023, and 2023, then average them. The results are shown in Tab. 8. From the results on four popular point cloud models, it can be seen that when all transformations are rigid, the test accuracy is still the lowest.

C.3 More Results of UMT Against Adaptive Attacks

To further explore the performance of UMT against adaptive attacks, we supplement the experimental results using four types of random augmentations RSHW in Tab. 10, and find that the conclusion is consistent with Tab. 2, i.e. , UMT exhibits a certain degree of robustness against the adaptive attack random transformations.

C.4 More Results of UMT Against Semantic Segmentation

To further evaluate the UMT performance against semantic segmentation, we employ the semantic scene understanding dataset SemanticKITTI [3] in Tab. 11. It can be observed that UMT is still effective against this real-world segmentation dataset.

C.5 Results of UMT Against SE(3) Equivariant Models

In the main text, we have discussed the robustness of UMT against rotation/scaling invariant models. The results for RIConv++ [58] (rotation invariant) and 3DGCN [26] (scaling invariant) networks in Tab. 1, as well as RIConv [57], LGR-Net [59] (rotation invariant), and 3DGCN in Tab. 4, demonstrating that these invariant networks can indeed defend against class-wise rotation and scaling. It is worth noting that these networks, which are invariant to a single transformation, cannot defend against UMT formed by a combination of multiple transformations. Thus, it appears that networks like SE(3) equivariant model, which are invariant to multiple transformations in space, can poten-

tially overcome UMT. Therefore, we include experimental results for the SE(3) equivariant network SE(3)-Transformer [11] in Tab. 12.

However, it can be seen that SE(3)-Transformer cannot defend against UMT (k=3, RSW ) and UMT(k=4, RSWH ). This is because existing transformation-invariant networks, even including SE(3) invariant networks, are designed only for rigid transformations (rotation, scaling, reflection, and translation). There are currently no invariant networks proposed for non-rigid transformations like shear and twisting. Therefore, if a data protector wants UMT to be more robust, they can include non-rigid class-wise transformations to defeat existing rigid transformation-invariant networks.

C.6 Results of Insightful Analysis

We train on clean training set and test on clean test set, train on clean training set and test on UMT test set, train on UMT training set and test on clean test set, and train on UMT training set and test on UMT test set to obtain the test accuracy results in Tab. 9 (we ensure that the UMT parameters for the UMT test set and UMT training set are consistent). Higher accuracy values indicate lower cross-entropy loss values, while lower accuracy values represent higher cross-entropy loss values. It can be observed that only when trained with the UMT training set, the loss after testing with the clean test set is high ( i.e. , low accuracy).

C.7 Boarder Hyper-Parameter Analysis

Additionally, we investigate the unlearnable effects across a wider range of hype-parameters r s , r p , b l , b u in UMT ( k = 2 with RS ), as shown in Tab. 13. It can be observed that our UMT scheme still exhibits a good unlearnable effect, and its key lies in the crucial role played by the class-wise setting.

C.8 Supplementary Ablation Results

In this section, we conduct ablation experiments on the UMT scheme on the ShapeNetPart dataset (keeping experimental parameters consistent with the main experiments), as shown in Tab. 14. It can be observed that whether using only the rotation module or only the scaling module, the final unlearnable effect is not as good as UMT. This clearly demonstrates that each module contributes to the overall UMT effectiveness.

C.9 Results of Mixture of Class-wise and Sample-wise Samples

In this section, we investigate the test accuracy results when using a mixture of different ratios of classwise samples and sample-wise (random) samples in the dataset, as shown in Tab. 15 (experiments are conducted on the ModelNet10 dataset with 10 categories, where '2 class-wise 8 sample-wise" denotes that samples from 2 categories undergo class-wise transformations, while the remaining 8 categories undergo random transformations, and so on). We can clearly observe from the experimental results that as the proportion of class-wise transformations gradually increases, the test accuracy gradually

decreases, indicating that the unlearnable effect becomes more pronounced. This strongly suggests that the class-wise setting is more effective than the sample-wise setting.

[𝟒𝟒. 𝟕𝟕

°

, 𝟎𝟎. 𝟗𝟗

°

,

𝟗𝟗𝟕𝟕. 𝟑𝟑

°

, 𝟎𝟎. 𝟕𝟕𝟕𝟕]

[𝟕𝟕. 𝟏𝟏

°

, 𝟑𝟑. 𝟏𝟏

°

,

𝟕𝟕𝟎𝟎. 𝟕𝟕

°

, 𝟎𝟎. 𝟖𝟖]

[𝟏𝟏𝟏𝟏. 𝟕𝟕

°

, 𝟖𝟖. 𝟕𝟕

°

,

𝟏𝟏𝟒𝟒. 𝟗𝟗

°

, 𝟎𝟎. 𝟕𝟕𝟕𝟕]

Clean samples

UMT

samples

UMT

samples

UMT

samples

ModelNet10

[

𝟎𝟎

.

𝟗𝟗

°

,

𝟔𝟔

.

𝟐𝟐

°

,

𝟗𝟗𝟗𝟗

.

𝟑𝟑

°

,

𝟎𝟎

.

𝟕𝟕𝟕𝟕

]

[

𝟗𝟗𝟎𝟎

.

𝟗𝟗

°

,

𝟑𝟑

.

𝟕𝟕

°

,

𝟕𝟕𝟗𝟗

.

𝟗𝟗

°

,

𝟎𝟎

.

]

[

𝟗𝟗𝟕𝟕

.

𝟔𝟔

°

,

𝟓𝟓

.

𝟗𝟗

°

,

𝟑𝟑𝟔𝟔

.

𝟗𝟗

°

,

𝟎𝟎

.

𝟕𝟕𝟗𝟗

]

Clean samples

UMT

samples

UMT

samples

UMT

samples

ModelNet40

[

𝟒𝟒

.

𝟔𝟔

°

,

𝟏𝟏𝟏𝟏

.

𝟕𝟕

°

,

𝟐𝟐𝟐𝟐

.

𝟒𝟒

°

,

𝟏𝟏

.

𝟔𝟔𝟐𝟐

]

[

𝟓𝟓

.

𝟕𝟕

°

,

𝟒𝟒

.

𝟖𝟖

°

,

𝟑𝟑𝟔𝟔

.

𝟒𝟒

°

,

𝟏𝟏

.

𝟔𝟔𝟏𝟏

]

[

𝟖𝟖

.

𝟏𝟏

°

,

𝟏𝟏𝟏𝟏

.

𝟖𝟖

°

,

𝟒𝟒𝟐𝟐

.

𝟓𝟓

°

,

𝟏𝟏

.

𝟕𝟕𝟒𝟒

]

Clean samples

UMT

samples

UMT

samples

UMT

samples

C.10 Additional Visual Presentations for 3D Point Cloud Samples

Scan We visualize clean point cloud samples and UMT ( k = 2 using RS ) point cloud samples on four benchmark datasets ModelNet10 [50], ModelNet40 [50], ScanObjectNN [41], and ShapeNetPart [4],

Benchmark datasets

Modules

↓

Models

-→

PointNet

PointNet++

DGCNN

PointCNN

PCT

AVG

ShapeNetPart [4]

Rotation module

53.51

44.05

57.69

49.48

43.84

49.71

Scaling module

28.74

77.21

37.93

44.02

51.84

47.95

UMT(k=2)

15.14

41.16

26.17

32.36

44.05

31.78

[

𝟓𝟓

.

𝟕𝟕

°

,

𝟓𝟓

.

𝟏𝟏

°

,

𝟏𝟏𝟏𝟏

.

𝟗𝟗

°

,

𝟎𝟎

.

𝟔𝟔𝟔𝟔

]

[

𝟔𝟔

.

𝟔𝟔

°

,

𝟏𝟏𝟏𝟏

.

𝟕𝟕

°

,

𝟕𝟕𝟔𝟔

.

𝟏𝟏

°

,

𝟎𝟎

.

𝟔𝟔𝟓𝟓

]

[

𝟕𝟕

.

𝟏𝟏

°

,

𝟏𝟏𝟏𝟏

.

𝟏𝟏

°

,

𝟗𝟗𝟕𝟕

.

𝟓𝟓

°

,

𝟎𝟎

.

𝟖𝟖𝟎𝟎

]

Clean samples

UMT

samples

UMT

samples

UMT

samples

Shape as depicted in Figs. 7 to 10, respectively. The UMT parameters consist of rotation and scaling parameters [ α , β , γ , λ ]. It can be observed that these unlearnable samples exhibit similar feature information to normal samples, presenting good visual effects and making it difficult to be detected as abnormalities.

D Proofs for Theories

D.1 Proof for Lemma 3

Lemma 3. The unlearnable dataset D u generated using UMT on D c can also be represented using a GMM, i.e., D u ∼ N ( y T y µ, λ 2 y I ) .

Proof: Assuming y = 1 , then D c 1 ∼ N ( µ, I ) , and we have

Similarly, assuming y = -1 , we can obtain

Thus we have: D u ∼ N ( y T y µ, λ 2 y I ) .

D.2 Proof for Lemma 4

Lemma 4. The Bayes optimal decision boundary for classifying D u is given by P u ( x ) ≡ A x ⊤ x + B ⊤ x + C = 0 , where A = λ -2 -1 -λ -2 1 , B = 2( λ -2 -1 T -1 + λ -2 1 T 1 ) µ , and C = ln | λ 2 -1 I | | λ 2 1 I | .

Proof: At the optimal decision boundary the probabilities of any point x ∈ R d belonging to class y = 1 and y = -1 modeled by D u are the same. Similar to the optimal decision boundary of the clean dataset D c , we have:

where A = λ -2 -1 -λ -2 1 , B = 2( λ -2 -1 T -1 + λ -2 1 T 1 ) µ , and C = ln | λ 2 -1 I | | λ 2 1 I | . Besides, note that if P u ( x ) is less than 0, the category of the Bayesian optimal classification is -1; otherwise, it is 1.

D.3 Proof for Lemma 5

Lemma 5. Let z ∼ N (0 , I ) , Z = z ⊤ z + b ⊤ z + c , and ∥·∥ 2 denote 2-norm of vectors. For any t ≥ 0 and γ ∈ R , we use Chernoff bound to have:

Proof: Since A is a constant, we have:

where b = B A , c = C A . Let Z = z ⊤ z + b ⊤ z + c and z ∼ N (0 , I ) ⊂ R d . Thus we have:

For any t ≥ 0 and x ∼ N (0 , I ) , we write the moment generating function for a quadratic random variable Y = x ⊤ x as 5 :

After that, we employ Chernoff bound, for some γ , we have:

D.4 Proof for Theorem 6

Theorem 6. For any constant t 1 and t 2 satisfying 0 ≤ t 1 < 1 2 and 0 ≤ t 2 < 1 2 , the accuracy of the unlearnable decision boundary P u on the dataset D c can be upper-bounded as:

̸

Furthermore, if µ ⊤ µ + b ⊤ µ + c + d < 0 and -µ ⊤ µ + b ⊤ µ -c -d < 0 , we have τ D c ( P u ) < 1 . Moreover, for any µ = 0 ∃ transformation matrix T i such that τ D c ( P u ) < τ D c ( P ) .

Proof: We note that if P u ( x ) is less than 0, the category of the Bayesian optimal classification is -1; otherwise, it is 1. Here, x = yµ + z where z ∼ N (0 , I ) and y ∈ {± 1 } since ( x, y ) ∼ D c .

We can see that:

Applying Lemma 5, with γ = γ 1 , t = t 1 for the computation of p 1 , as well as γ = γ 2 and t = t 2 for the computation of p 2 , where t 1 and t 2 are specific non-negative constants, we obtain:

This provides us with the upper bound for τ D c ( P u ) . Nonetheless, to ensure that this upper bound is less than 1, additional conditions need to be affirmed. As γ 1 and γ 2 increase, the values of p 1 and p 2 diminish ( p 1 > 0 , p 2 > 0 ), and as γ 1 increases, γ 2 decreases (since γ 1 + γ 2 = -2 b ⊤ µ ). We let || b +2 µ || 2 2 2 equal to α 1 ≥ 0 , µ ⊤ µ + b ⊤ µ + c (also equals to || µ || 2 2 + c -γ 1 + γ 2 2 ) equal to β 1 , resulting in:

Making | (1 -2 t 1 ) I | 1 2 meaningful requires satisfying the following condition:

We note that p 1 ( t 1 = 0) = 1 2 , p 1 ( t 1 → 1 2 ) = + ∞ . When we set d p 1 d t 1 = 0 , we obtain that:

Explanation of the Last Inequality. Assuming we define s ( α 1 ) = d 2 2 α 1 + α 1 2 . The minimum value of this function can be obtained using the Arithmetic Mean-Geometric Mean Inequality (AM-GM Inequality), which states that a + b ≥ 2 √ ab for a, b ≥ 0 . Thus, s ( α 1 ) ≥ 2 √ d 2 2 α 1 · α 1 2 = d . Since β 1 ≤ s ( α 1 ) , we can infer that β 1 ≤ s ( α 1 ) min, which means β 1 ≤ d .

.

The product of the roots of the equation is: t 11 t 12 = d + β 1 2(2 β 1 -α 1 ) (we assume that t 11 < t 12 ). We let f ( t 1 ) = 2(2 β 1 -α 1 ) t 2 1 +2( α 1 -2 β 1 -d ) t 1 + d + β 1 . Thus we have f (0) = d + β 1 , f ( 1 2 ) = α 1 2 > 0 Situation (i): β 1 < -d . At this time, f (0) < 0 , t 11 t 12 > 0 , based on the trend of quadratic functions and the distribution of roots, we can infer that 0 < t 11 < 1 2 , t 12 > 1 2 , and 2 β 1 < α 1 . Thus we conclude that there exists t such that p ( t = t ) < 1 .

11 1 1 11 2

Situation (ii): -d < β 1 < 0 . At this time, f (0) > 0 , t 11 t 12 < 0 , based on the trend of quadratic functions and the distribution of roots, we also can infer that t 11 < 0 , t 12 > 1 2 , and 2 β 1 < α 1 . Thus we have that p 1 ≥ 1 2 .

Situation (iii): 0 < β 1 < d . At this time, f (0) > 0 , the sign of t 11 t 12 simultaneously determines the direction of the opening of the quadratic function f ( t 1 ) . When t 11 t 12 < 0 , t 11 < 0 , t 12 > 1 2 , thus we have p 1 ≥ 1 2 ; when t 11 t 12 > 0 , 0 < t 11 < t 12 < 1 2 , the minimum point of p 1 is at p 1 ( t 12 ) . However, it is challenging to compare p 1 ( t 12 ) and 1 2 to determine which is greater or smaller.

Similarly for p 2 , we let || b -2 µ || 2 2 2 equal to α 2 > 0 , -µ ⊤ µ + b ⊤ µ -c -2 d equal to β 2 , resulting in:

Thus we have the similar situation, i.e. , β 2 < -d . At this time, f (0) < 0 , t 21 t 22 > 0 , based on the trend of quadratic functions and the distribution of roots, we can infer that 0 < t 21 < 1 2 , t 22 > 1 2 , and 2 β 2 < α 2 . Thus we conclude that there exists t 21 such that p 2 ( t 2 = t 21 ) < 1 2 .

Taking into account the above situations, we have that: when β 1 < -d , β 2 < -d (i.e., µ ⊤ µ + b ⊤ µ + c + d < 0 and -µ ⊤ µ + b ⊤ µ -c -d < 0 ), there exists t 11 and t 21 respectively, making p 1 < 1 2 , p 2 < 1 2 , i.e., p 1 + p 2 < 1 , where t 11 = 1 2 + d -√ α 2 1 -2 α 1 β 1 + d 2 2(2 β 1 -α 1 ) , t 21 = 1 2 + d -√ α 2 2 -2 α 2 β 2 + d 2 2(2 β 2 -α 2 ) .

̸

We know that for µ = 0 , τ D c ( P ) = ϕ ( µ ) > 1 2 . Therefore, we need to introduce additional conditions to further ensure that p 1 < 1 4 , and p 2 < 1 4 , and consequently τ D c ( P u ) = p 1 + p 2 < 1 2 < τ D c ( P ) .

To let p 1 satisfy p 1 < 1 4 ( α 1 > 0 , β 1 < -d , and 0 ≤ t 1 < 1 2 ), that is,

We assume that g ( t ) = α 1 t 2 1 -2 t + β 1 t -d 2 ln (1 -2 t ) . Let us assume α 1 = 1 2 and d = 3 for 3D point cloud data, then β 1 < -3 . Upon analyzing the function g ( t ) , we observe that as β 1 decreases, the minimum value of g ( t ) also decreases. We utilize various β 1 values and their corresponding function value to provide a more intuitive understanding as shown in Tab. 16.

̸

As for p 2 , since α 2 = α 1 , we need to reselect appropriate values to demonstrate the existence of p 2 < 1 4 . Similarly, to let p 2 satisfy p 2 < 1 4 ( α 2 > 0 , β 2 < -d , and 0 ≤ t 2 < 1 2 ), that is,

We assume that h ( t ) = α 2 t 2 1 -2 t + β 2 t -d 2 ln (1 -2 t ) . Let us assume α 2 = 1 3 and d = 3 , similarly, we utilize various β 2 values and their corresponding function value to provide a more intuitive understanding as shown in Tab. 17.

̸

Therefore, we conclude that there exist α 1 , β 1 , t 1 such that p 1 < 1 4 , α 2 , β 2 , t 2 such that p 2 < 1 4 , i.e. , p 1 + p 2 < 1 2 . At the same time, we observe that a smaller value of β makes it easier to satisfy the above conditions, i.e. , the more negative β 1 and β 2 are, the more likely it is to satisfy the above conditions. We formally combine and assert these conditions as, b ⊤ µ ≪ 0 , i.e. , µ ⊤ λ -2 -1 T ⊤ -1 + λ -2 1 T ⊤ 1 λ -2 -1 -λ -2 1 µ ≪ 0 (we sufficiently support this condition in the empirical results from Tabs. 16 and 17). Thus we conclude that for any µ = 0 ∃ transformation parameters T i such that τ D c ( P u ) < τ D c ( P ) .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the scope of the contributions made in the paper regarding the first proposed unlearnable scheme UMT and the data restoration scheme, as well as the theoretical and experimental evaluations.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discussed the limitations of the work in Sec. 6.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper clearly and accurately provides the assumptions and proofs for each theory.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The formulas in the methods section, the schematic diagrams of the schemes, Algorithm 1, as well as the details of the training process and descriptions of the datasets and models together support the reproducibility of the experimental results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper has open-sourced code, which are sufficient to reproduce the experimental results.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper provides sufficient training and testing details, which are adequate for understanding the experimental results.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper conducted experiments with three different random seeds, studying the results in terms of statistical significance.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper clearly specifies the parameters of the server used for the experiment and the code execution environment.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: After a thorough review of the paper, no violations of the NeurIPS Code of Ethics are found.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work in Sec. 6.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: The paper described safeguards that have been put in place for responsible release of data in Sec. 6.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The data used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces UMT (Unlearnable Multi-Transformations), the first approach designed to render 3D point cloud data unlearnable for unauthorized deep learning models, by applying class-wise transformations. It also presents a data restoration scheme to enable authorized users to effectively train on the unlearnable data. Theoretical analysis and extensive experiments on various datasets and models demonstrate the effectiveness of UMT in safeguarding sensitive 3D data while allowing authorized access for legitimate training purposes. The main contributions include the proposal of the first 3D unlearnable scheme, a novel data restoration approach, theoretical insights into the unlearnability mechanism, and empirical validation of UMT's superiority.\n\nSTRENGTHS:\n1. The paper is clearly written, making complex concepts easy to understand.\n2. Simple and Effective Unlearnable Scheme: The paper proposes a straightforward unlearnable scheme that leverages the characteristics of 3D data. By combining four types of 3D transformations, it effectively protects 3D point clouds data. Additionally, it allows authorized users to restore the transformed dataset using inverse transformations.\n3. The extensive experiments conducted in the paper demonstrate the effectiveness of the proposed method. The UMT dataset successfully prevents unauthorized users from achieving high performance on the target test set, thereby protecting the data.\n4. The method provides theoretical proof, which enhances the credibility and robustness of the approach.\n\nWEAKNESSES:\n1.\tThe experiments do not investigate the impact on UMT dataset performance when mixed with other datasets. Currently, transformations are applied to the entire training set, altering the distribution between the UMT training and test sets. If only part of the training set is UMT data, can the performance on the UMT test set still be maintained?\n2.\tThe experiments rely on randomness. Are the results averaged over multiple runs to ensure reliability? While the appendix provides experiments with different random seeds, it does not show the variance of results in other experiments.\n3.\tThe proposed transformation patterns, such as rotation and scaling, are easily detectable by comparing the UMT training set and test set. This predictability might allow unauthorized models to adapt to these transformations, potentially compromising the method's effectiveness.\n\nQUESTIONS:\nThe questions have been listed in the Weaknesses. Please check the weaknesses.\n\nLIMITATIONS:\nThe authors have discussed the limitations and Broader Impacts in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the protection scheme against unauthorized learning on 3D point cloud data. A simple class-wise transformation method is designed to mislead the model to learn the transformation patterns of points instead of categorical knowledge. The method is evaluated on popular used 3D datasets and models.\n\nSTRENGTHS:\n- This paper studies a new problem and presents a reasonable and simple method to tackle the problem.\n\n- Extensive experiments are conducted. I appreciate the comprehensive ablation studies and analyses provided in the paper.\n\nWEAKNESSES:\n- I am concerned about the value of the settings studied in the paper. Most classification datasets considered in the paper are used to develop 3D modeling algorithms and are usually far from real-world 3D/point cloud applications that require high safety standards like 3D face recognition and scene-level point clouds for autonomous driving. These scenarios usually have much more categories (e.g., facial ids) or cannot be modeled as a classification problem (e.g., scene understanding for autonomous driving). Can the proposed method be easy to transfer to these settings? \n\n- How about the results if we randomly apply R, S, W, H when we train the model? Such strong data augmentation may lead to lower classification accuracy but the model may achieve >30% test accuracy on the UMT datasets.\n\n- Rotation/scale/SE(3) equivariant 3D modeling has been extensively studied in previous work [r1-r4].  Will the proposed method fail if we use these models for 3D modeling?\n\n[r1] SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks, NeurIPS 2020\n\n[r2] Vector neurons: A general framework for so (3)-equivariant networks, ICCV 2021\n\n[r3] E2PN: Efficient SE(3)-Equivariant Point Network, CVPR 2023\n\n[r4] Scale-Equivariant Deep Learning for 3D Data\n\nQUESTIONS:\n- The title \"Class-wise Transformation Is All You Need\" is quite general, which may not be helpful for readers to understand the topic studied in the paper. I was a bit confused when I first read the title along with the abstract. \n\n\nOverall, this paper studies a new problem and proposes a simple approach against unauthorized learning on 3D point cloud data.  Extensive experiments and analyses are provided to evaluate the method. However, I still have concerns about the value of the settings considered in the paper and the robustness of the proposed method. I am not an expert on the topic (safety in machine learning), thus I would like to rate this paper as borderline and wait for further discussions.\n\nLIMITATIONS:\nThe limitations and broader impacts of the method have been discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAfter reading the author’s response, I would like to increase my evaluation to borderline accept.\n\n—\n\nThis paper addresses a critical issue by extending unlearnable strategies to 3D point cloud data, introducing the Unlearnable Multi-Transformations (UMT) approach. The use of a category-adaptive allocation strategy and multiple transformations is innovative and well-conceived. Notably, the paper highlights a gap in existing literature by acknowledging the challenges even authorized users face in learning from unlearnable data, proposing a data restoration scheme to address this. The theoretical and empirical validation across six datasets, sixteen models, and two tasks convincingly demonstrates the framework's effectiveness.\n\nSTRENGTHS:\n1. This paper studies an interesting and important problem. The problem is realistic but not widely explored.\n2. The experiments of this work are comprehensive, which is admirable.\n\nWEAKNESSES:\n1. The presentation and writing need to be polished to reach the acceptance bar. For the current form, there are a series of unclear descriptions and explanations.\n2. The theoretical contributions are overall weak, which provides limited insights to the research community. Besides, some theoretical analysis needs to be justified.\n\nQUESTIONS:\n1. The title of this paper is somewhat overclaimed or exaggerated. It is not related to the research topic of this work, which is also less informative.\n2. In the introduction, the paper analyzes three issues and challenges of the research problem. However, there is no intuition about why the proposed method can handle the issues and challenges. There are just some technical details. Could the paper supplement more intuitions for a better understanding of reviewers?\n3. In the introduction, there is an understanding gap between the method descriptions and theoretical analysis. Some definitions such as the decision boundary of the Bayes classifier in the Gaussian Mixture Model, are very strange, and cannot describe the work principle of the proposed method.\n4. For the data protector and authorized user, do we need to add some noise to implement them with the minimax optimization? More details are needed. Besides, are they mutually reversible in practice?\n5. For Eq. (3), is there some evidence about the choice of $\\mathcal{A}_N$?\n6. As for Property 2, why do we need all four transformation matrices we employ and the multiplicative combinations of any of these matrices are all invertible matrices? I can understand the theoretical analysis. However, it seems that this does not work for the main conclusion of this paper.\n7. Similarly, the bounds in Lemma 5 and Theorem 6 are very loose. With them, it is hard to believe the proposed method can work well (although the experimental results are good). More discussions are needed.\n8. The proof of Lemma 3 is simple since the space of $y$ is limited to 2. When the components of GMMs are more than 2, will the claim still hold? Could the paper add some discussions about this?\n9. From Line 782 to Line 783, could the paper supplement some details to describe why the last line of inequalities holds?\n10. For Line 811, the paper assumes that $\\alpha_2=\\frac{1}{3}$ and $d=3$. Will the assumption make the theory less general? More discussions are also needed.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach called Unlearnable Multi-Transformations (UMT) to make 3D point cloud data unlearnable by unauthorized users. This method employs a category-adaptive allocation strategy to apply class-wise transformations, thereby preventing unauthorized training on the data. Additionally, the authors propose a data restoration scheme that allows authorized users to learn from the unlearnable data effectively. The effectiveness of UMT is validated through theoretical analysis and extensive experiments on multiple datasets and models, demonstrating its potential in protecting sensitive 3D point cloud data from unauthorized exploitation while enabling authorized access.\n\nSTRENGTHS:\n+ A novel unlearnable scheme specifically designed for 3D point cloud data is proposed, and it addresses the issue of enabling authorized users to utilize the protected data effectively.\n+ The proposed method seems reasonable, and it has been theoretically demonstrated that the classification accuracy is lower than that of the clean dataset under the Bayes classifier's decision boundary in the Gaussian Mixture Model.\n+ Extensive experiments on three synthetic and three real-world datasets, using 16 widely adopted point cloud model architectures for classification and semantic segmentation tasks, verify the superiority of the proposed method.\n\nWEAKNESSES:\n- Are object categories sensitive to the combination of transformations? For instance, are there categories where using two different sets of transformations results in significant differences in outcomes? I'm not sure if I've missed something or not, but conducting multiple experiments to observe performance and its variance might be a more reasonable approach.\n- It's still not very clear how the point cloud segmentation experiments were conducted. Each object category would be in different regions of the point cloud, so would transformations be applied to the corresponding regions based on the GT? If so, would these transformations and the subsequent inverse transformations affect the segmentation results for authorized users?\n\nQUESTIONS:\nSee the weakness\n\nLIMITATIONS:\nYes, the authors mention some limitations of the proposed method, such as its vulnerability to rotation and scale transformations. However, they also note that other types of transformations are not easily compromised in the current context.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}