Unlearnable 3D Point Clouds: Class-wise Transformation Is All You Need

Authors: xianlong  wang
minghui  li
wei  liu
hangtao  zhang
shengshan  hu
yechao  zhang
ziqi  zhou
hai  jin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:56:34

Abstract: Traditional unlearnable strategies have been proposed to prevent unauthorized users from training on the 2D image data. With more 3D point cloud data containing sensitivity information, unauthorized usage of this new type data has also become a serious concern. To address this, we propose the first integral unlearnable framework for 3D point clouds including two processes: (i) we propose an unlearnable data protection scheme, involving a class-wise setting established by a category-adaptive allocation strategy and multi-transformations assigned to samples; (ii) we propose a data restoration scheme that utilizes class-wise inverse matrix transformation, thus enabling authorized-only training for unlearnable data. This restoration process is a practical issue overlooked in most existing unlearnable literature, i.e., even authorized users struggle to gain knowledge from 3D unlearnable data. Both theoretical and empirical results (including 6 datasets, 16 models, and 2 tasks) demonstrate the effectiveness of our proposed unlearnable framework. Our code is available at https://github.com/CGCL-codes/UnlearnablePC.

Main paper: 
 introduction: 
 Recently, 3D point cloud deep learning has been making remarkable strides in various domains, e.g., self-driving [6] and virtual reality [1, 46] . Specifically, numerous 3D sensors scan the surrounding environment and synthesize massive 3D point cloud data containing sensitive information such as pedestrian and vehicles [32] to the cloud server for deep learning analysis [12, 23] . However, the raw point cloud data can be exploited for point cloud unauthorized deep learning if a data breach occurs, posing a significant privacy threat. Fortunately, the privacy protection approaches for preventing unauthorized training have been extensively studied in the 2D image domain [9, 19, 28, 29, 39, 42] . They apply elaborate perturbations on images such that trained networks over them exhibit extremely low generalization, thus failing to learn knowledge from the protected data, known as "making data unlearnable". Nonetheless, the stark disparity between 2D images and 3D point clouds poses significant challenges for drawing lessons from existing 2D solutions. Specifically, migrating 2D unlearnable schemes to 3D suffers from following challenges: (i) Incompatibility with 3D data. Numerous model-agnostic 2D image unlearnable schemes operate in the pixel space, such as convolutional operations [39, 42] , making them fail to be directly transferred to the 3D point space. (ii) Poor visual quality. Migrating model-dependent 2D unlearnable methods [5, 9, 19, 28] to 3D point clouds requires perturbing substantial points, leading to irregular three-dimensional shifts which may significantly degrade visual quality. Hence, these challenges spur us to start directly from the characteristics of point clouds for proposing 3D unlearnable solutions.Recent works observe that 3D transformations can alter test-time results of models [7, 17, 44] . To explore this, we conduct an in-depth investigation into the properties of seven 3D transformations as shown in Fig. 1 and reveal the mechanisms by which transformations employed in a certain pattern serve as unlearnable schemes (Sec. 3.2). In light of this, we propose the first unlearnable approach in 3D point clouds via multi class-wise transformation (UMT), transforming samples to various forms for privacy protection. Concretely, we newly propose a category-adaptive allocation strategy by leveraging uniform distribution sampling and category constraints to establish a class-wise setting, thereby multiplying multi-transformations to samples based on categories. To theoretically analyze UMT, we define a binary classification setup similar to that used in [20, 33, 39] . Meanwhile, we employ a Gaussian Mixture Model (GMM) [38] to model the clean training set and use the Bayesian optimal decision boundary to model the point cloud classifier. Theoretically, we prove that there exists a UMT training set follows a GMM distribution and the classification accuracy of UMT dataset is lower than that of the clean dataset in a Bayesian classifier.Moreover, an incompatible issue in existing unlearnable works [9, 19, 28, 29, 39, 43] is identified [54] ,i.e., these approaches prevent unauthorized learning to protected data, but they also impede authorized users from effectively learning from unlearnable data. To address this, we propose a data restoration scheme that applies class-wise inverse transformations, determined by a lightweight message received from the protector. Our proposed unlearnable framework including UMT approach and data restoration scheme is depicted in Fig. 3 .Extensive experiments on 6 benchmark datasets (including synthetic and real-world datasets) using 16 point cloud models across CNN, MLP, Graph-based Network, and Transformer on two tasks (classification and semantic segmentation), verified the effectiveness of our proposed unlearnable scheme. We summarize our main contributions as follows:• The First Integral 3D Unlearnable Framework. To the best of our knowledge, we propose the first integral unlearnable 3D point cloud framework, utilizing class-wise multitransformation as its unlearnable mechanism (effectively safeguarding point cloud data against unauthorized exploitation) and proposing a novel data restoration approach that leverages class-wise reversible 3D transformation matrices (addressing an incompatible issue in most existing unlearnable works, where even authorized users cannot effectively learn knowledge from unlearnable data).• Theoretical Analysis. We theoretically indicate the existence of an unlearnable situation that the classification accuracy of the UMT dataset is lower than that of the clean dataset under the decision boundary of the Bayes classifier in Gaussian Mixture Model.• Experimental Evaluation. Extensive experiments on 3 synthetic datasets and 3 real-world datasets using 16 widely used point cloud model architectures on classification and semantic segmentation tasks verify the superiority of our proposed schemes. 
preliminaries: 
 Notation. Considering the raw point cloud data (X, Y) ∈ X × Y sampled from a clean distribution D for training a point cloud network, the user's goal is to obtain a model F : X → Y by minimizing the loss function (e.g., cross-entropy loss) L(F(X), Y). Let T ∈ T be a 3D transformation matrix that does not seriously damage the visual quality of point clouds. Note that X ∈ R 3×p , T ∈ R 3×3 , and p represents the number of points. In theoretical analysis, following [20, 33] , we simplify a training dataset D k to a Gaussian Mixture Model (GMM) [38] N (yµ, I), where y ∈ {±1} denotes the class labels, µ ∈ R d denotes the mean value, and I ∈ R d×d denotes the identity matrix. Thus the Bayes optimal decision boundary for classifying D k is defined by P (x) ≡ µ T x = 0. The accuracy of the decision boundary P on D k is equal to ϕ(||µ|| 2 ), where ϕ denotes the Cumulative Distribution Function (CDF) of the standard normal distribution.Data protector G p . G p aims to protect the knowledge from the clean training set (with size of n)D c = {X i , Y i } n i=1∼ D by compromising the unauthorized models who train on the unlearnable point cloud data {T i (X i ), Y i } n i=1 , resulting in extremely poor generalization on the clean test distribution D t ⊆ D. This objective can be formalized as:max E (X,Y)∼Dt L (F (X; θ u ) , Y) , s.t. θ u = arg min θ (Xi,Yi)∈Dc L (F (T i (X i ); θ) , Y i ) (1)where G p assumes that training samples are all transformed into unlearnable ones while maintaining normal visual effects, in line with previous unlearnable works [19, 28, 39, 42] . By the way, solving Eq. ( 1) directly is infeasible for neural networks because it necessitates unrolling the entire training procedure within the inner objective and performing backpropagation through it to execute a single step of gradient descent on the outer objective [8] .Authorized user G a . G a aims to apply another transformation T ′ ∈ T on the unlearnable sample, making the protected data learnable. This is formally defined as:min E (X,Y)∼Dt L (F (X; θ r ) , Y) , s.t. θ r = arg min θ (Xi,Yi)∈Dc L (F (T ′ i (T i (X i )); θ) , Y i ) (2)where G a assumes that, without access to any clean training samples, T ′ can be constructed by utilizing a lightweight message M received from data protectors.3 Our Proposed Unlearnable Schemes 
key intuition: 
 Several recent works [7, 13, 44] reveal employing 3D transformations can mislead the model's classification results. Such a phenomenon implies that there might be some defects in point cloud classifiers when processing transformed samples, leading us to infer that 3D transformations are probable candidates for data protection against unauthorized training. If the transformed point cloud data are used to train unauthorized DNNs, only simple linear features inherent in 3D transformations (at which transformations may act as shortcuts [14] ) are captured by the DNNs, successfully protecting point cloud data privacy. We summarize existing 3D transformations in Fig. 1 and formally define them in Appendix A. To seek clarity on the application and selection of transformations, we explore three aspects: (i) execution mechanism, (ii) exclusion mechanism, and (iii) working mechanism as follows.(i) Which execution mechanism successfully satisfy Eq. ( 1)? The extensively employed execution patterns in 2D unlearnable approaches are sample-wise [9, 19] and class-wise [19, 39] settings. We further complement the dataset-wise setting (using universal transformation) and implement the above execution mechanisms for training a PointNet classifier [35] on the transformed ModelNet10 [50] , obtaining test accuracy results in Fig. 2 (a) . We discover that model achieves considerably low test accuracy under the class-wise setting, satisfying Eq. ( 1). Sample-wise and dataset-wise settings do not obviously compromise model performance, which cannot serve as promising unlearnable routes. Moreover, we note that sample-wise transformation is often considered as a data augmentation scheme to improve generalization, which contradicts our aim of using class-wise transformation to lower model generalization.(ii) Which transformations need to be excluded? Not all transformations are suitable candidates. We exclude three transformations, tapering, reflection, and translation. (1) The tapering matrix may cause point cloud samples to become a planar projection when ηz defined in Eq. ( 18) equals to -1, rendering the tapered samples meaningless; (2) The reflection matrix has only three distinct transformation matrices, rendering it incapable of assigning class-wise transformations when the number of categories exceeds three; (3) The translation transformation is a straightforward and simple additive transformation that is too easily defeated by point cloud data pre-processing approaches.(iii) Why does class-wise transformation work? We conduct experiments using class-wise transformations (see Tab. 6), indicating that the model training on the class-wise transformed training set achieves a relatively high accuracy on the class-wise transformed test set (using the same transformation process as the training set). Besides, if we permute the class-wise transformation for the test set, we obtain a significant low accuracy on the test set. Therefore, we conclude that the reason why class-wise transformation works is that the model learns the mapping between class-wise transformations and corresponding category labels as shown in Fig. 2 (b) , which results in the model being unable to predict the corresponding labels on a clean test set lacking transformations. This analytical process yields conclusions that are in agreement with prior research [39, 44] .We assign transformation parameters based on categories to realize class-wise setting. For rotation transformation R ∈ R 3×3 , we refer to α and β as slight angles imposed on the x and y axes, γ as the primary angle for z axis. We generate random angles for A N times in three directions:EQUATIONwhere U denotes uniform distribution, N denotes the number of categories, r s is a small range that controls α and β, while r p is a large range that controls γ. A N is computed in such a way to ensure that the number of combinations of three angles is greater than or equal to N . Concretely, in the rotation operation, each of the three directions has A N distinct angles, which means that the final rotation matrix has A 3 N possible combinations. To satisfy the class-wise setup, A 3 N must be at least N , requiring A N to be no less than 3 √ N . Finally, we randomly select N combinations of angles for the allocation. The scaling transformation S ∈ R 3×3 resizes the position of each point in the 3D point cloud sample by a certain scaling factor λ, which is sampled N times from a uniform distribution U:EQUATIONwhere b l and b u represent the lower bound and upper bound of the scaling factor, respectively.For shear H ∈ R 3×3 defined in Eq. ( 13), twisting W ∈ R 3×3 defined in Eq. ( 16), the process of generating parameters within ranges (ω l , ω u ) and (h l , h u ) is consistent to scaling. The range of these parameters ensures the visual effect of the sample. Property 1. Since rotation matrices R α , R β , and R γ around three directions are all orthogonal matrices, R is also an orthogonal matrix, which can be defined as:EQUATION)where we can determine the orthogonality by matrix multiplication through the definitions of Eq. (11) .Since R = R α R β R γ and RR T = R α R β R γ R T γ R T β R T α = I,so R is also an orthogonal matrix. Property 2. All four transformation matrices we employ, R, S, W, and H, and the multiplicative combinations of any these matrices are all invertible matrices, which can be formally defined as:EQUATIONwhere the inverse matrices of R, S, W, and H are given in Appendix A. This property allows the authorized users to normally train on the protected data due to that multiplying a matrix by its inverse results in the identity matrix, leading us to propose a data restoration scheme in Sec. 3.4.Assuming the point cloud training set D c is defined as {(X c1 , Y i ), (X c2 , Y i ), ..., (X cni , Y i )} N i=1 , where n 1 , n 2 , ..., n N represent the number of samples in the 1st, 2nd, ..., N -th category, respectively. We formally define the spectrum of transformations as k to indicate the number of transformations involved. Thus the ultimate unlearnable transformation matrix T k is defined as:EQUATIONOnce we employ the proposed category-adaptive allocation strategy to T k , the unlearnable point cloud dataset D u is constructed as:EQUATIONOur proposed UMT scheme is described in Algorithm 1. We enumerate possible transformations in Eq. ( 7) to obtain the unlearnability in Tab. 7 and select one type of class-wise transformation for each k for more comprehensive experiments in Tab. 1. To facilitate the theoretical study of UMT2 , we opt for RS as the transformation matrix T, which achieves the best unlearnable effect as suggested in Tabs. 1 and 7. Thus, in the GMM scenario, the class-wise transformation matrix is defined as T y = R y S y = λ y R y ∈ R d×d , where λ y ∈ R is the scaling factor.Lemma 3. The unlearnable dataset D u generated using UMT on D c can also be represented using a GMM, i.e., D u ∼ N (yT y µ, λ 2 y I). Proof: See Appendix D.1. Lemma 3 demonstrates that the unlearnable dataset D u can also be represented as a GMM, which is derived from Property 1.Lemma 4. The Bayes optimal decision boundary for classifying D u is given by Pu (x) ≡ Ax ⊤ x + B ⊤ x + C = 0, where A = λ -2 -1 -λ -2 1 , B = 2(λ -2 -1 T -1 + λ -2 1 T 1 )µ, and C = ln |λ 2 -1 I| |λ 2Proof: See Appendix D.2. Lemma 4 reveals that the Bayesian decision boundary for classifying D u is a quadratic surface based on the GMM expression of D u .Lemma 5. Let z ∼ N (0, I), Z = z ⊤ z + b ⊤ z + c, where b = B A , c = CA , and ∥•∥ 2 denote 2-norm of vectors. For any t ≥ 0 and γ ∈ R, we employ Chernoff bound to have:P{Z ≥ E[Z] + γ} ≤ exp t 2 2(1-2t) ||b|| 2 2 -t(γ + d) |(1 -2t)I| 1 2Proof: See Appendix D.3. Lemma 5 enables us to establish an upper bound on the accuracy of the unlearnable decision boundary P u applied to the clean dataset D c , denoted as τ Dc (P u ), as presented in Theorem 6 below. Theorem 6. For any constant t 1 and t 2 satisfying 0 ≤ t 1 < 1 2 and 0 ≤ t 2 < 1 2 , the accuracy of the unlearnable decision boundary P u on D c can be upper-bounded as:τ Dc (P u ) ≤ exp t 2 1 2(1-2t1) ||b + 2µ|| 2 2 + t 1 (µ ⊤ µ + b ⊤ µ + c) 2|(1 -2t 1 )I| 1 2 + exp t 2 2 2(1-2t2) ||b -2µ|| 2 2 -t 2 (µ ⊤ µ -b ⊤ µ + c + 2d) 2|(1 -2t 2 )I| 1 2 := p 1 + p 2 Furthermore, if µ ⊤ µ + b ⊤ µ + c + d < 0 and -µ ⊤ µ + b ⊤ µ -c -d < 0, we have τ Dc (P u ) < 1.Moreover, for any µ ̸ = 0, ∃ matrix T i such that τ Dc (P u ) < τ Dc (P ), where P is the Bayes optimal decision boundary for classifying D c .Proof: See Appendix D.4. The unlearnable effect takes place when τ Dc (P u ) < τ Dc (P ). To achieve this, we elaborately choose T y , which is formalized asµ ⊤ λ -2 -1 T ⊤ -1 +λ -2 1 T ⊤ 1 λ -2 -1 -λ -2 1 µ ≪ 0. Therefore,Theorem 6 theoretically explains why UMT is effective in generating unlearnable point cloud data.To ensure that authorized users can achieve better generalization after training on unlearnable data, i.e., satisfying Eq. ( 2), we exploit the inverse properties of 3D transformations, presented in Property 2, to calculate the inverse matrix of T k as:EQUATIONIn particular, we note that R -1 = R T , S -1 = 1 λ I. Afterwards, the authorized user receives a lightweight message M containing class-wise parameters from the data protector through a secure channel, thereby assigning M to the inverse transformation matrix in Eq. ( 9) for multiplying the unlearnable samples. Our proposed integral unlearnable process is illustrated in Fig. 3 . 
experimental details: 
 Datasets and Models. Three synthetic 3D point cloud datasets, ModelNet40 [50] , ModelNet10 [50] , ShapeNetPart [4] , and three real-world datasets including autonomous driving dataset KITTI [32] and indoor datasets ScanObjectNN [41] , S3DIS [2] are used. We choose 16 widely used 3D point cloud models PointNet [35] , PointNet++ [36] , DGCNN [45] , PointCNN [25] , PCT [16] , PointConv [48] , CurveNet [51] , SimpleView [15] , 3DGCN [26] , LGR-Net [59] , RIConv [57] , RIConv++ [58] , PointMLP [31] , PointNN [56] , PointTransformerV3 [49] , and SegNN [62] for evaluation of classification and semantic segmentation tasks. Experimental Setup. The training process involves Adam optimizer [22] , CosineAnnealingLR scheduler [30] , initial learning rate of 0.001, weight decay of 0.0001. We empirically set r s , r p , b l , b u , ω l , ω u , h l , and h u to 15 • , 120 • , 0.6, 0.8, 0 • , 20 • , 0, and 0.4 respectively. The main results of different Evaluation Metrics. For classification, we report the test accuracy (in %) derived from the classification accuracy on clean test set, which aligns with the metrics used in the 2D unlearnable schemes [19, 28, 39] . For semantic segmentation, we use eval accuracy and mean Intersection over Union (mIoU) as evaluation metrics (in %), where eval accuracy is the ratio of the number of points classified correctly to the total number of points in the point cloud, mIoU calculates the IoU for each class between the ground truth and the predicted segmentation [61] , and then takes the average of these ratios. The lower these metrics are, the better the effect of the unlearnable scheme.PointNet PointNet++ DGCNN PointCNN Effectiveness. As shown in Tab. 1, UMT results in a significant decrease in test accuracy compared to clean baseline, indicating the unlearnable effectiveness of UMT. Moreover, all values of k achieve good unlearnable results. The average performance of RS is the best, which is employed as the default in the remaining experiments. We note that rigid transformations are easily defeated by invariance networks [10, 26, 58] . Therefore, in practical settings, we will include non-rigid transformations, using combined transformations to enhance the robustness of the UMT scheme.Robustness. (i) Data augmentations. Similar to [18, 24] , we employ data augmentations like random scaling, random jitter, random rotation, Statistic Outlier Removal (SOR) [60] , and Simple Random Sampling (SRS) [53] against UMT. Tab. 2 suggests UMT is robust to random data augmentations. SOR detects and removes outliers or noisy points but is ineffective in countering UMT because UMT does not introduce irregular perturbations or add outliers. SRS randomly selects a small subset from the entire set of points with equal probability. UMT is robust to SRS as it alters the coordinates of all points. Even if an arbitrary subset is chosen, all points within the subset have already undergone the unlearnable transformations. (ii) Adaptive attack. We assume the unauthorized user gains knowledge about the G p 's use of RS. Thus we propose random rotation & scaling as an adaptive scheme. In Tab. 2, the adaptive scheme exhibits a higher accuracy than other schemes, confirming its effectiveness. Nonetheless, it remains 28.67% lower than clean baseline, revealing the robustness of UMT against adaptive attack. More results of adaptive attacks are provided in Appendix C.3.Visual Effect. We visualize UMT samples in Figs. 7 to 10, indicating that the unlearnable point cloud samples still retain their normal feature structure with visual rationality.Evaluation of Semantic Segmentation. We evaluate UMT using common metrics for point cloud semantic segmentation tasks in Tab. 3. As can be seen, the performance of semantic segmentation of data protected by UMT significantly decreases. The underlying reason is that the DNNs learn the features of class-wise transformations and establish a new mapping, which leads to the inability of test samples without transformations to be correctly segmented by the segmentation model.Evaluation of Data Restoration. We multiply UMT samples by the transformation matrix in Eq. ( 9). The data becomes learnable after the restoration process, with test accuracy reaching a level comparable to the clean baseline as shown in Fig. 4 . This strongly validates the effectiveness of the proposed data restoration scheme. The high test accuracy demonstrated by 3DGCN [26] can be attributed to its scaling invariance, which endows it with robustness against scaling transformation.Ablation on Scaling Module. As also shown in Tab. 4, the average accuracy increases by 18.80% and 16.88% when only using the rotation module, respectively. The high test accuracy achieved on RIConv [57] and LGR-Net [59] is due to the fact that both networks are rotation-invariant, thus providing resistance against rotation transformations. These ablation results furthermore emphasize the importance of incorporating more non-rigid transformations.Hyper-Parameter Analysis. We analyze four hyperparameters r s , r p , b l , and b u in Fig. 5 . The influence of r s and r p on the accuracy remains relatively small, exhibiting their best unlearnable effect when set to 15 • and 120 • , respectively. We attribute this to the crucial role played by the class-wise setting, while it is not highly sensitive to the size of specific values. The unlearnable effect is the best when b l and b u are set to 0.6 and 0.8, respectively. Similarly, the variations in b l and b u do not significantly alter the effect due to the class-wise setting. 5 Related Work 
2d unlearnable schemes: 
 The development of 2D unlearnable schemes [19, 29, 37, 39, 42, 55] has been booming. Specifically, model-dependent methods are initially proposed in abundance [9, 19, 29] . Afterwards, numerous model-agnostic methods that significantly improve the generation efficiency have surfaced [39, 42, 47] . However, due to the structural disparities between 3D point cloud data and 2D images, applying unlearnable methods directly from 2D to 3D reveals significant challenges.Some works proposed an encryption scheme based on chaotic mapping [21] or optical chaotic encryption [27] , and a 3D object reconstruction technique was introduced [34] , both achieving privacy protection for individual 3D point cloud data. Nevertheless, no privacy-preserving solution has been proposed specifically for the scenario of unauthorized DNN learning on abundant raw 3D point cloud data. It is worth mentioning that both parallel works [44, 63] study availability poisoning attacks against 3D point cloud networks, which largely reduce model accuracy, and both have the potential to be applied as unlearnable schemes. However, the feature collision error-minimization poisoning scheme proposed by Zhu et al. [63] overlooks the problem of effective training for authorized users, which limits its practical use in real-world applications. The rotation-based poisoning approach proposed by Wang et al. [44] is easily defeated by rotation-invariant networks [57, 58, 59] , as revealed in Tabs. 1 and 4. 
conclusion, limitation, and broader impacts: 
 In this research, we propose the first integral unlearnable framework in 3D point clouds, which utilizes class-wise multi transformations, preventing unauthorized deep learning while allowing authorized training. Extensive experiments on synthetic and real-world datasets and theoretical evidence verify the superiority of our framework. The transformations include rotation, scaling, twisting, and shear, which are all common 3D transformation operations. If unauthorized users design a network that is invariant to all these transformations, they could potentially defeat our proposed UMT. So far, only networks invariant to rigid transformations like rotation and scaling have been proposed, while networks invariant to non-rigid transformations like twisting and shear, have not yet been introduced. Therefore, our research also contributes to the design of more transformation-invariant networks.Our research calls for the design of more robust point cloud networks, which helps improve the robustness and security of 3D point cloud processing systems. On the other hand, if our proposed UMT scheme is maliciously exploited, it may have negative impacts on society, such as causing a sharp decline in the performance of models trained on it, affecting the security and reliability of technologies based on 3D point cloud networks. More transformation-invariant 3D point cloud networks need to be proposed in the future to avoid potential negative impacts. 
 
 Appendix: Existing 3D transformations are summarized in Fig. 1 and formally defined in this section. The 3D transformations are mathematical operations applied to three-dimensional objects to change their position, orientation, and scale in space, which are often represented using transformation matrices. We formally define the transformed point cloud sample with transformation matrix T ∈ R 3×3 as:EQUATIONwhere X ∈ R 3×p is the clean point cloud sample, X t ∈ R 3×p is the transformed point cloud sample.The rotation transformation that alters the orientation and angle of 3D point clouds is controlled by three angles α, β, and γ. The rotation matrices in three directions can be formally defined as:R α = 1 0 0 0 cos α -sin α 0 sin α cos α , R β = cos β 0 sin β 0 1 0 -sin β 0 cos β , R γ = cos γ -sin γ 0 sin γ cos γ 0 0 0 1 (11) Thus we have T = R α R β R γ while employing rotation transformation. Besides, we have R -1 α = R T α , R -1 β = R T β , and R -1 γ = R T γ .The scaling matrix S can be represented as:EQUATIONwhere the scaling factor λ is used to perform a proportional scaling of the coordinates of each point in the point cloud.In the three-dimensional space, shearing3 is represented by different matrices, and the specific form depends on the type of shear being performed. Specifically, the shear transformation matrix H xy (employed in UMT) of shifting x and y by the other coordinate z, and its corresponding inverse matrix can be expressed as:EQUATIONThe shear transformation matrix H xz of shifting x and z by the other coordinate y, and its corresponding inverse matrix can be expressed as:EQUATIONThe shear transformation matrix H yz of shifting y and z by the other coordinate x, and its corresponding inverse matrix can be expressed as:EQUATION)where θ is the parameter of the twisting transformation, and z is the z-coordinate of the object. The inverse matrix of W z (θ, z) is:EQUATIONA.5 Tapering TransformationThe tapering transformation [7] is a linear transformation used to alter the shape of an object, causing it to gradually become pointed or shortened. In three-dimensional space, tapering transformation can adjust the dimensions of an object along one or more axes, creating a tapering effect. The specific matrix representation of tapering transformation depends on the chosen axis and the design of the transformation. Generally, tapering transformation can be represented by a matrix that is multiplied by the coordinates of the object to achieve the shape adjustment, which is defined as:EQUATION)where z is the z-coordinate of the object. Considering that ηz could indeed equal to -1, in such a case, the 3D point cloud samples would be projected onto the z-plane, losing their practical significance and the tapering matrix is also irreversible.Reflection transformation is a linear transformation that inverts an object along a certain plane. This plane is commonly referred to as a reflection plane or mirror. For reflection transformations in threedimensional space, we can represent them through a matrix. Regarding the reflection transformation matrices of the xy plane, yz plane, and xz plane, we have:EQUATIONThe 3D translation transformation 4 refers to the process of moving an object in three-dimensional space. This transformation involves moving the object along the x, y, and z axes, respectively, smoothly transitioning it from one position to another, which is defined as:EQUATION)where t x , t y , and t z represents the translation along the x, y, and z axes, respectively. This 4×4 matrix is a homogeneous coordinate matrix that describes the translation in three-dimensional space.Additionally, the translation matrix also can be represented as a additive matrix to original point x i ∈ R 3×3 , which can be defined as: L = t x tWe initially investigate which approach yields the best unlearnable effect among sample-wise, datasetwise, and class-wise settings in Fig. 2 (a). Specific experimental settings are as follows:In the dataset-wise setting, the same parameter values are applied to the entire dataset. Specifically, we have α = β = γ = 10 • in the rotation transformation, the scaling factor λ is set to 0.8, both shearing factors s and t are set to 0.2. The angle θ in twisting is 25 • . The tapering angle η is set to 25 • , and the parameters in translation transformation t x , t y , and t z are set to 0.15.In the sample-wise setting, each sample has its independent set of parameters, meaning the parameter values for each sample are randomly generated within a certain range. In the rotation transformation, α, β, γ are uniformly sampled from the range of 0 • to 20 • . The scaling factor λ is uniformly sampled from 0.6 to 0.8, shearing factors s and t are uniformly sampled from the range of 0 to 0.4. Both the twist angle θ and tapering angle η are sampled from 0 • to 50 • , and the parameters in translation transformation t x , t y , and t z are sampled from 0 to 0.3.In the class-wise setting, the parameters for transformations are associated with the point cloud's class. The selection of parameters for each class is also obtained by random sampling within a fixed range. The chosen ranges are generally consistent with those described for the sample-wise setting above. However, a difference lies in the consideration of slight angle range and primary angle range in the case of rotation transformations, where these ranges are 20 • and 120 • , respectively.The specific results of test accuracy under different transformation modes, including sample-wise (random), class-wise, and dataset-wise (universal), are provided in Tab. 5. It can be seen that under the class-wise setting, the final unlearnable effect is the best. Dataset Licenses. The dataset license information is listed as:• ModelNet10, ModelNet40 [50] : All CAD models are downloaded from the Internet and the original authors hold the copyright of the CAD models. The label of the data is obtained by us via Amazon Mechanical Turk service and it is provided freely. This dataset is provided for the convenience of academic research only. Link is https://modelnet. cs.princeton.edu/.• ShapeNetPart [4] : We use the ShapeNet database (the "Database") at Princeton University and Stanford University. Link is https://www.shapenet.org/.• ScanObjectNN [41] : The license is MIT License: Copyright (c) 2019 Vision & Graphics Group, HKUST. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. Link is https://hkust-vgd.github.io/ scanobjectnn/. • S3DIS [2] : The copyright is from Stanford University, Patent Pending, copyright 2016.Link is http://buildingparser.stanford.edu/dataset.html.The scaling factor in random scaling augmentation is set to a minimum of 0.8 and a maximum of 1.25.In the random rotation operation, the three directional rotation angles are identical and uniformly sampled from [0, 2π). The perturbations in the random jitter are sampled from a normal distribution with a standard deviation of 0.05, and the perturbation magnitude is constrained within 0.1. The parameters k and α in SOR are set to 2 and 1.1, respectively. The number of dropped points in SRS is 500.We present the specific experimental results of Fig. 2 (a) in Tab. 5. It can be observed that under the class-wise setting, the average test accuracy is the lowest, while the unlearnable condition yields the best performance. Furthermore, we conduct investigations into training the PointNet classifier with class-wise transformed training sets (ModelNet10 dataset), analyzing accuracy results across different test sets, as shown in Tab. 6. It is noteworthy that the accuracy on class-wise test sets (using consistent transformation parameters with class-wise transformed training set) is consistently above 90%, indicating that the model has learned the mapping between transformations and labels. This leads to correct classification on test samples with the same transformations. However, when the class-wise test set undergoes permutation, the accuracy drops to a level similar to that of the clean test set. This clearly demonstrates that the model can correctly classify samples only when they have corresponding transformations, and samples without transformations or with mismatched transformations cannot be correctly classified. This further validates that the model learns a one-to-one mapping between class transformations and labels.We investigate combinations of four transformations, i.e., rotation, scaling, shear, and twisting, and create unlearnable datasets using the class-wise setting. The test accuracy results across five point cloud models are presented in Tab. 7. It can be observed that the combination of rotation and scaling, To ensure the reliability of the experimental results, we combine two different transformations and obtain the results from three runs using random seeds with 23, 1023, and 2023, then average them.The results are shown in Tab. 8. From the results on four popular point cloud models, it can be seen that when all transformations are rigid, the test accuracy is still the lowest.To further explore the performance of UMT against adaptive attacks, we supplement the experimental results using four types of random augmentations RSHW in Tab. 10, and find that the conclusion is consistent with Tab. 2, i.e., UMT exhibits a certain degree of robustness against the adaptive attack random transformations.To further evaluate the UMT performance against semantic segmentation, we employ the semantic scene understanding dataset SemanticKITTI [3] in Tab. 11. It can be observed that UMT is still effective against this real-world segmentation dataset.In the main text, we have discussed the robustness of UMT against rotation/scaling invariant models.The results for RIConv++ [58] (rotation invariant) and 3DGCN [26] (scaling invariant) networks in Tab. 1, as well as RIConv [57] , LGR-Net [59] (rotation invariant), and 3DGCN in Tab. 4, demonstrating that these invariant networks can indeed defend against class-wise rotation and scaling. It is worth noting that these networks, which are invariant to a single transformation, cannot defend against UMT formed by a combination of multiple transformations. Thus, it appears that networks like SE(3) equivariant model, which are invariant to multiple transformations in space, can poten- Additionally, we investigate the unlearnable effects across a wider range of hype-parameters r s , r p , b l , b u in UMT (k = 2 with RS), as shown in Tab. 13. It can be observed that our UMT scheme still exhibits a good unlearnable effect, and its key lies in the crucial role played by the class-wise setting.In this section, we conduct ablation experiments on the UMT scheme on the ShapeNetPart dataset (keeping experimental parameters consistent with the main experiments), as shown in Tab. 14. It can be observed that whether using only the rotation module or only the scaling module, the final unlearnable effect is not as good as UMT. This clearly demonstrates that each module contributes to the overall UMT effectiveness.In this section, we investigate the test accuracy results when using a mixture of different ratios of classwise samples and sample-wise (random) samples in the dataset, as shown in Tab. 15 (experiments are conducted on the ModelNet10 dataset with 10 categories, where "2 class-wise 8 sample-wise" denotes that samples from 2 categories undergo class-wise transformations, while the remaining 8 categories undergo random transformations, and so on). We can clearly observe from the experimental results that as the proportion of class-wise transformations gradually increases, the test accuracy gradually decreases, indicating that the unlearnable effect becomes more pronounced. This strongly suggests that the class-wise setting is more effective than the sample-wise setting. E (x,y)∼Dc1 T 1 x = T 1 E (x,y)∼Dc1 x = T 1 µ, E (x,y)∼Dc1 (T 1 x -T 1 µ)(T 1 x -T 1 µ) ⊤ = T 1 E (x,y)∼Dc1 (x -µ)(x -µ) ⊤ T 1 ⊤ = T 1 IT 1 ⊤ = λ 1 2 R 1 R 1 ⊤ I = λ 1 2 ISimilarly, assuming y = -1, we can obtainE (x,y)∼Dc-1 T -1 x = -T -1 µ, E (x,y)∼Dc-1 (T -1 x -T -1 µ)(T -1 x -T -1 µ) ⊤ = λ -1 2 I.Thus we have: D u ∼ N (yT y µ, λ 2 y I). exp[-1 2 (x -T -1 µ -1 ) ⊤ (λ 2 -1 I) -1 (x -T -1 µ -1 )] (2π) d |λ 2 -1 I| = exp[-1 2 (x -T 1 µ 1 ) ⊤ (λ 2 1 I) -1 (x -T 1 µ 1 )] (2π) d |λ 2 1 I| ⇒ (x -T -1 µ -1 ) ⊤ λ -2 -1 (x -T -1 µ -1 ) + ln |λ 2 -1 I| = (x -T 1 µ 1 ) ⊤ λ -2 1 (x -T 1 µ 1 ) + ln |λ 2 1 I| ⇒ λ -2 -1 (x ⊤ x -x ⊤ T -1 µ -1 -µ ⊤ -1 T ⊤ -1 x + µ ⊤ -1 T ⊤ -1 T -1 µ -1 ) -λ -2 1 (x ⊤ x -x ⊤ T 1 µ 1 -µ ⊤ 1 T ⊤ 1 x + µ ⊤ 1 T ⊤ 1 T 1 µ 1 ) + ln |λ 2 -1 I| |λ 2 1 I| = 0 ⇒ (λ -2 -1 -λ -2 1 )x ⊤ x -2(λ -2 -1 µ ⊤ -1 T ⊤ -1 -λ -2 1 µ ⊤ 1 T ⊤ 1 )x + µ ⊤ -1 µ -1 -µ ⊤ 1 µ 1 + ln |λ 2 -1 I| |λ 2 1 I| = 0 ⇒ P u (x) ≡ Ax ⊤ x + 2[(λ -2 -1 T -1 + λ -2 1 T 1 )µ] ⊤ x + ln |λ 2 -1 I| |λ 2 1 I| = 0 ⇒ P u (x) ≡ Ax ⊤ x + B ⊤ x + C = 0 where A = λ -2 -1 -λ -2 1 , B = 2(λ -2 -1 T -1 + λ -21 T 1 )µ, and C = ln After that, we employ Chernoff bound, for some γ, we have: Furthermore, if µ ⊤ µ + b ⊤ µ + c + d < 0 and -µ ⊤ µ + b ⊤ µ -c -d < 0, we have τ Dc (P u ) < 1. Moreover, for any µ ̸ = 0 ∃ transformation matrix T i such that τ Dc (P u ) < τ Dc (P ).P{Z ≥ E[Z] + γ} ≤ E[Proof: We note that if P u (x) is less than 0, the category of the Bayesian optimal classification is -1; otherwise, it is 1. Here, x = yµ + z where z ∼ N (0, I) and y ∈ {±1} since (x, y) ∼ D c . We can see that:-γ 1 := E{z ⊤ z + (b + 2µ) ⊤ z + µ ⊤ µ + b ⊤ µ + c} = T r(I) + µ ⊤ µ + b ⊤ µ + c -γ 2 := E{-z ⊤ z -(b -2µ) ⊤ z -µ ⊤ µ + b ⊤ µ -c} = -T r(I) -µ ⊤ µ + b ⊤ µ -cApplying Lemma 5, with γ = γ 1 , t = t 1 for the computation of p 1 , as well as γ = γ 2 and t = t 2 for the computation of p 2 , where t 1 and t 2 are specific non-negative constants, we obtain: This provides us with the upper bound for τ Dc (P u ). Nonetheless, to ensure that this upper bound is less than 1, additional conditions need to be affirmed. As γ 1 and γ 2 increase, the values of p 1 and p 2 diminish (p 1 > 0, p 2 > 0), and as γ 1 increases, γ 2 decreases (since γ 1 + γ 2 = -2b ⊤ µ). We let 2 ) equal to β 1 , resulting in:p 1 = expdp 1 dt 1 = 1 2 d[exp{ α1t 2 1 1-2t1 + β 1 t 1 }/(1 -2t 1 ) d 2 ] dt 1 = exp{ α1t 2 1 1-2t1 + β 1 t 1 } 2 [(1 -2t 1 ) -d 2 -1 d + (1 -2t 1 ) -d 2 ( 2α 1 t 1 (1 -t 1 ) (1 -2t 1 ) 2 + β 1 )] = exp{ α1t 2 1 1-2t1 + β 1 t 1 } 2(1 -2t 1 ) d 2 [ d 1 -2t 1 + 2α 1 t 1 (1 -t 1 ) (1 -2t 1 ) 2 + β 1 ] = p 1 ( d 1 -2t 1 + 2α 1 t 1 (1 -t 1 ) (1 -2t 1 ) 2 + β 1 ) = 2(2β 1 -α 1 )t 2 1 + 2(α 1 -2β 1 -d)t 1 + β 1 + d (1 -2t 1 ) 2 p 1I| .In the subsequent theoretical descriptions, UMT refers to UMT with k=2 using transformation matrix RS.https://www.mauriciopoppe.com/notes/computer-graphics/ transformation-matrices/shearing/https://www.javatpoint.com/computer-graphics-3d-transformations 


Decision: This paper was reviewed by four experts in the field. The recommendations are (Accept, Borderline Accept x 3). Based on the reviewers' feedback, the decision is to recommend the acceptance of the paper. The reviewers did raise some valuable concerns (especially the **overclaimed and exaggerated** paper title by Reviewers BhZK and Yqki, missed detailed and convincing experimental evaluations by Reviewers Yqki and nf2f) that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel approach called Unlearnable Multi-Transformations (UMT) to make 3D point cloud data unlearnable by unauthorized users. This method employs a category-adaptive allocation strategy to apply class-wise transformations, thereby preventing unauthorized training on the data. Additionally, the authors propose a data restoration scheme that allows authorized users to learn from the unlearnable data effectively. The effectiveness of UMT is validated through theoretical analysis and extensive experiments on multiple datasets and models, demonstrating its potential in protecting sensitive 3D point cloud data from unauthorized exploitation while enabling authorized access.
strength_weakness: strength_weakness: - Are object categories sensitive to the combination of transformations? For instance, are there categories where using two different sets of transformations results in significant differences in outcomes? I'm not sure if I've missed something or not, but conducting multiple experiments to observe performance and its variance might be a more reasonable approach.
- It's still not very clear how the point cloud segmentation experiments were conducted. Each object category would be in different regions of the point cloud, so would transformations be applied to the corresponding regions based on the GT? If so, would these transformations and the subsequent inverse transformations affect the segmentation results for authorized users?
questions: questions: See the weakness
limitations: limitations: Yes, the authors mention some limitations of the proposed method, such as its vulnerability to rotation and scale transformations. However, they also note that other types of transformations are not easily compromised in the current context.
paper_summary: paper_summary: After reading the author’s response, I would like to increase my evaluation to borderline accept.

—

This paper addresses a critical issue by extending unlearnable strategies to 3D point cloud data, introducing the Unlearnable Multi-Transformations (UMT) approach. The use of a category-adaptive allocation strategy and multiple transformations is innovative and well-conceived. Notably, the paper highlights a gap in existing literature by acknowledging the challenges even authorized users face in learning from unlearnable data, proposing a data restoration scheme to address this. The theoretical and empirical validation across six datasets, sixteen models, and two tasks convincingly demonstrates the framework's effectiveness.
strength_weakness: strength_weakness: 1. The presentation and writing need to be polished to reach the acceptance bar. For the current form, there are a series of unclear descriptions and explanations.
2. The theoretical contributions are overall weak, which provides limited insights to the research community. Besides, some theoretical analysis needs to be justified.
questions: questions: 1. The title of this paper is somewhat overclaimed or exaggerated. It is not related to the research topic of this work, which is also less informative.
2. In the introduction, the paper analyzes three issues and challenges of the research problem. However, there is no intuition about why the proposed method can handle the issues and challenges. There are just some technical details. Could the paper supplement more intuitions for a better understanding of reviewers?
3. In the introduction, there is an understanding gap between the method descriptions and theoretical analysis. Some definitions such as the decision boundary of the Bayes classifier in the Gaussian Mixture Model, are very strange, and cannot describe the work principle of the proposed method.
4. For the data protector and authorized user, do we need to add some noise to implement them with the minimax optimization? More details are needed. Besides, are they mutually reversible in practice?
5. For Eq. (3), is there some evidence about the choice of $\mathcal{A}_N$?
6. As for Property 2, why do we need all four transformation matrices we employ and the multiplicative combinations of any of these matrices are all invertible matrices? I can understand the theoretical analysis. However, it seems that this does not work for the main conclusion of this paper.
7. Similarly, the bounds in Lemma 5 and Theorem 6 are very loose. With them, it is hard to believe the proposed method can work well (although the experimental results are good). More discussions are needed.
8. The proof of Lemma 3 is simple since the space of $y$ is limited to 2. When the components of GMMs are more than 2, will the claim still hold? Could the paper add some discussions about this?
9. From Line 782 to Line 783, could the paper supplement some details to describe why the last line of inequalities holds?
10. For Line 811, the paper assumes that $\alpha_2=\frac{1}{3}$ and $d=3$. Will the assumption make the theory less general? More discussions are also needed.
limitations: limitations: N/A
paper_summary: paper_summary: This paper studies the protection scheme against unauthorized learning on 3D point cloud data. A simple class-wise transformation method is designed to mislead the model to learn the transformation patterns of points instead of categorical knowledge. The method is evaluated on popular used 3D datasets and models.
strength_weakness: strength_weakness: - I am concerned about the value of the settings studied in the paper. Most classification datasets considered in the paper are used to develop 3D modeling algorithms and are usually far from real-world 3D/point cloud applications that require high safety standards like 3D face recognition and scene-level point clouds for autonomous driving. These scenarios usually have much more categories (e.g., facial ids) or cannot be modeled as a classification problem (e.g., scene understanding for autonomous driving). Can the proposed method be easy to transfer to these settings? 

- How about the results if we randomly apply R, S, W, H when we train the model? Such strong data augmentation may lead to lower classification accuracy but the model may achieve >30% test accuracy on the UMT datasets.

- Rotation/scale/SE(3) equivariant 3D modeling has been extensively studied in previous work [r1-r4].  Will the proposed method fail if we use these models for 3D modeling?

[r1] SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks, NeurIPS 2020

[r2] Vector neurons: A general framework for so (3)-equivariant networks, ICCV 2021

[r3] E2PN: Efficient SE(3)-Equivariant Point Network, CVPR 2023

[r4] Scale-Equivariant Deep Learning for 3D Data
questions: questions: - The title "Class-wise Transformation Is All You Need" is quite general, which may not be helpful for readers to understand the topic studied in the paper. I was a bit confused when I first read the title along with the abstract. 


Overall, this paper studies a new problem and proposes a simple approach against unauthorized learning on 3D point cloud data.  Extensive experiments and analyses are provided to evaluate the method. However, I still have concerns about the value of the settings considered in the paper and the robustness of the proposed method. I am not an expert on the topic (safety in machine learning), thus I would like to rate this paper as borderline and wait for further discussions.
limitations: limitations: The limitations and broader impacts of the method have been discussed in Section 6.
paper_summary: paper_summary: The paper introduces UMT (Unlearnable Multi-Transformations), the first approach designed to render 3D point cloud data unlearnable for unauthorized deep learning models, by applying class-wise transformations. It also presents a data restoration scheme to enable authorized users to effectively train on the unlearnable data. Theoretical analysis and extensive experiments on various datasets and models demonstrate the effectiveness of UMT in safeguarding sensitive 3D data while allowing authorized access for legitimate training purposes. The main contributions include the proposal of the first 3D unlearnable scheme, a novel data restoration approach, theoretical insights into the unlearnability mechanism, and empirical validation of UMT's superiority.
strength_weakness: strength_weakness: 1.	The experiments do not investigate the impact on UMT dataset performance when mixed with other datasets. Currently, transformations are applied to the entire training set, altering the distribution between the UMT training and test sets. If only part of the training set is UMT data, can the performance on the UMT test set still be maintained?
2.	The experiments rely on randomness. Are the results averaged over multiple runs to ensure reliability? While the appendix provides experiments with different random seeds, it does not show the variance of results in other experiments.
3.	The proposed transformation patterns, such as rotation and scaling, are easily detectable by comparing the UMT training set and test set. This predictability might allow unauthorized models to adapt to these transformations, potentially compromising the method's effectiveness.
questions: questions: The questions have been listed in the Weaknesses. Please check the weaknesses.
limitations: limitations: The authors have discussed the limitations and Broader Impacts in the last section.

Comments: title: Response to Reviewer nf2f [2/2]
Response to Reviewer nf2f [1/2]
Help Check the Rebuttal, Make Discussions, and Update the Final Recommendations
Response to Reviewer Yqki
Response to Reviewer Yqki [3/3]
Response to Reviewer Yqki [2/3]
Response to Reviewer BhZK [3/3]
Response to Reviewer BhZK [2/3]
comment: > Q3: The combination of several transformations is useless for RIConv++ on KITTI dataset.

**Re:** Thank you for your thorough review. This is a really great and insightful question! After our in-depth consideration, we believe that this counter-intuitive phenomenon is caused by RIConv++'s local feature extraction process overlooking the shortcut points of UMT KITTI samples that are distant from the original sample.

The following is our reasoning process: Because this phenomenon only occurs when training RIConv++ on KITTI (our KITTI dataset was created following the settings in [1,2]). Therefore, our intuition is that the unique input point cloud samples in RIConv++ requiring additional normal vectors caused this phenomenon, because RIConv++[3] claims the LRA-based normal vector is the key factor in its rotation invariance. To further investigate this, we replaced RIConv++ with PointNet (with normals) and conducted experiments on  KITTI, as shown in the table below (counterintuitive results are highlighted in bold, with "√" indicating the use of normals calculated by the LRA method in RIConv++, and "×" indicating the absence of normals). 

|  Datasets  |  ModelNet10  |  ModelNet40  | ShapeNetPart | ScanObjectNN  |    KITTI     |     KITTI      |     KITTI      |
| :--------: | :----------: | :----------: | :----------: | :-----------: | :----------: | :------------: | :------------: |
|   Models   | RIConv++ (√) | RIConv++ (√) | RIConv++ (√) | RIConv++ (√)  | RIConv++ (√) | PointNet （×） | PointNet （√） |
|   Clean    |    86.01     |    85.82     |    97.39     | 66.34 ± 1.21  | 99.64 ± 0.09 |  98.04 ± 2.23  |  99.39 ± 0.40  |
|  UMT (R)   |    85.16     |    81.90     |    96.96     | 62.33 ± 10.68 | 98.53 ± 1.19 | 36.23 ± 30.18  |  27.79 ± 5.81  |
|  UMT (RS)  |    18.64     |    11.32     |     2.62     | 10.76 ± 1.76  | **99.80 ± 0.09** |  31.24 ± 7.11  |  23.83 ± 0.72  |
| UMT (RSW)  |    13.77     |     8.63     |     3.51     | 11.69 ± 1.41  | **98.48 ± 1.61** |  19.13 ± 6.93  |  26.17 ± 6.32  |
| UMT (RSWH) |    29.24     |    13.68     |    32.68     | 31.83 ± 2.20  | **99.34 ± 0.23** | 26.84 ± 16.26  |  29.52 ± 4.09  |

We can see that: (1) Using normal vectors did not influence the final outcome of PointNet; (2) The unexpected results occurred only on the KITTI dataset. Thus, we exclude the possibility of influence from the normal vectors and shift our focus to considering the potential impacts between the KITTI samples themselves and other processes involved in the RIConv++ implementation.

After carefully reading the original RIConv++ paper [3] and thoroughly inspecting the KITTI samples, we infer that this is because following the experimental procedures outlined in [1-2], the only two categories in the KITTI samples, "pedestrian" and "vehicle"(extracted from autonomous driving dataset), tend to be relatively flat in 3D space and consist of only 256 points. For example, pedestrian samples typically stand upright, with most data points concentrated around this plane-like surface, while vehicle samples are concentrated near the horizontal plane, with limited height range. Other datasets like ModelNet10 avoid this issue due to their diverse categories and 1024 points, with objects like chair and bathtub showing distinct 3D characteristics. After applying transformations to KITTI with UMT, the limited 3D sample features resulted in restricted transformation influence: the plane-like surface's features constrained the range of UMT transformations, and the number of points limited the points UMT could change, leading to local anomalies (e.g., when shear operations are applied to pedestrian samples, some points may stretch far from the plane-like surface, creating anomalies). While these anomalies would still be considered class-wise shortcuts in other models, resulting in defense effects, they are overlooked by RIConv++ since RIConv++ is designed with convolutional operators that extract local rotation-invariant features, ignoring points distant from local features.

We are happy to add a discussion of this counterintuitive phenomenon in the revised manuscript and plan to delve deeper into the differing effects of UMT on the KITTI dataset and various models in future work, as this presents an intriguing area for more detailed and complete investigation. Thank you once again for your valuable feedback, we hope our response resolves your concerns. If you have any further questions, feel free to let us know. Wishing you a wonderful day :)

[1] A backdoor attack against 3D point cloud classifiers. ICCV'21

[2] PointCRT: Detecting backdoor in 3D point cloud via corruption robustness. MM'23

[3] RIConv++: Effective rotation invariant convolutions for 3D point clouds deep learning. IJCV'22
> Q1: Specify the dataset and provide a comparison using 0% UMT data.

**Re:** Thank you very much for your reply. The dataset we used here is the ModelNet10 dataset. We have provided the results for 0% UMT data in the table below, and it can be seen that the trend remains consistent with our previous conclusion: the higher the proportion of UMT data, the better the protection effect.

**Table X: Test accuracy (%) results when using different ratios of UMT(k=1, $\mathcal{R}$) ModelNet10 training data.**

| UMT ratio | PointNet++ | PointNet | PointCNN | DGCNN |  AVG  |
| :-------: | :--------: | :------: | :------: | :---: | :---: |
| **100%**  |   30.51    |  29.85   |  29.46   | 35.13 | 31.24 |
|  **80%**  |   47.03    |  47.14   |  46.92   | 48.57 | 47.42 |
|  **60%**  |   60.35    |  62.44   |  63.55   | 62.89 | 62.31 |
|  **40%**  |   73.79    |  66.63   |  67.40   | 69.49 | 69.33 |
|  **20%**  |   84.03    |  84.14   |  83.26   | 85.68 | 84.28 |
|  **0%**   |   92.95    |  89.32   |  89.54   | 92.73 | 91.14 |

**Table Y: Test accuracy (%) results when using different ratios of UMT(k=2, $\mathcal{R}\mathcal{S}$) ModelNet10  training data.**

| UMT ratio | PointNet++ | PointNet | PointCNN | DGCNN |  AVG  |
| :-------: | :--------: | :------: | :------: | :---: | :---: |
| **100%**  |   28.08    |  22.25   |  21.48   | 14.10 | 21.48 |
|  **80%**  |   24.44    |  20.98   |  21.65   | 25.11 | 23.05 |
|  **60%**  |   41.74    |  40.51   |  40.74   | 40.62 | 40.90 |
|  **40%**  |   58.48    |  59.04   |  58.04   | 59.04 | 58.65 |
|  **20%**  |   76.00    |  75.00   |  75.78   | 76.00 | 75.70 |
|  **0%**   |   92.95    |  89.32   |  89.54   | 92.73 | 91.14 |
For Q1, the experiment has a few unclear aspects. First, the dataset used in this experiment is not specified. Second, the experiment does not provide a comparison using 0% UMT data.

For Q2, the experiments have addressed my concern.

For Q3, I still have a concern. According to Table 3, although RIConv++ is a rotation-invariant network, the combination of several transformations is useless for RIConv++.
Dear Reviewers,

Thanks for serving as a reviewer for the NeurIPS. The rebuttal deadline has just passed.

The author has provided the rebuttal, could you help check the rebuttal and other fellow reviewers' comments, make necessary discussions, and update your final recommendations as soon as possible?

Thank you very much.

Best,

Area Chair
Dear Reviewer Yqki,

We want to extend our sincere thanks for your detailed review and increasing the score during this phase. Your feedback was instrumental in helping us improve the quality of the paper, contributed significantly to the development of the community, and we are truly grateful for your support.

Thank you for your time and understanding.

Sincerely, 

Submission19662 Authors
Thanks for your detailed response. My concerns about data augmentation and existing SE(3)-equivariant methods have been addressed. After reading other reviews, I would like to upgrade my score to Borderline Accept.
> **Q5:** The title is quite general, which may not be helpful for readers to understand the topic studied in the paper.

**Re:** Thank you for bringing this matter to our attention. We apologize for any confusion caused by the title. To better convey the focus of the paper, we will revise the title to: "Unlearnable 3D Point Clouds: Class-wise Transformation Is All You Need".
> **Q3:** 3D face recognition also need to be considered.

**Re:** We value your insightful and constructive feedback and fully concur with your suggestion to conduct experiments in more practical scenarios. We present the UMT performance on face recognition in the Table R5 below using Basel Face Model 2017 [1]  to generate 3D point cloud face scans. Our experimental setup for 3D face BFM 2017  is consistent with [2], which involves generating 100 classes of face scans (each of them contains 50 point clouds), dividing the whole 5000 face data into training and test part, containing 4000 and 1000 data respectively, and then randomly choosing 1024 points for each point clouds. It can be seen that UMT is still effective in this scenario, which is also because the class-wise transformation causes the model to overfit the transformation-based shortcut of the training data, making it difficult to generalize to clean test data.

**Table R5:** Evaluation of UMT on face recognition using BFM2017-generated 3D point cloud face dataset.

|     PointNet      | Clean baseline | UMT(k=1, $\mathcal{R}$) | UMT(k=2, $\mathcal{R}\mathcal{S}$) | UMT(k=3, $\mathcal{R}\mathcal{S}\mathcal{W}$) | UMT(k=4, $\mathcal{R}\mathcal{S}\mathcal{W}\mathcal{H}$) |
| :---------------: | -------------- | ----------------------- | ---------------------------------- | --------------------------------------------- | -------------------------------------------------------- |
| Test accuracy (%) | 98.10          | 0.81                    | 1.11                               | 0.91                                          | 1.01                                                     |

[1] *Morphable face models-an open framework.* International Conference on Automatic Face & Gesture Recognition'18

[2] *Toward availability attacks in 3D point clouds.* ICML'24







> **Q4:** The performance of proposed method against rotation/scale/SE(3) equivariant models.

**Re:** We appreciate your insight on this matter. In our original paper, we have indeed discussed the robustness of UMT against rotation/scaling invariant models. The results for RIConv++ (rotation invariant) and 3DGCN (scaling invariant) networks in Table 1, as well as RIConv, LGR-Net (rotation invariant), and 3DGCN in Table 4, demonstrating that these invariant networks can indeed defend against class-wise rotation and scaling. It is worth noting that these networks, which are invariant to a single transformation, cannot defend against UMT formed by a combination of multiple transformations. Thus, it appears that networks like SE(3), which are invariant to multiple transformations in space, can potentially overcome UMT. Therefore, following your insightful advice, we included experimental results for the SE(3) network SE (3)-Transformer [1] in the following Table R6.

**Table R6:** Test accuracy results (%) of SE(3) equivariant model SE (3)-Transformer using UMT and clean training set.

|                        Clean baseline                        |   49.07   |
| :----------------------------------------------------------: | :-------: |
|      **UMT(k=3, $\mathcal{R}\mathcal{S}\mathcal{W}$)**       | **17.51** |
| **UMT(k=4, $\mathcal{R}\mathcal{S}\mathcal{W}\mathcal{H}$)** | **13.55** |

However, it can be seen that SE (3)-Transformer cannot defend against UMT (k=3, $\mathcal{R}\mathcal{S}\mathcal{W}$) and UMT(k=4, $\mathcal{R}\mathcal{S}\mathcal{W}\mathcal{H}$). This is because existing transformation-invariant networks, even including SE(3) invariant networks [1], are designed only for rigid transformations (rotation, scaling, reflection, and translation, as shown in Fig. 7 in the original paper). There are currently no invariant networks proposed for non-rigid transformations like shear and twisting. Therefore, if a data protector wants UMT to be more robust, they can including non-rigid class-wise transformations to defeat existing rigid transformation-invariant networks. Of course, we also acknowledge that more robust invariant networks may be developed in the future, but it is exactly for this reason that the introduction of UMT will advocate for the design of more robust 3D point cloud networks, which is also where the value of our work lies.

 [1] *SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks*. NeurIPS'20
> **Q10:** From Line 782-783, could you describe why the last line of inequalities holds?

**Re:** Thank you for noting this point. Firstly, we define $\alpha_1$ as a variable $\geq 0$ on line 779. Therefore, assuming we define $s(\alpha_1) = \frac{d^2}{2\alpha_1} + \frac{\alpha_1}{2}$. The minimum value of this function can be obtained using the Arithmetic Mean-Geometric Mean Inequality (AM-GM Inequality), which states that $a + b \ge 2\sqrt{ab}$ for $a, b \ge 0$. Thus, $s(\alpha_1) \ge 2\sqrt{\frac{d^2}{2\alpha_1} \cdot \frac{\alpha_1}{2}} = d$. Since $\beta_1 \le s(\alpha_1)$, we can infer that $\beta_1 \le s(\alpha_1)_{\text{min}}$, which means $\beta_1 \le d$.  In the revised version, we will include these explanations between Line 782 and Line 783 to make the reason of this inequality more clearly.




> **Q11:** Line 811 assumes $\alpha_2 = \frac{1}{3}, d=3$. Will this make the theory less general?

**Re:** Thank you for your thorough review and for highlighting this issue. We recognize that setting specific values for these parameters may make the proof of the theorem less general, but in the special case of our existence theorem proof, this will not make Theorem 6 less general. First, since the dimensionality of the 3D point cloud data we study is 3, it is appropriate to use $d=3$ to represent the data dimension in this theoretical analysis. Secondly, assigning an appropriate value to $\alpha_2$ aims to prove the existence of a solution (as mentioned in Line 768) rather than finding the solution of the inequality $\tau\_{\mathcal{D}\_c} (P_u) < \tau_{\mathcal{D}\_c} (P)$. This similar existence theorem proof to demonstrate the effectiveness of an unlearnable scheme is also adopted by a previous unlearnable work [1]. We are very grateful for the valuable advice you have given, in the revised version, we will add the above explanation in Line 811 to make our theory more clearly.

[1] *CUDA: Convolution-based unlearnable datasets.* CVPR'23
> **Q6:** Why is the choice of $\mathcal{A}_N$ for Eq. (3)?

**Re:** Thank you for highlighting this issue. We set $\mathcal{A}\_N$ this way to ensure that the number of classes in the final rotation matrix is greater than or equal to N (the number of classes in the training set), thereby ensuring that the UMT scheme satisfies the class-wise setup. Concretely, in the rotation operation, each of the three directions has $\mathcal{A}\_N$ distinct angles, which means that the final rotation matrix has $\mathcal{A}\_N^3$ possible combinations. To satisfy the class-wise setup, $\mathcal{A}\_N^3$ must be at least $N$, requiring $\mathcal{A}\_N$ to be no less than $\lceil \sqrt[3]{N} \rceil$. Therefore, we configure $\mathcal{A}\_N$ in this way in Eq. (3). This point will be explained in more detail in Lines 144,145  in the revised version.





> **Q7:** Why transformation matrices are all invertible matrices in Property 2?

**Re:** Thank you for your careful review and for identifying this problem. The transformation matrices need to be invertible for the purpose of designing our proposed data restoration scheme (Sec. 3.4). Specifically, authorized users can build inverse matrices for the transformations after receiving the class-wise parameters from the data protector. This allows the authorized users to normally train on the protected data by leveraging the property that multiplying a matrix by its inverse results in the identity matrix (Lines 196-197). In the revised version, we will provide a detailed explanation of the necessity of Property 2 when it is mentioned for the first time in Line 159.





> **Q8:** The bounds in Lemma 5 and Theorem 6 are loose.

**Re:** Your feedback on this issue is greatly appreciated. We acknowledge that these bounds are quite loose. Our theorem aims to demonstrate the existence of an unlearnable situation that UMT satisfies (as stated in Line 67 and Line 189), rather than emphasizing the performance of UMT. Our theory is intended to prove that the equation $\tau\_{\mathcal{D}\_c} (P_u) < \tau\_{\mathcal{D}\_c} (P)$ has a solution, not to solve this equation. This theoretical analysis follows a previously well accepted theoretical analysis proposed by 2D unlearnable literature [1, 2]. We apologize that our theorem has given you the impression of a weak contribution, but within the current literature on unlearnable examples, this series of proofs is accepted in the theoretical analysis of the unlearnable effectiveness [1]. In the revised version, we will add a statement about the limitations of our theoretical analysis in Sec. 6.

[1] *CUDA: Convolution-based unlearnable datasets*. CVPR'23

[2] *Corrupting Convolution-based Unlearnable Datasets with Pixel-based Image Transformations.* arXiv:2311.18403

 



> **Q9:** Will the claim still hold when the components of GMMs are more than 2?

**Re:** We are thankful for your feedback on this matter. The claim of Lemma 3 will still hold. Here are our proofs: 

> **Proof (GMM with *n* components):** Assuming the dataset $\mathcal{D}\_c$ is represented by a GMM $\mathcal{N}(y\mu, \boldsymbol{I})$, where $y \in \\{y_i\\}\_{i=1}^n$. Let us take $y = y_i$, then $\mathcal{D}\_{cy_i} \sim 
> \mathcal{N}(y_i\mu, \boldsymbol{I})$, UMT data point is $(\mathbf{T}\_\{y\} x, y)$, then we have:
>
> **Mean:** $\mathbb{E}_ \{ (x,y) \sim  \mathcal{D}\_{cy_i} }  [\mathbf{T}\_\{y_i\} x] =  \mathbf{T}\_\{y_i\} \mathbb{E}\_\{(x,y) \sim  \mathcal{D}_{cy_i}\} x = \mathbf{T}\_\{y_i\} y_i\mu$
>
> **Variance:** $\mathbb{E}_ \{ (x,y) \sim  \mathcal{D}\_{cy_i} } [(\mathbf{T}\_\{y_i\} x - \mathbf{T}\_\{y_i\} y_i\mu) (\mathbf{T}\_\{y_i\} x - \mathbf{T}\_\{y_i\} y_i\mu)^{\top}] =  \mathbf{T}\_\{y_i\}      \mathbb{E}_ \{ (x,y) \sim  \mathcal{D}\_{cy_i} }  [(x - y_i\mu) (x - y_i\mu)^{\top}] \mathbf{T}\_\{y_i\}^{\top} = \mathbf{T}\_\{y_i\} \boldsymbol{I} \mathbf{T}\_\{y_i\}^{\top} = {\lambda^2\_\{y_i\} } \boldsymbol{I} $
>
> Thus we have: $\mathcal{D}\_u \sim \mathcal{N}(y \mathbf{T}_y \mu, \lambda\_{y}^{2} \boldsymbol{I})$.

We model a binary classification problem for theoretical analysis as it allows for a simpler expression of the Bayesian decision boundary, and this setting is widely used and accepted by the community [1,2,3].

[1] *CUDA: Convolution-based unlearnable datasets*. CVPR'23

[2] *Precise statistical analysis of classification accuracies for adversarial training.* The Annals of Statistics'22

[3] *The curious case of adversarially robust models: More data can help, double descend, or hurt generalization.* Uncertainty in Artificial Intelligence'21