Informative Point cloud Dataset Extraction for Classification via Gradient-based Points Moving

Authors: wenxiao  zhang
ziqi  wang
li  xu
xun  yang
jun  liu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 04:29:38

Abstract: Point cloud plays a significant role in recent learning-based vision tasks, which contain additional information about the physical space compared to 2D images. However, such a 3D data format also results in more expensive computation costs to train a sophisticated network with large 3D datasets. Previous methods for point cloud compression focus on compacting the representation of each point cloud for better storage and transmission. In this paper, we introduce a new open problem in point cloud field: Can we compress a large point cloud dataset into a much smaller synthetic dataset while preserving the important information of the original large dataset?} In other words, we explore the possibility of training a network on a smaller dataset of informative point clouds extracted from the original large dataset but maintaining similar network performance. Training on this small synthetic dataset could largely improve the training efficiency. To explore this new open problem, we formulate it as a parameter-matching issue where a network could get similar network parameters after training on the original set and the generated synthetic set, respectively. We find that we could achieve this goal by moving the critical points within each initial point cloud through an iterative gradient matching strategy. We conduct extensive experiments on various synthetic and real-scanned 3D object classification benchmarks, showing that our synthetic dataset could achieve almost the same performance with only 5\% point clouds of ScanObjectNN dataset compared to training with the full dataset.

Main paper: 
 introduction: 
 Point clouds have recently gained great popularity for representing objects in 3D vision [13, 16, 32, 36, 73, 74, 76] . Several 3D object datasets have been created, such as ModelNet [61] , ShapeNet [9] , and ScanObjectNN [53] . Also, networks specifically designed for processing point cloud have been proposed [20, 28, 30, 38, 41-43, 52, 56, 73] . Training networks that work on point clouds is often more computationally expensive than the 2D image counterpart, as point clouds have irregular structures where the traditional convolution operation cannot be directly applied [5, 6, 34, 35, 57, 62, 71, 72, 75] . Some attempts have been made to design a convolution operation that operates directly on points [28, 42, 52, 56] . However, they commonly need to search the nearest neighbor points to combine the local property, involving a greatly high computational cost. Though downsampling the point cloud to a smaller number of points is a solution, it will significantly reduce the testing accuracy.When a new network architecture is developed, as the new network involves new combinations of different layers, it must be trained from scratch again on the training sets. This process can take a long time, especially if the training set is large or if the network architecture is complex. Additionally, even though large volumes of point cloud data are requested by researchers to train neural networks, some data collectors may not want to make the data available to the general public for commercial reasons. For example, Shibuya et al. [51] have tried to transform point clouds into 3D line clouds to preserve point cloud data privacy.Considering both the computation cost and data privacy, we investigate the feasibility of condensing a point cloud dataset into a compact set of representative synthetic point clouds, while preserving the majority of its information and minimizing performance degradation for classification task. Our objective is to create a synthetic dataset from a large dataset which should be significantly smaller in size compared to the original dataset, yet preserve the inherent characteristics of the dataset. Meanwhile, the generated synthetic dataset should be generalizable enough to train point cloud networks of different architectures. In this paper, we have explored that utilizing only a few synthetic point clouds generated with our method could result in plausible results when training the same model (Figure 1 ).Similar tasks have been recently explored in 2D image areas, which are defined as image dataset condensation or distillation [55, 79, 81] . Most of these methods try to generate the synthetic image dataset to approximate the original training set via an iterative synthetic image optimization process. However, we find directly applying these methods to point cloud data is non-trivial where the generated synthetic point clouds are hard to converge to the optimal shapes by simply employing these methods. This challenge arises because 2D images are regular grids, allowing the optimization process to proceed effectively by simply altering pixel colors. In contrast, point clouds lack such regularity which allows points to move more freely. This complicates the optimization process because it becomes more difficult to determine the most effective optimization direction in 3D space.We observe that the initialization of the synthetic point cloud dataset is a delicate factor which heavily impacts the synthetic set quality. In the 2D image field, most methods start with a synthetic dataset initialized with Gaussian noise or random samples, and often achieve satisfactory results after optimization. However, we find that due to the irregularity characteristic of point clouds, the initialization of the synthetic point cloud dataset is a delicate factor that significantly influences the synthetic set convergence. In other words, a strategic and well-considered initialization is essential for obtaining the more informative synthetic point cloud dataset.To address this challenge, we structure our objective by introducing a two-stage approach. The first stage is centered on devising an effective method for initializing the synthetic point cloud dataset. Specifically, we design a strategy based on the mean of the nearest point cloud feature, which entails initializing our synthetic dataset with a carefully selected subset of point clouds. In the second stage, we propose a parameter-matching strategy to optimize the synthetic point cloud through moving the position of points. Specifically, given the same network initialization, the parameters of the deep network trained on the original and synthetic datasets should ideally be very similar. This parameter matching issue could be solved through an iterative gradient matching strategy where the gradients for the backpropagation of two networks should be similar in each training iteration. We illustrate our method pipeline in Figure 2 . Instead of leveraging point cloud generative models [1, 27, 33, 65] to generate synthetic point clouds, we directly regard the input synthetic point clouds as learnable parameters, and treat the network weights as a differentiable function to optimize the synthetic input. In each training iteration, we compute the difference between the gradients from the original and synthetic point clouds and finally the synthetic point clouds could be optimized via moving the points.Comprehensive experimental evaluation of our approach on the various synthetic and real-scanned point cloud classification datasets shows the effectiveness of our method, demonstrating that we could train point cloud networks with small synthetic dataset to significantly fasten the training process with acceptable performance drop. To further demonstrate the effectiveness of our method, we further test our approach on practical applications such as continual learning in our experiments.In summary, our contributions are as follows: 1) We define a new open problem in the point cloud field, exploring the possibility of synthesizing a smaller training dataset from a large point cloud dataset for efficient training. 2) We introduce a two-stage solution, in which we propose a nearest-feature-mean based initialization strategy, followed by a parameter-matching strategy through gradient-based points moving. 3) Extensive experiments on various classification benchmarks show the effectiveness of our synthetic dataset, demonstrating we could train point cloud networks with a small synthetic dataset to significantly fasten the training process with acceptable performance drop. 4) We further evaluate our approach through applications in point cloud continual learning, demonstrating the significance of the new task and the efficacy of our proposed method. 
related work: 
 Deep Learning on Point Cloud Analysis. The field of point cloud processing has undergone a significant transformation with the advent of deep learning technology [29, [66] [67] [68] [69] . Traditional methods that relied on hand-crafted local descriptors [10, 17, 31] have been gradually replaced by learning-based methods. Qi et al.proposed PointNet [41] as a solution to the challenge posed by the disordered nature of point clouds, incorporating shared MLP and max pooling layers. PointNet++ [42] was subsequently introduced to enhance feature discrimination through hierarchical local feature extraction. Wang et al. introduced DGCNN [56] , which utilized the distance between each point in the feature space instead of the Euclidean distance to determine the area surrounding each point for graph feature extraction. Other works have employed convolution neural networks to learn point cloud local characteristics [3, 14, 18-20, 26, 28, 30, 38, 43, 48, 49, 52, 63] . Despite the success achieved by these methods in analyzing point clouds, many of them require the calculation of point-wise distance information to aggregate local characteristics, resulting in a higher computational cost than the convolution of conventional 2D images. Generative models on point clouds. Our work is also closely related to generative models for point clouds, including point cloud generative adversarial networks [1, 27, 33] , optimal transport [70] , and probabilistic-based models [24, 37, 65] . PointFlow [65] , DPF-Net [22] , and SoftFlow [25] utilize Normalizing Flows, while SetVAE [23] employs VAEs to generate sets of point clouds. ShapeGF [4] learns the distributions of gradient fields that constitute shape surfaces, and SP-GAN [27] utilizes a prior based on a spherical point cloud. However, the aim of these point cloud generation methods is to generate realistic-looking point clouds capable of fooling humans, while the objective of our work is to create informative training examples that can be effectively utilized for the training of deep neural networks. Dataset Compression & Condensation. Recent studies [21, 58, 60, 64, 77, 80, 82, 83] have employed learning-based methods to compress point cloud data. Given the disorderly nature of point clouds, current methods [39, 46] usually use a sophisticated data structure, like an octree, to arrange the unprocessed point cloud data. However, these methods focus on improving storage and transmission efficiency rather than training efficiency. In the 2D image community, some researchers aim to choose a subset of the entire training dataset, referred to as a "coreset", which can be used to achieve good performance during training. Most of these methods [11, 47] progressively select important data points based on heuristic criteria. Recent works [8, 54, 55, 78, 79, 81] have been proposed to generate datasets that are not limited to the original dataset, but applying these works to point clouds is non-trivial due to the irregularity of point clouds as discussed in the introduction. To this end, we explore the possibility of synthesizing a smaller, informative training dataset from a larger point cloud set. 
problem definition: 
 Given a training set T consisting of 𝑁 point cloud and label pairs, (𝑥 𝑖 , 𝑦 𝑖 ), 𝑖 = 1, 2, . . . 𝑁 , our objective is to generate a smaller set of synthetic point cloud and label pairs, S = (𝑠 𝑖 , 𝑦 𝑖 )|𝑖 = 1, 2, . . . 𝑀, where 𝑀 (𝑀 ≪ 𝑁 ) is much smaller than 𝑁 . The ideal scenario would be a network trained on S converging faster and performing similarly to a network trained on T .The networks trained on T and S are denoted as 𝜙 𝜽 T and 𝜙 𝜽 S , respectively, where 𝜽 T and 𝜽 S represent their respective parameters. The performance of each network can be defined asE 𝒙∼𝑃 D ℓ 𝜙 𝜽 T (𝒙), 𝑦 and E 𝒙∼𝑃 D ℓ 𝜙 𝜽 S (𝒙), 𝑦, where 𝑃 D represents the expected real data distribution and ℓ (., .) is the classification loss function. Our objective can be formulated as: We learn an informative synthetic point cloud set which can get similar network parameters when a network is trained on it and the original dataset. We introduce a nearest feature mean based initialization strategy, and formulate our goal as an efficient gradient matching problem via points moving. The synthetic sets are regarded as learnable parameters and optimized iteratively.EQUATIONIn the experiments, the network performance is evaluated on the splited unseen testing set. 
method: 
 To get the ideal S for a large point cloud dataset, we propose a two-stage solution. In the first stage, we introduce a nearestfeature-mean based initialization strategy, in which we initialize the synthetic point clouds with representive samples facilitating the following optimization process. In the second stage, we formulate our goal as a parameter-matching problem and further solve it by a gradient-matching strategy to iteratively optimize S. We introduce our method details step-by-step in the following sections.As previously noted, the irregular nature of point clouds makes the initialization of S a sensitive factor in the subsequent optimization process. A good initialization can significantly enhance the convergence of S to get optimal performance. To define a good initial S, inspired the heuristic sample selection method Herding [59] , we consider S is well initialized if the initial S could approximate the overall distribution of the training set T . To this end, we propose to initialize S by selecting the most representative samples in T based on the nearest feature mean.Algorithm 1 describes our initialization strategy for S. Specifically, S are initialized class by class. For the training set point clouds T 𝑐 belonging to class 𝑐, we extract the latent representation of each point cloud 𝑥 in T 𝑐 through a feature function 𝜙 𝑓 : 𝑥 → R 𝑑 . For the feature function 𝜙 𝑓 , we train a point cloud classification network on T , and use the last layer feature 𝑓 ∈ 1 × 𝐷 as the latent representation of each point cloud. Taking PointNet as an example, we use the feature after the max-pooling layer as the feature representation.Point clouds 𝑠 1 , . . . , 𝑠 𝑚 are iteratively selected and accumulated until the desired total of 𝑚 is reached. At each iteration step, a point cloud from the current dataset T 𝑐 is incorporated into S 𝑐 . Specifically, the chosen sample is the one that most closely aligns the average feature vector of all samples across the entire training set. To this end, S 𝑐 could finally approximate the distribution of the original set T 𝑐 .Input: point cloud dataset T 𝑐 = {𝑥 1 , . . . , 𝑥 𝑛 } of class 𝑐 Input: 𝑚 target number of the synthetic dataset S 𝑐 of class 𝑐 Require: feature function𝜙 𝑓 : 𝑥 → R 𝑑 𝜇 ← 1 𝑛 𝑥 ∈ T 𝑐 𝜙 𝑓 (𝑥) // current class feature mean for 𝑘 = 1, . . . , 𝑚 do 𝑠 𝑘 ← argmin 𝑥 ∈ T 𝑐 𝜇 -1 𝑘 [𝜙 𝑓 (𝑥) + 𝑘 -1 𝑗=1 𝜙 𝑓 (𝑠 𝑗 )] end for S 𝑐 ← (𝑠 1 , . . . , 𝑠 𝑚 ) Output: Initialized synthetic dataset S 𝑐 of class 𝑐To fulfill the objective outlined in Equation ( 1), it is crucial to not only achieve similar network output with minimal loss after training on S and T , but also to obtain similar network parameters 𝜽 S ≈ 𝜽 T , thus ensuring similar parameter optimization progress for improved generalization ability. Our goal can be expressed as:EQUATION𝜽 S (S) = arg minEQUATIONEQUATIONwhere 𝐷 denotes the distance function between the parameters of the two networks, and L denotes the classification function.In the process of parameter learning, the final trained 𝜽 T and the S is dependent on the initialization parameter 𝜽 0 which is sampled from an initialization parameter distribution 𝑃 𝜽 0 . Thus the above Equation 2, 3, 4 can be modified as:EQUATION𝜽 S (S) = arg minEQUATION𝜽 T (T ) = arg minEQUATIONOriginal point cloud set T is time-invariant during training, therefore we can first get trained 𝜽 T in an offline manner, then 𝜽 S can be learned with 𝜽 T as the target.The traditional approach to solve Equation 6, 7 involves implicit differentiation [44] . However, this can be complicated or expensive due to the formation of the derivative matrix. It becomes infeasible when the models are large. An alternative approach, backoptimization [15] , is proposed where the network parameters (𝜽 ) are partially optimized according to the loss function:EQUATIONEQUATIONwhere O𝑝𝑡 means a specific optimization procedure with a fixed number of steps (𝑁 ). However, using such a specific optimization procedure may not yield favorable results in our formulation. One reason is that the distance between the trained network parameters 𝜽 T and the initial network parameters 𝜽 S may be too large, and there may be multiple local minima in the loss reduction process, making it difficult to pass through.Instead of matching the parameters of 𝜽 S and trained 𝜽 T , we match the updated 𝜽 S 𝑡 and 𝜽 T 𝑡 at each training iteration 𝑡 from the same initialization to overcome the problem of local minima traps. This converts the goal of parameter matching into smaller goals for each iteration which breaks down Equations (5, 6, 7) to:EQUATION)EQUATIONEQUATIONwhere 𝑇 denotes the total number of iterations. We hope that through this method of goal decomposition, we can train the network with the generated point cloud set S and original training set T to get the similar parameters after each iteration (𝜽 S 𝑡 ≈ 𝜽 T 𝑡 ). During the gradient descent optimization of iteration 𝑡 + 1, the update rule of O𝑝𝑡 is:EQUATION)EQUATION)where 𝜂 𝜽 refers to the learning rate. For a network designed for point clouds, e.g., PointNet, the assumption is that at each iteration 𝑡, 𝜽 S 𝑡 and 𝜽 T 𝑡 will become equal. This allows the synthetic input S to be optimized by minimizing the difference between the mean gradients, based on gradient matching theory [45, 50, 81] :EQUATIONEQUATIONEQUATIONwhere 𝐷 denotes the Mean Squared Error (MSE) between the two gradients at each layer. Instead of directly minimizing the difference between the parameters 𝜽 S 𝑡 and 𝜽 T 𝑡 , the gradient distance is used as a simplified alternative to calculate S. This is because the gradient directly reflects the change in parameters, avoiding the need for recalculating the previous parameters 𝜽 𝑛 (𝑛 = 0, ..., 𝑛 -1) when the parameters are not significantly different, which leads to faster convergence and lower memory consumption.The gradient matching loss 𝐷 (•; •) in Equation 15measures the distance between two gradients. According to the network layer type and width, the size of gradients is different in each layer. Here, we flatten the gradient vectors to 1 × 𝐷 𝑖 where 𝐷 𝑖 denotes the total gradient size in the 𝑖-th network layer. We compute the total layerwise gradient losses as:EQUATION)where G S i and G T i denote the gradients of S and T of the 𝑖-th layer, respectively. With gradient loss backpropagation, the points in S could move correspondingly to achieve similar gradients in each iteration. In our experiments, we find that those "critical points" move more significantly in the optimization process, which is discussed and illustrated in Sec. 6.2.Additionally, the point cloud set S generated by our algorithm contains both point cloud data and the corresponding labels. Due to the complexity and learning difficulty, it is hard to optimize data and labels at the same time to get good results. Therefore, we fix a certain class label in a minibatch for generating the corresponding point cloud in a certain class. 
training algorithm: 
 We illustrate our algorithm in Algorithm 2. We first choose a point cloud classification network backbone 𝜙, e.g., PointNet, and initialize the synthetic training set S with the introduced nearest-featuremean strategy. The entire algorithm involves an outer loop for network parameter 𝜃 optimization and an inner loop for synthetic data S optimization. For the outer loop, we re-randomly initialize 𝜃 0 after 𝑇 iterations for 𝐾 times to increase the data generalization ability. In the inner loop, given the iteration 𝑡, we sample two training batches 𝐵 S 𝑐 and 𝐵 T 𝑐 from both the synthetic and original training set from the same class 𝑐. Then we compute the gradient matching loss between gradients ∇ 𝜽 L S 𝑐 (𝜽 𝑡 ) and ∇ 𝜽 L T 𝑐 (𝜽 𝑡 ) by applying gradient descent with learning rate 𝜂 S . The gradient matching is performed class by class. After each iteration 𝑡, we optimize 𝜃 by computing the classification loss on the updated synthetic images S. Though we could optimize S that are from multiple classes, we find that imitating the mean gradients of the data from a single class is easier compared to those of multiple classes. 
experiments: 
 Evaluation Datasets. We evaluate our proposed method on both synthetic and real-scanned classification datasets. 1) ModelNet. ModelNet [61] is a widely used dataset for point cloud classification. Experiments are conducted on both ModelNet10 and ModelNet40. ModelNet10 contains 4,899 CAD models from 10 categories, with 3,991 shapes for training and 908 for testing. ModelNet40 contains 12,311 shapes from 40 categories, split into 9,843 for training and 2,468 for testing. 2) ShapeNet. We also evaluate our methods on ShapeNetPart for classification, a subset of ShapeNet [9] with 16 categories containing 12,137 and 2,874 objects for training and testing, respectively. 3) ScanObjectNN. ScanObjectNN [53] is a challenging real-scanned dataset containing 11,636 training objects and 2,814 testing objects that are categorized into 15 classes in the real world. We evaluated our method on ScanObjectNN without backgrounds. Evaluated Networks. We use PointNet as the default classification backbone in all experiments if not specified. In the crossarchitecture experiment, we test following 3D classifiers: PointNet [41] , PointNet++ [42] , DGCNN [56] , PointNeXt [43] , PointMLP [38] , covering the pointwise-MLP-based, convolution-based, and graph-based methods. Experimental Settings. We evaluate our method in three settings, generating a synthetic dataset containing one point cloud per category, 1% point clouds of the whole dataset, and 5% point clouds of the whole dataset. Each experiment involves two phases. First, we learn to synthesize a small synthetic set from a large real training set. Then we use the learned synthetic set to train randomly initialized neural networks and evaluate their performance on the real testing set. We use point clouds with 2048 points as inputs. We run each experiment 5 times and report the best performance of the learned synthetic set. The batch size 𝐵 T 𝑐 and 𝐵 S 𝑐 for sampling the original dataset and synthetic dataset are all set to 128 for PointNet and 32 for all other networks on a single A5000. We first use a learning rate of 0.0001 to optimize the synthetic point clouds for 1,000 iterations. After we get the synthetic dataset, we train new networks with a learning rate of 0.001 for 300 epochs with batch size 128. Baselines We evaluate the performance of our method against baselines as follows:• Random point clouds: It involves randomly selecting 𝑀 point clouds from each class in the training set. • Herding: It uses the Herding algorithm [2, 7, 11, 59] , which is the same as our initialization strategy to select representative samples without further optimization. • K-Center: It uses the K-Center algorithm [47] , another coreset approach, which selects multiple center samples and minimizes the largest distance between a data sample and its nearest center.• DC [81] is a 2D dataset condensation approach. The core idea is to generate the synthetic dataset in a way that produces the same path of model updates as if training on the original dataset, which is similar to our gradient matching progress.In our experiments, we evaluated the performance of our proposed method by comparing it with various baseline methods, as depicted in Table 1 . We employed PointNet as the default network backbone.Our results show that though some coreset methods, such as Herding and K-Center, outperformed random selection, our approach demonstrated a clear advantage over these selection methods as it was not restricted to a subset of the original dataset. Our approach outperforms those directly applying 2D dataset condensation methods, such as DC and DM, indicating the better convergence of our synthetic dataset. Our synthetic data achieved 73.5% accuracy using only 10 point clouds on the ModelNet10 dataset and 83.5% accuracy with only 5% of the point clouds. Our method also performed well on the ModelNet40 and ShapeNet datasets. On the real-world ScanObjectNN dataset, our method achieved 72.0 % accuracy using only 5% of the data, which is almost the same as the performance achieved using the entire dataset (73.1% accuracy). Table 2 : Time consumption of learning the synthetic data and training PointNet with the synthetic data. The synthetic point cloud dataset only needs to be generated once.We visualize the learned synthetic point clouds of ModelNet10 and ModelNet40 in Figure 3 . We find that the synthetic point clouds are visually recognizable, and we further conduct further analysis to explore why our generated synthetic point clouds yield better training performance.In the visualization from Figure 3 , we observe that the network changes some of the point positions, which leads us to consider whether our method is modifying some of the "critical points" of the original point clouds, which is discussed in PointNet and PointNet++ [41, 42] . Using PointNet as the backbone, we illustrate the critical points that activate the final max-pooling layer. We visualize the synthetic point clouds and their corresponding critical points at different epochs in Figure 4 . It can be seen that after optimization, the overall point clouds do not change much, but the critical points have been significantly altered, which verifies our hypothesis.In our default settings, we initialize our synthetic set with our proposed nearest-feature-mean based strategy. Here, we make a comparison via initializing the point clouds with random samples. In Table 3 , we show the averaged results with 20 different random seeds. We could observe that we could get better performance with our initialized strategy on all datasets, indicating that the synthetic dataset S could converge better with our initialization. We give a time consumption analysis in Table 2 conducted on a single A5000. We report the time for learning the synthetic set with different numbers of point clouds per class, and the time for training PointNet with the synthetic set. Table 2 shows that the time for learning the synthetic set with 1 sample, 1% samples, and 5% samples per class for 1,000 iterations is less than training on the whole set for 300 epochs. With a smaller training set size, the training efficiency has been improved significantly. It is worth noting that the synthetic point cloud dataset only needs to be generated once and then distributed to different users for efficient training without re-generation.It is expected that the synthetic dataset will also be effective in training networks with different architectures. Thus we could generate a synthetic set using one backbone to gain generalization performance to different networks. We verify this through conducting two experiments that aim to evaluate the cross-architecture generalization.To begin with, we analyze the synthetic datasets generated through three different PointNet architectures with different layer numbers and feature dimensions. Specifically, we employ three distinct PointNet configurations, namely the standard PointNet (which involves 5 shared Multi-Layer Perceptron (MLP) layers for feature extraction), the PointNet-small (which involves 3 shared MLP layers), and the PointNet-large (which involves 7 shared MLP layers). Specifically, we use shared MLP layers with (64, 128, 512) Nest, we generate synthetic datasets using a range of networks including PointNet, PointNet++, DGCNN, PointNeXt, PointMLP, and VoxelNet, all configured with default settings for classification. Cross-testing was performed across these networks and the results were evaluated on ModelNet10 with one point cloud per category, which are presented in Table 5 . Results indicate that the synthetic dataset generated by PointNet produces the best performance on all the different architectures. We hypothesize this is because all other baselines involve sorting and ranking operations (e.g., TopK operators in Pytorch) for performing local convolution, such as ball query (PointNet++) and group convolution (DGCNN) operators. These kinds of operations are demonstrated as not gradient backpropagating friendly in [12] . Therefore, leveraging PointNet as the classification backbone, which is without sorting and ranking operations, could better optimize the synthetic dataset. Furthermore, the generated synthetic set using PointNet also performs well on other baselines, demonstrating that it could well represent the original set and is generalized with different architectures. Note that we set a small learning rate to 0.00001 for all baselines except PointNet, as we found that a larger learning rate will make the results collapes. As outlined in our introduction, the synthesized compact point cloud dataset can be utilized in two key applications: 1) It can facilitate continual learning in point cloud field, when processing new, Point Cloud Continual Learning. We adapted our method for continual learning to incrementally learn point clouds from new classes while preserving the performance on existing classes. We use the setting in [40] as our baseline, in which it progressively stores training samples in memory to maintain class balance. We replaced its Herding sample selection process by using our generated synthetic point cloud dataset. This includes generating synthetic point clouds from current classes and storing them in memory. We then assess the model performance in a task-incremental learning scenario using the ModelNet40 dataset of 40 classes.Our method is benchmarked against Random selection, Herding, and DC, using a memory budget of 5 synthetic point clouds per class on the ModelNet40 dataset. We structure the learning into 5 and 10 steps, dividing the 40 classes into equal parts for each step. Utilizing the PointNet backbone, our approach consistently outperformed the others in both 5-step and 10-step learning scenarios which is shown in Figure 5 . This indicates that our condensed dataset is more representative of the original dataset. 
conclusion and future works: 
 The purpose of our paper is to explore the feasibility of training a point cloud classification network to obtain high accuracy using only a reduced number of synthetic point clouds derived from a large dataset. This objective is achieved by formulating it as a parameter-matching problem where a network can attain comparable network parameters after being trained on both the original and the generated synthetic point clouds. To resolve this challenge, a nearest-feature-mean based initialization strategy and effective iterative gradient matching approach are introduced. Experimental results demonstrate the effectiveness of the proposed method across various point cloud classification datasets and its application in point cloud continual learning and data privacy protection. In future work, besides the classification task, we aim to explore the application to more complex point cloud tasks. 
acknowledgement: 
 This research was supported by the National Natural Science Foundation of China (NSFC) under Grant U22A2094. This research was also supported by the advanced computing resources provided by the Supercomputing Center of the USTC. 
 
 Appendix:  


Decision: Initially the paper received 2 positive and 1 negative ratings. After the rebuttal, two reviewers raised their ratings, and all reviewers reached a concensus on accepting this work. After carefully reading both the reviews and the paper, the AC agrees with the reviewers' recommendations, since this submission hold merits in introducing an interesting task and approach for 3D point cloud learning. Although this work only conducts experiments on fundamental 3D classification task using a few baselines, the AC expects it to be a pioneering work that can inspire more following trials.

Reviews: review: paper_summary: paper_summary: This paper focuses on a very interesting task, i.e., information point cloud extraction task, where the authors tend to extract a small but informative sub-point set to achieve an algorithm with acceptable performance. Specifically, the authors develop a two-stage point removing algorithm, including nearest-feature-mean based initialization for small point set and gradient-matching strategy based parameter-matching solver. The empirical experiments show the effectiveness of the developed algorithm on popular ModelNet, ShapeNet, and ScanObjectNN datasets.
strength_weakness: strength_weakness: 1. This paper studies a very interesting problem, namely, how to sample a small-scale informative point cloud set from a large-scale raw point cloud dataset to maintain performance without degradation.
limitations: limitations: 1. How to choose the feature function in the nearest-feature-mean based initialization stage? The authors seem to choose an already trained PointNet encoder as the feature selection function. What dataset is the PointNet encoder trained on?

2. The number of the informative sub-point set is manually selected according to the prior knowledge. The authors should add additional ablation experiments to verify the impact of different number of point sets.

3. The strategy of extracting the subset of informative point clouds is similar to the idea of information bottleneck. I would like to see the author give more theoretical explanation from the perspective of information bottleneck. In addition, additional citations on information bottleneck should be added.

4. Is it possible to extend the proposed method to other point cloud analysis tasks, such as point cloud registration?
paper_summary: paper_summary: This paper proposes a method for distilling point cloud datasets to generate a more informative distilled dataset, referred to as the Informative Point Cloud Dataset. To achieve this objective, the authors propose a two-stage approach to generate the Informative point cloud synthetic dataset. Initially, a strategy based on the nearest feature mean is introduced to initialize the synthetic dataset. Subsequently, a gradient matching strategy is employed to iteratively refine these initial point clouds.
strength_weakness: strength_weakness: 1. The authors' idea of introducing dataset distillation from existing image datasets into point clouds is quite interesting.
2. The paper is well-organized and easy to read, with ample references provided.
limitations: limitations: 1. Although the concept of dataset distillation proposed by the authors is quite interesting, it seems that this work primarily involves generating a smaller point cloud guided by the original complete point cloud through classification loss. This smaller point cloud is then used for training or testing a new network. This approach appears to be quite similar to existing learnable point cloud downsampling methods. The authors should clarify the differences between their proposed dataset distillation and current learnable downsampling approaches.

2. Distilling denser point clouds into sparser, meaningful point clouds is a significant task as it can promote the practical application of dense point clouds. However, the experiments in the paper are overly simplistic.
 1) Simple Datasets: The authors only analyzed relatively simple object point cloud datasets such as ModelNet, ScanObjectNN, and ShapeNet, which inherently contain a small number of points. Dataset distillation experiments would be more meaningful on more realistic and complex datasets, where an object consists of hundreds of thousands or millions of points.
 2) Simple Tasks: The authors only experimented with classification tasks. In fact, classification is a simple task where a few key points are often sufficient to achieve the same accuracy as a point cloud with a large number of points. This does not adequately demonstrate the true significance of dataset distillation. It is recommended that the authors conduct more extensive experiments on more complex analytical or generative tasks.

For these reasons, the authors' simplistic experimental results do not sufficiently support their overall motivation and ideas. Therefore, I am inclined to reject this paper and encourage the authors to conduct more meaningful and comprehensive experiments to support their story.
paper_summary: paper_summary: In order to alleviate the high cost of data storage and training time for point cloud classification tasks, this paper innovatively proposes a gradient-based point movement method to extract small-scale data sets from large point cloud data sets. Furthermore, a nearest-feature mean based synthetic point cloud initialization method is introduced. The method proposed in this article can synthesize a new data set that is only 5% of the size of the original data set but can achieve comparable classification results.
strength_weakness: strength_weakness: 1.The article raises a question worth pondering: Can a large data set be compressed into a small data set while maintaining performance on downstream tasks?

2.The article innovatively proposes a gradient-based point moving method. This method provides a new perspective for tasks such as data set extraction, generation, compression, and federated learning, that is, generating data with similar training effects based on gradient matching.

3.The writing is standardized and well structured.
limitations: limitations: 1.Insufficient motivation. The gradient-based point movement method is the main argument of the article, however, the motivation for it is not fully demonstrated. The initialization method also does not have enough preparation.

2.The experimental results are not sufficiently discussed. It is recommended to supplement each experiment with a corresponding discussion.

3.From the visualization results, the initialization method almost dominates the characteristics of the generated data set. Therefore, more comparisons and discussions of different initialization methods are expected to be reported to illustrate the impact of initialization methods on point movement methods.

4.Few references in the past two years.