LinNet: Linear Network for Efficient Point Cloud Representation Learning

Authors: hao  deng
kunlei  jing
shengmei  cheng
cheng  liu
jiawei  ru
jiang  bo
lin  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 11:12:37

Abstract: Point-based methods have made significant progress, but improving their scalability in large-scale 3D scenes is still a challenging problem. In this paper, we delve into the point-based method and develop a simpler, faster, stronger variant model, dubbed as LinNet. In particular, we first propose the disassembled set abstraction (DSA) module, which is more effective than the previous version of set abstraction. It achieves more efficient local aggregation by leveraging spatial anisotropy and channel anisotropy separately. Additionally, by mapping 3D point clouds onto 1D space-filling curves, we enable parallelization of downsampling and neighborhood queries on GPUs with linear complexity. 
LinNet, as a purely point-based method, outperforms most previous methods in both indoor and outdoor scenes without any extra attention, and sparse convolution but merely relying on a simple MLP. It achieves the mIoU of 73.7\%, 81.4\%, and 69.1\% on the S3DIS Area5, NuScenes, and SemanticKITTI validation benchmarks, respectively, while speeding up almost 10x times over PointNeXt. Our work further reveals both the efficacy and efficiency potential of the vanilla point-based models in large-scale representation learning. Our code will be available upon publication.

Main paper: 
 introduction: 
 Appealed by the ongoing evolution progress of technologies in robotics, autonomous driving, augmented reality, etc., LiDAR sensors are incrementally integrated into hardware-constrained devices such as mobile devices and AR headsets. This has led to a growing interest in efficient point cloud processing models. Given the limited computational power of mobile devices and embedded systems, the design of mobile-friendly point cloud representation learning algorithms should not only focus on performance but also pay attention to high computational efficiency.Unlike images, point cloud data is irregular and unordered. There are various methods for processing point cloud data in 3D vision. Common approaches include multi-view methods [1, 2, 3] and voxel-based methods [4, 5] . Converting irregular data into the required formal representations often requires additional computation and memory and thus results in the loss of geometric information [6] . Therefore, point-based methods that directly operate on point clouds have emerged. PointNet [7] and PointNet++ [8] are the pioneers of this approach, introducing a general point cloud learning paradigm from local to global. The paradigm consists of two parts: the first part is a spatial neighborhood search strategy, which utilizes algorithms such as furthest point sampling (FPS) and K-nearest neighbors (KNN) to implement sub-sampling and neighborhood grouping of point clouds, respectively. The second part is a trainable local feature extractor. Following them, subsequent extensive research [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] have shown promising results by focusing on the design of more sophisticated extractors.Despite the impressive results that have been achieved in object recognition and semantic segmentation, most of these methods are limited to applications in small-scale 3D point clouds. The main reason is discouraged by the high time complexity of the neighborhood search strategy that the point-based methods adopted. As shown in Fig. 1a . FPS and KNN occupy 46% of the runtime. This draws forth the main motivation of this paper: enhancing the scalability of point-based approaches in large-scale scenes, while maintaining their excellent performance in small-scale tasks.In this paper, we introduce a novel model, named Linear Net (LinNet). Our LinNet derives from inheriting the innovations and overcoming the drawbacks of the PointNet++ paradigm, including a disassembled set abstraction (DSA) module and an efficient point search strategy. Specifically, inspired by MobileNet [20] , we first use two independent MLPs to separately learn depth-wise geometric features of the neighborhood and point-wise semantic features. Then, the geometric features are assigned as biases to the queried neighbors' semantic features, achieving spatially anisotropic neighborhood aggregation. Since the learning of high-dimensional semantic features is point-wise and does not involve the neighborhood, the required floating-point operations (FLOPs) are significantly lower than those needed for SA. Besides, a hash query and linearization sampling strategy are proposed for speeding up point searching. The core of our method is to map the 3D search space onto a segmented curve for acceleration. A sparse point cloud is ordered on that curve, and points adjacent to each other in the curve are also adjacent in space. For neighborhood queries, we store each segmented curve as a bucket in a hash table. When querying the neighborhood, we only need to search in the buckets corresponding to the neighboring curves, which drastically cuts down the search range. Linear sampling ensures uniform sampling by taking the point closest to the origin within each grid as the new sampling point. The method reduces the time complexity to be linear and supports GPU parallelism, resulting in very fast sampling. As shown in Fig. 1 , the additional point cloud search operations take up less than 10% of the model's runtime. By employing these techniques, our method achieves efficient point cloud representation learning and scalability, providing significant performance improvements for large-scale point cloud analysis.The contribution of our paper can be summarized in the following three folds:• We analyze the feature aggregation of the vanilla SA module and introduce a novel efficient and effective DSA module. This strategy effectively reduces computational overhead and achieves performance gains. Moreover, we discuss the superiority of this method from the perspective of weight initialization, emphasizing how these adjustments crucially enhance the overall performance of the network. • To improve scalability in large-scale scenes, a linear complexity point cloud search strategy is introduced, mapping a 3D point cloud to a 1D space-filling curve. This approach drastically reduces the time consumption associated with sub-sampling and neighbors query.N  4 N  3 N  2 N  1 N  (a) Overview (b) DSA Block Hash Query Hash Query PWConv Max Pooling MLP PWConv 4 N  3 N  2 N  1 N  (a) Overview (b) DSA Block• Experiments show that our approach achieves state-of-the-art performance on widely adopted 3D large-scale semantic segmentation benchmarks (S3DIS, NuScenes) and competitive results on small-scale classification tasks (ScanObjectNN and ModelNet40). Extensive ablation studies have also validated the effectiveness of our proposed components. 
related works: 
 Point cloud analysis. Point cloud analysis is primarily approached in two ways. Another approach, exemplified by the PointNet family, directly processes raw point clouds. They introduce a hierarchical feature learning paradigm to recursively capture local geometric structures. By adopting local point representation and multi-scale information, PointNet++ has demonstrated excellent performance and has become a cornerstone of modern point cloud methods [9, 16, 21, 17, 22, 23] . Our LinNet follows the design philosophy of PointNet++ but explores a simpler yet deeper network architecture.Voxel-based methods. Instead of learning directly on discrete points, the sparse convolution [5, 24] first converts the point cloud into a regular grid and then constructs a full convolutional neural network using the discrete sparse tensor. By building a hash table of discrete rasters, neighborhood query and sampling can be performed efficiently with a constant time complexity of O(1). In addition, the hash table construction and query can be implemented in parallel on CUDA, which significantly improves the computational efficiency. However, even though sparse convolution performs well in many large-scale point cloud tasks, it still faces challenges in capturing the fine-grained patterns of point clouds. This is due to the quantization artifacts that may be introduced during voxelization, resulting in the extracted features being limited by the voxel size [25] .Efficient network in computer vision. In computer vision, an efficient network typically refers to a deep learning architecture designed to balance performance and operational efficiency, including aspects like latency, FLOPs, memory, and power consumption. MobileNet [20] use depthwise separable convolutions, making them particularly efficient for mobile and embedded devices. EfficientNet [26] systematically scales the network's width, depth, and resolution, achieving a balance between efficiency and accuracy. Many works in 3D vision [27, 18, 19, 25] are dedicated to optimizing the efficiency and performance of point cloud processing. While such networks often slightly compromise on performance for reduced computational load, our network, as detailed in Section 4, uniquely enhances both efficiency and accuracy.3 Linear NetQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: [TODO]Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation: 
 Given a 3D point cloud V = (P, F) consisting of n points x. For the i-th point x i = (p i , f i ), p i ∈ R 3 and f i ∈ R c are the space coordinates and features, respectively. The task of point cloud semantic segmentation involves assigning a class label to each point x i , while scene classification entails predicting a class label for the entire scene C. The point-based methods usually employ several stages to classify the points or point clouds. In each stage, a downsample layer is first applied to sample the points, reducing the density of the point cloud.In this section, we progressively introduce the DSA modules. Initially, we explore the direct application of separable convolutions in 3D vision. Subsequently, we adopt a balanced approach to separate channel and spatial anisotropy. Finally, we explain the superiority of this method from the perspective of weight initialization, highlighting how these adjustments enhance the overall performance of the network.Revisiting Local Aggregation of Computer Vision. In a standard convolutional kernel, the anisotropy of the weights plays a critical role in capturing local information. This anisotropy can be classified into two categories: spatial and channel anisotropy. Spatial anisotropy refers to the variations among the features of neighboring points within the same feature channel, while channel anisotropy reflects the differences across various feature channels. To improve efficiency, MobileNet [20] achieves speedup by decomposing the standard convolutional kernel: it divides the convolutional kernel into point-wise convolution (PWConv) for channel anisotropy and depth-wise convolution (DWConv) for spatial anisotropy. Due to the sparsity of point clouds, it is impractical to apply DWConv to handle them directly. Instead of achieving anisotropy through parameters, point-based methods approach this by manipulating the input directly and adding anisotropy to the input data. Given a point cloud x i , a typical local aggregation in 3D vision can be formulated as:EQUATIONwhere R is the aggregation function (usually max-pooling) that aggregates the local feature from the neighbors of anchor point x i denoted as {j : (i, j) ∈ N }. || is the concatenate operation in channels. PWConv 3+c →c : R 3+c → R c is an MLP that consists of pointwise convolution, a batch normalization layer, and a ReLU activation function. Here, the neighborhood features of the different anchors come from the same query set, so they are isotropic. The coordinates are anisotropic as they are relative to their respective anchor. Concatenating together the isotropic features and anisotropic relative coordinates gives the input anisotropy.Depth-wise Separate Set Abstraction. Corresponding to the separate weights, we initially chose to separate the inputs directly as follows:EQUATION)Since all f ′ j come from the same set, Eq. ( 2) is identity to Eq. ( 3):EQUATIONFigure 3 : Training on S3DIS.This separation is necessary for several reasons. First, since the features are derived from the same query set, there is no need to apply a shared-weight PWConv on the neighboring features [18] . If the feature dimension is c and the number of neighbors is k, the computational complexity would be kc 2 . After separation, this complexity is reduced to c 2 . Second, the distribution patterns of coordinates and features differ significantly, and using independent convolutional kernels allows for better capture of these differences. However, as encountered with MobileNet, the speedup often comes at the cost of reduced accuracy. As illustrated in Fig. 3 , DSSA accelerates the model's convergence during the initial few epochs. Yet, as training progresses, the convergence slows down, and the model's overall performance starts to degrade. The main reason lies in the lack of spatial-wise anisotropy between neighbor features. When features are processed independently of their spatial relationships, it can lead to a loss of contextual information critical for certain tasks, such as those involving complex spatial structures or detailed textural information.Disassembled set abstraction. Building on the principles outlined above, we propose a novel method for lightweight local aggregation that addresses the inherent challenges of separating coordinates and features. The proposed disassembled set abstraction (DSA) module can be formulated as:EQUATIONwhere BN is a batch normalization layer [28] . The spatial anisotropy derived from relative positions is integrated into the neighborhood feature aggregation as a manner of bias. This integration ensures that the aggregation of neighborhood features is closely linked to the spatial distribution of the point cloud, thereby enhancing the model's robustness under varying spatial distributions. Interestingly, the Eq. ( 1) and Eq. ( 4) are actually mathematically equivalent during forward propagation. However, the DSA module exhibits faster convergence and lower loss compared to the SA modules (see Fig. 3 ). This phenomenon can be attributed to variations in the initialization of weights as follows.Excluding bias, Eq. ( 1) uses a combined weight matrix W (dimensions c×(c+3)) to process semantic and geometric data simultaneously. For a given neighbor j, the input vector x j = [f j , p j -p i ] results in the output:EQUATIONIn this case, Kaiming initialization sets W as a normal distribution N (0, 2 c+3 ), potentially reducing the impact of geometric data due to its smaller proportional weight. In stark contrast, the DSA module separates the processing of semantic and geometric data using two distinct weight matrices, W f for semantic (dimensions c × c) and W p for geometric data (dimensions c × 3), leading to:EQUATIONThe specific initialization W f ∼ N (0, 2 c ) and W p ∼ N (0, 2 3 ) allows for a more balanced influence of geometric data, thus enhancing the network's ability to extract and utilize geometric information effectively. It is noteworthy that the PWConv between high-dimensional semantic features is applied directly to the point cloud features, rather than to the neighborhood. Additionally, the number of input channels for PWConv applied to the neighborhood is only 3, which is significantly smaller than c (with a minimum of 64 in the segmentation model). As illustrated in Fig 4 , DSA requires substantially fewer FLOPs compared to SA, thereby improving computational efficiency and making it more suitable for large-scale applications. Recent literature [29, 30] employ space-filling curves to serialize point clouds, which are then uniformly divided into patches and fed into a transformer architecture. Inspired by this, we map points in 3D space onto a space-filling curve for accelerating point searching. Denote the coordinates p as (x, y, z) and the batch index as b. The shuffled key is defined as a 64-bit integer:EQUATIONwhere ≪ denotes left bit-shift, s denotes the grid size, and | denotes bitwise OR. It is worth noting that, unlike PTv3 [30] , which contains only one point per grid cell, our approach allows multiple points to share the same key. We utilize shuffled keys to store these points in memory in an ordered manner, ensuring that elements sharing the same key are close to each other in memory. As shown in Fig. 5b , points on the same solid line are in the same grid. Through this, the point cloudV is partitioned into m sub-regions [V 1 , V 2 , . . . V m ].Hash query. By storing coordinates in spatial order, a hash table can be constructed to efficiently manage and query neighbors. Each bucket represents a non-empty grid. The key is the shuffled key of the grid, and the value contains two parts: the index of the first point in the grid and the count of points in that grid. If the number of grids is m, the time complexity of building the hash table is O(m). During the query phase, for each point, we find the 27 (i.e., 3 × 3 × 3) neighboring grids and query the hash table with these keys. Finally, the top k nearest points are selected. Assuming each point's 27-grid neighborhood contains p points on average, identifying the closest k points involves maintaining a heap with a complexity of O(p log k) and a final sorting step costing O(k log k). Thus, the total computational complexity isO(m + N (p log k + k log k)), while that of kNN is O(kN 2 ).Linearization sampling. To achieve uniform and fast sampling, we select a point from each subset according to the following rule:EQUATIONwhere (j) ∈ M i are the points of i-th sub-regions. This rule ensures that the newly sampled points are the closest to the origin within their respective grids, guaranteeing uniform sampling. The method has a linear time complexity and supports GPU parallelism, resulting in very fast sampling speeds.The overall architecture is illustrated in Fig. 2 . For segmentation tasks, we use both encoders and decoders. To ensure a fair comparison, in the indoor dataset S3DIS, we configure the encoder depth as [4, 7, 4, 4] , which is the same as PointNeXt. For the outdoor dataset, the encoder depth is set to [4, 4, 7, 4] . Specifically, the channel numbers for these stages are set to [C, 2C, 4C, 8C], with C being 64. For the classification task, only the encoder is used. Considering that the dataset for the classification task is small and PointNeXt is already capable of real-time response, we do not use the proposed linear search strategy, ensuring a fairer comparison between the DSA module and the SA module. [33] point --RangeNet++ [36] point 65.5 -Salsanext [37] hybrid 72.2 -MinkUNet [4] voxel 73.3 -PolarNet [38] point 71.0 -PVKD [39] hybrid 76.0 -AMVNet [40] -76.1 -Cylender3D [41] cylender 76.1 77.2 SPVNAS [42] hybrid 77.4 -RPVNet [43] hybrid Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: [TODO] Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 To validate the effectiveness of LinNet, we conduct experiments in 3D semantic segmentation and 3D object classification tasks. We also conduct an extensive ablation study to analyze each component in LinNet. More details of the experiments can be found in the Appendix.Data and metric. S3DIS [49] (Stanford Large-Scale 3D Indoor Spaces) is a challenging benchmark that comprises 6 extensive indoor areas, 271 rooms, and 13 semantic categories, which represent different types of objects and room elements commonly found in indoor environments. Each point in the dataset is labeled with one of the 13 semantic categories, such as table, door, chair, column, and window, in addition to clutter. Following previous work [22] , we subsample the grid before sending the point cloud to the network. The grid size and maximum number of points are set to 0.04m and 24000 respectively. During training, we crop the center point by the pre-set maximum number of points and discard the rest. During testing, the entire scene is processed. The experiment results are shown in Tab. 1. For evaluation metrics, we choose class-wise intersection over union (mIoU), mean of class-wise accuracy (mAcc), and overall point-wise accuracy (OA). Given that S3DIS is relatively small, we conduct further experiments on the NuScenes [50] dataset to validate the efficiency of our model. In this dataset, each point is annotated with one of the 16 semantic categories. The dataset encompasses 1,000 scenes collected in Boston and Singapore, reflecting diverse urban environments. We adhered to the official segmentation protocol, allocating 700 scenes for training, 150 for validation, and another 150 for testing, ensuring a balanced and comprehensive evaluation of our model's performance across varied scenes.Performance. The results are shown in Tab. 1 and Tab. 2. Following Point Transformer v2 [11] and CondaFormer [35] , we also employ the test time augmentation (TTA) strategy to achieve fairer comparisons, and results using the TTA strategy are labeled with ‡ . In indoor dataset S3DIS, our LinNet outperforms the SoTA point-based method PointNeXt [22] by 2.9%, 0.2%, 2.5% in terms of mIoU, OA, and mAcc, respectively. We also visualize the segment result in Fig. 6 . In largescale dataset NuScenes, LinNet also outperforms all previous methods. It is worth noting that our approach utilizes a pure MLP network, employing solely point-wise convolutions. This highlights that sophisticated feature extractors, such as attention mechanisms and graph structures, are not essential for achieving robust segmentation capabilities. Moreover, our method is strictly based on point data, devoid of any sparse convolutions, which further underscores the scalability of point-based methods in managing large-scale point clouds effectively. We chose the ScanObjectNN [53] and ModelNet40 [54] datasets to assess the classification capabilities of our model, and the results are shown in Fig. 3 . We also report the parameters, FlOPs, and throughput. Following PointNeXt, the input channels of the models used in ScanobjectNN and ModelNet40 are 32 and 64 respectively. The model parameters are computed for C = 32.ScanObjectNN. It contains approximately 15,000 real scanned objects divided into 15 categories with 2,902 instances, which presents substantial challenges due to occlusions and noise. We conduct experiments on PB_T50_RS, the most challenging and frequently used variant of ScanObjectNN.According to the report, the proposed LinNet significantly outperformed existing methods in Overall Accuracy (OA) and mean Accuracy (mAcc), using fewer model parameters and achieving faster processing speeds. LinNet achieved an OA of 88.6% and a mAcc of 87.3% on ScanObjectNN. Note that we employ the same training protocols and experimental conditions as the SOTA benchmark, PointNeXt. Nonetheless, we still achieve an OA improvement of 0.4%, while other PointNeXt style architectures (e.g., PointMetaBase [17] ) using the same experimental setup only achieve an OA improvement of 0.1%.ModelNet40. This dataset is a widely-used object classification dataset, comprises 12,311 3D computer graphics CAD models across 40 categories. Our results, as indicated, are highly competitive and exceed those of most previous methods. LinNet achieved an Overall Accuracy of 93.9% on ModelNet40, surpassing graph-based models like DGCNN [13] , transformer-based models such as Point Transformer [9] , and KPConv [33] .We evaluate the efficiency of our model at four different scales of points: 20k, 50k, 100k, and 200k. The down sampling rate is about 4. The experiments are performed on an RTX 3090. The models we compared include PointNeXt-XL [22] and Point Transformer v2 [11] . As shown in Fig. 7a , the We also explore the latency of the proposed point cloud search strategy. Note that the horizontal axis is on a logarithmic scale. As shown in Fig. 7b , linearization sampling and hash query based on space-filling curves leads to greater speedup as the point cloud size increases. The proposed linearization sampling can be up to a thousand times faster than FPS.We conduct ablation experiments of the model to verify the validity of each component, and all experimental results are averaged over three times in the S3DIS area 5 unless otherwise stated.LinNet. We perform ablation experiments on different modules introduced in LinNet: linearization sampling (LS), hash query (HQ), depth-wise separate set abstraction (DSSA), and disassembled set abstraction (DSA), with the results shown in Tab. 4. The delay was measured on a 20k number of point clouds. The model used in Exp. I is PointNeXt, which is the baseline result of our design. In Exp. II through VI, we progressively incorporated each of our proposed components, improving the baseline accuracy to 73.1% and reducing the latency to 34 ms.Disassembled set abstraction. In Tab. 5, we investigate the design of the feature aggregation module to improve the aggregation of semantic and geometric information. We utilize ASSA [18] and PosPool [19] , instead of DSA module. Additionally, we evaluate the performance impact of replacing max pooling with average pooling. Exp. ( 1) and (2) show a 1% decrease in accuracy when using simple separation of inputs directly. Comparing Exp. (1) with Exp. (6) indicates that the proposed DSA module performs better than the SA module. Exp. ( 3) and ( 4), which employ ASSA and PosPool respectively, demonstrate performance degradation, highlighting that our datadriven approach outperforms the parameter-free strategy in merging low-dimensional coordinates and high-dimensional semantics. Exp. (5) shows that max pooling is more compatible with our network.Model scalability. We refer to the default LinNet as LinNet-Base and designed two variants with different numbers of trainable parameters: LinNet-Small, which has one-tenth the parameters of LinNet-Base, and LinNet-Large, which has four times the parameters of LinNet-Base. We test the performance of these models on the outdoor dataset NuScenes, and the results are summarized in Tab. 6. The mIoU on the validation set steadily improves with increasing model size. Additionally, since the models are linear, the increase in parameters does not result in significantly higher latency. Notably, without using TTA, our LinNet-Small achieves a validation accuracy of 77.6% with only 1.7M parameters, surpassing the 38M parameter sparse convolution method MinkUnet [4] .Memory footprint. We conduct experiments to evaluate memory usage during both training and inference phases on the NuScenes dataset, utilizing an RTX 4090 graphics card with all tests conducted at a batch size of 1. We include comparisons with the baseline model PointNeXt [22] and the sparse convolution method MinkUNet [4] . Our findings reveal that PointNeXt suffers from out-of-memory issues when handling large-scale scenes, highlighting scalability challenges. In contrast, our DSA module significantly reduces memory consumption by avoiding high-dimensional feature transformations on neighboring point clouds. Given that MinkUNet starts with 32 input channels, we conducted similar tests with our LinNet-Small model, which also has 32 initial feature channels, for a direct comparison. The results are shown in Tab. 7. Although LinNet-Small consumes more memory than MinkUNet, it is crucial to note that LinNet-Small, with only 1.7M parameters, achieves a validation accuracy of 77.6%, surpassing the 38M parameter sparse convolution method MinkUNet, which achieves 73.3%.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: [TODO] Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 Conclusion. In this paper, we implement a point-based approach with linear complexity. Unlike current point-based methods, our framework uses space-filling curves to achieve neighbor query and downsampling with linear complexity. Additionally, we introduce a disassembled set aggregation module, which aggregates local features simply and elegantly, significantly reducing the redundant computations in neighborhoods and greatly enhancing scalability. Extensive experiments on multiple benchmarks demonstrate the efficiency and state-of-the-art performance of our method.Limitation and Future Work. Although the proposed approach largely addresses the scalability challenges of point-based approaches in large-scale scenes, the distribution of point cloud data in memory in point-based approaches tends to be discontinuous, leading to inefficient memory access. This increases the cache miss rate, which in turn reduces the processing speed. Point-wise neighborhood aggregation also consumes a significant amount of memory. We hope that future work will address the significantly higher memory footprint than sparse convolution methods. [8] 83.0 - [14] 85.9 63. 9 [13] 83. Table 12 : Sem. seg. on Sem. KITTI. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: [TODO]Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: SPVNAS [42] 64.7 66.4 Cylinder3D [41] 64.3 67.8 PVKD [39] -71.2 2DPASS [44] 69.3 72.9 WaffleIron [47] 68.0 70.8 SphereFormer [46] 67.8 74.8 RangeFormer [45] 67.6 73.3 MinkUNet [4] 63.8 -LinNet (ours) 69.1 70.4 Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper initially received borderline to positive ratings (BR, BA, WA, SA). The authors appreciated the novelty of the proposed point cloud feature learning approach but had some specific constraints regarding aspects of the method. Most of these concerns appeared resolved after the rebuttal, with 3 reviewers leaning towards acceptance (6,6,8). The only reviewer left on a BR was unresponsive during the rebuttal and discussion stage. Taken all this into account, the final recommendation is to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a method for point cloud segmentation and classification. The main contribution is making the local aggregation dependent on the anchor point. The approach demonstrates improvements of one to two percent on S3DIS and NuScenes datasets compared to existing methods.
strength_weakness: strength_weakness: - The paper's writing seems overly complex with lots of unnecessary jargon making it difficult to follow the core ideas and contributions.
- The improvements in performance, while consistent, are relatively small (1-2%), which raises questions about the practical significance of -the method.
- The modifications proposed are individually well-motivated but seem somewhat ad hoc, lacking a strong theoretical foundation.
questions: questions: 1. Have you explored whether a more general model could learn the invariances you've built into LinNet without explicit architectural choices?
2. Given the relatively small improvements in accuracy, what do you see as the main practical advantages of LinNet over existing methods?
limitations: limitations: The discussion of limitations is sufficient.
paper_summary: paper_summary: The paper proposes a novel lightweight backbone network model for input point cloud data suitable for global and local per-point feature extraction. It relies on two main ideas: (1) the separate processing of point coordinates and features (and a further combination of these two streams of features before the neighborhood pooling operation), (2) the use of the space-filling curves to define local neighborhoods that allow hash queries and linear complexity sampling.

The approach is evaluated in the point cloud classification and segmentation tasks on ModelNet40, ScanObjectNN and S3DIS, NuScenes datasets respectively, demonstrating performance competitive to state of the art. Additional experiments include ablations exploring the efficacy of every proposed component and comparison to other methods in terms of model efficiency.
strength_weakness: strength_weakness: * All the improvements (except for the NuScenes dataset) are not particularly distinctive.
* While present the efficiency is not exploited in any presented applications (for classification point clouds are small, so other methods work fast as well, for segmentation there are no efficiency comparisons).
* The text is written well but some figures can be improved.
questions: questions: The proposed method relies on the space-filling curves introduced in PointTransformer v.3, which is mentioned multiple times in the paper. At the same time, it is not considered for comparison. While PTv3 is concurrent, for completeness, it would be nice to include the results from it, especially since this work already acknowledges the existence of PTv3. So my question is how this method compares to PTv3 in terms of performance and efficiency?

Figure 5 shows a nice uniformly distributed (over the grid) point cloud but in practice, any grid-based discretization suffers to some extent from the discretization artifacts. The proposed method can in principle have cells with single points. Do these cells exist in practice and do they cause any problems?

Figure 1: a) and b) have the same area but the total time of b) is significantly lower which is misleading.

Figure 3: The choice of the number of inputs, intermediate features, and outputs is either arbitrary or not clear which is confusing. Showing the operations for a single input point would be clearer.

Figure 5: Showing an empty table in c) is not informative. Showing a part of the actual hash table for this example in the figure might work better.
limitations: limitations: The authors properly address the limitations in the submitted draft.
paper_summary: paper_summary: In this work, the authors propose an efficient learning framework for point cloud representation learning. For the computational intensive local aggregation operation, this work proposes Disassembled set abstraction (DSA) to aggregate local features in terms of the spatial distributions of points in a simple and efficient manner. This work also proposes a Linearization sampling strategy and hash query operation to accelerate the sampling and neighbor searching processes. Experiments on classification and semantic segmentation demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: My major concern about this work is the higher performances than transformer-based methods. It may be a little hard to understand why the proposed DSA and hash-based searching operations can improve the performances so greatly. Because these operations are more like the approximation of corresponding operations in PointNet++.
questions: questions: 1. Why do we introduce batch normalization to the aggregated features in Eq.4? As the addition of features and neighborhood features have already introduce the spatial characteristics, I am not sure if this BN is necessary here, or other simple components may also work;

2. In the Hash query part, will the points in a same local grid share the same neighbors? I do not quite get the calculation of complexity for each point.

3. Is the hash grid is pre-constructed before training? Or created repeatedly during training?

3. From the results in Table 1 and Table 3, the proposed method even outperforms transformer-based methods in a more efficiency way. Could the author analyze the reasons behind this? As the improvements of this framework seem to be efficient simplification of existing PointNet++ framework, I am curious why it can improve the performances so obviously.

Please check the grammar also, e.g., In Line 101, the $x_i$ might actually be $p_i$;
limitations: limitations: Yes.
paper_summary: paper_summary: The submission #308 entitled "LinNet: Linear Network for Efficient Point Cloud Representation Learning" introduces a linear network designed for efficient point cloud representation learning. To achieve this task, the authors propose a novel disassembled set abstraction (DSA) module and a linear sampling strategy, which together enhance computational efficiency and scalability. The method maps 3D point clouds onto 1D space-filling curves, allowing for parallelization of downsampling and neighborhood queries on GPUs with linear complexity. This approach achieves state-of-the-art performance across various benchmarks.
strength_weakness: strength_weakness: - One negative point is the significant memory footprint of the approach, as mentioned in the limitations section.
questions: questions: - Would it be possible to conduct a small experiment to measure the memory footprint of the approach?
- It would be interesting to have a few cross-validation tests to know more about the generalization of the technique.
limitations: limitations: The limitations of the approach are well covered at the end of the manuscript.

Comments: title: Supplement on weakness 3
Response
comment: Dear Reviewer zyd3,

Thanks for your constructive suggestions. 

We will improve our paper's quality based on your guidance and comments.

Thank you for recognizing our work! We hope our simple yet effective LinNet would help the community towards a better understanding of point cloud analysis.

Best, Authors
I would like to thank the authors for their clarifications. After reading the comments from other reviewers, I realize I may have been slightly too generous in my initial assessment. However, I still believe this manuscript is worthy of appearing in NeurIPS, and I would like to maintain my initial rating.
Dear Reviewer aBR6,

Thanks for your constructive suggestions and the score increase. 

We will improve our paper's quality based on your guidance and comments.

We hope our simple yet effective LinNet would help the community towards a better understanding of point cloud analysis.

Best, Authors
Thanks for your responses. The initialization could be a potential reason. Considering its good overall performances, I will raise my score to weak accept.
Inspired by the comment from reviewer aBR6, we delve deeper into the reason why DSA performs better than SA from the standpoint of parameter initialization.

To clarify, we replace pointwise convolutions with linear layers and restructure both the DSA and SA modules.
In SA, a single linear layer merges the neighborhood features $\mathbf{f}_j$ and the positional differences $\Delta\mathbf{p}_j$, transforming them linearly to produce the output as follows:
$$
\mathbf{f}_i' = \mathcal{R} _{j:(i, j)\in \mathcal{N}} \\{\text{Linear}^{3+c \mapsto c}([\mathbf{f}_j, \Delta\mathbf{p}_j])\\}.
$$

Contrastingly, DSA utilizes two separate linear layers to process $\mathbf{f}_j$ and $\Delta\mathbf{p}_j$ individually and then combines the outputs. **Ignoring the removal of redundant calculations in the neighborhood and Batch Normalization (BN)**, the DSA module is depicted as:

$$
\mathbf{f}_i' = {\mathcal{R} _{j:(i, j)\in \mathcal{N}} \\{ \text{Linear}^{c \mapsto c}(\mathbf{f}_j) + \text{Linear}^{3 \mapsto c}(\Delta\mathbf{p}_j)} \\}.
$$

Both modules appear mathematically equivalent during forward propagation. Yet, **distinct initializations of the two linear layers in DSA encourage the network to prioritize geometric information more effectively.** Excluding bias, the SA module uses a combined weight matrix $\mathbf{W}$ (dimensions $c \times (c+3)$) to process semantic and geometric data simultaneously. The input vector $\mathbf{x}_j = [\mathbf{f}_j, \Delta\mathbf{p}_j]$ leads to the output:
$$
\mathbf{y}_j = \mathbf{W} \mathbf{x}_j^\text{T}.
$$
Each output channel's contribution is calculated by the product of the weights and inputs, with Kaiming initialization setting $\mathbf{W}$ as a normal distribution $\mathcal{N}(0, \sqrt{\frac{2}{c+3}})$, limiting geometric data's influence due to its smaller proportional weight.

In contrast, DSA segregates the handling of semantic and geometric data using two separate weight matrices, $\mathbf{W}_f$ for semantic (dimensions $c \times c$) and $\mathbf{W}_p$ for geometric data (dimensions $c \times 3$). This results in outputs:
$$
\mathbf{y}_j = \mathbf{W}_f \mathbf{f}_j + \mathbf{W}_p \Delta\mathbf{p}_j.
$$
The initialization $\mathbf{W}_f \sim \mathcal{N}(0, \sqrt{\frac{2}{c}})$ and $\mathbf{W}_p \sim \mathcal{N}(0, \sqrt{\frac{2}{3}})$ allows for a more balanced influence of geometric data, thus enhancing the network's ability to extract and utilize geometric information effectively.

Following your feedback, we have included the additional explanations and comparative analyses in the revised version of the manuscript.
**A:** Thank you for your continued interest in the Disassembled Set Abstraction (DSA) and Set Abstraction (SA). Your question regarding why DSA outperforms SA in terms of data fitting is insightful. I'll provide a more intuitive explanation focusing on the architectural differences and their impacts.

In a nutshell, the **DSA module places more emphasis on the extraction of geometric information which is crucial for point cloud learning**.  

For SA, the feature of an anchor is updated as:
$$
\mathbf{f}_i' = \mathcal{R} _{j:(i, j)\in \mathcal{N}} \\{\text{PWConv}^{3+c \mapsto c}(\mathbf{f}_j||(\mathbf{p}_j-\mathbf{p}_i))\\}.
$$

For DSA, the process is:
$$
\mathbf{\overline{f}}_i  = \text{PWConv}^{c \mapsto c}(\mathbf{f}_i);  
\mathbf{f}_i' = \text{BN} \\{{\mathcal{R} _{j:(i, j)\in \mathcal{N}} \{\overline{\mathbf{f}}_j + \text{PWConv}^{3 \mapsto c}((\mathbf{p}_j-\mathbf{p}_i))} }\\}.
$$

Excluding the Batch Normalization and focusing only on neighbor feature computations for simplicity, let $\mathbf{y}_j=[y_j^1, y_j^2,..., y_j^c]$ represent **the features of the $j$-th neighbor**. Treating pointwise convolution as a linear layer and without considering a bias, the SA model uses a weight matrix $\mathbf{W}$ of dimensions $c \times (c+3)$ to process both semantic and geometric information concurrently. The input $\mathbf{x}_j = [\mathbf{f}_j, \Delta\mathbf{p}_j]$ includes semantic features $\mathbf{f}_j$ (dimension $c$) and geometric features $\Delta\mathbf{p}_j$ (dimension $3$), with the output defined as $\mathbf{y}_j = \mathbf{W}\mathbf{x}_j^\text{T}$. The output for the $k$-th channel is given by:
$$
y_j^k = [w^{k1}, w^{k2}, \ldots, w^{kc}, w^{k(c+1)}, w^{k(c+2)}, w^{k(c+3)}][\mathbf{f}_j, \Delta\mathbf{p}_j]^\text{T}.
$$
With Kaiming initialization, the weight matrix $\mathbf{W}$ is initialized to a normal distribution $\mathcal{N}(0, \sqrt{\frac{2}{c+3}})$. This initialization ensures uniformity across all weights, meaning the weights for geometric inputs contribute $\frac{3}{c+3}$ to the total output. Consequently, the influence of geometric information on the overall results is significantly limited.

In contrast, the DSA model separates the processing of semantic and geometric information through two distinct linear layers. It incorporates two weight matrices, $\mathbf{W}_f$ and $\mathbf{W}_p$, corresponding to the dimensions $c \times c$ and $c \times 3$, respectively. The output is determined by:
$$
\mathbf{y}_j = \mathbf{W}_f \mathbf{f}_j + \mathbf{W}_p \Delta\mathbf{p}_j.
$$
For the $k$-th channel, the output is:
$$
y_j^k = [w_f^{k1}, w_f^{k2}, \ldots, w_f^{kc}]\mathbf{f}_j^\text{T} + [w_p^{k1}, w_p^{k2}, w_p^{k3}]{\Delta\mathbf{p}_j}^\text{T},
$$
where $\mathbf{W}_f \sim \mathcal{N}(0, \sqrt{\frac{2}{c}}),\mathbf{W}_p \sim \mathcal{N}(0, \sqrt{\frac{2}{3}})$ . This initialization strategy enables the DSA module to appropriately tailor the weights based on the number of input channels. Although there are only three channels dedicated to geometric information, their relatively larger weights enhance the model's capability to more effectively extract geometric information.

In summary, DSA and SA are equivalent in terms of mathematical expression during forward propagation. In SA, a single linear layer merges $\mathbf{f}$ and $\Delta\mathbf{p}$, transforming them linearly to produce the output. In contrast, DSA employs two distinct linear layers to separately process $\mathbf{f}$ and $\Delta\mathbf{p}$, and then sums the outputs. However, the different weight initialization of the two linear layers causes the network to preferentially learn from geometric information. This bias enables the network to more effectively detect patterns associated with geometry, which is particularly advantageous in point cloud analysis.

Thank you once again for your insightful suggestions, which have prompted us to further explore the underlying logic. We hope our response adequately addresses your concerns. If you still have any questions, we are looking forward to hearing them.
Thank you for your rebuttal. It has addressed most of my concerns. However, I am still curious about the reason why DSA outperforms SA. Although Fig.4 confirms that DSA can converge to lower loss than SA, DSA seems to be a more efficient simplification of SA. Could you provide some more intuitive explanation about the reasons behind such improvements? That is, why DSA exhibits a more robust capability to fit the data than SA?