LinNet: Linear Network for Efficient Point Cloud Representation Learning
Authors: Hao Deng
Kunlei Jing
Shengmei Chen
Cheng Liu
Jiawei Ru
Bo Jiang
Lin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Point cloud
Primary area: deep_learning_architectures

Abstract: Point-based methods have made significant progress, but improving their scalability in large-scale 3D scenes is still a challenging problem. In this paper, we delve into the point-based method and develop a simpler, faster, stronger variant model, dubbed as LinNet. In particular, we first propose the disassembled set abstraction (DSA) module, which is more effective than the previous version of set abstraction. It achieves more efficient local aggregation by leveraging spatial anisotropy and channel anisotropy separately. Additionally, by mapping 3D point clouds onto 1D space-filling curves, we enable parallelization of downsampling and neighborhood queries on GPUs with linear complexity. 
LinNet, as a purely point-based method, outperforms most previous methods in both indoor and outdoor scenes without any extra attention, and sparse convolution but merely relying on a simple MLP. It achieves the mIoU of 73.7\%, 81.4\%, and 69.1\% on the S3DIS Area5, NuScenes, and SemanticKITTI validation benchmarks, respectively, while speeding up almost 10x times over PointNeXt. Our work further reveals both the efficacy and efficiency potential of the vanilla point-based models in large-scale representation learning. Our code will be available upon publication.

stronger variant LinNet: Linear Network for Efficient Point Cloud Representation Learning
Hao Dengl 1.2
Kunlei Jing
Shengmei Cheng 1.2 Cheng Liu 1.2
Jiawei Rut.² Jiang Bot Lin Wang
Joint Engineering and Research Center of Advanced Networking and Intelligent
Information Services, School of Information Science and Technology, Northwest University
Shaanxi Key Laboratory of Higher Education Institution of Generative Artificial Intelligence and Mixed Reality
School of Software Engincering. Xi' an Jiaotong University Department of Computing, The Hong Kong Polytechnic University denghaoβ s umaal nwr edu cn, kunlei.. dii.jinnggnn iingijituu... edu. 5244211 80 5 50qq. com, lc@nwu edu, cn
ruji nwwe @stumaii nwu edu. cn, (jiangbo, wangl in) ) @nwu. edu.cd
Abstract
Point-based methods have made significant progress, but improving their scala- bility in large-scale 3D scenes still a challenging problem. In this paper, we delve into the point-based method and develop simpler, model, dubbed as LinNet. particular, we first propose the disassembled set abstraction (DSA) module, which is more effective than the previous version of set abstraction. It achieves more efficient local aggregation by leveraging spa- tial anisotropy and channel anisotropy separately. Additionally, by mapping 3D point clouds onto ID space-filling curves, we enable parallelization sampling and neighborhood queries GPUs with linear complexity. LinNet, as purely point-based method, outperforms most previous methods in both in- door and outdoor scenes without any extra attention, and sparse convolution but merely relying on a simple MLP. It achieves the mloU of 73.7%, 81.4%, and 69.1 % the S3DIS Area5, NuScenes, and SemanticKITTI validation bench- marks, respectively, while speeding up almost 10x times over PointNeXt. Our work further reveals both the efficacy and efficiency potential of the vanilla point- based models in large-scale representation learning. Our code will be available at https: github com wtwm///engg299///1eee0e
down-
Introduction
Appealed by the ongoing evolution progress of technologies in robotics, autonomous driving, aug- mented reality, etc., LiDAR sensors are incrementally integrated into such mobile devices and AR headsets. This has led to growing interest in efficient point cloud processing models. Given the limited computational power of mobile devices and embedded systems, the design of mobile-friende friendly point cloud representation learning algorithms should not only focus on performance also pay attention to high computational efficiency.
devices
Unlike images, point cloud data irregular and unordered. There are various methods for processing
point cloud data in 3D vision. Common approaches include multi-view methods [1, 2, 3] and voxel-based methods [4, 5]. Converting irregular data into the required formal representations often
*Corresponding Authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

S3D15 Azea 5mlol) 80
93.7-
Nin
P
FPS
KNN
SA
0th.
72.9
Point Transformce V2
72.3
PoinfVector
Stedlified Transfomer
96ms
21.3-
PointNeXi
LinNet:
1.5
HQ
DSA
Oth.
20.5
600
Lateney (mn))
(a) Latency decomposition.
(b) Accuracy speed tradcoff.
Figure Latency decomposition. We show the inference run time decomposition. (a) SA: set abstraction; DSA: disassembled set abstraction; LS: linearization sampling; HQ: hash query; Oth.: others. (b) LinNet achieves the highest mloU with extremely low latency compared the comparative point-based approaches. The latency of each network is measured on single Nvidia 3090 GPU, taking a batch of 80k points.
requires additional computation and memory and thus results in the loss of geometric information [6]. Therefore, point-based methods that directly operate on point clouds have emerged. PointNet [7] and PointNet++ [8] are the pioneers of this approach, introducing general point cloud learning paradigm from local to global. The paradigm consists of two parts: the first part is a spatial neighborhood search strategy, which utilizes algorithms such as furthest point sampling (FPS) and K-nearest neighbors (KNN) to implement sub-sampling sampling and neighborhood grouping of point clouds, respectively. The second part is a trainable local feature extractor. Following them, subsequent extensive research [9, 10. 11. 12, 13, 14, 15, 16, 17, 18, 19] have shown promising results by focusing on the design of more sophisticated extractors.
Despite the impressive results that have been achieved in object recognition and semantic segmen- tation, most of these methods are limited to applications in small-scale 3D point clouds. The main reason is discouraged by the high time complexity of the neighborhood search strategy that the point-based methods adopted. As shown in Fig. la. FPS and KNN occupy 46% of the runtime. This draws forth the main motivation of this paper: enhancing the scalability of point-base approaches in large-scale seage-scaleaaee scenes. while maintaining their excellent performance in small-scale tasks.
In this paper, we introduce a novel model, named Linear Net (LinNet). Our LinNet derives from inheriting the innovations and overcoming the drawbacks of the PointNet++ paradigm, including a disassembled set abstraction (DSA) module and an efficient point search strategy. Specifically, inspired by MobileNet [20], we first use two independent MLPs to separately learn depth-wise geometric features of the neighborhood and point-wise semantic features. Then, the geometric features are assigned as biases to the queried neighbors' semantic features, achieving spatially anisotropic neighborhood aggregation. Since the learning of high-dimensional semantic features is point-wise and does not involve the neighborhood, the required seaaiig--point operations (FL.Ps) are significantly lower than those needed for SA. Besides, a hash query and linearization sampling strategy are proposed for speeding up point scarching. The core of our method is to map the 3D search space onto a segmented curve for accelcration. A sparse point cloud is ordered on that curve, and points adjacent each other in the curve are also adjacent in space. For neighborhood queries, we store each segmented curve as bucket in a hash table. When querying the neighborhood, we only need search in the buckets corresponding the neighboring curves, which drastically cuts down the search range. Linear sampling ensures uniform sampling by taking the point closest to the origin within each grid as the new sampling point. The method reduces the time complexity to be linear and supports GPU parallelism, resulting in very fast sampling. As shown in Fig. 1, the additional point cloud search operations take up less than 10% of the model's ''s runtime. By employing these techniques, our method achieves efficient point cloud representation learning and scalability, providing significant performance improvements for large-scale point cloud analysis.
The contribution of our paper can be summarized in the following three folds:
We analyze the feature aggregation of the vanilla SA module and introduce novel efficient and effective DSA module. This strategy effectively reduces computational overhead and achieves performance gains. Moreover, we discuss the superiority of this method from the perspective of weight initialization, emphasizing how these adjustments crucially enhance the overall performance of the network.

---

F
P
PWCon
Hash Query
Down Sample
DSA Block
Down Sample
DSA Block
Down Sample
DSA Block
Down Sample
DSA Block
Hesh Quar
PWCorV
Point Cloud
Embedding
Max Pooling
MLP
xN
XN
xN
xN
(a) Overview
(b) DSA Block
Figure 2: Overall architecture. (a) Overview of the framework. The whole network consists of an embedding layer and four stages, each containing a downsampling layer and N, disassembled SA blocks. (b) Structure of the DSA blocks. Each DSA block consists of DSA module and extra MLPs.
To improve scalability in large-scale scenes, linear complexity point cloud search strategy is introduced, mapping 3D point cloud to ID space-filling curve. This approach drastically reduces the time consumption associated with sub-sampling and neighbors query.
Experiments show that our approach achieves satte-of-the-ar performance widely adopted 3D large-scall scale semantic segmentation benchmarks (S3DIS, NuScenes) and competitive results on small-scale classification tasks (ScanObjectNN and ModelNer40). Extensive ablation studies have also validated the effectiveness of our proposed components.
Related Works
Point cloud analysis. Point cloud analysis is primarily approached in ways. Another approach, exemplified by the PointNet family, directly processes raw point clouds. They introduce a hierarchical feature learning paradigm recursively capture local geometric structures. By adopting local point representation and multi-scale information, PointNet++ has demonstrated excellent performance and has become cornerstone of modern point cloud methods [9, 16, 21, 17, 22, 23). Our LinNet follows the design philosophy of PointNet++ but explores a simpler yet deeper network architecture.
Voxel-based methods. Instead of learning directly discrete points, the sparse convolution [5, 24] first converts the point cloud into a regular grid and then constructs a full convolutional neural network using the discrete sparse tensor. By building hash table of discrete rasters, neighborhood query and sampling can be performed efficiently with a constant time complexity of ((1). In addition, the hash table construction and query be implemented in parallel on CUDA, which significantly improves the computational efficiency. However, even though sparse convolution performs well in many large-scale point cloud tasks, it still faces challenges in capturing the fine-grained patterns of point clouds. This is due to the quantization artifacts that may be introduced during voxelization, resulting in the extracted features being limited by the voxel size [25].
Efficient network computer vision. In computer vision, an efficient network typically refers to deep learning architecture designed to balance performance and operational efficiency, including aspects like latency, FLOPs, memory, and power consumption. MobileNet [20] use depthwise separa- ble convolutions, making them particularly efficient for mobile and embedded devices. EfficientNet [26] systematically scales the network's width, depth, and resolution, achieving a balance between efficiency and accuracy. Many works in 3D vision [27, 18. 19, 25] are dedicated optimizing the efficiency and performance of point cloud processing. While such networks often slightly compro- mise performance for reduced computational load, network, detailed in Section 4, uniquely enhances both efficiency and accuracy.
3

---

Linear Net
3.1 Problem Formulation
Given a 3D point cloud V = (P, consisting of n points æ. For the i-th point x; (p:: f.). P: E IR's and fü E I' are the space coordinates and features, respectively. The task of point cloud semantic segmentation involves assigning a class label to each point æ;, while scene classification entails predicting class label for the entire scene C. The point-based methods usually employ several stages classify the points or point clouds. In each stage, downsample layer is first applied to sample the points, reducing the density of the point cloud.
3.2 Disassembled Set Abstraction
In this section, we introduce the DSA modules. First, we progressively explore the direct application of separable convolutions in 3D vision. Then, we adopt a harmonized approach to separate channel and spatial anisotropy. Finally, we discuss the superiority of this method in terms of weight initialization, showing why the adjustment can improve overall network performance.
Revisiting Local Aggregation of Computer Vision. In standard convolutional kernel, the anisotropy of the weights plays a critical role in capturing local information. This anisotropy can be classified into categories: spatial and channel anisotropy. Spatial anisotropy refers to the variations among the features of neighboring points within the same feature channel, while channel anisotropy reflects the differences across various feature channels. To improve efficiency, MobileNet [20] achieves speedup by decomposing the standard convolutional kernel: it divides the convolutional kernel into point-wise convolution (PWConv) for channel anisotropy and depth -iiee convolution (DWConv) for spatial anisotropy. Due to the sparsity of point clouds, it is impractical to apply DWConv to handle them directly. Instead of achieving anisotropy through parameters, point-based methods approach this by manipulating the input directly and adding anisotropy to the input data, Given a point cloud æ;, a typical local aggregation in 3D vision can formulated as:
f = RAMAEENN P (fsl((p - p:))},
where R is max- max-pooling that aggregates the local feature from the neighbors of anchor point æ; denoted as {j (i.j) W]. is the concatenate operation in channels. PWCony g3+e + I is an MLP that consists of pointwise convolution, batch normalization layer, and ReLU activation function. Here, the neighborhood features of the different anchors come from the same query set, so they are isotropic. The coordinates are anisotropic as they are relative to their respective anchor. Concatenating together the isotropic features and anisotropic coordinates gives the input anisotropy.
Depth-wise Separate Set Abstraction. Corresponding to the separate weights, we initially chose to separate the inputs directly as follows:
(<p( R):((,,) (PWConV C ((,)) + Ritis ( - -P:)).
(2)
Since all Jj come from the same set, Eq. (2) identity to Eq. (3):
f. P
(3)
f RRUGEE (F,}
p.))).
This separation necessary for several reasons. First, since the features are derived from the same query set, there is no need apply
-eieh PWConv on the neighboring features [18]. If the feature dimension is and the number of neighbors is k. the computational complexity would be kc? After separation, this complexity is reduced c². Second, the distribution patterns of coordinates and features differ significantly, and using independent convolutional kernels allows for better capture of these differences. However, as encountered with MobileNet, the speedup often comes at the cost of
1.6
SA
DSSA
Training Loss
1.4
1.2
1.0
0
20
40
60
80
100
Training Epochs
Figure 3: Training S3DIS.

---

Sum
Sum
Sum
Sum
Neighbor Query
Neighbor Query
Neighbor Query
NaighborQuay
Neighhbr Query
Neighbor Query
(a) SA
(b) DSSA
(c) DSA
Figure 4: Comparison of various local aggregation. Each square represents the semantic feature, while each rectangle represents relative coordinates. The number of neighbors indicates mapping in high- figh-dimensional space (e.g., from 3 + or c), and the red line indicates a mapping from a lower dimension to a higher dimension (e.g., from 3 to c). More blue lines indicate more computation.
The blue line
reduced accuracy. As illustrated in Fig. 3, DSSA accelerates the model's convergence during the initial few epochs. Yet, as training progresses, the convergence slows down, and the model's overall performance starts to degrade. The main reason lies in the lack of spatial-wise anisotropy between feature.... When features are processed independently lead to a loss of contextual information critical for certain tasks, such as those involving complex spatial structures or detailed textural information.
their spatial relationships, it can
Disassembled set abstraction. Building on the principles outlined above, we propose a novel method for lightweight local aggregation that addresses the inherent challenges of separating coordinates and features. The proposed disassembled set abstraction (DSA) module can be formulated as:
f. PWConv (fi);
f BN J.. PWCony " p.))}},
where BN is a batch normalization layer [28]. The spatial anisotropy derived from relative positions integrated into the neighborhood feature aggregation
manner of bias. This integration ensures that the aggregation of neighborhood features is closely linked to the spatial distribution of the point cloud, thereby enhancing the model's 's robustness under varying spatial distributions. Interestingly, the Eq. (1) and Eq. (4) are actually mathematically equivalent during forward propagation. However, the DSA module exhibits faster convergence and lower loss compared the SA modules (see Fig. 3). This phenomenon can be attributed to variations in the initialization of weights as follows.
Excluding bias, Eq. (1) uses combined weight matrix W (dimensions ex (c+3)) to process semantic and geometric data simultaneously. For given neighbor j. the input vector X₃- = [f - pi] results in the output:
yJ = Wx].
In this case, Kaiming initialization sets W normal distribution N(O, - potentially reducing the impact of geometric data due to its smaller proportional weight. In stark contrast, the DSA module separates the processing semantic and geometric data using two distinct weight matrices, W, for semantic (dimensions e x c) and W, p for geometric data (dimensions e x 3), leading to:
y, W W (P) P:)
The specific initialization W, No(0, v and W, NO / -3( allows for more balanced influence of geometric data, thus enhancing the network's ability to extract and utilize geometric information effectively. is noteworthy that the PWConv between high- nigh-dimensional semantic features is applied directly to the point cloud features, rather than to the neighborhood. Additionally, the number of input channels for PWConv applied to the neighborhood is only 3, which is significantly smaller than (with minimum of 64 in the segmentation model). As illustrated in Fig 4, DSA requires substantially fewer FLOPSs compared to SA, thereby improving computational efficiency and making it more suitable for large-scale applications.
5

---

Cell
Startinder
Count
0
4
1
4
3
2
7
4
11
3
30
6
19
2
7
21
4
Input points
(b) Space-filling curve
(c) Hash table
(d) Linearization sampling
Figure 5: Efficient point clouds searching for query and sampling. (a) The input point cloud. (b) Point cloud after linearization by space-filling curves. Points connected by solid arrows are within the same grid, while dashed lines connect points between different grids. (c) Store each segment of the solid line as a hash table. (d) The point closest to the center of each region represented by segments connected by solid arrows is chosen the new sampling point.
3.3 Point Searching Strategy
Recent literature [29, 30] employ space-filling curves to serialize point clouds, which are then uniformly divided into patches and fed into transformer architecture. Inspired by this, we map points in 3D space onto a space-filling curve for accelerating point searching. Denote the coordinates p (x, y, 2) and the batch index as b. The shuffled key is defined 64-bit integer:
(7) where < denotes left bit-shift, denotes the grid size, and denotes bitwise OR. it worth noting that, unlike PTv3 [30], which contains only one point per grid cell, our approach allows multiple points to share the same key. We utilize shuffled keys to store these points in memory in an ordered manner, ensuring that elements sharing the same key are close to each other in memory. As shown in Fig. 5b, points the same solid line are in the same grid. Through this, the point cloud V is partitioned into m sub-regions [Vi, V2,... Vm].
Key (b 34) (1=/s] < 36) (ly/s] < 18) ([z/s]),
Hash query. By storing coordinates in spatial order, a hash table can be constructed to efficiently manage and query neighbors. Each bucket represents non-empty grid. The key is the shuffled key of the grid, and the value contains two parts: the index of the first point in the grid and the count of points in that grid. If the number of grids is m, the time complexity of building the hash table is (m.) During the query phase, for each point, we find the 27 (i.e., 3 x 3 x 3) neighboring grids and query the hash table with these keys. Finally, the top & nearest points are selected. Assuming each point's 27-grid neighborhood contains p points maintaining a heap with a complexity of O(p log k) and a final sorting step costing O log k). Thus, the total computational complexity is O(m + N (p log k k klog k)), while that of &NN is Ok:N2)
average, identifying the closest & points involves
Linearization sampling. To achieve uniform and fast sampling, we select point from each subset according to the following rule:
idx; = arg min (lp, - [p5/s] x s|l2),
(8)
()EM:
where (j) E M, are the points of i-th sub-regions. This rule ensures that the newly sampled points are the closest to the origin within their respective grids, guaranteeing uniform sampling. The method has a linear time complexity and supports GPU parallelism, resulting in very fast sampling speeds.
3.4 Network Architecture
The overall architecture is illustrated in Fig. 2. For segmentation tasks, we use both encoders and decoders. To ensure fair comparison, in the indoor dataset S3DIS, we configure the encoder depth as [4, 7, 4, 4], which is the same as PointNeXt. For the outdoor dataset, the encoder depth is set to [4, 7, 4J. Specificallyy, the channel numbers for these stages are set IC, 2C, 4C, 8CJ, with C being 64. For the classification task, only the encoder is used. Considering that the dataset for the classification task is small and PointNeXt is already capable of real-time response, we do not use the proposed linear search strategy, ensuring a fairer comparison between the DSA module and the SA module.

---

Table I: Indoor sem. seg. on S3DIS Area 5.
Table Outdoor sem. seg. on NuScenes.
Methods
Input
mloU
OA
Acc
Methods
Input
Val
Test
PointNet [7]
point
41.1
66.2
RandLA-Ner [27]
point
PointCNN [31]
point
75.6
KPConv [33]
point
PointWeb [32]
point
87.0
66.6
RangeNet++ [36]
point
65.5
PointNct++ [8]
point
68.6
87.7
67.1
Salsanext [37]
hybrid
72.2
KPConv [33]
point
67.1
79.1
MinkUNet [4]
voxel
73.3
RandLA-Net [27]
point
82.0
PolarNet [38]
point
71.0
PTv1 [9]
point
70.4
90.8
76.5
PVKD [39]
hybrid
76.0
CBL [34]
point
69.4
90.6
AMVNet [40]
76.1
PointMeta [17]
point
72.0
91.4
Cylender3D [41]
cylender
76.1
77.2
ASSANet [18]
point
66.8
SPVNAS [42]
hyhrid
77.4
Str. Trans. [21]
point
72.0
91.5
78.1
RPVNet [43]
hybrid
77.6
Fast PT [25]
point
77.3
2DPASS [44]
hyhrid
80.8
PTv2 [11]
point
72.6
91.6
78.0
RangeFormer [45]]
78.1
80.1
PTv3* [30]
point
73.4
SphereFormer [46]
voxel
78,4
81.9
ConDaFormer" [35]
point
73.5
92.4
78.9
Wafflelron' [47]
point
79.1
PointVector [16]
point
72.3
91.0
78.1
OACNN [48]
voxel
78.9
PointNeXt [22]
point
70.8
91.7
77.5
PTv3* [30]
point
80.4
82.7
LinNet (ours)
point
72.9
91.3
78.6
LinNet(ours)
point
80,4
LinNet (ours)
point
73.7
91.9
79.0
LinNet *(ours)
point
81.4
82.3
4
Experiments
To validate the effectiveness of LinNet, we conduct experiments in 3D semantic segmentation and 3D object classification tasks. We also conduct extensive ablation study to analyze each component in LinNet. More details of the experiments can be found in the Appendix.
4.1 Semantic Segmentation
Data and metrie. S3DIS [49] (Stanford Large-Scale 3D Indoor Spaces) a challenging benchmark that comprises 6 extensive indoor areas, 271 rooms, and 13 semantic categories, which represent different types of objects and room elements commonly found indoor environments. Each point in the dataset is labeled with one of the 13 semantic categories, such table, door, chair, column, and window, in addition to clutter. Following previous work [22], we subsample the grid before sending the point cloud to the network. The grid size and maximum number of points 0.04m and 24000 respectively. During training, we crop the center point by the pre-set maximum number points and discard the rest. During testing, the entire scene is processed. The experiment results are shown in Tab. 1. For evaluation metries, we choose class-wise intersection over union (mloU), mean of class-wise accuracy (mAcc), and overall point-wise accuracy (OA). Given that S3DIS is relatively small, we conduct further experiments on the NuScenes [50] dataset to validate the efficiency of our model. In this dataset, each point annotated with one of the 16 semantic categories. The dataset encompasses 1,000 scenes collected in Boston and Singapore, reflecting diverse urban environments. We adhered the official segmentation protocol, allocating 700 scenes for training, 150 for validation, and another 150 for testing, ensuring a balanced and comprehensive evaluation of our model's performance across varied scenes.
set
Performance. The results are shown Tab. and Tab. 2. Following Point Transformer v2 [11] and CondaFormer [35]. we also employ the test time augmentation (TTA) strategy to achieve fairer comparisons, and results using the TTA strategy are labeled with f, In indoor dataset S3DIS, our LinNet outperforms the SoTA point-based method PointNeXt [22] by 2.9%. 0.2%. 2.5% in terms of mloU, OA, and mAcc, respectively. We also visualize the segment result in Fig. 6. In large- scale dataset NuScenes, LinNet also outperforms all previous methods. It worth noting that our approach utilizes a pure MLP network, employing solely wisnt-wise convolutions. This highlights that sophisticated feature extractors, such as attention mechanisms and graph structures, are not essential for achieving robust segmentation capabilities. Moreover, our method is strictly based on point data, devoid of any sparse convolutions, which further underscores the scalability of point-based methods in managing large-scale point clouds effectively.

---

Input
Ground tuuhh
LinNet
Input
Ground truth
LinNet
ceiling
floor
wall
beam
column
window
door
table
chair
sofa
bookcase
board
clutter
Figure 6: Comparative Visualization of Semantic Segmentation S3DIS.
Table 3: 3D object classification in ScanObjectNN and ModelNet40. Averaged results in three random runs using 1024 points input without normals and without voting are reported.
ScanObjectNN (PB. T50 RS)
ModelNet40
Params.
FLOPs
Throughput (ins./sec)
Method
OA (%)
mAce (%)
OA
mAce (%)
M
G
PoiniNet [7]
68.2
63.4
89.2
86,2
3.5
0.9
4199
PoinCCNN [14]
78.5
75.1
92.2
88.1
0.6
DGCNN [13]
78.1
90.2
1.8
DeepGCN [51]
93.6
90.9
2.2
3.9
KPConv [33]
92.9
14.3
ASSANet-L [18]
92.9
118.4
Vimp [3]
80.5++0.3
93.0+H..4
90.5++0.8
0.8
MVTN [1]
82.8
93.5
92.2
3.5
1.8
2-
Point Cloud Transfomerr [10]
93.2
2.9
2.3
CurveNet [52]
93.8
2.0
PoinMLP [23]
85.4:11.3
83.911.5
94.1
91.3
13.2
31.3
220
PoietMetaBase [17]
87.9110.2
0.6
PointNeXt [22]
87.7:10.4
85.8:0.6
93.7110.
90.9:10.5
1.4
2126
LinNet (ours)
88.2++0.4
86.6:10.7
93.6+H0.2
91.0++0.5
1.4
0.6
1852
4.2 Object classification
chose the ScanObjectNN [53] and ModelNet40 [54] datasets to assess the classification capa- bilities of our model, and the results shown in Fig. 3. We also report the parameters, FIOPs, and throughput. Following PointNeXt, the input channels of the models used in ScanobjectNN and ModelNet40 are 32 and 64 respectively. The model parameters
computed for C 32.
ScanObjectNN. It contains approximately 15,000 real scanned objects divided into 15 categories with 2,902 instances, which presents substantial challenges due to occlusions and noise. We conduct experiments on PB_T50 RS, the most challenging and frequently used variant of ScanObjectNN. According to the report, the proposed LinNet significantly outperformed existing methods in Overall Accuracy (OA) and mean Aceuracy (mAcc), using fewer model parameters and achieving faster processing speeds. LinNet achieved OA of 88.6% and mAce of 87.3% on ScanObjectNN. Note that we employ the same training protocols and experimental conditions as the SOTA benchmark, PointNeXt. Nonetheless, architectures (c.g., PointMetaBase [17]) using the same experimental setup only achieve an ÓA improvement of 0.1%.
still achieve OA improvement of 0.4%, while other PointNeXt style
ModelNet40. This dataset is a widely-used object classification dataset, comprises 12,311 1 3D computer graphics CAD models across 40 categories. Our results, as indicated, are highly competitive and exceed those most previous methods. LinNet achieved an Overall Accuracy of 93.9% on MedelNet40, surpassing graph-based models like DGCNN [13], transformer-based models such as Point Transformer [9], and KPConv [33].
4.3 Model Efficiency
We evaluate the efficiency of our model at four different scales of points: 20k, 50k, 100k, and 200k. The down sampling rate is about 4. The experiments are performed on an RTX 3090. The models compared include PointNeXt-XL [22] and Point Transformer v2 []1]. As shown in Fig. 7a, the

---

3000
PoiniNeXi
KNN
Linesrization Sampling
2
10'
Hish Query
RPS
2500
LinNet
2000
10"
1500
1000
10'
13
500
0---
20%
200k
50k
200k
(a) Efficiency comparison of different models.
(b) Efficiency comparison of different components.
Figure 7: Efficiency comparisons. The horizontal axis represents the number of points in the input tensor and the vertical axis represents the running time in milliseconds.
Table 4: Model design ablation.
Table 5: Ablation on the DSA design.
ID
S.
HQ
DDSA
DSA
mloU
Latency(ms)
Methods
mloU
A
70.8
89
Vanilla SA
72.0
-1.1
72.0
45
(2) DSSA
71.0
-2.1
III
70.8
80
(3) ASSA
70.5
-2.6
IV
72.0
38
(4)
PosPool
69.9
-3.2
V
71.0
32
(5) Avg. pooling
71.2
-1.9
VI
73.1
34
(6) DSA
73.1
latency of LinNet grows linearly with the scale of the point cloud. Point Transformer v2 employs grid pooling, a technique similar to our linearization sampling, both characterized by linear time complexity. However, thanks to the simplicity and efficiency of our disassembled set aggregation, our model exhibits only half the latency Point Transformer v2. Notably, at the 200k level, our LinNet model operates 13 times faster than PointNeXt, demonstrating significant improvements in processing speed. We also explore the latency of the proposed point cloud search strategy. Note that the horizontal axis is on logarithmic scale. As shown in Fig. 7b, linearization sampling and hash query based on space-filling curves leads to greater speedup as the point cloud size increases. The proposed linearization sampling can be up to thousand times faster than FPS.
4.4 Ablation Study
We conduct ablation experiments of the model to verify the validity of each component, and all experimental results are averaged over three times in the S3DIS area 5 unless otherwise stated.
LinNet. We perform ablation experiments on different modules introduced LinNet: lincarization sampling (LS), hash query (HQ), depth-wise separate set abstraction (DSSA), and disassembled set abstraction (DSA), with the results shown Tab. 4. The delay was measured on 20k number of point clouds. The model used in Exp. I is PointNeXt, which is the baseline result of our design. In Exp. through VI, we progressively incorporated each of our proposed components, improving the baseline accuracy to 73.1% % and reducing the latency to 34 ms.
Disassembled set abstraction. module to improve the aggregation of semantic and geometric information. We utilize ASSA [18] and PosPool [19], instead of DSA module. Additionally, we evaluate the performance impact of replacing max pooling with average pooling. Exp. (1) and (2) show 1% decrease in accuracy when using simple separation of inputs directly. Comparing Exp. (1) with Exp. (6) indicates that the proposed DSA module performs better than the SA module. Exp. (3) and (4), which employ ASSA and PosPool respectively. demonstrate performance degradation, highlighting that our data- driven approach outperforms the parameter-free strategy in merging oow-iiensioona coordinates and high-dimensional semantics. Exp. (5) shows that max pooling is more compatible with our network.
Tab. 5, we investigate the design of the feature aggregation
Model scalability. We refer to the default LinNet as LinNet-Base and designed two variants with different numbers of trainable parameters: Linael,Small, which has one-tenth the parameters of LinNet-Bace and LinNet-Large, which has four times the parameters of LinNet-Bass We test the

---

Table 6: Model scalability. Latency and FLOPs are measured with 24k points.
Name
Channels
Depths
Param(M).
FLOPs (G)
Latency (ms)
mloU(%)
Small
32
[2,2,2,2]
1.5
2.1
27
77.6
Base
64
[4,4,7,4]
16.5
7.8
34
80.4
Large
128
[4,4.7,4]
65.6
24.9
42
81.3
Table 7: Memory footprint during training and inference the NuScenes dataset.
Model
Training Memory
Inference Memory
MinkUNet PointNeXt
2.6 GB
1.4 GB
Out of Memory
Out of Memory
LinNet-Smmll
5.2 GB
4.9 GB
LinNet
16 GB
13 GB
performance of these models on the outdoor dataset NuScenes, and the results are summarized in Tab. 6. The mloU on the validation set steadily improves with increasing model size. Additionally, since the models are linear, the increase in parameters does not result in significantly higher latency. Notably, without using TTA, our LinNet-Small achieves validation accuracy of 77.6% with only 1.7M parameters, surpassing the 38M parameter sparse convolution method MinkUnet [4].
Memory footprint. We conduct experiments to evaluate memory usage during both training and inference phases on the NuScenes dataset, utilizing an RTX 4090 graphics card with all tests conducted at batch size of 1. We include comparisons with the baseline model PointNeXt [22] and the sparse convolution method MinkUNet [4]. Our findings reveal that PointNeXt suffers from out-of-memory issues when handling large-scale scenes, highlighting scalability challenges. In contrast, our DSA module significantly reduces memory consumption by avoiding high-dimensional feature transformations on neighboring point clouds. Given that MinkUNet starts with 32 input channels, we conducted similar tests with our LinNet-Small model, which also has 32 initial feature channels, for direct comparison. The results are shown in Tab. 7. Although LinNet-Sman consumes more memory than Mink it is crucial to note that NinNet-Small, with only 1.7M parameters, achieves a validation accuracy of 77.6%. surpassing the 38M parameter sparse convolution method MinkUNet which achieves 73.3%.
Conclusion
Conclusion. this paper. we implement a point-based approach with linear complexity. Unlike current boint-based methods, framework uses space-filling curves to achieve neighbor query and downsampling with linear complexity. Additionally, we introduce a disassembled set aggregation module, which aggregates local features simply and elegantly, significantly reducing the redundant computations in neighborhoods and greatly enhancing scalability. Extensive experiments on multiple benchmarks demonstrate the efficiency and state-of-the-art performance of method.
Limitation and Future Work. Although the proposed approach largely addresses the scalability challenges of point-based approaches in large-scale scenes, the distribution of point cloud data in memory in point-based approaches tends to be discontinuous, leading to inefficient memory access. This increases the cache miss rate, which in turn reduces the processing speed. Point-wise neighborhood aggregation also consumes a significant amount of memory. We hope that future work will address the significantly higher memory footprint than sparse convolution methods.
Acknowledgment. The authors would like to thank the reviewers of NeurlPS'24 24 for their constructive suggestions. This work was supported by the Key Research and Development Program of Shaanxi Province of China under Grant 2024GX-YBXM-140 in part by the Qinchuangyuan "Scientist + Engineer" Team Construction Project of Shaanxi Province of China under Grant 2022KXJ-OW in part by the National Natrral Science Foundation of China under Grant 42271 1 400 and in part by Northwest University Graduate Innovation Project under Grant CX2024194.
10

---

References
[1]
Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mytn: Multi-view transformation network for 3d shape recognition. CVPR, 2021. 1,8
[2] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Leamnd-MMllerr Multi-view convolational neural networks for 3d shape recognition. In ICCV, 2015.
[3] Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with simple and effective baseline. In ICML, 2021. 1,8
[4] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski
convolutional neural networks. In CVPR, 2019. 7, 10, 15
[5] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In IROS, 2015. 3.3
[6] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel cnn for efficient 3d deep learning. In NewrIPS, 2019.
[7] Charles R Qi. Hao Su. Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In CVPR, 2017. 2, 7,8, 15
[8] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hicrarchical feature learning point sets in metric space. NewrIPS, 2017. 2, 7, 15
[9] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In CVPR, 2021. 2, 3, 7, 8, 15
[10] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. CVM, 7:197-199, 2021. 2, 8
[11] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-base pooling. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, NeurlPS, 2022. 2, 7,
[12] Lci Wang, Yuchun Huang, Yaolin Hou, Shenman Zhang, and Jie Shan. Graph attention convolution for point cloud semantic segmentation. In CVPR, 2019, 2
[13] Yue Wang, Yonghin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning point clouds. ACM TOG, 38 2019. 2, 8, 15
[14] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x-transformed points. NewrlPS, 2018. 2, 8, 15
[15] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In CVPR, 2019. 2
[16] Xin Deng, WenYu Zhang, Qing Ding, and XinMing Zhang. Pointvector: A vector representation in point cloud analysis. In CVPR, 2023. 2. 3,7
[17] Haojia Lin, Xiawu Zheng, Lijiang Li, Fei Chao, Shanshan Wang, Yan Wang. Yonghong Tian, and Rongrong Ji. Meta architecure for point cloud analysis. arXiv:2211. 2022. 3,
[18] Guocheng Qian, Hasan Hammoud, Guohao Li, Ali Thabet, and Bernard Ghanem. Assanct: An anisotropic separable set abstraction for efficient point cloud representation lcarning. NeurlPS, 2021. 2, 4, 7, 15
[19] Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look local aggregation operators in point cloud analysis. In ECCV, 2020. 2, 3,9
[20] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.0464), 2017. 3,
[21] Xin Lai, Jianhui Liu, Li Jiang. Liwei Wang. Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. In CVPR, 2022. 3.7
[22] Guocheng Qian, Yuchen Li, Houwen Peng. Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In NeurIPS. 2022. 7. 8, 10, 15

---

[23] Xu Ma. Can Qin, Haoxuan You. Haoxi Ran. and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual mip framework. ICLR, 2022. 3,
[24] Benjamin Graham, Martin Engeleke, and Laurens van der Maaten. 3d semantic segmentation with
submanifold sparse convolutional networks. In CVPR, 2018. 3
[25] Chunghyun Park, Yoonwoo Jeong. Minsu Cho, and Jacsik Park. Fast point transformer. In CVPR, 2022. 3,
[26] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML,, 2019.
(27] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Learning semantic segmentation of large-scale point clouds with random sampling. PAMI, 44(1 44((3):838-8354 2021. 7,7
[28] Sergey Loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[29] Peng-Shin Shuai Wang. Octformer: Octree-based transformers for 3D point clouds. ACM Transactions on Graphics (SIGGRAPH), 42(4), 2023.
[30] Xiaoyang Wu, Li Jiang, Peng-Shune Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster, stronger. In CVPR, 2024. 6,
[31] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In CVPR, 2019. 7
[32] Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia. Pointweb: Enhancing local neighhorhood features for point cloud processing. In CVPR, 2019. 7
[33] Hugues Thomas, Charles R Qi, Jean-Emmanus Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpeonv: Flexible and deformable convolution for point clouds. In ICCV, 2019. 7, 15
[34] Liyao Tang. Yibing Zhan, Zhe Chen, Baosheng Yu, and Dacheng Tao. Contrastive boundary learning for point cloud segmentation. In CVPR, 2022. 7
[35] Lunhao Duan, Shanshan Zhao, Nan Xue, Mingming Gong, Gui-Song Xia, and Dacheng Tao. Condaformer: Disassembled transformer with local structure enhancement for 3d point cloud understanding. In NeurlPS, 2023. 7
[36] Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss, Rangenet++: Fast and accurate lidar semantic segmentation. In IROS, 2019. 15
[37] Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. ISVC, 2020. 15
[38] Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, Hassan Foroosh. Polamett: An improved grid representation for online lidar point clouds semantic segmentation. In CVPR, 2020. 7, 15
[39] Yuenan Hou, Xinge Zhu, Yuexin Ma, Chen Change Loy, and Yikang Li. Point-to-voxel knowledge distillation for lidar semantic segmentation. In CVPR, 2022. 7, 15
[40] Venice Erin Liong, Thi Ngoe Tho Nguyen, Sergi Widjaja, Dhananjai Sharma, and Zhuang Jie Chong. Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation, 2020. 15
[41] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong. Yuexin Ma. Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. CVPR, 2021. 7, 15
[42] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse poxet-voxel convolution. In ECCV, 2020. 7, 15
[43] Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu. Rpvnet: A deep and efficient range-point- voxel fusion network for lidar point cloud segmentation. In CVPR, 2021. 7, 15
[44] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shuguang Cui, and Zhen Li. 2dpass: 2d priors assisted semantic segmentation lidar point clouds. In ECCV, 2022. 7, 15
12

---

[45] Lingdong Kong, Youquan Liu, Runnan Chen, Yuexin Ma, Xinge Zhu, Yikang Li. Yuenan Hou, Yu Qiao, and Ziwei Liu. Rethinking range view representation for lidar segmentation. In CVPR, 2023. 7, 15
[46] Xin Lai, Yukang Chen, Fanbin Lu, Jianhui Liu, and Jiaya Jia. Spherical transformer for lidar-based 3d recognition. In CVPR, 2023. 7, 15
[47] Gilles Puy. Alexandre Boulch, and Renaud Marlet. Using waffle iron for automolive point cloud semantic segmentation. In CVPR, 2023. 15
[48] Bohao Peng, Xiaoyang Wu, Li Jiang, Yukang Chen, Hengshuang Zhao, Zhuotao Tian, and Jiaya Jia. Oa-cnns: Omni-adaptive sparse cnns for 3d semantic segmentation. In CVPR, 2024. 7
[49] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Joannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In CVPR, 2016. 7, 14
[50] Holger Cacsar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020. 7,
[51] Guohao Li, Matthias Muller, Ali Thahet, and Bernard Ghanem. Deepgens: Can gens go deep as cnns? In CVPR, 2019.
[52] Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. Walk in the cloud: Learning curves for point clouds shape analysis. In CVPR, 2021.
[53] Mikacla Angelina Uy, Quang- Hicu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset classification model on real-world data. ICCV, 2019. 8, 14
[54] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR, 2015. 8, 14
[55] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In CVPR, 2019. 14
[56] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Lconidas Guibas. A scalable active framework for region annotation in 3d shape collections. ToG, 35(6):1-12, 1-12, 2016.
[57] Haoxi Ran, Jun Liu, and Chengjie Wang. Surface representation for point clouds. In CVPR, 2022. 15
[58] Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep lcarning on point sets with parameterized convolutional filters. In ECCV, 2018.
13

---

Appendix
In the appendix, we provide more experiment details Sec. A, and more experiment results in Sec.
B.
A
Experimental Details
This section provides a detailed description of the experimental setup for each dataset.
Experimental environment.
CUDA version: 11.3
PyTorch version: 1.12.1
GPU: Nvidia RTX 4090D x
CPU: AMD EPYC 9754 128-Core
Training details. The specific model training settings are shown in Tab. 8 and Tab. cross-entropy loss in all experiments.
We used
Table 8: Data augmentation.
Rotate
Flip
Scale
Jitter
Chromatic Drop
Height
Grid Size
ScanOhjeetNN
ModelNet40
S3DIS
/
0.04
NuScenes
0.05
Sem. KITTI
0.05
Table 9: Training setting.
Epoch
Learning Rate
Weight Decay
Scheduler
Optimizer
Batch Size
ScanObjectNN
250
0.002
0.05
Cosine
Adam
ModeINet40
600
0.001
0.05
Cosine
Adam W
32
S3DIS
3000
0.01
0.0001
Cosine
Adam W
NuScenes
50
0.002
0.05
Cosine
AdamW
12
Sem. KITTI
50
0.002
0.05
Cosine
AdamW
Data license. Our experiments use open-source datasets widely applied for 3D recognition research.
The ScanObjeCtNN [53], SemanticKITTI [55], dataset is under the MIT license, while S3DIS [49], NuScenes [50], and ModelNet40 [54] have custom licenses that only allow academic use.
B Additional Quantitative Results
In this section, we present additional quantitative results of SemanticKITT [55] for 3D semantic segmentation. addition, we provide semantic segmentation results for each category NuScenes (see Tab. 10) and S3DIS Area 5 (see Tab. 11).
SemanticKITTI. The SemanticKITTI dataset consists of sequences from the original KITTI dataset, comprising a total of 22 sequences. Fach sequence contains approximately 1,000 LiDAR scans, amounting to around 20,000 individual frames. The result shown Tab. 12. The mloU of validation set and test set are 69.1% % and 70.4% respectively.
S3DIS 6-fold cross- wros-validatiin. To evaluate the generalization capabilities, we perform 6-fold cross- validation on the S3DIS dataset to ensure a robust assessment of our model's performance across different subsets of data. The results are shown in Fig. 13.
Nomalization layer type. We conducted ablation studies on the S3DIS dataset to further assess necessity and effectiveness of BN. As shown in Tab. 14, models with BN outperform those with Layer Normalization (L.N) and without any normalization, indicating that BN is particularly effective for our specific architecture.
14

---

Table 10: Semantic segmentation results on NuScenes val set.
t denotes using rotation and
translation testing-time time augmentations.
bicycle
podestriam
tallie cone
triler
driveablee
other Bat
sddwwalk
tommin
marmake
Mothod
mlooi
Range No133++ [36]
NolarNer [38]
74.7
35.1
71.3
58.8
76.
71.1
74.7
Sabaneet [37]
72.2
74.8
34.1
85.9
83.4
12.2
72.1
72.2
63.1
61.3
76.5
96.0
70.8
71.2
71.5
$6.7
$4.4
AMVNet (40)
76.1
79.8
32.4
82.2
86.4
62.5
81.9
75.3
72.3
83.5
65.]
97.4
67.0
78.
74.6
90.8
87.9
Cylinder3D
84.4
87.4
PVKD [39]
76.0
76.2
90.2
94.0
50.9
77.4
78.8
61.7
62.0
$1.1
96.6
71.4
76.4
76.3
90.3
36,9
RPVNei
77.6
78.2
43.4
92.7
93.2
49.0
85.7
80.5
66.0
66.9
84.0
96.9
73.5
75.9
76.0
90.6
88.9
3DPASS
96.8
87.2
SpheneFommer [46]*
79.5
78.7
46.7
95.2
93.7
54.0
83.9
81.1
68.0
74.2
86.2
97.2
74.3
76.3
75.8
91.4
89.7
LinNet(ours)
80.4
79.2
96.6
93.2
53.9
39.0
83.7
70.6
73.3
83.5
96.9
73.8
76.1
75.1
91.5
39.7
LinNet'(ours)
81.4
80.0
56.9
96.9
94.0
58.4
90.0
84.4
72.1
74.2
89.7
97.0
74.4
76.8
76.0
91.7
89.9
Table 11: Semantic segmentation results on S3DIS Area 5.
mikes
mlnU
oalling
beam
Window
door
bahee
bookcooe
board
Method
PounNeu
89.0
11.1
88.8
97.3
69.8
3.9
16.3
10.8
39.0
32.6
10.3
26,1
33.2
PoimNa++1SS]
PoinCNN]
63.9
57.3
92.3
98.2
79.4
0.0
22.8
62.1
74.4
80.6
31.7
66.7
62.1
56.7
DGCNN|L33
83.6
47.9
DeepGCN[S1]
52.5
ASSANa-L[1S]
66.8
Point [rans.99]
90.8
76.5
70.4
94.0
98.5
86.3
0.0
38.0
63.4
74.3
82.4
89.1
74.3
80.2
76.0
59.3
90.2
76.0
68.9
Soctoocc
PoindNeXt (22)
90.7
77.5
20.8
94.1
98.5
84.4
0.0
37.7
59.3
74.0
83.1
91.6
77.4
77.2
78.8
60.6
LinNet(ours)
91.9
79.0
73.7
94.8
98.1
86.2
0.0
45.5
61.6
82.8
85.1
92.3
85.5
80,0
80.4
66.4
Table 12: Sem. seg. on Sem. KITTI.
Table 13: S3DIS 6-fold cross-validation.
Methods
Val
Test
Methods
mloU (%)
mAce (%)
OA (%)
SPVNAS [42]
64.7
66.4
Cylinder3D [41]
64.3
67.8
PointNeXt 74.9
83.0
90.3
PVKD [39]
71.2
LinNet
78.6
86.3
91.9
2DPASS [44]
69.3
72.9
Wafflelron [47]
68.0
70.8
Table 14: Normalization layer.
SphereFormer [46]
67.8
74.8
RangeFormer [45]
67.6
73.3
None
BN
LN
Mink [4]
63.8
71.8%
72.9%
71.9%
LinNet (ours)
69.1
70.4
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize other setrings.
It is fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not diseussed in the paper.
The authors are encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low or images taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and
complete (and correct) proof?
Answer: [NA]
16

---

Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should deseribe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [NA]
Justification: Our code will available upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips) cc/
public/ /triiccciiiii des CodeSubmi as onnol oolicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips. cc/ public/ wwwc//uuiiesseee Code Submiss is oonooliyy))) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
18

---

It is OK to report 1- I-sigma enror bars, but one should state it. The authors should preferably report 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /neurips..... hc/ppuullce/ttttccccleeee...e
Answer:[Yes Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, legitimate to point out that an improvement in the quality of generative models could be used to generate
19

---

deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include a URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thoode. com/ da aaateset s has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
20

---

Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate details of the dltaset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "faJlUPOv9i", "reviewer": "Reviewer_J8xV", "summary": "This paper proposes a method for point cloud segmentation and classification. The main contribution is making the local aggregation dependent on the anchor point. The approach demonstrates improvements of one to two percent on S3DIS and NuScenes datasets compared to existing methods.", "questions": "1. Have you explored whether a more general model could learn the invariances you've built into LinNet without explicit architectural choices?\n2. Given the relatively small improvements in accuracy, what do you see as the main practical advantages of LinNet over existing methods?", "limitations": "The discussion of limitations is sufficient.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The architecture is based purely on an MLP (with some hashing operations) which is a significant benefit in terms of implementation and potentially in terms of computational efficiency.\n- The approach of making local aggregation anisotropic and the proposed DSA module has a good motivation.\n- The method shows improved performance on standard benchmarks (S3DIS and NuScenes), outperforming existing approaches by a small but consistent margin.", "weaknesses": "- The paper's writing seems overly complex with lots of unnecessary jargon making it difficult to follow the core ideas and contributions.\n- The improvements in performance, while consistent, are relatively small (1-2%), which raises questions about the practical significance of -the method.\n- The modifications proposed are individually well-motivated but seem somewhat ad hoc, lacking a strong theoretical foundation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and constructive comments. In the following, we address your concerns carefully.\n\n**W1: The paper's writing seems overly complex with lots of unnecessary jargon making it difficult to follow the core ideas and contributions.**\n\n**A:** We sincerely apologize for any confusion caused by the writing complexity of our manuscript and appreciate your feedback on the use of jargon. In response, we have revised the manuscript to simplify the language and ensured that key concepts are explained more thoroughly. For example, in the context of DSA module, we use terms like **anisotropy** and **isotropy**. To clarify, we employ the term **spatial-wise anisotropy** to refer to variations among the features of neighboring points within the same feature channel, and **channel-wise anisotropy** to describe differences across various feature channels.\n\nThe core idea of our paper is to **enhance the scalability of existing point-based methods through a more lightweight feature aggregation strategy and a point cloud search strategy with reduced linear complexity**. \n\n-----\n\n**W2: The performance improvements (1-2%) are relatively small, which raises questions about the practical significance.** \n\n**A:** Firstly, **in the domain of point cloud processing, even seemingly modest improvements of 1-2% are indeed substantial.** For instance, Point Transformer [1] (first released in Dec 2020)  achieved a result of 70.4%, while the result of Point Transformer V2 [2] (Oct 2022) is 71.6%. \n\nSecondly, we want to highlight that our approach **achieves dual improvements in both speed and accuracy**. Previous work, such as the Fast Point Transformer [3], sacrifices accuracy in the pursuit of efficiency. In contrast, we achieve a 2.9% mIoU improvement while much faster than PointNeXt [5] on S3DIS. More critically, by employing a linear sampling strategy and a DSA module, **we have largely addressed the long-standing scalability challenges associated with point-based networks.** This allows our network to be easily applied to large-scale point cloud scenes. We believe these aspects affirm the practical significance of our reported improvements, offering meaningful contributions to the field.\n\n-----\n\n**W3: The modifications proposed are individually well-motivated but seem somewhat ad hoc, lacking a strong theoretical foundation.**\n\n**A:** Thank you for your endorsement of the motivation for our work. \n\n**Our modifications are not ad hoc but are derived from a thorough analysis and adaptation of established concepts within both 2D and 3D vision technologies.** To improve the efficiency of point-based networks, we started by systematically reviewing local aggregation techniques used in computer vision. Inspired by the success of separable convolutions in 2D imaging, we explored their applicability to 3D point clouds.\n\n**We conducted a detailed theoretical analysis of the set abstraction (SA) module used in 3D vision, examining it from the perspective of anisotropy in Sec. 3.2.** It shows that principles of separable convolution, effective in 2D vision, could be adapted for 3D point clouds, thus motivating our modifications. Initially, we attempted to directly separate spatial anisotropy and channel anisotropy from the input features. However, the initial experimental outcomes were not as anticipated, which prompted further in-depth analysis.\n\nThis led to the development of the DSA module, where we refined our approach based on our theoretical insights and experimental findings. **Finally, we validated our modifications with explanatory analysis and empirical evidence, clearly demonstrating the advantages of the DSA module over the vanilla SA.** \n\nFollowing your feedback, we will further **strengthen the theory by incorporating references such as ASSANet [4], which provides an in-depth theoretical analysis of anisotropy in point cloud processing.** \n\n-----\n\n**Q1: Have you explored whether a more general model could learn the invariances you've built into LinNet without explicit architectural choices?** \n\n**A:** Thank you for your insightful question about exploring a more generalized model without explicit architectural choices. \n\nIn the manuscript, for the segmentation used in S3DIS, the number of layers of our model was set to [4, 7, 4, 4] for a fair comparison with the baseline PointNeXt . For the other dataset, NuScenes, which is much larger, we adjusted the number of layers in the encoder part to [4, 4, 7, 4] for efficiency. In response to your question, we used a uniform [4, 7, 4, 4] configuration across tasks to determine if a more generalized structure would impact performance. The results show that the mIoU of NuScenes decreases by only 0.2% (80.2% vs. 80.4%), which indicates that the model performs robustly in both configurations.\n\n-----\n\n**Q2: Apart from accuracy, the practical advantages of LinNet over existing methods.**\n\n**A:** Thank you for your question.  In addition to improved accuracy,  **the substantial increase in processing speed represents a significant practical benefit of our approach, particularly for large-scale applications**. As demonstrated in Fig. 7(a), LinNet achieves a remarkable improvement in response speed; when handling point clouds of up to 200k points, it performs **13 times faster** than the current state-of-the-art method, PointNeXt++, and **twice as fast as PTv2**. \n\n-----\n\n[1] Zhao et al. Point Transformer. ICCV 2019.\n\n[2] Wu et al. Point Transformer V2: Grouped Vector Attention and Partition-based Pooling. NeurIPS 2022.\n\n[3] Park et al. Fast Point Transformer. CVPR 2022.\n\n[4] Qi et al. ASSANet: An Anisotropic Separable Set Abstraction for Efficient Point Cloud Representation Learning. NeurIPS 2021.\n\n[5] Qian et al. PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies. NeurIPS 2022. \n\n------\n\nWe hope our response adequately addresses your concerns. If you still have any questions, we are looking forward to hearing them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oB6ejqmdRA", "reviewer": "Reviewer_7sY3", "summary": "The paper proposes a novel lightweight backbone network model for input point cloud data suitable for global and local per-point feature extraction. It relies on two main ideas: (1) the separate processing of point coordinates and features (and a further combination of these two streams of features before the neighborhood pooling operation), (2) the use of the space-filling curves to define local neighborhoods that allow hash queries and linear complexity sampling.\n\nThe approach is evaluated in the point cloud classification and segmentation tasks on ModelNet40, ScanObjectNN and S3DIS, NuScenes datasets respectively, demonstrating performance competitive to state of the art. Additional experiments include ablations exploring the efficacy of every proposed component and comparison to other methods in terms of model efficiency.", "questions": "The proposed method relies on the space-filling curves introduced in PointTransformer v.3, which is mentioned multiple times in the paper. At the same time, it is not considered for comparison. While PTv3 is concurrent, for completeness, it would be nice to include the results from it, especially since this work already acknowledges the existence of PTv3. So my question is how this method compares to PTv3 in terms of performance and efficiency?\n\nFigure 5 shows a nice uniformly distributed (over the grid) point cloud but in practice, any grid-based discretization suffers to some extent from the discretization artifacts. The proposed method can in principle have cells with single points. Do these cells exist in practice and do they cause any problems?\n\nFigure 1: a) and b) have the same area but the total time of b) is significantly lower which is misleading.\n\nFigure 3: The choice of the number of inputs, intermediate features, and outputs is either arbitrary or not clear which is confusing. Showing the operations for a single input point would be clearer.\n\nFigure 5: Showing an empty table in c) is not informative. Showing a part of the actual hash table for this example in the figure might work better.", "limitations": "The authors properly address the limitations in the submitted draft.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The proposed feature aggregation method is likely novel and improves the results according to the ablation studies.\n* The proposed method is on par with the state-of-the-art non-transformer-based approaches but is better in terms of scalability to larger point clouds.\n* The extensive evaluation shows the importance of every component in the ablations.", "weaknesses": "* All the improvements (except for the NuScenes dataset) are not particularly distinctive.\n* While present the efficiency is not exploited in any presented applications (for classification point clouds are small, so other methods work fast as well, for segmentation there are no efficiency comparisons).\n* The text is written well but some figures can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and constructive comments. In the following, we address your concerns carefully.\n\n**W1: The improvements (except for the NuScenes dataset) are not particularly distinctive.**\n\n**A:**   Thank you for your comments.\n\nFirstly, for small-scale classification tasks, we employed the same training protocols and experimental conditions as the SOTA benchmark, PointNeXt [1]. Nonetheless, we still achieve an overall accuracy (OA) improvement of 0.4% on the ScanObjectNN dataset, while other PointNeXt style architectures (e.g., PointVector [2], PointMetaBase [3]) using the same experimental setup only achieve an OA improvement of 0.1%.  \n\nSecondly, for large-scale segmentation tasks, except for the NuScenes dataset, we also achieved a 2.5% improvement in mIoU without Test Time Augmentation (TTA) and a **2.9% improvemen**t with TTA. The improvements are particularly impressive and distinctive. **The speedup is even more impressive than the accuracy improvement**, and **the magnitude of the speedup increases with the size of the point cloud**. Moreover, **the enhancement in processing speed**—a critical factor for large-scale applications—**is even more striking**. Specifically, at a 20k point cloud size, our inference speed is **1.7 times faster** than PointNeXt (47ms vs. 87ms), scaling up to a speedup ratio of **13 times** (232 ms vs. 3147ms) when handling 200k points.\n\n**This dual achievement of improved speed and accuracy, especially at larger scales, is noteworthy**. \n\n-----\n\n**W2: The efficiency is not exploited in any presented applications.**\n\n**A:** Thank you for pointing out this issue. Visual comparisons of model efficiency at four different point cloud sizes (20k, 50k, 100k, 200k) are shown in Fig. 7(a). Following your suggestion, we further add a table to more clearly and intuitively demonstrate efficiency. Corresponding to the number of points pre-sampled by PointNeXt on the S3DIS dataset, we replaced 20k with 24k. Thanks to the simplicity and efficiency of our DSA module, the proposed LinNet exhibits only half the latency of Point Transformer v2 [4]. Remarkably, at the 200k level, our LinNet model performs 13 times faster than PointNeXt. \n\nModel latency on different scales (ms):\n\n| Methods   | 24 k | 50 k | 100 k | 200 k |\n| --------- | ---- | ---- | ----- | ----- |\n| PointNeXt | 87   | 266  | 878   | 3147  |\n| PTv2      | 55   | 163  | 228   | 493   |\n| LinNet    | 47   | 62   | 123   | 232   |\n\n-----\n\n**W3: Some figures can be improved.**\n\n**A:** Thank you for the constructive suggestions and valuable feedback. \n\nThe revised figure can be found in the **PDF** of the global rebuttal. Fig. 1, Fig. 2, and Fig. 3 of the global rebuttal correspond to Fig. 1, Fig. 3, and Fig. 5 of the manuscript.\n\n- **Figure 1:** We have redrawn Fig. 1 (a) and added the total inference latency to it.\n- **Figure 3:** We are sorry for this confusion. Following your suggestion, we take a single point as input and set the number of neighbors to 3. We represent the features using rectangles of uniform size to maintain consistency across the data representation.\n- **Figure 5:** Thank you for your insightful feedback on Fig. 5, and your concerns about artifacts from grid-based discretization. Given the sparse nature of the data distribution and the characteristics of the discretization process, our method accommodates scenarios where a grid may contain only a single point. This point is then directly used as a new sampling point under our strategy, ensuring that the distribution of the newly sampled point cloud closely mirrors that of the original. Following your suggestions, we have made specific optimizations in the revised version to ensure that the number of points in each grid varies, better reflecting the variability found in real-world data distributions. Additionally, in Fig.5 (c), we have incorporated actual data into the hash table.\n\n-----\n\n**Q1: Without result of Point Transformer v3**.\n\n**A:** Thank you for pointing out this issue. A detailed comparison with PointTransformerv3 (PTv3) is available in the comments below. Our model consistently achieves better results than PTv3 on the S3DIS and NuScenes validation sets, also demonstrating competitive performance on NuScenes testing sets. \n\nRegarding latency, PTv3 utilizes spatial curves to divide the point cloud into patches. This allows the model to compute attention on the patches rather than on individual points, avoiding the need for point-wise local attention. In contrast, our method leverages point-wise local features, resulting in higher latency compared to PTv3. However, we would like to highlight that our LinNet is a pure MLP network and does not require any additional operational assistance. Instead, PTv3 relies on sparse convolution for positional encoding and the sparse convolution kernel incurs more model parameters.\n\nModel performance:\n\n|        | S3DIS Area 5 | S3DIS 6-fold | NuScenes (val) | NuScenes (test) |\n| ------ | ------------ | ------------ | -------------- | --------------- |\n| PTv3   | 73.4         | 77.4         | 80.4           | 82.7            |\n| LinNet | 73.7         | 78.6         | 81.4           | 82.3            |\n\nModel size and execution time, latency are measured with 24k points:\n\n|        | Model Size (M) | Forward Latency (ms) |\n| ------ | -------------- | -------- |\n| PTv3   | 46.2           | 26                   |\n| LinNet | 14.7           | 47                   |\n\n-----\n\n[1] Qian et al. PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies. NeurIPS 2022.\n\n[2] Deng et al. PointVector: A Vector Representation In Point Cloud Analysis. CVPR 2023.\n\n[3] Lin et al. Meta Architecture for Point Cloud Analysis. CVPR 2023.\n\n[4] Wu et al. Point Transformer V2: Grouped Vector Attention and Partition-based Pooling. NeurIPS 2022. \n\n------\n\nWe hope our response adequately addresses your concerns. If you still have any questions, we are looking forward to hearing them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V28l1766Ei", "reviewer": "Reviewer_aBR6", "summary": "In this work, the authors propose an efficient learning framework for point cloud representation learning. For the computational intensive local aggregation operation, this work proposes Disassembled set abstraction (DSA) to aggregate local features in terms of the spatial distributions of points in a simple and efficient manner. This work also proposes a Linearization sampling strategy and hash query operation to accelerate the sampling and neighbor searching processes. Experiments on classification and semantic segmentation demonstrate the effectiveness of the proposed method.", "questions": "1. Why do we introduce batch normalization to the aggregated features in Eq.4? As the addition of features and neighborhood features have already introduce the spatial characteristics, I am not sure if this BN is necessary here, or other simple components may also work;\n\n2. In the Hash query part, will the points in a same local grid share the same neighbors? I do not quite get the calculation of complexity for each point.\n\n3. Is the hash grid is pre-constructed before training? Or created repeatedly during training?\n\n3. From the results in Table 1 and Table 3, the proposed method even outperforms transformer-based methods in a more efficiency way. Could the author analyze the reasons behind this? As the improvements of this framework seem to be efficient simplification of existing PointNet++ framework, I am curious why it can improve the performances so obviously.\n\nPlease check the grammar also, e.g., In Line 101, the $x_i$ might actually be $p_i$;", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed local aggregation operation seems to be simple, efficient, and effective;\n\n2. The point searching strategy including the Linearization sampling strategy and hash query can indeed improve the effciency while keeping the overall performances;", "weaknesses": "My major concern about this work is the higher performances than transformer-based methods. It may be a little hard to understand why the proposed DSA and hash-based searching operations can improve the performances so greatly. Because these operations are more like the approximation of corresponding operations in PointNet++.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and constructive comments. In the following, we address your concerns carefully.\n\n**W1 & Q4: Why the proposed DSA and hash-based searching operations can improve the performances so greatly? (Higher than transformer-based methods)**\n\n**A:** Thank you for your insightful concerns regarding the superior performance. It is important to clarify that though our method is built upon the PointNet++ style framework, LinNet is not a superficial approximation but a substantial exploration and design to specifically address scalability and efficiency. Our model, LinNet, addresses a longstanding issue with the PointNet++ style framework: the lack of scalability. \n\nWe try to explain the significant performance improvements from two aspects.\n\n**(a) Comparison with PointNet++ style framework**\n\n- **Accuracy**: Fig. 4 of the manuscript illustrates that after several training epochs, both the DSA and vanilla SA stabilize in terms of loss; however, DSA consistently maintains a lower loss compared to vanilla SA. This considerable difference demonstrates that vanilla SA may not sufficiently adapt to the datasets, whereas DSA exhibits a more robust capability to fit the data, thereby achieving higher accuracy.\n\n- **Speed**: In DSA, point-wise convolutions are applied directly to anchor features rather than to an expansive neighborhood, thus **DSA requires significantly fewer FLOPs compared to vanilla SA**. Additionally, the adoption of a linear complexity search strategy shifts away from the costly traditional algorithms like FPS and KNN—which have complexities of $\\mathcal{O}(N^2)$ and $\\mathcal{O}(kN^2)$ respectively—to more efficient, GPU-friendly linear complexity algorithms. This enhances computational efficiency remarkably.\n\n**(b) Comparison with Transformer-based method.**\n\nThe Point Transformer v2 (PTv2) we are comparing is also essentially a point-based approach, which implements downsampling through grid pooling similar to the resize operation in image and uses local attention for feature aggregation.\n\n- **Accuracy**: We observe that the limited receptive field during the downsampling stages of PTv2 and similar methods could restrict their accuracy. PTv2 samples only within a single grid (typically about 6 points), whereas our method captures the nearest $k$ points from the target grid and its 26 surrounding grids, providing a broader receptive field.\n- **Speed**: Although PTv2's grid sampling avoids the need for farthest point sampling, its reliance on KNN for neighborhood queries and the complexity of its local attention calculations significantly hamper its processing speed. In contrast, our DSA module simplifies these processes, leading to faster data processing.\n\n-----\n\n**Q1: Why adapted BN in Eq. (4). Is this BN necessary, or other simple components also work?**\n\n**A:** Thank you for your interest in batch normalization. As you mentioned, the addition has already introduced spatial characteristics. Note that BN is placed **after the max pooling layer to normalize the aggregated feature**, facilitating subsequent convergence. In the ScanObjectNN dataset, BN is necessary and has led to a 0.4% OA improvement. As you suggested, we conducted ablation studies on the S3DIS dataset to further assess the necessity and effectiveness of BN. Here are the results averaged over three experiments:\n\n| None   | BN     | LN     |\n| ------ | ------ | ------ |\n| 71.8 % | 72.9 % | 71.9 % |\n\nAs shown, models with BN outperform those with Layer Normalization (LN) and without any normalization, indicating that BN is particularly effective for our specific architecture. \n\n-----\n\n**Q2: In the Hash query part, will the points in the same local grid share the same neighbors?** \n\n**A:** Thank you for your interest in the hash query part.  Our hash query confines the search range of each point to its own grid and the adjacent 26 neighborhood grids (i.e., $3\\*3\\*3-1$), selecting the closest $k$ points as neighbors from these grids. **Although points in the same local grid have an identical search range, the specific neighbors selected for each point (i.e., the $k$ nearest points) can differ.** This variation arises because **neighbor points are chosen based on the actual spatial distances between points,** not merely by their presence in the same grid.\n\nRegarding the computation complexity discussed in the manuscript, we apologize for any confusion caused by omitting the complexity of heap sorting. For a point cloud comprising $N$ points distributed across $m$ non-empty grids, constructing the hash table entails a complexity of $\\mathcal{O}(m)$. Assuming each point’s 27-grid neighborhood contains $p$ points on average, identifying the closest $k$ points involves maintaining a heap with a complexity of $\\mathcal{O}(p \\log k)$ and a final sorting step costing $\\mathcal{O}(k \\log k)$. Thus, the total computational complexity is $\\mathcal{O}(m + N(p \\log k + k \\log k))$. We will clarify and elaborate on these calculations in the revised version of our manuscript to prevent confusion.\n\n-----\n\n**Q3: Is the hash grid is pre-constructed before training? Or created repeatedly during training?**\n\n**A:** Thank you for your question regarding the hash table. In our approach, the hash table is not pre-constructed but is instead dynamically built-in real-time during the training process. This allows for hash query and linearization sampling to be integrated seamlessly into each training iteration. As demonstrated in Fig. 1, both operations are efficiently parallelized on the GPU and collectively account for less than 10% of the model's forward time. \n\n-----\n\n**Q4: The typo in line 101.**\n\n**A:** Thank you for pointing out this typo. You are correct that the symbol \"$x_i$\" should indeed be \"$p_i$\". We have made this correction in the revised manuscript.\n\n------\n\nWe hope our response adequately addresses your concerns. If you still have any questions, we are looking forward to hearing them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sHfkLbdDRU", "reviewer": "Reviewer_zyd3", "summary": "The submission #308 entitled \"LinNet: Linear Network for Efficient Point Cloud Representation Learning\" introduces a linear network designed for efficient point cloud representation learning. To achieve this task, the authors propose a novel disassembled set abstraction (DSA) module and a linear sampling strategy, which together enhance computational efficiency and scalability. The method maps 3D point clouds onto 1D space-filling curves, allowing for parallelization of downsampling and neighborhood queries on GPUs with linear complexity. This approach achieves state-of-the-art performance across various benchmarks.", "questions": "- Would it be possible to conduct a small experiment to measure the memory footprint of the approach?\n- It would be interesting to have a few cross-validation tests to know more about the generalization of the technique.", "limitations": "The limitations of the approach are well covered at the end of the manuscript.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The linear sampling strategy is very elegant.\n- The large-scale comparison across multiple datasets and many approaches is highly appreciable.\n- The authors offer to open their code upon acceptance.\n- The ablation studies are well-conducted, clearly showing the contribution of each proposed module.\n- The reasoning behind DSA is well introduced and motivated.\n- The editorial quality of the paper is excellent. It is easy to read, and the illustrations are pleasant and informative.\n- The assessments are numerous and conclusive, showing that the proposed approach is innovative and achieves state-of-the-art results on multiple metrics.", "weaknesses": "- One negative point is the significant memory footprint of the approach, as mentioned in the limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and constructive comments. In the following, we address your concerns carefully.\n\n**W1: One negative point is the significant memory footprint of the approach, as mentioned in the limitations section.**\n\n**A:** Thank you for addressing the concern regarding the memory footprint of our approach, which we have acknowledged in the \"Limitations\" section of our manuscript.\n\nWhile our local aggregation methods have indeed improved the scalability of point-based approaches in large-scale scenarios by reducing memory usage, the overall memory footprint remains a substantial challenge. We are actively working on strategies to further minimize memory consumption, which we plan to implement in future iterations of our model.\n\nFirstly, we plan to integrate hash query strategies with our local aggregation methods. Employing hash tables to build rulebooks, similar to those used in sparse convolutions, will allow us to manage memory more efficiently.\n\nSecondly, we aim to make better use of shared memory on GPUs to enhance processing efficiency. Shared memory is much faster than global memory and using it effectively can greatly reduce the need for frequent global memory accesses, which are more costly in terms of time and energy consumption. \n\n-----\n\n**Q1: Experiment to measure the memory footprint of the approach**.\n\n**A:** Thank you for your constructive suggestion regarding the measurement of our model's memory footprint. To address this, we conducted experiments to evaluate memory usage during both training and inference phases on the NuScenes dataset, utilizing an RTX 4090 graphics card with all tests conducted at a batch size of 1.\n\nWe included comparisons with the baseline model PointNeXt [1] and the sparse convolution method MinkUNet [2]. Our findings reveal that PointNeXt suffers from out-of-memory issues when handling large-scale scenes, highlighting scalability challenges. In contrast, our DSA module significantly reduces memory consumption by avoiding high-dimensional feature transformations on neighboring point clouds.\n\nGiven that MinkUNet starts with 32 input channels, we conducted similar tests with our LinNet-Small model, which also has 32 initial feature channels, for a direct comparison:\n\n|              | Training Mem. (NuScenes) | Inference Mem. (NuScenes) |\n| ------------ | ------------------------ | ------------------------- |\n| MinkUNet     | 2.6 GB                   | 1.4 GB                    |\n| PointNeXt    | Out of Memory            | Out of Memory             |\n| LinNet-Small | 5.2 GB                   | 4.9 GB                    |\n| LinNet       | 16 GB                    | 13 GB                     |\n\nAlthough LinNet-Small consumes more memory than MinkUNet, it is crucial to note that LinNet-Small, with only 1.7M parameters, achieves a validation accuracy of 77.6%, surpassing the 38M parameter sparse convolution method MinkUNet, which achieves 73.3%. This demonstrates that our model, despite its higher memory footprint, provides superior accuracy, offering a significant advantage in scenarios where performance is critical.\n\n-----\n\n**Q2: Conduct cross-validation.**\n\n**A:** Thank you for suggesting the inclusion of cross-validation tests to evaluate the generalization capabilities of our technique. We have performed 6-fold cross-validation on the S3DIS dataset to ensure a robust assessment of our model's performance across different subsets of data. Here are the results:\n\n| Methods   | mIoU (%) | mAcc (%) | OA (%) |\n| --------- | -------- | -------- | ------ |\n| PointNeXt | 74.9     | 83.0     | 90.3   |\n| LinNet    | 78.6     | 86.3     | 91.9   |\n\nThese results demonstrate that LinNet consistently outperforms the baseline model, PointNeXt, across multiple metrics, which suggests superior generalization abilities.\n\n-----\n\n[1] Qian et al. PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies. NeurIPS 2022. \n\n[2] Choy et al. Minkowski convolutional neural networks. CVPR 2019.\n\n------\n\nWe hope our response adequately addresses your concerns. If you still have any questions, we are looking forward to hearing them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their positive comments about the novelty (7sY3, zyd3), motivation (J8xV, zyd3), writing quality (7sY3, zyd3), and experiments (J8xV, 7sY3, aBR6, zyd3) of this work.

  As suggested by the four reviewers, we conduct sufficient additional experiments on our LinNet and demonstrate more inspirable abilities.

1. For the comments raised by reviewer **J8xV**, we elaborated on the core ideas and contributions of our model. We conducted experiments on the NuScenes dataset to explore a general model without explicit architectural choices, confirming that the model performs robustly in both configurations.
2. For the comments raised by reviewer **7sY3**, we highlighted the significance of our improvements. We added latency measurements to better demonstrate the model's efficiency and modified the figures based on constructive comments from the reviewer. 
3. For the comments raised by reviewer **aBR6**, we conducted experiments to verify the necessity of Batch Normalization (BN) in the DSA module and addressed the complexity of the hash query.
4. For the comments raised by reviewer **zyd3**, we implemented cross-validation tests to assess the generalization capabilities of our model and measured its memory footprint.

In conclusion, to enhance the scalability of point-based methods, we developed Linear Net (LinNet). LinNet achieves more efficient local aggregation by leveraging spatial anisotropy and channel anisotropy separately. Additionally, by mapping 3D point clouds onto 1D space-filling curves, we performed downsampling and neighborhood queries with linearly reduced complexity.  LinNet **achieves dual improvements in both speed and accuracy, largely addressing the long-standing scalability challenges associated with point-based networks.** Extensive experimental results demonstrated the superiority of the methodology design, which also shows that **even without the support of any additional techniques (e.g., sparse convolution, attention, graph convolution), purely point-based methods can achieve good results by relying only on a simple MLP.** We anticipate that these insights will encourage the community to rethink methods for efficiently point clouds learning.

We include three figures in **the PDF** and cite them in rebuttal.

Meta-review: This paper initially received borderline to positive ratings (BR, BA, WA, SA). The authors appreciated the novelty of the proposed point cloud feature learning approach but had some specific constraints regarding aspects of the method. Most of these concerns appeared resolved after the rebuttal, with 3 reviewers leaning towards acceptance (6,6,8). The only reviewer left on a BR was unresponsive during the rebuttal and discussion stage. Taken all this into account, the final recommendation is to accept this paper.
Decision: Accept (poster)