On-the-fly Point Feature Representation for Point Clouds Analysis

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 12:56:52

Abstract: Point cloud analysis is challenging due to its unique characteristics of unorderness, sparsity and irregularity. Prior works attempt to capture local relationships by convolution operations or attention mechanisms, exploiting geometric information from coordinates implicitly. These methods, however, are insufficient to describe the explicit local geometry, e.g., curvature and orientation. In this paper, we propose On-the-fly Point Feature Representation (OPFR), which captures abundant geometric information explicitly through Curve Feature Generator module. This is inspired by Point Feature Histogram (PFH) from computer vision community. However, the utilization of vanilla PFH encounters great difficulties when applied to large datasets and dense point clouds, as it demands considerable time for feature generation. In contrast, we introduce the Local Reference Constructor module, which approximates the local coordinate systems based on triangle sets. Owing to this, our OPFR only requires extra 1.56ms for inference (65$\times$ faster than vanilla PFH) and 0.012M more parameters, and it can serve as a versatile plug-and-play module for various backbones, particularly MLP-based and Transformer-based backbones examined in this study. Additionally, we introduce the novel Hierarchical Sampling module aimed at enhancing the quality of triangle sets, thereby ensuring robustness of the obtained geometric features. Our proposed method improves overall accuracy (OA) on ModelNet40 from 90.7\% to 94.5\% (+3.8\%) for classification, and OA on S3DIS Area-5 from 86.4\% to 90.0\% (+3.6\%) for semantic segmentation, respectively, building upon PointNet++ backbone. When integrated with Point Transformer backbone, we achieve state-of-the-art results on both tasks: 94.8\% OA on ModelNet40 and 91.7\% OA on S3DIS Area-5.

Main paper: 
 related work 2.1 deep learning on point clouds: 
 Many prior works [4, 22, 27, 30, 31, 61, 64] learn from raw point clouds via careful network designs. PointNet [30] pioneers this trail by handling coordinates of each point with shared-MLP and consolidating the final representation with a global pooling operation. However, it is susceptible to a deficiency in preserving local structures due to the use of global pooling operation. PointNet++ [31] is an extension of the original PointNet architecture, which applies PointNet to multiple subsets of point clouds. It further leverages a hierarchical feature learning paradigm to capture the local structures. However, PointNet++ still processes points individually in Figure 1 : The workflow of Point Feature Histogram, which can be decomposed into two steps. Firstly, for one interested point (pink), each neighboring point (blue) pair is described via angles. Secondly, for each angle, its distribution over all point pairs is summarized using histograms. Here, (e 1 , e 2 , e 3 ) is the local coordinate system and n is the normal vector.each local region, neglecting explicit consideration of relationships between centroids and their neighbors.As PointNet++ establishes the hierarchical point clouds analysis framework, the focus of many works has shifted towards the development of local feature extractors, including convolutionbased [14, 17, 20, 25, 54] , attention-based [8, 60, 65] , and graphbased [46, 50, 58, 59, 63] approaches. PointCNN [20] learns a 𝜒transformation from input point clouds, which attempts to reorganize inputs into canonical order. Subsequently, it utilizes vanilla convolution operations to extract local features. Point Transformer [65] replaces conventional shared-MLP modules with Transformer [49] blocks, serving as feature extractors within localized patch processing. DGCNN [50] utilizes dynamic graph structures to enhance feature learning and capture relationships between points. However, these works rely heavily on the learnability of feature extractors, potentially missing inherent local shape information. More recently, RepSurf [33] leverages triangle sets with connected triangles formed by 𝑘 nearest neighbors (𝑘-NNs), to learn location and orientation-aware representations from geometric features explicitly. Although location and orientation features are explicitly injected into network architecture in RepSurf, the usage of curvature information still remains underexplored. Moreover, RepSurf relies on naive 𝑘-NNs to produce triangle sets and obtain geometric features, which are vulnerable to noisy points [33] .Many works in 3D computer vision attempt to build sophisticated feature descriptors [35, 37, 39] , which help to understand point clouds through hand-crafted features. Point Feature Histogram (PFH) [37] , one of the feature descriptors, is commonly used in computer vision tasks like object recognition, registration and model retrieval [11, 19, 36] . It develops point cloud representations by summarizing the distribution of certain geometric attributes within a local neighborhood around each point.We depict the workflow of PFH derivation for one of our interested point in Fig. 1 . The whole process can be decomposed into two steps. Firstly, for each point pair within 𝑘 nearest neighbors (𝑘-NNs) of interested point, curvature features are characterized using angles calculated from normal vectors and relative positions. Secondly, for each angle, we achieve its histogram within 𝑘-NNs. The histograms of different angles are concatenated together, yielding the final PFH representation. Unfortunately, many point cloud datasets [7, 48] collected in real-world scenarios lack normal vectors, and estimating normal vectors [13, 52] for sparse point clouds often leads to significant deviations from ground truth. Nonetheless, PFH calculation involves establishing local coordinate systems and constructing curvature features, which is computationally expensive. As a result, the practical application of vanilla PFH is limited. 
methodology: 
 The pipeline of On-the-fly Point Feature Representation (OPFR) is depicted in Fig. 2 , and we illustrate the OPFR generating process for the right corner of table (highlighted in pink), which is one of our interested points. Firstly, we propose Hierarchical Sampling module, which takes each point in the point cloud as input and outputs several clusters (highlighted in blue) and corresponding centroids (highlighted in orange). This hierarchical sampling strategy improves the quality of triangle set for each point, thereby facilitating the development of subsequent geometric features. Then, for each point pair within the clusters, we design Curve Feature Generator module to generate geometric features, including location, orientation, and curvature. The inclusion of explicit curvature information allows us to more effectively capture the local geometry surrounding these point pairs. To enhance the efficiency and enable on-the-fly processing, we present Local Reference Constructor module. It approximates a local coordinate system (highlighted in red) for each point pair using adjacent points from triangle sets. Lastly, these obtained geometric features are further fed into shared-MLP followed by a pooling operation, constituting our final OPFR representation r 𝑖 . The resultant OPFR representation r 𝑖 along with coordinate x 𝑖 can be directed into various point clouds analysis backbones, e.g., PointNet++ [31] and Point Transformer [65] , for end-to-end training.As mentioned earlier, the connected triangle sets produced by naive 𝑘 nearest neighbors (𝑘-NNs) are susceptible to noisy points [33] , leading to significant distortion. Given that our Local Reference Constructor module inherently relies on triangle sets to approximate local reference frames, we propose the novel Hierarchical Sampling module to alleviate this distortion issue. For each individual point, our Hierarchical Sampling module generates several clusters, forming a triangle set. Specifically, we firstly conduct 𝑘-NN algorithm to select the 𝑘 1 nearby points (highlighted in purple). Secondly, we utilize farthest point sampling [5] algorithm to identify 𝑘 2 surface centroids (highlighted in orange) from these 𝑘 1 nearby points. Lastly, for each centroid, we retrieve its 𝑘 3 nearest neighbors (highlighted in blue). The selected 𝑘 3 neighbors are used to further develop geometric features. The detailed implementation is presented in Algorithm 1.As illustrated in Fig. 2 , Hierarchical Sampling module is designed to decouple the right corner of the table into 𝑘 2 distinct clusters (e.g., table top and table leg). These 𝑘 2 clusters exhibit simpler geometric structures, allowing resultant triangle sets to better approximate the original local surface. Therefore, compared to the naive 𝑘-NN approach, our hierarchical sampling scheme effectively releases the distortion issue of triangle sets, and ensures the robustness against original noisy points. As a result, it greatly enhances the development of subsequent geometric features. We provide additional visualization examples comparing triangle sets generated by Hierarchical Sampling and those produced by 𝑘-NN sampling in the supplementary material. A local reference frame is a local system of Cartesian coordinates at each point [26] , which provides a reference for understanding local structures. Denote a point set asX = {x 1 , x 2 , • • • , x 𝑁 } ⫅ R 𝑁 ×3 , normal vector set as N = {n 1 , n 2 , • • • , n 𝑁 } ⫅ R 𝑁 ×3 . Assumex 𝑖 as our interest point, and the objective is to extract geometric features for point x 𝑖 . Then, the local reference frame {u, v, w} [37] for {(x 𝑖 , x 𝑗 ), 𝑖 ≠ 𝑗 } is defined as:EQUATION)Although Equ. 1 achieves the construction of local reference frames, it comes with two major problems. Firstly, it relies on normal vectors, which are often unavailable in many benchmarks [7, 48] and real-life scenarios. Despite normal estimation [13] is feasible, its computational cost escalates significantly with dense point clouds, and its accuracy diminishes considerably with sparse point clouds.Secondly, it involves multiple cross-product operations sequentially, which cannot be effectively parallelized in terms of tensor operations. This leads to the inevitable computational overheads.To circumvent normal estimation and overcome the computational bottlenecks, we design approximated local reference frames through Local Reference Constructor (LRCon) module. Within each cluster generated by Hierarchical Sampling module, we establish point pairs between centroid and neighboring points. For each point pair, LRCon module leverages two adjacent neighbors along with their cross-product to serve as the approximate local reference frames. Denote number of neighbors as 𝐾, neighbors of centroidx 𝑖 as X 𝑖 = {x 𝑖1 , x 𝑖2 , • • • , x 𝑖𝐾 } ⊆ R 𝐾 ×3 .Based on this setting, we can construct the approximated local reference frame { û, v, ŵ} for point pair {(x 𝑖 , x 𝑖 𝑗 ), 𝑗 = 1, 2, • • • , 𝐾 }, which is defined as:EQUATIONwhere x + 𝑖 𝑗 , x - 𝑖 𝑗 are the most adjacent points for x 𝑖 𝑗 in neighbor set X 𝑖 clockwise and counterclockwise. To maintain the consistency of local frame orientation, we apply clockwise cross-product [33] to compute ŵ. When setting up the approximated local reference frames, the LRCon module basically finds the adjacent neighbors from the corresponding triangles in the triangle sets, i.e., Δ 1 = {x 𝑖 , x 𝑖 𝑗 , x + 𝑖 𝑗 } and Δ 2 = {x 𝑖 , x 𝑖 𝑗 , x - 𝑖 𝑗 }. As mentioned earlier, the Hierarchical Sampling module can boost the quality of triangle sets, ensuring the robustness against noisy points. This implicitly guarantees the reliability of the approximated local reference frames.This approximation scheme allows us to establish local reference frames that are independent with normal vectors of point clouds. By re-ordering these neighbors in X 𝑖 based on their projected angles in 𝑥𝑦-plane, we can efficiently derive approximated reference frames through tensor operations. Notably, with the integration of the LRCon module, our OPFR only requires an additional 1.56ms for inference, making it 65× faster than vanilla PFH. Furthermore, since LRCon module eliminates the need of normal estimation, it is compatible with point clouds of varying densities.We propose to approximate the local curve at point (𝑡, 𝑓 (𝑡)) by excluding high-order derivatives using Taylor Series [43] :EQUATIONIntuitively, the derivatives 𝑓 ′ (𝑡) and 𝑓 ′′ (𝑡) can reflect how the local curve is oriented and skewed near point (𝑡, 𝑓 (𝑡)) respectively. From this Taylor approximation, it can be observed that, 1-order derivative information is inadequate for accurately characterizing local curves. To illustrate this point, consider the 2D example shown in Fig. 3 . In the left figure, the curve may appear relatively flat compared to the right figure. However, both curves share the same normal vector (1-order derivative) at point 𝑥 = 1, implying that these two points possess identical orientation. Therefore, if we neglect 2-order derivative, it would lead to confusion between these two distinct curves. In fact, these curves exhibit significant differences in how surfaces curve at point 𝑥 = 1.To exploit 2-order curvature information, we propose the Curve Feature Generator (CFGen) module. This module processes input point pairs along with their approximated local reference frames, generating geometric features that encompass location, orientation, and curvature. Denote the approximate local reference frames for (x 𝑖 , x 𝑖 𝑗 ) as { û𝑖 𝑗 , v𝑖 𝑗 , ŵ𝑖 𝑗 }. The location and orientation can be naturally [33] characterized by relative position x ′ 𝑖 𝑗 = x 𝑖 𝑗x 𝑖 and frame cross-product n 𝑖 𝑗 = û𝑖 𝑗 × v𝑖 𝑗 , respectively. Furthermore, we propose the curvature proxy p 𝑖 𝑗 for point clouds, which is an approximation of curvature definition [40] from differential geometry. We provide the theoretical analysis for this part in the supplementary material. The curvature proxy p 𝑖 𝑗 is defined as: where ⊙ is the entry-wise dot product. Note that, curvature proxy p 𝑖 𝑗 approximates the limit definition of curvature from differential geometry, making it inherently curvature-aware. Intuitively, p 𝑖 𝑗 effectively captures how the surface is curved in three reference frames { û𝑖 𝑗 , v𝑖 𝑗 , ŵ𝑖 𝑗 } in terms of normalized angles.EQUATIONPoint Feature Histogram (PFH) [37] utilizes histogram operations to aggregate regional geometric features and generate final representation for each point. We argue that, these predefined transformation functions are task-agnostic, which making the final representations not fitting well for specific tasks. To this end, motivated by PointNet++ [31] , we employ shared-MLP to learn the final representations from point clouds. Therefore, the proposed OPFR representation r 𝑖 for point x 𝑖 is defined as:EQUATIONwhere A is a pooling operation (e.g., sum), F is a shared-MLP, and [x ′ 𝑖 𝑗 ; n 𝑖 𝑗 ; p 𝑖 𝑗 ] are explicit geometric features obtained from CFGen module for one point pair (x 𝑖 , x 𝑖 𝑗 ). By feeding OPFR representation r 𝑖 along with coordinate x 𝑖 into the backbone, the whole learning process can be achieved through end-to-end training.In Fig. 4 , the three-view drawing depicts the OPFR values of 1-st channel for an airplane. The blue hues represent areas with smaller OPFR values, typically the airplane body, while the red hues indicate larger OPFR values, primarily associated with the airplane wings. This color differentiation underlines our OPFR is sensitive to curvature variation across the airplane's structure, demonstrating the curvature-aware property of OPFR. We provide more visualization examples in the supplementary material. Additionally, it is important to highlight that, as shown in Fig. 5 , our OPFR can outperform vanilla PFH by a large margin with the help of learnable shared-MLP. Furthermore, the introducing of shared-MLP only increases 0.012M learnable parameters, which is approximately negligible for most popular backbones [31, 65] . 
experiments: 
 We evaluate our OPFR on two primary tasks: point cloud classification and semantic segmentation. We choose two representative point cloud understanding models, PointNet++ [31] and Point Transformer [65] , as our backbones to evaluate the effectiveness and compatibility of OPFR representations across different backbone architectures. Additionally, we carry out ablation studies to demonstrate the effectiveness of our OPFR network designs and quantitatively evaluate the efficiency and quality of OPFR feature representations. Moreover, due to space constraints, we present qualitative results in the supplementary material.Table 2 : Performance of semantic segmentation on S3DIS 6-fold and S3DIS Area-5 benchmarks. We evaluate different approaches in terms of mean Intersection over Union (mIoU, %), mean per-class accuracy (mAcc, %), overall accuracy (OA, %), number of parameters (#Params) and FLOPs. Bold means outperforming other models on corresponding dataset. Green means an improvement from our OPFR compared with the original backbone. Implementation details. For the Hierarchical Sampling module, we set 𝑘 1 = 20 and 𝑘 2 = 4 to control the number of candidate centroids and selected centroids respectively. The shared-MLP consists of three layers with 30 OPFR dimensions (r 𝑖 ∈ R 30 ), followed by a sum pooling operation. These are achieved via empirical studies, which will be further discussed in Sec. 4.3. Following RepSurf [33] , we set 𝑘 3 = 8, considering the trade-off between performance and efficiency. We use CrossEntropy loss and label smoothing [44] techniques with a ratio of 0.3 for both tasks. We provide more details about implementation in the supplementary material.We evaluate our OPFR on two commonly used benchmarks for point cloud classification: ModelNet40 [56] and ScanObjectNN [48] . Experimental setups. Following RepSurf [33] , we implement two versions to integrate OPFR with PointNet++ [31] , one standard version and one scaled-up version. The scaled-up version doubles the channels of standard version and exploits deeper networks. If not specified, we default to the standard version. We also apply the channel de-differentiation design [33] when integrated with PointNet++. We opt Adam [16] optimizer with default parameters to train our models for 250 epochs with a batch size of 64 and initial learning rate of 0.002. We apply exponential learning rate decay scheme with decay rate of 0.7. The whole training and testing process are conducted through one NVIDIA Quadro P5000 16GB GPU. For evaluation metrics, we use overall accuracy (OA) and mean accuracy within each classes (mAcc). For efficiency metrics, we use number of learnable parameters (#Params) and floating point operations (FLOPs). For a fair comparison, we calculate FLOPs from 1024 input point clouds, and utilize single-scale grouping (SSG) set abstraction [31] for all PointNet++ based [24, [31] [32] [33] methods. Classification on ModelNet40. ModelNet40 [56] is one synthetic object classification benchmark, which contains 9843 training samples and 2468 testing samples. They contain 100 unique CAD models from 40 object categories. The experimental results are presented in Tab. 1. The results reveal that our OPFR significantly improves PointNet++ [31] backbone by 3.8% OA and 3.2% mAcc, with just an additional 0.012M more parameters and 0.15G more FLOPs. The scaled-up OPFR further attains a slight improvement of 0.1% OA and 0.2% mAcc. Moreover, when integrated with transformer-based backbone, Point Transformer [65] , our OPFR achieves the state-ofthe-art 94.8% OA and 92.0% mAcc (+1.1% OA and +1.4% mAcc). Classification on ScanObjectNN. ScanObjectNN [48] is a challenging, real-world object classification benchmark. It is composed of 2902 point cloud samples from 15 categories, including occlusion and background. Following the typical protocol [29, 31, 33] , we verify our OPFR on the hardest variant (PB_T50_RS_variant) of ScanObjectNN. In Tab. 1, the proposed OPFR achieves 85.7% OA and 83.8% mAcc (+7.8% OA and +8.4% mAcc) on PointNet++ backbone, which outperforms RepSurf [33] by a large margin of 1.4% OA and 2.5% mAcc with comparable model size. Our result surpasses PointMLP [24] by 0.3% OA as well, and utilizes 9× fewer parameters. Furthermore, we scale up our proposed OPFR and achieve 88.5% OA and 86.6% mAcc, which demonstrates a superiority of 0.8% OA and 0.8% mAcc compared with state-of-the-art MLP-based backbone, PointNeXt [32] . Our result is also comparable to prior state-ofthe-art transformer-based backbone, SPoTr [29] , with around 2.1× fewer FLOPs. When integrated with Point Transformer, our OPFR attains a notable improvement of 5.8% OA and 5.6% mAcc, which only increases 0.003M more parameters and 0.04G more FLOPs.We evaluate our proposed OPFR representations on a challenging benchmark, S3DIS [1] , for semantic segmentation task. Experimental setups. When integrated with PointNet++ [31] , we apply the channel de-differentiation design [33] . We opt AdamW [23] with default parameters to train our models for 100 epochs with a batch size of 8 and initial learning rate of 0.006. Here, we employ multi-step learning rate decay scheme and decay at [60, 80] epochs with a decay rate of 0.1. The whole training and testing process are conducted through two NVIDIA A40 48GB GPU. For evaluation metrics, we use mean of classwise intersection over union (mIoU), mean of classwise accuracy (mAcc), and overall accuracy (OA). For a fair comparison, we calculate FLOPs from 15000 input point clouds [32] , and leave test-time-augmentation [3] in absence. Semantic Segmentation on S3DIS. S3DIS [1] encompasses 271 scenes which are distributed across 6 indoor areas, with each individual point being classified into one of 13 semantic labels. Following a common protocol [31, 45] , we evaluate the presented approach in two modes: (a) Area-5 is withheld for training and is used for testing, and (b) 6-fold cross-validation. In Tab. 2, our proposed OPFR considerably enhances PointNet++ [31] by 14.7%/16.9%/3.0% (mIoU/mAcc/OA) on S3DIS 6-fold benchmark. Our result is comparable to PointNeXt-XL [32] , with around 40× fewer parameters and 11× fewer FLOPs. When integrated with Point Transformer [65] , the performance of OPFR exceeds previous state-of-the-art Superpoint Transformer [41] by 0.9%/0.1%/1.6% (mIoU/mAcc/OA) for S3DIS 6-fold. Meanwhile, on S3DIS Area-5, our OPFR attains mIoU/mAcc/OA of 72.6%/78.6%/91.7% (+2.2%/+2.1%/+0.9%), surpassing the prior state-of-the-art ConDaFormer [3] . Furthermore, as shown in Tab. 3, we present quantitative segmentation results for each semantic class on S3DIS Area-5 in terms of mIoU. In Tab. 3, the top performance gain comes from the most challenging columns semantic class for both PointNet++ and Point Transformer backbones. Within all classes columns exhibit a distinct columnar structure, which consists of two or three planes in S3DIS dataset. This multi-plane structure can be effectively captured by different clusters generated from the proposed Hierarchical Sampling module, which facilitates the recognition of column pattern with greater ease. Furthermore, we provide qualitative results in the supplementary material. We ablate some critical designs of our standard OPFR with Point-Net++ [31] backbone on ModelNet40 [56] and ScanObjectNN [48] dataset for an insightful exploration. Effectiveness of different OPFR modules. Shown in Tab. 4, as we remove Hierarchical Sampling module, Curve Feature Generator module, and 3-layer shared-MLP, the overall accuracy (OA) decreases by 1.17%, 2.02%, 1.53% and mean accuracy (mAcc) drops by 0.91%, 1.76%, 1.34% respectively. From this empirical study, we can confirm that, explicit geometric features are crucial for 3D object understanding, and shared-MLP is necessary as well to enhance the semantics of obtained geometric features. Furthermore, due to the use of Hierarchical Sampling module, we can effectively release the distortion of triangle sets, thereby improving the quality of geometric features. Additionally, we argue that, the Hierarchical Sampling module can be applied to RepSurf [33] to handle the distorted triangle sets from 𝑘 nearest neighbors. Due to space limits, we provide the ablation study in the supplementary material. Designs of OPFR network architecture. We ablate the designs of OPFR network architecture in terms of pooling operation A and shared-MLP F in Tab. 5. Empirical results demonstrate that, usage of summation pooling, batch normalization, and three-layer shared-MLP with 30 OPFR dimensions outperforms other options.From our experiments, we hypothesize that, the network tends to [13, 52] . We propose novel Local Reference Constructor module to eliminate the needs of normal estimation and overcome the computational overheads. We achieve an impressive inference speed of 1.56ms (65× faster) with a marginal increase of 0.012M number of parameters. Therefore, OPFR can serve as a versatile plug-and-play module for various backbones. Furthermore, the efficiency of our OPFR is close to the previous state-of-the-art plug-and-play feature representation RepSurf [33] , with only 0.004M more parameters and 0.44ms more inference time.Quality of OPFR representations. Shown in Fig. 5 , we compare the performance between PFH [37] , RepSurf [33] , and proposed OPFR using PointNet++ [31] backbone. All of them are injected to PointNet++ as extra features. By incorporating vanilla PFH, overall accuracy (OA) and mean accuracy (mAcc) are enhanced by 2.0% and 2.6% on ModelNet40, 4.7% and 5.3% on ScanObjectNN, emphasizing the effectiveness of regional curvature knowledge. This gain further escalates to 3.8% and 3.2% on ModelNet40, 7.8% and 8.4% on ScanObjectNN in OA and mAcc respectively, when equipped with the proposed OPFR. This demonstrates the significance of shared-MLP, which enriches the obtained geometric features. Furthermore, compared with the previous state-of-the-art feature representation RepSurf, our OPFR outperforms it dramatically on ScanObjectNN, with a considerable margin of 1.4% and 2.5% higher OA and mAcc. We hypothesize that, this phenomenon is attributed to the uses of explicit curvature knowledge and robust sampling strategy, which are underexplored in RepSurf. 
conclusion: 
 We propose the novel plug-and-play module On-the-fly Point Feature Representation (OPFR) for various backbones. It explicitly captures local geometry including location, orientation and curvature through Curve Feature Generator module. We further develop the Local Reference Constructor module to improve efficiency and enable on-the-fly processing. Additionally, we introduce the Hierarchical Sampling module to mitigate the distortion of triangle sets that occurs in the naive 𝑘 nearest neighbors sampling, thereby enhancing the robustness of obtained geometric features.We evaluate the proposed OPFR on ModelNet40 [56] and ScanOb-jectNN [48] benchmarks for point cloud classification task, S3DIS [1] for semantic segmentation task. For both PointNet++ [31] and Point Transformer [65] backbones, our presented OPFR achieves the state-of-the-art results on different benchmarks. The comprehensive empirical results demonstrate the backbone-agnostic nature of our proposed method. We believe that our work can prompt consideration of how to better leverage geometric knowledge in network architecture designs for understanding point clouds.  
: 
 driven by advancements in sensor technologies like LiDAR and photogrammetry. This growing interest attributes to two key advantages: 1) It can accurately represent complex objects with numbers of points. 2) It can be quickly created by using 3D scanning devices. Compared to 2D image data, point clouds provide a more powerful 3D sparse representation containing abundant geometry and layout information of the environment.Deep learning technology [10, 18] has achieved significant improvements in various image processing tasks. However, the typical deep learning technology requires highly regular input data formats. The unordered and irregular point clouds bring great challenges to apply the image processing techniques directly. PointNet [30] , the pioneering work of network architecture that directly works with point clouds, overcomes the challenges of the unordered and irregular inputs. It uses point-wise shared-MLP followed by a pooling operation to extract global features from point clouds, but global pooling operation leads to the loss of valuable local information. PointNet++ [31] further proposes set abstraction (SA) to process local regions hierarchically. This step aggregates features from neighboring points, thereby capturing local information. However, it still learns from individual points without incorporating local relationships [21] . This could hinder the model from leveraging inherent point clouds geometric structures.Local geometric structures are vital for understanding point clouds. In an effort to capture this information, some prior works attempt to learn local relationships from convolutions [14, 20, 54] , attentions [8, 60, 65] , or graphs [50, 59, 63] . However, these methods require huge amount of labelled data to learn local geometry implicitly [33] , while getting large amount of labelled 3D annotations is difficult. Recently, RepSurf [33] has emerged as a novel approach that explicitly learns geometric information based on umbrella surface [6] , which is a triangle set 1 with connected triangles formed by 𝑘 nearest neighbors (𝑘-NNs). While triangle sets are effective in capturing location and orientation information, they often fall short in incorporating curvature knowledge, which is essential for accurate point cloud recognition [2, 42] . Moreover, as depicted in the supplementary material, in certain 𝑘-NNs, the 𝑘 points may come from different surfaces of the object. These "noisy points" can lead to the distortion of 𝑘-NN triangle sets, significantly impacting the quality of the obtained geometric features [33] .To integrate curvature information explicitly, we draw inspiration from Point Feature Histogram (PFH) [37] , a notable handcrafted feature descriptor for capturing regional curvature knowledge. PFH exploits the histogram of curvature angles within local neighborhoods to characterize individual points. As shown in Fig. 1 , these angles are calculated between the normal vector and the local coordinate system, which demand substantial computing resources. Nonetheless, many point cloud datasets [7, 48] lack normal vectors and necessitate additional normal estimation [12, 13, 38] . Normal estimation poses significant computational challenges, particularly for dense point clouds, while its accuracy degenerates considerably for sparse point clouds. These limitations can potentially lead to the breakdown of vanilla PFH approach, further underscoring the challenges of its direct integration with deep learning models.In view of PFH's potentials and limitations, we explore curvature information and propose On-the-fly Point Feature Representation (OPFR), which includes Local Reference Constructor module and Curve Feature Generator module. This provides an efficient way to leverage explicit curvature knowledge without the prerequisite of normal estimation, which inherently relies on the quality of triangle sets. Additionally, we propose the novel Hierarchical Sampling module to mitigate the distortion of triangle sets that occurs in the naive 𝑘-NN approach. Our sampling method demonstrates the robustness against noisy points by employing a hierarchical sampling strategy and a farthest point sampling strategy. As a result, it can significantly improve the obtained geometric features. These innovations confer the following properties:• Curvature Awareness. The usage of curvature information remains underexplored by prior works. Our proposed OPFR obtains the capability to explicitly capture not only location and orientation knowledge, but also curvature geometry via Curve Feature Generator module. • Computational Efficiency. Vanilla PFH is computationally expensive due to normal estimation. Our proposed OPFR introduces Local Reference Constructor module, which approximates the local coordinate systems based on triangle sets to overcome the computational bottlenecks. • Robustness. Naive 𝑘-NN sampling causes distortion of triangle sets, which compromises the obtained geometric features.In contrast, our proposed OPFR presents Hierarchical Sampling module to enhance the quality of triangle sets, ensuring robust geometric features for noisy points. Moreover, our OPFR is backbone-agnostic, making it compatible with different 3D point clouds analysis architectures. We demonstrate its model-agnostic nature by adapting two representative backbones: PointNet++ [31] and Point Transformer [65] . It serves as an efficient plug-and-play module, and achieves substantial performance improvements. Empirical results prove its compatibility with different backbones. When incorporating with Point Transformer backbone, our OPFR achieves state-of-the-art performance for both point cloud classification and semantic segmentation tasks. 
 
 Appendix: In this paper, we refer triangle set to a collection of connected or disconnected triangles. For those connected ones, they form a surface. 


Decision: Initially the paper receives 3 positive but 1 negative ratings. After the rebuttal, all reviewers are satisfied with the authors' follow-up answers and efforts on addressing their questions. Particularly, as one reviewers raised the rating to borderline accept, all four reviewers reached a concensus on accpeting this submission. After carefully reading both the reviews and the paper, the AC agrees with the reviewers' recommendations, given its merits in presenting a new point cloud feature representation method with strong performances in fundamental 3D analysis tasks. The AC decides to recommend an acceptance, and suggests the authors to prepare a better camera ready paper by referring to some rebuttal content.

Reviews: review: paper_summary: paper_summary: This paper proposes a local point feature representation, On-the-fly Point Feature Representation (OPFR), for point cloud analysis. The method consists of three main modules: Hierarchical Sampling, Local Reference Constructor and Curve Feature Generator. OPFR obtains the capability to explicitly capture not only location and orientation knowledge, but also curvature geometry. The experiments show that the method facilitates the classification and semantic segmentation performances.
strength_weakness: strength_weakness: - The local reference constructor utilizes the two nearest points (clockwise and counterclockwise) of a point pair to construct a local reference frame. It avoids using the normal vector to build the frame and improve the efficiency.

- The curve feature generator uses a curvature proxy to depict the surface curvature and add more geometric details to point features.

- The experiments show that OPFR can facilitate classification and semantic segmentation tasks based on different backbones.
limitations: limitations: - The Hierarchical Sampling is kind of trivial. The sampling procedure can directly use FPS to sample more points and create clusters by k-NN. The Hierarchical Sampling module doubles the sampling and clustering step, which is not necessary and more complicated.

- The efficiency of OPFR should be further tested.  The construction of a local reference frame is applied to each point pair in each local cluster, which is a little fussy. I am curious about the inference time change before and after adding the OPFR to PointNet++ and PointTransformer.

- The procedure for finding the clockwise and counterclockwise most adjacent points x_{ij}^+, x_{ij}^- needs more details. Besides, the efficiency of getting these points for each point pair needs further analysis.
paper_summary: paper_summary: This paper presents a novel approach to point cloud representation and analysis through the introduction of the On-the-fly Point Feature Representation (OPFR) module. This module aims to provide a more explicit depiction of local geometric information within point clouds. The method is composed of three key components: Hierarchical Sampling, Local Reference Constructor, and Curve Feature Generator. Hierarchical Sampling improves triangle set generation robustness, while the Local Reference Constructor offers an alternative method for generating point clouds independent of local geometric data. The Curve Feature Generator utilizes second-order derivative curvature information to generate detailed curvature features. The paper demonstrates the effectiveness of OPFR across various benchmarks for point cloud classification and semantic segmentation tasks, achieving state-of-the-art results with different backbone architectures like PointNet++ and Point Transformer. Overall, the paper highlights the importance of integrating geometric knowledge into network architecture designs for better understanding and processing of point clouds.
strength_weakness: strength_weakness: The manuscript presents innovative approaches in leveraging curvature information and refining sampling algorithms. These methodologies are rigorously validated through a comprehensive array of experiments. The clarity and coherence of the presentation are commendable. It is noteworthy that the method exhibits strong reproducibility and simplicity, with no undue complexity.
limitations: limitations: The current state of the art remains uncertain, particularly with regard to the selection of the 2021 experimental baseline. Additionally, the paper referenced for the utilization of curvature information dates back to 2016, lacks comparison with more recent studies, and fails to convincingly demonstrate the advancement and effectiveness of curvature information utilization.
paper_summary: paper_summary: In this paper, the authors introduce a novel module, On-the-fly Point Feature Representation (OPFR), which is designed to enhance point cloud analysis by explicitly capturing geometric information such as curvature and orientation. The OPFR module incorporates a Curve Feature Generator that builds upon the concept of the Point Feature Histogram (PFH), yet it is significantly optimized to improve performance. By introducing the Local Reference Constructor module, the authors manage to approximate local coordinate systems using triangle sets, enabling the OPFR module to operate much faster (65 times faster than the traditional PFH) and with minimal additional parameters.
strength_weakness: strength_weakness: 1.	This paper introduces a novel Curve Feature Generator module that effectively leverages explicit features from point clouds, significantly improving performance.
2.	The methodology is presented clearly and in detail, making it easy to follow.
3.	The experiments conducted are both thorough and comprehensive.
limitations: limitations: 1.	This paper experimentally follows the methodologies of RepSurf but does not include a comparison and analysis with RepSurf-U. Notably, some ideas presented in this paper bear similarities to those in RepSurf.
In Table 3, the values in the beam column are listed as 0. Is this indicative of actual zero results, or were these values possibly not filled in?
paper_summary: paper_summary: This paper addresses the challenges in point cloud analysis due to their unorderness, sparsity and irregularity. Traditional methods struggle with capturing explicit local geometry like curvature and orientation. The authors propose an On-the-fly Point Feature Representation (OPFR) to explicitly capture geometric information through a Curve Feature Generator module, inspired by Point Feature Histogram (PFH). The OPFR is efficient, requiring only 1.56ms for inference, making it 65 times faster than vanilla PFH. The proposed method significantly improves classification and semantic segmentation accuracy on various datasets.
strength_weakness: strength_weakness: 1. **Efficiency**: The On-the-fly Point Feature Representation (OPFR) significantly improves computational efficiency, requiring only 1.56ms for inference, which is 65 times faster than the traditional Point Feature Histogram (PFH). This efficiency is crucial for real-time applications and large datasets.

2. **Explicit Geometric Information**: The OPFR captures explicit geometric information, including location, orientation, and curvature, which are essential for accurate point cloud analysis. This approach overcomes the limitations of previous methods that implicitly capture geometric information.

3. **Versatility and Compatibility**: The OPFR is designed as a plug-and-play module that can be integrated with various backbones, including MLP-based and Transformer-based architectures. This versatility allows it to enhance different types of neural network architectures used in point cloud analysis.
limitations: limitations: 1. **Complexity of Hierarchical Sampling**: The Hierarchical Sampling module, while improving the quality of geometric features, introduces additional complexity in the preprocessing stage. This might limit its scalability for extremely large datasets or real-time applications where rapid data processing is crucial.
2. **Dependency on Accurate Triangle Sets**: The effectiveness of the Local Reference Constructor and Curve Feature Generator modules relies heavily on the accuracy and quality of the triangle sets generated.
3. **Sensitivity to Hyperparameters k1 and k2**: Experimental results show that OPFR is relatively insensitive to the choices of k1 and k2. Please provide a clearer explanation.