Multi-fineness Boundaries and the Shifted Ensemble-aware Encoding for Point Cloud Semantic Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 15:54:50

Abstract: Point cloud segmentation forms the foundation of 3D scene understanding. Boundaries, the intersections of regions, are prone to mis-segmentation. Current point cloud segmentation models exhibit unsatisfactory performance on boundaries. There is limited focus on explicitly addressing semantic segmentation of point cloud boundaries. We introduce a method called Multi-fineness Boundary Constraint (MBC) to tackle this challenge. By querying boundaries at various degrees of fineness and imposing feature constraints within these boundary areas, we enhance the discrimination between boundaries and non-boundaries, improving point cloud boundary segmentation. However, solely emphasizing boundaries may compromise the segmentation accuracy in broader non-boundary regions. To mitigate this, we introduce a new concept of point cloud space termed ensemble and a Shifted Ensemble-aware Perception (SEP) module. This module establishes information interactions between points with minimal computational cost, effectively capturing direct point-to-point long-range correlations within ensembles. It enhances segmentation performance for both boundaries and non-boundaries. We conduct experiments on multiple benchmarks. The experimental results demonstrate that our method achieves performance surpassing or comparable to state-of-the-art methods, validating the effectiveness and superiority of our approach.

Main paper: 
 introduction: 
 Point cloud semantic segmentation is a fundamental task in understanding 3D scenes. Due to point clouds' unordered and unstructured nature, mature convolutional methods [19, 20] cannot be directly applied to them. [9, 33, 61] preprocess point clouds into structured voxels suitable for 3D convolutions. [7, 12, 15] project point clouds from different views and extracts features using 2D convolutions. PointNet [35] introduces the symmetric functions, enabling direct input of raw point clouds into neural networks. Subsequently, numerous excellent methods [32, 36, 38] have emerged. However, most of them are unsatisfactory for segmenting boundaries [43] . Boundaries are transitional areas between regions, blending features from points belonging to various categories. These mixed features are characterized by complexity and ambiguity. As the network deepens, these ambiguous features encompassing information from multiple semantic categories inevitably spread within boundary regions, resulting in an unsatisfactory performance on semantic segmentation at boundaries [10] .Boundary plays a critical role in semantic segmentation. However, explicit research on semantic segmentation of point cloud boundaries remains relatively limited compared to 2D image segmentation. JSENet [23] and BGE [10] build a boundary prediction branch network to assist segmentation. CBL [43] utilizes contrastive learning to optimize the representation of boundary features. The commonality among these methods is the requirement to pre-know the set of boundary points by querying. The quality of the boundary query will affect the performance of boundary semantic segmentation. In 2D images, pixels are semantically connected and continuous, and a set of pixels can accurately define boundaries. However, the increased dimensions and the sparsity of point clouds make objects have more complex spatial interactions with others, bringing challenges to the boundary definition. [10, 43] use the neighbor query method KNN (K-Nearest Neighbors) to iterate over points and label the point as a boundary point if there are points with different classes in its neighbor. The accuracy of boundaries heavily depends on the single neighbor query setting, which have a non-negligible impact on subsequent semantic segmentation.Different query degrees of fineness result in different boundaries. High-confidence boundaries can be obtained with a high degree of fineness query (i.e., reducing the neighbor query range or the number of query points), but some boundary points may be missed. Conversely, with a low degree of fineness query, incorrect boundary delineation may occur, but the integrity of boundary points increases. Given this, we propose a method called Multi-fineness Boundary Constraint (MBC). As shown in the bottom-right of Figure 1 , it conducts boundary queries at different degrees of fineness to obtain more precise and complete boundaries. Boundary points with the same category and located nearby should exhibit similar features. MBC randomly selects multiple points as key points for feature constraints in the boundary neighbors. It encourages similarity among points with the same class and dissimilarity between points with different classes, enhancing the discrimination between boundaries and non-boundaries. Without additional networks, this approach reduces the negative impact of erroneous boundary queries and improves the segmentation accuracy of boundaries.Introducing additional boundary feature constraints may divert the model's attention from semantic segmentation, affecting the segmentation performance of extensive non-boundary areas [43] . Balancing global segmentation accuracy is another challenge we encounter. Long-range correlations refer to dependencies between distant points, such as geometric and semantic relationships [25] . They aid the model in comprehending the semantic structure and context of the entire point cloud, enabling a broader perception of the distinctions between boundaries and non-boundaries, thereby enhancing overall segmentation performance. Most models extract and aggregate features within neighbors consisting of several tens of points. They rely on the deepening of the network to expand the receptive field progressively. This indirect way makes capturing long-range correlations among points difficult, potentially resulting in the blurring or even loss of crucial features during multiple abstraction processes. As illustrated in the bottom-left of Figure 1 , we introduce a novel concept in point cloud space called ensemble, along with a shifted ensemble-aware perception module (SEP) to address this issue. Unlike methods like BGE [10] that employ masks to prevent feature aggregation, we encourage information exchange between points. The point cloud is randomly partitioned into multiple ensembles with an equal point count. SEP captures long-range correlations within each ensemble, enabling feature abstraction and aggregation beyond local neighbors. For better global features, SEP generalizes the ensemble feature pattern to the global point cloud space with minimal computational cost through shifting and parameter-sharing on multi-scale point clouds, which improves the overall semantic segmentation performance.The MBC and the SEP are detailed in Section 3.1 and Section 3.2, respectively. We name our method MBSE (Multi-fineness Boundaries and the Shifted Ensemble-aware Encoding framework), which enhances the segmentation performance of both boundaries and non-boundaries. Our main contributions are as follows:• We propose a method called multi-fineness boundary constraint (MBC). Without additional networks, it enhances the feature discrimination between boundaries and nonboundaries, assisting the discriminator in making better segmentation decisions for boundaries. • We propose a new concept in point cloud space called ensemble and a shifted ensemble-aware perception (SEP) module. SEP delicately captures direct point-to-point long-range correlations within ensembles. Through shifting and parameter sharing, SEP generalizes the ensemble-level patterns to the entire point cloud with low computational cost, enhancing the overall semantic segmentation accuracy. • The extensive experimental results demonstrate that MBSE notably improves the performance of baselines, achieving state-of-the-art or highly competitive performance on multiple benchmarks with an increase of just 1M parameters and 0.5 GFLOPs in computational cost. 
related work 2.1 boundary segmentation: 
 Research on boundary semantic segmentation of 2D images has a long history. Initially, researchers use low-level image features such as color to predict boundaries. SBD [17] introduces the first semantic boundary dataset for 2D images, accelerating the development of 2D boundary segmentation. HFL [2] establishes a two-stage prediction network for separately predicting boundaries and boundary categories. CaseNet [57] proposes an end-to-end deep semantic edge learning architecture based on ResNet [19] with category awareness. PED [22] aggregates semantic and instance boundary segmentation tasks into a multi-branch network. SEAL [58] proposes a method of correcting noisy labels to help networks generate high-quality boundaries.Compared to the mature 2D image boundary segmentation, relatively fewer studies explicitly focus on point cloud boundary segmentation. Some methods [10, 23] divide 3D semantic segmentation into boundary prediction and semantic segmentation subtasks. The segmentation accuracy of these methods is closely related to the quality of boundary prediction, incorrect boundary predictions can adversely affect semantic segmentation. CBL [43] demonstrates the poor performance of current point cloud semantic segmentation models in boundary segmentation and proposes a method based on contrastive learning to enhance boundary feature representation. These explicit boundary semantic segmentation methods rely on querying to obtain boundary points. However, none of the above methods realize the impact of boundary queries on semantic segmentation. Our work does not treat the boundary prediction as a subtask. We introduce feature constraints on boundaries at different degrees of fineness to optimize the feature representation and enhance the semantic discrimination between boundaries and nonboundaries, aiding the discriminator in making better segmentation decisions on boundary areas.The inherent non-structural and unordered characteristics of point clouds pose challenges for feature abstraction. Various approaches have been developed for preprocessing point clouds to structured data. Voxel-based methods [31, 47, 54, 61] transform point clouds into fixed-size voxels, treating them as the smallest processing units for feature abstraction. SEGCloud [44] utilizes a 3D fully convolutional neural network (CNN) to extract features from point cloud voxels and then restores the prediction results to the original point cloud resolution through trilinear interpolation. To mitigate unnecessary computations caused by sparse point clouds when applying 3D dense convolution, OctNet [41] uses an unbalanced octree to partition the point cloud hierarchically, with its leaf nodes storing feature representations extracted by 3D convolution. Voxelbased methods can directly utilize mature CNNs to extract point cloud features, but the voxelization process may lead to the loss of information such as spatial position.Methods based on multi-view projection [7, 15, 16, 42, 49] project 3D point clouds into 2D images from different viewpoints. CNNs are employed to extract and fuse features that can be utilized for downstream tasks. SnapNet [3] generates RGB and depth images, employs fully convolutional networks for 2D image feature abstraction and labeling, and then reprojects them back to the point cloud. SnapNet-R [11] improves upon SnapNet by independently using FuseNet [18] to extract features and performing semantic segmentation on multi-view images. Methods based on multi-view are susceptible to the projection angles. Additionally, this method may lose features due to internal occlusions within point clouds.Point-based methods [24, 48, 55, 59 ] directly extract features from the raw point cloud, making them a current research hotspot in point cloud semantic segmentation. PointNet++ [36] introduces hierarchical and multi-scale grouping structures to capture features at multiple scales and local neighbors. Subsequently, Point Transformer [60] and StratifiedTransformer [25] establish transformerbased frameworks for point cloud semantic segmentation, which achieve excellent performances. Meanwhile, MLP-based methods PointNext [38] and PointVector [6] demonstrate the competitiveness of MLPs (Multi-Layer Perceptions) and Transformer [46] . However, the methods mentioned above progressively aggregate local features in neighbors, making it difficult for distant points to interact. Although Transformer structures have the advantage of large receptive fields, they still apply self-attention at a local feature level due to the high volume of point cloud data. The locally aggregated features may blur the information of key points. Unlike existing works, we focus on the direct interaction between points. We generalize the patterns to the entire point cloud through shifting and parameter-sharing with a low computational cost, capturing better local and global feature representations. 
method: 
 In this section, we present our MBSE framework, comprising the multi-fineness boundary constraint (MBC) and the shifted ensembleaware perception (SEP). The two components are described in Section 3.1 and 3.2, respectively.Explicitly improving the performance of point cloud boundary segmentation requires accurate localization of boundaries. We observe that point cloud boundary queries can impact semantic segmentation performance, a problem that has yet to be systematically explored in previous research. Incorrect boundary queries can adversely affect boundary feature representations, thereby misleading semantic segmentation discriminators. In this section, we present a multi-fineness boundary constraint method that imposes feature similarity constraints on multiple sets of boundaries queried with different degrees of fineness. This approach enhances the semantic discrimination between boundaries and non-boundaries, improving the segmentation performance on boundaries.Query. [10, 43] uses KNN to query the point's neighbor and determines whether it is a boundary point according to the categories of the points in its neighbor, which has a severe drawback. Due to the uneven distribution of point clouds, KNN, which lacks distance restrictions, may group distant points into the same neighbor, leading to errors in boundary neighbor queries in sparse areas. We refer to this situation as the enclave phenomenon. To address this issue, we use a ball query to limit the query distance of the neighbors. Specifically, to classify a point 𝑥 𝑖 , which belongs to the point cloud X, we query the 𝑘 nearest points of 𝑥 𝑖 within the preset 3D spatial range. If there is a point that is inconsistent with the class of 𝑥 𝑖 , 𝑥 𝑖 is identified as a boundary point. The set of boundary points B can be represented by the following formula: Here, 𝑙 𝑖 represents the ground truth of 𝑥 𝑖 , and 𝑄 𝑘 𝑟 (𝑥 𝑖 ) denotes the operation of querying the 𝑘 nearest points within a distance 𝑟 from 𝑥 𝑖 . Although the method alleviates the enclave phenomenon, inappropriate settings of 𝑘 and 𝑟 may lead to inaccurate boundary queries for point clouds. A high degree of fineness query (i.e., reducing 𝑟 or 𝑘) can enhance the precision of boundary queries but may overlook some boundary points. Conversely, a low degree of fineness query (i.e., increasing 𝑟 or 𝑘) can obtain more complete boundaries but might mistakenly classify non-boundary points as boundary points. Considering this, we conduct boundary queries at multiple degrees of fineness, obtaining multiple sets of boundaries by setting different values of 𝑘 and 𝑟 . The multi-fineness boundary set B 𝑚 can be formalized as:EQUATIONB 𝑚 = {𝑥 𝑖 ∈ X | ∃ 𝑥 𝑗 ∈ 𝑄 𝑘 𝑚 𝑟 𝑚 (𝑥 𝑖 ) ∧ 𝑙 𝑖 ≠ 𝑙 𝑗 }, 𝑚 = {0, ..., 𝑀 -1}. (2)Where 𝑀 represents the number of fineness degrees. Conducting boundary queries at multiple degrees of fineness helps mitigate potential issues from a single query, such as boundary misclassification and incompleteness. To address this issue, we impose feature constraints within the boundary neighbor whose center point is the boundary point. In the neighbor, we enhance the feature similarity between the constraint key points and the points belonging to the same category while reducing it otherwise. CBL [43] takes the center points as constraint key points. As shown in Figure 2a , The single center point constraint only imposes constraints between the constraint key point and its neighboring points, neglecting the establishment of constraint relationships among other points except for the constraint key point. The similarity information of features within the boundary neighbor is underutilized, limiting the effectiveness of the feature constraints. As illustrated in Figure 2b , we extend the feature constraints to the entire boundary neighbor. Specifically, in a boundary neighbor, 𝑞 non-repeating points are randomly selected as constraint key points for 𝑞 times of feature constraints. Inspired by [8, 14, 34] , the feature constraint for the boundary neighbor centered at 𝑥 (𝑥 ∈ B) can be formulated as follows:EQUATION)Where 𝑑 represents the Euclidean metric. 𝜏 represents the constraint temperature. 𝑅(•) represents the operation of randomly non-repeated selection. Note that in low-density regions, there may be cases where the number of points is less than 𝑘 in some neighbor queries. In our implementation, the points are randomly duplicated to ensure each boundary neighbor contains the same number of points. We independently apply random multi-point boundary feature constraints to boundaries at different fineness degrees. The multi-fineness boundary feature constraint L 𝑀𝐵𝐶 can be formulated as follows:EQUATIONHere, 𝜆 𝑚 represents the weight of the feature constraint for the 𝑚-𝑡ℎ boundary query fineness degree. Boundary constraints at higher fineness degrees are assigned higher weights to ensure the accuracy of feature constraints. In contrast, those at lower fineness degrees are given lower weights to balance the completeness of boundary feature constraints. Finally, the total criterion of our MBSE framework isEQUATIONWhere L 𝑠𝑒𝑔 stands for the loss of semantic segmentation. The above settings will be described in Section 4.1. The multi-fineness boundary constraints enhance the feature discrimination between boundaries and non-boundaries, reducing the difficulty of classification for the semantic segmentation discriminator. It improves the segmentation performance of boundary regions. The detailed evaluation and analysis of MBC is given in Section 4.4.1.Introducing boundary feature constraints may distract the network from supervising segmentation, leading to problems such as unbalanced segmentation accuracy [43] . Long-range correlations enable the network to better comprehend the semantic structure of point clouds and discern differences between boundaries and non-boundaries across larger spatial extents. In this section, we introduce a new point cloud space called ensemble and propose a novel shifted ensemble-aware perception module. This module captures the direct point-to-point long-range correlations within ensembles. Additionally, It generalizes the ensemble patterns to the global point cloud with minimal computational cost, enhancing the overall segmentation performance. Ensemble Point Cloud Space. Long-range correlation refers to the association or relationship that exists between distant elements [25] . Capturing the long-range correlation can help the model identify semantic relationships between different parts of the point cloud, thereby improving semantic segmentation accuracy. Obtaining long-range correlations requires establishing remote information interaction. Current point cloud segmentation models [6, 29, 38] mainly rely on aggregating features in local neighbors, which consist of several tens of points, to obtain high semantic features for segmentation. These models' receptive fields gradually increase as the networks deepen and local features are aggregated. Obtaining relevant information between two non-adjacent points may require multiple rounds of feature abstractions or aggregations. In this complex process, some crucial long-range correlation information may be lost. However, due to the high volume of point cloud data, establishing direct global long-range interactions requires enormous computational costs.To avoid this, we propose a novel concept in point cloud space called ensemble. The input point cloud X is randomly divided into 𝑛 non-overlapping ensembles 𝜖 𝑖 (i.e., 𝜖 𝑖 ⊂ X and 𝜖 𝑖 ∩ 𝜖 𝑗 = ∅). Each ensemble contains the same number of points. The spatial size of the ensemble depends on the size of the input point cloud, with the detailed configuration described in Section 4.1. Without extra computational overhead, the random partitioning of ensembles allows distant points in the point cloud to be grouped into the same feature encoding unit, facilitating direct information exchange between distant points. Direct information interactions and contextual relationships can be established within each ensemble, capturing long-range correlation. Compared to traditional local neighbor aggregation, ensembles extend the feature encoding space. The method for obtaining long-range correlation encodings within an ensemble is described in the following subsection.Ensemble-aware Perception Encoding. The feature encodings embedded with long-range correlations are obtained in each ensemble. The diagram of the ensemble-aware perception module is illustrated in Figure 3 . We take the inherent unordered nature of point clouds as prior knowledge, where semantic features should remain invariant to translation or rotation. Specifically, for each ensemble 𝜖 𝑖 , the difference between its position 𝑝 𝜖 𝑖 and the average position of the point cloud 𝑝 X is embedded into the input features 𝑓 𝜖 𝑖 . This embedding ensures the position-invariant representation of the perception module and facilitates the generalization of the perception module across different ensembles in subsequent stages. This operation F can be formulated as follows:EQUATIONWhere [•, •] denotes the concatenation operation. ℎ 𝜃 𝑝 is an operation implemented with an MLP that integrates features with position information and reduces the feature dimension to 1/𝑠 of its original size. 𝑛 denotes the number of ensembles in the point cloud. The output of F (𝜖 𝑖 ) serves as the initial encoding for the 𝑖-𝑡ℎ ensemble. The ensemble-aware perception module captures long-range correlations between points by computing associations and transforming the feature space. We argue that long-range correlations should exist in diverse aspects, such as semantics, geometries, color spaces, and density distributions. A single feature space transformation is insufficient to characterize point clouds' complex spatial and semantic features. We employ 𝐺 space transformations to capture long-range correlations within ensembles from multiple perspectives. Additionally, we perform contribution perception for different space transformations. The long-range correlation set M, which encompasses 𝐺 long-range correlations for 𝜖 𝑖 , can be formulated as:EQUATIONHere, A and T represent operations for feature alignment and space transformation, respectively, implemented by MLPs. 𝑤 𝑗 denotes the contribution perception weight of the 𝑗-𝑡ℎ space transformation, implemented through a set of learnable parameters processed by Softmax function. Then, we multiply the 𝐺 long-range correlation matrixes with the initial feature encoding F (𝜖 𝑖 ) and sum them point-wise to integrate long-range correlation representations. The perception encoding of the 𝑖-𝑡ℎ ensemble can be formulated as: deepening the network and feature extraction, the ensemble-aware perception module's direct receptive field spans the ensemble rather than just the neighbor with several dozen points.EQUATIONMulti-layer Shifting and Parameter Sharing. Hierarchical structures are widely applied in point cloud segmentation [6, 29, 38] . Existing models utilize hierarchical structures to downsample point clouds, obtaining multiple levels of sub-point clouds.With increasing downsamplings and feature abstraction iterations, high-layer sub-point clouds containing fewer points encapsulate richer semantic information conducive to semantic segmentation. While low-layer sub-point clouds with more points exhibit lower semantic feature levels, they compensate for detailed information such as position lost in high semantics. Ensemble-aware perception modules are applied separately at different layers to fully utilize features from different hierarchical layers. At lower layers, it enhances models' capability to capture spatial geometric patterns, while at higher layers, it strengthens global interactions among high-level semantic features. Like images, point clouds also possess positional invariance, necessitating ensemble-aware perception to be spatially agnostic. The ensemble-aware perception module shifts and shares its parameters across the point cloud to capture and generalize long-range correlations. We obtain the shifted ensembleaware perception encoding of the point cloud X 𝑙 at the 𝑙-𝑡ℎ layer by concatenating its 𝑛 ensemble-aware perception encodings at spatial locations aligned with the input and establishing a shortcut connection, which can be formulated as follows:EQUATIONMulti-layer shifting and parameter sharing establish an information exchange pathway from ensembles to the entire point cloud with low computational cost, further expanding the receptive fields. It enables the model to interact information with a global receptive field at different layers and generalize local patterns among different ensembles, thereby enhancing its ability to capture global spatial geometry and long-range correlations. The detailed evaluation and analysis of SEP is given in Section 4.4.2. 
experiments: 
 In this section, we showcase and analyze the performance of MBSE on point cloud semantic segmentation and part segmentation on multiple benchmarks. Through comparative experiments with various recent state-of-the-art methods and ablation studies, we demonstrate the effectiveness and superiority of our MBSE.We apply the MBSE framework to the popular and representative PointNext [38] family, taking its four variants as baselines. We add an SEP after each of their four sub-sampling modules and replace the loss function with our L 𝑀𝐵𝑆𝐸 . For fairness, we employ the same training and evaluation strategies as the baselines. Except for ShapePartNet [56] , we do not use any voting strategy for evaluation.For MBC, we conduct boundary queries on three fineness degrees (i.e., 𝑀 = 3). The boundary neighbor query distances 𝑟 𝑚 and point numbers 𝑘 𝑚 are set to (0.05, 0.1, 0.2) and (4, 8, 12) , respectively. The number of random constraint key points is set to (2, 4, 6) . The constraint temperature 𝜏 is set to 1. The weights 𝜆 𝑚 for multifineness constraints are set to (0.5, 0.3, 0.2). For SEP, the downscale factor 𝑠 is set to 4. The four downsampled sub-point clouds are evenly divided into (36, 28, 20, 12) ensembles, respectively. The criterion used for segmentation (i.e., L 𝑠𝑒𝑔 ) is CrossEntropy. 24k points are fed as a batch to train the models. We evaluate the models using the entire scene as input. Unless otherwise specified, all experiments are conducted according to the above settings.For point cloud semantic segmentation, we evaluate our MBSE on the widely used S3DIS [1] and ScanNet [5] . S3DIS is a challenging large-scale 3D point cloud dataset. The results on S3DIS Area 5 are shown in Table 1 . MBSE improves the performance of all Point-Next variants. Compared to the PointNext-XL [38] , MBSE increases mIoU, OA, and mAcc by 1.3%, 0.4%, and 0.8%, respectively, and surpasses PointVector-XL [6] in terms of mIoU and OA. It achieves the best or second-best performance in IoU for almost half of all categories. Compared to PointVector, MBSE extends the direct receptive field to a larger ensemble level rather than the traditional neighbor.Based on the Transformer architecture, StratifiedFormer [25] has a larger receptive field, which is an advantage over the models based on MLPs. Nevertheless, MBSE performs highly competitively because it can capture direct correlations between points over long distances without requiring feature aggregation in neighbors. Furthermore, MBSE performs well in segmenting objects with high boundary proportions, such as beams and columns. On PointNext-L and PointNext-XL, the IoU of the beam is improved to 0.2% and 0.4% by MBSE, respectively, while this metric remains at 0.0% for almost all other comparative models.We further evaluate MBSE on S3DIS Area 6-fold and ScanNet. The results are reported in Table 2 . ScanNet comprises 1,613 3D indoor scans. We train and evaluate the models on training and validation sets, respectively. MBSE significantly improves the performance of all baselines. On S3DIS Area 6-fold, except for mIoU, the PointNext-XL model equipped with MBSE achieves the best performance. On ScanNet, the number of input points is set to 64k. The MBSE versions of PointNext-L and PointNext-XL achieve a 1.1% and 1.4% improvement in mIoU compared to the baselines, respectively. MBSE can provide more considerable performance boosts for larger baseline models. It is primarily due to larger models being able to extract richer high-level semantic information. Increasing feature discrimination and capturing long-range correlations for high semantic features yields superior semantic segmentation performance. To accurately evaluate the impact of MBSE on baselines, we did not modify the structure of the baselines. MBSE is applied with the same experimental settings on all baselines, with parameters and computational costs of 1M and 0.5 GFLOPs, respectively. The volume of the baselines primarily determines the parameters and computational costs of the models.ShapeNetPart [56] is a widely used dataset for 3D object part segmentation. It contains a total of 16 different object categories with 16,880 models. Each category is comprised of 2 to 6 parts, resulting in a maximum of 50 labeled parts. The results are reported in Table To further verify the effectiveness of our MBSE, we perform ablation studies on S3DIS Area 5 with PointNext-XL as the baseline. 86.2 84.0 KPConv [45] 86.4 85.1 CurveNet [50] 86.8 -ASSANet [37] 86.1 -Point Transformer [60] 86.6 83.7 PointMLP [32] 86.1 84.6 StratifiedFormer [25] 86.6 85.1 PointVector [6] 86.9 -PointMetaBase [29] 87. To explore the effectiveness of SEP, we conducted ablation studies on its key components. The results of its performance and computational cost are reported in Table 5 . Removing the relative position embedding led to a decrease in all metrics, demonstrating the effectiveness of position-invariant representation. Non-shared parameter results in a 4.8M increase in the number of parameters and a sharp drop in performance. The main reason is that ensembles are randomly sampled sub-point clouds and the non-shared parameter poses challenges for pattern convergence. Furthermore, we explore how the number of space transformation blocks affects SEP. Increasing the number of blocks enables the model to perceive the interrelations between points from more aspects, capturing long-range correlations comprehensively. However, an excessive number of the blocks might overfit the training data, leading to a decrease in performance. After comparison, we set the default number of space transformation blocks to 8. Different long-range correlations contribute differently to the segmentation. Contribution perception enabled SEP to achieve better performance. SEP boosted the baseline's mIoU, OA, and mAcc by 0.9%, 0.2%, and 0.6% in low computational cost, proving the effectiveness of SEP.  
conclusion: 
 We conduct the first systematic study on the impact of boundary queries on the semantic segmentation of point clouds. We introduce a multi-fineness boundary constraint approach, which explicitly enhances the feature discrimination between boundaries and non-boundaries, improving the accuracy of difficult-to-segment categories with high boundary proportions. Additionally, we proposed a shifted ensemble-aware perception module that establishes direct point-to-point interactions in novel ensemble spaces. With low computational costs, SEP effectively generalizes the discrimination patterns between boundaries and non-boundaries and the long-range correlations across the entire point cloud. Experimental results demonstrate that MBSE significantly improves segmentation performance for classes with high boundary proportions and the overall point cloud. We hope this work will inspire further exploration into studying the explicit enhancement of boundary segmentation and the long-range correlations in point clouds.  
 
 Appendix:  


Decision: Initially the paper received 3 positive and 1 negative ratings. After rebuttal, one reviewer raised the rating, and thus all four reviewers reached a consensus on "borderline accept". After carefully reading both the reviews and the paper, the AC agrees with the reviewers' recommendations, and concludes that the submission hold values in proposing a novel method to enhance semantic segmentation of point cloud boundaries. However, the paper also has some weaknesses such as lack of testing with more recent backbones, marginal improvements, etc. The AC leans towards an acceptance, and would encourage the authors to further improve the quality of camera ready paper by carefully referring to the reviewers comments.

Reviews: review: paper_summary: paper_summary: The author introduces Multi-fineness Boundary Constraint (MBC) and shifted ensemble-aware perception (SEP). The former enhances the discrimination between boundaries and non-boundaries, thereby improving point cloud boundary segmentation. The latter delicately captures direct point-to-point long-range correlations within ensembles. Extensive experiments validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: Without additional networks, it enhances the feature discrimination between boundaries and nonboundaries, assisting the discriminator in making better segmentation decisions for boundaries.
limitations: limitations: Weakness:
1. The motivation of this paper is to emphasize the importance of boundary query for segmentation prediction, but there is no “BIoU” to measure the effectiveness of the proposed MBC in the experimental results. Please explain the reason and make experimental supplements.
2. Line 346-350, this part is confused, the author states that the current method based on boundary query only relies on K nearest neighbors and lacks distance restrictions, but in CBL, the boundary query is constrained by a constant radius of 0.1. Can an identical distance constraint (0.1) be given to measure the effectiveness of “Multi-point Boundary Feature Constraints”?
3. For query distances r_m and point numbers k_m, whether there is any basis for comparison of results, e.g. “fixed r_m and adjusted k_m” or “fixed k_m and adjusted r_m”.
4. In Eq.(3), what does the meaning of notation f_p, f_j, f_v?
5. In line 501, the author divides the input into n non-overlapping ensembles. I do not see the detailed configuration of splits in section 4.1.
6. The method proposed by the author is similar to CBL in both its core idea and method framework. Please supplement the comparison of the results with CBL under the same network. (e.g. PointNext-B + CBL or RandLA-Net + MBSE)
7. In summary, lack of novelty. MBC looks more like a trick in terms of experimental results and method details. There is no effective basis for selecting Random multiple points. Compared to single selection, multi-point or multi-prototype is always the better choice.
8. As I understand, ConDaFormer [Ref1] can achieve 73.5% mIOU on S3DIS Area 5. What is the competitiveness of your method?
[Ref1] Duan, Lunhao, et al. "ConDaFormer: Disassembled Transformer with Local Structure Enhancement for 3D Point Cloud Understanding." Advances in Neural Information Processing Systems (2024).

Suggestion:
1. “enclave phenomenon” is a confusing word without citing the source, and if the definition proposed by the author himself needs to be explained or proved.
2. Losing visualization results with competitors.
paper_summary: paper_summary: This paper introduces a Multi-finenessBoundary Constraint (MBC) method to improve point cloud boundaries for semantic segmentation. By querying boundaries of different fineness and imposing feature constraints in these boundary areas, the distinction between boundaries and non-boundaries is enhanced. In addition, Shifted Ensemble-aware Perception (SEP) module is proposed, which establishes information interaction between points with minimal computational cost, captures the direct point-to-point long-range correlation in the ensemble and improves the segmentation performance of boundaries and non-boundaries. Experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1.The proposed MBC can enhance the feature distinction between boundaries and non-boundaries and improve segmentation decisions without additional networks.
2.The proposed SEP module captures direct point-to-point long-range dependencies in a collection while keeping the computational requirements low.
2.This manuscript designs many experiments to demonstrate the effectiveness of the proposed methods.
limitations: limitations: 1. In section 3.2.2, the manuscript states that “We take the inherent unordered nature of point clouds as prior knowledge”. Since the point cloud is unordered, how is the location information of the point cloud expressed?
2. Please provide more details about feature alignment, and what operations can achieve feature alignment?
3. EPM captures the long-term correlation between points by multiplying matrices. However, the result of the matrix product calculation represents the similarity of the two matrices, and it does not seem to be effectively related to long-range correlations simply through the matrix product operation.
4. Many hyperparameters are used in the proposed methods. However, the experimental section lacks analysis on hyperparameters, such as r, k or the number of ensembles. Do these hyperparameters affect algorithm stability, and how to carefully design a reasonable range?
paper_summary: paper_summary: This paper introduces a method called Multi-fineness Boundary Constraint (MBC) to tackle the challenge that the boundaries of point cloud are prone to mis-segmentation. By querying boundaries at various degrees of fineness and imposing feature constraints within these boundary areas, the discrimination between boundaries and non-boundaries is enhanced. Moreover, a new concept of point cloud space termed ensemble and a Shifted Ensemble-aware Perception (SEP) module is introduced to mitigate the problem that solely emphasizing boundaries may compromise the segmentation accuracy in broader non boundary regions. This module establishes information interactions between points with minimal computational cost, effectively capturing direct point-to-point long-range correlations within ensembles. The experimental results on multiple benchmarks validate the effectiveness and superiority of proposed method. However, considering the compared baselines are relatively weak and I am not familiar with the area, I recommend borderline accept score.
strength_weakness: strength_weakness: 1.This paper conducts the first systematic study on the impact of boundary queries on the semantic segmentation of point clouds. It introduces a multi-fineness boundary constraint approach, which explicitly enhances the feature discrimination between boundaries and non-boundaries, improving the accuracy of difficult-to-segment categories with high boundary proportions.

2.The experimental results demonstrate the effectiveness of the proposed method, including Multi-fineness Boundary Constraint (MBC) and Shifted Ensemble-aware Perception (SEP).

3.The paper is well-written with all key concepts are well described and easy to follow.
limitations: limitations: 1.The Comparative experiments are not convincing. Several state-of-the-art papers are ignored, such as PTv2 [1], Cluster3DSeg [2], PTv3 [3] and so on. Please compare with these strong baselines.

2.The experiments for adaptable of proposed method are not enough. Only PointNext are adopted to present results with proposed MBSE. It is better to combine with more strong baselines to make the adaptable ability more convincing.

[1]Point transformer v2: Grouped vector attention and partition-based pooling, NeurIPS 2022.

[2]Clustering based Point Cloud Representation Learning for 3D Analysis, ICCV 2023.

[3]Point Transformer V3: Simpler, Faster, Stronger, CVPR 2024.
paper_summary: paper_summary: This paper attempt to deal with 3D point cloud segmentation. It proposed Multi-fineness Boundary Constraint to enhance the feature discrimination in boundary areas. It also proposed shifted ensemble-aware perception to learn the long-range relationship. Results in S3DIS Area 5, ScanNet, and ShapeNetPart shows improvement over backbone.
strength_weakness: strength_weakness: 1. The motivation make sense to me to utilize contrastive loss in boundary area and learn the long-range relationship.
2. Experiments on various datasets shows the improvement over backbone PointNext.
limitations: limitations: 1. The definition of ensemble is not clear, whether it is similar to super-point or it is formed by collecting points randomly from a point cloud. There is no visual illustration. Points are grouped according to the feature in “Dynamic Graph CNN for Learning on Point Clouds”, is ensemble a better way to learn the long-range relationship?
2. The notation like A and H in Eq 7 and Eq 8 is not shown in the Figure 3 making it hard to understand the design of SEP.
3. The constrain in the boundary area is similar to the region similarity loss in “SceneEncoder: Scene-Aware Semantic Segmentation of Point Clouds with A Learnable Scene Descriptor”, whether such constrain works better.
4. Only PointNext is utilized as the backbone, it will be better to show the improvement on other backbone like “Kpconv: Flexible and deformable convolution for point clouds”or “4d spatio-temporal convnets: Minkowski convolutional neural networks” to prove the generalization ability.