Multi-scale Twin-attention for 3D Instance Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 03:53:46

Abstract: Recently, transformer-based techniques incorporating superpoints have become prevalent in 3D instance segmentation. However, they often encounter an over-segmentation problem, especially noticeable with large objects. Additionally, unreliable mask predictions stemming from superpoint mask prediction further compound this issue. To address these challenges, we propose a novel framework called MSTA3D. It leverages multi-scale feature representation and introduces twin-attention mechanisms to effectively capture them. Furthermore, MSTA3D integrates a box query with a box regularizer, offering a complementary spatial constraint alongside semantic queries. Experimental evaluations on ScanNetV2, ScanNet200, and S3DIS datasets demonstrate that our approach surpasses state-of-the-art 3D instance segmentation methods.

Main paper: 
 introduction: 
 Given 3D point clouds, 3D instance segmentation refers to a task that involves identifying and separating individual objects within a 3D scene, including detecting object boundaries and assigning a unique label to each identified object. Its significant role in computer vision has surged corresponding to the demand for 3D perception in various applications, such as augmented/virtual reality [16, 24] , autonomous driving [37] , robotics [35] , and indoor scanning [17] .In the literature, 3D instance segmentation methods are commonly categorized into four main approaches: proposal-based [5, 10, 12, 19, 26, 36] , grouping-based [3, 11, 18, 32, 33, 39] , kernelbased [8, 9, 11, 23, 34] , and transformer-based methods [14, 21, 29, 30] . Proposal-based methods begin by generating a 3D bounding box and then using it to segment into an instance mask. Groupingbased methods aggregate points into clusters using per-point features, such as semantic or geometric cues, and then segment instances based on these clusters. Kernel-based methods are similar to grouping-based techniques but treat each potential instance as a kernel for dynamic convolution. However, these methods require a high-quality proposal or clustering algorithm that heavily relies on per-point prediction, resulting in a significant demand for computational resources.To address these issues, transformer-based methods have been proposed, treating each potential instance as an instance query and refining it through a series of transformer decoder blocks. However, predicting instances from point clouds inherently presents substantial challenges due to their typically lacking clear structure, unlike the regular grid arrangement found in images. Moreover, managing large-scale input point clouds further requires costly computations and extensive memory resources. Thus, recent transformer approaches leverage superpoints, which roughly offer contextual relationships between object parts with reduced memory usage. Nevertheless, existing transformer-based approaches employing superpoints often suffer from performance degradation due to over-segmentation problems and the unreliability of mask predictions. These challenges are described as follows: (1) Existing methods are prone to over-segmentation, especially when dealing with large objects such as doors, curtains, bookshelves, and backgrounds. Additionally, converting labels from superpoints to points can introduce unreliability into the categorical grouping. (2) The learning process for point-wise classification and aggregation encounters challenges due to the sparse and irregular distribution of observed scene points.Hence, we propose a novel framework that leverages multi-scale superpoint features and simultaneously incorporates global/local spatial constraints. This framework is aimed at mitigating previously mentioned over-segmentation challenges and overcoming the limitations in point-wise classification. Specifically, to capture features at various scales, we generate superpoints at different scales, enabling effective feature representation of large objects and backgrounds as well as small objects. Correspondingly, we introduce a novel attention scheme, named twin-attention, to effectively fuse features at different scales. Moreover, we introduce the concept of box query, in addition to semantic query, which is trained using the proposed twin-attention decoder and refined by the spatial regularizer to enhance the confidence of mask predictions. Furthermore, we utilize this bounding box prediction to enhance the reliability of mask predictions during the inference phase.In summary, the contributions of this study are as follows:• We propose a twin-attention-based decoder for effectively representing multi-scale features to tackle over-segmentation challenges observed in large objects and backgrounds. • We introduce the notion of box query with box regularizer to provide complementary supervision without additional annotation requirements. This enforces spatial constraints for each instance during the query learning process, resulting in enhanced object localization and reduced background noise.• Extensive experiments are conducted on widely-used benchmark datasets, including ScanNetV2 [4] , ScanNet200 [28] , and S3DIS [1] , demonstrating the effectiveness of the proposed approach and achieving state-of-the-art results. 
related work: 
 Current approaches of instance segmentation on 3D point clouds can be classified into four categories: proposal-based, groupingbased, kernel-based, and transformer-based methods.Proposal-based methods. In the early stages of this approach, 3D-BoNet [36] was introduced, employing global characteristics derived from PointNet++ [26] to generate bounding boxes. These bounding boxes were then integrated with point features to produce instance masks. GICN [19] utilized a Gaussian distribution to estimate the center and size of each object instance for proposal prediction. Meanwhile, 3D-MPA [5] predicted instance centers and employed a graph convolutional network to group points around these centers, refining the features of proposals.Grouping-based methods. SSTNet [18] represented the scene comprehensively by constructing a superpoint tree and traversing it to merge nodes, thus creating instance masks. PointGroup [11] predicted the 3D displacement of each point from its instance's center and identified clusters from both the original and center-shifted points. HAIS [3] introduced a hierarchical clustering technique, allowing smaller clusters to be either eliminated or merged into larger ones. SoftGroup [32] permitted each point to be associated with multiple clusters representing diverse semantic classes to mitigate prediction inaccuracies. Softgroup++ [33] extended SoftGroup to reduce computation time and search space for the clustering process.Kernel-based methods. DyCo3D [8] employed a clustering algorithm from PointGroup [11] and lightweight 3D-UNet to generate kernels, while PointInst3D [9] opted for farthest-point sampling to produce kernels. DKNet [34] introduced candidate localization to yield more distinctive instance kernels. ISBNet [23] proposed a sampling-based instance-wise encoder to obtain a faster and more robust kernel for dynamic convolution.Transformer-based methods. SPFormer [30] and Mask3D [29] utilized the mask-attention to produce query instances based on voxel or superpoint features while MAFT [14] improved efficiency by using position query instead of mask-attention. QueryFormer [21] optimized query instances with the query initialization module and proposes an affiliated transformer decoder to eliminate noise background. 
box guided twin-attention decoder for multi-scale superpoint: 
 In this section, we provide an in-depth discussion of the design choices underlying the proposed model. The problem addressed in this paper and the overall structure devised to tackle this problem are elucidated in Section 3.1. Section 3.2 explains the backbone network and the reasoning behind incorporating multi-scale superpoint features. Section 3.3 elaborates on the novel twin attention mechanism, and Section 3.4 outlines the approach for constraining spatial regions for each instance. Finally, Section 3.5 provides a comprehensive explanation of each component of the loss function and the matching method employed.The goal of 3D instance segmentation in this paper is to predict precise point-wise boundaries of 𝑁 𝐼 individual object instances given 𝑁 ℎ superpoints generated from point clouds, denoted as p ∈ R 𝑁 ×6 . Each point cloud contains positional coordinates (𝑥, 𝑦, 𝑧) and color information (𝑟, 𝑔, 𝑏), where 𝑁 is the total number of point clouds. A binary mask represents each of these 𝑁 𝐼 instances, and a set of binary masks is collectively referred to M ∈ {0, 1} 𝑁 𝐼 ×𝑁 ℎ , where a value of 1 indicates the presence of objects, and 0 indicates their absence. Moreover, it is necessary to predict the semantic category associated with each instance, denoted as C ∈ Z 𝑁 𝐼 ×𝑁 𝐶 , where 𝑁 𝐶 is the total number of semantic categories.To tackle the problem, we propose a model consisting of three key components, as illustrated in Figure 2: (1) backbone network, (2) twin-attention decoder, and (3) box regularizer. First, the backbone network extracts multi-scale features S ℓ and S ℎ from the given inputs. These extracted features are then fed into the novel twin-attention decoder to generate instance proposals, represented as X, guided by newly introduced box queries. Additionally, employing the box regularizer, the prediction of bounding boxes B along with their corresponding scores B 𝑠 are utilized to confine instance regions. Finally, through the twin-attention-based decoder and box regularizer, the final outputs include instance masks M and corresponding labels C.As previously mentioned in Section 3.1, the backbone network, a 3D U-Net [6, 27] , takes voxelized point clouds as input and extracts point-wise features. The voxelization method utilized in this paper is based on the approach outlined in [14, 21, 30] .We choose to utilize superpoint-wise labels instead of point-wise labels to reduce training time and memory consumption (See Figure 5 ). Superpoints, as proposed by [15] , comprise multiple points sharing similar geometric properties and serve as a method to subsample point clouds. However, one potential drawback of using superpoints is the possibility of over-clustering scene [18, 21, 23, 30] , which depends on the chosen grouping size. Therefore, we extract features from superpoints at multiple scales to ensure that the decoder accurately identifies objects of various sizes during the learning process.For this purpose, we pre-compute superpoint at two different scales by adjusting the number of neighbors for each point, as described in [15] . Following this, a pooling layer is applied to aggregate information from the point-wise features into the multi-scale features. We then denote the outputs passing through this pooling layer as S ℓ ∈ R 𝑁 ℓ ×𝐷 𝑏 for the lower-scale features and S ℎ ∈ R 𝑁 ℎ ×𝐷 𝑏 for the higher-scale features. Here, 𝑁 ℓ and 𝑁 ℎ represent the number of low and high-scale superpoints, respectively, and 𝐷 𝑏 is the embedding dimension from the backbone.To leverage the advantages of multi-scale features 𝑆 ℓ and 𝑆 ℎ defined in Section 3.2, we introduce a novel decoder structure consisting of a series of twin attentions. The proposed twin-attention-based decoder is meticulously designed to integrate the multi-scale features. This decision is driven by the need to ensure that the proposed model effectively harnesses the diverse information present across different scales of the input features. The twin decoder incorporates a spatial regularizer for guiding instance delineations. Further details regarding this spatial regularization are provided in the following section. Region Constraint Instance Query. We propose the concept of a box query, along with a semantic query [14, 21, 30] , to guide the regions of instance masks for more accurate predictions. This guidance enables the model to concentrate more effectively on regions of interest, potentially reducing instance region ambiguity and improving segmentation accuracy. Concretely, we construct a set of learnable instance queries X 0 ∈ R 𝑁 𝑜 ×𝐷 𝑜 by concatenating the box queries with 6 elements, denoted as X 𝑏 ∈ R 𝑁 𝑜 ×6 and the semantic queries with 𝐷 𝑠 elements, denoted asX 𝑠 ∈ R 𝑁 𝑜 ×𝐷 𝑠 . The concatenation is expressed as X 0 = [X 𝑠 ; X 𝑏 ],where 𝑁 𝑜 denotes the number of queries (proposals), initially set randomly with 𝑁 𝑜 > 𝑁 𝐼 , and 𝐷 𝑜 is the total dimensionality of a query vector (i.e., 𝐷 𝑜 = 𝐷 𝑠 +6). Note that the number of instance proposals 𝑁 𝑜 is configured to be larger than the ground truth 𝑁 𝐼 . Consequently, the proposals selected as the final ones are those with the highest confidence. Let 𝜋 𝑐 (•) be the linear projection in the attention modules, where 𝑐 = {𝑞, 𝑣, 𝑘 } represents query, value, and key, respectively. In the cross-attention modules, the query matrix Q 𝐿 ∈ R 𝑁 𝑜 ×𝐷 𝑜 of the 𝐿-th block is obtained after linearly projecting instance queries, computed as 𝜋 𝐿 𝑞 X 𝐿-1 . Similarly, for low-scale feature S ℓ , the key matrix K 𝐿 ℓ ∈ R 𝑁 ℓ ×𝐷 𝑜 and value matrix V 𝐿 ℓ ∈ R 𝑁 ℓ ×𝐷 𝑜 are given by linearly projecting 𝜋 𝐿 𝑘 (S ℓ ) and 𝜋 𝐿 𝑣 (S ℓ ), respectively. Correspondingly, the key and value matrices of the high-scale features S ℎ are derived as K 𝐿 ℎ = 𝜋 𝐿 𝑘 (S ℎ ) ∈ R 𝑁 ℎ ×𝐷 𝑜 and V 𝐿 ℎ = 𝜋 𝐿 𝑣 (S ℎ ) ∈ R 𝑁 ℎ ×𝐷 𝑜 , respectively, using the same procedure. The proposed twin attention (TATT) is then computed on the shared set of instance queries and multi-scale features to attend to relevant information, as follows:EQUATIONEQUATIONwhere 𝜎 (•) denotes the softmax function, and 𝑑 is computed as the division of the dimension of the query (𝐷 𝑜 ) by the number of attention heads (8 in the experiments of this study). For high-scale attention in (1a), we employ the superpoint mask attention 𝐴 𝐿-1 ℎ ∈ R 𝑁 𝑜 ×𝑁 ℎ with a threshold 𝜏 = 0.5, similar to the works in [21, 29, 30] . However, we further enhance the mask attention prediction by incorporating the proposed box query and box regularizer, as elaborated in Section 3.4. Next, denoting the outputs of the cross-attention modules passing through a residual connection [7] and layer norm [2, 31] Y 𝐿 ℓ ∈ R 𝑁 𝑜 ×𝐷 𝑜 and Y 𝐿 ℎ ∈ R 𝑁 𝑜 ×𝐷 𝑜, the subsequent self-attention layer computes twin attention on a set of linearly projected queries, keys, and values, in a similar manner to (1a) and (1b), without applying mask attention. The computed outputs of the self-attention layer, denoted as Z 𝐿 ℓ ∈ R 𝑁 𝑜 ×𝐷 𝑜 and Z 𝐿 ℎ ∈ R 𝑁 𝑜 ×𝐷 𝑜 , are then fed into the multi-scale feature fusion module to abstract both low-scale and high-scale contextual information. To accomplish this, we utilize element-wise multiplication to merge the features extracted from inputs of different scales, followed by further aggregation using a feedforward layer (FFN), as follows:EQUATIONwhere ⊗ denotes element-wise multiplication. The output X 𝐿 ∈ R 𝑁 𝑜 ×𝐷 𝑜 of this feature fusion module for 𝐿-th block is then utilized for the inputs of the next twin-attention blocks and box regularizer.In the training, the proposed twin-attention blocks are sequentially trained using an iterative prediction strategy [30] . The output of the last decoder block (X 6 ) serves as the final instance proposals during inference.We have found that this introduced twin-attention-based decoder indeed enhances the performance of instance segmentation by leveraging multi-scale features. It enables the representation of objects of various sizes and captures the background details of the entire scene, effectively mitigating the possibility of oversegmentation (See Table 6 and Figure 6 ). Instance and Box Prediction: Given the fused feature output X 𝐿 of the 𝐿-th block, the instance prediction module computes the mask score M 𝑠 ∈ R 𝑁 𝑜 for each instance and its corresponding class C ∈ R 𝑁 𝑜 ×𝑁 𝐶 through multilayer perceptron (MLP) layers. These MLP layers employ linear activation for the mask score and softmax activation for classification. Moreover, to fully exploit the information from latent instance queries, we introduce two additional regressors with linear activation: B ∈ R 𝑁 𝑜 ×6 of instance box predictions and the corresponding box scores B 𝑠 ∈ R 𝑁 𝑜 . These spatial feature-associated regressors are introduced without requiring additional annotation efforts. The proposed framework autonomously derives box information from the instance labels. Specifically, we choose axis-aligned bounding boxes due to the simplicity of constructing ground truths B ∈ R 𝑁 𝐼 ×6 from existing instance annotations. Each bounding box is defined by six coordinates, representing the minimum and maximum 3D coordinates (i.e., [𝑥 𝑚𝑖𝑛 , 𝑦 𝑚𝑖𝑛 , 𝑧 𝑚𝑖𝑛 , 𝑥 𝑚𝑎𝑥 , 𝑦 𝑚𝑎𝑥 , 𝑧 𝑚𝑎𝑥 ] 𝑇 ) that enclose the instance. We observe that adopting this approach improves the overall performance of the proposed method because it utilizes bounding boxes as local spatial references for the box regularizer. These references provide contextual information about the local space occupied by each object relative to the entire scene during training, helping the model generate high-quality superpoint masks.In addition to incorporating box queries, we introduce the box regularizer to identify coarse object regions and constrain local spatiality. This regularization aims to enhance the representation of spatial latent features, utilizing box queries passed through the proposed twin-attention, as demonstrated in Table 6 .The proposed box regularizer takes the box predictions B for each instance from the twin-attention-based decoder as input, along with features comprising scene-wise semantic score F 𝑚 ∈ R 𝑁 ℎ ×𝐷 𝑠 for each superpoint and scene-wise box information F 𝑏 ∈ R 𝑁 ℎ ×6 . These scene-wise features are obtained by passing 𝑆 ℎ through multilayer perceptron layers with linear activation. By utilizing these features, the goal is to provide the box regularizer with comprehensive scene-associated features, ensuring effective representation of local instance regions. This approach guarantees the enhanced confidence of superpoint mask predictions by analyzing instance features in the context of the entire scene. Concretely, the box regularizer integrates instance-specific positional features b 𝑖 ∈ R 6 , which denotes 𝑖-th row of B, and scene-wise positional features F 𝑏 for all instances. By employing the broadcasting subtraction operator, we compute the relative positional difference R 𝑖 ∈ R 𝑁 ℎ ×6 for each instance box in relation to the entire scene, as follows:EQUATIONwhere ⊖ denotes a broadcasting subtraction operator, and { b 𝑖 } 𝑁 ℎ represents the stretched version of b 𝑖 by 𝑁 ℎ .To predict the binary mask in each twin-attention block, we concatenate these relative positional features R 𝑖 with the scenewise semantic features F 𝑚 ∈ R that simultaneously capture spatial and semantic information. The predicted binary masks are computed through batch matrix multiplication, as follows:EQUATIONIn ( 4), {R 𝑖 } 𝑁 𝑜 𝑖=1 constructs to a tensor in R 𝑁 𝑜 × (𝑁 ℎ ×6) , and {F 𝑚 } 𝑁 𝑜 represents the stretched version of {F 𝑚 } by 𝑁 𝑜 using broadcasting operators, resulting in the concatenated features in R 𝑁 𝑜 × (𝑁 ℎ ×𝐷 𝑜 ) (where 𝐷 𝑜 = 𝐷 𝑠 + 6); ⊙ denotes batch matrix multiplication, and 𝐿𝑖𝑛𝑒𝑎𝑟 (•) denotes a linear activation function.Intuitively, the relative positional variation R 𝑖 captures the prediction differences between the bounding box predicted from scenewise global dependency and the estimated box derived from instancewise local dependency. This suggests that mask prediction can be enhanced by abstracting both scene-wise features and individual instance-wise features. Moreover, this prediction of relative positional disparities can offer valuable feedback to the model, enabling it to refine its predictions based on discrepancies observed at different scales.Training: To train the proposed framework, we formulate the loss function L, as follows:L = 𝛽 𝑐𝑙𝑠 •L 𝑐𝑙𝑠 +𝛽 𝑚𝑎𝑠𝑘 •(L 𝑏𝑐𝑒 +L 𝑑𝑖𝑐𝑒 )+𝛽 𝑠 •L 𝑠 +𝛽 𝑏 •L 𝑏 +𝛽 𝑏𝑠 •L 𝑏𝑠 (5)where the coefficients 𝛽 𝑐𝑙𝑠 , 𝛽 𝑠 , 𝛽 𝑚𝑎𝑠𝑘 , 𝛽 𝑏𝑠 , and 𝛽 𝑏 determine the contribution of each loss function to the total loss function. The classification loss L 𝑐𝑙𝑠 is defined by the multi-class cross-entropy loss for object class categorization, and mask prediction utilizes a combination of binary cross-entropy L 𝑏𝑐𝑒 and dice loss L 𝑑𝑖𝑐𝑒 . Additionally, we use L2 loss L 𝑠 for predicting mask scores used for matching during training, L1 loss L 𝑏 for box coordinate prediction, and L2 loss L 𝑏𝑠 for box score prediction, as follows:EQUATIONEQUATIONEQUATION)where || • || 𝑝 denotes the vector 𝑝-norm, and || • || 𝑝,𝑞 denotes the entry-wise matrix 𝑝, 𝑞-norm; 𝟙 {𝑖𝑜𝑢 𝑚𝑠 } indicates whether the Intersection over Union (IoU) between mask prediction and assigned ground truth is higher than a threshold 𝜂 𝑚𝑠 ; and 𝟙 {𝑖𝑜𝑢 𝑏𝑠 } indicates whether IoU between box prediction and assigned ground truth is higher than a threshold 𝜂 𝑏𝑠 . In the experiments, we used 𝛽 𝑐𝑙𝑠 = 0.5 for class classification, 𝛽 𝑚𝑎𝑠𝑘 = 1 for mask prediction, 𝛽 𝑠 = 0.5 for mask score prediction, 𝛽 𝑏𝑠 = 0.5 for box score prediction, 𝛽 𝑏 = 1 for box prediction, and 𝜂 𝑚𝑠 = 𝜂 𝑏𝑠 = 0.5.To assign proposals to each ground truth instance, we formulate a pairwise matching cost 𝐶 𝑖 𝑗 to assess the similarity between the 𝑖-th proposal and the 𝑗-th ground truth. The matching cost 𝐶 𝑖 𝑗 is defined as [30] :EQUATIONwhere 𝑝 𝑖,𝑐 𝑗 denotes the 𝑐 𝑗 -th semantic category probability of the 𝑖-th proposal; 𝐶 𝑚𝑎𝑠𝑘 𝑖 𝑗 denotes superpoint mask matching score; 𝜆 𝑐𝑙𝑠 and 𝜆 𝑚𝑎𝑠𝑘 are the coefficients of each term respectively. In the experiments, we set 𝜆 𝑐𝑙𝑠 = 0.5 and 𝜆 𝑚𝑎𝑠𝑘 = 1. The superpoint mask matching cost 𝐶 𝑚𝑎𝑠𝑘 𝑖 𝑗 is calculated based on a binary crossentropy (BCE) and a dice loss with a Laplace smoothing [22] :EQUATIONwhere m 𝑖 is the 𝑖-th row of the superpoint mask prediction M, and m 𝑗 is the 𝑗-th row of the ground truth mask M. Additionally, we treat the task of matching proposals with ground truth as an optimal assignment problem. Hence, we employ the Hungarian algorithm [13] to identify the optimal solution for this task. Once completing the assignment, we categorize the proposals not assigned to ground truth as the "no instance" class.Inference. During inference, we compute instance confidence, considering that the number of proposals 𝑁 𝑜 can be greater than the number of ground truth instances 𝑁 𝐼 . The confidence score of 𝑖-th proposal is computed based on classification probability c 𝑖 , IoU box score 𝑏 𝑠,𝑖 , IoU score 𝑚 𝑠,𝑖 of corresponding superpoint mask. Follow [14, 30] , we utilize a superpoint mask score 𝑎 𝑖 by computing by averaging the probabilities of superpoint that have a probability greater than 0.5 in each superpoint mask prediction. The final confidence score for each instance is defined as{𝑠} 𝑁 𝑜 𝑖=1 = c 𝑖 • 𝑏 𝑠,𝑖 • 𝑚 𝑠,𝑖 • 𝑎 𝑖 𝑁 𝑜 𝑖=0and utilized for ranking predicted instances.In this work, we do not employ post-processing (non-maximum suppression or DBSCAN) as [21, 23, 29] to enhance inference speed. 
experimental results: 
 In this section, we introduce the datasets and evaluation metrics utilized to validate the effectiveness of the proposed model in Section 4.1, and implementation details are described in Section 4.2. Following this, performance comparisons are presented in Section 4.3, and analyses of additional studies on the model are demonstrated in Section 4.4.Dataset: To validate the effectiveness of the proposed framework, we conducted experiments on the ScanNetV2 dataset [4] , Scan-Net200 dataset [28] , and S3DIS dataset [1] .ScanNetV2 comprises 1613 scans, with 1201 scans for training, 312 for validation, and 100 for testing. It includes 18 object categories commonly used for 3D instance segmentation evaluation. ScanNet200 [28] extends ScanNetV2 [4] with fine-grained categories and includes 198 instances with an additional 2 semantic classes. The dataset division into training, validation, and testing sets mirrors the original ScanNetV2 dataset.S3DIS consists of 271 scenes across 6 different areas, with each area containing 13 categories. For the experiments in this paper, evaluations were conducted using datasets of Area 5. Evaluation Metric: The evaluation of instance segmentation typically relies on task-mean average precision (mAP), a widely-used metric that averages scores across various IoU thresholds ranging [34] 50.8 66.7 76.9 SoftGroup [32] 45.8 67.6 78.9 PBNet [38] 54.3 70.5 78.9 TD3D [12] 47.3 71.2 81.9 ISBNet [23] 54.5 73.1 82.5 Mask3D [29] 55.2 73.7 83.5 SPFormer [30] 56.3 73.9 82.9 QueryFormer [21] 56.5 74.2 83.3 MAFT [14] 58.4 75.6 84.5 MSTA3D (Ours) 58.4 77.0 85.4 from 50% to 95%, incremented by 5%. mAP 50 and mAP 25 represent the scores at IoU thresholds of 50% and 25%, respectively. Our evaluation of the ScanNetV2 and ScanNet200 datasets included mAP, mAP 50 , and mAP 25 metrics, and we also utilized mean precision (mPrec) and mean recall (mRec) in our analysis of the S3DIS dataset.We implemented the proposed model using the PyTorch deep learning framework [25] and conducted training with the AdamW optimizer [20] . The training was performed on a single A100 GPU with 40GB of memory. We used an initial learning rate of 0.0001, weight decay of 0.05, and batch size of 2, and employed a polynomial scheduler with a base of 0.9 for 512 epochs. For data augmentation, we applied random scaling, elastic distortion, random rotation, horizontal flipping around the z-axis, and random scaling for 𝑥, 𝑦, and 𝑧 coordinates. Additionally, we utilized noise and brightness augmentation for red, green, and blue colors. Both ScanNetV2 and ScanNet200 datasets were processed with a voxel size of 2𝑐𝑚. For S3DIS dataset, we used a voxel size of 5𝑐𝑚. The proposed model adopted the same backbone design described in [14, 30] , resulting in a feature map with 32 channels (𝐷 𝑏 = 32). During training, scenes were limited to a maximum of 250, 000 points. During inference, the entire scenes were inputted into the network without any cropping, and the top 100 instances were selected based on their highest scores.ScanNetV2. The instance segmentation results of the ScanNetV2 test and validation sets are presented in Tables 1 1 and 2 . Overall, the proposed model shows a significant improvement in mAP compared to previous studies, indicating its capability to capture intricate details and generate high-quality instance segmentation. In particular, compared to SPFormer [30] , the proposed model achieved a performance increase of +2.0 mAP, +2.5 mAP 50 , and +2.8 mAP 25 .Compared to Mask3D [29] , the improvement was +1.7 mAP, +1.5 mAP 50 , and +0.9 mAP 25 . Compared to MAFT [14] , the gains were +0.9 mAP 50 and +1.9 mAP 25 . Finally, compared to QueryFormer [21] , the increases were +0.8 mAP 50 and +0.5 mAP 25 on the hidden test set. On the validation set, the proposed model outperformed other methods across all three metrics: mAP, mAP 25 , and mAP 50 . Specifically, compared to QueryFormer [21] , the proposed model achieved a +1.9 mAP improvement, +2.8 mAP 50 improvement, and +2.1 mAP 25 improvement. Compared to MAFT [14] , the gains were +1.4 mAP 50 and +0.9 mAP 25 . As evidenced by the mAP scores for each class, the proposed model exhibited superior performance, particularly for relatively large objects such as beds or bookshelves. Notably, it established a considerable lead of over 10% mAP 25 specifically for the bookshelf category. This demonstrates that our proposed method effectively captures large objects to mitigate over-segmentation. ScanNet200. We also evaluated the proposed model on the Scan-Net200 dataset using the validation set. As shown in Table 3 , the results of the proposed model indicate a significant improvement compared to other methods. Specifically, the proposed model achieved an increase of +1 in mAP, +1.4 in mAP 50 and +0.5 in mAP 25 compared to SPFormer [30] . Especially, compared to TD3D [12] , our model achieved a +3.1 mAP and +0.4 mAP 50 improvement. Nevertheless, this dataset presents challenges for superpoint-wise labels due to its limited number of classes, indicating that point-wise methods will likely remain dominant. S3DIS. We conducted an evaluation of the proposed model on S3DIS using Area 5. As shown in Table 4 , the results demonstrate slight improvements compared to previous works such as MAFT or QueryFormer. In comparison to SPFormer, we achieved a notable improvement of +3.2 mAP 50 . Additionally, the proposed method outperformed other methods in terms of the mPrec metric.In this section, we conduct a thorough analysis to validate each component of the proposed approach, as outlined below.Multi-scale Feature Representation and Loss Functions. We assessed performance to further analyze the effectiveness of multiscale feature representations and the proposed loss functions by disabling one or more components. The results of this study are tabulated in Table 6 . The results when utilizing instance bounding boxes confirmed that the introduced box queries improved the accuracy of instance recognition. Furthermore, utilizing bounding box features with box scores enhanced recognition accuracy even further. The introduced box regularizer also improved the accuracy When comparing the performance of multi-scale feature representation against single-scale feature representation, the results demonstrate that multi-scale superpoints enhanced the feature representation at different scales. This finding is consistent with the results presented in Table 1 , where the proposed model outperformed relatively large objects while performing well in recognizing small objects. Number of Queries. We conducted a performance comparison of the proposed model using varying numbers of queries as shown in Table 5 . To ensure fairness in comparison, we exclusively evaluated models that utilize superpoint-wise information for both training and inference, namely SPFormer [30] and MAFT [14] . Remarkably, the proposed model achieved superior performance across all three metrics with only 150 queries, compared to SPFormer's 400 queries. Note that the results presented were obtained from the ScanNet benchmark on March25, 2024 2024-04-12 07:17. Page 6 of 1-9.CONCLUSIONIn this paper, we presented MSTA3D, a transformer-based method designed for 3D point cloud instance segmentation. To address the challenge of over-segmentation, particularly with background or large objects in the scene, we devised a multi-scale superpoint strategy. Furthermore, we introduced a twin-attention decoder to leverage both high-scale and low-scale superpoints simultaneously. This enhancement expands the model's capacity to capture features at various scales, thereby enabling better performance on large objects and reducing over-segmentation. In addition to the semantic query, we introduced the notion of a box query. This provides spatial context for generating high-quality instance proposals, assists the box regularizer in producing reliable instance masks, and contributes to box score regression, leading to significant performance improvements. Finally, we rigorously evaluated each of these components through extensive experiments. 
 
 Appendix:  


Decision: This paper presents a new transformer-based 3D instance segmentation framework with three main novelties:
1) Multi-scale point feature representation
2) Box regularization integrating scene-wise (global) and instance-wise (local) feature fusion
3) Efficient model architecture optimization

This paper is well written and clarity.  The experiments are extensive and solid. The experimental results did not completely surpass the best results, but they are competitive overall considering the trade-off between accuracy and computational complexity.

Reviews: review: paper_summary: paper_summary: This paper presents a new transformer-based 3D instance segmentation framework that utilizes multiscale superpoints for feature aggregation. A twin-attention mechanism takes input features from both high-scale superpoints and low-scale superpoints for better representation. Additionally, a box query is used to regularize semantic predictions. Experiments conducted on popular 3D indoor datasets (ScanNet-V2, ScanNet200, and S3DIS) show improved performance over existing methods.
strength_weakness: strength_weakness: •	The paper is generally well-written and easy to follow. The figures clearly illustrate the proposed framework.

•	The motivation for applying multiscale superpoints is straightforward and sensible.

•	The experiments are solid. This method achieves SOTA performance across different datasets. The ablation study demonstrates the effectiveness of each proposed component.
limitations: limitations: I have some questions need clarification.

•	The training of this network is conducted on a single 40G A100 GPU. How is the number of high-scale and low-scale superpoints controlled to avoid out-of-memory issues during training? As state in l.254, the authors adjust the number of neighbors to obtain different scales. The total number of superpoints can be very large in some scenarios (i.e. the theatre hall in S3DIS). There are some superpoint method can perform segmentation based on the given number of superpoints. Can the authors compare the pros and cons of these two types of superpoints generation strategies?

•	In Figure 5, the model complexity of the proposed method is lower than other methods. Where does this reduction likely come from? For example, is it due to a lighter backbone with fewer layers, a better-arranged network, or the elimination of some unnecessary components?

•	Can this network be trained from scratch, or does it rely on the pretrained backbone from other method?

•	There are a few relevant publications that are not cited. Since the total number of references in this paper is only 39, the authors might consider including more related works on grouping-based methods in Section 2.

[1] OccuSeg: Occupancy-aware 3D Instance Segmentation
[2] 3D Instance Segmentation via Multi-Task Metric Learning
[3] Learning Regional Purity for Instance Segmentation on 3D Point Clouds
paper_summary: paper_summary: This paper introduces a novel method for 3D instance segmentation that aims to address key issues such as the over-segmentation problem and unreliable mask predictions. To tackle these challenges, the paper proposes to combine multi-scale feature representation with a twin-attention mechanism and introduces a box regularizer.
strength_weakness: strength_weakness: - Extensive experiments have been conducted to demonstrate the effectiveness of the proposed method, achieving competitive results compared to existing baselines.
- This paper is well-written with a clear structure.
limitations: limitations: - As a key component of the method, further robustness experiments on the multi-scale feature representation could be provided. Specifically, it would be beneficial to explore how different hyperparameters, such as the number of neighbors mentioned in L254, affect the final performance of the method. Additionally, an investigation into the performance impacts of utilizing more levels of different scale features could provide deeper insights into the scalability of the approach.

- The paper shows inconsistent improvements, notably exhibiting inferior results (e.g., mAP50 and mRec) on the S3DIS dataset. Providing a detailed analysis of these results would be beneficial.
paper_summary: paper_summary: The author propose a twin-attention-based decoder for effectively representing multi-scale features to tackle over-segmentation challenges observed in large objects and backgrounds. Secondly, The author introduce the notion of box query with box regularizer to provide complementary supervision without additional annotation requirements. This enforces spatial constraints for each instance during the query learning process, resulting in enhanced object localization and reduced background noise.
strength_weakness: strength_weakness: The author has verified the feasibility of their method through a large number of experiments.
limitations: limitations: 1. Twin attention was proposed and widely applied a few years ago. The author did not cite twin attention literature in the introduction section. We suggest that the author cite twin attention's work in appropriate places.
2. In the section 4.3, the author did not state why the mAP metric of our method in Table 1 is lower than that of the MAFT and QueryFormer methods.
3. In table 4, what is the reason why the MSTA3D method is lower than other comparison methods in both mAP50 and mRec methods?
4. We believe that the method proposed in this article has limitations in terms of novelty.