ConDaFormer: Disassembled Transformer with Local Structure Enhancement for 3D Point Cloud Understanding
Authors: Lunhao Duan
Shanshan Zhao
Nan Xue
Mingming Gong
Gui-Song Xia
Dacheng Tao
Venue: NeurIPS 2023 poster (2023)
Keywords: Point Cloud
Transformer
3D Segmentation
3D object detection

Abstract: Transformers have been recently explored for 3D point cloud understanding with impressive progress achieved. A large number of points, over 0.1 million, make the global self-attention infeasible for point cloud data. Thus, most methods propose to apply the transformer in a local region, e.g., spherical or cubic window. However, it still contains a large number of Query-Key pairs, which requires high computational costs. In addition, previous methods usually learn the query, key, and value using a linear projection without modeling the local 3D geometric structure. In this paper, we attempt to reduce the costs and model the local geometry prior by developing a new transformer block, named ConDaFormer. Technically, ConDaFormer disassembles the cubic window into three orthogonal 2D planes, leading to fewer points when modeling the attention in a similar range. The disassembling operation is beneficial to enlarging the range of attention without increasing the computational complexity, but ignores some contexts. To provide a remedy, we develop a local structure enhancement strategy that introduces a depth-wise convolution before and after the attention. This scheme can also capture the local geometric information. Taking advantage of these designs, ConDaFormer captures both long-range contextual information and local priors. The effectiveness is demonstrated by experimental results on several 3D point cloud understanding benchmarks. Our code will be available.

ConDaFormer:
Disassembled Transformer with Local Structure Enhancement for 3D Point Cloud Understanding
Lunhao Duan*1
Shanshan Zhao
Nan Xue
Mingming Gong 3
Gui-Song Xiat Dacheng Tao
School of Computer Science, Wuhan University
JD Explore Academy
3 University of Melbourne
4 University of Sydney
Abstract
Transformers have been recently explored for 3D point cloud understanding with impressive progress achieved. A large number of points, over 0.1 million, make the global self-attention infeasible for point cloud data. Thus, most methods propose to apply the transformer in local region, e.g. spherical or cubic window. However, it still contains large number of Query-Key pairs, which require high computational costs. In addition, previous methods usually learn the query, key, and value using linear projection without modeling the local 3D geometric structure, In this paper, we attempt to reduce the costs and model the local geometry prior by developing a new transformer block, named ConDaFormer 4 Technically, ConDaFormer disas- sembles the cubic window into three orthogonal 2D planes, leading to fewer points when modeling the attention in similar range. The disassembling operation is beneficial enlarging the range of attention without increasing the computational complexity but ignores some contexts. To provide remedy, we develop a local structure enhancement strategy that introduces a depth-wise convolution before and after the attention. This scheme can also capture the local geometric informa- tion. Taking advantage of these designs, ConDaFormer captures both long-range contextual information and local priors. The effectiveness is demonstrated by experimental results on several 3D point cloud understanding benchmarks. Our code will be available.
Introduction
As fundamental vision task, 3D point cloud analysis [56, 58, 12, 97] has been studied extensively due to its critical role in various applications, such as roboties and augmented reality. The inherent irregularity and sparsity of point clouds make the standard convolution fail to extract the geometric features directly. As result, different strategies have been proposed for 3D point cloud data processing, which can be roughly divided into the voxel-based methods [12], the projection-based [7], and the point-based [58, 97]. Many methods achieve remarkable performance in point cloud analysis tasks, such point cloud segmentation [15] and 3D object detection [64]]
In recent years, transformers [70] have shown powerful capability of modeling long-ram long-range de- pendencies in the natural language processing community [17, 60]. Following ViT [19], trans- formers [18, 22, 13, 93, 44] also achieve competitive or better performance in comparison with the CNN architectures in many 2D vision tasks, such as object detection and semantic segmentation.
'Equal Contribution. This work was done when Lunhao Duan was a research intern at Explore Academy.
Correspondence Author.
Coon, Da, and Former indicate Convolution, Disassembled, and Transformer, respectively.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Regarding the 3D point cloud data, the graph structure makes it natural to apply transformers in hierarchical feature representation, and there are indeed many attempts [97, 80, 34] focusing on efficient designs. Since the 3D scene data usually contains more than 0.1 million points [15], it is impractical to apply self-attention globally. Instead, a local region obtained by KNN search [80] or spatially non-overlapping partition [34] is selected to perform the attention. Nevertheless, due to the 3D spatial structure, it still requires high computational costs.
Additionally, for 2D image data, convolution operation has been explored to assist the transformer in modeling local visual structures [22, 87], whereas such strategy is under-explored for 3D point cloud data. As the density varies across the point cloud, we cannot concatenate the raw pixels within a patch for query, key, and value learning as in 2D image data. As result, it is important to model the local geometric structure for better feature representation. In fact, in Point Transformer v2 [80] and Stratified Transformer [34], although grouping- -eriggttheehppee theennpollng used aggregate local information in the downsampling operation between adjacent encoding stages, we deem it is still inadequate for modeling local geometry prior.
Keeping the issues aforementioned in mind, in this paper we develop new transformer block for 3D point cloud under- standing, named ConDaFormer. Specifically, regarding the computational costs, we seek an alternative window partition strategy to replace the 3D cubic window, which can model the dependencies in similar range but involve fewer points. In- spired by CSnin [18], we disassemble the cubic window into three orthogonal 2D planes. In this way, for query point, only the points located in three directions are considered the key. Therefore, we can enlarge the attention range with negligible additional computational costs. The triplane-style strategy is also used by EG3D [5] in 3D generation, which aims at repre- senting the intermediate 3D volume into three planes to reduce memory consumption. As shown in Figure 1, we can observe that for similar distances, the multi-plane windows contain fewer Query-Key pairs. However, as shown in Figure 3 (a), the disassembling operation inevitably causes fewer contexts be modeled, To alleviate this issue, we apply a depth-wise sparse convolution (DSConv) before the attention to aggregate more contextual information for the query, key, and value as a complement. Moreover, after the attention, also propagate the updated feature of each point to its local neighbors with another DSConv operation. Such local structure enhancement based on sparse convolution not only compensates for the context loss caused by disassembling but also benefits local 3D geometric prior learning. Thanks to the efficient designs, our ConDaFormer is able to model both dependencies in a larger range and the local structure.
10
Cubic Window
8
Disassembled Window
OOK Pains
4
2-
0.16
0.24
0.32
0.48
Window Size (m)
Figure 1. Query-Key pairs E.S. Win- dow Size. We count the average ber Query-Key pairs for computing self-attention under different window sizes in scenes of S3DIS [1] with 80k points input. We can observe that the disassembled window partition can duce the number of pairs remarkably.
In total, the main contributions of this paper can be summarized as follows:
We propose novel disassembled window attention module for 3D point cloud understanding by disassembling the 3D window into three orthogonal planes for self-attention. This strategy effectively reduces computational overhead with negligible performance decreases.
To enhance the modeling of local features, we introduce depth-wise sparse convolution within the disassembled window attention module. This combination of self-attention and convolution provides comprehensive solution for capturing both long-range contextual information and local priors in 3D point cloud data.
Experiments show that method achieves state-of-the-att performance widely adopted large- scale 3D semantic segmentation benchmarks and comparable performance in 3D object detection tasks. Extensive ablation studies also verify the effectiveness of our proposed components.
2 Related Work
Vision transformers. Inspired by the remarkable success of transformer architecture in natural lan- guage processing [70], Vision Transformer (ViT) [19] has been proposed, which decomposes image into non-overlapping patches and leverages the multi-headed self-attention by regarding each patch as

---

a token. However, global self-attention across the entire image imposes a substantial computational burden and is not applicable for pixel-level image understanding tasks with high-resolution image input. To tackle this issue and extend VIT' to downstream tasks, Swin Transformer [44] proposes to confine self-attention to local non-overlapping windows and further introduces shift operations to foster information exchange between adjacent windows. To enlarge the receptive field, some tech- niques, such as Cenet [30], Axial-Attention [72], and CSWin Transformer [18], propose to perform self-attention in striped window. Such designs enable pixels to capture long-range dependencies while maintaining an acceptable computational cost. In addition, several recent approaches [78, 22, 87] attempt to integrate convolutions into transformer blocks, leveraging the benefits of local priors and
long-range contexts.
Point cloud understanding, Existing methods for point cloud understanding can be broadly classified into three categories: voxel-based, projection- -based, and point-based. Voxel-based methods [57, 12] divide the point cloud space into small voxels (i.e,, voxelization) and represent each voxel with
binary value indicating whether it occupied or not. Earlier approaches [8] directly apply
convolution all voxels, which causes a large computational effort. To improve computational efficiency, some methods [61, propose octrees representation for point cloud data to reduce the computational and memory costs. Recently, considering the sparsity property, many methods [12, 21] develop sparsity-aware convolution where only non-empty voxels are involved. For example, Choy et al. [12] represent the spatially sparse 3D data sparse tensors and develop open-source library that provides general operators (e.g., sparse convolution) for the sparse data. Projection- based methods [7. 65] project the point cloud data onto different 2D views. In this way, each view can be processed as a 2D image and the standard convolution can be used straightforwardly. Both voxel- based and projection-based methods might suffer from the loss of geometric information due the voxelization and projection operation. In comparison with them, point-based methods directly operate on the raw point cloud data. Following the pioncering work, PointNet [56] and PointNet++ [58], large number of approaches for local feature extraction have been proposed [69, 79, 76, 96, 39, 86, 28]. These methods utilize learnable function to transform each point into a high-dimensional feature embedding, and then employ various geometric operations, such as point-wiss feature fusion and feature pooling, to extract semantic information.
Point cloud transformers. As transformers have shown powerful capability in modeling long- range dependencies in 2D image data, it is natural to explore the application point cloud data with graph structure [97, 23, 50, 24, 20, 80, 45]. Although most transformer-based point cloud understanding networks can be classified into the point-based aforementioned, we review previous works especially here since they are very close to this paper. Point Cloud Transformer (PCT) [23] and Point Transformer v1 (PTv1) [97] make earlier attempts design transformer block for point cloud data. However, similar to ViT [19]. PCT performs global attention on all input points, leading to high computation and memory costs. In comparison, PTv1 achieves self-attention within local region for each query point, which largely reduces the computational effort and can be applied to scene-level point clouds. Following 1 Point Transformer V2 (PTv2) [80] further promotes the study in point cloud transformer and achieves better performance on typical point cloud analysis tasks with developed grouped vector attention and partition based pooling schemes. Instead of the overlapping ball regions, Stratified Transformer [34], inspired by Swin Transformer [44], splits the 3D space into non-overlapping 3D cubic windows to perform local self-attention within each window. To enlarge the receptive field and capture long-range contexts, it also develops a stratified strategy to select distant points as well as nearby neighbours as the keys. Considering the special structure of LiDAR data, SST [20] first voxelizes 3D LiDAR point cloud into sparse Bird's Eye View (BEV) pillars and then splits the space into oon-overlap 2D square windows to perform self-attention. To solve the problem of inconsistent token counts within each window in SST, FlatFormer [45] first re-orders these pillars in the BEV space and then partitions these pillars into groups of equal sizes to achieve parallel processing the GPU. CpT [32] and 3DCTN [48] explore the local structure encoding in the transformer block but only focus on small-scale data, like the CAD model. To provide more exhaustive progress on the point cloud transformer, here we further briefly discuss two recent works that were uploaded to the arXiv website shortly before the manuscript submission, Swin3D [92] and OctFormer [74]. Swin3D mainly focuses on the 3D backbone pre-training by constructing a 3D Swin transformer and pre-training it on a large-scale synthetic dataset. OctFormer aims to reduce the computation complexity of attention by developing octree-based transformers with the octree representation of point cloud data. Both of them and ours aim to explore the transformer in point cloud analysis in different ways.
3

---

Lincar
ConDaForner Block
ConDal Fommer Block
Con CanDDalrmmer Former Block
ML.P
Point Embeeding
Downsamping
Downsampling
Downsampling
Downsmpling
LSE module
Con DorDaarrrmer
LN
Disascembled Altn.
Point
ConibnFormer
Linear
LN
LSE module
x/1
x2
xNV3
XN4
(a) Overview
(b) The ConDaFormer Block
Figure (a) Framework overview. The entire network consists a Point Embedding layer and four stages each one which contains a downsampling layer and N, ConDaFormer blocks. (b) Structure of ConDaFormer block. Each ConDaFormer block consists of two Layer Normalization layers, an MLP, and a ConDaFormer module. The module contains disassembled window attention, two linear layers, and two local structure enhancement (LSE) modules before and after the attention.
3 Method
3.1 Preliminary
Regarding the point cloud understanding task, our goal is to develop a backbone network to generate high-level hierarchical features for a given point cloud, which can be further processed by specific task head, such as semantic segmentation and object detection. We focus on designing transformer block that can be used construct such a backbone. Specifically, let X (P R Nx3 F E R N NxC') represent the point cloud feed into the transformer block, where P represents the 3D position of N points and F denotes the features with channel number C. The transformer block processes X by exploiting the attention mechanism [70] and generates new features. We design the transformer block by disassembling the typical 3D window into multiple planes and introducing the local enhancement strategy, resulting in our ConDaFormer block. Based ConDaFormerr, we can construct networks for different point cloud analysis tasks, including point cloud semantic segmentation and 3D object detection. In the following, we detail CConDaFormes and the proposed networks for two high-level point cloud analysis tasks. Before that, we first give a brief introduction to the transformer based on the shifted 3D window, i.e., basic Swin3D block.
Overview.
Basic Swin3D block. VIT [19] achieves global self-attention 2D images by partitioning the space into non overlapplgpin patches tokens. However, for high-resolution images, global attention is high-cost due to the quadratic increase complexity with image size. To encourage the transformer to be more feasible for general vision tasks, Swin Transformer [44] develops hierarchical transformer block where self-attention applied within non-overlapping windows, and the adjacent windows are connected with a shifted window partitioning strategy. For the 3D point cloud data, we can naively extend the 2D window to the 3D version [34, 92]. Specifically, denote the points the t-th window with the size of S x S S and containing Ne points. The self-attention with H heads can be formulated as:
X; = (P R N. Fg E R.Nxc)
Q = Linear(Xx)) K. = Linear Linear(X,), V.A = (enear(X,),4r (I....H.. Attn (Xt) Soft((Q))(())) / /C/H) [1.... H], Swin3D(X,) Lineer([AAth) (Xt)
Attn .... @ Attn Attn#(X))),
where 1) h indicates the h-th head; 2) Qi, K and V represent query, key, and value, respectively;
3) Linear denotes the Linear function (Note that these linear functions do not share parameters); 4) Soft denotes the Softmax function; 5) Attn denotes the self-attention operation; 6) [- O represents the concatenation operation along the feature channel dimension. To achieve cross- winddw connection, the shift along three dimensions with displacements utilized in a successive transformer block with similar operations in Eq. 1.
Although the local window attention largely reduces the computational complexity in comparison with global attention, for each partitioned 3D window, the computation for self-attention is still

---

Cubic Window
Disussembled Windows
N
Concatenation
1xixi, C/2
1x1x1, C/2
BN. GELU
3x333,
X xC/3
X
XZ,N
The disassembled window attention
(b) LSE module
Two components the ConDaFormer block. (a) illustration of window partition: from cubic window to disassembled windows. The light blue areas indicate the regions involved in self-attention. (b) The detailed structure of the LSE module.
Figure 3.
high-cost. In this paper, we explore how to further reduce the computational costs with small changes in the range of attention. Specifically, we develop new block ConDaFormer, which enlarges the attention range with negligible additional computational costs.
3.2 ConDaFormer
In this part, we introduce our ConDaFormer by detailing the two main components: disassembling operation and pparse-convolution based local structure enhancement.
Cube to Multiple planes: Disassembling the 3D window. As analyzed before, the 3D window attention requires high computational costs for modeling long-range dependencies. To reduce the computational cost while maintaining the equivalent attention range, we propose to convert the cubic window three orthogonal 2D planes, as shown in Figure 3. Technically, we split the whole 3D space into three orthogonal planes, XY-plane, XZ-plane, and YZ-plane. For each 2D plane, we generate series of non-overlapping 2D windows, in each of which self-attention is performed. Taking an example of the XY -plane for illustration, we have the following equations:
Qny, = Linear(Xi), K = Linear(X,). Val,t Linear Linear(Xi)) h [I,..., H/3],
Altn (X.) = Sof(((Q 2q. )
CCC//CC//VVVV// 1,, / [ H/3].
Aytn [Attn (X,) @ Attn (X,) @
Attn / (X,))
where the notations have identical meanings to those in Eq. 1, except that Linear maps the channel number from C to C/3. such basic formulation, the attention is only related to the similarity between query Q and key K but does not contain the position bias which is important for self- /C/CH to encode
attention learning. We can add learned relative position bias (Q%,,))(K) (KK),,)² /T/ the position information as Swin Transformer does [44]. But, to better capture the position information with contextual information, scheme developed Stratified Transformer [34]. Specifically, we can re-write the computation of Attn my (X)) in Eq. as follows:
(Kr,
...T
adopt efficient context-based adaptive relative position encoding
Altn (X,) = So SSofsSoossss ((((')
(K (K), E + Kh ( CC///////VVVVV my,t + Eb)),
where Eg Ex. and E. are corresponding learnable position encodings for Qay Kay and Vag. For the other two planes, we exploit similar operations and share the same position encoding for three planes. We denote the attention as Attny= (X,) (Y (Y--pp -llaane) and Attna: (X,) (X Z Z-llane), respectively. Such a shared strategy makes the learning of position encoding more effective, which is demonstrated in the experiments. Then, we merge the three attention results obtain the final output of the disassembled transformer (DaFormer):
DaFormer Xt) Sinear((Attnnag(5 e Attmyz Atim (X+)))
Local structure enhancement with sparse convolution. As shown in Figure 3 (a), in comparison with the window, although our plane-based window easy to capture long-range dependency, ignores the contexts not located in the planes. To address this issue and model the local geometry prior, we propose to apply a sparse convolution operation to encode the input before the query, key, and value learning. As depicted in Figure 3 (b), the local structure enhancement module has
5

---

two branches: 1 x 1 X convolution and depth-wise sparse convolution to aggregate the local information. Mathematically, the local structure enhancement (LSE) module can be written as: (S, = (Linearr))) @
(5)
where BN, GELU, and DConu indicate the Batch Normalization, GELU activation function, and depth-wise sparse convolution (3 x 3 kernel), respectively. After the self-attention, we also apply another local enhancement operation to propagate the long-range contextual information learned by self-attention local neighbors to further improve the local structure. As result, the full operation consisting of two local enhancements and one self-attention can be written as with the notations above:
=
(6)
Formerr)r)))()) Linear [.. (I.SF.(AAttm @ Allmys (X)) Atin (XII).
ConDaFormer block structure. The overall architecture of our ConDaFormer block is depicted in Figure (b). Our ConDaFormer block comprises several essential components, including two local enhancement modules (before and after self-attention). a disassembled window self-attention module, a multi-layer perceptron (MLP), and residual connections [25].
3.3 Network Architecture
Point embedding. As mentioned by Stratified Transformer [34], the initial local feature aggregation important for point transformer networks. To address this, employ a sparse convolution layer to lift the input feature to a higher dimension C. Subsequently, we leverage ResBlock [25] to extract the initial point embedding. facilitating the representation learning process.
Downsampling. The downsampling module composed of linear layer, which serves to increase the channel dimension, and max pooling layer with a kernel size of 2 and stride of 2 to reduce the spatial dimension.
Network settings. Our network architecture is designed with four stages by default and each stage is characterized by specific channel configuration and a corresponding number of blocks. Specifically, the channel numbers for these stages are set to {C, 2C, 4C, 4C) and the corresponding block numbers {N1, N2, N3, 3 are 2, 6.2). And C is set to 96 and the head numbers H are set to 1/16 of the channel numbers in our experiments. For the task of semantic segmentation, following the methodology of Stratified Transformer [34], we employ a U-Net structure to gradually upsample the features from the four stages back to the original resolution. Additionally, we employ an MIP to perform point-wise prediction, enabling accurate semantic segmentation. In the case of object detection, we adopt FCAF3D [63] and CAGroup3D [71] as the baseline and replace the network backbone with our proposed architecture, leveraging its enhanced capabilities for improved object detection performance.
4 Experiments
To validate the effectiveness of our ConDaFormer, we conduct experiments on 3D semantic seg- mentation and 3D object detection tasks. We also perform extensive ablation studies to analyze each component in our ConDaFormer. Additional ablation results of window size and position encoding and more experiment results on outdoor perception tasks and object-level tasks are available in the appendix.
4.1 Semantic Segmentation
Datasets and metrics. For 3D semantic segmentation, we conduct comprehensive experiments on three benchmark datasets:
ScanNet v2 [15], S3DIS [1], and the eccently-itrooduce ScanNet200 [62].
The ScanNet v2 [15] dataset comprises collection of 1513 3D scans reconstructed from RGB-D frames. The dataset split into 1201 scans for training and 312 scans for validation. The input point cloud is obtained by sampling vertices from the reconstructed mesh and annotated with 20 semantic categories. In addition, we utilize the ScanNer200 [62] dataset, which shares the same input point cloud as ScanNet v2 but provides annotations for 200 fine-gras ine-grained semantic categories.

---

Table Semantic segmentation ScanNet v2.
Table Semantic segmentation S3DIS Area
Method
Input
Val
Test
Method
Input
OA
mAce
mloU
PointNct++ [58]
point
53.5
55.7
PointNet [56]
point
49.0
41.1
3DMV [16]
point
SegCloud [68]
point
57.4
48,9
PanopticFusion [53]
point
52.9
TanConv [67]
point
62.2
52.6
PointCNN [39]
point
45.8
PointCNN [9]
point
85,9
63.9
57.3
PointConv [79]
point
61.0
66.6
PointWeb [96]
point
87.0
66.6
60.3
JointPointBased [1]]
point
69.2
63.4
HPEIN [31]
point
87.2
68.3
61.9
PointASNL [90]
point
63.5
66.6
GACNet [73]
point
87.8
62.9
SegGCN [38]
point
58.9
PAT [91]
point
70.8
60.1
RandLA-NNt [28]
point
64.5
ParamConv [75]
point
67.0
58.3
KPConv [69]
point
69.2
68.6
SPGraph [35]
point
86.4
66.5
58.0
JSENet [29]
point
SegGCN 38
point 88.2
63.6
SparseConvNet [2]]
voxel
69.3
72.5
MinkUNet [12]
voxel
71.7
65.4
MinkUNet [12]
voxel
72.2
73.6
PAConv [86]
point
66.6
PTv1 [97]
point
70.6
KPConv [69]
point
72.8
67.1
PointNeXt [59]
point
71.5
71.2
PTvl [97]
point
90.8
76.5
70.4
[54]
voxel
72.1
PointNeXt [59]
point
90.6
70.5
LargeKemnel [8]
voxel
73.2
73.9
FPT [54]
voxel
77.3
70.1
Stratified [34]
point
74.3
73.7
Stratified [34]
point
91.5
78.1
72.0
PTv2 [80]
point
75.5
75.2
PTv2" [80]
point
91.6
78.0
72.6
ConDeFormer
point
75.1
74.7
ConDaFormer ConDaFormer'
point
91.6
78.4
72.6
ConDaFormer"
point
76.0
75.5
point
92.4
78.9
73.5
The S3DIS [I] dataset consists of 271 room scans from six areas. Following the conventions of previous methods, we reserve Area for validation while utilizing the remaining areas for training. The input point cloud of the S3DIS dataset is sampled from the surface of the reconstructed mesh and annotated with 13 semantic categories.
For evaluating the performance of our ConDaFormer on the ScanNet v2 and ScanNet200 dataset, we employ the widely adopted mean intersection over union (mloU) metric. In the case of the S3DIS dataset, we utilize three evaluation metrics: mloU, mean of class-wise accuracy (mAcc), and overall point-wise accuracy (OA).
Experimental Setup. For ScanNet v2 and ScanNet200, train for 900 epochs with voxel size and batch size set to 0.02m and 12 respectively. We utilize an Adam optimizer [47] with an initial learning rate of 0.006 and weight decay of 0.02. The learning rate decreases by factor of 10 after 540 and 720 epochs respectively. The initial window size set to 0. 1.66m and increases by factor of 2 after each downsampling. Following Point Transformer v2 [80] and Stratified Transformer [34], we use some data augmentation strategies, such as rotation, scale, jitter, and dropping color.
For S3DIS, we train for 3000 epochs with voxel size and batch size set to 0.04m and 8 respectively, We utilize an AdamW optimizer with an initial learning rate of 0.006 and a weight decay of 0.05. The learning rate decreases by a factor of 10 after 1800 and 2400 epochs respectively. The initial window size S is set to 0.32m and increases by a factor of 2 after each downsampling. The data augmentations are identical to those used in Stratified Transformer [34]]
Quantitative results. In experiments, compare the performance of ConDaFormer against state-of-the-arr methods 3D semantic segmentation. The results on ScanNet v2 and S3DIS datasets are shown in Table and Table respectively. Since Point Transformer v2 [80] employs the test- aumeenagmenttiion (TTA) strategy to improve the performance, we also adopt such strategy for fair comparison and the results with TTA strategy are marked with Clearly, on the validation of SeanNet v2, our ConDaFormer achieves the best performance and surpasses Point Transformer v2 [80] by 0.5% mloU (TTA also exploited). Similarly, on S3DIS, our ConDaFormer outperforms prior methods and achieves a new state-of-the-an performance of 73.5% mloU. Additionally, on the challenging ScanNet200 dataset, as shown in Table 3, our ConDaFormer still performs better than Point Transformer v2 [80] and substantially outperforms the rest of competitors that are pre-trained with additional data or a large vision-language model. The state-of-thh-ar performance on these datasets demonstrates the effectiveness of our ConDaFormer.

---

Input
Ground truth
Cubic
Stratified [34]
Ours
floor
wall
cabinct
bed
chair
sofa
table
door
window
bookshelf
picture
counter
desk
curtain
refrigerator
bathtub shower curtain
toilet
sink
other fumnituee
Figure 4. Visualization of semantic segmentation results on ScanNet v2.
Input
Ground truth
Cubic
Stratified [3]]
Ours
ceiling
floor
wall
beam
column
window
door
table
chair
sofa
bookease
board
clutter
Figure 5. Visualization of semantic segmentation results on S3DIS.
Qualitative results. Figure and Figure show the visualization comparison results of semantic seg- mentation on ScanNet v2 and S3DIS datasets, respectively. Compared to Stratified Transformer [34] and the baseline with cubic window attention, our ConDaFormer is able produce more accurate segmentation results for some categories, such as Wall, Window, and Sofa.
4.2 Object Detection
Data and metric. We evaluate our ConDaFormer on the SUN RGB-D [64] dataset annotated with oriented 3D bounding boxes of 37 categories. The dataset consists of about 10k indoor RGB-D scans, which are split into 5285 scans for training and 5050 scans for validation. Following prior works, we the average precision (mAP) under loU thresholds of 0.25 (mAP@0.25) and 0.5 (mAP@0.50) for evaluating the performance.
Experimental Setup. We implement the object detection network based on FCAF3D [63] and CAGroup3D [71], respectively. We replace the backbone with our ConDaFormer while keeping the detection head unchanged. The detection network built upon our ConDaFormer is trained with Adam W optimizer for epochs with batch size and weight decay set 32 and 0.01 respectively. The initial learning rate is set to 0.001 and decreases by factor of 10 after 12 and 20 epochs, respectively, The data augmentation techniques are the same as those FCAF3D and CAGroup3D.
Quantitative results. Following FCAF3D [63], we run ConDaFormer times and record the best and average (in bracket) performance reduce the impact caused by random sampling. As shown in Table 5, in comparison with FCAF3D and CAGroup3ID, our method achieves comparable performance and the gap between the best and average performance of method is smaller. In addition, ours has fewer parameters than FCAF3D (23 million v.s. 70 million).
4.3 Ablation Study
perform ablation experiments of semantic segmentation on the S3DIS dataset to demonstrate the effectiveness of each component of our proposed method.

---

Table Semantic segmentation ScanNer200.
Table 5. Detection results on SUN RGB-D.
Method
Input
mloU (%)
Method
mAP@0.25
mAP@0.5
CSC [26]
voxel
26.4
VoteNet [55]
57.7
LGround [62]
voxel
28.9
MLCCNet [84]
59.8
PTv2" [80]
point 31.9
3DETR [52]
59.1
32.7
H3DNet [95]
39.0
ConDaFormer"
point
32.3
BRNet [
61.1
43.7
HGNet [6]
61.6
Table 4. Ablation of window attention type.
VENet [83]
62.5
39.2
Type
mIoU (%)
Time (hours)
GroupFree [46]
63.0 (62.6)
45.2 (44.4)
Cubic
71.2
40.8
FCAF3D [63]
64.2 (63.8)
48.9 (48.2)
ConDaFormer
64.9 (64.7)
48.8 (48.5)
DaFormer Split
71.1
39.9
DaFormer w.o Share
69.9
24.1
CAGroup3D [71]
66.8 (66.4)
50.2 (49.5)
DaFormer
70.7
24.1
ConDaFormer
67.1 (66.8)
49.9 (49.5)
Table 6. Ablation of L.E.
Table 7. Ablation of window size.
Before
After
mloU
Window Size
0.16m
0.32m DaFormer
70.7
Method
Cubic
DaFormer
Ours
Cubic
Ours
72.0
Params (M)
7.4
7.0
8.2
8.0
8.4
71.5
Time (hours)
26.5
21.2
26.7
40.8
24.1
29.8
72.6
mloU (%)
69.9
69.7
71.7
71.2
70.7
72.6
Window disassembly. We begin by investigating the impact of disassembling the 3D cubic window into three orthogonal 2D planes. We first compare disassembled attention with the vanilla 3D window attention, which we refer to as "Cubic". We also further analyze our DaFormer by examining some involved settings. Specifically, 1) "DaFormer W.O Split": the number of channels equals that of the input for all three planes and the position embedding is also shared across the three planes, and the Concatenation in Eq. 4 replaced by Add. 2) "DaFormer 1/3 of that of the input for all three planes and the position embedding is not shared. Table 4 shows the results with the window size set 0.32m. The computational cost is evaluated with the GPU hours for training the network with a single NVIDIA Tesla V100 GPU. Compared to "Cubic" and "DaFormer w.o Split", although ours ("DaFormer") has a slightly lower score, the computational effort is significantly reduced. Notably, the comparison between "DaFormer w.o Share" and ours suggests that sharing the position embedding across different planes yields substantial performance gains. The potential reason may be the fact that when not shared across planes, the position embedding is not sufficiently trained due to the small number of neighborhood points within each plane.
Share": the number of channels equals
Local structure enhancement. Next, we explore the influence of the proposed local structure enhancement module, which aims to capture local geometric information and address the challenge of context ignorance introduced by window disassembly. The results presented in Table 6 indicate that incorporating local sparse convolution before or after self-ateention leads improved performance. And the best result is achieved when both of them are adopted. These comparisons highlight the significance of leveraging local prior information in the context of 3D semantic segmentation, emphasizing the importance of capturing detailed local geometric cues alongside global context.
Window size. Finally, we evaluate the effect of window size. Specifically, we evaluate the segmen- tation performance and computational costs of the vanilla cubic window attention, disassembled window attention, and our ConDaFormer (i.e., disassembled window attention with local structure enhancement) by setting the initial window size to O. 1.m and 0.32m. We have also experimented with window size of 0.08m on the cubic window and got 67.7% mIoU, sigmificantly worse than the result of 69.9% mIoU obtained with a window size of 16m. As shown in Table 7, compared to the cubic window attention, our proposed disassembled window attention ("DaFormer") significantly reduces the training time, especially when the window size is set to 0.32m, a relatively large window. Concomitantly. under both window sizes, the "DaFormer" brings slight degradation in segmentation performance. However, compared to the "Cubic" under the window size of 0.16m, the "DaFormer" under the window size of 0.32m not only improves the mloU by 0.8% but also has less computational cost. Moreover, compared to the "DaFormer",, our ConDaFormer significantly improves the seg- mentation performance with small increase in the number of parameters and training time, which verifies the effectiveness and efficiency of our model.

---

Conclusion
This paper focused the transformer for point cloud understanding. Aiming reducing the computational costs in the previous 3D non-overlapping window partition, we disassemble the 3D window into three mutually perpendicular 2D planes. In this way, the number of points decreases within similar attention range. However, the contexts also accordingly reduce for each query point. To alleviate this issue and also model the local geometry prior, we introduce the local structure enhancement strategy which inserts the sparse convolution operation before and after the self-attention. We evaluate our developed transformer block, i.e., ConDaFormer, for 3D point cloud semantic segmentation and 3D object detection. The experimental results can demonstrate the effectiveness.
Limitation: In our experiments, we find that larger attention window might cause drops in performance. For example, if we further enlarge the window size from 0.32m to 0.48m, the training loss drops from around 0.048 to around 0.045 while the mloU does not increase on the S3DIS dataset. As pointed out by LargeKernel [8] large convolutional kernel might cause difficulties optimizing proliferated parameters and leads to over-fitting. We guess that in our transformer block larger attention range requires more positional embeddings, which might cause similar issues. However, this phenomenon has not been fully explored in the 3D point cloud community. We think it would be worth studying in the future with efficient learning strategies, such as exploring self-supervised learning large-scale datascts.
Acknowledgements
This research is supported by the NSFC Grants under the contracts No. 62325111 and No. U22B201 1.
10

---

References
[1] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, loannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 1534 1543, 2016. 2, 6.
[2] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang. Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfu- sion: Robust lidar-camera fusion for 3d object detection with transformers. IEEE Conf. Comput. Vis.
Pattern Recog. pages 1090-1099, 2022. 16
[3] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic seene understanding of lidar sequences. In Int. Conf. Comput.
Vis., pages
2019. 15
[4] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 11621-11631, 2020. 15
[5] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Lconidas J Guibas, Jonathan Tremblay, Samch Khamis, et al. Efficient geometry-await aware 3d generative adversarial networks. In IEEE Conf. Comput. Vis, Pattern Recog,, pages 16123 16133, 2022. 2
[6] Jintai Chen, Biwen Lei, Qingyu Song, Haochao Ying, Danny Z Chen, and Jian Wu. A hicrarchical graph network for 3d object detection point clouds. In IEEE Conf. Comput. Vis. Pattern Recog,, pages
392-401, 2020. 9
[7] Xiaozhi Chen. Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In IEEE Conf. Comput. Vis. Puttern Recog., pages 1907-1915, 2017. 1,
[8] Yukang Chen, Jianhui Liu, Xiaojuan Qi. Xiangyu Zhang, Jian Sun, and Jiaya Jia. Scaling up kernels in 3d cnns. Adv. Neural Inform. Process. Syst., 2022. 7, 10, 15
[9] Bowen Cheng, Lu Sheng, Shaoshuai Shi, Ming Yang, and Dong Xu. Back-tracing representative points for onting-based 3d object detection in point clouds. In IEEF Conf. Comput. Vis. Pattern Recog., pages
8963-8977 2021. 9
[10] Ran Cheng. Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. 2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[1I] Hung-Yuehl Chiang, Yen-Liang Lin, Yueh-Cheng Liu, and Winston H Hsu. A unified point-based framework for 3d segmentation. In Int. Conf. 3D Vis., pages 155-163. IEEE, 2019.
[12] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3075-3084 2019. 3, [13] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Adv. Neural Inform. Process.
Syst., 34:9355 91655-93662222 2021.
[14] Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: Fast, uncertainty-ave semantic
segmentation of lidar point clouds. In International Symposium Visual Computing, 2020. 16
[15] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly- -nnottean 3d reconstructions of indoor scenes. In IEEE Conf. Comput. Vis. Pattern Recog..
2017. 2, 6
[16] Angela Dai and Matthias Nießner. 3dmv: Joint 3d- 3d-multi-view prediction for 3d semantic scene segmenta- tion. In Eur. Conf. Compur. Vis., pages 452-468, 2018.
[17] Jacob Devlin. Ming-Wes Ming-Weiiee Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec-
tional transformers for language understanding. arXiv preprint arXiv:1810.044: 04.005 2018. I
[18] Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu. Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In IEFF Conf. Comput. Vis. Pattern Recog., pages 12124-12134, 2022. 2,
[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192. 2020. 2,
16x16 words:
[20] Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang. Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In JEEE Conf. Comput. Vis. Pattern Recog,, pages 8458-8466 2022. 3
[21] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. semantic segmentation with submanifold sparse convolutional networks. In IEFE Conf. Comput. Vis. Pattern Recog., pages 9224-9232,
2018. 3, 7. 16
[22] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 22175-12188 2022. 1,2.
11

---

[23]
Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai- Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet:
Point cloud transformer. Comput. Vis. Media, 7:187-199. 3. 16
[24] Chenhang Hc, Ruihuang Li, Shuai Li, and Lei Zhang. Voxel set transformer: A set-to-set approach to 3d object detection from point clouds. In IEEE Conf. Comput. Vis. Puttern Recog,, pages 8417-8427, 2022.
[25] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conf. Comput. Vis. Pattern Recog. pages 770-778, 2016. 6
[26] Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene under- standing with contrastive scene contexts. In JEEE Conf. Comput. Vis. Pattern Recog,, pages 15587-15597,
2021. 9
[27] Yuenan Hou, Xinge Zhu, Yuexin Ma. Chen Change Loy, and Yikang Li. voiell knowledge distillation for lidar semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog,, 2022. 16
[28] Qingyong Hu, Bo Yang. Linhai Xie. Stefano Rosa, Yulan Guo. Zhihua Wang. Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. JEEE Conf. Comput.
Vis. Pattern Recog.. pages 11 1118-11117, 2020. 3, 7, 16
[29] Zeyu Hu, Mingmin Zhen, Xuyang Bai, Hongbo Fu, and Chiew-lan Tai. Jsenet: Joint semantic segmentation and edge detection network for 3d point clouds. Eur. Conj. Compus. Vis., pages 222-239, 2020.
[30] Zilong Huang, Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss- attention for semantic segmentation. In Int. Conf. Comput. Vis., pages 603- 2019. 3
[31] Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, and Jiaya Jia. Hierarchical point-edge interaction network for point cloud semantic segmentation. In Int. Conf. Comput. Vis., 2019. 7
[32] Chaitanya Kaul, Joshua Mitton, Hang Dai, and Roderick Murray-Smin -mitt Convolutional point transformer.
In Asian Conf. Comput. Vis., pages 303-319, 2022. 3
[33] Xin Lai, Yukang Chen, Fanbin Lu, Jianhui Liu, and Jiaya Jia. Spherical transformer for lidar-based 3d recognition. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 17545 -1555,, 2023. 16
[34] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. In IEEE Conf. Comput. Vis. Pattern Recog.. pages
8500-850 8509, 2022. 2, 3, 4, 5, 6, 7, 8, 15
[35] Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. IEEE Conf. Comput. Vis. Pattern Recog., 2018. 7
[36] Sanghyeok Lee, Minkyu Jeon, Injae Kim, Yunyang Xiong, and Hyunwoo J Kim. Sagemix: Saliency-guided
mixup for point clouds. In Adv. Neural Inform. Process. Syst., 2022. 15
[37] Huan Lei, Naveed Akhtar, and Ajmal Mian, Octree guided enn with spherical kernels for 3d point clouds.
In IEEE Conf. Comput. Vis. Pattern Recog,, pages 9631-9640, 2019. 3
[38] Huan Lei, Naveed Akhtar, and Ajmal Mian. Seggen: Efficient 3d point cloud segmentation with fuzzy
spherical kernel. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 11611-11622 2020. 7
[39] Yangyan Li, Rui Bu, Mingehao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on
x- transformed points, Ady. Neural Inform. Process. Syst. 31, 2018. 3, 7, 16
[40] Venice Erin Liong, Thi Ngoc Tho Nguyen, Sergi Widjaja, Dhananjai Sharma, and Zhuang Jie Chong- Amvnet: Assertion-basee multi-view fusion network for lidar semantic segmentation. arXiv:2012.0499 2020. 16
[41] Yongcheng Liu, Bin Fan, Gaofeng Meng, Jiwen Lu, Shiming Xiang, and Chunhong Pan. Densepoint: Learning densely contextual representation for efficient point cloud processing. In Int. Conf. Comput. Vis., pages 5239-5228,, 2019. 16
[42] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network
for point cloud analysis. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 8895-8904, 2019. 16
[43] Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look at local aggregation operators in point cloud analysis. In Eur. Conf. Compur. Vis., pages 326-342, 2020. 16
[44] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Int. Conf. Comput. Vis., 2021. 1, 3,
4, 5, 15
[45] Zhijian Liu, Xinyu Yang, Haotian Tang, Shang Yang, and Song Han. Flatformer: Flattened window
attention for efficient point cloud transformer. arXiv preprint ar arXi::2301.087333 2023. 3
[46] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers.
In Int. Conf. Comput. Vis., pages 2949-2958, 2021. 9
[47] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations. 7
[48] Dening La,, Qian Xie, Kyle Gao, Linlin Xu, and Jonathan Li. 3detm: 3d convoluiion network for point cloud classification. IEEF trans. Intell. Transp. Syst., 23(12))24854-48665 2022. 3
[49] Xu Ma. Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework. arXiv preprint arXiv:2202.0772 2022. 16
[50] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In Int. Conf. Comput. Vis., pages 11733317,,, 2021. 3
12

---

[51] Andres Milioto. Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. Rangenet++: Fast and accurate lidar semantic segmentation. In JEEE/RSJ international conference on intelligent robots and systems (IROS).
2019. 16
[52]
Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Inr. Conf. Comput, Vis., pages 2906-2917, 2021. 9
[53] Gaku Narita, Takashi Seno, Tomoya Ishikawa, and Yohsuke Kaji. Panopticfusion: Online volumetric
semantic mapping at the level of stuff and things. In IEEE Int. Conf. Intell. Rob. Syst., pages 4205-4212.
IEEE, 2019. 7
[54]
Chunghyun Park, Yoonwoo Jeong, Minsu Cho, and Jaesik Park. Fast point transformer. In IEEE Conf.
Comput. Vis. Pattern Recog., pages 16949-16958, 2022. 7
[55]
Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas, Deep hough voting for 3d object detection in point clouds. In Int. Conf. Comput. Vis., pages 9277- 2019. 9
[56] Charles R Qi, Hao Su, Kaichun Mo. and Leonidas J Guibas. Pointnet: Deep lcarning on point sets for 3d classification and segmentation. In IEEE Conf. Comput. Vis. Puttern Recog,, pages 652-660, 2017. 1, 1, 7,
16
[57] Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas Guibas. Volumetric and multi-view cans for object classification on 3d data. In IEEE Conf. Comput. Vis. Pattern Recog, pages 5648 5656, 2016. 3
[58] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hicrarchical feature learning on point sets in a metric space. Adv. Neural Inform. Process. Syst. 30, 2017. 1, 7, 16
[59] Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. Adv. Neural
Inform. Process. Syst., 35:23192-23204, 2022. 7, 16
[60]
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding
by generative pre pre-training. 2018. 1
[61] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high
resolutions. In IEEE Conf. Compur. Vis. Pattern Recog., 2017. 3
[62] David Rozenberszki, Or Litany, and Angela Dai. Laaguagg-grooudder indoor 3d semantic segmentation in the wild. In Eur. Conf. Comput. Vis., pages 125-141, 2022. 6,
[63]
Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Feaf3d: fully convolutional anchor-free 3d
object detection. In Eur. Conf. Comput. Vis., pages 477-493, 2022. 6, 8, 9
[64] Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding
benchmark suite. In IEEE Conf. Comput. Vis. Pottern Recog., pages 567-576 2015, 1, [65] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned neural networks for 3d shape recognition. In Int. Conf. Comput. Vis., pages 945-953, 2015. 3
Multi-view convolutional
[66]
Haotian Tang. Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient architectures with sparse point-voxel convolution. Eur. Conf. Comput. Vis., 2020.
[67]
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian- Yi Zhou. Tangent convolutions for dense prediction in 3d. In JEEE Conf. Comput. Vis. Pottern Recog.. 2018.
[68] Lyne Tchapmi, Christopher Choy, Iro Armeni, Jun Young Gwak, and Silvio Savarese. Segcloud: Semantic
segmentation of 3d point clouds. In Int. Conf. 3D Vis., 2017. 7
[69] Hugues Thomas, Charles Qi, Jean-Jaannnaaeee -mmmauem Deschaud, Beatriz Marcotegui. François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Int. Conf. Comput.
Vis., pages 641 1-6420, 2019. 3, 7, 16
[70] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Adv. Neural Inform. Process. Syst., 30, 2017. 1, 2, 4 [71] Haiyang Wang. Shaocong Dong. Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, Liwei Wang, et al. Cagroup3d: Class-aware grouping for 3d object detection point clouds. Adv. Neural Inform. Process.
Syst., 35:29975--29988, 2022. 6, 8, 9
[72] Huiyu Wang. Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Axial- deeplab: Stand-alone axial-attention for panoptic segmentation. In Eur. Conf. Comput. Vis., pages 108-126, 2020. 3
[73] Lei Wang. Yuchun Huang, Yaolin Hou, Shemman Zhang, and Jie Shan. Graph attention convolution for point cloud semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2019. 7
[74] Peng-Shuai Wang. Octformer: Detree-based transformers for point clouds. arXiv preprint
arXiv:2305.033055 2023. 3
[75] Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrowsky, and Raquel Urtasun. Deep parametric
continuous convolutional neural networks. In IEEF Conf. Compur. Vis. Pattern Recog,, 2018. 7
[76] Yue Wang, Yonghin Sun, Ziwei I Liu, Sanjay F. Sarma, Michael M. Bronstein, and Justin M. Solomon.
Dynamic graph enn for learning on point clouds. ACM Trans. Graph., 2019. 3.16
[77] 3D Warehouse. Sketchup. https:/ / /3dwarehcuse, sket com/ 2022. 15
[78] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions vision transformers. In Int. Conf. Comput. Vis., pages 22 31, 2021. 3
13

---

[79] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds.
In IEEE Conf. Comput. Vis. Pattern Recug,, pages 9621-9630. 2019. 7. 16
[80] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-based pooling. In Adv. Neural Inform. Process. Syst. 2022. 2, 3. 7, 15, 16 [81] Zhirong Wu, Shuran Song. Aditya Khosla, Fisher Yu, Linguang Zhang. Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In JEEE Conf. Comput. Vis. Pattern Recog..
pages 1912-1920, 2015. 3, 15
[82] Tiange Xiang. Chaoyi Zhang. Yang Song, Jianhui Yu, and Weidong Cai. Walk in the cloud: Learning curves for point clouds shape analysis. In Int. Conf. Comput. Vis., pages 915-924, 2021. 16
[83] Qian Xie, Yu-Kun Lai. Jing Wu, Zhoutao Wang, Dening Lu, Mingqiang Wei, and Jun Wang. Venet: Voting enhancement network for 3d object detection. In Int. Conf. Comput. Vis., pages 3712-3721, 2021. 9
[84] Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang. Yiming Zhang, Kai Xu, and Jun Wang. Mlevnet: Multi-level context votenet for 3d object detection. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 10447-10456, 2020.
[85] Jianyun Xu. Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun. and Shiliang Pu. Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation. In Int. Conf. Comput. Vis., 2021. 16 [86] Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. In JEEE Conf. Comput. Vis. Pattern Recog., pages 3173-3182,
2021. 16
[87] Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Adv. Nearal Inform. Process. Syst., 34, 2021. 2,
[88] Xu Yan. Jiantao Gao, Li. Ruimao Zhang, Zhen Li, Rui Huang, and Shuguang Cui. Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion. In AAAI, 2021.
16
[89] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng. Ruimao Zhang, Shuguang Cui, and Zhen Li. 2dpass: 2d priors assisted semantic segmentation on lidar point clouds. In Eur: Conf. Comput. Vis., 2022. 16 [90] Xu Yan. Chaoda Zheng. Zhen Li. Sheng Wang, and Shuguang Cui. Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling. In IEEE Conf. Comput. Vis. Pattern
Recog., pages 5589-5598, 2020. 7, 16
[91] Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point clouds with atte-attention and gumbel subset sampling. JEEE Conf. Comput. Vis. Puttern Recog..
2019. 7
[92] Yuqi Yang. Yuxiao Guo, Jianyu Xiong, Yang Liu, Hao Pan, Pengshuai Wang, Xin Tong, and Baining Guo. Swin3d: pretrained transformer backhone for 3d indoor scene understanding. arXiv preprint arXiv:2304.06900 2023. 3, 4
[93] Qiming Zhang. Yufei Xu, Jing Zhang, and Dacheng Tao. Vitaev2: Vision transformer advanced by
exploring inductive bias for image recogmition and bcyond. Int. J. Compur. Vis., pages 1-22, 2023. 1
[94] Yang Zhang. Zixiang Zhou, Philip David. Xiangyu Yue, Zerong Xi. Boqing Gong, and Hassan Foroosh. Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In IEEE Conf. Compur. Vis. Pattern Recog., 2020. 16
[95] Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primitives. In Eur. Conf. Comput. Vis., pages 311-329. Springer, 2020. 9
[96] Hengshuang Zhao, Li Jiang, Chi- ting Fu, and Jiaya Jia. Pointweb: Enhancing local neighborhood features for point cloud processing. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 5565-5573, 2019. 3,7
[97] Hengshuang Zhao, Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Int. Conf.
Comput. Vis., pages 16259-16268, 2021. 1, 3, 7, 16
[98] Haoran Zhou. Yidan Feng. Mingsheng Fang. Mingqiang Wei, Jing Qin, and Tong Lu. Adaptive graph convolution for point cloud amalysis. In Int. Conf. Comput. Vis., pages 4965-4974, 2021. 16
[99] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma. Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IEEE Conf. Comput. Vis. Pattern Recog,, 2021. 16
14

---

Appendix
In the appendix, we provide additional ablation results window size and position encoding in Sec. A, more experiment results on outdoor perception tasks and object-level tasks in Sec. B.
A Additional Ablation Results
Ablation of window size. As stated in the main paper, our proposed disassembled window attention offers a notable advantage over the vanilla 3D cubic window attention by significantly reducing computational effort, enabling the potential enlargement of the receptive field through increase in the window size. However, we also acknowledge a limitation of our ConDaFormer, which is the degradation in performance when utilizing larger attention window. In this section, present detailed ablation results to further investigate this issue. To assess the impact of window size on the performance of ConDaFormer, we conduct experiments using varying window sizes: (0.32m, 0,48m, 0.64m), and present the corresponding results in Table 8. It is worth noting that as the window size expands, the training loss (Loss,) gradually decreases, and the performance on the training set (mIoU,) steadily improves. However, contrary to expectations, the performance on the validation set experiences a decline, indicating the occurrence of over over-fitting. This phenomenon aligns with the observation made in LargeKernel [8] when increasing the convolutional kernel size. The introduction potentially resulting in optimization difficulties and leading to over-fitring. To address this issue, future research can explore techniques such as self-supervised or supervised pre-training on large- scale datasets. These approaches have shown promise in mitigating over-fitting and improving generalization performance. By leveraging such techniques, is possible to enhance the robustness of ConDaFormer and enable the utilization of larger attention windows without suffering from performance degradation.
larger attention window incorporates additional positional embeddings,
Ablation of position encoding. To enhance the modeling of crucial position information necessary for self-attentio learning, we employ the contextual relative position encoding (cRPE) scheme introduced by Stratified Transformer [34]. In this context, we compare the performance of cRPE with two alternative position encoding schemes: Swin [44], wherein the learned relative position bias directly added to the similarity between query and key. and PTv2 [80], which generates the position bias through an MLP that takes the relative position as input and subsequently adds it to the similarity between query and key, As shown in Table 9, cRPE outperforms the other schemes in two out of three metrics, indicating the significance of contextual features effectively capturing fine-grained position information.
Table 8. Ablation of window size.
Table Ablation of position encoding.
Window
mIoU
mAcc
OA
Loss;
mloU,
Method
mloU
mAcc
OA
0.32m
72.6
78.4
91.6
0.048
95.9
cRPE
70.7
76.9
90.6
0.48m
72.1
78.3
91.5
0.045
96.0
Swin
69.6
76.1
90.6
0.64m
71.6
78.4
91.4
0.044
96.1
PTv2
70.1
76.4
91.2
B Additional Quantitative Results
In this section, we present additional quantitative results four benchmark datasets: Model- Net40 [81] for shape classification, ShapeNet-Part [77, 36] for part segmentation, SemanticKITTI [3] for 3D semantic segmentation, and nuScenes [4] for both 3D semantic segmentation and 3D object detection.
Results on ModelNet40 and ShapeNet-Part, As shown in Table 10 and 11, we can find that for small-scale point cloud data, our method still achieves comparable or even better performance in comparison with Point Transformer vl, Point Transformer v2, and Stratified Transformer.
Results on SemanticKITTI and nuScenes. In summary, our ConDaFormer achieves comparable or slightly better performance on outdoor perception tasks compared to current methods, demonstrating ConDaFormer's potential usage as generalized 3D backbone. Specifically, for semantic segmen- tation, as shown in Table 12 and 13, ConDaFormer achieves 72.0% mloU and 79.9% mloU on the
15

---

test set of the SemanticKITTI and the validation set of nuScenes datasets, respectively. Obviously, ConDaFormer outperforms most prior methods designed specifically for outdoor LiDAR data, such as Cylinder31) [99] and RPVNet [85], and performs only slightly worse than 2DPASS [89]. which utilizes additional image information. For 3D object detection, we choose TransFusion-L. [2] as the baseline model and replace the backbone with our ConDaFormer. ConDaFormer achieves 68.5% NDS and 63.0% mAP on the validation set of the nuScenes dataset. The comparable performance on different datasets and tasks demonstrates the effectiveness of our ConDaFormer.
Table 10. Shape classification on ModelNet4O.
Table 11. Part segmentation on ShapeNet- Part.
Method
mAcc (%)
OA (%)
Methods
cls. mlou
ins. mloU
PointNet [56]
86.0
89.2
PointNet [56]
80.4
83.7
PointNet++ [58]
91.9
PointNet++ [58]
81.9
85.1
PoinCNNN [39]
88.1
92.5
PointCNN [39]
84.6
86.1
PointConv [79]
92.5
DGCNN [76]
82.3
85.1
KPConv [69]
92.9
RSCNN [42]
84.0
86.2
DGCNN [76]
90.2
92.9
KPConv [69]
85.1
86.4
RS-CNN [42]
92.9
PointConv [79]
82.8
85.7
PointASNL [90]
92.9
PointASNL [90]
86.1
DensePoint [4]]
93.2
PCT [23]
86.4
PosPool [43]
93.2
PAConv [86]
84.6
86.
PCT [2:]]
93.2
AdaptCony [98]
83.4
86.4
PA-DGC [86]
93.9
PTvl [97]
83.7
86.6
CurveNet [82]
94.2
CurveNet [82]
86.8
PTv1 [97]
90.6
93.7
PointMLP [49]
84.6
86.1
PTv2 [80]
91.6
94.2
PointNeXt [59]
85.2
87.0
ConDaFormer
90.8
94.0
ConDaFormer
84.6
86.8
Table 12. Semantic segmentation results on SemanticKITTI test set.
sdowolk
parking
huiding
biggode
metorryde
0fhe veth
iemiin
mncorrydis
Lance
pole
Italic kip
Method
mloU
Randi A-Ne [18]
55.9
90.5
74.0
61.8
24.5
39.7
94.2
43.9
29.8
32.2
39.1
83.8
63.6
68.6
48.4
47.4
9.4
60.4
51.0
50.7
KPCun [1]]
58.8
90.1
71.7
61.3
31.5
90.5
95.0
19.4
90.2
43.5
3
84-8
69.2
69.
61.5
61.6
11.8
64.2
56.4
47.4
Polathis [04]
54.3
90.8
74.4
61.7
21.7
90.0
93.8
22.9
40.3
30.1
28.5
84.0
65.5
67.8
43.2
40.2
5.6
61.3
51.8
57.5
JS3C-Nal [65]]
66.0
58.9
72.1
61.9
31.9
92.5
958
54.3
91.3
$2.9
46.0
84.5
69.3
67.9
69.5
65.4
39.9
708
60.7
68.7
SPVNAS
67.0
90.2
75.4
67.6
21.8
91.6
92.2
56.5
50.6
58.0
86
73.4
71.0
67.4
67.1
50.3
66.9
64.3
67.3
Cylinde3D [X]]
68.9
92.2
73.0
65.0
32.3
90.
92.1
67.6
63.8
58.5
85.6
72.5
69.8
73.7
68.2
48.0
66.5
62.4
6622
RPVNo [15]
70.3
91.4
80.7
70.3
33.3
93.5
97.6
44.2
68.4
68.7
61.1
86.5
75.1
71.3
75.9
74.4
43.4
73.1
64.
614
(AF)-$3Nax [10]]
70.8
PVXD
71.2
2DPASS
72.9
Ous
720
53.3
72.8
703
35.7
927
973
02.5
66.5
09.3
6488
849
738
09.7
73.6
73.1
631
70.7
05.
713
Table 13. Semantic segmentation results on nuScenes val set. denotes using rotation and translation testing-
time augmentations.
polestiion
traffic 0000
driveshle
other Bat
sidewalk
munimade
Regatasionn
bertler
bicycle
traler
track
somain
Methed
mloU
Ra53eN [S]]
65.5
66.0
21.3
77.2
80.9
30.2
66.8
09.6
32.1
54.2
72.3
94.1
66.6
63.5
70
83.
79.8
PolarNet [S[]]
71.0
74.7
28.2
85.3
90.9
35.1
77.5
71.3
58.8
57.4
76.1
96.5
71.1
74.7
74.0
87.3
85.7
Salsanest [14]
72.2
9488
34.1
85.9
83.4
42.2
72.4
72.2
63.1
61.3
76.5
96.0
70.8
71.2
86.7
84.4
AMVNet [40]
86.4
75.3
72.3
97.4
Cylinder1D [99]
76.1
76.4
40.3
91.2
93.8
51.3
78.0
78.9
64.9
62.1
84.4
96.8
71.6
76.4
75.4
0I.5
87.4
PVKD [:7]
76.0
76.2
40.0
90.2
94.0
50.9
77.4
78.8
64.7
62.0
34.1
96.6
714
76.4
76.3
90.3
86.9
RPVNet [15]
77.5
78.2
43.4
92.7
93.2
49.0
80.5
66.0
66.9
84.0
96.9
73.5
76.0
90.6
83.9
2DPASS [89]
79.4
78.8
49.6
95.6
93.6
60.0
84.1
82.2
67.5
72.6
88.1
96.8
72.8
76.2
76.5
89.4
87.2
FomereFomerr pagt
79.5
78.7
46.7
95.2
93.7
54.0
68.0
742
862
97.2
74.
76.3
75.8
91.4
89.7
Our
78.2
78.1
44.6
94.9
92.2
54.
84.9
80.7
66.0
70.1
85.8
96.9
73.4
75.7
74.88
90.7
89.0
94.1
97.1
74.8
91.2
Table 14. Object detection results nuScenes val set.
Method
NDS
mAP
Sparse Conv [2]]
68.48
63.07
Ours
68.54
62.95
16


Reviews:
[{"review_id": "j8vKkLaGMW", "reviewer": "Reviewer_BFnK", "summary": "Recent advancements in 3D point cloud understanding have explored the use of Transformers, resulting in notable progress. However, the computational demands of applying global self-attention to large point cloud datasets, which contain over 0.1 million points, present a significant challenge. To mitigate this issue, researchers have proposed using Transformers within local regions, such as spherical or cubic windows. Nevertheless, these approaches still involve a considerable number of Query-Key pairs, leading to high computational costs. Moreover, previous methods often neglect the local 3D geometric structure by employing linear projections to learn the query, key, and value.\nIn this paper, a new transformer block named ConDaFormer is introduced to address these challenges while also considering the local geometry prior. ConDaFormer decomposes the cubic window into three orthogonal 2D planes, reducing the number of points involved in attention modeling within a similar range. Although this disassembling operation sacrifices some contextual information, a local structure enhancement strategy is implemented using depth-wise convolutions before and after the attention step. This strategy effectively captures local geometric information.\nBy leveraging these innovative designs, ConDaFormer is capable of capturing both long-range contextual information and local priors. Experimental results on various benchmarks for 3D point cloud understanding demonstrate the effectiveness of ConDaFormer.", "questions": "I'm positive about this paper. I really like the idea to disassemble 3D cubic window into three orthogonal planes for self-attention. It can reduce the computational cost. However, I still have some questions about the experimental results. Please see the Weaknesses and respond to those questions. Thank you.", "limitations": "This paper still have some limitations when using larger attention window size. If the authors enlarge the window size from 0.32m to 0.48m, the training loss drops from around 0.52 to around 0.47 while the mIoU does not increase on the S3DIS dataset.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The authors propose a novel disassembled window attention module for 3D point cloud semantic segmentation by disassembling the 3D window into three orthogonal planes for self-attention. This strategy effectively reduces computational overhead with negligible performance decrease.\n(2) To enhance the modeling of local features, the authors introduce depth-wise sparse convolution within the disassembled window attention module. This combination of self-attention and convolution provides a comprehensive solution for capturing both long-range contextual information and local priors in 3D point cloud data.\n(3) Experiments show that our method achieves state-of-the-art performance on widely adopted large scale 3D semantic segmentation benchmarks and comparable performance in 3D object detection task. Extensive ablation studies also verify the effectiveness of the proposed components.", "weaknesses": "(1) In Table 1, why is there no results on test set for ConDaFormer?\n(2) In Table 5, you said \"in comparison with FCAF3D, our method achieves comparable performance but performs more steadily.\". What do you mean by more steadily? Do you have any experimental results to support that?\n(3) In Table 7, why don't try the window size smaller than 0.16m? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and constructive comments. We are encouraged by your positive comments on our method and experiments (novelty and effectiveness). In the following, we address your concerns carefully.\n\n**Q1: Why is there no result on test set for ConDaFormer in Table 1**\n\nA: In Table 1, we provided the results of ConDaFormer without the test-time augmentation (TTA) on both validation and test sets of ScanNet but reported the performance with TTA (marked by a Star) only on the validation set. We would like to apologize for that. Here we explain the reason and hope to clarify this.\n\nFollowing most of previous works, we first did not use TTA technique for model evaluation and reported the performance on both validation and test sets of ScanNet (stated in L260-263). However, as Point Transformer v2 employs TTA, for fair a comparison, we further evaluated our method with TTA on the validation set. Unfortunately, due to the submission rule of ScanNet benchmark, we only reported the performance on the validation set (marked by a Star) and submitted the results on the test set to the benchmark server after the paper submission deadline. We **got 75.5% mIoU, surpassing Point Transformer v2's 75.2% mIoU**. We would like to apologize for that again and will include the test set result in the revised paper.\n\n---\n\n**Q2: Comparison with FCAF3D**\n\nA: Following previous works, we ran ConDaFormer 5 times to reduce the impact caused by random sampling and provided the best and average (in bracket) performance in Table 5 (as we stated in L287-288). \n\nThe reason that we think our model performs more steadily than FCAF3D is that the best score and average score on mAP\\@0.25 and mAP\\@0.50 are very close (64.9 v.s. 64.7, 48.8 v.s. 48.5), while FCAF3D got (64.2 v.s. 63.8) and (48.9 v.s. 48.2). \n\nThank you for pointing out this issue. We will make this explanation clearer in the revised paper.\n\n---\n\n**Q3: In Table 7, why don't try the window size smaller than 0.16m?**\n\nA: If the voxel size is set to 0.04m, when the window size is less than 0.16m, we think the area of window attention is too small, resulting in a limited receptive field of the network. We have experimented with a window size of 0.08m on the Cubic window and got 67.7% mIoU, significantly worse than the result of 69.9% mIoU obtained with a window size of 0.16m. Thank you for pointing out this issue. We will include this information in the revised version to provide a comprehensive understanding of the window size's impact on performance.\n\n---\n\nWe hope our response adequately addresses your concerns. If you still have any questions, we are looking forward to hearing them."}, {"review_id": "okRCJ5iTKj", "reviewer": "Reviewer_JCcK", "summary": "In this paper, CondaFormer, an innovative 3D transformer architecture, is presented. It ingeniously dissects the cubic window into three orthogonal 2D planes and incorporates a local structure enhancement strategy that uses depth-wise convolutions to capture local geometric information. Through rigorous experiments on point cloud semantic segmentation and 3D detection, CondaFormer's efficacy is demonstrably validated.", "questions": "see the strengths and weaknesses.", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The rationale behind dissecting the cubic window into tri-planes is well-articulated and has a solid foundation. This approach considerably reduces computational cost by limiting query-key pairs.\n2. CondaFormer showcases remarkable improvements in performance in the context of semantic segmentation, as evidenced by the results.\n3. A series of comprehensive ablation studies are conducted, providing evidence for the effectiveness of the window disassembly design, local structure enhancement, and the impact of hyper-parameter choices.\n4. The paper is well-structured and clearly presented, making it accessible and easy to follow.", "weaknesses": "1. The overall concept, though practical, does not break new ground in terms of novelty. The disassembly of 3D windows into 2D planes can be seen as a straightforward adaptation of the Axial Transformer[1], which similarly disassembles 2D windows into 1D axis attention. \n([1] Ho J, Kalchbrenner N, Weissenborn D, et al. Axial attention in multidimensional transformers[J]. arXiv preprint arXiv:1912.12180, 2019.)\n2. The performance in 3D object detection leaves room for improvement, as CondaFormer does not exhibit a substantial edge over the baseline FCAF3D.\n3. It is recommended that the authors further validate the model's performance and robustness by conducting additional experiments on outdoor perception tasks such as 3D object detection or segmentation on datasets like KITTI or Waymo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and constructive comments. We are encouraged by your positive comments on the rationale behind our method, the improvement in segmentation, ablations (comprehensive), and presentation.\nIn the following, we address all your concerns carefully.\n\n**Q1: The overall concept, though practical, does not break new ground in terms of novelty.**\n\nA: We first sincerely appreciate your recognition of the inspiration behind the disassembly of 3D windows into 2D planes from 2D window attention. \n\nRegarding your concern about the disassembly operation, we agree with you that some similar strategies have been explored by previous works on 2D transformers. As mentioned in L52 of the submission, we also indicated that our idea is indeed inspired by CSWin. In L87-89, we also list several works using similar window partitioning strategies, including Axial-Attention.\n\nIn this paper, we explore such a strategy for transformer in 3D point cloud understanding to enlarge the range of attention without increasing the computational costs. But, we would like to explain that our work **does not just bring the design in CSWin or Axial-Attention into the 3D transformer**. \n\n**As shown in Table 7, while reducing the computational costs, such a straightforward adaptation inevitably causes performance degradation due to fewer contexts to be modeled. \nWe thus introduce the depth-wise sparse convolution within the disassembled window attention module to enhance the local structure representation. Moreover, we also make detailed ablations to analyze the components involved in the block.**\n\nIn fact, applying strategies inspired by some advanced techniques in 2D to the 3D community has been explored. For example, Stratified Transformer motivated by the SWin Transformer extends the non-overlapped 2D window partitioning strategy to the transformer in the 3D point cloud. To capture long-range contexts, it introduces a stratified sampling strategy. This method has become an important backbone for point cloud understanding. We hope our study in the 3D window disassembly strategy could motivate more explorations for 3D point cloud transformers.\n\n---\n\n**Q2: The performance in 3D object detection leaves room for improvement.**\n\nA: Thank you for pointing out this issue. To validate the effectiveness of our ConDaFormer as a generalized 3D backbone, we selected a simple 3D object detection method, FCAF3D, as our baseline and did not make elaborate adjustments to our method to improve the performance. To address your concern, we further take the SoTA detector, CAGroup3D  (Table 5 in the sumission), as our baseline and conducted experiments on SUN RGB-D dataset. The best and average (in bracket) performance are listed below. As a generalized 3D backbone, ConDaFormer's potential usage is demonstrated by its comparable performance to FCAF3D and CAGroup3D. We will add this new comparison in the final version to further show the capacity of our method.\n\nDetection results on SUN RGB-D: \n|   Method | mAP\\@0.25 | mAP\\@0.50 |\n|----------|:----------:|:-------:|\n| CAGroup3D    |\t66.8 (66.4) | 50.2 (49.5) |\n| ConDaFormer  |\t67.1 (66.8) | 49.9 (49.5) |\n\n---\n\n**Q3: Additional experiments on outdoor perception tasks.**\n\nA: Thank you for your constructive suggestions. In response to your recommendation, we conducted experiments on SemanticKitti for 3D semantic segmentation and nuScenes for both 3D semantic segmentation and 3D object detection. The detailed results are listed below. \n\nSpecifically, for 3D semantic segmentation, ConDaFormer achieves 72.0% mIoU and 79.9% mIoU on the test set of the SemanticKitti and the validation set of nuScenes datasets, respectively. Obviously, ConDaFormer outperforms most prior methods designed specifically for outdoor LiDAR data, such as Cylinder3D [3] and RPVNet [4], and performs only slightly worse than 2DPASS [5], which utilizes additional image information.\n\nFor 3D object detection, we choose TransFusion-L [7] as the baseline model and replace the backbone with our ConDaFormer. ConDaFormer achieves 68.5% NDS and 63.0% mAP on the validation set of the nuScenes dataset. \n\nIt is worth noting that for point cloud understanding backbone design (not aimed at outdoor LiDAR data), previous works (such as Stratified Transformer and Point Transformer) did not conduct experiments on outdoor LiDAR data. \n\nSemantic segmentation results on SemanticKITTI test set: \n|   Method | mIoU |\n|:----------|:----------:|\n| KPConv [1]       | 58.8 |\n| SPVNAS [2]          | 67.0 |\n| Cylinder3D [3]      | 68.9 |\n| RPVNet [4]          | 70.3 |\n| 2DPASS [5]          | 72.9 |\n| ConDaFormer      | 72.0 |\n\nSemantic segmentation results on nuScenes val set: \n|   Method | mIoU |\n|:----------|:----------:|\n| Cylinder3D [3]      | 76.1 |\n| PVKD [6]            | 76.0 |\n| RPVNet [4]          | 77.6 |\n| 2DPASS [5]          | 79.4 |\n| ConDaFormer      | 79.9 |\n\nObject detection results on nuScenes val set: \n|   Method | NDS | mAP |\n|:----------|:----------:|:----------:|\n| TransFusion-L [7]  | 68.48  | 63.07 |\n| ConDaFormer    | 68.54  | 62.95 |\n\n[1] Thomas, et al. \"Kpconv: Flexible and deformable convolution for point clouds.\" ICCV 2019.\n\n[2] Tang, et al. \"Searching efficient 3d architectures with sparse point-voxel convolution.\" ECCV 2020.\n\n[3] Zhu, et al. \"Cylindrical and asymmetrical 3d convolution networks for lidar segmentation.\" CVPR 2021.\n\n[4] Xu, et al. \"Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation.\" ICCV 2021.\n\n[5] Yan, et al. \"2dpass: 2d priors assisted semantic segmentation on lidar point clouds.\" ECCV 2022.\n\n[6] Hou, et al. \"Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation.\" CVPR 2022.\n\n[7] Bai, et al. \"TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers.\" CVPR 2022.\n\n---\n\nWe hope our response adequately addresses your concerns. If you still have any questions, we are looking forward to hearing them.\n"}, {"review_id": "Ea86lEWKTe", "reviewer": "Reviewer_teZK", "summary": "This paper studies point cloud understanding. They propose ConDaFormer, a novel Transformer architecture for 3D point cloud. Specifically, ConDaFormer disassembles the cubic window into three orthogonal 2D planes, leading to fewer points when modeling the attention in a similar range. Together with local sparse convolutions, ConDaFormer can capture both long-range contextual information and local priors. They evaluate their method on point cloud detection and segmentation datasets and achieve good performance.\n", "questions": "Please refer to the weaknesses.", "limitations": "Please refer to the weaknesses.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Their method has substantial improvements over the existing point cloud Transformers.\n\n- The proposed architecture is simple and effective. The method has sufficient novelty.\n\n- They evaluate their approach on widely-used point cloud datasets and achieve satisfactory results.\n\n- Paper writing is good and easy to follow.", "weaknesses": "- The proposed model has some similarities with FlatFormer [1], which groups the patches by axis sorting. The axis sorting is similar to disassembled window attention proposed in this paper. The authors need to clarify the difference with the existing works.\n\n- The proposed model failed to outperform the SoTA detectors on SUN-RGBD. \n\n[1] Liu et al. FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and constructive comments. We are encouraged by your positive comments on our method (novel, effective, substantial improvements over the existing point cloud Transformers) and the writing. In the following, we address your concerns carefully.\n\n**Q1: The similarities with FlatFormer.**\n\nA: Thank you for your suggestion for discussing the differences between ours and FlatFormer.\n\nInspired by Swin Transformer [1], SST [2] first projects 3D point cloud into Bird's Eye View (BEV) space and then splits the space into non-overlapping 2D square windows to perform self-attention. FlatFormer [3] proposes flattened window attention mainly to solve the problem of inconsistent token counts within each window in SST and then to achieve parallel processing on the GPU. FlatFormer does not partition the window into planes and instead it first voxelizes the point cloud into sparse Bird's Eye View (BEV) pillars and then re-orders these pillars in the BEV space. Both the motivation and designs are different from ours. In comparison with them, we first voxelize 3D point cloud to voxels and then process the voxels with our proposed disassembled window attention module to enlarge the attention range with minimal additional computational costs.\nWe will add the discussion in the final version.\n\n[1] Liu et al. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV 2021.\n\n[2] Fan et al. Embracing single stride 3d object detector with sparse transformer. CVPR 2022.\n\n[3] Liu et al. FlatFormer: Flattened window attention for efficient point cloud transformer. CVPR 2023.\n\n---\n\n**Q2: The proposed model failed to outperform the SoTA detectors on SUN-RGBD.**\n\nA: Thank you for pointing out this issue. To validate the effectiveness of our ConDaFormer as a generalized 3D backbone, we selected a simple 3D object detection method, FCAF3D, as our baseline and did not make elaborate adjustments to our method to improve the performance. To address your concern, we further take the SoTA detector, CAGroup3D (Table 5 in the sumission), as our baseline and conducted experiments on SUN RGB-D dataset. The best and average (in bracket) performance are listed below. As a generalized 3D backbone, ConDaFormer's potential usage is demonstrated by its comparable performance to FCAF3D and CAGroup3D. We will add this new comparison in the final version to further show the capacity of our method.\n\nDetection results on SUN RGB-D: \n|   Method | mAP\\@0.25 | mAP\\@0.50 |\n|----------|:----------:|:-------:|\n| CAGroup3D    |\t66.8 (66.4) | 50.2 (49.5) |\n| ConDaFormer  |\t67.1 (66.8) | 49.9 (49.5) |\n\n---\n\nWe hope our response adequately addresses your concerns. If you still have any questions, we are looking forward to hearing them.\n"}, {"review_id": "yfAZCb17Es", "reviewer": "Reviewer_Kwdg", "summary": "This paper proposes a new window partitioning method, which can save a lot of computation costs by sacrificing a small amount of precision. At the same time, it proposes a kind of depth wise sparse convolution, which can better capture local structure by using before and after self attention. The experimental results fully prove the effectiveness of the method", "questions": "1.\tI think a better interaction between the planes might improve task performance even more", "limitations": "limitations have benn discussed in the paper", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper innovatively puts forward a new window partitioning method, which divides 3D cubic into 3 2D planes and then divides the Windows. This method can save a lot of calculation costsThis paper presents a new regularization method to optimize networks by predicting relative position differences\n2.\tBy using depth wise sparse convolution to capture local structures, the experimental results fully demonstrate the effectiveness of this design\n3.\tThe content is sufficient and the experimental results are abundant\n", "weaknesses": "1.\tThe interaction between different planes is only additive operation, which may lead to the loss of 3D structure information,\n2.\tMore relevant work on window partitioning methods is basically not discussed and compared\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and constructive comments. We are encouraged by your positive comments on our method (innovative, effective) and experiments (abundant). In the following, we address your concerns carefully.\n\n\n**Q1: The interaction between different planes.**\n\nA: Thank you for pointing out this issue. But, we have to clarify that the interaction between different planes is not a summantion but a concatenation ($\\bigoplus$) in Eq. 4 and Eq. 6, as stated in L161. We apologize for the misunderstanding.\n\nIn addition, although using the concatenation, the loss of 3D structure information might exist, as you pointed out. In fact, we alleviated the issue by introducing depth-wise sparse convolution (DSConv). Specifically, before the attention, we apply DSConv to aggregate more contextual information. After the self-attention within each plane, we also propagate the updated feature of each point to its local neighbors with another DSConv operation, which enables information exchange between different planes. We appreciate your suggestion to further improve the interaction between planes, which we will explore in future work.\n\n---\n\n**Q2: Relevant work on window partitioning methods.**\n\nA: In the Related work of the submission, we briefly reviewed some works on window partitioning in 2D transformers, such as Swin Transformer [1], Ccnet [2], Axial-Attention [3], and CSWin Transformer [4], and several related works in 3D, including Stratified Transformer, Swin3D, and OctFormer. Here, we discuss more related works on 3D and will incorporate them into the final version.\n\nInspired by Swin Transformer, both the Stratified Transformer[5] and Swin3D[6] partition the 3D space into non-overlapping 3D cubic windows to perform local self-attention. SST [7] first projects 3D point cloud into Bird's Eye View (BEV) space and then splits the space into non-overlapping 2D square windows. To avoid the expensive cost of window partitioning and padding due to inconsistent token counts within each window in SST, FlatFormer[8] partitions the 3D point cloud into groups of equal sizes using axis sorting, leading to improved computational regularity.\n\n[1] Liu et al. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV 2021.\n\n[2] Huang et al. Ccnet: Criss-cross attention for semantic segmentation. ICCV 2019.\n\n[3] Wang et al. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. ECCV 2020.\n\n[4] Dong et al. Cswin transformer: A general vision transformer backbone with cross-shaped windows. CVPR 2022.\n\n[5] Lai et al. Stratified transformer for 3d point cloud segmentation. CVPR 2022.\n\n[6] Yang et al. Swin3D: A pretrained transformer backbone for 3d indoor scene understanding. arXiv:2304.06906 (2023).\n\n[7] Fan et al. Embracing single stride 3d object detector with sparse transformer. CVPR 2022.\n\n[8] Liu et al. FlatFormer: Flattened window attention for efficient point cloud transformer. CVPR 2023.\n\n---\n\nWe hope our response adequately addresses your concerns. If you still have any questions, we are looking forward to hearing them.\n"}, {"review_id": "cC9uFX82Ba", "reviewer": "Reviewer_32ZL", "summary": "This paper proposed a network for point cloud understanding. The basic idea is to process point clouds in 3 orthogonal 2D planes (triplanes). This can largely reduce the computation cost compared to processing point clouds in 3D space. Experiments on several tasks (segmentations and detection) are conducted to show the effectiveness of this method.", "questions": "See the weakness section.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper showed an insight of point cloud processing: when we are dealing with data of large dimensionality, it is better that we can first project the data into a smaller space. The figures nicely illustrated the core idea. Even though the triplane-style network has been used in many other works, I still believe the paper did a great job in designing the backbone network. Some experiments on segmentation and detection can also show the potential usage of the proposed network.\n\nOverall, I believe the paper delivered a great idea in designing 3D networks and some good results. My major concern is about the experiement part (see the weakness section).", "weaknesses": "1. As a general backbone network for point cloud understanding. Some experiments are missing. For example, object-level classification (ShapeNet, ModelNet, ScanObjectNN), object-level segmentation (ShapeNetPart, PartNet).\n2. As mentioned above, the triplane-style network has been used in some prior works (e.g., [1]).\n3. A minor issue, it is difficult to understand the notations of attention matrix for 3 planes (-, ~ and ^). I know because of the superscripts and subscripts, the authors had to choose (-, ~ and ^) to denote different planes. But maybe we can find something better.\n\n[1] Efficient Geometry-aware 3D Generative Adversarial Networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and constructive comments. We are encouraged by your positive comments on our method (great idea in designing 3D networks, potential usage of the proposed network). In the following, we address your concerns carefully.\n\n**Q1: Missing experiments on object-level classification and object-level segmentation.**\n\nA: Thank you for pointing out this issue. During the submission, we did not conduct experiments on object-level tasks. It is because this work mainly aims to deal with large-scale point clouds that have more points requiring higher computational costs. Thus, we neglected to experiment on object-level point clouds. \n\nHere, with your suggestions and our interest in the performance on small-scale point clouds, we conduct experiments on ModelNet and ShapeNetPart and the results are listed below.  We can find that for small-scale point cloud data, our method still achieves comparable or even better performance in comparison with Point Transformer v1, Point Transformer v2, and Stratified Transformer. We will add these results in the final version.\n\nShape classification results on ModelNet40: \n|   Method | mAcc (%) | OA (%) |\n|:----------|:----------:|:-------:|\n| PointNet         | 86.0 | 89.2 |\n| PointNet++       | -    | 91.9 |\n| Point Transformer v1 | 90.6 | 93.7 |\n| Point Transformer v2 | 91.6 | 94.2 |\n| PointNeXt        | 90.8 | 93.2 |\n| ConDaFormer      | 90.8 | 94.0 |\n\nPart segmentation results on ShapeNet-Part: \n|   Method | cls. mIoU | ins. mIoU |\n|:----------|:----------:|:-------:|\n| PointNet         | 80.4 | 83.7 |\n| PointNet++       | 81.9 | 85.1 |\n| Point Transformer v1 | 83.7 | 86.6 |\n| Stratified Transformer | 85.1 | 86.6 |\n| PointNeXt        | 85.2 | 87.0 |\n| ConDaFormer      | 84.9 | 86.8 |\n\n---\n\n**Q2: The triplane-style network has been used in some prior works.**\n\nA: Thank you for providing an interesting work using the triplane-style strategy. EG3D [1] aims at representing the intermediate 3D volume in 3D generation into three planes to reduce the memory. For each 3D point, we can get its features by projecting it to the three planes and then summing the queried three feature vectors. Although EG3D and ours both use the triplane to reduce the computational costs, EG3D mainly focuses on the more efficient representation (2D planes instead of 3D dense volumes) to capture greater details by using higher resolution features, while ours focuses on reducing the involved points in self-attention to enlarge the attention range. Therefore, we think our method is different from EG3D. According to your suggestion, we will add the discussion to the final version.\n\n[1] Chan et al. Efficient geometry-aware 3D generative adversarial networks. CVPR 2022.\n\n---\n\n**Q3: The notations of attention matrix are difficult to understand.**\n\nA: Thank you for your suggestion. We are considering placing the plane indicator to the subscript, such as $\\bar{Q}^h_t$ to $Q^h_{xy,t}$ and $\\bar{Attn}$ to $Attn_{xy}$. We would appreciate it if you could further give your feedback.\n\n---\n\nWe hope our response adequately addresses your concerns. If you still have any questions, we are looking forward to hearing them.\n"}]

Author rebuttal:

Meta-review: The submission received 5 positive recommendations. Initially, the reviewers were concerned about the similarities to the latest methods, the evaluations, and the demerit of the proposed attention mechanism. The authors addressed most of the concerns in the rebuttal and the responses to reviewers’ additional comments. The reviewers reached a consensus of acceptance after the discussion period. The AC read through the submission, the review, the rebuttal, and the discussions. The AC agrees with the reviewers that the idea is interesting and novel and the evaluation is convincing. Per this, the AC supports the reviewers’ decision, and accepts this submission. The decision was discussed with and approved by the SAC. Please follow the reviewers’ comments to improve the manuscript.
Decision: Accept (poster)