SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lunhao Duan*1 Shanshan Zhao*2
Nan Xue<sup>1</sup> Mingming Gong<sup>3</sup> Gui-Song Xia<sup>†</sup> Dacheng Tao<sup>4</sup>
<sup>1</sup> School of Computer Science, Wuhan University
$^{2}$  JD Explore Academy  $^{3}$  University of Melbourne  $^{4}$  University of Sydney
Transformers have been recently explored for 3D point cloud understanding with impressive progress achieved. A large number of points, over 0.1 million, make the global self-attention infeasible for point cloud data. Thus, most methods propose to apply the transformer in a local region, e.g., spherical or cubic window. However, it still contains a large number of Query-Key pairs, which require high computational costs. In addition, previous methods usually learn the query, key, and value using a linear projection without modeling the local 3D geometric structure. In this paper, we attempt to reduce the costs and model the local geometry prior by developing a new transformer block, named ConDaFormer. Technically, ConDaFormer disassembles the cubic window into three orthogonal 2D planes, leading to fewer points when modeling the attention in a similar range. The disassembling operation is beneficial to enlarging the range of attention without increasing the computational complexity but ignores some contexts. To provide a remedy, we develop a local structure enhancement strategy that introduces a depth-wise convolution before and after the attention. This scheme can also capture the local geometric information. Taking advantage of these designs, ConDaFormer captures both long-range contextual information and local priors. The effectiveness is demonstrated by experimental results on several 3D point cloud understanding benchmarks. Our code will be available.
As a fundamental vision task, 3D point cloud analysis [56, 58, 12, 97] has been studied extensively due to its critical role in various applications, such as robotics and augmented reality. The inherent irregularity and sparsity of point clouds make the standard convolution fail to extract the geometric features directly. As a result, different strategies have been proposed for 3D point cloud data processing, which can be roughly divided into the voxel-based methods [12], the projection-based [7], and the point-based [58, 97]. Many methods achieve remarkable performance in point cloud analysis tasks, such as point cloud segmentation [15] and 3D object detection [64].
In recent years, transformers [70] have shown a powerful capability of modeling long-range dependencies in the natural language processing community [17, 60]. Following ViT [19], transformers [18, 22, 13, 93, 44] also achieve competitive or better performance in comparison with the CNN architectures in many 2D vision tasks, such as object detection and semantic segmentation.
*Equal Contribution. This work was done when Lunhao Duan was a research intern at JD Explore Academy.
$\dagger$  Correspondence Author.
$^{\ddagger}$ Con, Da, and Former indicate Convolution, Disassembled, and Transformer, respectively.
Regarding the 3D point cloud data, the graph structure makes it natural to apply transformers in hierarchical feature representation, and there are indeed many attempts [97, 80, 34] focusing on efficient designs. Since the 3D scene data usually contains more than 0.1 million points [15], it is impractical to apply self-attention globally. Instead, a local region obtained by KNN search [80] or spatially non-overlapping partition [34] is selected to perform the attention. Nevertheless, due to the 3D spatial structure, it still requires high computational costs.
Additionally, for 2D image data, convolution operation has been explored to assist the transformer in modeling local visual structures [22, 87], whereas such a strategy is under-explored for 3D point cloud data. As the density varies across the point cloud, we cannot concatenate the raw pixels within a patch for query, key, and value learning as in 2D image data. As a result, it is important to model the local geometric structure for better feature representation. In fact, in Point Transformer v2 [80] and Stratified Transformer [34], although grouping-then-pooling is used to aggregate local information in the downsampling operation between adjacent encoding stages, we deem it is still inadequate for modeling local geometry prior.
Keeping the issues aforementioned in mind, in this paper we develop a new transformer block for 3D point cloud understanding, named ConDaFormer. Specifically, regarding the computational costs, we seek an alternative window partition strategy to replace the 3D cubic window, which can model the dependencies in a similar range but involve fewer points. Inspired by CSWin [18], we disassemble the cubic window into three orthogonal 2D planes. In this way, for a query point, only the points located in three directions are considered the key. Therefore, we can enlarge the attention range with negligible additional computational costs. The triplane-style strategy is also used by EG3D [5] in 3D generation, which aims at representing the intermediate 3D volume into three planes to reduce memory consumption. As shown in Figure 1, we can observe that for similar distances, the multi-plane windows contain fewer Query-Key pairs. However, as shown in Figure 3 (a), the disassembling operation inevitably causes fewer contexts to be modeled. To alleviate this issue, we apply a depth-wise sparse convolution (DSConv) before the attention to aggregate more
Figure 1. Query-Key pairs v.s. Window Size. We count the average number of Query-Key pairs for computing self-attention under different window sizes in scenes of S3DIS [1] with 80k points input. We can observe that the disassembled window partition can reduce the number of pairs remarkably.
contextual information for the query, key, and value as a complement. Moreover, after the attention, we also propagate the updated feature of each point to its local neighbors with another DSConv operation. Such local structure enhancement based on sparse convolution not only compensates for the context loss caused by disassembling but also benefits local 3D geometric prior learning. Thanks to the efficient designs, our ConDaFormer is able to model both dependencies in a larger range and the local structure.
In total, the main contributions of this paper can be summarized as follows:
- We propose a novel disassembled window attention module for 3D point cloud understanding by disassembling the 3D window into three orthogonal planes for self-attention. This strategy effectively reduces computational overhead with negligible performance decreases.
- To enhance the modeling of local features, we introduce depth-wise sparse convolution within the disassembled window attention module. This combination of self-attention and convolution provides a comprehensive solution for capturing both long-range contextual information and local priors in 3D point cloud data.
- Experiments show that our method achieves state-of-the-art performance on widely adopted large-scale 3D semantic segmentation benchmarks and comparable performance in 3D object detection tasks. Extensive ablation studies also verify the effectiveness of our proposed components.
Vision transformers. Inspired by the remarkable success of transformer architecture in natural language processing [70], Vision Transformer (ViT) [19] has been proposed, which decomposes image into non-overlapping patches and leverages the multi-headed self-attention by regarding each patch as
a token. However, global self-attention across the entire image imposes a substantial computational burden and is not applicable for pixel-level image understanding tasks with high-resolution image input. To tackle this issue and extend ViT to downstream tasks, Swin Transformer [44] proposes to confine self-attention to local non-overlapping windows and further introduces shift operations to foster information exchange between adjacent windows. To enlarge the receptive field, some techniques, such as Ccnet [30], Axial-Attention [72], and CSWin Transformer [18], propose to perform self-attention in striped window. Such designs enable pixels to capture long-range dependencies while maintaining an acceptable computational cost. In addition, several recent approaches [78, 22, 87] attempt to integrate convolutions into transformer blocks, leveraging the benefits of local priors and long-range contexts.
Point cloud understanding. Existing methods for point cloud understanding can be broadly classified into three categories: voxel-based, projection-based, and point-based. Voxel-based methods [57, 12] divide the point cloud space into small voxels (i.e., voxelization) and represent each voxel with a binary value indicating whether it is occupied or not. Earlier approaches [81, 57] directly apply convolution on all voxels, which causes a large computational effort. To improve computational efficiency, some methods [61, 37] propose octrees representation for point cloud data to reduce the computational and memory costs. Recently, considering the sparsity property, many methods [12, 21] develop sparsity-aware convolution where only non-empty voxels are involved. For example, Choy et al. [12] represent the spatially sparse 3D data as sparse tensors and develop an open-source library that provides general operators (e.g., sparse convolution) for the sparse data. Projection-based methods [7, 65] project the point cloud data onto different 2D views. In this way, each view can be processed as a 2D image and the standard convolution can be used straightforwardly. Both voxel-based and projection-based methods might suffer from the loss of geometric information due to the voxelization and projection operation. In comparison with them, point-based methods directly operate on the raw point cloud data. Following the pioneering work, PointNet [56] and PointNet++ [58], a large number of approaches for local feature extraction have been proposed [69, 79, 76, 96, 39, 86, 28]. These methods utilize a learnable function to transform each point into a high-dimensional feature embedding, and then employ various geometric operations, such as point-wise feature fusion and feature pooling, to extract semantic information.
Point cloud transformers. As transformers have shown powerful capability in modeling long-range dependencies in 2D image data, it is natural to explore the application to point cloud data with graph structure [97, 23, 50, 24, 20, 80, 45]. Although most transformer-based point cloud understanding networks can be classified into the point-based aforementioned, we review previous works especially here since they are very close to this paper. Point Cloud Transformer (PCT) [23] and Point Transformer v1 (PTv1) [97] make earlier attempts to design transformer block for point cloud data. However, similar to ViT [19], PCT performs global attention on all input points, leading to high computation and memory costs. In comparison, PTv1 achieves self-attention within a local region for each query point, which largely reduces the computational effort and can be applied to scene-level point clouds. Following PTv1, Point Transformer V2 (PTv2) [80] further promotes the study in point cloud transformer and achieves better performance on typical point cloud analysis tasks with developed grouped vector attention and partition-based pooling schemes. Instead of the overlapping ball regions, Stratified Transformer [34], inspired by Swin Transformer [44], splits the 3D space into non-overlapping 3D cubic windows to perform local self-attention within each window. To enlarge the receptive field and capture long-range contexts, it also develops a stratified strategy to select distant points as well as nearby neighbours as the keys. Considering the special structure of LiDAR data, SST [20] first voxelizes 3D LiDAR point cloud into sparse Bird's Eye View (BEV) pillars and then splits the space into non-overlapping 2D square windows to perform self-attention. To solve the problem of inconsistent token counts within each window in SST, FlatFormer [45] first re-orders these pillars in the BEV space and then partitions these pillars into groups of equal sizes to achieve parallel processing on the GPU. CpT [32] and 3DCTN [48] explore the local structure encoding in the transformer block but only focus on small-scale data, like the CAD model. To provide more exhaustive progress on the point cloud transformer, here we further briefly discuss two recent works that were uploaded to the arXiv website shortly before the manuscript submission, Swin3D [92] and OctFormer [74]. Swin3D mainly focuses on the 3D backbone pre-training by constructing a 3D Swin transformer and pre-training it on a large-scale synthetic dataset. OctFormer aims to reduce the computation complexity of attention by developing octree-based transformers with the octree representation of point cloud data. Both of them and ours aim to explore the transformer in point cloud analysis in different ways.
Figure 2. (a) Framework overview. The entire network consists of a Point Embedding layer and four stages each one of which contains a downsampling layer and  $N_{i}$  ConDaFormer blocks. (b) Structure of ConDaFormer block. Each ConDaFormer block consists of two Layer Normalization layers, an MLP, and a ConDaFormer module. The module contains a disassembled window attention, two linear layers, and two local structure enhancement (LSE) modules before and after the attention.
Overview. Regarding the point cloud understanding task, our goal is to develop a backbone network to generate high-level hierarchical features for a given point cloud, which can be further processed by a specific task head, such as semantic segmentation and object detection. We focus on designing a transformer block that can be used to construct such a backbone. Specifically, let  $X = (P \in \mathbb{R}^{N \times 3}, F \in \mathbb{R}^{N \times C})$  represent the point cloud feed into the transformer block, where  $P$  represents the 3D position of  $N$  points and  $F$  denotes the features with channel number  $C$ . The transformer block processes  $X$  by exploiting the attention mechanism [70] and generates new features. We design the transformer block by disassembling the typical 3D window into multiple planes and introducing the local enhancement strategy, resulting in our ConDaFormer block. Based on ConDaFormer, we can construct networks for different point cloud analysis tasks, including point cloud semantic segmentation and 3D object detection. In the following, we detail ConDaFormer and the proposed networks for two high-level point cloud analysis tasks. Before that, we first give a brief introduction to the transformer based on the shifted 3D window, i.e., basic Swin3D block.
Basic Swin3D block. ViT [19] achieves global self-attention of 2D images by partitioning the 2D space into non-overlapping patches as tokens. However, for high-resolution images, global attention is high-cost due to the quadratic increase in complexity with image size. To encourage the transformer to be more feasible for general vision tasks, Swin Transformer [44] develops a hierarchical transformer block where self-attention is applied within non-overlapping windows, and the adjacent windows are connected with a shifted window partitioning strategy. For the 3D point cloud data, we can naively extend the 2D window to the 3D version [34, 92]. Specifically, let  $X_{t} = (P \in \mathbb{R}^{N_{t} \times 3}, F_{t} \in \mathbb{R}^{N_{t} \times C})$  denote the points in the  $t$ -th window with the size of  $S \times S \times S$  and containing  $N_{t}$  points. The self-attention with  $H$  heads can be formulated as:
$$
Q _ {t} ^ {h} = \operatorname {L i n e a r} \left(X _ {t}\right), K _ {t} ^ {h} = \operatorname {L i n e a r} \left(X _ {t}\right), V _ {t} ^ {h} = \operatorname {L i n e a r} \left(X _ {t}\right), h \in [ 1, \dots , H ],
$$
$$
A t t n ^ {h} \left(X _ {t}\right) = S o f t \left(\left(Q _ {t} ^ {h}\right) \left(K _ {t} ^ {h}\right) ^ {T} / \sqrt {C / H}\right) \left(V _ {t} ^ {h}\right), h \in [ 1, \dots , H ], \tag {1}
$$
$$
S w i n 3 D (X _ {t}) = \text {L i n e a r} ([ A t t n ^ {1} (X _ {t}) \oplus \dots \oplus A t t n ^ {h} (X _ {t}) \oplus \dots \oplus A t t n ^ {H} (X _ {t}) ]),
$$
where 1)  $h$  indicates the  $h$ -th head; 2)  $Q_{*}^{*}, K_{*}^{*}$ , and  $V_{*}^{*}$  represent query, key, and value, respectively; 3) Linear denotes the Linear function (Note that these linear functions do not share parameters); 4) Soft denotes the Softmax function; 5) Attn denotes the self-attention operation; 6)  $[\cdot \oplus \cdot ]$  represents the concatenation operation along the feature channel dimension. To achieve a cross-window connection, the shift along three dimensions with displacements is utilized in a successive transformer block with similar operations in Eq. 1.
Although the local window attention largely reduces the computational complexity in comparison with global attention, for each partitioned 3D window, the computation for self-attention is still
(a) The disassembled window attention
(b) LSE module
Figure 3. Two components in the ConDaFormer block. (a) An illustration of window partition: from cubic window to disassembled windows. The light blue areas indicate the regions involved in self-attention. (b) The detailed structure of the LSE module.
high-cost. In this paper, we explore how to further reduce the computational costs with small changes in the range of attention. Specifically, we develop a new block ConDaFormer, which enlarges the attention range with negligible additional computational costs.
3.2 ConDaFormer
In this part, we introduce our ConDaFormer by detailing the two main components: disassembling operation and sparse-convolution-based local structure enhancement.
Cube to Multiple planes: Disassembling the 3D window. As analyzed before, the 3D window attention requires high computational costs for modeling long-range dependencies. To reduce the computational cost while maintaining the equivalent attention range, we propose to convert the cubic window to three orthogonal 2D planes, as shown in Figure 3. Technically, we split the whole 3D space into three orthogonal planes,  $XY$ -plane,  $XZ$ -plane, and  $YZ$ -plane. For each 2D plane, we generate a series of non-overlapping 2D windows, in each of which self-attention is performed. Taking an example of the  $XY$ -plane for illustration, we have the following equations:
$$
Q _ {x y, t} ^ {h} = \operatorname {L i n e a r} (X _ {t}), K _ {x y, t} ^ {h} = \operatorname {L i n e a r} (X _ {t}), V _ {x y, t} ^ {h} = \operatorname {L i n e a r} (X _ {t}), h \in [ 1, \dots , H / 3 ],
$$
$$
A t t n _ {x y} ^ {h} \left(X _ {t}\right) = S o f t \left(\left(Q _ {x y, t} ^ {h}\right) \left(K _ {x y, t} ^ {h}\right) ^ {T} / \sqrt {C / H}\right) \left(V _ {x y, t} ^ {h}\right), h \in [ 1, \dots , H / 3 ], \tag {2}
$$
$$
A t t n _ {x y} (X _ {t}) = \left[ A t t n _ {x y} ^ {1} (X _ {t}) \oplus \dots \oplus A t t n _ {x y} ^ {h} (X _ {t}) \oplus \dots \oplus A t t n _ {x y} ^ {H / 3} (X _ {t}) \right],
$$
where the notations have identical meanings to those in Eq. 1, except that Linear maps the channel number from  $C$  to  $C / 3$ . In such a basic formulation, the attention is only related to the similarity between query  $Q$  and key  $K$  but does not contain the position bias which is important for self-attention learning. We can add learned relative position bias to  $(Q_{xy,t}^{h})(K_{xy,t}^{h})^{T} / \sqrt{C / H}$  to encode the position information as Swin Transformer does [44]. But, to better capture the position information with contextual information, we adopt an efficient context-based adaptive relative position encoding scheme developed in Stratified Transformer [34]. Specifically, we can re-write the computation of  $Attn_{xy}^{h}(X_{t})$  in Eq. 2 as follows:
$$
A t t n _ {x y} ^ {h} \left(X _ {t}\right) = S o f t \left(\left(\left(Q _ {x y, t} ^ {h}\right)\left(K _ {x y, t} ^ {h}\right) ^ {T} + Q _ {x y, t} ^ {h} E _ {q} ^ {h} + K _ {x y, t} ^ {h} E _ {k} ^ {h}\right) / \sqrt {C / H}\right)\left(V _ {x y, t} ^ {h} + E _ {v} ^ {h}\right)\left. \right), \tag {3}
$$
where  $E_{q}, E_{k}$ , and  $E_{v}$  are corresponding learnable position encodings for  $Q_{xy}, K_{xy}$ , and  $V_{xy}$ . For the other two planes, we exploit similar operations and share the same position encoding for three planes. We denote the attention as  $Attn_{yz}(X_t)$  ( $YZ$ -plane) and  $Attn_{xz}(X_t)$  ( $XZ$ -plane), respectively. Such a shared strategy makes the learning of position encoding more effective, which is demonstrated in the experiments. Then, we merge the three attention results to obtain the final output of the disassembled transformer (DaFormer):
$$
\operatorname {D a F o r m e r} \left(X _ {t}\right) = \operatorname {L i n e a r} \left(\left[ \operatorname {A t t n} _ {x y} \left(X _ {t}\right) \oplus \operatorname {A t t n} _ {y z} \left(X _ {t}\right) \oplus \operatorname {A t t n} _ {x z} \left(X _ {t}\right) \right]\right). \tag {4}
$$
Local structure enhancement with sparse convolution. As shown in Figure 3 (a), in comparison with the 3D window, although our plane-based window is easy to capture long-range dependency, it ignores the contexts not located in the planes. To address this issue and model the local geometry prior, we propose to apply a sparse convolution operation to encode the input before the query, key, and value learning. As depicted in Figure 3 (b), the local structure enhancement module has
two branches: a  $1 \times 1 \times 1$  convolution and a depth-wise sparse convolution to aggregate the local information. Mathematically, the local structure enhancement (LSE) module can be written as:
$$
L S E \left(X _ {t}\right) = \left[ B N \left(\text {L i n e a r} \left(X _ {t}\right)\right) \oplus B N \left(\text {L i n e a r} \left(\text {G E L U} \left(B N \left(\text {D C o n v} \left(X _ {t}\right)\right)\right)\right)\right), \right. \tag {5}
$$
where  $BN$ ,  $GELU$ , and  $DConv$  indicate the Batch Normalization, GELU activation function, and depth-wise sparse convolution ( $3 \times 3 \times 3$  kernel), respectively. After the self-attention, we also apply another local enhancement operation to propagate the long-range contextual information learned by self-attention to local neighbors to further improve the local structure. As a result, the full operation consisting of two local enhancements and one self-attention can be written as with the notations above:
$$
X _ {t} ^ {\prime} = L S E (X _ {t}),
$$
$$
C o n D a F o r m e r \left(X _ {t}\right) = L i n e a r \left(L S E \left(\left[ A t t n _ {x y} \left(X _ {t} ^ {\prime}\right) \oplus A t t n _ {y z} \left(X _ {t} ^ {\prime}\right) \oplus A t t n _ {x z} \left(X _ {t} ^ {\prime}\right) \right]\right)\right). \tag {6}
$$
ConDaFormer block structure. The overall architecture of our ConDaFormer block is depicted in Figure 2 (b). Our ConDaFormer block comprises several essential components, including two local enhancement modules (before and after self-attention), a disassembled window self-attention module, a multi-layer perceptron (MLP), and residual connections [25].
3.3 Network Architecture
Point embedding. As mentioned by Stratified Transformer [34], the initial local feature aggregation is important for point transformer networks. To address this, we employ a sparse convolution layer to lift the input feature to a higher dimension  $C$ . Subsequently, we leverage a ResBlock [25] to extract the initial point embedding, facilitating the representation learning process.
Downsampling. The downsampling module is composed of a linear layer, which serves to increase the channel dimension, and a max pooling layer with a kernel size of 2 and a stride of 2 to reduce the spatial dimension.