Point Transformer V2: Grouped Vector Attention and Partition-based Pooling
Authors: Xiaoyang Wu
Yixing Lao
Li Jiang
Xihui Liu
Hengshuang Zhao
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D Computer Vision
Point Cloud
Transformer

Abstract: As a pioneering work exploring transformer architecture for 3D point cloud understanding, Point Transformer achieves impressive results on multiple highly competitive benchmarks. In this work, we analyze the limitations of the Point Transformer and propose our powerful and efficient Point Transformer V2 model with novel designs that overcome the limitations of previous work. In particular, we first propose group vector attention, which is more effective than the previous version of vector attention. Inheriting the advantages of both learnable weight encoding and multi-head attention, we present a highly effective implementation of grouped vector attention with a novel grouped weight encoding layer. We also strengthen the position information for attention by an additional position encoding multiplier. Furthermore, we design novel and lightweight partition-based pooling methods which enable better spatial alignment and more efficient sampling. Extensive experiments show that our model achieves better performance than its predecessor and achieves state-of-the-art on several challenging 3D point cloud understanding benchmarks, including 3D point cloud segmentation on ScanNet v2 and S3DIS and 3D point cloud classification on ModelNet40. Our code will be available at https://github.com/Gofinge/PointTransformerV2.

Point Transformer V2: Grouped Vector Attention and Partition-based Pooling
Xiaoyang Wu'
Yixing Lao² Li Jiang? Xihui Liu
Hengshuang Zhao
"The University of Hong Kong (xywu3, hszhao les. hku.hh.
Intel Labs
Max Planck Institute
Oct 2022
Abstract
aXXiv:221005666VV [cs.CV] 12 Oct
As a pioneering work exploring transformer architecture for 3D point cloud under- standing, Point Transformer achieves impressive results multiple highly compet- itive benchmarks. this work, we analyze the limitations of the Point Transformer and propose our powerful and efficient Point Transformer V2 model with novel designs that overcome the limitations of previous work. In particular, we first propose group vector attention, which is more effective than the previous version of vector attention. Inheriting the advantages of both learnable weight encoding and multi-head attention, present a highly effective implementation of grouped vector attention with novel grouped weight encoding layer. We also strengthen the position information for attention by an additional position encoding multiplier. Furthermore, we design novel and lightweight partition-based pooling methods which enable better spatial alignment and more efficient sampling. Extensive experiments show that our model achieves better performance than its predecessor and achieves tatte-of-theaat on several challenging 3D point cloud understanding benchmarks, including 3D point cloud segmentation on ScanNet v2 and S3DIS and 3D point cloud classification on ModelNet40. Our code will be available https: /ttes:///ttttttt github. com/ fo
Introduction
Point Transformer (PTv1) [1] introduces the self-attention networks 3D point cloud understanding. Combining the vector attention [2] with a U-Net style encoder-decoder framework, PTvl achieves remarkable performance several 3D point cloud recognition tasks, including shape classification. object part segmentation, and semantic scene segmentation.
In this work, analyze the limitations of Point Transformer (PTv1) [1] and propose elegant and powerful backbone named Point Transformer V2 (PTv2). Our PTv2 improves upon PTv1 with several novel designs, including the advanced grouped vector attention with improved position encoding, and the efficient partition-based pooling scheme.
The vector attention layers in PTvl utilize MLPs as the weight encoding to map the subtraction relation query and key into an attention weight vector that can modulate the individual channels of the value vector. However, the model goes deeper and the number of channels increases. the number of weight encoding parameters also increases drastically, leading to severe overfitting and limiting the model depth. To address this problem, parameter-effice efficiiette formulation, where the vector attention is divided into groups with shared vector attention weights. Meanwhile, show that the well-known multi-head attention [3] and the vector attention [2, 1] are degenerate cases of our proposed grouped vector attention. Our proposed grouped
present grouped vector attention with more
"Corresponding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

vector attention inherits the merits of both vector attention and multi head attention while being more powerful and efficient.
Furthermore, point positions provide important geometric information for 3D semantic understanding. Hence, the positional relationship among 3D points is more critical than 2D pixels. However, previous 3D position encoding schemes mostly follow the 2D ones and do not fully exploit the geometric knowledge in 3D coordinates. To this end, we strengthen the position encoding mechanism by apply- ing an additional position encoding multiplier to the relation vector. Such a design strengthens the positional relationship information the model, and we validate its effectiveness our experiments.
Moreover, it is worth noting that the irregular, non-uniform spatial distributions of points are signif- icant challenges to the pooling modules for point cloud processing. Previous point cloud pooling approaches rely on a combination of sampling methods (e.g. farthest point sampling [4] or grid sampling [5]) and neighbor query methods (e.g. kNN radius query), which is time-consuming and not spatially well-aligned. To overcome this problem, we go beyond the pooling paradigm of combining sampling and query, and divide the point cloud into non-overlapping partitions to directly fuse points within the same partition. We use uniform grids as partition divider and achieve significant improvement.
In conclusion, we propose Point Transformer V2, which improves Point Transformer [1] from several perspectives:
We propose an effective grouped vector attention (GVA) with a novel weight encoding layer that enables efficient information exchange within and among attention groups.
We introduce an improved position encoding scheme utilize point cloud coordinates better and further enhance the spatial reasoning ability of the model.
We design the partition-based pooling strategy to enable more efficient and spatially better- aligned information aggregation compared to previous methods.
We conducted extensive analysis and controlled experiments to validate our designs. Our results indicate that PTv2 outperforms predecessor works and sets the new state-of-the-a the-art various 3D understanding tasks.
2 Related Works
Image transformers. With the great success of ViT [6], the absolute dominance of convolution in vision tasks is shaken by Vision Transformer, which becomes a trend in 2D image understanding [7, 8, 9. 10]. VIT introduces the far-reaching scaled dot-product self-attention and multi-head self-attention theory [3] in NLP into vision by considering image patches as tokens. However, operating global attention on the entire image consumes excessive memory. To solve the memory consumption problem, Swin Transformer [7] introduces the grid-based local attention mechanism to operate the transformer block in a sequence of shifted windows.
Point cloud understanding. Learning-based methods for processing 3D point clouds can be classi- fied into the following types: projection-bad based, voxel- and point- based networks. An intuitive way to process irregular inputs like point clouds is to transform irregular representations into regular ones. Projection- based methods project 3D point clouds into various image planes and utilize 2D CNN-based backbones to extract feature representations [11, 12, 13, 14]. An alternative approach operates convolutions in 3D by transforming irregular point clouds into regular voxel representa- tions [15, 16). Those voxel-based methods suffer from inefficiency because of the sparsity of point clouds until the introduction and implementation of sparse convolution [17, 18]. Point-based methods extract features directly from the point cloud rather than projecting or quantizing irregular point clouds onto regular grids in 2D or 3D [19, 4, 20, 5]. The recently proposed transformer-basee point cloud understanding approaches, introduced in the next paragraph, are also categorized into point-based methods.
Point cloud transformers. Transformer- based networks belong to the category of point-based networks for point cloud understanding. During the research upsurge of vision transformers, at almost the same period, Zhao et al. [1] and Guo et al. [21] published their explorations of applying attention to point cloud understanding, becoming pioneers in this direction. The PCT [21] proposed by Guo et al. performs global attention directly on the point cloud. Their work, similar to VIT, is limited

---

Re
Sottmax
AGG
PTvl
MLP
AP
U9
U
U
Fr (p))
Pooling
Unpooling
Sollmax
Gouped AGG
PTv2
MLP
U
U
Pr
Pr ft
(PP)
Figure 1: Comparison of the attention, position encoding, and pooling mechanisms between PTvl and PTv2. Top-left: the vector attention (Sec. 3.1) with position encoding (Sec. 3.3) in PTv1. Bottom-left: our grouped vector attention (Sec. 3.2, denoted by red) with improved position encoding (Sec. 3.3, denoted by blue) in PTv2. Top-right: the sampling- based pooling and interpolation-bant -based unpooling PTvl. Bottom-right: our partition-base pooling and unpooling PTv2 (Sec. 3.4).
by memory consumption and computational complexity. Meanwhile, based on the vector attention theory proposed SAN [2], Point Transformer [1] proposed by Zhao al. directly performs local attention between each point and its adjacent points, which alleviated the memory problem mentioned above. Point Transformer achieves remarkable results in multiple point cloud understanding tasks and state-of-art results for several competitive challenges. In this work, we analyze the limitations of the Point Transformer [1], and propose several novel architecture designs for the attention and pooling module, to improve the effectiveness and efficiency of the Point Transformer. Our proposed model, Point Transformer V2, performs better than the Point Transformer across a variety of 3D scene understating tasks.
3 Point Transformer V2
We analyze the limitations of Point Transformer VI [PTv1) and propose our Point Transformer V2 (PTv2), including several improved modules upon PTv1. We begin by introducing the mathematical formulations and revisiting the vector self-attention used in PTv1 in Sec. 3.1. Based on the observation that the parameters PTv1 increases drastically with the increased model depth and channel size, propose our powerful and efficient grouped vector attention in Sec. 3.2. Further, introduce improved position encoding in Sec. 3.3 and the new pooling method in Sec. 3.4. We finally describe network architecture in Sec. 3.5.
3.1 Problem Formulation and Background
Problem formulation. Let M (P.F) be 3D point cloud scene containing a set of points x; = (p; fi) E M, where E R represents the point position, and f: R represents the point features. Point cloud semantic segmentation aims to predict class label for each point æ; and goal of scene classification is to predict class label for each scene M. M(p) denotes mapping function that maps the point at position p subset of M denoted as "reference set". Next, we revisit the self-attention mechanism used in PTvl [1].

---

Local attention. Conducting the global attention [6, 21] over all points in a scene is computationally heavy and infeasible for large-scale 3D scenes. Therefore, we apply local attention where the attention for each point a; works within subset of points, i.e., reference point set, M(p)).
Shifted-gri attention [7], where attention is alternatively applied over sets of non-overlapping image grids, has become is a common practice [22, 23, 24, 25] for image transformers. Similarly, the 3D space can be split into uniform non-overlapping grid cells, and the reference set defined as the points within the same grid, i.c., M(p.) = {(p, fo) P, in the same grid cell as p. However, such attention relies on a cumbersome shift grid operation achieve a global receptive field, and does not work well on point clouds where the point densities within different grids are not consistent.
PTv1 adopts neighborhood attention, where the reference point set is a local neighborhood of the given point, i.e., M(p:) = ((P; fz) Pj E Neighborhood((::) Specifically, the neighhorhood point set. M(p))) is defined the k nearest neighboring (kNN) points of p; in PTv1. Our experiments (Sec. 4.3) show that neighborhood attention is more effective than shifted-grid attention, so our approach adopts the neighborhood attention.
Scalar attention and vector attention. Given a point æ; = (pi, f.) E M,, we apply linear projec- tions or MLPs to project the point features St to the feature vectors of query qi key k; and value v, each with с channels. The standard sealar attention (SA) operated on the point x; and its reference point set (p,) can be represented follows,
Wij (q: /Ch/
fullhn
Σ
Softmax(w:),,j
my m.EM(p)
The attention weights in the above formulation are scalars computed from the scaled dot-product [3] between the query and key vectors. Multi-head scalar attention (MSA) [3] is extension of SA which runs several scalar attentions in parallel. MSA is widely applied in transformers, and we will show in Sec. 3.2 that MSA is degenerate case of our proposed grouped vector attention.
Instead of the sealar attention weights, PTvl applies vector attention, where the attention weights are vectors that can modulate the individual feature channels. In SA, the scalar attention is computed by the scaled dot product between the query and key vectors. In vector attention, a weight encoding function encodes the relation between query and key to vector. The vector attention [2] is formulated as follows,
Wij = w(r(q, k2)).
Fatta
(ettmax(W.)) ① Uy
(2)
By ((P)))))
where @ is the Hadamard product. is relation function (e.g., subtraction). IRC + R is learnable weight encoding (e.g., MLP) that computes the attention vectors to re-weight vy by channels
before aggregation. Fig. 2 (a) shows method using vector attention with linear weight encoding.
3.2 Grouped Vector Attention
In vector attention, the network goes deeper and there are more feature encoding channels, the number of parameters for the weight encoding layer increases drastically. The large parameter size restricts the efficiency and generalization ability of the model. In order overcome the limitations of vector attention, we introduce the grouped vector attention, as illustrated in Fig. (left).
Attention groups. We divide channels of the value vector R evenly into g groups (1 S S c). The weight encoding layer outputs grouped attention vector with channels instead of channels. Channels of within the same attention group share the same scalar attention weight from the grouped attention vector. Mathematically,
M(p:)
c/g
Wij = ww>(g;,kk))
fottn
Softmax (W;) 3e
te/gtm
(3)
m=1
where y is the relation function and w R IR is the learnable grouped weight encoding defined in the next paragraph. The second equation in Eq. the grouped vector aggregation. Fig. (a) presents vanilla GVA implemented by fully connected weight encoding, the number of the grouped weight encoding function parameters reduced compared with the vector attention (Fig. (b)), leading more powerful and efficient model.

---

(a) VA Lincar
(b) GVA Lincar
GVA MSA
(d) GVA
Grouped Linear
k)
((4
Figure 2: Comparison of various weight encoding functions. Each square represents a scalar, and each row of them represents a vector. The three rows represent relation vector, weight vector, and value vector from top to bottom. The attention groups are separated by dash lines. For demonstration, assume the feature dimension is 4 and the number of attention groups (applicable to b, c, d) is 2. Lines with different colors refer to different operations, blue lines represent learnable parameters act on input relation scalar, while red lines represent multiply by the input relation scalar. Orange lines identify which value feature is affected by the input scalar weight.
GVA is generalized formulation of VA and MSA. Our GVA degenerates to vector attention (VA) when g = c, and it degenerates to multi-heas self-attention (MSA) if w Eq. is defined follows,
1ixes
0 01xep
0 O1xea 01xe,
01xc,
11x0g
(r) =
(4)
VCg
O1xe, 0 01xe,
1 11xe
Cq
where Cg = c/g and r e Rxce
Grouped linear. Inspired by the weight encoding function of MSA, we design the grouped linear layer ((r) : R + R2 where different groups the input vector are projected with different parameters independently. Grouped linear further reduce the number of parameters in the weight encoding function. Our final adopted grouped weight encoding function is composed of the grouped linear layer, normalization layer, activation layer, and a fully connected layer allow inter-group information exchange. Mathematically,
P
0 1xca
0 Oxxea
T
0 01xeg
P2
0 01xe,
((r)
(5)
0 Oxxe
0 O1xer
Pg
g Cg
((r) Linear Act Norm(C(r)),
where cy = c/g. P...... Py R are learnable parameters, and represents function composition.
3.3 Position Encoding Multipler
Different from the discrete, regular-grid pixels in 2D images, points in the 3D point cloud are unevenly distributed in a continuous Euclidean Metric space, making the spatial relationship in 3D point cloud much more complicated than 2D images. In transformers and attention modules, the spatial information is obtained with the position encoding dvias (p. p;) added the relation vector r(q;, ky) bias.
Due to the generalization limitation vector attention in PTv1 mentioned in Sec. 3.2, adding more position encoding capacity to vector attention will not help to improve the performance. In PTv2, the grouped vector attention has an effect of reducing overfitting and enhancing generalization. With grouped vector attention restricting the capacity of the attention mechanism, we strengthen the position encoding with an additional multiplier 6mut (p; P,) to the relation vector, which focuses on learning complex point cloud positional relations. As shown in Fig. I (left), improved position
5

---

encoding is as follows,
Wi = w(omot - P,) @ 7(q; k:j) + dbias - -P))),
where @ is the Hadamard product. Smul, Obios IRd + Rd are two MLP position encoding functions, which take relative positions as input. Position encoding multiplier compliments group vector attention to achieve good balance of network capacity.
3.4 Partition-based Pooling
Traditional sampling-basee based pooling procedures adopted by other point-based methods use a com- bination of sampling and query methods. In the sampling stage, farthest point sampling [4] or grid sampling [5] is used to sample points reserved for the following encoding stage. For each sampled point, a neighbor query is performed aggregate information from the neighboring points. In these camplingg-bsee pooling procedures, the query sets of points are not spatially-aligned since the information density and overlap among each query set are not controllable. To address the problem, we propose a more efficient and effective partition-based pooling approach, as shown in Fig. 1.
Pooling. Given point set M (P,F), we partition M into subsets MM,.M2,.... MM,' by separating the space into non-overlapping partitions. We fusion each subset of points M; = (P., from single partition follows,
f MaxPool U [s Fi)),
= M(an Pj E.J),,
(8)
where (p', f() is the position and features of pooling point aggregated form subset M,, and U e Rexe' is the linear projection. Collecting the pooling points from n' subsets gives us the point set M = {p'.. fi for the next stage of encoding. In our implementation, we uniform grids to partition the point cloud space, and thus our partition- partition-based pooling is also called grid pooling.
Unpooling, The common practice of unpooling by interpolation is also applicable to partition- based pooling. Here we introduce a more straightforward and efficient unpooling method. To unpool the fused point set .M' back to M, the point locations in M are record from the pooling process, and we only need obtain the features for each point in M. With the help of the grid biddd partitioning subset,
during the pooling stage, we can map point feature to all points from the same
f.P fj
if if ( E
M,
3.5 Network Architecture
Backbone structure. Following previous works [18, 1]. we adopt the U-Net architecture with skip connections. There are four stages of encoders and decoders with block depths [2, 2, 2] and 1, 1. 1], respectively. The grid size multipliers for the four stages are [x3.0, x2.5, x2.5, x2.5], representing the expansion ratio over the previous pooling stage. The attention is conducted in local neighborhood, described in "neighborhood attention" in Sec. 3.1. In Sec. 4.3 we compare the neighborhood attention with shift-grid attention.
The initial feature dimension is 48, and we first embed the input chamnels to this number with a basic block with attention groups of 6. Then, double this feature dimension and attention groups each time entering the next encoding stage. For the four encoding stages, the feature dimensions are [96, 192, 384, 384], and the corresponding attention groups [12, 24, 48, 48].
Output head. For point cloud semantic segmentation, we apply an MLP to map point features produced by the backbone the final logits for each point in the input point set. For point cloud classification, we apply global average pooling over the point features produced by the encoding stages to obtain a global feature vector, followed by MLP classifier for prediction.
Experiments
To validate the effectiveness of the proposed method, we conduct experimental evaluations on ScanNet v2 [44] and S3DIS [45] for semantic segmentation, and Mode/Net40 [46] for shape classification. Implementation details are available in the appendix.

---

Table Semantic segmentation on ScanNet v2.
Table 2: Semantic segmentation on S3DIS Area
Method
Input
Val
Test
Method
Input
OA
mAcc
mloU
PointNet++ [4]
point
53.5
55.7
PointNet [19]
point
49.0
41.1
3DMV [26]
point
48.4
SegCloud [36]
point
57.4
48.9
PanopticFusion [27]
point
52.9
TanConv [37]
point
62.2
52.6
PointCNN [28]
point
45.8
PointCNN [28]
point
85.9
63.9
57.3
PointConv [29]
point
61.0
PointWeb [20]
point
87.0
66.6
JointPointBased [30]
point
69.2
63.4
HPEIN [38]
point
87.2
68.3
61.9
PointASNL [31]
point
63.5
66.6
GACNet [39]
point
87.8
62.9
SegGCN [32]
point
58.9
PAT [40]
point
70.8
60.1
RandLA-Net [33]
point
ParamConv [41]
point
67.0
58.3
KPConv [5]
point
69.2
68.6
SPGraph [42]
point
86.4
66.5
ISENet [34]
point
69.9
SegGCN [32]
point
88.2
70.4
63.6
FusionNet [35]
point
68.8
Mink [18]
voxel
71.7
65.4
SparseConvNet [17]
voxel
69.3
72.5
PAConv [43]
point
66.6
MinkUNet [18]
voxel
72.2
73.6
KPConv [5]
point
72.8
PTvl [1]
point
70.6
PTvl []]
point
90.8
76.5
70.4
PTv2 (ours)
point
75.4
75.2
PTv2 (ours)
point
91.1
77.9
71.6
4.1 Semantic Segmentation
Data and metric. For semantic segmentation, experiment on ScanNet v2 [44] and S3DIS [45]. The ScanNet v2 dataset contains 1,513 room scans reconstructed from RGB-D frames. The dataset divided into 1,201 scenes for training and 312 for validation. Point clouds for the model input sampled from vertices of reconstructed meshes, and each sampled point is assigned semantic label from 20 categories (wall, floor, table, etc.). The S3DIS dataset for semantic scene parsing consists of 271 rooms in six areas from three different buildings. Following a common protocol [36, 4, 1], area 5 is withheld during training and used for testing. Different from ScanNet v2, points of S3DIS are densely sampled on the mesh surfaces and annotated into 13 categories. Following standard protocol [4], we use mean class-wise intersection over union (mloU) as the evaluation metric for validation and test set of ScanNet v2. And we use mean class-wise intersection over union (mloU), mean of class-wiss accuracy (mAcc), and overall point-wise accuracy (OA) for evaluating performance on S3DIS area5.
Performance comparison. Table and Table 2 show the results of our PTv2 model compared with previous methods on ScanNet v2 and S3DIS, respectively. Our PTv2 model outperforms prior methods in all evaluation metrics. Notably, PTv2 significantly outperforms PTv1 [1] by 4.8% mloU on the ScanNet v2 validation set.
Visualization. The qualitative results of point cloud semantic segmentation shown in Fig. 3 and Fig. 4. Our PTv2 model is able to predict semantic segmentation results that are quite close the ground-truth. It is worth noting that model can capture the detailed structure information and predict the correct semantics for challenging scenarios. For example, in the S3DIS scenes with chairs, PTv2 is able cleanly predict the chair legs and armrests.
4.2 Shape Classification
Data and metric. We test our proposed PTy2 model for 3D point cloud classification on ModelNet40 dataset. The ModelNet40 [46] dataset consists of 12,311 CAD models belonging to 40 object categories. 9,843 models are split out for training, and the rest 2,468 models are reserved for testing. Following the common practice in the community, report the class-average accuracy (mAcc) and overall accuracy (OA) on the test set.
Performance comparison. We test our PTv2 model and compare it with previous models on the ModelNet40 dataset for shape classification. Results are shown Table 3, demonstrating that our proposed PTv2 model achieves state-of-the-art performance on ModelNet40 shape classification.
7

---

Input
Ground truth
PTv2
Input
Ground truth
PTv2
floor
wall
cabinet
bed
chair
sofa
table
door
window
bookshelf
picture
counter
desk
curtain
refrigerator
buthlub
shower curtain
toilet
sink
other furniture
Figure 3: Visualization of semantic segmentation results on ScanNet v2.
Input
Ground truth
PTv2
Input
Ground truth
PTv2
coiling
floor
wall
beam
columm
window
door
table
chuir
sofa
bookease
board
clutter
Figure 4: Visualization of semantic segmentation results on S3DIS.
4.3
Ablation Study
We conduct ablation studies to examine the effectiveness of each module in our design. The ablation study results reported ScanNet v2 validation
Attention type. We first investigate the effects of different attention designs. We experiment with two types of local attention introduced in Sec. 3.1, namely shifted-grid attention and neighborhood attention [1]. Then, to validate the effectiveness of our proposed grouped vector attention (denoted as "GVA"), compare it with the commonly-used multi-heal self-attentin (denoted as "MSA"). We use the vanilla position encoding in PTv1 [i] and our proposed partition- based pooling scheme in all of the experiments in Table 4. It shows neighborhood attention performs significantly better than shifted- grid attention, indicating that the neighborhood attention is better suited for point clouds which are non-uniformly distributed. Moreover, our proposed grouped vector attention consistently outperforms the commonly-used multi-head self-attention with both shifted-grid attention and neighborhood attention. So our grouped vector attention multi-head self seff-attention. The comparison between GVA and MSA indicates the effectiveness of the learnable parameters in the grouped linear layer of the grouped weight encoding in Sec. 3.2.
only more efficient, but also more effective, than
Weight encoding. We study the effects of different weight encoding functions w in Table 6. The weight encoding functions introduced in Sec. 3.1 and Sec. 3.2, and different attention mechanisms adopt different weight encoding functions. We use the vanilla position encoding in PTv1 [1] and our proposed grid pooling scheme in all of the experiments in Table 6. We experimented with the following weight encoding functions: (1) The weight encoding for multi-head scalar attention Eq. 4, denoted "MSA". (2) Weight encoding
linear layer denoted "I.". (3) The grouped linear layer, which is in Eq. 5, denoted as "GI." (4) The linear layer followed by batch normalization, activation, and another linear layer, denoted "L+N+A+L". (5) The grouped linear layer, followed by batch normalization, activation, and a linear layer, denoted as "GL+N+A+L". (5) is also the grouped weight encoding function used for our grouped vector attention, introduced in Eq. Results in Table 6 demonstrate that our grouped weight encoding function outperforms other compared designs. Specifically, comparing (1), (3) and (5), GL slightly outperforms MSA but adding additional inter- sieerggropp information exchange combined with proper normalization and activation can boost performance to be better than MSA. Moreover, the comparison between (5) and (4) and comparison between (3) and (2) both indicate that our grouped linear layer outperforms the naive linear layer, even though the grouped linear layer has y times fewer parameters and requires less computing than the linear layer.

---

Table Shape classification on ModelNet40.
Table 4: Attention type ablation.
Method
mAce (%)
OA (%)
Local Type
Mechanism Type
mloU (%)
PointNct [19]
86.0
89.2
MSA
71.6
Shifted-Grid
PointNet++ [4]
91.9
GVA
72.5
PointCNN [28]
88.1
92.5
MSA
73.9
PointConv [29]
92.5
Neighborhood
GVA
75.0
KPConv [5]
92.9
DGCNN [47]
90.2
92.9
RS-CNN [48]
92.9
Table 5: Pooling method ablation.
PointASNL [31]
92,9
DensePoint [49]
93.2
Pooling Method
Pooling Ratio
Grid Size
mioU
PosPool [50]
93.2
Multipliers
(%)
GBNet [51]
91.0
93.8
1/4
74.4
PCT [21]
93.2
FPS
1/6
72.9
PA-DGC [43]
93.9
-1 -1/4
[x3.0, x2.0, x2.0, x2.0]
75.2
CurveNet [52]
94.2
-1 1/4
[x4.0, x2.0, x2.0, x2.0]
75.0
Grid
PTv1 [1]
90.6
93.7
-1/6
[x3.0, x2.5, x2.5, x2.51
75.4
PTv2 (ours)
91.6
94.2
1/6
[x4.0, x2.5, x2.5, x2.5]
74.7
Table Weight encoding ablation.
Table Module design ablation.
mloU (%)
PE Mul
Grid Pool
Map Unpool
mloU
ID
Weight encoding
ID
GVA
(%)
MSA
73.9
72.3
L
73.8
II
V
73.8
(3) GL
74.1
III
V
74.4
(4) L+N+A+L
74.7
IV
74.9
(5) GLAN+A+L (ours)
75.0
Pooling methods. Sec. 3.4 we discuss the potential limitations of the sampling-buss -bsti pooling in PTv1 and propose new pooling and unpooling scheme based non-overlapping partitions. We also name a simple and effective grid-base implement of our partition- pooling as grid pooling. To further examine the superiority of method, experiment with different molling-unpoolng schemes in Table
For our partition-baat based pooling implemented by grid, the base grid size is 0.02 meters, which is identical to the voxelization grid size during data pre-processing. The grid size multipliers are the grid size expansion ratio over the previous pooling stage. For example, [.4.0, x 2.0, x2.0, x2.0] means that the grid sizes are: [0.08, 0616, 0.32, 0.64] meters, respectively. We choose relatively large value for initial grid sizes (x3.0 and x x..0) to provide sufficiently large receptive fields, which is analogous to the common practice in image transformers [6]. For subsequent pooling stages, we observe that x2.0 grid size increase results in approximate pooling ratio of for the point cloud, while x2.5 grid size increase results an approximate pooling ratio of 6. We choose the same sampling ratio of 4 and 6 for sampling-base pooling to ensure fair comparison.
The results Table 5 illustrate that our partition- based pooling achieves higher mloU than the sampling-basse based method. For sampling-bee -baedd pooling with farthest point sampling, the performance decreases significantly when the sampling ratio increases from 4 However, for our partition- based pooling implemented by grid, we observe that initial grid size and subsequent grid size multipliers do not significantly affect the overall performance, so we can use larger grid sizes to reduce the number of points in each stage to save memory.
Module design. We ablate different modules introduced our PTv2: grouped vector attention (VGA), position encoding multiplier (PE Mul), partition-base pooling implemented by grid (Grid Pool), and partition map unpooling (Map Unpool) and the results are illustrated in Table The model adopts in Experiment is PTv1 [1], which serves as a baseline result of our design. Benefiting from structural parameter adjustments and better data processing, which are also shared with rest of the experiments, our baseline result increased from 70.6% to 72.3%. Experiment II to V add each of our proposed components in turns, gradually increasing our baseline result to 75.4% The increasing mIOU indicates the effectiveness of each component.

---

Table 8: Model performance and amortized latency with different pooling methods.
FPS-KNN [4]
Grid-KNN [5]
Grid pooling (ours)
Pooling Rate
1/4
1/6
-1/4
~1/
-1/4
-16
Time (ms)
1007
785
318
266
mloU (%)
74.4
72.9
74.1
73.4
75.2
75.4
Table 9: Model parameters and amortized latency of several networks.
4
6
PTv1 ① + GVA (L)
! + GVA (GL)
! GVA (GL-N-A-L)
4 + GP
PEM
Params (M)
11.4
9.8
9.6
9.6
9.6
12.8
Time (ms)
1023
951
971
220
266
mloU (%)
72.3
73.0
73.2
74.2
75.0
75.4
4.4 Model Complexity and Latency
We further conduct model complexity and latency studies examine the superior efficiency of several design in our work. We record the amortized forward time for each scan in the ScanNet v2 validation set with batch size 4 on single TITAN RTX.
Pooling methods. Table shows the forward time and mIoU of PTv2 with different pooling methods and pooling ratios. We compare our pooling method with two classical sampling-bee baaiinggg pooling methods: FPS-KNN and Grid-kNN. FPS-KNN pooling [4, 1] uses farthest point sampling (FPS) to sample specified number of points and then query k nearest neighbor points for pooling. We call the pooling method in Strided KPConv [5] Grid-kNN pooling, as it uses a uniform grid to sample points and then applies the kNN method to index neighbors. This leads to uncontrollable overlaps of the pooling receptive fields. As shown in the table, our grid pooling method is not only faster but also achieves higher mloUs.
Module design. Table summarizes comparisons of model complexity, time consumption, and evaluation performances on ScanNet v2 validation set. Meanwhile, drop the first batch of forwarding time for GPU preparation. In Table 9, GVA refers to grouped vector attention. L refers to grouped weight encoding implemented by single Linear. GL refers to grouped weight encoding implemented by a Grouped Linear. GL-N-A-L refers to the grouped lincar layer, followed by batch normalization, activation, and a linear layer grouped weight encoding function. GP refers to partition- partition-based pooling implemented by grid. PEM refers to the position encoding multiplier. To ensure fair comparison, PTv1 is to be the same depth and feature dimensions our model architecture of PTv2. By comparing experiment (1 and 2. we can study the effect of GVA. The same spirit goes for experiment G ④ and (), where each experiments adds one additional module, so that we can study the effect of the added module respectively.
Comparing experiments !, 2. Q. and ④ the introduction of grouped vector attention (GVA) with grouped weight encoding dramatically improves the model performance and slightly reduces exccution time. The comparison between 4 and (5) indicates that the grid pooling strategy can significantly speed up the network and further enhance the generalization ability of our model. Position encoding multiplier is the only design that increases the number of model parameters, but experiment c demonstrates its effectiveness in improving performance. Meanwhile, our model is still lightweight compared voxel-basee backbones, such as MinkUNet42 [18] with 37.9M parameters.
Conclusion
We propose Point Transformer V2 (PTv2), a powerful and efficient transformer-basee backbone for 3D point cloud understanding. Our work makes several non-trivial improvements upon Point Transformer V1 [1], including the grouped vector attention, improved position encoding, and partition- based pooling. Our PTv2 model achieves state-of-the-art performance on point cloud classification and semantic segmentation benchmarks.
Acknowledgements
This work is supported in part by HKU Startup Fund and HKU Seed Fund for Basic Research. We also appreciate the supporting of computing resources by SmartMore Corporation.
10

---

References
[1] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021. 1,2,3, 6, 7, 8, 9. 10, 14,
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In CVPR, 2020. 1, 3,4 4
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrIPS, 2017. 1,
[4] Charles Qi, Li Yi, Hao Su, Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NeurlPS, 2017. 2, 6, 7, 9, 10, 15, 16
[5] Hugues Thomas, Charles Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In ICCV, 2019. 2, 6, 7, 9, 10, 15, 16
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. ICLR, 2021.
2, 4,
[7] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV, 2021, 2,
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In ICCV, 2021. 2
[9] Xiaoyi Dong Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. urXiv:2107.00652 2021. 2
[10] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 2
[11] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In ICCV, 2015.
[12] Bo Li, Tianlei Zhang, and Tian Xia. Vehicle detection from 3d lidar using fully convolutional network. In RSS, 2016.
[13] Xiaozhi Chen, Huimin Ma. Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In CVPR, 2017. 2
[14] Alex H Lang, Sourahh Vora, Holger Caesar, Lubing Zhou, Jiong Yang. and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
[15] Daniel Maturana and Sebastian Scherer. Voxnet: 3d convolutional neural network for real-time object recognition. In IROS, 2015. 2
[16] Shuran Song. Fisher Yu, Andy Zeng. Angel Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In CVPR, 2017. 2
[17] Benjamin Graham, Martin Engeleke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR, 2018. 2,7
[18] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, 2019. 2, 6, 7, 10
[19] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017. 2, 7.9
[20] Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia. Pointweh: Enhancing local neighborhood features for point cloud processing. In CVPR, 2019. 2.7
[21] Meng-Har Hao Guo, Jun- Xiong Cai, Zheng-Ning Ning Liu, Tai- Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 2021. 2, 4.9.

---

[22] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, 2021. 4
[23] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022. 4
[24] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. CVPR, 2022.
[25] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang. Hang Su, Jun Zhu, Lionel M. Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.0366 2022. 4
[26] Angela Dai and Matthias NieBner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmenta- tion. In ECCV, 2018.7 7
[27] Gaku Narita, Takashi Seno, Tomoya Ishikawa, and Yohsuke Kaji. Panopticfusion: Online volumetric semantic mapping at the level of stuff and things. In IROS, 2019. 7
[28] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x- trannfooreed points. NewrIPS, 2018. 7.9
[29] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In CVPR, 2019. 7,9
[30] Hung-Yuse Yueh Chiang, Yen-Liang Lin, Yuch-Cheng Liu, and Winston H Hsu. A unified point-based framework for 3d segmentation. In 3DV, 2019. 7
[31] Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui. Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling. In CVPR, 2020. 7.9
[32] Huan Lei. Naveed Akhtar, and Ajmal Mian. Seggen: Efficient 3d point cloud segmentation with fuzzy spherical kernel. In CVPR, 2020. 7
[33] Qingyong Hu, Bo Yang. Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In CVPR, 2020.
[34] Zeyu Hu, Mingmin Zhen, Xuyang Bai, Hongho Fu, and Chiew-lan Tai. Jsenet: Joint semantic segmentation and edge detection network for 3d point clouds. In ECCV, 2020.
[35] Feihu Zhang, Jin Fang. Benjamin Wah, and Philip Torr. Deep fusionnet for point cloud semantic segmenta- tion. ECCV, 2020.
[36] Lyne Tchapmi. Christopher Choy, Iro Armeni, Jun Young Gwak, and Silvio Savarese. Segcloud: Semantic segmentation of 3d point clouds. In 3DV, 2017. 7
[37] Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian- Zhou. Tangent convolutions for dense prediction 3d. In CVPR, 2018. 7
[38] Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, and Jiaya Jia. Hierarchical point-edge interaction network for point cloud semantic segmentation. ICCV, 2019. 7
[39] Lei Wang, Yuchun Huang, Yaolin Hou, Shenman Zhang, and Jie Shan. Graph attention convolution for point cloud semantic segmentation. In CVPR, 2019. 7
[40] Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point clouds with self-attention and gumbel subset sampling. In CVPR, 2019. 7
[41] Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. Deep parametric continuous convolutional neural networks. In CVPR, 2018. 7
[42] Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In CVPR, 2018. 7
[43] Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. In CVPR, 2021. 7.9
[44] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 6, 7, 14
12

---

[45] Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang. Ioannis Brilakis, Martin Fischer, and Silvio Savarese.
3d semantic parsing of large-scale indoor spaces. In CVPR, 2016. 6, 14
[46] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang. Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: deep representation for volumetric shapes. In CVPR, 2015. 6, 7, 14
[47] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph for learning on point clouds. TOG, 2019. 9
[48] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. Relation-shape convolutional neural network for point cloud analysis. In CVPR, 2019. 9
[49] Yongcheng Liu, Bin Fan, Gaofeng Meng. Jiwen Lu, Shiming Xiang, and Chunhong Pan. Densepoint: Learning densely contextual representation for efficient point cloud processing. In ICCV, 2019. 9
[50] Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look local aggregation operators in point cloud analysis. In ECCV, 2020. 9
[51] Shi Qiu, Saced Anwar, and Nick Barnes. Geometric hack-projection network for point cloud classification. TMM, 2021.9
[52] Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. Walk in the cloud: Learning curves for point clouds shape analysis. In 2021, 9
2021.
13


Reviews:
[{"review_id": "yyj9uAUCqxP", "reviewer": "Reviewer_EpHU", "summary": "The paper propose grouped vector attention (GVA) module, which allows information to be exchanged within and between attention groups through a new weight encoding layer. It enhances the location encoding by adding a multiplier to the relational vector. And replaces the farthest-point-sampling-based downsampling method and the interpolation-based upsampling method used in PTv1 with an efficient grid upsampling method of its own setting.", "questions": "1) I hope the authors can add a visual explanation of the results of the grouped vector attention module that they used in the ablation experiments.\n2) The proposed Position Encoding Multipler seems reasonable. I want to know the impact of the Multipler on the overall network params, inference cost time and downstream task accuracy compared with the encoding scheme $\\delta = \\theta(p_i - p_j)$ in PTv1[1].\n3) The authors keep emphasizing that their grid upsampling is more efficient than the farthest-point-sampling-based downsampling method and the interpolation-based upsampling method. However, I only see the metric results for the downstream task. And only in \\textbf{Appendix B. Table 4}, there are a few quantitative results related to cost time and params (I want to see a more detailed comparison of the grid upsampling and downsampling method with PTv1[1], e.g. different \\textbf{grid sizes})\n4) I don't see any relevant explanation of network loss in the paper and the supporting material. I hope the authors can give detailed loss settings for their experiments on downstream tasks.\n\nIn the current version, there are some issues as well. I look forward to the response by the authors. For now, I would recommend a borderline accept rating for the paper.\n\n[1]Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021.", "limitations": "Yes, I don't see any potential negative social impact of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Originality: The main idea of the proposed approach is to use the transformer. To my limited knowledge about it, it lacks a little novelty, but the originality is reasonable.\n- Quality：While the approach seems reasonable and the experimental results look promising, I have the following concerns (See Questions) about the paper.\n- Clarity：This manuscript is clearly written.\n- Significance：There are not many works in the field of point cloud feature learning by transformer, so I think this paper makes a positive contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mjKUBIIAJz", "reviewer": "Reviewer_N2zu", "summary": "Summary:\n\n---\n- Point Transformer V2 (PTv2) proposes a new attention mechanism called group vector attention (GVA) that is more efficient than its multi-head self-attention (MHSA) and vector attention of Point Transformer (PTv1) [[1]] counterparts. The effectiveness of GVA is then tested on the tasks of 3D semantic segmentation and 3D shape classification where it achieves competitive performance.\n\n- PTv2 proposes a novel positional encoding scheme that introduces a multiplier $\\delta_{mul}$ to the relation vector in addition to the bias that was introduced in PTv1 $\\delta_{bias}$.\n \n- PTv2 proposes to introduce grid fusion downsampling and grid mapping upsampling as more effective, efficient, and more spatially aligned alternative to furthest point downsampling and trilinear interpolation upsampling used in PTv1.\n\n[1]: https://openaccess.thecvf.com/content/ICCV2021/papers/Zhao_Point_Transformer_ICCV_2021_paper.pdf\n", "questions": "Questions and Suggestions:\n\n---\n- Is there a particular reason why PTv2 does not compare against Stratified Transformer[[2]] but does compare to another CVPR2022 work titled _\"Fast Point Transformer\"_[[5]]? [[2]] shows that a vanilla MHSA is effective for 3D discriminative tasks and demonstrates the benefits of using shifted-grid locality type for long range contextual information aggregation.\n\n- I would like to see an ablation where the PTv1 architecture is fixed and only the vector attention is replaced with the proposed GVA. I think this is a more fair comparison and a real test for the effectiveness of the proposed GVA.\n\n- I would like to see a comparison of the final points/second throughput against PTv1 and Stratified Transformer.\n\n- I would like a clarification from the authors on the novelty of the used downsampling and upsampling grid based methods.\n\nI am willing to change my rating if my questions and concerns were addressed reasonably in the rebuttal.\n\n[2]: https://openaccess.thecvf.com/content/CVPR2022/papers/Lai_Stratified_Transformer_for_3D_Point_Cloud_Segmentation_CVPR_2022_paper.pdf \n\n[5]: https://openaccess.thecvf.com/content/CVPR2022/papers/Park_Fast_Point_Transformer_CVPR_2022_paper.pdf", "limitations": "Even though it is not critical, the paper does not include shape-part segmentation experiments on ShapeNet part or PartNet.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n---\n\n- The proposed GVA is more efficient than the vector attention proposed in PTv1 and MHSA while still maintaining competitive performance on semantic segmentation benchmarks such as S3DIS Area 5 and ScanNet test set and shape classification on ModelNet40.\n\n- The paper provides a thorough ablation study most important of which are attention type (GVA vs MHSA) and neighborhood (local vs shifted-grid). Table 7 highlights the effectiveness of the proposed GVA and positional encoding multiplier.\n\nWeaknesses:\n\n---\n\n- PTv2 does not compare to CVPR2022 work titled _“Stratified Transformer for 3D Point Cloud Segmentation”_ [[2]] that is currently the state-of-the-art point-based method on S3DIS Area 5 benchmark with 72 mIoU and reaches 74.7 mIoU on ScanNet test set as per the ScanNet benchmark website (however paper reports 73.7 mIoU). The work uses vanilla multi-head self-attention (MHSA) and uses shifted grid to aggregate information beyond KNN local neighborhood. [[2]] also introduces positional encoding multiplier but in a much more involved scheme. Having said that, PTv2 does include an interesting ablation study in section 4.3 on the attention type used (MHSA vs GVA) and locality type (Shifted-Grid vs Neighborhood) that supports the authors’ choices.  \n\n- The comparison with PTv1, in its current form, is not fair because of the architectural differences. The PTv2 uses an encoder with block depths [2,2,6,2] while PTv1 uses [1,1,1,1]. I think a better assessment of the effectiveness of the proposed GVA would be to fix the PTv1 encoder-decoder architecture (including block depths, positional encoding, embedding dimensions, and downsampling/upsampling methods), and just replace the vector attention operator with the proposed GVA.\n\n- The paper claims that the proposed GVA is more efficient. It is evident from the mathematical elaboration in section 3.2 and Table 4 of supplementary material that GVA is more efficient than vector attention of PTv1 and MHSA. However, I would like to see the final points/second throughput of the network as compared with PTv1 and Stratified Transformer (comparison with Stratified Transformer might be tricky because they use CUDA kernels for their attention mechanism).\n\n- The downsampling and upsampling grid based methods are not novel. They were first introduced in KPConv[[3]] and used by later works such as CloserLook3D[[4]]. Maybe the authors meant to say that they introduced these sampling methods to PTv1 that uses furthest point sampling for downsampling, and trilinear interpolation for upsampling.\n\n[2]: https://openaccess.thecvf.com/content/CVPR2022/papers/Lai_Stratified_Transformer_for_3D_Point_Cloud_Segmentation_CVPR_2022_paper.pdf \n\n[3]: https://openaccess.thecvf.com/content_ICCV_2019/papers/Thomas_KPConv_Flexible_and_Deformable_Convolution_for_Point_Clouds_ICCV_2019_paper.pdf\n\n[4]: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123680324.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XIokL4a202", "reviewer": "Reviewer_KrWR", "summary": "This paper upgrades Point Transformer into its version 2 in three ways. First, group vector attention is designed and compared with other alternatives. Second, an additional multiplier is added to the position encoding. Third, a light-weight grid-based sampling is proposed. Point Transformer V2 achieves competitive results in point cloud semantic segmentation and classification tasks.", "questions": "Please see the Weaknesses section.\n", "limitations": "No additional limitations or potential negative societal impact.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-presented with clear illustrations such as Figure 2.\n2. The discussion on the  attention mechanism is interesting and insightful. I especially like the degenerate analysis.\n3. Solid improvements on fundamental point cloud tasks are obtained.\n\nWeaknesses:\n1. It is more complete to include a benchmark for part segmentation.\n2. Line 113, I do not think shifted-grid attention is a “common” practice especially the references are Swin Transformer V1 and V2. Besides, the shifted window technique that Swin Transformer employs essentially leads to overlapping windows. Author’s comments on using non-overlapping windows. \n3. The investigation in the position encoding multiplier is relatively insufficient, other than one row of ablation experiment. What does it really do?\n4. A complexity analysis of FPS vs grid-based sampling will be helpful.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9esCMUn_Rl9", "reviewer": "Reviewer_MnQR", "summary": "In this paper, the authors proposed Point Transformer V2, which is based on the previous state-of-the-art Point Transformer. Compared to PT V1, PT V2 introduces several improvements, including group vector attention, grouped weight encoding layer, and a novel sampling method. Experimental results on ScanNet V2, S3DIS, and ModelNet40 show demonstrate considerable improvements upon previous methods.", "questions": "Please see above weakness. ", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nI like the presentation and organization of this paper. \nExperimental results show significant improvement over previous work.\nThe grid upsampling/downsampling at the feature level is interesting.\n\nWeakness:\n1. One of my concerns is the Group Vector Attention, as presented in Sec. 3.2. From my understanding, it simply applies the group operation to the vector attention in PTV1. I don't think adding a group operation  (or using a group operation to replace MHA) is novel. \nThe authors claimed, \"GVA is a generalized formulation of VA and MSA\". This claim is trivial, and we can easily achieve this statement. Besides, when generalizing Group Attention to Multi-head Attention, it can be mathematically generalized, but the implementation may not be easily achieved (we may need pad 0 manually). This generalization is not only suitable for GVA, but is common in all group operations. In Eq. 4, the value should be learnable parameters instead of 1. The Grouped linear is still a simple group operation.\n\n2. In Sec. 3.3, I could not get the motivation of the additional multiplier, which further emphasizes (I would not say enhance) the positional information. Why authors introduce the multiplier to the relation vector? I would like to see a more detailed analysis or theoretical proof, not barren \"explanation\".\n\n-- For the Sec. 3.4 Grid-based Sampling Methods are more like feature-level voxel or grid sampling. I like this modification, but it is not that novel. \n\n3. About the experiments. The ModelNet40 and S3DIS datasets are relatively simpler, and the performance of models trained on these two datasets varies greatly (maybe because of the overfitting problem?). This is a common issue in the community. Also, ScanNetV2 is not that complex. I would suggest the authors report the mean-std on simple datasets and add more experiments on complex tasks, like 3d object detection. \n\n4. In Table 4, I would like to see an ablation study that uses max-pooling to replace attention, and this study can serve as a baseline. \n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept