PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences
Authors: Hehe Fan
Xin Yu
Yuhang Ding
Yi Yang
Mohan Kankanhalli
Venue: ICLR 2021 Poster (2021)
Keywords: Point cloud
spatio-temporal modeling
video analysis
action recognition
semantic segmentation
convolutional neural network

Abstract: Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension. Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences. In this paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences. The proposed PST convolution first disentangles space and time in point cloud sequences.  Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension.  Furthermore, we incorporate the proposed PST convolution into a deep network, namely PSTNet, to extract features of point cloud sequences in a hierarchical manner.  Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of PSTNet to model point cloud sequences.

Published as a conference paper at ICLR 2021
PSTNET: POINT SPATIO-TEMPORAL CONVOLUTION ON POINT CLOUD SEQUENCES
Hehe Fan', Xin Yu², Yuhang Ding', Yi Yang² & Mohan Kankanhalli'
School of Computing, National University of Singapore
PReLER, University of Technology Sydney
Baidu Research
ABSTRACT
Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension. Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences. In this paper, we propose point
(PST) convolution to achieve informative representations of point cloud sequences. The proposed PST convolution first disentangles space and time in point cloud sequences. Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension. Furthermore, we incorporate the proposed PST convolution into deep network, namely PSTNet, extract features of point cloud sequences in hierarchical manner. Extensive experiments on widely-used 3D action recogmition and 4D semantic segmentation datasets demonstrate the effectiveness of PSTNet model point cloud sequences.
INTRODUCTION
Modern robotic and automatic driving systems usually employ real-time depth sensors, such as LiDAR, to capture the geometric information of scenes accurately while being robust to different lighting conditions. A scene geometry is thus represented by a 3D point cloud, i.e., a set of measured point coordinates {x, =- Moreover, when RGB images are available, they are often used as additional features associated with the 3D points to enhance the discriminativeness of point clouds. However, unlike conventional grid based videos, dynamic point clouds are irregular and unordered in the spatial dimension while points are not consistent and even flow in and out over time. Therefore, existing 3D convolutions on grid based videos (Tran al., 2015; Carreira & Zisserman, 2017; Hara et al., 2018) are not suitable to model raw point cloud sequences, as shown in Fig. 1.
To model the dynamics of point clouds, solution is converting point clouds to sequence of 3D voxels, and then applying 4D convolutions (Choy et al., 2019) to the voxel sequence. However, directly performing convolutions quantization errors are inevitable during voxelization, which may restrict applications that require precise measurement of scene geometry. Another solution MeteorNet (Liu et al., 2019e) is extending the static point cloud method PointNet++ (Qi et al., 2017b) to process raw point cloud sequences by appending ID temporal dimension to 3D points. However, simply concatenating coordinates and time together and treating point cloud sequences as unordered 4D point sets neglect the temporal order of timestamps, which may not properly exploit the temporal information and lead inferior performance. Moreover, the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible. Treating them equally not conducive for network optimization. Besides, MeteorNet only considers spatial neighbors and neglects the local dependencies of neighboring frames. With the use of whole sequence length as its temporal receptive field, MetcorNet cannot construct temporal hierarchy. As points not consistent and even flow in and out of the region, especially for long sequences and fast motion embedding points in a spatially local area along an entire sequence handicaps capturing accurate local dynamics of point clouds.
voxel sequences require large amount of computation. Furthermore,
In this paper, we propose a point ppatio-tempore (PST) convolution to directly process raw point cloud sequences. As dynamic point clouds are spatially irregular but ordered in the temporal dimension,

---

Published as a conference paper at ICLR 2021
[o] Conventional Grid based Video Encoding
(b) Point Cloud Sequencee Encoding
ppoel
Teature
4
coorditate
4
01
Figure 1: Illustration of grid based and point based convolutions on spatio- spatio-temporal temporal sequences. (a) For grid based video, each grid represents a feature of a pixel, where C. L,, H and W denote the feature dimension, the number of frames, height and width, respectively. A 3D convolution encodes an input to an output size C' x L' H' x W', (b) A point cloud sequence consists of a coordinate part (3 x L x N) and a feature part (C x L x N), where N indicates the number of points in frame. Our PST convolution encodes an input to an output composed of coordinate tensor (3 x L' x N)) and feature tensor (C' L' x N'). Usually, L' S L and N' S N so that networks can model point cloud sequences in a spatio-temporally hierarchical manner. Note that points in different frames are not consistent, and thus it is challenging to capture the spatio-temporal correlation.
decouple the spatial and temporal information model point cloud sequences. Specifically, PST convolution consists of (i) a point based spatial convolution that models the spatial structure of 3D points and (ii) a temporal convolution that captures the temporal dynamics of point cloud sequences. In this fashion, PST convolution significantly facilitates the modeling of dynamic point clouds and reduces the impact of the spatial irregularity of points on temporal modeling. Because point cloud sequences emerge inconsistently across frames, it is challenging to perform convolution them. To address this problem, we introduce a point tube to preserve the spatio-tempon local structure.
To enhance the feature extraction ability,, we incorporate the proposed PST convolution into spatio- temporally hierarchical network, namely PSTNet. Moreover, we extend our PST convolution to a transposed version address point-level prediction tasks. Different from the convolutional version, the PST transposed convolution is designed to interpolate temporal dynamics and spatial features. Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of the proposed PST convolution and the superiority of PSTNet in modeling point clouds sequences. The contributions of this paper are fourfold:
To the best of our knowledge, we are the first attempt to decompose spatial and temporal information in modeling raw point cloud sequences, and propose a generic point based convolutional operation, named PST convolution, to encode raw point cloud sequences.
We propose PST transposed convolution to decode raw point cloud sequences via interpo- lating the temporal dynamics and spatial feature for point-level prediction tasks.
We construct convolutional neural networks based on the PST convolutions and transposed convolutions, dubbed PSTNet, to tackle sequence-level classification and point-level pre- diction tasks. To the best of our knowledge, our PSTNet is the first deep neural network to model raw point cloud sequences in a both spatially and temporally hierarchical manner.
Extensive experiments on four datasets indicate that method improves the accuracy of 3D action recognition and 4D semantic segmentation.
RELATED WORK
Learning Representations on Grid based Videos. Impressive progress has been made on generat- ing compact and discriminative representations for RGB/RGBD videos due to the success of deep neural networks. For example, two-stream convolutional neural networks (Simonyan Zisserman, 2014; Wang et al., 2016) utilize a spatial stream and an optical flow stream for video modeling. To summarize the temporal dependencies of videos, recurrent neural networks (Ng et al., 2015; Fan et al., 2018) and pooling techniques (Fan et al., 2017) are employed. In addition, by stacking multiple 2D frames into 3D tensor, 3D convolutional neural networks (Tran et al., 2015; Carreira & Zisserman, 2017; Tran et al., 2018; Hara et al., 2018) widely used to learn spatio-temporal representations for videos, and achieve promising performance. Besides, interpretable video or action reasoning methods (Zhuo et al., 2019; Fan al., 2021b) are proposed by explicitly parsing changes in videos.
2

---

Published as a conference paper at ICLR 2021
Statie Point Cloud Processing. Static point cloud analysis has been widely investigated for many problems, such as classification, object part segmentation, scene semantic segmentation (Qi et al., 2017a;;; Li et al., 2018b; Wu et al., 2019; Thomas et al., 2019; Wang et al., 2019), reconstruction (Dai et al., 2017; Yu et al., 2018 8) and object detection (Chen et al., 2017: 7; Qi et al., 2019). Most recent works aim directly manipulate point sets without transforming coordinates into regular voxel grids. Since a point cloud is essentially a set of unordered points and invariant to permutations of its points, static point cloud processing methods mainly focus on designing effective point based spatial correlation operations that do not rely on point orderings.
Dynamic Point Cloud Modeling. Compared with static point cloud processing, dynamic point cloud modeling is a fairly new task. Fast and Furious (FaF) (Luo et al., 2018) converts a point cloud frame into a bird's view voxel and then extracts features via 3D convolutions. MinkowskiNet (Choy et al., 2019) converts point cloud sequence into 4D occupancy grid and then applies 4D Spatio-Temporal ConvNets. PointRNN (Fan & Yang. 2019) leverages point based recurrent neural networks for raw point cloud sequence forecasting. MeteorNet (Liu al., 2019e) extends 3D points to 4D points and then appends temporal dimension to PointNet++ to process these 4D points. 3DV (Wang al., 2020) first integrates 3D motion information into regular compact voxel set and then applies PointNet++ to extract representations from the set for 3D action recognition. P4Transformer (Fan al., 202 employs transformer avoid point tracking for raw point cloud sequence modeling. Niemeyer et al. (2019) learned a temporal and spatial vector field for 4D reconstruction. Prantl al. (2020) learned stable and temporal coherent feature spaces for point based super-resolution. CaSPR (Rempe et al., 2020) learns to encode spatio-temporal changes in object shape from point clouds for reconstruction and camera pose estimation. In this paper, we propose a point based convolution to model raw point cloud sequences in spatio-temporall hierarchical manner.
3 PROPOSED POINT SPATIO-TEMPE
CONVOLUTIONAL NETWORK
In this section, we first briefly review the grid based 3D convolution operation as our point spatio- temporal (PST) convolution is motivated by it. Then, introduce how the proposed PST convolution extracts features from dynamic point clouds in detail. In order to address dense point prediction tasks, i.e., semantic segmentation, we develop PST transposed convolution. Finally, incorporate operations into deep hierarchical networks to address different dynamic point cloud tasks.
3.1 MOTIVATION FROM GRID BASED 3D CONVOLUTION
The power of convolutional neural network (CNNs) comes from local structure modeling by con- volutions and global representation learning via hierarchical architectures. Given an input feature map F E IRCxLxxx where C, L, H and W denote the input feature dimension, length, height and width, 3D convolution is designed to capture the spatio-temporal local structure of F. written as:
[122)
[1/2]
[2/3]
Σ
Σ
Σ
W
F(zti0->) S+
[K/2]
[=-11/
where W C'xxxxxxxxx is the convolution kernel, (1, h, w) is the kernel size, C" represents the output feature dimension and is matrix multiplication. The w(t)) E CR'' xC denotes the weight kernel position (k, i,j) and E+* (t + k, x y j). Usually, CNNs employ small kernel sizes, e.g., (3, 3). which are much smaller than the input size, to effectively model the local structure. To construct hierarchical CNNs, stride (> 1) is used to subsample input feature maps during the convolution operation. In this fashion, following convolutional layers will have relatively larger receptive fields.
((++,++)) [RCx1 denotes the feature of pixel at input position
3.2 POINT SPATIO-TEMPORA, (PST) CONVOLUTION
Let P. E 3 and F. E CN denote the point coordinates and features of the 1-th frame in a point cloud sequence, where N and C' denote the number of points and feature channels, respectively. Given point cloud sequence ([P1; F1),
P2: Fz). [Pt; FL]), the proposed PST convolution will
encode the sequence to ( (PP: F(, [P:: F. [PL: Fi,,), where and indicate the number of frames, and P E 333N'' and F C''N'' represent the encoded coordinates and features.
3

---

Published as a conference paper at ICLR 2021
3.2.1 DECOMPOSING SPACE AND TIME IN POINT CLOUD SEQUENCE MODELING
Because point clouds are irregular and unordered, grid based 3D convolution cannot be directly applied to point cloud sequences. As point cloud sequences are spatially irregular and unordered but temporally ordered, this motivates us decouple these two dimensions in order to reduce the impacts of the spatial irregularity of points on temporal modeling. Moreover, the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible. Treating them equally is not conducive for network optimization. By decoupling the spatio-temporal information in point cloud sequences, we not only make the spatio-temporal modeling easier but also significantly improve the ability to capture the temporal information. Therefore, our PST convolution is formulated as:
1//2]
Σ
Σ
L4k
k=- 1/2]
L/2]]
Σ
(S (1(6))8)))))
Feth p(ettwwthy.stt.)
where (z. 2) Ps. (82, Sy: 82) represents displacement and is spatial search radius. The convolu- tion kernel W E RC'xxxx x R is decomposed into spatial convolution kernel S E RR xCxl x R and a temporal convolution kernel T IRC xCC"xCm** x! x R and C. is the dimension of the intermediate feature. Moreover, because space and time are orthogonal and independent of each other, we further decompose spatial and temporal modeling as:
L1/21 Σ A=- 1/2
F F((
T
Σ
S(62.59.62). 8
t+k
(3)
where S RC R and T E RC (C''xxxX The above decomposition can also be expressed as applying temporal convolution first and then spatial convolution to input features. However, doing so requires point tracking to capture point motion. Because it is difficult to achieve accurate point trajectory and tracking points usually relies on point colors and may fail to handle colorless point clouds, we opt to model the spatial structure of irregular points first, and then capture the temporal information from the spatial regions. In this fashion, Eq. (3) can be rewritten
11/21
M (2.V.2)
Σ
Σ To TR-M
(Z..2) M
(4)
k=-11/2
The spatial convolution captures spatial local structures of point clouds in 3D space, while the temporal convolution aims to describe the temporal local dynamics of point cloud sequences along the time dimension. In order to capture the distributions of neighboring points, it is required to learn a convolution kernel S for all displacements. However, this impossible because point displacements are not discrete. To address this problem, we convert the kernel to function of displacements, defined as:
M ((
Σ (
f(162, 8y, 62):0) F 61
(5)
where f: RI3 R.....C XC is function of (8z, 8y. 82) parametrized by 0 to generate different RC xC according different displacements. There can be many ways to implement the f function. In principle, the f function should be both computation-effiee convolution is able to encode long sequences. Existing static point cloud convolution methods (Li et al., 201 8b; Wu et al., 2019; Wang et al., 2019) usually contains multilayer perceptrons (MLPs), which are computation and memory consuming. In this paper, we design lightweight implementation for f. which contains only few parameters. Specifically, we further decompose the f function as f(62, 5z); 0) 0a (6g, 6,)
and nemorr-efficienn so that PST
0.. where 0 0., 0.), 0₂ R x3 is a displacement transform kernel. O, E Cm xC a sharing kernel, (1. ... 1) E RIxC for broadcasting, and c is element-wise product. The sharing kernel O, is increase point feature dimension to improve the feature representation ability. The displacement kernel 0 is to capture spatial local structure based on displacements. In this fashion, f generates unique spatial kernel for each displacement so that the lightweight spatial convolution able to increase feature dimension while capturing spatial local structure like conventional convolutions.
4

---

Published as a conference paper at ICLR 2021
temporal stride =
temporalstride
[P atcchars
P22 1
Ps
P F
[Ps: Fsl
a @ Xox
pubsampling
anchors
anchors
point tube
(h)
poddatoug
MC
P1
M2
MG
171
MG
AG
padding
P'S
poolding
AP
MP
MF
MT
P11
pradling
temperal lorrnll size 3+
Htemporal kenmel sine Sa
H (emmppral lorrnell size 3
Figure 2: Illustration of the proposed point spatio-temporal (PST) convolution. The input contains L= 5 frames, with N points per frame. (a) Based on the temporal kernel size I 3. temporal stride s, = 2. and temporal padding 1, the 1st. 3rd, 5th frames are selected as temporal anchor frames. According to a spatial subsampling rate s, 4. 2 spatial anchor points are sampled by FPS in each anchor frame. The sampled anchor points are then transferred to the [ J nearest neighboring frames. point tube constructed with spatial search radius for the anchor points. (b) The spatial convolution encodes the local structure around each anchor point. (c) The temporal
convolution encodes the spatial features to a spatio-temporal feature. The original L x = 5 x sequence is encoded as L' x N' = 3 x 2 sequence.
3.2.2 POINT TUBE
Grid based 3D convolutions can be easily performed on regular conventional videos by sliding along the length, height and width. Because point cloud sequences are irregular and unordered in 3D space and emerge inconsistently across different frames, it is challenging perform convolution on them. To address this problem, we introduce point tube to preserve the spatio-temporal local structure. In contrast to pixel cubes in 3D convolution, in which pixels distributed regularly, point tubes dynamically generated based on the input sequences so that dense areas have more tubes than sparse ones. Specifically, the point tube is constructed as follows:
Temporal anchor frame selection. For an entire sequence of point clouds, we need to select some anchor frames to generate our tubes. Temporal anchor frames in a point cloud sequence are automatically selected based on temporal kernel size (!), temporal stride (st) and temporal padding (p), where 1 is set to an odd number so that an anchor frame located in the middle of point tube. Moreover, we set
to avoid selecting a padding frame an anchor frame.
Spatial anchor point sampling. Once a temporal anchor frame selected, we need to choose spatial anchor points that can represent the distribution of all the points in the frame. Given spatial subsampling rate 8s, this operation aims to subsample N points to N' = points. We employ the farthest point sampling (FPS) (Qi et al., 201 to sample points in each anchor frame. Point tubes are generated according to sampled anchor points.
Transferring spatial anchor points. 3D convolutions can effectively capture local changes within cube of kernel size (1, h, w) rather than tracking specific pixel. Following this idea, we propagate the positions of sampled anchor points to the neighboring frames without tracking, and they are regarded as the anchor points in those frames. Specifically, each anchor point is transferred to the [ nearest frames. The original and transferred anchor points form the central axis a tube.
Spatial neighbor. This step aims to find the spatial neighbors of every anchor point in each frame for performing spatial convolution. As indicated in Eq. (5), radius neighborhood is used to search neighbors within the tube slice, where local structure of points is depicted. Note that, padding is usually used in grid based convolution to align feature maps. However, our point based spatial convolution is not conducted on grids and thus spatial padding is not employed.
5

---

Published as a conference paper at ICLR 2021
By performing PST convolution on point tubes, network is able to capture the dynamic changes in local areas. The temporal kernel size I and spatial search radius r allow our PST convolution to capture temporal and spatial local structure, respectively. Frame subsampling (according to 8)) and point subsampling (according to 8₂) make our PSTNet be both temporally and spatially hierarchical. Global movement can be summarized by merging the information from these tubes in spatio-temporally hierarchical manner. An illustration of PST convolution is shown in Fig. 2.
3.3 POINT OPATIO-TEMPORE TRANSPOSED CONVOLUTION
After PST convolution, the original point cloud sequence is both spatially and temporally subsampled. However, for point-level prediction tasks, we need to provide point features for all the original points. Thus, we develop a PST transposed convolution. Suppose (IP); Fi), [P): F2l. ... ( F',1) is the encoded sequence of the original one [[ [P:: F2). [PI: F2l). PST transposed convolu- tion propagates features (Fi, ; F2..... ..... F') to the original point coordinates (Pı, P2,
Pt.. thus outputting new features (F"', FT,.... FL), where F. E <RC" xN and C" denotes the new feature dimension. To this end, PST transposed convolution first recovers the temporal length by temporal transposed convolution:
T T. (F.a)))
[M ((..e)) (-1/2 ... M ((....)) U-U/j
(6)
where T' E R xC" is the temporal transposed convolution kernel, and then increases the number of points by assigning temporal features to original points. Inspired by (Qi et al., 2017b), the interpolated features are weighted by inverse distance between an original point and neighboring anchor points:
F Fllla)))) = S'. 5116,,18 w(oz 52) M
wo.... 8y. 8z)
211(5 w(.,, dy. 6z)
166 òz)12
where S' E RCO x is a sharing kernel to enhance the interpolated features and change the feature dimension from C C", An illustration of PST transposed convolution shown in Appendix A.
3.4 PSTNET ARCHITECTURES
PSTNet for 3D action recognition. We employ 6 PST convolution layers and a fully-connected (FC) layer for 3D action recognition. In the 1st, 2nd, 4th, 6th layers, the spatial subsampling rate is set to 2 to halve the spatial resolution. Because point clouds are irregular and unordered, spatial receptive fields cannot be relatively enlarged by spatial subsampling. To address this problem, we progressively increase the spatial search radius. In the 2nd and 4th layers, the temporal stride is set to to halve the temporal resolution. In the 2-4 layers, the temporal kernel size is set to to capture temporal correlation. Temporal paddings are added when necessary. After the convolution layers, average and max poolings are used for spatial pooling and temporal pooling, respectively. Finally, the FC layer maps the global feature to action predictions.
PSTNet for 4D semantic segmentation. We use 4 PST convolution layers and 4 PST transposed convolution layers for 4D semantic segmentation. The spatial subsampling rate is set to 4 to reduce the spatial resolution in the 1-3 PST convolutions and 2 in the fourth PST convolution. Similar to 3D) action recognition, the spatial search radius progressively increases to grow spatial receptive fields along the network layers. In the 3rd PST convolution and 2nd transposed convolution, the temporal reduce and increase temporal dimension, respectively. Skip connections are
kernel size is set added between the corresponding convolution layers and transposed convolution layers. After the last PST transposed convolution layer, a ID convolution layer is appended for semantic predictions.
In addition, batch normalization and ReLU activation are inserted between PST convolution and transposed convolution layers. Our PSTNet architectures are illustrated in Appendix B.
EXPERIMENTS
4.1 3D ACTION RECOGNITION
To show the effectiveness in sequence-level classification, we apply PSTNet to 3D action recognition. Following (Liu et al., 2019e; Wang et al., 2020), sample 2,048 points for each frame. Point

---

Published as a conference paper at ICLR 2021
Table Action recognition accuracy (%) on the MSR-Action3D dataset.
Figure 3: Influence of temporal kernel size and (initial) spatial search radius on MSR-Acttio3D with 24 frames.
Method
Input
# Frames
Accuracy
Vieira et al. (Vicira et al., 2012)
depth
20
78.20
91.20
Kläser ef al. (Kläser et al., 2008)
depth
18
81.43
90.55
Actionlet (Wang et al., 2012)
skeleton
all
88.21
accuree P50
89
17,
PointNet++ (Qi et al., 2017b)
point 1
61.61
86
4
78.11
04-
8
81.14
82 80.000
(a) temporal kernel size
MeteorNet (Liu al., 2019c)
point
12
86.53
16
88.21
91.20
24
88.50
g 91.640.94
30,86
4
81.14
20.77
8
83.50
90.6
PSTNet (ours)
point
12
87.88
00.41
90.4
16
89.90
24
91.20
(b) spatial search radius
90.03
90.0
0.5
0.9
1.3
2.1
Figure 4: Visualization of PST convolution's output. Top: input point cloud sequences, where color encodes depth. Bottom: output of PST convolution, where brighter color indicates higher activation. Interestingly, PST convolution outputs high activation to salient motion. Best viewed in color.
cloud sequences split into multiple clips (with a fixed number of frames) inputs. For training, sequence-lever labels are used as clip-level labels. For evaluation, the mean of the clip-level predicted probabilities used as the sequence-level prediction. Point colors are not used.
4.1.1 MSR-ACTION3D
The
(Li et al., 2010) dataset consists of 567 Kinect depth videos, with 20 action categories and 23K frames in total. We the same training/test split previous works (Wang et al., 2012; Liu et al., 2019e). We conduct experiments with 10 times and report the mean.
Comparison with the state-of-the-art. We compare PSTNet with skeleton skeleton-based,, depth- basth and point-based 3D action recognition methods on this dataset. As shown in Table 1, the proposed PSTNet significantly outperforms all the state-of-the-ar methods, demonstrating the superiority of our PST convolution on feature extraction. Moreover, when encoding long point sequences, our spatio-temporally hierarchical PSTNet is more effective than the spatially hierarchical MeteorNet. Specifically, from 16 frames to 24 frames, MeteorNet only achieves a slight improvement of 0.29%, while our method increases the accuracy by 1.30%.
What does PST convolution learn? To investigate what PST convolution learns, we visualize the output of the middle layer (i.e., the 3rd layer) Fig. 4. As ReLU is employed as the activation function, larger values indicate higher activation. As expected, PSTNet outputs higher activation on moving areas, demonstrating that our PSTNet captures the most informative clues action reasoning.
4.1.2 NTU RGB+D 60 AND NTU RGB+D 120
The NTU RGB+D 60 (Shahroudy et al., 2016) is the second largest dataset for 3D action recognition. It consists of 56K videos, with 60 action categories and 4M frames in total. The videos captured
7

---

Published as a conference paper at ICLR 2021
Table 2: Action recognition accuracy (%) on the NTU RGB+D 60 and NTU RGB+D 120 datasets.
NTU RGB+D 60
NTU RGB+D 120
Method
Input
Subject
View
Subject
Setup
SkeleMotion (Cactano al., 2019)
skeleton
69.6
80.1
67.7
66.9
GCA-LSTM (Liu et al., 2017)
skeleton
74.4
82.8
58.3
59.3
FSNet (Liu et al., 2019b)
skeleton
59.9
62.4
Two Stream Attention LSTM (Liu et al., 2018)
skeleton
77.1
85.1
61.2
63.3
Body Pose Evolution Map (Liu & Yuan, 2018)
skeleton
64.6
66.9
AGC-LSTM (Si et al., 2019)
skeleton
89.2
95.0
AS-GCN (Li al., 2019)
skeleton
86.8
94.2
VA-fusion (Zhang et al., 2019)
skeleton
89.4
95.0
2s-AGCN (Shi et al., 2019b)
skeleton
88.5
95.1
DGNN (Shi et al., 2019a)
skeleton
89.9
96.1
HON4D (Oreifej & Liu, 2013)
depth
30.6
7,3
SNV (Yang & Tian, 2014)
depth
31.8
13.6
HOG (Ohn-Bar & Trivedi, 2013)
depth
32.2
22.3
Li ct al. (2018a)
depth
68.1
83.4
Wang et al. (2018)
depth
87.1
84.2
MVDI (Xiao al., 2019)
depth
84.6
87.3
NTU RGB+D 120 Baseline (Liu et al., 2019a)
depth
48.7
40.1
PointNet++ (appearance) (Qi al., 2017b)
point
80.1
85.1
72.1
79.4
3DV (motion) (Wang et al., 2020)
voxel
84.5
95.4
76.9
92.5
3DV-PoinINet++ (Wang et al., 2020)
voxel + point
88.8
96.3
82.4
93.5
PSTNet (ours)
point
90.5
96.5
87.0
93.8
using Kinect v2, with 3 cameras and 40 subjects (performers). The dataset defines two types of evaluation, i.e., cross- suoss-t and cross-view. The cross-subject evaluation splits the 40 performers into training and test groups. Each group consists of 20 performers. The cross- cross-view evaluation uses all the samples from camera for testing and samples from cameras and for training.
The NTU RGB+D 120 (Liu et al., 2019a) dataset, the largest dataset for 3D action recognition, is an extension of NTU RGB+D 60. consists of 1 114K videos, with 120 action categories and 8M frames in total. The videos captured with 106 performers and 32 collection setups (locations and backgrounds). Besides cross-subject evaluation, the dataset defines a new evaluation setting, i.e., cross-setup, where 16 setups are used for training, and the others are used for testing.
Comparison with state-of-the-art methods. As indicated in Table 2, PSTNet outperforms all the other approaches in all evaluation settings. Particularly, as indicated by the cross-setup evaluation on NTU RGB+D 120, PSTNet outperforms the second best 3DV-PointNett+ (Wang et al., 2020) by 4. 4.6%. Moreover, compared 3DV that extracts motion from voxels, PSTNet directly models the dynamic information of raw point cloud sequences and thus is efficient.
Computational efficiency. We provide running time comparison with the second best 3DV- PointNet++ (Wang et al., 2020). The average running time video is shown in Fig. 5. Experiments are conducted using Nvidia RTX 2080Ti GPU on NTU RGB+D Impressively, compared to 3DV-PointNet++. PSTNet reduces running time by about 2s, showing that PSTNet is efficient.
4.2 4D SEMANTIC SEGMENTATION
To demonstrate that our PST convolution can be generalized to point-level dense prediction tasks, apply PSTNet to 4D semantic segmentation. Following the works (Choy al., 2019; Liu et al., 2019e), we conduct experiments on video clips with length of 3 frames. Note that, although semantic segmentation be achieved from single frame, exploring temporal consistency would facilitate exploring the structure of scenes, thus improving segmentation accuracy and robustness to noise. The widely-used standard mean Intersection over Union (mloU) is adopted the evaluation metric.
Synthia 4D (Choy et al., 2019) uses the Synthia dataset (Ros et al., 2016) to create 3D video sequences. The Synthia 4D dataset includes 6 sequences of driving scenarios. Each sequence consists of 4 stereo
8

---

Published as a conference paper at ICLR 2021
Table 3: Semantic segmentation results on the Synthia 4D dataset.
Figure 5: Comparison of
recognition rumning time per video on NTU RGB+D 60.
Method
Input #Frms #Params (M) mloU (%)
3D MinkNet14 (Choy al., 2019) voxel
19.31
76.24
4D MinkNet14 (Choy et al., 2019) voxel
23.72
77.46
2205
CPU
2000
PointNct++ (Qi et al., 2017b)
point
0.88
79.35
meneg tome
MetcorNet (Liu et al., 2019c)
point
1.78
81.80
1500
1000
PSTNet (1
point
1.42
80.79
500
PSTNet (1 (t=33
point
1.67
82.24
3DV-PaitlNet+
(ours)
RGB-D images taken from the top of moving car. Following (Liu et al., 2019e), we use the same training/validationnesst split, with 19,8 0/88888/1/118888 frames, respectively.
As in Table 3, PSTNet (1 3) exploits the temporal information and outperforms the state-of- the-art. Moreover, our method saves 0. 0.M,M, relative 6% of parameters compared to the second best method MeteorNet (Liu et al., 2019e). We visualize few segmentation examples in Appendix N.
4.3 ABLATION STUDY
Clip length. Usually, information is not equally distributed in sequences along time. Short point cloud clips may miss key frames and thus confuse the models as noise. Therefore, as shown in Table increasing clip length (i.e., the number of frames) benefits models for action recognition.
Temporal kernel size. The temporal kernel size controls the temporal dynamics modeling of point cloud sequences. Fig. 3(a) shows the accuracy MSR-Action3D with different (. (a) When is set to 1, temporal correlation is not captured. However, PSTNet can still observe 24 frames and the pooling operation allows PSTNet capture the pose information of an entire clip. Moreover, some actions (e.g., "golf swing") can be easily recognized by certain pose, and thus PSTNet with / = 1 can still achieve satisfied accuracy. (b) When is greater than 1, PSTNet models temporal dynamics and therefore improves accuracy on actions that rely on motion trajectory reasoning (e.g., "draw x", "draw tick" and "draw circle"). (c) When / greater than the accuracy decreases. This mainly depends motion in sequences. Because most actions in MSR-Action3D are fast (e.g.. "high arm wave"), using smaller temporal kernel size facilitates capturing fast motion, and long- mangrange temporal dependencies will be captured in high-level layers. Since aim to present generic point based convolution operations, we do not tune the kernel size for each action but use the same size. When PSTNet is used in 4D semantic segmentation, 1.45% compared to PSTNet (1 = 1) that neglects temporal structure (shown in Table 3).
observe that PSTNet (1 3) improves mIoU by
Spatial search radius. The spatial search radius controls the range of the spatial structure be modeled, As shown in Fig. 3(b), using too small r cannot capture sufficient structure information while using large r will decrease the discriminativeness of spatial local structure for modeling.
CONCLUSION
In this paper, we propose an point spatio-temporal (PST) convolution to learn informative repre- sentations from raw point cloud sequences and PST transposed convolution for point-level dense prediction tasks. We demonstrate that, by incorporating the proposed convolutions into deep net- works, dubbed PSTNets, our method competent to address various point- tasks. Extensive experiments demonstrate that our PSTNer significantly improves the 3D action recognition and 4D semantic segmentation performance by effectively modeling point cloud sequences.
ACKNOWLEDGMENTS
This research is supported by the Agency for Science, Technology and Research (A* STAR)) under its AME Programmatic Funding Scheme (#A18A2h)) (AA1AA20004)).

---

Published as a conference paper at ICLR 2021
REFERENCES
Paul J. Besl and Neil D. McKay. A method for registration of 3-d shapes. JEEF Trans. Pattern Anal. Mach. Intell., (4(2):2992556 1992. doi: 10.1 10.1109/3..11199.
Carlos Caetano, Jessica Sena de Souza, François Brémond. Jefersson A. dos Santos, and William Rob- son Schwartz. Skelemotion: A new representation of skeleton joint sequences based on motion information for 3d action recognition. In 16th IEEE International Conference Advanced Video and Signal Based Surveillance, AVSS, 2019.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In CVPR, 2017.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, 2019.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias
Nießner. Scannet: Richly-annotated reconstructions of indoor scenes. CVPR, 2017.
Hehe Fan and Yi Yang. PointRNN: Point recurrent neural network for moving point cloud processing. arXiv, 1910.08287, 2019.
Hehe Fan, Xiaojun Chang, De Cheng. Yi Yang, Dong Xu, and Alexander G. Hauptmann. Complex event detection by identifying reliable shots from untrimmed videos. ICCV, 2017.
Hehe Fan, Zhongwen Xu, Linchao Zhu, Chenggang Yan, Jianjun Ge, and Yi Yang. Watching a small portion could be as good as watching all: Towards efficient video classification. In IJCAI, 2018.
Hehe Fan, Yi Yang, and Mohan S. Kankanhalli. Point 4d transformer networks for spatio-temporal modeling in point cloud videos. In CVPR, 2021a.
Hehe Fan, Tao Zhuo, Xin Yu, Yi Yang, and Mohan S. Kankanballi. Understanding atomic hand-object interaction with human intention. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2021b. doi: 10. 1109TCCST.22221335555553
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In CVPR, 2018.
Alexander Kläser, Marcin Marszalek, and Cordelia Schmid. A spatio-tempore descriptor based on 3d-gradients. In BMVC, 2008.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. Unsupervised learning of view- invariant action representations. In NeurIPS, 2018a.
Maosen Li. Siheng Chen, Xu Chen, Ya Zhang. Yanfeng Wang, and Qi Tian. struuuural graph convolutional networks for skeleton based action recognition. In CVPR, 2019.
Wanqing Li, Zhengyou Zhang, and Zicheng Liu. Action recognition based on bag of 3d points, In CVPR Workshops, 2010.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x-transformed points. In NeurlPS, 2018b.
Jun Liu, Gang Wang, Ping Hu, Ling-Yo Yu Duan, and Alex C. Kot. Global context-aware attention LSTM networks for 3d action recognition. In CVPR, 2017.
Jun Liu. Gang Wang, Ling- Yu Duan, Kamila Abdiyeva, and Alex C. Kot. Skeleton-based human action recognition with global context-aware attention LSTM networks. IEEE Trans. Image Processing, 27(4):1586-159) 2018. doi: 10. 109/TT1.2001.2782277
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C. Kot. Ntu rgb+d 120: large-scale benchmark for human activity understanding. IEEE Trans. Pattern Anal. Mach. Intell. 201 9a. doi: 10. 1..11099PPAMI 0010//9A.........222 20119229167777
10

---

Published as a conference paper at ICLR 2021
Jun Liu, Amir Shahroudy, Gang Wang. Ling- Yu Duan, and Alex C. Kot. Skeleton-based online action prediction using scale selection network. IEEE Trans. Pattern Anal. Mach. Intell., 2019b. doi: 10.1 (009//TAA..21199299999999999
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Prototype propagation networks (PPN) for weakly-supervised few-shot learning category graph. In IJCAI, 2019c.
Lu Liu, William Hamilton, Guodong Long, Jing Jiang, and Hugo Larochelle. universal representa- tion transformer layer for few-shot image classification. In ICLR, 2021.
Mengyuan Liu and Junsong Yuan. Recognizing human actions the evolution of pose estimation maps. In CVPR, 2018.
Xingyu Liu, Charles R. Qi, and Leonidas J. Guibas. Flownet3d: Learning scene flow in 3d point clouds. In CVPR, 2019d.
Xingyu Liu, Mengyuan Yan, and Jeannette Bohg. Meteornet: Deep learning dynamic 3d point cloud sequences. In ICCV, 2019e.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In ICLR, 2019f.
Wenjie Luo, Bin Yang, and Raquel Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with single convolutional net. In CVPR, 2018.
Joe Yue-Hei Ng, Matthew J. Hausknecht, Sudheendra Vjjyanarasimman, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, 2015.
Michael Niemeyer, Lars M. Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. ICCV, 2019.
Eshed Ohn-Bar and Mohan M. Trivedi. Joint angles similarities and HOG2 for action recognition. In CVPR Workshops, 2013.
Omar Oreifej and Zicheng Liu. HON4D: histogram of oriented 4d normals for activity recognition from depth sequences. In CVPR, 2013.
Lukas Prantl, Nuttapong Chentanez, Stefan Jeschke, and Nils Thuerey. Tranquil clouds: Neural
networks for learning temporally coherent features in point clouds. In ICLR, 2020.
Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017a
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas, Pointnet++: Deep hicrarchical feature learning point sets in metric space. In NeurIPS, 2017b.
Charles Ruizhongtai Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, 2019.
Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojeic, Srinath Sridhar, and Leonidas J. Guibas. Caspr: Learning canonical spatiotemporal point cloud representations. In NeurlPS, 2020.
Germán Ros, Laura Sellart, Joanna Materzynska, David Vázquez, and Antonio M. López. The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016.
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. NTU RGB+D: A large scale dataset for 3d human activity analysis. In CVPR, 2016.
Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Skeleton Ekeleton-based action recognition with directed graph neural networks. In CVPR, 2019a.
11

---

Published as a conference paper at ICLR 2021
Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-stream adaptive graph convolutional networks for sicleton-bascd action recognition. In CVPR, 2019b.
Chenyang Si, Wentao Chen, Wei Wang, Liang Wang, and Tieniu Tan. An attention enhanced graph
convolutional LSTM network for skeleton-based action recognition. In CVPR, 2019.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurlPS, 2014.
Hugues Thomas, Charles Ruizhongtai Qi, Jean-Ernaanue Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. In ICCV, 2019.
Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. CVPR, 2018.
António Wilson Vieira, Erickson R. Nascimento, Gabriel L. Oliveira, Zicheng Liu, and Mario Fermando Montenegro Campos. STOP: space-time occupancy patterns for 3d action recognition from depth map sequences. In Progress Pattern Recognition, Image Analysis, Computer Vision, and Applications 17th Iberoamerican Congress, CIARP, 2012.
Jiang Wang, Zicheng Liu, Ying Wu, and Junsong Yuan. Mining actionlet ensemble for action recognition with depth cameras. In CVPR, 2012.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
Pichao Wang. Wanqing Li, Zhimin Gao, Chang Tang, and Philip O. Ogunbona. Depth pooling based large-scale 3-d action recognition with convolutional neural networks. IEEE Trans. Multimedia, 1055)1105-- 2018. doi: 10. (09/TMM.2011.2888333333
Yancheng Wang, Yang Xiao, Fu Xiong, Wenxiang Jiang, Zhiguo Cao, Joey Tianyi Zhou, and Junsong Yuan. 3dv: 3d dynamic voxel for action recognition depth video. In CVPR, 2020.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph CNN for learning on point clouds. ACM Trans. Graph., 166 2019. doi: 10. 16./3333366...
Wenxuan Wu, Zhongang Qi, and Fuxin Li. Pointcony: Deep convolutional networks on 3d point clouds. In CVPR, 2019.
Yang Xiao, Jun Chen, Yancheng Wang, Zhiguo Cao, Joey Tianyi Zhou, and Xiang Bai. Action recognition for depth video using multi-view dynamic images. Inf. Sci., 480:287-304, 2019. doi: 00.10066/j.nss.200 12.050.
Xiaodong Yang and Yingli Tian. Super normal vector for activity recognition using depth sequences. In CVPR, 2014.
Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Pu-net: Point cloud upsampling network. In CVPR, 2018.
Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, and Nanning Zheng, View adaptive neural networks for high performance skeleton-basee human action recognition. IEEE Trans. Pattern Anal. Mach. Intell., 1163):1963-1978, 2019, doi: 10. 10.11///MM.222199296666.
Linchao Zhu and Yi Yang. Label independent memory for semi-supervised few-shot video classifica- tion. IEEE Trans. Puttern Anal. Mach. Intell., 2020. doi: 10. 0.1119//PAMMI.2000330757
Tao Zhuo Zhiyong Cheng, Peng Zhang, Yongkang Wong, and Mohan S. Kankanhalli. Explainable video action reasoning via prior knowledge and state transitions. In ACM Multimedia, 2019.
12

---

Published as a conference paper at ICLR 2021
A PST TRANSPOSED CONVOLUTION
temperaal leerndl size Se
Hetemgoosa Icernel size Se]
Homporral lorrne sizur
Se
discord
M
M'
M'SS 2
Aiscard
(a)
M
M
11 [er (
Mi (M Amta
P&s (
(b)
(
Py
Pg
P3
Pa
Pg
P2:
[Pg F3'
[ F4"
[pst Fg"
(c)
temporal
2
temporal stride =
Figure 6: IIlustration of the proposed PST transposed convolution. The input contains T,' = 3 frames, with N' points per frame. The PST transposed convolution is to generate new original point features for the original point cloud sequence, which contains L = 5 frames, with N = 8 points per frame. The temporal kernel size is 3 and the temporal stride is 8 2. (a) Temporal transposed convolution. (b) Temporal interpolation. (c) Spatial interpolation.
We illustrate an example of PST transposed convolution in Fig. Given a convolved sequence (IP); Fi1. [P: F2l, [P F)..). and its original coordinate sequence (P1, P2. Ps. Ps, Po), PST trans- posed convolution aims to generate new features (F F" E F according to the original point coordinates. First, with the temporal kernel size = 3, each input point feature is decoded to 3 features by a temporal transposed convolution. Second, 5 frames are interpolated in accor- dance with temporal stride s, 2, and the input points with the decoded features are assigned to interpolated frames. In this way, the temporal cross-correlation is constructed. Third, given the original coordinates of a point cloud frame (e.g., P1) and the corresponding interpolated frame (i.e., [P]: [M'] M])). ), we generate features (i.e., F!") for all the original point coordinates.
B PSTNET ARCHITECTURES
B.1 PSTNET FOR 3D ACTION RECOGNITION
Sesperer
spale 872
12 quii Xp1
qurai B0s
teppont..t Cureers 1836
Cusee 5 317
coald 8216
Ieaperai Bdboo
durnd
dolc Borare
PSTCawic 4.
ISTDev NC
C-66
Co 94
131
192
X
A.p
[14]
En L.y La [8.)]
446
- 126
Figure 7: Hierarchical PSTNet architecture for 3D action recognition.
As shown in Fig. 7, the architecture of PSTNet for 3D action recognition consists of 6 PST con- volutions, i.e., PSTConv1, PSTConv2a, PSTConv2b, PSTConv3a, PSTConv3b, PSTConv4, and a fully-connected (FC) layer. In PSTConv1, PST PSTConv3a and PSTConv4, the spatial subsampling rate is set to 2 to halve the spatial resolution. In PSTConv2a 2a and PSTConv3a, the temporal stride is set to to halve the temporal resolution. The ro denotes the initial spatial search radius. The temporal padding p is split as [p,, P2], where Pı and p2 denote the left padding and the right padding, respectively. After these convolution layers, average and max poolings are used for spatial pooling and temporal pooling, respectively. The FC layer is then used classifier.
13

---

Published as a conference paper at ICLR 2021
B.2 PSTNET FOR 4D SEMANTIC SEGMENTATION
spicil a
quad Vou
spate 37120
matak Vo54
ESTGRe 294.200
POTAA Tead
1-3 (34)
1a- [V1
C 139
Figure 8: Hierarchical PSTNet architecture for 4D semantic segmentation.
As shown in Fig. 8, we employ 4 PST convolutional layers, i.e., PSTConv1, PSTConv2, PSTConv3, and PSTConv4, and 4 PST transposed convolutional layers, i.e., PSTConyTransi, PSTConvTrane PSTCony Trans3, and PSTConv Trans4 for 4D Semantic Segmentation. The spatial subsampling rate set reduce the spatial resolution in each PST convolution layer. Following (Choy al., 2019; Liu et al., 2019c), we conduct 4D semantic segmentation on clips with the length of 3 frames, i.., fixed to In PSTConv3 and ASTConv'raas22 the temporal kernel size set 3. Skip connections are added from input to PSTConvTrann 1,ans,, PSTConv] to PSTConv Trans2, PSTConv2 to Trans3,, and PSTConv3 to OSTCCnyTTaanse After PSTCon'TTrane a 1D convolution layer appended for semantic predictions.
C IMPLEMENTATION DETAILS
To build a unified network and train the network with mini-batch, the number of spatial neighbors needs to be fixed. We follow existing point based works (Qi et al., 2017b; Liu al., 2019e) to randomly sample fixed number of neighbors. For 3D action recognition, the number is set to 9. For 4D semantic segmentation, we follow (Liu et al., 2019e) to set the number to 32. If the actual number of neighbors of point is less than the set one (e.g., 9), we randomly repeat some neighbors.
We train our models for 35 epochs with the SGD optimizer. Learning rate is set to 0.01, and decays with rate of 0.1 at the 10th epoch and the 20th epoch, respectively.
MSR-Action3D (Li al., 2010): Following (Liu et al., 2019e), batch size is set to 16 and frame sampling stride are set to and 1, respectively. We set the initial spatial search radius ro to 0.5.
NTU RGB+D /120 (Shahroudy et al., 2016; Liu et al., 2019a): Following (Wang et al., 2020), batch size is set to 32. We set the clip length, frame sampling stride and the initial spatial search radius r, to 23, 2 and 0.1, respectively.
Synthia 4D (Choy et al., 2019): Following (Liu et al., 2019e), batch size and frame sampling stride are set to 12 and 1, respectively. We set the spatial search radius ro to 0.9.
We have implemented our PSTNets using both PyTorch and PaddlePaddle, which have shown similar performance.
D INFLUENCE OF TEMPORAL KERNEL SIZE ON DIFFERENT ACTIONS
We show the influence of temporal kernel size on different actions in MSR-Action3D Fig. Clip length is 24.
When temporal kernel size / = 1, temporal correlation not exploited. However, PSTNet can leverage pose information for recognition, especially for actions whose poses are unique in the dataset. For example, PSTNet with = 1 can correctly recognize the "jogging" action because the action's 's pose is discriminative in the MSR-Action3D dataset. However, PSTNet with motion modeling (I 2 3) can distinguish similar actions such as "draw x", "draw tick" and "draw circle". The results support intuition that PSTNet effectively captures dynamics in point cloud sequences.
14

---

Published as a conference paper at ICLR 2021
00330005
W00100 7011
2
(a) draw
(b) draw tick
(c) draw circle
(d) high arm wave
hammer
080580235
16
18YOVE harelsi
kord
lame
temposi lardiiie
(I) hand catch
(g) high throw
(h) tennis swing
ii) bend
forward kick
lamparal
aerpera
tampori
side kick
(I) jogging
(m) tennis serve
(n) two hand wave
(0) golf swing
000000 5350882885
BCCATO
Inperd Sene size
(p) pick up & throw (q) horizont arm wave (r) forward punch
(S) hand clap
side-boxing
Figure 9: Influence of temporal kernel size on different actions of MSR-Action3D.
E
FEATURE VISUALIZATION
MeteorNet
PS (ours)
(a) MeteorNet
(b) PSTNet
Figure 10: Feature visualizations on MSR-Action31) using t-SNE. Each sequence is visualized as a point and sequences belonging the same action have the same color. PSTNet features are semantically separable compared to MeteorNet, suggesting that it learns better representations for point cloud sequences.
We qualitatively evaluate PSTNet's ability to encode point cloud sequence by visualizing the learned features. We compare PSTNet with the sate-of-thh-and art MeteorNet (Liu et al., 2019e). Features are extracted from the layer before classifier (FC). These features are then projected 2-dimensional space using t-SNE. As shown in Fig. 10, PSTNet feature more compact and discriminative than MeteorNet.
15

---

Published as a conference paper at ICLR 2021
F SYNTHIA 4D SEMANTIC SEGMENTATION RESULT DETAILS
We list the segmentation result for each class in Table The Synthia 4D dataset contains 12 categories. Our PSTNet achieves five best accuracies among them, demonstrating the effectiveness of PSTNet.
Table 4: Semantic segmentation result (mloU %) details on the Synthia 4D dataset.
Method
Frames Bldn Road Sdwik Fence Vegiltn
Pole
Car
TT. Sign Peusttm
Bicycl
Lane
Light (Aerrage
4 (Chay et al., 2019)
92.27
4D MinkNet14 (Choy et al,, 2019)
3
90.13
98.26 73.47
87.19
99.10
97.50 94.01
79.04
92.62
0.00
50.01
68.14
77.46
PoimNet=+ (Qi al. 2017b)
1
96.88
97.72
86.20
92.75
97.12
97.09
90.85
66.87
78.64
0.00
72.93
75.17
79.35
MeteorNet (l.iu al,, 2019c)
3
98.10 97.72
88.65
94.00
97.98
97.65
93.83
84.07
80.90
0.00
71.14
77.60
81.80
PSTNet (1 = 1)
3
96.32
98.07
85.40
94,66
97.16
97.51
94.83
76.65
76.99
0.00
75.39
76.45
80.79
PSTNet
97.61
75.71
77.28
G PST CONVOLUTION WITHOUT DISENTANGLING SPACE AND TIME
To confirm the effectiveness of our disentangling structure for spatio-temporal modeling, we perform 3D action recognition on MSR-Action3D with non-decomposing convolution. To combine space and time, 3D displacements and time differences are encoded together to generate convolution kernel. We evaluate our PSTNet with different clip lengths. As shown in Fig. I, the disentangling PST convolution achieves better accuracy than non-decomposing structure, demonstrating the effectiveness of disentangling space and time for point cloud sequence modeling.
w/o disentangling
w/ disentangling
accuracy 196)
86
84
82
80
8
12
16
24
clip length
Figure 11: Comparison of PST convolution with (w/) and without (w/o) disentangling structure MSR-Action3D.
H IMPACT OF DISPLACEMENT KERNEL AND SHARING KERNEL
In this paper, design lightweight implementation for the function in Eq. (5), i.e., f((5,, &y, 0z); = 0a (ōn, ōy, 52) 1 0., where 0₂ is a displacement transform kernel, 0, is sharing kernel. In this section, investigate the influence of these kernels. We conduct 3D) action recognition on MSR-Action3D, The experimental results are shown Table 5.
Table 5: Impact of displacement kernel and sharing kernel on 3D action recognition accuracy (%).
The MSR-Action3D dataset is used.
0
0,
16-frame
24-frame
58.25
61.67
86.35
87.46
/
89.90
91.20
16

---

Published as a conference paper at ICLR 2021
Without Os, the spatial convolution degenerates into M, (1.1))) Σ v O2 (6x, By, 6z)= In this way, only the points in the final layer of our PSTNet are used. Therefore, the accuracy decreases dramatically.
Without 0g. the spatial convolution becomes M, (....2) Σ ,6, O, 1 2,,66
In this fashion, the points neighborhood are treated equally. Point features are leveraged but their positions are ignored. Because spatial structure is not well captured, the accuracy decreases compared to the case where both 0g and 0g are used.
I COMPUTATIONAL EFFICIENCY AND MEMORY USAGE
In this section, we evaluate the computational efficiency and memory usage, i... the running time and number of parameters, of our method. We conduct 3D action recognition with a clip length of 16 on MSR-Action3D using Nvidia Quadro RTX 6000 GPU.
As shown in Table 6, the proposed PSTNet is 41.5% faster than MeteorNet (Liu al., 2019e). This because that MeteorNet is based on MLPs, which are less efficient than convolutions.
Table 6: Comparison of parameter and rumning time 3D action recognition accuracy (%). The MSR-Action3D dataset used. Clip length 16.
Method
Parameters (M) Running time per clip (ms)
Accuracy (%)
MeteorNet (Liu et al., 2019e)
17.60
54.56
88.21
PSTNet (lightweight)
8.44
31.92
89.90
PSTNet (with PointConv)
20.46
102.72
90.24
Second, we study the impact of the number of PSTNet layers on running time and parameter. As shown in Fig. 12, when PSTNet becomes deep, the number of parameters significantly grows. This is because, like most deep neural networks, the point feature dimension exponentially increases to improve the feature representation ability PSTNet, which needs more parameters. However, the running time does not increase significantly. This is because our PSTNet is spatio-temporally hicrarchical, points are exponentially reduced along both spatial and temporal dimensions, thus saving running time.
35
running time
+ parameter
other operations
30
8
25
Parameters (M)
convolution 39%
running
spotial neighbor search 22%
farthest paint sampling
1
number of PSTNet layers
Figure 12: Impact of the number of PSTNet layers on running time and parameter.
Figure 13: Running time proportion of main operations PSTNet.
Third, we investigate the running time of main operations in PSTNet, including convolution, farthest point sampling and spatial neighbor search. We show the running time proportion of these operations in Fig. 13.
Finally, because the f function in Eq. (5) can be implemented in different ways, we replace our lightweight spatial convolution with PointConv (Wu et al., 2019) investigate the influence of the spatial modeling in our PSTNet. As shown in Table 6, using PointConv only improves accraucy slightly compared lightweight spatial convolutions, but significantly increases parameters and
17

---

Published as a conference paper at ICLR 2021
running time. This is because PointConv utilizes MLP based operations to process points and their features, which are less efficient than our lightweight convolutions.
J IRREGULAR FRAME SAMPLING
Our PSTNet can also process irregularly sampled point cloud sequences by frame interpolation. In this section, randomly remove frames in sequences. Specifically, conduct 3D action recognition on MSR-Action with 24 frames. We randomly remove 8 frames from each clip. Then, replication and Iterative Closest Point (ICP) (Besl & McKay, 1992) to interpolate missing frames, respectively. We compare our PSTNet with MeteorNet (Liu et al., 2019e), which explicitly encodes time and does need interpolation.
Table 7: 3D action recognition MSR-Action3D with irregularly sampled frames. Clip length is originally 24 and then frames are randomly removed from each clip.
Method
Accuracy (%)
MeteorNet (Liu et al., 2019e)
88.40
PSTNet (replication interpolation)
90.24
PSTNet (ICP interpolation)
90.94
As shown in Table 7, PSTNet still achieves the best accuracy, indicating that PSTNet is able to model point cloud sequences with irregular frame sampling.
K SCENE FLOW ESTIMATION
To evaluate our method on the LiDAR data where points in a frame can be highly irregular, we apply method to scene flow estimation. We follow the setting proposed by MeteorNet (Liu et al., 2019e), Le., given a point cloud sequence, estimating a flow vector for every point in the last frame. However, because point tube is constructed according the middle frame, which is not applicable the last-frame scene flow estimation, we adapt temporal anchor frame selection and spatial anchor point transferring. Specifically, we select the last frame in each tube the anchor frame. Then, after spatial sampling, each anchor point is transferred to its previous nearest frames. Following MeteorNet, we first train our model on 20190 2019e), and then fine-tune the model on a KITTI scene flow dataset (Liu et al., 2019e). Point tracking used.
dataset according the synthetic method in (Liu al.,
Table 8: Scene flow estimation accuracy on the KITTI scene flow dataset.
Method
Input
Frames
End-Point
FlowNet3D (Liu et al., 2019d)
points
2
0,287
MetcorNet (Liu et al., 2019e)
points
0,282
PSTNet (ours)
points
0.278
As shown in Table 8, our PSTNet achieves promising accuracy scene flow estimation.
L MOTION RECOGNITION ON THE SYNTHETIC MOVING MNIST POINT CLOUD DATASET
In the real world, it is impossible to obtain point IDs. To evaluate our PSTNet in the scenario where points are correlated and can be accurately tracked across frames, conduct motion recognition on a synthetic Moving MNIST Point Cloud dataset. To synthesize moving MNIST digit sequences, we generation process similar to that described in (Fan & Yang. 2019). Each synthetic sequence consists of 16 consecutive point cloud frames. Each frame contains one handwritten digit moving and bouncing inside 64 x 64 area. The digits are chosen randomly from the MNIST dataset. We
18

---

Published as a conference paper at ICLR 2021
the same training/test split the original MNIST dataset. For each digit, sample 128 points.
Point order is maintained across frames so that tracking can be employed. We design 144 motions, including
nine initial locations:
eight velocities: (1, (-,,)),,,,),,,
(-2,2), (-2, -2), (2, -2).
kinds of digit distortion: scaling digits horizontally vertically, i.e., (0.4 0.05t 0.6) x (z Cz. + C2 or (0..4 - 0.05t| - 0.6) x (y cy) ) + Cy. where (Co, Cy) is the digit center and [6].
few motion examples shown in Fig. 14, Because motion and appearance are independent of each other, it is challenging to recognize motion while avoiding interference from digit appearance.
motion
motion 2
0
0
0
0
3
3
3
3
3
3
3
3
L;
4
4
4
4
4
4
4
4
5
$
5 5
>
5
Figure 14: Motion examples in the synthetic Moving MNIST Point Cloud dataset.
To exploit point correlation, anchor points and their neighbors are selected in the first frame, which are then propagated to other frames. this fashion, anchors and their neighbors are tracked across frames.
Table Motion recognition accuracy on the Moving MNIST Point Cloud dataset.
Method
Accuracy (%)
PSTNet (original)
97.15
PSTNet (with tracking)
98.74
As shown in Table 9, both the original PSTNet and the tracking based PSTNet achieve promising accuracy on the Moving MNIST Point Cloud dataset. Our original PSTNet achieves similar accuracy with the tracking based PSTNet in this simulated case, demonstrating that our PSTNet does not heavily rely on point IDs tracking.
M VISUALIZATION OF THE OUTPUT OF EACH PST CONVOLUTION LAYER IN PSTNET
We visualize the output of each layer in the PSTNet architecture trained on MSR-Action3D in Fig. 15. Because we use ReLU as the activation function, all outputs are greater than zero and large outputs indicate high activation. To visualize outputs, we squeeze output vectors to scalars via norm. We observe that:
For PSTConv1, because temporal kernel size I = 1, it does not capture the temporal correlation. this case, PSTConv] focuses on the appearance and therefore outputs high activation to performer contours.
PSTConv2a to PSTConv4. When modeling the spatio-temporal structure of point cloud sequence, PST convolutions mainly focus on moving areas so as to achieve informative representations of actions.
19

---

Published as a conference paper at ICLR 2021
The visualization results support intuition that the proposed PST convolution effectively captures dynamics of point cloud sequences.
N VISUALIZATION OF 4D SEMANTIC SEGMENTATION
We visualize segmentation results from the Synthia 4D dataset in Fig. 16 and Fig. 17. Our PSTNet can accurately segment most objects.
O LIMITATION
Like most deep neural networks, the proposed PSTNet relies large-scale labeled datasets for training. A potential improvement is to integrate some learning methods, egg., few-shot learning (Liu al., 2019c;f; Zhu & Yang, 2020; Liu al., 2021), into point cloud sequences modeling to reduce the reliance on human-annotated data.
20

---

Published as a conference paper at ICLR 2021
Visualization of the output of each layer in PSTNet. For the input point cloud sequence, color encodes depth. For the outputs, brighter color indicates higher activation. Input sequences consist of 24 frames. Due to the spatial subsampling 8, and the temporal stride Sr, points and frames progressively decrease along the network depth.
xt
Maswe
3
w/o tempora modeling
w/o temporal modeling
PSTConv1: N = 1024
PSTConv2a: N 512
PSTConv2b: 512
PSTConv3a: 256
PSTComv3b: N = 256
PSTConv4: = 128
PSTConv1: N = 1024
PSTConv2a: N 512
PSTConv2b: 512
PSTConv3a: V = 256
PSTConv3b: N 256
PSTConv4: N = 128
Input: N = 2048
Input: N = 2048
Figure 15:
21

---

Published as a conference paper at ICLR 2021
Figure 16: Visualization of semantic segmentation examples from Synthia 4D (I). Top: inputs. Middle: ground truth. Bottom: PSTNet predictions.
22

---

Published as a conference paper at ICLR 2021
TARK
TARK
PURK
Figure 17: Visualization of semantic segmentation examples from Synthia 4D (II). Top: inputs.
Middle: ground truth. Bottom: PSTNet predictions.
23


Reviews:
[{"review_id": "lnY3AUdv8g", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces point spatio-temporal convolutions, which are used for the feature extraction of point cloud sequences. A trainable kernel is used which is applied locally as a continuous convolution. An important aspect is that the temporal dimension is processed separately, with an additional convolution, instead of simply using a 4D convolution. The authors claim that in this way the network will achieve a better understanding of the dynamics of the input.\n\nThe presented method is very interesting in my eyes. I find the argumentation concerning the construction of the spatio-temporal conclusive and it is supported by the experiments. Together with the presented striding, based on farthest point sampling, and the transposed convolution, the method seems to be very versatile, as it was shown in the results. However, a few points are still not quite clear to me:\n* Why is striding used for the semantic segmentation test? Theoretically, this problem could be solved much easier without striding and transposed convolutions.\n* The results were mainly sequences where the points of the frames do not correlate. What about data where the points correlate, like in physical simulations? Would the presented point tubes make sense there?\n\nThe paper is written understandably and the evaluations are sufficient, IMO, to confirm the claims given by the authors. Maybe the following papers could be relevant as related work: Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics and Tranquil Clouds: Neural Networks for Learning Temporally Coherent Features in Point Clouds, since they also deal with point sequences. \nIn section 3.1 there is a small typo (... CNNs usually adopt small ~~kennel~~ **kernel** sizes ...), just like at the beginning of section 4.3 (Temporal ~~kennel~~ **kernel** size...).\n\nAll in all I find the method simple but promising. The paper seems well elaborated and I would tend to accept it.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "jkW1n7SJ8x8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to process the point cloud data in a convolution manner. The authors propose the PST convolution and deconvolution operations to handle the different tasks such as the classification and segmentation on point cloud. The extensive experiments verify the effectiveness of the proposed method and achieve state-of-the-art results on multiple benchmark datasets. Overall, the paper is well-written and organized. \n \nStrength:\n •\tIn this paper, the authors introduce a method named PST convolution which could directly handle the sequential point cloud data. The PST convolution could capture the spatio-temporal in a convolution manner. They decompose the spatial and temporal information and conduct convolution on them respectively. \n•\tIn order to handle both the abstraction level and point-wise level tasks, they build a complete convolution system including convolution and deconvolution. Also, the point tube concept is introduced here to preserve the spatio-temporal structure. \n•\tThe experiments are exhaustive and impressive. Especially the author adopts various kinds of demonstration to show the effectiveness. \n \n Weakness:\n •\tEven through the experiments results look pretty good, the novelty of this paper is still limited. In this paper, compared to the previous paper such as MeteorNet, the authors explicitly promotes the concept of capturing the spatial and temporal information. There is no significant difference with MeteorNet or even PointNet++. The MeteorNet also extracts the spatio-temporal information by chain-flow radius search. No matter the point tube or the anchor points extracted in the spatial and temporal domain, the main idea is finding a good neighbor for the current point. It should be described clearly in details for the novelty in the future revised version. \n•\tAlso, even the convolution adopted in this paper could make the point processing more convenient, I have a concern about the efficiency of the proposed method. The computation cost including stacked convolution layers, the FPS sampling, and the nearest neighbor search are all heavy. So the authors may provide some efficiency comparison in the future. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DThyUilwG-Q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new convolutional approach to directly process raw spatiotemporal (ST) point cloud data. The proposed point spatio-temporal (PST) convolution operates on \"point tubes\" and decouples space and time through a shared spatial convolution at each timestep, followed by a temporal convolution. It also introduces a transposed PST to enable point-wise predictions in an encoder-decoder framework (PSTNet). The presented experiments demonstrate the effectiveness of these convolutions by using PSTNet for action recognition and semantic segmentation on point cloud sequences, showing improvement over relevant recent work.\n\nStrengths:\n+ The approach is technically novel. Processing raw 4D point clouds of arbitrary length is a relatively new direction, and I have not seen a convolutional approach before.\n+ I like the idea of decoupling spatial and temporal convolutions. As the paper points out, this should allow better handling of varied spatial sampling over time.\n+ Though the ST neighborhood size (i.e. temporal kernel size and spatial search radius) must be manually set and tuned, these neighborhoods are defined and structured in a principled way similar to grid-based CNNs which could make tuning easier than in prior work (e.g. MeteorNet).\n+ Experiments show slight improvement over the most relevant SOTA MeteorNet and MinkowskiNet for action recognition and semantic segmentation from depth sequences. I also appreciate the analysis of sensitivity to ST neighborhood parameters and the informative visualization of learned features (Fig 4, 12, and 13).\n+ The paper is, for the most part, well-written and nicely presented (e.g. Fig 2). The detailed supplementary material and promised code release enabled reproducibility. \n\nWeaknesses:\n- The paper could better situate the proposed method in the context of relevant prior work (see \"Related Work\" below).\n- The proposed PST convolution is described as \"generic\", but as far as I can tell it would struggle with irregular temporal sampling. The architecture operates on ordered point cloud sequences but does not leverage timestamps, so there is an underlying assumption that point clouds are sampled at a fixed rate that does not change from training to test time.\n- Various claims and design decisions could be better supported through additional experiments; in particular, the choice to decouple space and time and the spatial kernel design (see \"Experiments\" below). \n- There is no discussion about the limitations of the proposed method in the paper.\n\nBased on these points and the ones detailed below, I initially lean towards accepting the paper. The proposed method is technically novel and principled, and the experiments are sufficient to show PSTNet can extract useful features. The ability to handle irregular timestamps and additional experiments to characterize the advantages of space/time decoupling would improve the paper, but their absence does not outweigh the contributions for me. My other concerns about related work and a discussion of limitations can be addressed in a revision.\n\nRelated work:\n- PSTNet should be compared and contrasted to MeteorNet in more detail since this is the most relevant prior work. Currently, MeteorNet is described in an oversimplified way as applying PointNet++ to 4D points, but there are actually many similarities to the proposed method (\"point tubes\" are analogous to neighborhood grouping and the PST convolution to PointNet aggregation).\n- The proposed spatial kernel (Eq 5) is not compared to or motivated by related static point cloud convolutional methods in the Related Work or Methods sections. Additionally, some important relevant works are not cited, e.g., [1][2].\n- Other relevant works that consider spatiotemporal point clouds in a less general setting may be worth mentioning: e.g. in scene flow estimation [3] or learning from spatiotemporal object point clouds [4].\n\n[1] PointCNN: Convolution on X-Transformed Points, Li et al., 2018\n\n[2] Dynamic Graph CNN for Learning on Point Clouds, Wang et al., 2019\n\n[3] HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-scale Point Clouds, Gu et al., 2019\n\n[4] CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations, Rempe et al., 2020\n\nExperiments:\n- The decision to decouple spatial/temporal convolutions would be more convincing with: (1) a comparison to a baseline PST convolution that does not fully decompose space and time (Eq 2), and (2) an evaluation on LIDAR data where spatial sampling can be highly irregular and variable over time. For example, the task of scene flow estimation from LIDAR would indicate if PSTNet can deal with high spatial irregularity while still encoding necessary local details.\n- The design of the learned spatial kernel f (Eq 5 and following paragraph) is not extensively motivated or justified in the text or by experimental results. How does the displacement/sharing kernel design affect performance or compare to using an MLP or other prior work like PointConv which considers point density? \n- If possible, a runtime comparison to MeteorNet/MinkowskiNet would be useful since these are similar generic ST point processing approaches.\n\nMinor Suggestions:\n- The formulation in Sec 3.2 unclear and inconsistent at times: e.g. (1) the displacement vector is described as a real number following Eq 3, but as a vector following Eq 5, and (2) the shape of F is not detailed for Eq 2-5. In general, I think the exposition in Sec 3.2 would be improved by intuitively describing the \"Point Tube\" idea before the precise mathematical formulation.\n- In Fig 5, GPU runtime is longer than CPU?\n- Typo in Sec 4.3, second paragraph: \"kennel size\" -> \"kernel size\".", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a construction for deep learning on point clouds that evolve over time. The key characteristics of the data are irregular sampling in the spatial domain and regular sampling in the temporal domain. The presented construction addresses both these aspects of the data. The review by R3 was negative but was addressed by the authors and R3 did not participate in the discussion. The AC supports acceptance.
Decision: Accept (Poster)