PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences

1 INTRODUCTION . Modern robotic and automatic driving systems usually employ real-time depth sensors , such as LiDAR , to capture the geometric information of scenes accurately while being robust to different lighting conditions . A scene geometry is thus represented by a 3D point cloud , i.e. , a set of measured point coordinates { ( x , y , z ) } . Moreover , when RGB images are available , they are often used as additional features associated with the 3D points to enhance the discriminativeness of point clouds . However , unlike conventional grid based videos , dynamic point clouds are irregular and unordered in the spatial dimension while points are not consistent and even flow in and out over time . Therefore , existing 3D convolutions on grid based videos ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Hara et al. , 2018 ) are not suitable to model raw point cloud sequences , as shown in Fig . 1 . To model the dynamics of point clouds , one solution is converting point clouds to a sequence of 3D voxels , and then applying 4D convolutions ( Choy et al. , 2019 ) to the voxel sequence . However , directly performing convolutions on voxel sequences require a large amount of computation . Furthermore , quantization errors are inevitable during voxelization , which may restrict applications that require precise measurement of scene geometry . Another solution MeteorNet ( Liu et al. , 2019e ) is extending the static point cloud method PointNet++ ( Qi et al. , 2017b ) to process raw point cloud sequences by appending 1D temporal dimension to 3D points . However , simply concatenating coordinates and time together and treating point cloud sequences as unordered 4D point sets neglect the temporal order of timestamps , which may not properly exploit the temporal information and lead to inferior performance . Moreover , the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible . Treating them equally is not conducive for network optimization . Besides , MeteorNet only considers spatial neighbors and neglects the local dependencies of neighboring frames . With the use of whole sequence length as its temporal receptive field , MeteorNet can not construct temporal hierarchy . As points are not consistent and even flow in and out of the region , especially for long sequences and fast motion embedding points in a spatially local area along an entire sequence handicaps capturing accurate local dynamics of point clouds . In this paper , we propose a point spatio-temporal ( PST ) convolution to directly process raw point cloud sequences . As dynamic point clouds are spatially irregular but ordered in the temporal dimension , we decouple the spatial and temporal information to model point cloud sequences . Specifically , PST convolution consists of ( i ) a point based spatial convolution that models the spatial structure of 3D points and ( ii ) a temporal convolution that captures the temporal dynamics of point cloud sequences . In this fashion , PST convolution significantly facilitates the modeling of dynamic point clouds and reduces the impact of the spatial irregularity of points on temporal modeling . Because point cloud sequences emerge inconsistently across frames , it is challenging to perform convolution on them . To address this problem , we introduce a point tube to preserve the spatio-temporal local structure . To enhance the feature extraction ability , we incorporate the proposed PST convolution into a spatiotemporally hierarchical network , namely PSTNet . Moreover , we extend our PST convolution to a transposed version to address point-level prediction tasks . Different from the convolutional version , the PST transposed convolution is designed to interpolate temporal dynamics and spatial features . Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of the proposed PST convolution and the superiority of PSTNet in modeling point clouds sequences . The contributions of this paper are fourfold : • To the best of our knowledge , we are the first attempt to decompose spatial and temporal information in modeling raw point cloud sequences , and propose a generic point based convolutional operation , named PST convolution , to encode raw point cloud sequences . • We propose a PST transposed convolution to decode raw point cloud sequences via interpolating the temporal dynamics and spatial feature for point-level prediction tasks . • We construct convolutional neural networks based on the PST convolutions and transposed convolutions , dubbed PSTNet , to tackle sequence-level classification and point-level prediction tasks . To the best of our knowledge , our PSTNet is the first deep neural network to model raw point cloud sequences in a both spatially and temporally hierarchical manner . • Extensive experiments on four datasets indicate that our method improves the accuracy of 3D action recognition and 4D semantic segmentation . 2 RELATED WORK . Learning Representations on Grid based Videos . Impressive progress has been made on generating compact and discriminative representations for RGB/RGBD videos due to the success of deep neural networks . For example , two-stream convolutional neural networks ( Simonyan & Zisserman , 2014 ; Wang et al. , 2016 ) utilize a spatial stream and an optical flow stream for video modeling . To summarize the temporal dependencies of videos , recurrent neural networks ( Ng et al. , 2015 ; Fan et al. , 2018 ) and pooling techniques ( Fan et al. , 2017 ) are employed . In addition , by stacking multiple 2D frames into a 3D tensor , 3D convolutional neural networks ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Tran et al. , 2018 ; Hara et al. , 2018 ) are widely used to learn spatio-temporal representations for videos , and achieve promising performance . Besides , interpretable video or action reasoning methods ( Zhuo et al. , 2019 ; Fan et al. , 2021b ) are proposed by explicitly parsing changes in videos . Static Point Cloud Processing . Static point cloud analysis has been widely investigated for many problems , such as classification , object part segmentation , scene semantic segmentation ( Qi et al. , 2017a ; b ; Li et al. , 2018b ; Wu et al. , 2019 ; Thomas et al. , 2019 ; Wang et al. , 2019 ) , reconstruction ( Dai et al. , 2017 ; Yu et al. , 2018 ) and object detection ( Chen et al. , 2017 ; Qi et al. , 2019 ) . Most recent works aim to directly manipulate point sets without transforming coordinates into regular voxel grids . Since a point cloud is essentially a set of unordered points and invariant to permutations of its points , static point cloud processing methods mainly focus on designing effective point based spatial correlation operations that do not rely on point orderings . Dynamic Point Cloud Modeling . Compared with static point cloud processing , dynamic point cloud modeling is a fairly new task . Fast and Furious ( FaF ) ( Luo et al. , 2018 ) converts a point cloud frame into a bird ’ s view voxel and then extracts features via 3D convolutions . MinkowskiNet ( Choy et al. , 2019 ) converts a point cloud sequence into a 4D occupancy grid and then applies 4D Spatio-Temporal ConvNets . PointRNN ( Fan & Yang , 2019 ) leverages point based recurrent neural networks for raw point cloud sequence forecasting . MeteorNet ( Liu et al. , 2019e ) extends 3D points to 4D points and then appends a temporal dimension to PointNet++ to process these 4D points . 3DV ( Wang et al. , 2020 ) first integrates 3D motion information into a regular compact voxel set and then applies PointNet++ to extract representations from the set for 3D action recognition . P4Transformer ( Fan et al. , 2021a ) employs transformer to avoid point tracking for raw point cloud sequence modeling . Niemeyer et al . ( 2019 ) learned a temporal and spatial vector field for 4D reconstruction . Prantl et al . ( 2020 ) learned stable and temporal coherent feature spaces for point based super-resolution . CaSPR ( Rempe et al. , 2020 ) learns to encode spatio-temporal changes in object shape from point clouds for reconstruction and camera pose estimation . In this paper , we propose a point based convolution to model raw point cloud sequences in a spatio-temporally hierarchical manner . 3 PROPOSED POINT SPATIO-TEMPORAL CONVOLUTIONAL NETWORK . In this section , we first briefly review the grid based 3D convolution operation as our point spatiotemporal ( PST ) convolution is motivated by it . Then , we introduce how the proposed PST convolution extracts features from dynamic point clouds in detail . In order to address dense point prediction tasks , i.e. , semantic segmentation , we develop a PST transposed convolution . Finally , we incorporate our operations into deep hierarchical networks to address different dynamic point cloud tasks . 3.1 MOTIVATION FROM GRID BASED 3D CONVOLUTION . The power of convolutional neural network ( CNNs ) comes from local structure modeling by convolutions and global representation learning via hierarchical architectures . Given an input feature map F ∈ RC×L×H×W , where C , L , H and W denote the input feature dimension , length , height and width , 3D convolution is designed to capture the spatio-temporal local structure of F , written as : F′ ( x , y ) t = bl/2c∑ k=−bl/2c bh/2c∑ i=−bh/2c bw/2c∑ j=−bw/2c W ( i , j ) k · F ( x+i , y+j ) t+k , ( 1 ) where W ∈ RC′×C×l×h×w is the convolution kernel , ( l , h , w ) is the kernel size , C ′ represents the output feature dimension and · is matrix multiplication . The W ( i , j ) k ∈ RC ′×C denotes the weight at kernel position ( k , i , j ) and F ( x+i , y+j ) t+k ∈ RC×1 denotes the feature of pixel at input position ( t+k , x+ i , y+ j ) . Usually , CNNs employ small kernel sizes , e.g. , ( 3 , 3 , 3 ) , which are much smaller than the input size , to effectively model the local structure . To construct hierarchical CNNs , stride ( > 1 ) is used to subsample input feature maps during the convolution operation . In this fashion , following convolutional layers will have relatively larger receptive fields . 3.2 POINT SPATIO-TEMPORAL ( PST ) CONVOLUTION . Let Pt ∈ R3×N and Ft ∈ RC×N denote the point coordinates and features of the t-th frame in a point cloud sequence , where N and C denote the number of points and feature channels , respectively . Given a point cloud sequence ( [ P1 ; F1 ] , [ P2 ; F2 ] , · · · , [ PL ; FL ] ) , the proposed PST convolution will encode the sequence to ( [ P ′1 ; F ′ 1 ] , [ P ′ 2 ; F ′ 2 ] , · · · , [ P ′L′ ; F ′L′ ] ) , where L and L′ indicate the number of frames , and P ′t ∈ R3×N ′ and F ′t ∈ RC ′×N ′ represent the encoded coordinates and features . 3.2.1 DECOMPOSING SPACE AND TIME IN POINT CLOUD SEQUENCE MODELING . Because point clouds are irregular and unordered , grid based 3D convolution can not be directly applied to point cloud sequences . As point cloud sequences are spatially irregular and unordered but temporally ordered , this motivates us to decouple these two dimensions in order to reduce the impacts of the spatial irregularity of points on temporal modeling . Moreover , the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible . Treating them equally is not conducive for network optimization . By decoupling the spatio-temporal information in point cloud sequences , we not only make the spatio-temporal modeling easier but also significantly improve the ability to capture the temporal information . Therefore , our PST convolution is formulated as : F ′ ( x , y , z ) t = bl/2c∑ k=−bl/2c ∑ ‖ ( δx , δy , δz ) ‖≤r W ( δx , δy , δz ) k · F ( x+δx , y+δy , z+δz ) t+k , = bl/2c∑ k=−bl/2c ∑ ‖ ( δx , δy , δz ) ‖≤r T ( δx , δy , δz ) k · ( S ( δx , δy , δz ) k · F ( x+δx , y+δy , z+δz ) t+k ) , ( 2 ) where ( x , y , z ) ∈ Pt , ( δx , δy , δz ) represents displacement and r is spatial search radius . The convolution kernel W ∈ RC′×C×l × R is decomposed into a spatial convolution kernel S ∈ RCm×C×l × R and a temporal convolution kernel T ∈ RC′×Cm×l × R and Cm is the dimension of the intermediate feature . Moreover , because space and time are orthogonal and independent of each other , we further decompose spatial and temporal modeling as : F ′ ( x , y , z ) t = bl/2c∑ k=−bl/2c Tk · ∑ ‖ ( δx , δy , δz ) ‖≤r S ( δx , δy , δz ) · F ( x+δx , y+δy , z+δz ) t+k , ( 3 ) where S ∈ RCm×C × R and T ∈ RC′×Cm×l . The above decomposition can also be expressed as applying temporal convolution first and then spatial convolution to input features . However , doing so requires point tracking to capture point motion . Because it is difficult to achieve accurate point trajectory and tracking points usually relies on point colors and may fail to handle colorless point clouds , we opt to model the spatial structure of irregular points first , and then capture the temporal information from the spatial regions . In this fashion , Eq . ( 3 ) can be rewritten as : M ( x , y , z ) t = ∑ ‖ ( δx , δy , δz ) ‖≤r S ( δx , δy , δz ) · F ( x+δx , y+δy , z+δz ) t , F ′ ( x , y , z ) t = bl/2c∑ k=−bl/2c Tk ·M ( x , y , z ) t+k . ( 4 ) The spatial convolution captures spatial local structures of point clouds in 3D space , while the temporal convolution aims to describe the temporal local dynamics of point cloud sequences along the time dimension . In order to capture the distributions of neighboring points , it is required to learn a convolution kernel S for all displacements . However , this is impossible because point displacements are not discrete . To address this problem , we convert the kernel to a function of displacements , defined as : M ( x , y , z ) t = ∑ ‖ ( δx , δy , δz ) ‖≤r f ( ( δx , δy , δz ) ; θ ) · F ( x+δx , y+δy , z+δz ) t , ( 5 ) where f : R1×3 → RCm×C is a function of ( δx , δy , δz ) parametrized by θ to generate different RCm×C according to different displacements . There can be many ways to implement the f function . In principle , the f function should be both computation-efficient and memory-efficient so that PST convolution is able to encode long sequences . Existing static point cloud convolution methods ( Li et al. , 2018b ; Wu et al. , 2019 ; Wang et al. , 2019 ) usually contains multilayer perceptrons ( MLPs ) , which are computation and memory consuming . In this paper , we design a lightweight implementation for f , which contains only a few parameters . Specifically , we further decompose the f function as f ( ( δx , δy , δz ) ; θ ) = θd · ( δx , δy , δz ) T · 1 θs , where θ = [ θd , θs ] , θd ∈ RCm×3 is a displacement transform kernel , θs ∈ RCm×C is a sharing kernel , 1 = ( 1 , · · · , 1 ) ∈ R1×C is for broadcasting , and is element-wise product . The sharing kernel θs is to increase point feature dimension to improve the feature representation ability . The displacement kernel θd is to capture spatial local structure based on displacements . In this fashion , f generates a unique spatial kernel for each displacement so that the lightweight spatial convolution is able to increase feature dimension while capturing spatial local structure like conventional convolutions .

Summary: This paper introduces a new convolutional approach to directly process raw spatiotemporal (ST) point cloud data. The proposed point spatio-temporal (PST) convolution operates on "point tubes" and decouples space and time through a shared spatial convolution at each timestep, followed by a temporal convolution. It also introduces a transposed PST to enable point-wise predictions in an encoder-decoder framework (PSTNet). The presented experiments demonstrate the effectiveness of these convolutions by using PSTNet for action recognition and semantic segmentation on point cloud sequences, showing improvement over relevant recent work.