PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences

1 INTRODUCTION . Modern robotic and automatic driving systems usually employ real-time depth sensors , such as LiDAR , to capture the geometric information of scenes accurately while being robust to different lighting conditions . A scene geometry is thus represented by a 3D point cloud , i.e. , a set of measured point coordinates { ( x , y , z ) } . Moreover , when RGB images are available , they are often used as additional features associated with the 3D points to enhance the discriminativeness of point clouds . However , unlike conventional grid based videos , dynamic point clouds are irregular and unordered in the spatial dimension while points are not consistent and even flow in and out over time . Therefore , existing 3D convolutions on grid based videos ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Hara et al. , 2018 ) are not suitable to model raw point cloud sequences , as shown in Fig . 1 . To model the dynamics of point clouds , one solution is converting point clouds to a sequence of 3D voxels , and then applying 4D convolutions ( Choy et al. , 2019 ) to the voxel sequence . However , directly performing convolutions on voxel sequences require a large amount of computation . Furthermore , quantization errors are inevitable during voxelization , which may restrict applications that require precise measurement of scene geometry . Another solution MeteorNet ( Liu et al. , 2019e ) is extending the static point cloud method PointNet++ ( Qi et al. , 2017b ) to process raw point cloud sequences by appending 1D temporal dimension to 3D points . However , simply concatenating coordinates and time together and treating point cloud sequences as unordered 4D point sets neglect the temporal order of timestamps , which may not properly exploit the temporal information and lead to inferior performance . Moreover , the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible . Treating them equally is not conducive for network optimization . Besides , MeteorNet only considers spatial neighbors and neglects the local dependencies of neighboring frames . With the use of whole sequence length as its temporal receptive field , MeteorNet can not construct temporal hierarchy . As points are not consistent and even flow in and out of the region , especially for long sequences and fast motion embedding points in a spatially local area along an entire sequence handicaps capturing accurate local dynamics of point clouds . In this paper , we propose a point spatio-temporal ( PST ) convolution to directly process raw point cloud sequences . As dynamic point clouds are spatially irregular but ordered in the temporal dimension , we decouple the spatial and temporal information to model point cloud sequences . Specifically , PST convolution consists of ( i ) a point based spatial convolution that models the spatial structure of 3D points and ( ii ) a temporal convolution that captures the temporal dynamics of point cloud sequences . In this fashion , PST convolution significantly facilitates the modeling of dynamic point clouds and reduces the impact of the spatial irregularity of points on temporal modeling . Because point cloud sequences emerge inconsistently across frames , it is challenging to perform convolution on them . To address this problem , we introduce a point tube to preserve the spatio-temporal local structure . To enhance the feature extraction ability , we incorporate the proposed PST convolution into a spatiotemporally hierarchical network , namely PSTNet . Moreover , we extend our PST convolution to a transposed version to address point-level prediction tasks . Different from the convolutional version , the PST transposed convolution is designed to interpolate temporal dynamics and spatial features . Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of the proposed PST convolution and the superiority of PSTNet in modeling point clouds sequences . The contributions of this paper are fourfold : • To the best of our knowledge , we are the first attempt to decompose spatial and temporal information in modeling raw point cloud sequences , and propose a generic point based convolutional operation , named PST convolution , to encode raw point cloud sequences . • We propose a PST transposed convolution to decode raw point cloud sequences via interpolating the temporal dynamics and spatial feature for point-level prediction tasks . • We construct convolutional neural networks based on the PST convolutions and transposed convolutions , dubbed PSTNet , to tackle sequence-level classification and point-level prediction tasks . To the best of our knowledge , our PSTNet is the first deep neural network to model raw point cloud sequences in a both spatially and temporally hierarchical manner . • Extensive experiments on four datasets indicate that our method improves the accuracy of 3D action recognition and 4D semantic segmentation . 2 RELATED WORK . Learning Representations on Grid based Videos . Impressive progress has been made on generating compact and discriminative representations for RGB/RGBD videos due to the success of deep neural networks . For example , two-stream convolutional neural networks ( Simonyan & Zisserman , 2014 ; Wang et al. , 2016 ) utilize a spatial stream and an optical flow stream for video modeling . To summarize the temporal dependencies of videos , recurrent neural networks ( Ng et al. , 2015 ; Fan et al. , 2018 ) and pooling techniques ( Fan et al. , 2017 ) are employed . In addition , by stacking multiple 2D frames into a 3D tensor , 3D convolutional neural networks ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Tran et al. , 2018 ; Hara et al. , 2018 ) are widely used to learn spatio-temporal representations for videos , and achieve promising performance . Besides , interpretable video or action reasoning methods ( Zhuo et al. , 2019 ; Fan et al. , 2021b ) are proposed by explicitly parsing changes in videos . Static Point Cloud Processing . Static point cloud analysis has been widely investigated for many problems , such as classification , object part segmentation , scene semantic segmentation ( Qi et al. , 2017a ; b ; Li et al. , 2018b ; Wu et al. , 2019 ; Thomas et al. , 2019 ; Wang et al. , 2019 ) , reconstruction ( Dai et al. , 2017 ; Yu et al. , 2018 ) and object detection ( Chen et al. , 2017 ; Qi et al. , 2019 ) . Most recent works aim to directly manipulate point sets without transforming coordinates into regular voxel grids . Since a point cloud is essentially a set of unordered points and invariant to permutations of its points , static point cloud processing methods mainly focus on designing effective point based spatial correlation operations that do not rely on point orderings . Dynamic Point Cloud Modeling . Compared with static point cloud processing , dynamic point cloud modeling is a fairly new task . Fast and Furious ( FaF ) ( Luo et al. , 2018 ) converts a point cloud frame into a bird ’ s view voxel and then extracts features via 3D convolutions . MinkowskiNet ( Choy et al. , 2019 ) converts a point cloud sequence into a 4D occupancy grid and then applies 4D Spatio-Temporal ConvNets . PointRNN ( Fan & Yang , 2019 ) leverages point based recurrent neural networks for raw point cloud sequence forecasting . MeteorNet ( Liu et al. , 2019e ) extends 3D points to 4D points and then appends a temporal dimension to PointNet++ to process these 4D points . 3DV ( Wang et al. , 2020 ) first integrates 3D motion information into a regular compact voxel set and then applies PointNet++ to extract representations from the set for 3D action recognition . P4Transformer ( Fan et al. , 2021a ) employs transformer to avoid point tracking for raw point cloud sequence modeling . Niemeyer et al . ( 2019 ) learned a temporal and spatial vector field for 4D reconstruction . Prantl et al . ( 2020 ) learned stable and temporal coherent feature spaces for point based super-resolution . CaSPR ( Rempe et al. , 2020 ) learns to encode spatio-temporal changes in object shape from point clouds for reconstruction and camera pose estimation . In this paper , we propose a point based convolution to model raw point cloud sequences in a spatio-temporally hierarchical manner . 3 PROPOSED POINT SPATIO-TEMPORAL CONVOLUTIONAL NETWORK . In this section , we first briefly review the grid based 3D convolution operation as our point spatiotemporal ( PST ) convolution is motivated by it . Then , we introduce how the proposed PST convolution extracts features from dynamic point clouds in detail . In order to address dense point prediction tasks , i.e. , semantic segmentation , we develop a PST transposed convolution . Finally , we incorporate our operations into deep hierarchical networks to address different dynamic point cloud tasks . 3.1 MOTIVATION FROM GRID BASED 3D CONVOLUTION . The power of convolutional neural network ( CNNs ) comes from local structure modeling by convolutions and global representation learning via hierarchical architectures . Given an input feature map F ∈ RC×L×H×W , where C , L , H and W denote the input feature dimension , length , height and width , 3D convolution is designed to capture the spatio-temporal local structure of F , written as : F′ ( x , y ) t = bl/2c∑ k=−bl/2c bh/2c∑ i=−bh/2c bw/2c∑ j=−bw/2c W ( i , j ) k · F ( x+i , y+j ) t+k , ( 1 ) where W ∈ RC′×C×l×h×w is the convolution kernel , ( l , h , w ) is the kernel size , C ′ represents the output feature dimension and · is matrix multiplication . The W ( i , j ) k ∈ RC ′×C denotes the weight at kernel position ( k , i , j ) and F ( x+i , y+j ) t+k ∈ RC×1 denotes the feature of pixel at input position ( t+k , x+ i , y+ j ) . Usually , CNNs employ small kernel sizes , e.g. , ( 3 , 3 , 3 ) , which are much smaller than the input size , to effectively model the local structure . To construct hierarchical CNNs , stride ( > 1 ) is used to subsample input feature maps during the convolution operation . In this fashion , following convolutional layers will have relatively larger receptive fields . 3.2 POINT SPATIO-TEMPORAL ( PST ) CONVOLUTION . Let Pt ∈ R3×N and Ft ∈ RC×N denote the point coordinates and features of the t-th frame in a point cloud sequence , where N and C denote the number of points and feature channels , respectively . Given a point cloud sequence ( [ P1 ; F1 ] , [ P2 ; F2 ] , · · · , [ PL ; FL ] ) , the proposed PST convolution will encode the sequence to ( [ P ′1 ; F ′ 1 ] , [ P ′ 2 ; F ′ 2 ] , · · · , [ P ′L′ ; F ′L′ ] ) , where L and L′ indicate the number of frames , and P ′t ∈ R3×N ′ and F ′t ∈ RC ′×N ′ represent the encoded coordinates and features . 3.2.1 DECOMPOSING SPACE AND TIME IN POINT CLOUD SEQUENCE MODELING . Because point clouds are irregular and unordered , grid based 3D convolution can not be directly applied to point cloud sequences . As point cloud sequences are spatially irregular and unordered but temporally ordered , this motivates us to decouple these two dimensions in order to reduce the impacts of the spatial irregularity of points on temporal modeling . Moreover , the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible . Treating them equally is not conducive for network optimization . By decoupling the spatio-temporal information in point cloud sequences , we not only make the spatio-temporal modeling easier but also significantly improve the ability to capture the temporal information . Therefore , our PST convolution is formulated as : F ′ ( x , y , z ) t = bl/2c∑ k=−bl/2c ∑ ‖ ( δx , δy , δz ) ‖≤r W ( δx , δy , δz ) k · F ( x+δx , y+δy , z+δz ) t+k , = bl/2c∑ k=−bl/2c ∑ ‖ ( δx , δy , δz ) ‖≤r T ( δx , δy , δz ) k · ( S ( δx , δy , δz ) k · F ( x+δx , y+δy , z+δz ) t+k ) , ( 2 ) where ( x , y , z ) ∈ Pt , ( δx , δy , δz ) represents displacement and r is spatial search radius . The convolution kernel W ∈ RC′×C×l × R is decomposed into a spatial convolution kernel S ∈ RCm×C×l × R and a temporal convolution kernel T ∈ RC′×Cm×l × R and Cm is the dimension of the intermediate feature . Moreover , because space and time are orthogonal and independent of each other , we further decompose spatial and temporal modeling as : F ′ ( x , y , z ) t = bl/2c∑ k=−bl/2c Tk · ∑ ‖ ( δx , δy , δz ) ‖≤r S ( δx , δy , δz ) · F ( x+δx , y+δy , z+δz ) t+k , ( 3 ) where S ∈ RCm×C × R and T ∈ RC′×Cm×l . The above decomposition can also be expressed as applying temporal convolution first and then spatial convolution to input features . However , doing so requires point tracking to capture point motion . Because it is difficult to achieve accurate point trajectory and tracking points usually relies on point colors and may fail to handle colorless point clouds , we opt to model the spatial structure of irregular points first , and then capture the temporal information from the spatial regions . In this fashion , Eq . ( 3 ) can be rewritten as : M ( x , y , z ) t = ∑ ‖ ( δx , δy , δz ) ‖≤r S ( δx , δy , δz ) · F ( x+δx , y+δy , z+δz ) t , F ′ ( x , y , z ) t = bl/2c∑ k=−bl/2c Tk ·M ( x , y , z ) t+k . ( 4 ) The spatial convolution captures spatial local structures of point clouds in 3D space , while the temporal convolution aims to describe the temporal local dynamics of point cloud sequences along the time dimension . In order to capture the distributions of neighboring points , it is required to learn a convolution kernel S for all displacements . However , this is impossible because point displacements are not discrete . To address this problem , we convert the kernel to a function of displacements , defined as : M ( x , y , z ) t = ∑ ‖ ( δx , δy , δz ) ‖≤r f ( ( δx , δy , δz ) ; θ ) · F ( x+δx , y+δy , z+δz ) t , ( 5 ) where f : R1×3 → RCm×C is a function of ( δx , δy , δz ) parametrized by θ to generate different RCm×C according to different displacements . There can be many ways to implement the f function . In principle , the f function should be both computation-efficient and memory-efficient so that PST convolution is able to encode long sequences . Existing static point cloud convolution methods ( Li et al. , 2018b ; Wu et al. , 2019 ; Wang et al. , 2019 ) usually contains multilayer perceptrons ( MLPs ) , which are computation and memory consuming . In this paper , we design a lightweight implementation for f , which contains only a few parameters . Specifically , we further decompose the f function as f ( ( δx , δy , δz ) ; θ ) = θd · ( δx , δy , δz ) T · 1 θs , where θ = [ θd , θs ] , θd ∈ RCm×3 is a displacement transform kernel , θs ∈ RCm×C is a sharing kernel , 1 = ( 1 , · · · , 1 ) ∈ R1×C is for broadcasting , and is element-wise product . The sharing kernel θs is to increase point feature dimension to improve the feature representation ability . The displacement kernel θd is to capture spatial local structure based on displacements . In this fashion , f generates a unique spatial kernel for each displacement so that the lightweight spatial convolution is able to increase feature dimension while capturing spatial local structure like conventional convolutions .

Summary: The paper introduces point spatio-temporal convolutions, which are used for the feature extraction of point cloud sequences. A trainable kernel is used which is applied locally as a continuous convolution. An important aspect is that the temporal dimension is processed separately, with an additional convolution, instead of simply using a 4D convolution. The authors claim that in this way the network will achieve a better understanding of the dynamics of the input.