Learning Transferable Features for Point Cloud Detection via 3D Contrastive Co-training

1 Introduction . 3D point cloud detection shows remarkable significance in real-world scenarios , such as autonomous driving [ 14 , 43 , 38 , 44 ] , in which the recent progress is largely driven by the emergence of highprecision LiDAR sensors and large-scale , densely annotated point cloud datasets [ 1 , 6 , 26 ] . Most existing 3D detection models assume that the training domain and testing domain are independently and identically distributed . In practice , however , domain shifts are often inevitable due to differences in physical environments or LiDAR sensor configurations , including different numbers of laser beams and installation positions , etc . To address this issue , we present an early study of the unsupervised domain adaptation problem for point cloud detection , which aims to effectively adapt 3D detectors from the labeled source domain to a novel unlabeled target domain by learning transferable features . Previous domain adaptation approaches for image data [ 27 , 12 , 24 , 2 ] are not readily applicable to point clouds . As shown in Fig . 1 , different from the domain shifts of 2D scenes that usually ∗ The first two authors have equal contributions . † C. Ma is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . exists in the image appearance , those of 3D scenes are mainly reflected in the geometry variations of point clouds . Since 2D images from different domains have the same grid topology of uniformly distributed pixels , most of the domain adaptation methods exploit image encoders with domainsharing parameters , which is also adopted by existing 3D transfer learning models such as PointDAN [ 20 ] . However , due to the severe low-level geometry shifts between different point sets , we believe that some features are transferable while some features are not on 3D object detection , which requires a rethink of the transferability of different levels of the point cloud representations . To this end , we propose a novel framework named 3D Contrastive Co-training ( 3D-CoCo ) , whose architecture contains separate 3D encoders with domain-specific parameters , a domain-agnostic BEV transformation module , and the final detection head . The key idea of the architecture design is that the BEV features with similar grid structures to images can be more transferable than low-level 3D features , so that they can be better integrated with advanced transfer learning techniques in 2D vision and thus greatly reduce the geometry shift . Another benefit of the co-training architecture with domain-specific encoders is that in addition to improving domain adaptation results , it also maintains the in-domain performance . Furthermore , 3D-CoCo is also featured by a new end-to-end contrastive learning framework , which contains two main components , i.e. , contrastive instance alignment based on bird-eye-view ( BEV ) features and hard sample mining with transformed point clouds . The insight of the contrastive instance alignment is to push the feature centroids of similar sample clusters , induced by pseudo-labels , closer to each other , no matter they are in the same domain or different domains . Besides , we consider the mismatch between the true distribution of BEV features and the fictitious one used for contrastive learning , which is caused by biased pseudo-labels . Specifically , we take advantage of the editability of point clouds and perform hard sample mining by applying specific transformation functions to 3D data . The hard samples , as an effective supplement to contrastive co-training , can further reduce the geometry shift across domains and prevent the adaptation model from falling into the local minima . Notably , there exists another line of work discussing transfer learning for point cloud detection [ 28 , 35 , 3 ] , which utilizes the self-training pipeline that retrains the model with pseudo-labels on target data . Compared to these approaches , we adopt a different problem setup , using labeled source domain data and unlabeled target domain data for co-training . Through ablation studies , we observe that the separate encoders and the contrastive co-training scheme can progressively filter domain-specific features and learn more transferable knowledge across domains . We evaluate the effectiveness and generalizability of 3D-CoCo across three widely-used autonomous driving datasets collected by heterogeneous LiDAR sensors , including Waymo [ 26 ] , nuScenes [ 1 ] , and KITTI [ 6 ] . 3D-CoCo is shown to significantly outperform existing approaches for point cloud detection on different unsupervised domain adaptation benchmarks . 2 Preliminaries . Problem setup . The traditional setup of point cloud detection is to learn a base 3D object detector D that classifies and localizes m objects represented by Y from a point cloud P : D : P → Y , ( 1 ) where P = { p1 , . . . , pn } consists of n points pi ∈ Rd scattered over the 3D coordinate space . The dimension d is set to 4 , including the coordinate ( x , y , z ) and an additional intensity i . The annotations Y = { y1 , . . . , ym } consists of a set of class labels and bounding box coordinates , which is represented by the size ( l , w , h ) , center ( cx , cy , cz ) , and heading angle r of each 3D object . Typically , D is trained in a supervised manner by minimizing the classification and localization errors between the ground truth Y and the prediction Ŷ over the training ( source ) dataset . In this paper , we specifically study the problem of unsupervised domain adaptation for point cloud detection , which aims to adapt the detector Dθ parametrized by θ from a labeled source domain { ( PSi , Y Si ) } NS i=1 to an unlabeled target domain { PTi } NT i=1 , where NS and NT are the numbers of training samples : Dθ : P S ∪ PT → Y S ∪ Y T . ( 2 ) Compared to the typical setup of point cloud detection , we focus more on the performance of the model on the test set of the target domain , which requires additional well-designed modules for learning transferable features . Modern point cloud detectors . Modern 3D point cloud detectors [ 38 , 43 , 14 ] usually consist of three modules : a 3D encoder E , a bird-eye-view ( BEV ) transformation module U , and a detection head H . The 3D encoder quantizes the point cloud into regular grids for feature extraction . The BEV transformation module , usually in forms of 2D convolutional layers , produces fixed-size BEV feature maps M ∈ RW×L×F , where width W and length L are based on the resolution of bins , and F denotes the channel number . The detection head takes M as inputs and produces detection results Ŷ . To demonstrate the generality of the proposed method , we adopt two mainstream architectures , i.e. , VoxelNet [ 43 ] and PointPillars [ 14 ] , with different point cloud processing pipelines as alternative options of the 3D encoder . VoxelNet quantizes the point cloud into small 3D voxel features and then uses a 3D CNN to compress them into 2D BEV space along the height of the voxel , while PointPillars quantizes the point cloud into vertical pillars on fixed-size 2D grids and then performs linear transformation and max-pooling on each pillar to obtain the BEV representation . Neither of them explicitly considers the domain shift in transfer learning setups . 3 Method . We present 3D-CoCo as a feasible solution to the unsupervised domain adaptation task in point cloud detection . It has two contributions to learning transferable features from heterogeneous geometries , which respectively reside in the new architecture design , as shown in Fig . 2 , and the framework of contrastive instance alignment enhanced by hard sample mining , as shown in Alg . 1 and Fig . 3 . 3.1 3D-CoCo Architecture . Domain-specific 3D encoders . Transfer learning in 3D scenes may suffer from dramatic geometry shifts , such as density variations and different occlusion ratios of point clouds , due to different physical environments and sensor configurations . Although some work has explored model pretraining on 3D pretext tasks [ 41 , 30 ] , in contrast with 2D scenarios , 3D vision still lacks a transferable , wellpretrained backbone . One possible reason is that it is very difficult to reduce the domain shift in geometric representations at the bottom of the 3D encoder . Intuitively , we expect the 3D detection network to progressively process domain-specific non-transferable features and learn domain-invariant semantic features . As shown in Fig . 2 , we present a novel model architecture with domain-specific 3D encoders , which learn different mapping functions to parse and convert LiDAR points into the Algorithm 1 : The learning procedure of 3D contrastive co-training ( 3D-CoCo ) Input : The labeled point set from the source domain { ( PSi , Y Si ) } NS i=1 , the unlabeled point set from the target domain { PTi } NT i=1 , the maximum number of update stages K Output : Learned network weights θ 1 D′ ← { ( PSi , Y Si ) } NS i=1 . Pretrain the base detector according to Eq . ( 3 ) 2 { Y Ti } NT i=1 ← ( D′ , { PTi } NT i=1 ) . Generate pseudo-labels for target domain samples 3 ( { ( PTi , Y T i ) } NT i=1 ) ← HSM ( { ( PTi , Y T i ) } NT i=1 ) . Mine hard samples to augment the target set 4 Dθ ← D′ . Initialize the model with the base detector 5 D0θ ← ( { ( PSi , Y Si ) } NS i=1 , { ( PTi , Y T i ) } NT i=1 ) . Warm-up according to Eq . ( 6 ) 6 for k = 1 , . . . , K do 7 { Y Ti } NT i=1 ← ( D k−1 θ , { PTi } NT i=1 ) . Update pseudo-labels 8 ( { ( PTi , Y T i ) } NT i=1 ) ← HSM ( { ( PTi , Y T i ) } NT i=1 ) . Append new hard samples to the target set 9 Dkθ ← ( { ( PSi , Y Si ) } NS i=1 , { ( PTi , Y T i ) } NT i=1 ) . Model update according to Eq . ( 6 ) 10 Dk−1θ ← Dkθ bird-eye-view ( BEV ) space for different domains . It is worth noting that the co-training architecture not only benefits the adaptation performance on the target domain but also contributes to maintaining the performance on the source domain , in the sense that learning transferable features upon separate encoders can facilitate bi-directional knowledge sharing . Domain-agnostic BEV transformation module . The 2D transformation module is co-trained with data samples from both source and target domain . It further compresses the outputs of the domain-specific 3D encoders into BEV feature maps M . The BEV features are supposed to be more transferable , because they have similar structures to the grid-based feature maps in 2D vision and can therefore be easily integrated into off-the-shelf transfer learning techniques . Based on M , we perform the contrastive alignment training scheme to encourage the learning of domain-invariant features . Detection head . The detection head classifies and localizes 3D objects from the BEV feature maps M . Given a labeled sample from the source domain , the detection head is trained to minimize Ldet = LScls + LSloc , ( 3 ) where LScls and LSloc respectively indicates the classification and localization error . For an unlabeled sample from the target domain , we use a regularization of the localization error LTreg between the predictions and the pseudo-labels , which encourages a rapid adaptation to the target domain .

Summary: This paper proposes a domain adaptation point cloud object detection method termed 3D-COCO to deal with geometry shifts caused by different physical environments or LiDAR sensors. The authors use the shared BEV module to learn the domain-invariant for both source and target domain, while the augmented contrastive alignment schemes are also introduced to discover the hard samples. The mismatch is greatly reduced by 3D-CoCo with transformed point clouds. The experimental results on three benchmarks demonstrate the effectiveness of the proposed method.