Learning Transferable Features for Point Cloud Detection via 3D Contrastive Co-training

1 Introduction . 3D point cloud detection shows remarkable significance in real-world scenarios , such as autonomous driving [ 14 , 43 , 38 , 44 ] , in which the recent progress is largely driven by the emergence of highprecision LiDAR sensors and large-scale , densely annotated point cloud datasets [ 1 , 6 , 26 ] . Most existing 3D detection models assume that the training domain and testing domain are independently and identically distributed . In practice , however , domain shifts are often inevitable due to differences in physical environments or LiDAR sensor configurations , including different numbers of laser beams and installation positions , etc . To address this issue , we present an early study of the unsupervised domain adaptation problem for point cloud detection , which aims to effectively adapt 3D detectors from the labeled source domain to a novel unlabeled target domain by learning transferable features . Previous domain adaptation approaches for image data [ 27 , 12 , 24 , 2 ] are not readily applicable to point clouds . As shown in Fig . 1 , different from the domain shifts of 2D scenes that usually ∗ The first two authors have equal contributions . † C. Ma is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . exists in the image appearance , those of 3D scenes are mainly reflected in the geometry variations of point clouds . Since 2D images from different domains have the same grid topology of uniformly distributed pixels , most of the domain adaptation methods exploit image encoders with domainsharing parameters , which is also adopted by existing 3D transfer learning models such as PointDAN [ 20 ] . However , due to the severe low-level geometry shifts between different point sets , we believe that some features are transferable while some features are not on 3D object detection , which requires a rethink of the transferability of different levels of the point cloud representations . To this end , we propose a novel framework named 3D Contrastive Co-training ( 3D-CoCo ) , whose architecture contains separate 3D encoders with domain-specific parameters , a domain-agnostic BEV transformation module , and the final detection head . The key idea of the architecture design is that the BEV features with similar grid structures to images can be more transferable than low-level 3D features , so that they can be better integrated with advanced transfer learning techniques in 2D vision and thus greatly reduce the geometry shift . Another benefit of the co-training architecture with domain-specific encoders is that in addition to improving domain adaptation results , it also maintains the in-domain performance . Furthermore , 3D-CoCo is also featured by a new end-to-end contrastive learning framework , which contains two main components , i.e. , contrastive instance alignment based on bird-eye-view ( BEV ) features and hard sample mining with transformed point clouds . The insight of the contrastive instance alignment is to push the feature centroids of similar sample clusters , induced by pseudo-labels , closer to each other , no matter they are in the same domain or different domains . Besides , we consider the mismatch between the true distribution of BEV features and the fictitious one used for contrastive learning , which is caused by biased pseudo-labels . Specifically , we take advantage of the editability of point clouds and perform hard sample mining by applying specific transformation functions to 3D data . The hard samples , as an effective supplement to contrastive co-training , can further reduce the geometry shift across domains and prevent the adaptation model from falling into the local minima . Notably , there exists another line of work discussing transfer learning for point cloud detection [ 28 , 35 , 3 ] , which utilizes the self-training pipeline that retrains the model with pseudo-labels on target data . Compared to these approaches , we adopt a different problem setup , using labeled source domain data and unlabeled target domain data for co-training . Through ablation studies , we observe that the separate encoders and the contrastive co-training scheme can progressively filter domain-specific features and learn more transferable knowledge across domains . We evaluate the effectiveness and generalizability of 3D-CoCo across three widely-used autonomous driving datasets collected by heterogeneous LiDAR sensors , including Waymo [ 26 ] , nuScenes [ 1 ] , and KITTI [ 6 ] . 3D-CoCo is shown to significantly outperform existing approaches for point cloud detection on different unsupervised domain adaptation benchmarks . 2 Preliminaries . Problem setup . The traditional setup of point cloud detection is to learn a base 3D object detector D that classifies and localizes m objects represented by Y from a point cloud P : D : P → Y , ( 1 ) where P = { p1 , . . . , pn } consists of n points pi ∈ Rd scattered over the 3D coordinate space . The dimension d is set to 4 , including the coordinate ( x , y , z ) and an additional intensity i . The annotations Y = { y1 , . . . , ym } consists of a set of class labels and bounding box coordinates , which is represented by the size ( l , w , h ) , center ( cx , cy , cz ) , and heading angle r of each 3D object . Typically , D is trained in a supervised manner by minimizing the classification and localization errors between the ground truth Y and the prediction Ŷ over the training ( source ) dataset . In this paper , we specifically study the problem of unsupervised domain adaptation for point cloud detection , which aims to adapt the detector Dθ parametrized by θ from a labeled source domain { ( PSi , Y Si ) } NS i=1 to an unlabeled target domain { PTi } NT i=1 , where NS and NT are the numbers of training samples : Dθ : P S ∪ PT → Y S ∪ Y T . ( 2 ) Compared to the typical setup of point cloud detection , we focus more on the performance of the model on the test set of the target domain , which requires additional well-designed modules for learning transferable features . Modern point cloud detectors . Modern 3D point cloud detectors [ 38 , 43 , 14 ] usually consist of three modules : a 3D encoder E , a bird-eye-view ( BEV ) transformation module U , and a detection head H . The 3D encoder quantizes the point cloud into regular grids for feature extraction . The BEV transformation module , usually in forms of 2D convolutional layers , produces fixed-size BEV feature maps M ∈ RW×L×F , where width W and length L are based on the resolution of bins , and F denotes the channel number . The detection head takes M as inputs and produces detection results Ŷ . To demonstrate the generality of the proposed method , we adopt two mainstream architectures , i.e. , VoxelNet [ 43 ] and PointPillars [ 14 ] , with different point cloud processing pipelines as alternative options of the 3D encoder . VoxelNet quantizes the point cloud into small 3D voxel features and then uses a 3D CNN to compress them into 2D BEV space along the height of the voxel , while PointPillars quantizes the point cloud into vertical pillars on fixed-size 2D grids and then performs linear transformation and max-pooling on each pillar to obtain the BEV representation . Neither of them explicitly considers the domain shift in transfer learning setups . 3 Method . We present 3D-CoCo as a feasible solution to the unsupervised domain adaptation task in point cloud detection . It has two contributions to learning transferable features from heterogeneous geometries , which respectively reside in the new architecture design , as shown in Fig . 2 , and the framework of contrastive instance alignment enhanced by hard sample mining , as shown in Alg . 1 and Fig . 3 . 3.1 3D-CoCo Architecture . Domain-specific 3D encoders . Transfer learning in 3D scenes may suffer from dramatic geometry shifts , such as density variations and different occlusion ratios of point clouds , due to different physical environments and sensor configurations . Although some work has explored model pretraining on 3D pretext tasks [ 41 , 30 ] , in contrast with 2D scenarios , 3D vision still lacks a transferable , wellpretrained backbone . One possible reason is that it is very difficult to reduce the domain shift in geometric representations at the bottom of the 3D encoder . Intuitively , we expect the 3D detection network to progressively process domain-specific non-transferable features and learn domain-invariant semantic features . As shown in Fig . 2 , we present a novel model architecture with domain-specific 3D encoders , which learn different mapping functions to parse and convert LiDAR points into the Algorithm 1 : The learning procedure of 3D contrastive co-training ( 3D-CoCo ) Input : The labeled point set from the source domain { ( PSi , Y Si ) } NS i=1 , the unlabeled point set from the target domain { PTi } NT i=1 , the maximum number of update stages K Output : Learned network weights θ 1 D′ ← { ( PSi , Y Si ) } NS i=1 . Pretrain the base detector according to Eq . ( 3 ) 2 { Y Ti } NT i=1 ← ( D′ , { PTi } NT i=1 ) . Generate pseudo-labels for target domain samples 3 ( { ( PTi , Y T i ) } NT i=1 ) ← HSM ( { ( PTi , Y T i ) } NT i=1 ) . Mine hard samples to augment the target set 4 Dθ ← D′ . Initialize the model with the base detector 5 D0θ ← ( { ( PSi , Y Si ) } NS i=1 , { ( PTi , Y T i ) } NT i=1 ) . Warm-up according to Eq . ( 6 ) 6 for k = 1 , . . . , K do 7 { Y Ti } NT i=1 ← ( D k−1 θ , { PTi } NT i=1 ) . Update pseudo-labels 8 ( { ( PTi , Y T i ) } NT i=1 ) ← HSM ( { ( PTi , Y T i ) } NT i=1 ) . Append new hard samples to the target set 9 Dkθ ← ( { ( PSi , Y Si ) } NS i=1 , { ( PTi , Y T i ) } NT i=1 ) . Model update according to Eq . ( 6 ) 10 Dk−1θ ← Dkθ bird-eye-view ( BEV ) space for different domains . It is worth noting that the co-training architecture not only benefits the adaptation performance on the target domain but also contributes to maintaining the performance on the source domain , in the sense that learning transferable features upon separate encoders can facilitate bi-directional knowledge sharing . Domain-agnostic BEV transformation module . The 2D transformation module is co-trained with data samples from both source and target domain . It further compresses the outputs of the domain-specific 3D encoders into BEV feature maps M . The BEV features are supposed to be more transferable , because they have similar structures to the grid-based feature maps in 2D vision and can therefore be easily integrated into off-the-shelf transfer learning techniques . Based on M , we perform the contrastive alignment training scheme to encourage the learning of domain-invariant features . Detection head . The detection head classifies and localizes 3D objects from the BEV feature maps M . Given a labeled sample from the source domain , the detection head is trained to minimize Ldet = LScls + LSloc , ( 3 ) where LScls and LSloc respectively indicates the classification and localization error . For an unlabeled sample from the target domain , we use a regularization of the localization error LTreg between the predictions and the pseudo-labels , which encourages a rapid adaptation to the target domain .

Summary: This paper contributes as an early study of unsupervised domain adaptation (UDA) on 3D point cloud detection, an important step towards zero cost of labeling. Different from 2D image data that have regular grid structures, 3D point cloud data could have huge geometric variations such as point density and occlusion due to different physical environments or LiDAR sensor configurations; consequently, the low-level features of point cloud representation may be not transferable across domains. Taking it into account, this paper uses two individual encoders for the source and target domains to capture domain-specific characteristics and a shared Bird-Eye-View (BEV) transformation module to learn domain-agnostic BEV features, which have similar grid structures as images and thus are more transferable. Specifically, this paper extends contrastive instance alignment to point cloud detection, based on BEW features of foreground and background proposals. Direct use of contrastive instance alignment can cause the mismatch in point density and occlusion ratio between pseudo-labeled and true samples distributions in the target domain. To address it, this paper proposes to augment positive easy target samples (with dense points and complete geometry) by their artificial counterparts with sparse points and occlusion. Moreover, this paper constructs a new benchmark using three large-scale datasets. Empirical results show that the proposed method is effective and achieves the new state of the art.