SSAL: Synergizing between Self-Training and Adversarial Learning for Domain Adaptive Object Detection

1 Introduction . Deep convolutional neural network based object detectors have shown promising results , through learning representative features from large annotated datasets [ 7 , 32 , 10 ] . However , like other supervised deep learning methods , object detection methods trained on the source domain do not generalize adequately to a new target domain . This problem , known as domain shift [ 49 ] could be exhibited by change in style , camera pose , or object size and orientation , or the number or location of objects in the scene , among other things . Often , collecting large annotated dataset for fine-tuning the model to the target domain is expensive , error prone and in many cases not possible . Unsupervised Domain Adaptation ( UDA ) is a promising research direction towards solving this problem by transferring knowledge from a labelled source domain to an unlabelled target domain . ∗Corresponding author , Intelligent Machines Lab , Department of Computer Science , Information Technology University of the Punjab , Lahore , Pakistan . Email : akhtar.munir @ itu.edu.pk Project Page : http : //im.itu.edu.pk/synergizing-domain-adaptation/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) Many unsupervised domain adaptive detectors rely on adversarial adaptation or self-training techniques . Methods based on adversarial adaptation [ 4 , 43 , 15 , 17 , 54 , 50 , 3 , 36 ] , mostly rely on domain discriminator for aligning features at image or instance levels . However , due to the absence of labels in target domain they suffer from the challenges of how to pick samples for the adaptation . Selecting uniformly , one ends up missing on infrequent classes or instances . Most importantly adversarial alignment do not explicitly incorporates the class discriminative information , resulting in non-optimal alignment for classification and object detection tasks [ 43 , 4 , 45 ] . A potential solution to this problem is self-training based adaptation , however , it faces the challenge of how to avoid noisy pseudo-labels . Some methods choose high confidence predictions as pseudo-labels [ 27 , 19 , 42 ] , but the likely poor calibration of model under domain shift renders this solution inefficient [ 38 ] . Further , in the case of object detection , prediction probability can not directly capture object localization inaccuracies . We present a principled approach , dubbed as SSAL ( Synergizing between Self-Training and Adversarial Learning for Domain Adaptive Object Detection ) , to achieve right balance between self-training and adversarial alignment for adaptive object detection via leveraging model ’ s predictive uncertainty . To estimate predictive uncertainty of a detection , we propose taking into account variations in both the localization prediction and confidence prediction across Monte-Carlo dropout inferences [ 8 ] . Certain detections are taken as pseudo-labels for self-training , while uncertain ones are used to extract tiles ( regions in image ) for adversarial feature alignment . This synergy between adversarial alignment via tiling around the uncertain object regions and self-training with pseudo-labels from certain object regions lets us include instance-level context for effective adversarial alignment and improve feature discriminability for class-specific alignment . Since we select pseudo-labels with low uncertainty and take relatively uncertain as potential , object-like regions with context ( i.e . tiles ) for adversarial alignment , we tend to reduce the effect of poor calibration under domain shift , thereby improving model ’ s generalization across domains . Our key contributions include the following : ( 1 ) We introduce a new uncertainty-guided framework that strikes the right balance between self-training and adversarial feature alignment for adapting object detection methods . Both pseudo-labelling for self-training and tiling for adversarial alignment are impactful due to their simplicity , generality and ease of implementation . ( 2 ) We propose a method for estimating the object detection uncertainty via taking into account variations in both the localization prediction and confidence prediction across Monte-Carlo dropout inferences . ( 3 ) We show that , selecting pseudo-labels with low uncertainty and using relatively uncertain regions for adversarial alignment , it is possible to address the poor calibration caused by domain shift , and hence improve model ’ s generalization across domains . ( 4 ) Unlike most of the previous methods , we build on computationally efficient one-stage anchor-less object detectors and achieve state-of-the-art results with notable margins across various adaptation scenarios . 2 Related Work . Object detection . Deep learning based object detection algorithms can be classified into either anchor-based [ 40 , 30 , 46 , 2 ] or anchor-free methods [ 26 , 6 , 47 ] . Anchor-based methods , such as Faster RCNN [ 40 ] , uses region proposal network ( RPN ) to generate proposals . Anchor-free detectors , on the other hand , skip proposal generation step and through leveraging fully convolutional network ( FCN ) [ 33 ] directly localize objects . For instance , [ 47 ] proposed per-pixel prediction and directly predicted the class and offset of the corresponding object at each location on the feature map . In this work , we capitalize on the computationally inexpensive characteristic in anchor-free detectors to study adapting trained object detectors . Tiling for object detection . The process of cropping regions of an input image , a.k.a tiling , in a uniform [ 39 ] , random , or informed [ 51 , 16 , 29 ] fashion before applying object detection is typically used to tackle scale variation problem and improve detection accuracy over small objects . Informed tiling can be achieved by first generating a set of regions of object clusters , and then cropping them for subsequent fine detection [ 51 ] . Domain-adaptive object detection . The pioneering work of [ 4 ] on domain-adaptive ( DA ) object detection proposed reducing domain shift at both image and instance levels via embedding adversarial feature adaptation into anchor-based detection pipeline . Global feature alignment could suffer as domains may manifest distinct scene layouts and complex object combinations . Several subsequent approaches attempted to achieve a right balance between the global and instance-level alignments [ 55 , 50 ] . Other methods [ 15 , 23 , 1 , 18 ] improved feature alignment in various ways e.g. , through exploiting hierarchical feature learning in CNNs [ 15 ] . While above methods are built on two-stage pipeline , a few approaches have built domain adaptive detectors on one-stage pipeline [ 22 , 17 ] . [ 17 ] proposed to predict pixel-wise objectness and center-aware feature alignment , building on [ 47 ] , to focus on the discriminative parts of objects . Uncertainty for DA object detection . Exploiting model ’ s predictive uncertainty and entropy optimization have remained subject of interest in prior cross-domain recognition [ 34 , 14 , 35 , 41 ] and detection [ 12 , 37 ] works . For cross-domain recognition , [ 41 ] employed uncertainty for filtering training data and aligning features in Euclidean space . For DA object detection , [ 12 ] proposed an uncertainty metric to regulate the strength of adversarial learning for well-aligned and poorly-aligned samples adaptively . Pseudo-labelling for DA object detection . In DA object detection , pseudo-labelling aims at acquiring pseudo instance-level annotations for incorporating discriminative information . Inoue et al . [ 19 ] generated pseudo instance-level annotations by choosing the top-1 confidence detections . Similarly , [ 42 ] obtained the same by using high-confidence detections and further refined them using tracker ’ s output . Towards refining ( noisy ) pseudo instance-level annotations , [ 21 ] employed auxiliary component and [ 22 ] devised a criterion based on supporting RoIs . Confidence-based pseudo-label selection is prone to generating noisy labels since the model is poorly calibrated under domain shift , eventually causing degenerate network re-training . Unlike most prior methods we build on computationally inexpensive one-stage anchor-free detector . Different to existing methods , we leverage model ’ s predictive uncertainty , considering variations in localization and confidence predictions across MC simulations , to achieve the best of both selftraining and adversarial alignment through mining highly certain target detections as pseudo-labels and relatively uncertain ones as guides in the tiling process . 3 Proposed Method . In this section , we describe the technical details of our method . Fig . 1 displays the overall architecture of our method . We propose to leverage model ’ s predictive uncertainty to strike the right balance between adversarial feature alignment and self-training . To this end , we introduce uncertainty-guided pseudo-labels selection ( UGPL ) for self-training and uncertainty-guided tiling ( UGT ) for adversarial alignment . The former allows generating accurate pseudo-labels to improve feature discriminability for class-specific alignment , while the latter enables extracting tiles on uncertain , object-like regions for effective domain alignment . 3.1 Preliminaries . Problem Setting . Let Ds = { ( xsi , ysi ) } Ns i=1 be the labeled source dataset and Dt = { xtj } Nt j=1 be the unlabeled target dataset . Where ysi = { bsi , csi } is set of bounding boxes bsi for the objects in the image xsi and their corresponding classes c s i ∈ { 1 , . . . , C } . The source and target domains share an identical label space , however , violate the i.i.d . assumption since they are sampled from different data distributions . Our goal is to learn a domain-adaptive object detector , given labeled Ds and unlabeled Dt , capable of performing accurately in the target domain . One-stage anchor-free object detection . Owing to the computationally inexpensive feature of one-stage anchor-free detection pipelines , we build our uncertainty-guided domain-adaptive detector on fully convolutional one-stage object detector ( FCOS ) [ 47 ] . Inspired from the fully convolutional architecture [ 33 ] , FCOS incorporates per-pixel predictions and directly regresses object location . Specifically , it outputs aC-dimensional classification vector , a 4D vector of bounding box coordinates , and a centerness score . The loss function for training FCOS is : Ldet ( cu , v , bu , v ) = 1 Npos ∑ u , v Lcls ( ĉu , v , cu , v ) + 1 Npos ∑ u , v 1ĉu , v > 0Lbox ( b̂u , v , bu , v ) ( 1 ) where Lcls is the classification loss ( i.e . focal loss [ 31 ] , and Lbox ( i.e . IoU loss [ 52 ] ) is the regression loss . ĉu , v , b̂u , v denotes class and bounding box predictions at location ( u , v ) . Npos denotes the number of positive samples . Adversarial feature alignment . Several methods [ 43 , 4 ] align feature maps on the image-level to reduce domain shift via adversarial learning . It involves a global discriminator Dadv that identifies whether the pixels on each feature map belong to the source or the target domain . Specifically , let F ∈ RH×W×K be the K-dimensional feature map of spatial resolution H ×W extracted from the feature backbone network . The output of Dadv is a domain classification map of the same size as F . The discriminator can be optimized using binary cross-entropy loss : Ladv ( xs , xt ) = − ∑ u , v q log ( Dadv ( F s ) u , v ) , + ( 1− q ) log ( 1−Dadv ( F t ) u , v ) ( 2 ) where q is the domain label ∈ { 0 , 1 } . We perform adversarial feature alignment by applying gradient reversal layer ( GRL ) [ 9 ] to source F s and target F t feature maps , in which the sign of gradient is flipped when optimizing the feature extractor via GRL layer . Global alignment is prone to focusing on ( unwanted ) background pixels . We introduce uncertainty-guided tiling , that involves cropping tiles ( regions with context ) around object-like regions for effective adversarial alignment ( sec . 3.2.1 ) . Self-Training . Self-training is a process of training with pseudo-labels , which are generated for unlabelled samples in the target domain with a model trained on labelled data . Hard pseudo instancelevel labels are obtained directly from network class predictions . Let pj , k be the probability outputs vector of a trained network corresponding to a detection ŷj , k , such that pcj , k denotes the probability of class c being present in the detection . With these probabilities , the pseudo-label can be generated for ŷj , k as : ỹcj , k = 1 [ p c j , k ≥ α ] , where α = maxcpcj , k . There could be a significant fraction of incorrectly pseudo-labelled detections used during training . A common strategy to reduce noise during training is to select pseudo-labels corresponding to high-confidence detections [ 19 , 42 ] . Let gj , k be a boolean variable denoting the selection or rejection of ỹj , k i.e . where gj , k = 1 when ỹj , k is selected or otherwise . Formally , in confidence-based selection , a pseudo-label ỹj , k is selected as : gj , k = 1 [ pcj , k ≥ τ ] , where τ is the confidence threshold . These high confidence detections are often noisy because the model is poorly calibrated under domain shift . Instead , we propose to select pseudo-labels utilizing uncertainty in both class prediction and localization prediction to mitigate the impact of poor network calibration ( sec . 3.2.1 ) .

Summary:  In this paper, authors propose an unsupervised domain adaptation method for object detection based on pseudo-labeling and adversarial training. The main idea is two-fold: 1) Monte-Carlo dropout is used to produce the detection uncertainty information that is used to generate pseudo-labels; and 2) regions surrounding uncertain detections are used for adversarial training against the regions from the source domain surrounding the ground truth bounding boxes. The proposed method is compared with the state-of-the-art single-stage and two-stage object detectors on Cityscapes-to-Foggy Cityscapes, KITTI-to-Cityscapes and Sim10K-to-Cityscapes datasets. Additionally, an ablation study examining the contribution of each of the proposed modules to the increased detection precision is presented.