SSAL: Synergizing between Self-Training and Adversarial Learning for Domain Adaptive Object Detection

1 Introduction . Deep convolutional neural network based object detectors have shown promising results , through learning representative features from large annotated datasets [ 7 , 32 , 10 ] . However , like other supervised deep learning methods , object detection methods trained on the source domain do not generalize adequately to a new target domain . This problem , known as domain shift [ 49 ] could be exhibited by change in style , camera pose , or object size and orientation , or the number or location of objects in the scene , among other things . Often , collecting large annotated dataset for fine-tuning the model to the target domain is expensive , error prone and in many cases not possible . Unsupervised Domain Adaptation ( UDA ) is a promising research direction towards solving this problem by transferring knowledge from a labelled source domain to an unlabelled target domain . ∗Corresponding author , Intelligent Machines Lab , Department of Computer Science , Information Technology University of the Punjab , Lahore , Pakistan . Email : akhtar.munir @ itu.edu.pk Project Page : http : //im.itu.edu.pk/synergizing-domain-adaptation/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) Many unsupervised domain adaptive detectors rely on adversarial adaptation or self-training techniques . Methods based on adversarial adaptation [ 4 , 43 , 15 , 17 , 54 , 50 , 3 , 36 ] , mostly rely on domain discriminator for aligning features at image or instance levels . However , due to the absence of labels in target domain they suffer from the challenges of how to pick samples for the adaptation . Selecting uniformly , one ends up missing on infrequent classes or instances . Most importantly adversarial alignment do not explicitly incorporates the class discriminative information , resulting in non-optimal alignment for classification and object detection tasks [ 43 , 4 , 45 ] . A potential solution to this problem is self-training based adaptation , however , it faces the challenge of how to avoid noisy pseudo-labels . Some methods choose high confidence predictions as pseudo-labels [ 27 , 19 , 42 ] , but the likely poor calibration of model under domain shift renders this solution inefficient [ 38 ] . Further , in the case of object detection , prediction probability can not directly capture object localization inaccuracies . We present a principled approach , dubbed as SSAL ( Synergizing between Self-Training and Adversarial Learning for Domain Adaptive Object Detection ) , to achieve right balance between self-training and adversarial alignment for adaptive object detection via leveraging model ’ s predictive uncertainty . To estimate predictive uncertainty of a detection , we propose taking into account variations in both the localization prediction and confidence prediction across Monte-Carlo dropout inferences [ 8 ] . Certain detections are taken as pseudo-labels for self-training , while uncertain ones are used to extract tiles ( regions in image ) for adversarial feature alignment . This synergy between adversarial alignment via tiling around the uncertain object regions and self-training with pseudo-labels from certain object regions lets us include instance-level context for effective adversarial alignment and improve feature discriminability for class-specific alignment . Since we select pseudo-labels with low uncertainty and take relatively uncertain as potential , object-like regions with context ( i.e . tiles ) for adversarial alignment , we tend to reduce the effect of poor calibration under domain shift , thereby improving model ’ s generalization across domains . Our key contributions include the following : ( 1 ) We introduce a new uncertainty-guided framework that strikes the right balance between self-training and adversarial feature alignment for adapting object detection methods . Both pseudo-labelling for self-training and tiling for adversarial alignment are impactful due to their simplicity , generality and ease of implementation . ( 2 ) We propose a method for estimating the object detection uncertainty via taking into account variations in both the localization prediction and confidence prediction across Monte-Carlo dropout inferences . ( 3 ) We show that , selecting pseudo-labels with low uncertainty and using relatively uncertain regions for adversarial alignment , it is possible to address the poor calibration caused by domain shift , and hence improve model ’ s generalization across domains . ( 4 ) Unlike most of the previous methods , we build on computationally efficient one-stage anchor-less object detectors and achieve state-of-the-art results with notable margins across various adaptation scenarios . 2 Related Work . Object detection . Deep learning based object detection algorithms can be classified into either anchor-based [ 40 , 30 , 46 , 2 ] or anchor-free methods [ 26 , 6 , 47 ] . Anchor-based methods , such as Faster RCNN [ 40 ] , uses region proposal network ( RPN ) to generate proposals . Anchor-free detectors , on the other hand , skip proposal generation step and through leveraging fully convolutional network ( FCN ) [ 33 ] directly localize objects . For instance , [ 47 ] proposed per-pixel prediction and directly predicted the class and offset of the corresponding object at each location on the feature map . In this work , we capitalize on the computationally inexpensive characteristic in anchor-free detectors to study adapting trained object detectors . Tiling for object detection . The process of cropping regions of an input image , a.k.a tiling , in a uniform [ 39 ] , random , or informed [ 51 , 16 , 29 ] fashion before applying object detection is typically used to tackle scale variation problem and improve detection accuracy over small objects . Informed tiling can be achieved by first generating a set of regions of object clusters , and then cropping them for subsequent fine detection [ 51 ] . Domain-adaptive object detection . The pioneering work of [ 4 ] on domain-adaptive ( DA ) object detection proposed reducing domain shift at both image and instance levels via embedding adversarial feature adaptation into anchor-based detection pipeline . Global feature alignment could suffer as domains may manifest distinct scene layouts and complex object combinations . Several subsequent approaches attempted to achieve a right balance between the global and instance-level alignments [ 55 , 50 ] . Other methods [ 15 , 23 , 1 , 18 ] improved feature alignment in various ways e.g. , through exploiting hierarchical feature learning in CNNs [ 15 ] . While above methods are built on two-stage pipeline , a few approaches have built domain adaptive detectors on one-stage pipeline [ 22 , 17 ] . [ 17 ] proposed to predict pixel-wise objectness and center-aware feature alignment , building on [ 47 ] , to focus on the discriminative parts of objects . Uncertainty for DA object detection . Exploiting model ’ s predictive uncertainty and entropy optimization have remained subject of interest in prior cross-domain recognition [ 34 , 14 , 35 , 41 ] and detection [ 12 , 37 ] works . For cross-domain recognition , [ 41 ] employed uncertainty for filtering training data and aligning features in Euclidean space . For DA object detection , [ 12 ] proposed an uncertainty metric to regulate the strength of adversarial learning for well-aligned and poorly-aligned samples adaptively . Pseudo-labelling for DA object detection . In DA object detection , pseudo-labelling aims at acquiring pseudo instance-level annotations for incorporating discriminative information . Inoue et al . [ 19 ] generated pseudo instance-level annotations by choosing the top-1 confidence detections . Similarly , [ 42 ] obtained the same by using high-confidence detections and further refined them using tracker ’ s output . Towards refining ( noisy ) pseudo instance-level annotations , [ 21 ] employed auxiliary component and [ 22 ] devised a criterion based on supporting RoIs . Confidence-based pseudo-label selection is prone to generating noisy labels since the model is poorly calibrated under domain shift , eventually causing degenerate network re-training . Unlike most prior methods we build on computationally inexpensive one-stage anchor-free detector . Different to existing methods , we leverage model ’ s predictive uncertainty , considering variations in localization and confidence predictions across MC simulations , to achieve the best of both selftraining and adversarial alignment through mining highly certain target detections as pseudo-labels and relatively uncertain ones as guides in the tiling process . 3 Proposed Method . In this section , we describe the technical details of our method . Fig . 1 displays the overall architecture of our method . We propose to leverage model ’ s predictive uncertainty to strike the right balance between adversarial feature alignment and self-training . To this end , we introduce uncertainty-guided pseudo-labels selection ( UGPL ) for self-training and uncertainty-guided tiling ( UGT ) for adversarial alignment . The former allows generating accurate pseudo-labels to improve feature discriminability for class-specific alignment , while the latter enables extracting tiles on uncertain , object-like regions for effective domain alignment . 3.1 Preliminaries . Problem Setting . Let Ds = { ( xsi , ysi ) } Ns i=1 be the labeled source dataset and Dt = { xtj } Nt j=1 be the unlabeled target dataset . Where ysi = { bsi , csi } is set of bounding boxes bsi for the objects in the image xsi and their corresponding classes c s i ∈ { 1 , . . . , C } . The source and target domains share an identical label space , however , violate the i.i.d . assumption since they are sampled from different data distributions . Our goal is to learn a domain-adaptive object detector , given labeled Ds and unlabeled Dt , capable of performing accurately in the target domain . One-stage anchor-free object detection . Owing to the computationally inexpensive feature of one-stage anchor-free detection pipelines , we build our uncertainty-guided domain-adaptive detector on fully convolutional one-stage object detector ( FCOS ) [ 47 ] . Inspired from the fully convolutional architecture [ 33 ] , FCOS incorporates per-pixel predictions and directly regresses object location . Specifically , it outputs aC-dimensional classification vector , a 4D vector of bounding box coordinates , and a centerness score . The loss function for training FCOS is : Ldet ( cu , v , bu , v ) = 1 Npos ∑ u , v Lcls ( ĉu , v , cu , v ) + 1 Npos ∑ u , v 1ĉu , v > 0Lbox ( b̂u , v , bu , v ) ( 1 ) where Lcls is the classification loss ( i.e . focal loss [ 31 ] , and Lbox ( i.e . IoU loss [ 52 ] ) is the regression loss . ĉu , v , b̂u , v denotes class and bounding box predictions at location ( u , v ) . Npos denotes the number of positive samples . Adversarial feature alignment . Several methods [ 43 , 4 ] align feature maps on the image-level to reduce domain shift via adversarial learning . It involves a global discriminator Dadv that identifies whether the pixels on each feature map belong to the source or the target domain . Specifically , let F ∈ RH×W×K be the K-dimensional feature map of spatial resolution H ×W extracted from the feature backbone network . The output of Dadv is a domain classification map of the same size as F . The discriminator can be optimized using binary cross-entropy loss : Ladv ( xs , xt ) = − ∑ u , v q log ( Dadv ( F s ) u , v ) , + ( 1− q ) log ( 1−Dadv ( F t ) u , v ) ( 2 ) where q is the domain label ∈ { 0 , 1 } . We perform adversarial feature alignment by applying gradient reversal layer ( GRL ) [ 9 ] to source F s and target F t feature maps , in which the sign of gradient is flipped when optimizing the feature extractor via GRL layer . Global alignment is prone to focusing on ( unwanted ) background pixels . We introduce uncertainty-guided tiling , that involves cropping tiles ( regions with context ) around object-like regions for effective adversarial alignment ( sec . 3.2.1 ) . Self-Training . Self-training is a process of training with pseudo-labels , which are generated for unlabelled samples in the target domain with a model trained on labelled data . Hard pseudo instancelevel labels are obtained directly from network class predictions . Let pj , k be the probability outputs vector of a trained network corresponding to a detection ŷj , k , such that pcj , k denotes the probability of class c being present in the detection . With these probabilities , the pseudo-label can be generated for ŷj , k as : ỹcj , k = 1 [ p c j , k ≥ α ] , where α = maxcpcj , k . There could be a significant fraction of incorrectly pseudo-labelled detections used during training . A common strategy to reduce noise during training is to select pseudo-labels corresponding to high-confidence detections [ 19 , 42 ] . Let gj , k be a boolean variable denoting the selection or rejection of ỹj , k i.e . where gj , k = 1 when ỹj , k is selected or otherwise . Formally , in confidence-based selection , a pseudo-label ỹj , k is selected as : gj , k = 1 [ pcj , k ≥ τ ] , where τ is the confidence threshold . These high confidence detections are often noisy because the model is poorly calibrated under domain shift . Instead , we propose to select pseudo-labels utilizing uncertainty in both class prediction and localization prediction to mitigate the impact of poor network calibration ( sec . 3.2.1 ) .

Summary: The work proposes a UDA pipeline for object detection based on self-training (via pseudo-labeling) and adversarial alignment. To this end, Monte-Carlo dropout inference is used to estimate detection confidence (or uncertainty). Certain detections (with  associated pseud-labels) are used for self-training while  uncertain ones are used to extract tile for alignment.