Towards Noise-resistant Object Detection with Noisy Annotations

1 INTRODUCTION . The remarkable success of modern object detectors largely relies on large-scale datasets with extensive bounding box annotations . However , it is extremely expensive and time-consuming to acquire high-quality human annotations . For example , annotating each bounding box in ILSVRC requires 42s on Mechanical Turk ( Su et al. , 2012 ) , whereas the recent OpenImagesV4 Kuznetsova et al . ( 2018 ) reports 7.4 seconds with extreme clicking ( Papadopoulos et al. , 2017b ) . On the other hand , there are ways to acquire annotations at lower costs , such as limiting the annotation time , reducing the number of annotators , or using machine-generated annotations . However , these methods would yield annotations with both label noise ( i.e . wrong classes ) and bounding box noise ( i.e . inaccurate locations ) , which could be detrimental for learning . Learning with label noise has been an active area of research . Some methods perform label correction using the predictions from the model and modify the loss accordingly ( Reed et al. , 2015 ; Tanaka et al. , 2018 ) . Other methods treat samples with small loss as those with clean labels , and only allow clean samples to contribute to the loss ( Jiang et al. , 2018b ; Han et al. , 2018 ) . However , most of those methods focus on the image classification task where the existence of an object is guaranteed . Several recent works have studied object detection with noisy annotations . Zhang et al . ( 2019 ) focus on the weakly-supervised ( WS ) setting where only image-level labels are available , and find reliable bounding box instances as those with low classification loss . Gao et al . ( 2019 ) study a semisupervised ( SS ) setting where the training data contains a small amount of fully-labeled bounding boxes and a large amount of image-level labels , and propose to distill knowledge from a detector pretrained on clean annotations . However , these methods require access to some clean annotations . In this work , we address a more challenging and practical problem , where the annotation contains an unknown mixture of label noise and bounding box noise . Furthermore , we do not assume access to any clean annotations . The entanglement of label noise and bounding box noise increases the difficulty to perform noise correction . A commonly used noise indicator , namely the classification loss , is incapable to distinguish label noise from bounding box noise . Furthermore , it is problematic to correct noise directly using the model predictions , because label correction requires accurate 1Code will be released . bounding box coordinates to crop the object , whereas bounding box correction requires accurate class labels to produce the regression offset . To overcome these difficulties , we propose a two-step noise correction procedure . In the first step , we perform class-agnostic bounding box correction ( CA-BBC ) , which seeks to decouple bounding box noise from label noise , and optimize the noisy ground-truth ( GT ) bounding box regardless of its class label . An illustration of CA-BBC is shown in Figure 1 . It is based on the following intuition : if a bounding box tightly covers an object , then two diverged classifiers would agree with each other and produce the same prediction . Furthermore , both classifiers would have low scores for the background class , i.e. , high objectness scores . Therefore , we directly regress the noisy GT bounding box to minimize both classifier discrepancy and background scores . CA-BBC also has the option to reject a bounding box as false positive if the objectness score is too low . In the second step , we leverage the model ’ s output for label noise correction and class-specific bounding box refinement . It has been shown that co-training two models can filter different types of noise and help each other learn ( Blum & Mitchell , 1998 ; Han et al. , 2018 ; Yu et al. , 2019 ; Chadwick & Newman , 2019 ) . Therefore , we distil knowledge from the ensemble of dual detection heads for noise correction , by generating soft labels and bounding box offsets . We show that soft labels with well-adjusted temperature lead to better performance even for a clean dataset . To summarize , this paper proposes a noise-resistant learning framework to train object detectors with noisy annotations . The proposed framework jointly optimizes object labels , bounding box coordinates , and model parameters by performing alternating noise correction and model training . We conduct experiments on two benchmarks : PASCAL VOC and MS-COCO , which contain different levels of synthetic noise as well as machine-generated noise . The proposed method outperforms previous methods by a large margin . We also provide qualitative results to demonstrate the efficacy of the two-step noise correction , and ablation studies to examine the effect of each component . 2 RELATED WORK . 2.1 CROWDSOURCING FOR OBJECT DETECTION . Crowdsourcing platforms such as Amazon Mechanical Turk ( AMT ) have enabled the collection of large-scale datasets . Due to the formidable cost of human annotation , many efforts have been devoted to reduce the annotation cost . However , even an efficient protocol still report 42.4s to annotate one object in an image ( Su et al. , 2012 ) . Other methods have been proposed which trade off annotation quality for lower cost , by using click supervision ( Papadopoulos et al. , 2017a ) , human-inthe-loop labeling ( Russakovsky et al. , 2015 ; Papadopoulos et al. , 2016 ; Konyushkova et al. , 2018 ) , or exploiting eye-tracking data ( Papadopoulos et al. , 2014 ) . These methods focus on reducing human effort , rather than combating the annotation noise as our method does . 2.2 LEARNING WITH LABEL NOISE . Deep Neural Networks ( DNNs ) can easily overfit to noisy labels in the training data , leading to poor generalization performance ( Zhang et al. , 2017 ) . Many works have addressed learning with label noise . Some approaches correct noise by relabeling the noisy samples ( Vahdat , 2017 ; Veit et al. , 2017 ; Lee et al. , 2018 ) , but they rely on a small set of clean samples for noise correction . Iterative relabeling methods ( Tanaka et al. , 2018 ; Yi & Wu , 2019 ) have been proposed which produce hard or soft labels using the model predictions . Other approaches filter noise by reweighting or selecting training samples ( Jiang et al. , 2018b ; Ren et al. , 2018 ; Chen et al. , 2019b ; Arazo et al. , 2019 ; Li et al. , 2020 ) . Since DNNs learn clean samples faster than noisy ones , samples with smaller classification loss are usually considered to be clean ( Arpit et al. , 2017 ) . To avoid error accumulation during the noise correction process , co-teaching ( Han et al. , 2018 ) trains two networks simultaneously , where each network selects small-loss samples to train the other . Co-teaching+ ( Yu et al. , 2019 ) further keeps the two networks diverged by training on disagreement data . 2.3 WEAKLY-SUPERVISED AND SEMI-SUPERVISED OBJECT DETECTION . Weakly-supervised object detection aims to learn object detectors with only image-level labels . Most existing works formulate it as a multiple instance learning ( MIL ) task ( Dietterich et al. , 1997 ) , where each label is assigned to a bag of object proposals . A common pipeline is to iteratively alternate between mining object instances using a detector and training the detector using the mined instances ( Deselaers et al. , 2010 ; Cinbis et al. , 2017 ) . To address the localization noise in the object proposals , Zhang et al . ( 2019 ) propose an adaptive sampling method which finds reliable instances as those with high classification scores , and use the reliable instances to impose a similarity loss on noisy images . Different from weakly-supervised object detection which assumes that the correct object label is given , our method deals with label noise and bounding box noise at the same time . Semi-supervised methods train object detectors using training data with bounding box annotations for some images and only image-level labels for other images ( Hoffman et al. , 2014 ; Tang et al. , 2016 ; Uijlings et al. , 2018 ; Gao et al. , 2019 ) . Gao et al . ( 2019 ) propose an iterative training-mining framework consisting of detector initialization , box mining , and detector retraining . To address the annotation noise of the mined boxes , they use a detector pretrained on clean annotations for knowledge distillation . Different from all semi-supervised learning methods , our method does not need access to any clean annotations . 3 METHOD . 3.1 OVERVIEW . Given a training dataset with images X , noisy object labels Y , and noisy bounding boxes B , our method aims to train an object detector parameterized by ⇥ , by jointly optimizing Y , B and ⇥ . We first warm-up ⇥ where we train the detector in a standard manner using the original noisy annotations . After the warm-up , we perform alternating optimization on the annotations and the model . Specifically , for each mini-batch of data X = { xi } , Y = { yi } , B = { bi } , we first keep ⇥ fixed and perform noise correction to update Y and B , then we used the corrected annotations to update ⇥ . An overview of the algorithm is shown in Algorithm 1 . We use a popular two-stage object detector ( i.e . Faster-RCNN ( Ren et al. , 2015 ) ) , which consists of a backbone feature extractor parameterized by ✓cnn , a Region Proposal Network ( RPN ) ✓rpn , a classification head ✓c , and a bounding box ( bbox ) regression head ✓b . Note that ✓c and ✓b have shared layers . Let detection head with parameters ✓d denote the union of the classification head and the bbox regression head . During training , we simultaneously train two detection heads ✓1d = { ✓1c , ✓1b } and ✓2d = { ✓2c , ✓2b } , which are kept diverged from each other by different ( random ) parameter initializations and different ( random ) training instance ( i.e . RoI ) sampling . Due to the entanglement of an unknown mixture of label noise and bbox noise , it is difficult to correct both types of noise in a single step . Therefore , we propose a two-step noise correction method . In the first step , we perform class-agnostic bounding box correction ( CA-BBC ) , which disentangles bbox noise from label noise . In the second step , we utilize the outputs from dual detection heads for label noise correction and class-specific bbox refinement . Figure 2 shows an illustration of our framework . Next we delineate the details . Algorithm 1 : alternating two-step noise correction and model training . 1 Input : model ⇥ = { ✓cnn , ✓rpn , ✓1d , ✓2d } , noisy training dataset ( X , Y , B ) . 2 while not MaxIters do 3 Mini-batch X = { xi } , Y = { yi } , B = { bi } . 4 for b in B do 5 Update b ! b⇤ with CA-BBC ( Eq . 2 & 3 ) . 6 end 7 for ( y , b⇤ ) in ( Y , B⇤ ) do 8 Update y ! y⇤ with dual-head soft label correction ( Eq . 4 & 5 ) . 9 Update b⇤ ! b⇤⇤ with class-specific bbox refinement ( Eq . 6 ) . 10 end 11 Update ⇥ by SGD on Lrpn ( B⇤⇤ ) , L1+2cls ( Y ⇤ ) , L1+2loc ( B ⇤⇤ , Y ⇤ ) . 12 end

Summary: In this work the authors propose a framework to perform object detection when there is noise present in class labels as well as bounding box annotations. The authors propose a two-step process, where in the first step the bounding boxes are corrected in class-agnostic way, and in the second step knowledge distillation has been used to correct the class labels. The propose method has been evaluated on two different datasets with synthetic noise.