Towards Noise-resistant Object Detection with Noisy Annotations
Authors: Junnan Li
Caiming Xiong
Steven Hoi
Venue:  (2021)
Keywords: noisy annotation
object detection
label noise

Abstract: Training deep object detectors requires large amounts of human-annotated images with accurate object labels and bounding box coordinates, which are extremely expensive to acquire.  Noisy annotations are much more easily accessible, but they could be detrimental for learning. We address the challenging problem of training object detectors with noisy annotations, where the noise contains a mixture of label noise and bounding box noise. We propose a learning framework which jointly optimizes object labels, bounding box coordinates, and model parameters by performing alternating noise correction and model training. To disentangle label noise and bounding box noise, we propose a two-step noise correction method. The first step performs class-agnostic bounding box correction, and the second step performs label correction and class-specific bounding box refinement. We conduct experiments on PASCAL VOC and MS-COCO dataset with both synthetic noise and machine-generated noise. Our method achieves state-of-the-art performance by effectively cleaning both label noise and bounding box noise.

Under review as a conference paper at ICLR 2021
TOWARDS NOISE-RESISTANT OBJECT DETECTION WITH NOISY ANNOTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Training deep object detectors requires large amounts of anman-annotated images with accurate object labels and bounding box coordinates, which are extremely expensive to acquire. Noisy annotations are much more easily accessible, but they could be detrimental for learning. We address the challenging problem of train- ing object detectors with noisy annotations, where the noise contains a mixture of label noise and bounding box noise. We propose a learning framework which jointly optimizes object labels, bounding box coordinates, and model parameters by performing alternating noise correction and model training. To disentangle la- bel noise and bounding box noise, we propose a two-step noise correction method. The first step performs class-agnostic bounding box correction, and the second step performs label correction and class-specific bounding box refinement. We conduct experiments on PASCAL VOC and MS-COCO dataset with both syn- thetic noise and geeeerre-geeerteed noise. Our method achieves state-of-the-art performance by effectively cleaning both label noise and bounding box noise
INTRODUCTION
The remarkable success of modern object detectors largely relies on large-scale datasets with exten- sive bounding box annotations. However, it is extremely expensive and time-consuming to acquire high-quality human annotations. For example, annotating each bounding box in ILSVRC requires 42s on Mechanical Turk (Su et al. 2012). whereas the recent OpenImagesVa Kuznetsova et al. 2018)) reports 7.4 seconds with extreme clicking (Papadopoulos et al. 2017b). On the other hand, there are ways to acquire annotations at lower costs, such as limiting the annotation time, reducing the number of annotators, or using srcciie--genrrtee annotations. However, these methods would yield annotations with both label noise (i.e. wrong classes) and bounding box noise (i.e. inaccurate locations), which could be detrimental for learning.
Learning with label noise has been an active area of research. Some methods perform label correc- tion using the predictions from the model and modify the loss accordingly (Reed al. 2015 Tanaka et al. 2018). Other methods treat samples with small loss as those with clean labels, and only allow clean samples contribute the loss Tiang et al. 20186 Han al. 2018). However, most of those methods focus on the image classification task where the existence of an object is guaranteed.
Several recent works have studied object detection with noisy annotations. Zhang et al. (2019) focus on the weakky-supervised (WS) setting where only image-level labels are available, and find reliable bounding box instances as those with low classification loss. [iao supervised (SS) setting where the training data contains a small amount of ful fully- lalleed bounding boxes and a large amount image-level labels, and propose to distill knowledge from a detector pretrained on clean annotations. However, these methods require access to some clean annotations.
(2019) study a semi-
In this work, we address a more challenging and practical problem, where the annotation contains an unknown mixture of label noise and bounding box noise. Furthermore, we do not assume access label noise and bounding box noise increases the difficulty perform noise correction. A commonly used noise indicator, namely the classification loss, is incapable to distinguish label noise from bounding box noise. Furthermore, is problematic to correct noise directly using the model predictions, because label correction requires accurate
to any clean annotations. The entanglement
Code will he released.

---

Under review as a conference paper at ICLR 2021
Minimize Clossifier Discreponcy (s)
Maximize Region Objectness -(p,
Noisy GT
Feature Extractor
PrRO1 Pooling
Classification
Head A
No
Reject?
Classification Head
Yes
DC(b)
6*
Remove
b*
Our Class-Agnoste Bounding Box Correction (CA-BBC) disentangles bounding box (hbox) noise from label noise, by directly optimizing the noisy bbox coordinates regardless of its class label. We use diverged classifiers to predict the same image region, and update the bbox & to b" by minimizing classifier discrepancy and maximizing region objectness. Boxes with very low objectness are rejected as false positives.
Figure 1:
bounding box coordinates to crop the object, whereas bounding box correction requires accurate class labels to produce the regression offset.
To overcome these difficulties, we propose two-step noise correction procedure. In the first step, we perform class-agnostic bounding box correction (CA-BBC), which seeks to decouple bounding box noise from label noise, and optimize the noisy ground-truth (GT) bounding box regardless of its class label. An illustration of CA-BBC is shown in Figure T It is based on the following intuition: if bounding box tightly covers object, then two diverged classifiers would agree with each other and produce the same prediction. Furthermore, both classifiers would have low scores for the background class, i.e., high objectness scores. Therefore, we directly regress the noisy GT bounding box to minimize both classifier discrepancy and background scores. CA-BBC also has the option to reject a bounding box false positive the objectness score too low.
In the second step, we leverage the model's output for label noise correction and class-specific bounding box refinement. has been shown that co co-training two models can filter different types of noise and help each other learn (Blum & Mitchell, 1998 Han et al. 2018 Yu et al. 2019 Chadwick Newman 2019p. Therefore, we distil knowledge from the ensemble of dual detection heads for noise correction, by generating soft labels and bounding box offsets. We show that soft labels with well-adjusted temperature lead to better performance even for clean dataset.
To summarize, this paper proposes noise-resistant learning framework to train object detectors with noisy annotations. The proposed framework jointly optimizes object labels, bounding box co- ordinates, and model parameters by performing alternating noise correction and model training. We conduct experiments on two benchmarks: PASCAL VOC and MS-COCO, which contain different levels of synthetic noise well machine- generated noise. The proposed method outperforms previous methods by a large margin. We also provide qualitative results to demonstrate the efficacy of the two-step noise correction, and ablation studies examine the effect of each component.
machine-gemerated
RELATED WORK
2.1 CROWDSOURCING FOR ORJECT DETECTION
Crowdsourcing platforms such as Amazon Mechanical Turk (AMT) have enabled the collection of large-scale datasets. Due to the formidable cost of human annotation, many efforts have been devoted to reduce the annotation cost. However, even an efficient protocol still report 42.4s to annotate one object in image (Su et al. 2012). Other methods have been proposed which trade off annotation quality for lower cost, by using click supervision (Papadopoulos et al. 2017a), human-in- loop labeling (Russakovsky al. 2015; Papadopoulos et al. [016 Konyushkova exploiting eye-tracking data (Papadopoulos et al., 2014). These methods focus on reducing human effort, rather than combating the annotation noise method does.
2018). or
2

---

Under review as a conference paper at ICLR 2021
2.2
LEARNING WITH LABEL NOISE
Deep Neural Networks (DNNs) can easily overfit to noisy labels in the training data, leading to poor generalization performance Zhang et al., 2017). Many works have addressed learning with label noise. Some approaches correct noise by relabeling the noisy samples (Vahdar 2017; Veit et al. 2017 Lee et al. 2018). , but they rely on a small set of clean samples for noise correction. Iterative relabeling methods (Tanaka al. 2018 Yi & Wu 2019) have been proposed which produce hard or soft labels using the model predictions. Other approaches filter noise by reweighting or selecting
training samples Jiang et al. 2018 Ren Re al. 2018 Chen et al. 2019b Arazo al.. 2019 Li et al. 2020). Since DNNs learn clean samples faster than noisy ones, samples with smaller classification loss are usually considered be clean (Arpit et al. 2017). To avoid error accumulation during the noise correction process, co-teaching (Han et al. 20 8) trains two networks simultancously. where each network selects small-loss samples to train the other. Co-teaching+ YYu et al. 2019 further
keeps the two networks diverged by training on disagreement data.
2.3 WEARLY-SUUPERVISED AND SEMI-SUPERVISED OBJECT DETECTION
Weakly-supervised object detection aims to learn object detectors with only image-level labels. Most existing works formulate it as a multiple instance learning (MIL) task (Dietterich et al.] 1997), where each label is assigned to bag of object proposals. A common pipeline is to iteratively alternate between mining object instances using detector and training the detector using the mined instances (Deselaers al. 2010 Cinbis et al. 2017 To address the localization noise in the object proposals. Zhang et al. (2019) propose an adaptive sampling method which finds reliable instances as those with high classification scores, and use the reliable instances impose a similarity loss on noisy images. Different from weakly-supervisee object detection which assumes that the correct object label is given, our method deals with label noise and bounding box noise at the same time.
Semi- supervied methods train object detectors using training data with bounding box annotations for some images and only image-level labels for other images (Hofhman et al. 2014 Tang et al.) 2016 Uijlings et al. 2018, Gao et al., 2019). Gao et al. (2019) propose an iterative maining-mining framework consisting of detector initialization, box mining, and detector retraining. To address the annotation noise of the mined boxes, they use detector pretrained on clean annotations for knowledge distillation. Different from all semi-supervised learning methods, our method does not need access any clean annotations.
3 METHOD
3.1
OVERVIEW
Given a training dataset with images X, noisy object labels V, and noisy bounding boxes B, our method aims to train an object detector parameterized by θ, by jointly optimizing V, B and e. We first warm-up θ where we train the detector in standard manner using the original noisy annotations. After the warm-up, we perform alternating optimization on the annotations and the model. Specifically, for each mini-batch of data X (x.)). Y = {, B (bi), first keep θ fixed and perform noise correction to update Y and B, then we used the corrected annotations to update O An overview of the algorithm is shown in Algorithm
We use a popular two-stage object detector (i.e. Faster-RCNN (Ren et al. 2015)), )) which consists of a backbone feature extractor parameterized by 0can. Region Proposal Network (RPN) 0 Orpm. a clas- sification head 6., and a bounding box (bbox) regression head 6g. Note that 6g and 6, have shared layers. Let detection head with parameters 6g denote the union of the classification head and the bbox regression head. During training, we simultaneously train two detection heads 02 (0), 0,) and 02 = (02, 02) which are kept diverged from each other by different (random) parameter initial- izations and different (random) training instance (i.e. Rol) sampling.
Due to the entanglement of an unknown mixture of label noise and bbox noise, it is difficult to correct both types of noise in single step. Therefore, we propose two-step noise correction method. In the first step, we perform class-agnostic bounding box correction (CA-BBC), which disentangles bbox noise from label noise. In the second step, we utilize the outputs from dual detection heads for label noise correction and class specific bbox refinement. Figure 2 shows an illustration of our framework. Next we delineate the details.
3

---

Under review as a conference paper at ICLR 2021
Noiey GT 68ox
Class Agnostic BBox Correction
Detection
Two- Two-Step
scalel
Caat"
Noise Correction
Feature Extractor
PYROI
Detection
Average
Sharpen
Noisy GT Label
"Dos"
Arpp(("")
RPN
Sampling
Detection Head
4'00")
chxc("")e")
Model
PIROI Pooling
Training
Detection
4800))
Sampling
Figure 2: The proposed framework alternately performs noise correction (with fixed model parameters) and model training (with corrected annotations) for each mini-batch. The noise correction procedure consists of steps: (1) the class-agnostic bounding box correction (Figure [ disentangles bbox noise and label noise; (2) class-specific correction step uses dual detection heads generate soft labels label correction and refine bounding boxes using class spesssicciifcc bbox offsets. The two detection heads are kept diverged by different random parameter initialization and different random Rol sampling during training.
Algorithm 1: alternating two-step noise correction and model training.
Input: model {0ean, Orpns f1 (8), noisy training dataset (X, V. B). while not MaxIters do
Mini-batch X = {. ) [. {b:).
for b in B do
Update + b* with CA-BBC (Eq. end
(y.b") in (Y, B²) do for
Update y y with dual-head soft label correction (Eqq.
Update 3 - h** with class-specific bhox refinement (Eq.
10
end
Update + by SGD on (B* 1 (Y*), 1+2 ( (B*, Y*).
11
12
end
3.2 CLASS-AGNOSTIC BOUNDING BOX CORRECTION
We first correct bounding box noise by updating - B" regardless of the label noise Y. As illustrated Figure CA-BBC uses two diverged classification heads to produce two sets of class predictions on the same image region, and updates the bounding box to minimize classifier discrep- ancy and maximize region objectness. The intuition is: if bounding box tightly covers an object, then two classifiers would agree with each other and produce the same predictions. Moreover, both predictions would have low scores on the background class.
Specifically, given an image each noisy GT bounding extract a fixed-sized feature Φ(x, b). Then we give the Rol feature to the two classification heads to produce two sets of softmax predictions over C + classes (including the background class), Pi (ő(x, b); 02) and p2 (2(((, b); 02). For simplicity we denote them P1 and p2. The discrepancy between the two predictions defined as their L.2 distance:
X, the backbone first extracts a convolutional feature map. For b E B, we perform a Rol- Kol-Pooling operation on the feature map to
D(p,, P2)
1| P2l12
4

---

Under review as a conference paper at ICLR 2021
Minimizing the classifier discrepancy w.zf the bounding box will push it region where the two classifiers agree on its class label. To prevent the bounding box from simply moving to a background region, we also minimize the classifiers' scores on the background class, P1 and bg In other words, want maximize the objectness of the region covered by the bounding box.
Therefore, we aim to find the optimal b* that minimizes the following objective function:
[b) P2) + X + P2 bR).
where À controls the balance of the two terms and to 0.1 in our experiments.
For faster speed, we estimate b* by performing single step of gradient descent to update b:
aL(b)
b" = - ab
3
where a is the step size.
Since Rol-Pooling (Ren et al. 2015) or Rol-Align (He al., 2017) performs discrete sampling on the feature map to generate ø xz b). L(b) is not differentiable w.r.t b. Therefore, adopt the Precise Rol-Pooling method (Jiang et al. 2018a), which avoids any quantization of coordinates and has a continuous gradient b.
In order to handle false positive bboxes that do not cover any object, we add a reject option which removes b from the ground-truth if both classifiers give low objectness score (high background bg 0.9.
score), P1 bg 0.9 and P2
3.3 DUAL-HEAD DISTILLATION FOR NOISE CORRECTION
In the second step, we perform class-specific self-distillation for label noise correction and bbox refinement. We simultancously train two diverged detection heads which can filter different types of noise, and distil knowledge from their ensemble to clean the annotation noise. Using the ensemble of two heads helps alleviate the confirmation bias problem (i.e. model confirms its own mistakes) that commonly occurs in self-training.
Soft label correction. Given the Rol feature ((x, b*), the two classification heads produce two sets of softmax predictions over object classes, pi and pi- Inspired by the bootstrapping method (Reed the classifiers' predictions to update the noisy GT label. Let y {0, 1j
ctt al. 2015). represent the GT label as a one-hot vector over C' classes, we create the soft label by first averaging the classifiers' predictions and the GT label:
ÿ (pi + P2 3)/3.
(4)
Then we apply a sharpening function the soft label to reduce the entropy of the label distribution. The sharpening operation is defined as:
w" = 5 -.t.c=L..... 1,2.......
(S)
where hft is the score for class The temperature T controls the 'softness" of the label and is set to 0.4 in our experiments. A lower temperature decreases the softness and has the implicit effect of entropy minimization, which encourages the model produce high confidence predictions and allows better decision boundary to be learned (Grandvalet & Bengio 2005 Berthelot et al. 2019).
Class-specific bounding box refinement. The bbox regression heads produce two sets of per- class bounding box regression offsets, t and fz. Let c* denote the class with the highest score in the soft label, i.e. c* arg max, yc: e 1,2, ..., C. We refine the bounding box b* by merging the class-specific outputs from both bbox regression heads:
1 (1 + 15 1/2
(6)
6** = 6" + pl,
where 1e and E5" are the bounding box offsets for class ct, and controls the magnitude of the refinement.
5

---

Under review as a conference paper at ICLR 2021
BBox Noise
20%
40%
Label Noise
0%
20%
40%
0%
20%
40%
vanilla (Ren et al. 2015)
75.5
70.7
66.9
59.3
54.2
50.0
objectness maximization
75.8
71.1
67.4
63.2
59.6
55.7
CA-BBC
76.8
72.4
68.0
67.8
64.7
61.7
Evaluation of class class-agnostic bounding box correction. Numbers are mAP@ .5 on VOC2007 test set for models trained with different mixtures of lahel noise and bbox noise.
Table I:
3.4 MODEL TRAINING
Let Y and 8* denote mini-batch of soft labels and refined bounding boxes, respectively, We use them as the new GT to train the model. Specifically, we update @ = {0em Orpa 02. 62) to optimize the following losses: (1) the loss function of RPN defined in (Ren et al. 2015). (xpn(B"") (2) the classification loss for the two detection heads, Ldx and (Y"), defined as the cross- entropy loss Σ, - lpg(p:); (3) the localization loss for the detection heads, Lic Y*) and
Lioe (B*r, Y"), defined as the smooth LI loss (Gishicck 2015
4 EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION DETAILS
Since most available datasets for object detection have been extensively verified by human annota- tors and contain little noise, we created noisy annotations using two popular benchmark datasets, PASCAL VOC (Everingham
al. 2010) and MS-COCO (Lin et al.] 2014). First, we generated synthetic noise to simulate human mistakes of different severity, by corrupting the training anno- tation with mixture of label noise and bounding box noise. For label noise, we follow previous works Ciang et al. 2018b Arazo et al., 2019) and generate symmetric label noise. Specifically, we randomly choose 4% of the training samples and change each of their labels another random label. For bounding box noise, we perturb the coordinates of all bounding boxes by a number of pixels uniformly drawn from [-wN,, %, +NN (w is bbox width) for horizontal coordinates or N,,,,% +hN,%% (h is bbox height) for vertical coordinates. We experiment with multiple com- binations of label noise ranging from 0% to 60% and bounding box noise ranging from 0% to 40%. Under 40% bbox noise, the average IoU between noisy bbox and its corresponding clean bbox is only 0.45. For VOC, we use the union set of trainva/2007 and trainval2012 as training data, and test2007 as test data. We report mean average precision (mAP@ .5) the evaluation metric. For MS-COCO, we use train2017 as training data, and report mAp@ 5 and mAP@ .55. on val2017.
We also mined large amounts of free training data with noisy annotations by using machine- generated annotations on unlabeled images. We first train a Faster R-CNN detector on 10% of labeled data from COCO train2017, which has a validation mAP@.5 5 of 40.5. Then we use the trained detector to annotate unlabeled2017, which contains 123k unlabeled images. We use COCO unlabeled2017 with machine- generated annotations as our noisy training data.
We use the common Faster-RCNN (Ren et al. 2015) architecture with ResNet-50 (He et al. 2016 and FPN (Lin al. 2017) the feature extractor. We train the model using SGD with a learning rate of 0.02, a momentum of 0.9, and weight decay of 1e 4. The hyper-parameters are set as À 0.1, 0.4, p = 0.5, and a E {0, 100, 200), which are determined by the validation performance on 10% of training data with clean annotations (only used for validation). We implement our framework based on the mmdetection toolbox (Chen et al. 2019a). In terms of computation time, our method increases training time by ~44 compared to vanilla training. During inference, we only use the first detection head unless otherwise specified, which does not increase inference time.
4.2 EVALUATION ON CA-BBC
First, we evaluate the effect of the proposed CA-BBC method by itself. We train detector fol- lowing the proposed learning framework, except that we only perform the first step of noise cor- rection (i.e. CA-BBC). Table [ shows the results VOC with different mixtures of label noise and bounding box noise. Compared to vanilla training without any noise correction (Ren et al. 2015 Chen al. 2019a), performing CA-BBC can significantly improve performance, especially

---

Under review as a conference paper at ICLR 2021
Figure 3: Examples of class-agnostic bounding box correction on VOC with 40% label noise and 40% bound- ing box noise. Noisy GT bounding boxes are in red and the corrected bounding hoxes are in green.
Figure 4: Examples of failed bbox corrections from CA-BBC. Noisy GT bounding boxes in red and the corrected bounding boxes are in green. CA-BBC could be confused when the GT bboxes cover multiple object instances of the same class. It also sometimes fails to extend the box boundaries to cover the entire object.
BBox Noise
0%
20%
40%
Label Noise
0%
20%
40%
60%
20%
40%
60%
0%
20%
60%
Vanilla (Ren et al. 2015
78.2
72.9
69.3
62.1
75.5
70.7
66.9
61.2
59.3
54.2
50.0
45.9
Co-teaching (Han al.
2018
78.3
76.5
74.1
69.9
75.6
73.2
69.7
65.1
60.6
59.7
55.8
50.4
SD-LocNet Zhang et al..
2019
78.0
75.3
73.0
66.2
75.3
72.1
67.5
64.0
59.7
58.7
54.5
49.2
NOTE-RCNN (Gao et al.
2019
78.6
76.7
74.9
69.9
76.0
73.7
70.1
65.8
63.4
61.5
57.8
53.7
Ours
80.1
79.1
77.7
74.1
77.9
76.7
74.8
71.9
71.9
70.6
69.1
64.5
Table 2: Results VOCO7 test with different levels of training noise.
Numbers indicate mAP@.5.
for higher level of bbox noise. The improvement is consistent despite the increase of label noise, which demonstrates the ability of CA-BBC to disentangle the two types of noise and effectively correct bbox noise. We also demonstrate the effect of the proposed discrepancy minimization by removing D(P1-P2) from the loss in Eq. and only maximize the objectness of the bbox region. which leads to lower performance. Figure show qualitative examples of CA-BBC. The noisy GT bboxes are shown in red whereas the corrected bboxes are shown in green. CA-BBC can update the bounding boxes to more accurately capture the objects of interest.
4.3 COMPARISON WITH THE STATE-OF-THE-AN
We evaluate our full learning framework with two-step noise correction and compare it with multiple existing methods for learning with noisy annotations. We implement all methods using the same network architecture. Since previous methods operate in different settings ours, we adapt them for our problem to construct strong baselines as described in the following:
Co-teaching (Han al. 2018 simultaneously trains two models where each model acts teacher for the other by selecting its small-loss samples as clean data to train the other. It has been employed by Chadwick Newman (209) for training object detectors with noisy data. We adapt co-teaching into our dual-head network, where each detection head selects box samples with small classification loss train the other head. Note that the RPN is trained on all boxes.
SD-LocNet (Zang al. 2019) proposes adaptive sampling method that assigns reliable weight to each box sample. Higher weights are assigned to samples with higher classification scores and lower prediction variance consecutive training epochs.

---

Under review as a conference paper at ICLR 2021
N3 = 20%, N; 20%
N6 = 10%, N; = 10%
machine-gennraate
Method
mAP@... 5 MAP@ (5, .95
mAP@.5 MAP@ [5, .95)
mAP@.5 .5 mAP@ (.5, .95
Vanilla (Ren al. 2015
47.9
23.9
29.7
10.3
41.5
21.5
Co-teaching (Han ct al.
2018]
49.7
24.6
35.9
14.6
41.4
21.5
SD-LocNet Zhang
2019
49.3
24.5
35.1
13.9
42.8
21.9
NOTE-RCNN (Gao et al.
2019
50.4
25.1
38.5
15.2
43.1
22.0
Ours
53.5
27.7
47.4
21.2
46.5
23.2
Oracle (clean)
54.5
31.4
54.5
31.4
54.5
31.4
Table 3: Results on COCO val2017 with different levels of training noise.
Forward
Dual Heads
CA BBC
Dual Infer.
0%
20%
40%
(N6
Cor..
20%
40%
60%
20%
40%
60%
20%
40%
60%
(NV)
78.9
77.4
73.4
75.9
73.6
68.8
67.2
65.3
59.1
79.1
77.7
74.1
76.2
74.1
69.8
67.9
66.0
60.3
79.1
77.7
74.1
76.7
74.8
71.9
70.6
64.5
79.6
78.3
74.8
77.3
75.2
72.5
71.3
69.8
65.4
Table Ablation study to examine the effect of each component in the proposed framework. Numbers indicate mAP@ on VOC 2007 test set. The results validate the efficacy of the proposed CA-BBC and dual-head noise correction method. Ensemble of the two detection heads during inference can further boost performance.
NOTE-RCNN (Gao et al.) 2019) uses clean seed box annotations to train the bbox regression head. It also pretrains a teacher detector on the clean annotations for knowledge distillation. Because we do not have clean annotations, we follow previous works (Han et al. 2018 Arazo ct al.) 2019) and consider box samples with smaller classification loss as clean ones. We first train detector standard manner to mine clean samples. Then we utilize the clean samples following NOTE-RCNN (Gao et al. 2019).
Table shows the comparison results on VOC, where the training data contains different mixtures of label noise and bbox noise. Our method significantly outperforms all other methods across all noise settings. For high levels of noise (N₂ 40%, N2 (40%. 60%)). our method achieves 20% improvement in mAP compared to vanilla training, and > 100 improvement compared to the state-of-the-art NOTE-RCNN (Gao et al. 2019
On clean training data with 0% annotation noise, our method can still improve upon vanilla training by +1.9%, mostly due to the proposed soft labels. Compared to the one-hot GT labels, soft labels contain more information about an image region in cases where multiple objects co-exists in the same bounding box. Moreover, using soft labels has the effect of label smoothing, which could prevent overfitting and improve model's generalization performance (Müller et al., 2019).
Table 3 shows the results on COCO. Our method outperforms baselines by a large margin. Under 40% of label and bbox noise, vanilla training results in catastrophic degradation of 24.88% in mAP@.5 compared to training on clean data (oracle), whereas our method can reduce the perfor- mance drop 7.1%. The proposed method also achieves improvement under machine-generater noise, which validates its practical usage to train detectors by utilizing free unlabeled data.
4.4 ABLATION STUDY
In Table we add or drop different components in our framework to examine their effects. Below we explain the results in detail. More ablation study and qualitative results shown in the appendix.
In the first row, perform noise correction with only detection head, by using its output create soft labels and regress bounding boxes. Compared with the proposed dual-head network where knowledge is distilled from the ensemble, using single head suffers from confirmation bias where the model's prediction error would accumulate and thus degrade the performance.
In the second row, we remove CA-BBC from the proposed framework and only perform the dual-head noise correction (Section 3.3). Compared with the results using the proposed two-step noise correction (the third row), the performance decreases considerably for higher level (40%) of bounding box noise, which validates the importance of the proposed CA-BBC.
The third row shows the results using the proposed method.
8

---

Under review as a conference paper at ICLR 2021
In the last row, the ensemble of both detection heads during inference by averaging their outputs, which leads to further performance improvement.
CONCLUSION
To conclude, this paper addresses a new challenging research problem, which aims to train object detectors from noisy annotations that contain entangled label noise and bounding box noise. We propose a noise-resistant learning framework which jointly optimizes noisy annotations and model parameters. A two-step noise correction method is proposed, where the first step performs class- agnostic bbox correction to disentangle bbox noise and label noise, and the second step performs dual-head noise correction by self-distillation. Experiments on both synthetic noise and machine- generated noise validate the efficacy of the proposed framework. We believe that our work is one step forward towards alleviating human from the tedious annotation effort.
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In ICML, pp. 312-321, 2019.
Devansh Arpit, Stanislaw Jastrzkehski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In ICML, pp. 233-242, 2017.
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurlPS, 2019.
Avrim Blum and Tom M. Mitchell. Combining labeled and unlabeled data with co-training. In COLT, pp. 92-100, 1998.
Simon Chadwick and Paul Newman. Training object detectors with noisy data. In JEEE Intelligent Vehicles Symposium, pp. 1319-1325, 2019.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 990.007555. 2019a.
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, pp. 1062-1070, 2019b.
Ramazan Gokberk Cinbis, Jakob Verbeek, and Cordelia Schmid. Weakly supervised object local- ization with multi-fold multiple instance learning. TPAMI, (9(1):189-203 2017.
Thomas Deselaers, Bogdan Alexe, and Vittorio Ferrari. Localizing objects while learning their appearance. In ECCV, pp. 452-466 2010.
Thomas G. Dietterich, Richard H. Lathrop, and Tomás Lozano-Pérez Solving the multiple instance problem with axis aaiallarallel rectangles. Artif. Intell., 89(1-2)311-1, 1997.
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisser- man. The pascal visual object classes (VOC) challenge. IJCV, 88(2)3003-338, 2010.
Jiyang Gao, Jiang Wang, Shengyang Dai, Li-Jia Li, and Ram Nevatia. NOTE-RCNN: noise tolerant ensemble RCNN for semi-supervised object detection. In ICCV, 2019.
Ross B. Girshick. Fast R-CNN. In ICCV, pp. 1440-1448, 2015.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NIPS, pp. 529-536, 2005.

---

Under review as a conference paper at ICLR 2021
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pp. 2336-856,, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, pp- 2980-2988, 2017.
Judy Hoffman, Sergio Guadarrama, Eric Tzeng, Ronghang Hu, Jeff Donahue, Ross B. Girshick, Trevor Darrell, and Kate Saenko. LSDA: large scale detection through adaptation. In NIPS, pp. 3536-3544, 2014.
Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In ECCV, pp. 816-832, 2018a.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven currieulum for very deep neural networks corrupted labels. In ICML, pp. 2309-2318, 2018b.
Ksenia Konyushkova, Jasper R. R. Uijlings, Christoph H. Lampert, and Vittorio Ferrari. Learning intelligent dialogs for bounding box annotation. CVPR, pp. 9175-9184, 2018
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Sha- hab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv: 11///...00..... 2018
Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In CVPR, pp. 5447-5456, 2018
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In ICLR, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zimick. Microsoft COCO: common objects in context. In ECCV, pp- 740-755,15 2014.
Tsung- Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Be- longie. Feature pyramid networks for object detection. CVPR, pp. 936-944, 2017.
Rafael Müller, Simon Korbllihh, and Geoffrey E. Hinton. When does label smoothing help? In NeurlPS, pp. 4696-4705, 2019.
Dim P. Papadopoulos, Alasdair D. F. Clarke, Frank Keller, and Vittorio Ferrari. Training object class detectors from eye tracking data. In ECCV, pp. 361-376, 2014.
Dim P. Papadopoulos, Jasper R. Uijlings, Frank Keller, and Vittorio Ferrari. We don't need no bounding-boxes boxes: Training object class detectors using only human verification. In CVPR, pp- 854-863, 2016.
Dim P. Papadopoulos, Jasper R. R. Uijlings, Frank Keller, and Vittorio Ferrari. Training object class detectors with click supervision. In CVPR, pp. 180-189, 2017a
Dim P. Papadopoulos, Jasper R. R. Uijlings, Frank Keller, and Vittorio Ferrari. Extreme clicking for efficient object annotation. ICCV, pp.
2017b.
Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In /CLR, 2015.
Mengye Ren. Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for
robust deep learning. In ICML., pp. 4331-43400 2018.
10

---

Under review as a conference paper at ICLR 2021
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R- R-CNN: towards real-time object detection with region proposal networks. In NIPS, pp. 91-99, 2015.
Olga Russakovsky, Li-Jia Li, and Fei-Fei Li. Best of both worlds: Human-machine collaboration for object annotation. In CVPR, pp. 2121-2131, 2015.
Hao Su, Jía Deng, and Li Fei-Fei. Crowdsourcing annotations for visual object detection. In AAAI Human Computation Workshop, 2012.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame-
work for learning with noisy labels. In CVPR, pp. 5552-5560, 2018.
Yuxing Tang, Josiah Wang, Boyang Gao, Emmanuel Dellandréa, Robert J. Gaizauskas, and Liming Chen. Large scale semi-supervised object detection using visual and semantic knowledge transfer. In CVPR, pp. 2119-2128, 2016.
Jasper R. R. Uijlings, Stefan Popov, and Vittorio Ferrari. Revisiting knowledge transfer for training object class detectors. CVPR, pp. 1101-1110, 2018.
Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In NIPS. pp. 5601-5610 2017.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge J. Belongie. Learn- ing from noisy large-scale datasets with minimal supervision. In CVPR, pp. 6575-6583, 2017.
Kun Yi and Jianxin Wu. Probabilistic end-to and-to-end noise correction for learning with noisy labels. In CVPR, 2019.
Xingrui Yu, Han, Jiangchao Yao, Gang Niu, Ivor W. Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In ICML, pp. 7164-7173, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Xiaopeng Zhang, Yang Yang, and Jiashi Feng. Learning to localize objects with noisy labeled instances. In AAAI, pp. 9219-99199999 99229,,,22222 2019.


Reviews:
[{"review_id": "k5NUJ4s0cUo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros\n- The writing of the paper is good and clear. The problem paper it tries to solve is very practical and meaningful — annotation noise for training object detectors. \n- The proposed idea is interesting and the dual heads and two-step training strategy are critical for achieving the good performance. The method is described well with details and references. The paper also provide some intuitive explanation on why the method works.\n\nCons \n- The noisy annotation problem exists in practice and the paper’s motivation is also valid. However the experiments are carried on simulated noisy datasets. Although the results are promising, the practical value is limited. The way it simulates the noise may not reflect the most common case in practice. Since most object detection datasets are usually annotated in two steps where the first step identifies the class labels for instances in each image and the second step tries to draw bounding boxes for each present category. Even if the label noise happens in the first step, the second step is able to alleviate it by skipping drawing bounding box for that category. Thus the bounding box noise might be more common in real datasets and missing annotation is also another source of noise we should consider. \n- Different classes may suffer differently to the noise. Objects of different sizes also suffer differently. The paper lacks the analysis on class-specific performance and object size-specific performance on different noise settings. \n- In Table 2, when training on clean dataset, the proposed method still outperforms the vanilla training, which is somewhat surprising. It would be interesting to investigate the real reason behind this result by removing different components from the proposed method.\n- Since one-stage detectors also play an important role in real applications due to their efficiency. How will this method perform on one-stage detectors? \n- It would be interesting to see more qualitative results to compare the proposed method and the state-of-the-art methods.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dA4M1CUa8xP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work the authors propose a framework to perform object detection when there is noise present in class labels as well as bounding box annotations. The authors propose a two-step process, where in the first step the bounding boxes are corrected in class-agnostic way, and in the second step knowledge distillation has been used to correct the class labels. The propose method has been evaluated on two different datasets with synthetic noise.\n\nPros:\n\nThe idea of class-agnostic noise correction for bounding-box is interesting as it avoids the effect of noisy class label. \n\nThe idea of knowledge distillation has been extended for correcting bounding boxes which has some novelty.\n\nThe evaluation is comprehensive with ablations for all the components of the proposed framework.\n\nCons:\n\nThe class-agnostic update for bounding box is based on the assumption that, if a bounding box tightly covers an object, then two diverged classifiers would agree with each other and produce the same prediction. It is not clear why this assumption should hold true. Also the assumption that both classifiers would have low scores for the background class,i.e., high objectness scores seems difficult to understand as if the classifiers are trained on noisy labels, how can we be confidence in their predictions?\n\nThe knowledge distillation for noisy learning has been well studied and is not a significant novel contribution. The extension to bounding box correction is new but it does not include any novel ideas. \n\nIn general, this is an interesting work and solves a very interesting problem. But some of the assumptions used in the class-agnostic bounding-box correction are not easy to understand and also not well motivated. The results are motivating, but as such there is not enough novelty in the proposed solution which stands out.\n\n-- post rebuttal---\nAfter carefully reading the authors response, I still think the assumptions are not well motivated. For example, 'On a clean bounding box, even a less well-trained classifier can produce a lower-entropy (high-confidence) prediction with low background scores. Furthermore, it is less likely that two different classifiers both predict the same wrong class with high scores.'. Both of these assumptions seems very strong. They might hold sometimes, but they can not be generalized. Also, I still think the technical novelty is not enough in this submission. Therefore, I am not very positive about this submission.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "A_QYWYiX3Kw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the task of training object detectors from noisy labels. Unlike other methods that operate in the weakly- or semi-supervised regime, this method operates on the full set of bounding box annotations but assumes that all of them have been synthetically corrupted: \n- Each bounding box coordinate is drawn from a uniform distribution centered around the ground truth coordinate and with a range that is some pre-specified fraction of the bounding box height/width.\n- Each class label is either retained or flipped to some random class, with the decision to flip drawn from a Bernoulli distribution.\n\nTo nonetheless reliably train an object detector from this data, the following procedure is proposed:\n- preparation: A Faster R-CNN detector with a fully differentiable feature pooling layer (PrRoi) is extended with a second detection head. This detector is trained for a warm-up period on the corrupted labels in the usual fashion.\n- for each mini-batch:\n  - two-stage noise correction:\n    1. class-agnostic bounding box correction: Both detection heads classify the incoming bounding box proposal from the RPN. If both agree that it is background, the proposal is discarded. Otherwise, with the help of the differentiable pooling layer the bounding box coordinates are optimised to maximise agreement of softmax scores between both detection heads. The optimisation runs for a single step to save time.\n    2. label update and second bounding box update: given the corrected bbox from the last step, the class predictions of both heads are averaged and \"sharpened\" to reduce entropy, and the regressed bounding boxes are averaged together with the ground truth box\n  - updating the network parameters:\n    - Given the corrected labels and (twice-corrected) bounding boxes, the network parameters are updated in the usual manner.\n\nThis approach is evaluated against other weakly-supervised object detection approaches and performs quite favourably across multiple noise levels. As far as I can tell, the authors re-implemented the baselines for a fairer comparison, which is a plus. So overall, the results are quite good and the approach is sensible and seems to work. I don't have any major deal-breaking complaints, but a series of smaller ones mainly regarding gaps in the experiments:\n\nThe introduction motivates this approach by citing a number from a 2012 paper that is very much outdated and should be replaced. According to Su et al., the median time for a single bounding box annotation is 42 seconds. In the much more recent OpenImagesV4 paper (Kuznetsova et al., 2018), they find that they need 7s on average per bounding box with (among other things) a much more efficient box drawing procedure, which is well below the average of 88s from Su et al. 2012. This more recent number also tracks with my own annotation experience.\n\nGiven this emphasis on wanting to save annotation time, I would have expected the \"machine-generated\" annotations to figure much more prominently in the experiments (i.e. train a network on 10% of the ground truth and use detections on the rest of the training images). These only appear once in the sota comparison. It is still interesting to see that the proposed method handles high synthetic noise significantly better than competing methods, but this is the less realistic setting as the synthetic noise is applied to the entire set of ground truth annotations. I would thus also dispute the description of this setting as being \"more challenging and practical\" than other settings considered for weakly-supervised training (e.g. mostly relying on image-level labels).\n\nThe warm-up phase seems like a critical part of this approach and it's only mentioned in passing without any discussion. How much does the length of the warm-up phase matter? Did you experiment with this? Is there a trade-off between (a) getting some amount of training for the correction to work, vs. (b) over-training on the noisy labels? Is the recommended warm-up length dependent on the amount of noise?\n\nThe first noise correction stage (CA-BBC) is explicitly based on the assumption that if a bounding box tightly covers an object, the two classifiers will agree. This is a testable assumption and I would have been curious to see to what degree it actually holds.\n\nGiven that the proposed method focuses on correcting the bounding box annotations, I think it would have been important to report results on the corrected training boxes, especially since you have access to the un-corrupted annotations. Based on the final results, the method obviously must be doing something right but some distribution of IoU values before and after the correction/training process would have been interesting w/ some more qualitative examples of boxes that weren't successfully corrected.\n\nI may have overlooked this, but since training involves multiple epochs (and with it multiple corrections to the same bounding box) are the corrections retained or discarded? That is: Are individual boxes cumulatively corrected throughout the training?\n\nAlpha is the hyperparameter for the step size for CA-BBC and in section 4.1 you specify three different values for it {0, 100, 200}. Why three values?\n\nThe first result table is inconsistent with others, only reporting a subset of the results. What happens in the 0% label noise case?\n\nThe optimisation in CA-BBC is only run for one step for efficiency reasons. What about the effect on performance? Do additional steps bring any improvements? Currently, the noise correction only adds 25% computation time to training so I assume that running the optimisation for a few more steps won't make the approach prohibitively slow.\n\nCorrections/editing suggestions:\n- A correction needs to be made to the Chadwick & Neuman reference, as the paper was published at IEEE IV 2019.\n- p2, par 2: \"distil\" -> \"distill\"\n- 4.3 title: \"state-of-the-arts\" -> \"state-of-the-art\"\n- I would maybe re-organise the experimental section a little, as 4.2 focuses on the CA-BBC part on its own, the 4.3 compares the full method against the state-of-the-art, and 4.4 returns to a more complete ablation study. I would switch the 4.3 and 4.4, i.e. first get all the ablations out of the way then compare against the state-of-the-art.\n- Fig. 1 shows one part of the method in detail and Fig. 2 an overview of full method. I would either switch the order here, or merge these somehow.\n\nConclusion:\n-------------------\n\nOverall, I think this is a sensible aproach to training with noisy bounding box annotations, which compares favourably against competing methods. I have several reservations with regards to the experiments, especially the emphasis on synthetic noise as opposed to the more realistic setting where only a small set of hand-annotated labels are available. I would have liked to see more analysis (esp. quantitative) of the corrections of the training data, as this is what the method does. The warm-up phase appears to be critical and is also given short shrift. These are all things that are not fundamental problems, hence the (slightly) positive rating.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received borderline scores, before and after the rebuttal. Thus, support for paper acceptance isn't sufficiently strong. While the reviewers see merit, concerns which remain after the discussion phase, include how convincing the experimental settings and results are, and uncertainty about the motivation and practical value.
Decision: Reject