Learning a unified label space
Authors: Xingyi Zhou
Vladlen Koltun
Philipp Kraehenbuehl
Venue:  (2021)
Keywords: object detection
image recognition
computer vision

Abstract: How do we build a general and broad object detection system? We use all labels of all concepts ever annotated. These labels span many diverse datasets with potentially inconsistent semantic labels. In this paper, we show how to integrate these datasets and their semantic taxonomies in a completely automated fashion. Once integrated, we train an off-the-shelf object detector on the union of the datasets. This unified recognition system performs as well as dataset-specific models on each training domain, but generalizes much better to new unseen domains. Entries based on the presented methodology ranked first in the object detection and instance segmentation tracks of the ECCV 2020 Robust Vision Challenge.

Under review as a conference paper at ICLR 2021
LEARNING A UNIFIED LABEL SPACE
Anonymous authors Paper under double-blind review
ABSTRACT
How do we build a general and broad object detection system? We use all labels all concepts ever annotated. These labels span many diverse datasets with poten- tially inconsistent semantic labels. In this paper, we show how to integrate these datasets and their semantic taxonomies in completely automated fashion. Once integrated, we train an off-the-shelf object detector on the union of the datasets. This unified recognition system performs as well dataset-specific models each training domain, but generalizes much better to new unseen domains. En- tries based on the presented methodology ranked first in the object detection and instance segmentation tracks of the ECCV 2020 Robust Vision Challenge.
COCO
7S2
Opanimane
Mapilary
Mip
trashh hn C11365
COCO
Figure I: Different datasets span diverse semantic and visual domains. We learn to unify the label spaces of multiple datasets and train single object detector that generalizes across datasets.
INTRODUCTION
Computer vision aims to produce broad, general-purpose perception systems that work in the wild. Yet object detection is fragmented into datasets (Lin al., 20144, Neuhold et al.) [2017] Shao al.)
2019 Kuznetsova et al.) 2020) and our models are locked into specific domains. This fragmentation
brought rapid progress in object detection (Ren al., 2015) and instance segmentation (He et al.,
2017 but comes with a drawback. Single datasets are limited and do not yield general-purpose
recognition systems. Can we alleviate these limitations by unifying diverse detection datasets?
In this paper, we make training object detector the union of disparate datasets straight- forward as training on a single one. The core challenge lies in integrating different datasets into common taxonomy and label space. A traditional approach is to create this taxonomy by hand (Lam- bert et al. 2020 Zhao et al., 2020), which is both time-consuming and error error-prone. prone. We present a
fully automatic way unify the output space of multi-dataset detection system using visual data only. We use the fact that object detectors for similar concepts from different datasets fire on similar novel objects. This allows us to define the cost of merging concepts across datasets, and optimize for a common taxonomy fully automatically. Our optiiization jointly finds a unified taxonomy, mapping from this taxonomy each dataset, and detector over the unified taxonomy using a novel 0-1 integer programming formulation. An object detector trained on this unified taxonomy has large, automatically constructed vocabulary of concepts from all training datasets.
We evaluate our unified object detector at an unprecedented scale. We train a unified detector on large and diverse datasets: COCO (Lin et al. 2014), Objects365 (Shao et al., 2019). OpenIm- ages (Kuznetsova et al. (020)), and Mapillary (Neuhold et al.) 2017). Experiments show that our learned taxonomy outperforms the best expert-annotated label spaces, as well as language-based alternatives. For the first time, we show that single detector performs as well as dataset-specific models on each individual dataset. Crucially, we show that models trained on the diverse training sets generalize zero-shot to domains, and outperform single-dataset models. Our models ranked first in the object detection and instance segmentation tracks of the ECCV 2020 Robust Vision Chal- lenge across all evaluation datasets. Code and models will be released upon acceptance.

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
Training on multiple datasets. In recent years, training on multiple diverse datasets has emerged as an effective tool to improve model robustness for depth estimation (Ranftl et al., 2020) and stereo matching (Yang et al. 2019). In these domains unifying the output space involves modeling different camera models and depth ambiguities. In contrast, for recognition, the unification involves merging different semantic concepts. MSeg (Lambert et al.) 2020) manually created unified label taxonomy of 7 semantic segmentation datasets and used Amazon Mechanical Turk to resolve the inconsistent annotations between datasets. Different from MSeg, our solution does not require any manual effort and unifies the label space directly from visual data in a fully automatic way.
Wang et al. (2019) train universal object detector on multiple datasets, and gain robustness by joining diverse sources of supervision. However, they produce a dataset-specific prediction for each input image. When evaluated in-domain, they require knowledge of the test domain. When evaluated out- -ft-of-domain, they produce multiple ouiputs for a single concept. This limits the generalization ability of detection, as we show in experiments (Section. (522). Our approach, on the other hand, merges visual concepts at training time and yields a single consistent model that does not require knowledge of the test domain and can be deployed cleanly in new domains. Both WWang et al.] de
(019) and MSeg (Lambert et al. 2020) observe performance drop in single unified model. With our unified label space and a dedicated training framework, this is not the case: the unified model performs as well as single-dataset models on the training datasets.
Zhao et al. (2020) trains universal detector on multiple datasets: COCO (Lin et al. 2014), Pascal VOC (Everingham et al. 2010), and SUN- (Song et al. (015)), with under 100 classes in total. They manually merge the taxonomics and then train with cross-datase pseudo-labels generated by dataset-specific models. The pseudo-label idea is complementary to work. Our unified label space learning removes the manual labor, and works on a much larger scale: we unify COCO, Objects365, and OpenImages, with more complex label spaces and 900+ classes.
YOLO9000 (Redmon & Farhadi 2017) combines detection and classification datasets to expand the detection vocabulary. LVTS (Gupta et al., 2019) extents COCO annotations to 1000 classes in federated way. Our approach of fusing multiple readily annotated datasets is complementary and can be operationalized with no manual effort to unify disparate object detection datasets.
Zero-shot classification and detection reason about novel object categories outside the training set (Fu et al. 2018, Bansal et al., 2018). This is often realized by representing a novel class by a semantic embedding (Norouzi et al.) 2014) or auxiliary attribute annotations (Farbadi et al. 2009) In zero-shot detection, Bansal et al. (2018) proposed a statically assigned background model to avoid novel classes being detected as background. Raahman et al. (2019) included the novel class word embedding test-time training to progressively generate novel class labels. Li et al. (2019) leveraged external text descriptions for novel objects. Our program is complementary:
aim to
build a sufficiently large label space by merging diverse detection datasets during training, such that the trained detector transfers well across domains even without machinery such word embeddings or attributes. Such machinery can be added, if desired, to further expand the model's vocabulary.
3 PRELIMINARIES
An object detector jointly predicts the locations by E R2 and classwise detection scores dę RI4 of all objects in a scene. The detection score describes the confidence that a bounding box belongs to an object with label E L, where L is the of all classes. Figure 2a provides an overview. On single dataset, the detector is trained to produce high scores only for the ground-truth class.
Consider multiple datasets, each with its own label space L' i?..... A detector now needs to learn a common label space for all datasets, and define a mapping between this common label space and dataset-specific labels L + L'. In this work, we only consider direct mappings. Each common label maps to at most one dataset-specific label per dataset, and each dataset-specific label maps to exactly one common label. In particular, we do not hierarchically relate concepts across datasets. When there are different label granularities between datasets, space, and expect to predict all of them. Mathematically, the mapping from the joint output space to dataset- one is Boolean linear transformation of the output of the recognition system
keep them all in our label
2

---

Under review as a conference paper at ICLR 2021
detector
Inpul
detector
bourding boteee
object
detecion
soom
domaln
Indaal
detanet
(a) Object detection pipeline
(h) Our training framework
Figure 2: A standard object detection pipeline (a) fits bounding boxes to objects and predicts detec- tion scores over fixed set of output classes. In single-dataset training, class scores are supervised directly from annotations in the dataset. In multi-dataset training (b), detector learns its own output space and links it to the label space of each dataset.
di = d'd., with di E IR L.1, T E {0, 1114 and constraints T 1, T < 1. The two constraints ensure that only direct mappings are learned. For simplicity, let T = [TLT NT] be the mapping all dataset-specific output spaces. Figure 2b provides overview.
Prior work defined L and T by hand (Lambert et al. 2020 Zhao et al. 2020) or used a trivial mapping T = / with completely disjoint outputs (Wang al. 2019). They then trained detector given the fixed label space and mapping. In the next section, we show how to jointly learn the label space, the mapping to the individual datasets, and the detection scores in globally optimal manner.
4
METHOD
We start with training detector on the trivial disjoint label space U f. In this section, we show how to automatically learn a unified label space by converting the disjoint label space into a unified label space. Once the unified label space is learned, retrain the detector end-to-end with the unified label space. An overview of our workflow can be found in Appendix
4.1 LEARNING A UNIFIED LABEL SPACE
We first consider only fine-tuning the last linear layer of the disjoint-label space detectpr. Specifi- cally, let f1, f2.... be the D-dimensional features fi E R of the penultimate layer of the pretrained model for object locations 61, 62................. for all objects
dataset. Our goal is to learn a new detection score de = W fa with parameters W [W1, W2,.... .... ]iاl and wi E RD, label space L, and dataset-specific transformations T. The pretrained detector allows us to formulate this objective over fixed set of precomputed detections and their features F [f1, fe....:
minimizer,
Σ 4(Tww) F) NLL JEL
subject to
T1-1 and T11
V.e(L...N)
Here & general loss function that factorizes over the labels L,, and N the number of datasets. The weight wy controls the output of the detector for the joint label 1, and Ti is column of the dataset-specific transformation that maps each joint label to all training datasets. The cardinality penalty A|L encourages small and compact label set. A factorization of the loss & over the output space E L may seem restrictive. However, does include the most common loss functions in detection: sigmoid cross-entropy and mean average precision. Section 4.2 discusses the exact loss functions used in our optimization.
For fixed label I. and mapping T. objective reduces standard training objective of detec- tor. However, the joint optimization I. and T significantly complicates the optimization. It mixes combinatorial optimization
I. with continuous optimization of W, and 0-1 integer program over T. However, there is a simple reparametrization that lends itself to efficient optimization.
First, observe that the label set L simply corresponds to the number of columns in T. Furthermore, we merge at most one label per dataset 7 1 < 1. Hence, for each dataset a column T E T takes one of [ + values: T (0, 41, 4,,- ..., where 1 E {0, 1) is an indicator vector. Each column T, E T then only chooses from small set of potential values = T x T2 x x..., where represents the Cartesian product. Instead of optimizing over the label set I. and transformation T
3

---

Under review as a conference paper at ICLR 2021
directly, instead use combinatorial optimization over the potential column values of t T. Let It E {0, 1] be the indicator of combination t E T. In this combinatorial formulation, the constraint T'1 = 1 translates to EteTro;=1 It = 1 for all datasct-specific labels (. Furthermore, the objective of the optimization simplifies to
Σ
(((Tw) XLl- = E xxli(bui P)
F)+ + Σ Zt.
Crucially, the weights we of the detection score are now independent of the combinatorial optimiza- tion, and can be precomputed for each column value E T in a merge cost:
C min la (tw, F). M2
(3)
This leads to compact integer linear programming formulation of objective
minimize,
Σ
Et (ct + A)
tET
subject to
Σ BET14==1
x
1/
For two datasets, the above objective is equivalent to a weighted bipartite matching. For a higher number of datasets, it reduces weighted graph matching and is NP-hard, but is practically solvable with integer linear programming (Linderoth & Ralphs] 2005).
One of the most appealing properties of the above formulation is that it separates the integer opti- mization over from the continuous optimization of the last linear layer wy. Section 4.2 explores this separation and shows how to precompute the merge cost ct for various loss functions.
One major drawback of the combinatorial reformulation is that the set of potential combinations T However, most
grows exponentially in the number of datasets used: |T] 0( merges E are arbitrary combinations of labels and incur a large merge cost ce. Section 4.3
presents linear-time greedy enumeration algorithm for low-cost merges.
Considering only low-cost matches, standard integer linear programming solvers find an optimal solution within second for all label spaces we tried, even for [L] > 600 and up to 6 datasets.
4.2 LOSS FUNCTIONS
The loss function in our constrained objective is quite general and captures wide range of com- monly used losses. We highlight two: an unsupervised objective based on the distortion of the output compared to the pretrained model, and mean Average Precision (mAP).
Distortion. A natural objective is learn a joint label space that stays close the pretrained model. Let D = W F' be the detection scores of the pretrained model with weights W = [[], ......] for each dataset-specific label. Distortion then measures the difference between the joint and pretrained models on all dataset-specific outputs:
edian (tw, F) Σ li [ F.FF-wih ue (
(5)
This distortion has a closed-form solution W/ En4n E, The merge cost e corresponds to the variance in detector outputs between dataset-specific labels and can be computed efficiently from pairwise differences. The main drawback of this distortion measure is that does not take task performance into consideration when optimizing the joint label space. Next, we show how to learn a label space using annotations for dataset-specific predictions on the original datasets.
Mean Average Precision. Let mAP; be the mean Average Precision for each dataset-specific class i on the corresponding dateset-specific validation set. Let mAP, be the mAP of merged output over all dataset-specific outputs used in t. Our loss is then the difference in mAP:
AM (tw) F) Σ (tmAP; mAP:).
(6)
4

---

Under review as a conference paper at ICLR 2021
It is hard to optimize we for mAP directly, since it operates sorted set of detection scores. We instead use the averaging solution of the distortion metric (5). and simply evaluate сt- The mAP computation is computationally quite expensive. but we will provide an optimized joint evaluation with our code.
4.3 COMPUTATION AND PRUNING
The size of our optimization problem scales linearly in the number of potential merges [T], which grow exponentially in the number of datasets. To counteract this exponential growth, we only consider sets of classes
T'
C t
For an aggressive enough threshold т, the number of potential merges [T'] remains manageable. We greedily grow T by first enumerating all feasible two-class merges (|t) 2), then three-class merges, and so on. We use À = 0.1 and T = 0.2 in our experiments. The runtime of this greedy algorithm is O([T'] max; [E) In practice, the cost computation took few seconds for the distor- tion loss function and about 10 minutes for the mAP loss (due to the need repeatedly recompute mAP). The integer programming solver finds the optimal solution within one second in both cases.
5 EXPERIMENTS
Our goal aims to facilitate the training of models that perform well across datasets. Our main training datasets are adopted from the ECCV 2020 Robust Vision Challenge (RVC). These are four large object detection datasets: COCO (Lin et al. 2014), Openlmages (Kuznetsova et al. 2020), Objects365 (Shao et al.) 2019), and Mapillary (Neuhold al., 2017). To evaluate zero-shot cross- dataset generalization, we use the RVC instance segmentation datasets (all of which have bounding box annotations): VIPER (Richter et al. (0117). CityScapes (Cordts et al. (016)) ScanNet (Dai et al.) 2017). WildDash (Zendel et al. 2018), and KITTI (Geiger et al. 2012)). 2). In addition. we test zero-shot on two object detection datasets: Pascal VOC (Everingham et al. 2010) and CrowdHuman (Shao
ct al. 2018). The datasets are described in more detail in Appendix
In our implementation, we first train a unified detector on the large and general datasets: COCO, Objects365, and Openlmages. As Mapillary is small and domain-specifer we add it in subsequent fine-tuning stage. See Appendix [B] for details.
Evaluation metric. We evaluate a detector both its training dataset(s) and new datasets that may contain objects out of the training label space. On the training datasets, we use the official metrics of each datasets: for COCO, Objects365, and Mapillary, we use the mAP at IoU thresholds 0.5 to 0.95. For OpenImages, we use the official modified mAP@0.5 that excludes unlabeled classes and enforces hierarchical labels (Kuznetsova al. 2020).
For evaluating on new test datasets, standard mAP evaluation requires an expert-annotated test-to- train class correspondence. This is unavailable our setting. We initially invited volunteer anno- tators to link the classes between each test dataset and the unified label space. However, we observe considerable disagreement between annotators. For example, "rider" in CityScapes linked to 'bi- cyclist" or 'motorcyclist" by different human annotators. For reproducible and scalable evaluation, we instead define new metric, mean Expected AP (mEAP). We use word embedding (Penning- ton al., 2014) to find set of correspondences between each test class and the unified label space. Each test class can have multiple correspondences. We calculate the expected AP (EAP) of a test class as the average AP of all corresponding unified classes. The summary metric per test dataset over all classes is mean EAP (mEAP). most cases, a test class corresponds to only one joint label, and the EAP and AP are equivalent. Appendix C specifies the evaluation protocol in more detail.
5.1 IMPLEMENTATION
We use the CascadeRCNN detector (Cai & Vasconcelos 2019). A single region proposal network (RPN) is shared by all datasets. Each proposed bounding box from RPN is classified by a cascaded classifier. For disjoint label-space baseline, the last classification layer of each cascade stage is split between datasets. We only apply a training loss to the source classifier, following Wang et al. 2019). Our unified detector uses CaScadeRCNN as is.
5

---

Under review as a conference paper at ICLR 2021
VOC
VIPER
CityScapes ScanNet
WildDash CrowdH. KITTI
mean
COCO
80.0
13.9
39.6
17.4
25.9
73.9
30.5
40.2
Objects365
71.9
20.7
43.4
24.9
27.6
71.8
32.2
41.8
OpenImages
64.4
10.4
29.8
24.2
20.3
66.7
21.8
33.9
Mapillary
11.4
15.2
44.7
0.0
23.4
49.3
37.8
26.0
Disjoint label-space
80.5
19.5
44.7
32.4
30.0
65.3
35.9
44.0
Unified (ours)
82.5
20.9
49.1
30.8
31.6
70.7
37.9
46.2
Unified (expert human)
82.5
20.9
50.3
31.4
31.9
71.3
37.1
46.5
Dataset-specific
80.3
31.8
54.6
44.7
80.0
Table 1: Zero-shot cross- cross-dataset object detection performance on the validation sets of datasets that were not seen during training. The metric is mEAP@0.. (see the text for definition). We compare models trained on each single training dataset (top 4 rows), a unified detector with dataset-specific output classifier (5th row), a unified detector with our learned unified label space (6th row), and unified detector with the human expert label space (7th row). For reference, we show an "oracle" model that is trained on the training set of each test dataset on the bottom row, The columns refer to test datasets.
Training loss. The standard CascadeRCNN uses softmax cross-entropy as the classification loss (Ren et al. 2015). This is infeasible in our case. Softmax assumes one object has at most class label, while OpenImages (Kuznetsova al. 2020) requires predicting label hierarchy (e.g., requires predicting "vehicle" and "car" for all cars) and may have overlapping labels for the same object (e.g., both "toy" and "car" for a toy car). In our implementation, we use the sigmoid cross-entropy loss for all multi-dataset models and baselines, but use the hest dataset-specific loss when training baseline on just dataset. For OpenImages, cross-entropy loss that sets parent classes as positives and ignores the losses over descendant classes. When training on multiple datasets, also ignore labels outside the training dataset for each image. Appendix reports ablation experiments on the hierarchy- aware and multi-dataset losses.
hierarchy- awaare sigmoid
Data sampling. There significant spread in the sizes of the training datasets (see Appendix AA. In our experiments, we found that sampling images evenly from all training datasets works best. Fol- lowing the current best practice (Peng et al.) 2020), we use class-aware sampling (Shen et al. 2016) for OpenImages and Objects365 to tackle the long-tail class distribution. A controlled evaluation of the different sampling strategies can be found in Appendix
Training details. Our implementation is based on detectron2 (Wu et al. (019)) and we adopt most of the default hyperparameters for training. We use the standard object detection data augmentation, including random flip and random scaling of the short edge in the range [640, 800). We use SGD with a base learning rate 0.01 and batch size 16 over GPUs. We use ResNet50 (He ct al. 2016) as the backbone in our controlled experiments unless specified otherwise.
For our label space evaluation experiments, we use a standard 2x schedule (180k iterations with learning rate dropped at the 120k and 160k iterations) (Wu et al., 2019) to save computation. This results in 8 epochs on COCO, 0.5 epochs on OpenImages, and 1.6 epochs on Objects365. When comparing models trained on different datasets (Section 5.2 and Section 5.4)) we use an 8x schedule (720k iterations with learning rate dropped at the last 60k and 20k iterations) (Wu et al. [209 Goyal al. 2017) 7) or until the model converges (for small datasets). A comparison of training schedules provided in Appendix
5.2 ZERO-SHIOT CROSS-DATASET EVALUATION
For our main evaluation, we train a detector on all four RVC training datasets and evaluate it on new datasets that were not seen during training. We train strong disjoint label-space detector using a ResNeSt101 (Zhang et al. 2020) backbone with the same training procedure as described in Section 5.1 We obtain the predicted bounding boxes in the validation sets to run our label space optimization algorithm. We compare our unified detector to the disjoint label-space baseline fine- tuned with the same schedule, hyperparameters, and detection models. For reference, we also show the performance of detectors trained on the training set of each test dataset. This serves as an oracle "upper bound" that has seen the test domain and label space. Note that KITTI (RVC version) and

---

Under review as a conference paper at ICLR 2021
Dijwnoos Bocer
Open Ieeiu
OipecidEs Areican Roohal
DOOD
Loamndd
footbal
Aemorian Tootbol
COW
Coari ragen wan
CANCERRE ovan
smiil muzuuM
Dreet sith atorelop
pane
Ove
Figure 3: Example differences between an expert designed label space provided as part of the ECCV 2020 Robust Vision Challenge (top each row, blue) and our learned label space (bottom of each row, pink). Best viewed the screen.
WildDash small and do not have validation set. We thus directly evaluate the training set and do not provide the oracle model.
Table 1 shows the results under the mEAP@0.5 (loU threshold 0.5) metric. The COCO model al- ready exhibits reasonable performance of some test datasets, such as Pascal VOC and CrowdHuman. However, its performance is less than satisfactory differs significantly from COCO. Training on the more diverse Objects365 dataset yields higher ac- curacy in the indoor domain, but loses ground on VOC and CrowdHuman, which are more similar to COCO. Training on all datasets, either with disjoint label space for each dataset or with a unified label space yields generally good performance on all datasets. The individual test datasets are closer to the span of all training datasets than to any individual dataset. Note that on Pascal VOC, our unified model outperforms the VOC oracle model without seeing VOC training images.
datasets such ScanNet, whose label space
Our unified model consistently produces high-quality detections for all classes, while the quality of the outputs of the disjoint baseline depends on the respective dataset-specifie label space. For example, Mapillary person detector generalizes much more poorly than COCO one. In addition, a disjoint baseline yields redundant detections per object.
5.3 EVALUATION OF THE UNIFIED LABEL SPACE
Next, we analyze our unified label space on COCO, Objects365, and OpenImages. We compare to a human expert label space officially provided part of the ECCV Robust Vision Challenge, derived by the challenge organizers" Our optimization gives 701 classes in the unified label space, which more than the 659 classes in the human expert label space. This is because some semantically similar concepts between datasets have different visual expressions.
Figure 3 shows some examples of differences between our learned unified label space and a human expert label space. The learning algorithm can separate visually different categories with similar words ("American football" and "football"), and merge same concept expressed in different words ("Cow" and "Cattle"). Interestingly, the learned label space splits COCO, Objects365 and OpenImages oven, even though they share exactly the same word. However, they are visually dis- similar: COCO ovens include the cooktop. OpenImages only the control panel, and Objects365 oven focuses just on the front door. This signal is only present the visual data.
To quantitatively evaluate the label spaces in practice, train unified detector each label space using the same training procedure (ResNet-50 and the 2x schedule), and compare their detection performance. Table 2 shows the results. We additionally compare language-based baseline (Glove embedding). Specifically, we replace the cost measurement defined in Section 4.2 with the cosine distance between the Glove word embeddings (Pennington et al. 2014). and run the same integer linear program. The three label spaces obtained are similar in most classes, hence the overall mAP does not change much. However, our label space consistently outperforms the human expert,
ht httpee///stthub.comm//eeeee ot

---

Under review as a conference paper at ICLR 2021
COCO
Objects365 OpenImages
mean
Glove embedding
41.6
20.3
62.3
41.4
Learned, distortion
41.5
20.7
62.6
41.6
Learned, mAP (ours)
42.0
20.9
62.8
41.9
Expert human
41.5
20.6
62.6
41.6
Table 2: Evaluation of unified label spaces. We measure mAP on the validation sets of the training
datasets. We compare to language-based baseline and manual unification by human expert.
COCO
Objects365
OpenImages
Mapillary
mean
Unified
44.9
23.9
65.7
14.8
37.3
Disjoint label-space oracle Dataset-specific oracle
45.1
24.0
65.1
14.9
37.3
42.5
24.9
65.7
15.5
37.2
Detection mAP on the validation sets of the training datasets. We show the performance of our unified model, the disjoint label-space detector with the oracle head at test time, and dataset- specific models (the last row, where each column from a different model).
Table 3:
with a healthy 0.3 mAP margin in average. The relative improvement of our model over the expert larger than the experts' improvement over the language-basee baseline.
5.4 PERFORMANCE ON TRAINING DATASETS
compee]so the unified detector, the disjoint label-space detector, and dataset-specific detectors on the four training domains. Our unified detector is competitive with the disjoint label-space detec- tor without knowing the image domain at test time. On COCO, our unified detector outperforms the COCO-specific detector by 2.4 mAP, likely because it benefits from 20x more data provided by the other training datasets. On the other three datasets, the unified model matches the dataset-speciff models within mAP. Our work shows that training on multiple datasets can increase the model's generality across domains without compromising accuracy within domains.
5.5 ECCV ROBUST VISION CHALLENGE
We submitted a model trained with the presented approach the ECCV 2020 Robust Vision Chal- lenge (RVC). We used heavy ResNeSt200 backbone (Zhang et al. 2020) and followed the same training procedure as in (o.ttio[5.4ec to train the model using an 8x schedule. We used a unified label space of 682 classes learned with the distortion loss. The training took ~6 days on a server with 8 Quadro RTX 6000 GPUs Table summarizes the results of the challenge. Our model outperforms all other RVC entries all datasets by a large margin. Notably, WiseDet_RVC used a stronger detector QQao et al. 2020). but without our learned label space or multi-dataset training setup (Sec- tion [.1]. The bottom rows of Table 4 show the fate-of-the-ar results reported on each individual dataset. On COCO, our result is comparable with DetectoRS (Qiao et al.] (020). which is by default 2.4 mAP higher than our ResNeSt200 backbone (50.9 mAP) (Zhang al. 2020)) On OpenImages, our result matches the best single-model performance of the Openlmages 2019 Challenge winner,
TSD (Song et al. 2020). with a comparable backbone (5ENet154 (Hu et 2018 with deformable
COCO OpenImages
Mapillary
Objects365
Ours
52.9
60.6 / 56.8
25.3
33.7
WiseDet_RVC R50 GN RVC
40.0
56.1 / 53.3
22.5
34.0
21.4/ 19.9
8.1
DetectoRS (Qia et al. 2020)
53.3
TSD (Song et al 2020
60.5
CACascade RCNN Gao et al. 2019)
31.6
Table 4:
Test set performance on RVC datasets: COCO test-challenge set, OpenImages challenge 2019 test sets (shown in public test set/ private test set), Mapillary test set, and Objects365 validation set. Top: results of RVC challenge participants. Bottom: the published sfate-of-the -iie----tthhrrrrr performance on each specific dataset (without model ensembles test-time augmentation). Objects365 was initially part of the challenge but was removed in the final evaluation.
8

---

Under review as a conference paper at ICLR 2021
COCO CityScapes Mapillary
VIPER
ScanNet
OpenImages
KITTI WildDash
COCO
35.6
19.6
3,2
8.5
5.2
7.2
15.7
8.4
CityScapes
0.0
21.5
0.8
2.3
0,0
0.0
13.0
2.4
Mapillary
0.6
11.7
10.6
9.0
1.2
0.0
13.4
5.4
VIPER
0.1
2.8
1.1
17.8
0.0
0.0
6.5
1.4
ScanNet
0.4
0.0
0.0
0.0
35.6
0.0
0.0
0,0
OpenImages 12.9
9.5
1.1
3.5
1.7
52.8
7.2
4.9
Unified (ours)
24.0
28.3
8.1
16.5
28.7
41.8
16.9
11.3
Table 5: Instance segmentation performance on six training datasets and two new datasets (KITTI and WildDash). We show mask mAP when the test dataset is included training, and show mask mEAP when the testing new datasets.
COCO CityS. Mapillary VIPER ScanNet Openlmages KITTI WildDash
Ours
33.0
29.8
13.0
18.9
20.5
35.0
23.2
21.0
seamseg-rvest -rveubbet
22.1
20.9
MifPS_blbs4.om
21.3
Table 6: Leaderboard of RVC instance segmentation challenge. We show results the test set for each datasets (test-challenge for COCO and private test set for Openlmages).
convolution Zhu et al. 2019 On Objects365, we outperform the 2019 challenge winner by 2 mAP.
INSTANCE SEGMENTATION
We further evaluate our label space learning algorithm and unified training framework on instance segmentation. We follow the ECCV Robust vision challenge set to use datasets: COCO, Openlmages, Mapillary, ScanNet, VIPER, CityScapes, WildDash and KITTI (the same as Table 7 except OpenImages segmentation set has 300 instead of 500 classes.). Again, we leave WildDash and KITTI as testing only as they are small and similar to CityScapes and Mapillary. We run our label space learning algorithm (Section. 4.1 the remaining six datasets, resulting unified label space of 358 classes. We use CascadeRCNN (Cai & Vasconcelos 2019) with standard mask head the detector, and train 2 schedule with ResNet50. The fataset-specingg models trained with 1x or 2x schedule depending on their size.
Table. 5 compares the unified detector (with instance segmentation) dataset specific models. As expected, no single dataset-speccfic model performs well on all test domains. Our unified model performs consistently good on all training datasets. More importantly, it generalizes the best to the new test datasets (KITTI and WildDash) than any single dataset model. Table. 6 compares our method with others on the test sets of RVC instance segmentation challenge. We outperform other entries on all datasets that have a valid submission.
7 CONCLUSIONS
We presented an automated way to unify the label spaces of multiple datasets. This enables training single detection model that works across the training domains and beyond. We showed that the resulting detector is robust in zero-shot cross-dataset testing. Our current implementation unifies matching concepts, but does not yet merge hierarchical concepts. Neither do adapt our model to unseen data using unsupervised objectives. These are exciting avenues for future work.
REFERENCES
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In ECCV, 2018.

---

Under review as a conference paper at ICLR 2021
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cmn: High quality object detection and instance segmentation. TPAMI, 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.
The PASCAL visual object classes (VOC) challenge. IJCV, 2010.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In CVPR, 2009.
Yanwei Fu, Tao Xiang, Yu-Gang Jiang, Xiangyang Xue, Leonid Sigal, and Shaogang Gong. Recent advances in zero-shot recognition: Toward data-efficient understanding of visual content. IEEE Signal Processing Magazine, 2018.
Yuan Gao, Hui Shen, Donghong Zhong, Jian Wang, Zeyu Liu, Ti Bai, Xiang Long, and Shilei Wen. A solution for densely annotated large scale object detection task. 2019.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv: 02677, 2017.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance seg- mentation. In CVPR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In CVPR, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciaation networks. In CVPR, 2018.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. IJCV, 2020.
John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. MSeg: A composite dataset for multi-domain semantic segmentation. In CVPR, 2020.
Zhihui Li, Lina Yao, Xiaoqin Zhang, Xianzhi Wang, Salil Kanhere, and Huaxiang Zhang. Zero-shot object detection with textual descriptions. In AAAI, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Jeffrey T Linderoth and Ted K Ralphs. Noncommercial software for mixed-integer linear program- ming. Integer programming: theory and practice, 3(253-303): 144-189,,, 2005. 2005.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Buld, and Peter Kontschieder: The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. In ICLR, 2014.
10

---

Under review as a conference paper at ICLR 2021
Junran Peng, Xingyuan Bu, Ming Sun, Zhaoxiang Zhang, Tieniu Tan, and Junjie Yan. Large-scale object detection in the wild from imbalanced multi-labels. In CVPR, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Siyuan Qiao, Liang-Chite Chieh Chen, and Alan Yuille. Detectors: Detecting objects with recursive
feature pyramid and switchable atrous convolution. arXiv:2006.023344 2020.
Shafin Rahman, Salman Khan, and Nick Barnes. Transductive learning for zero-shot object detec- tion. In JCCV, 2019.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI, 2020.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In CVPR, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Stephan R Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In ICCV, 2017.
Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun. Crowdhu- man: A benchmark for detecting human in a crowd. arXiv: 1805.00/23, 2018.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: large-scate scale, high-quality dataset for object detection. ICCV, 2019.
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In ECCV, 2016.
Guanglu Song, Yu Liu, and Xiaogang Wang. Revisiting the sibling head in object detector. In CVPR, 2020.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, 2015.
Xudong Wang, Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detec- tion by domain attention. In CVPR, 2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
Gengshan Yang. Joshua Manela, Michael Happold. and Deva Ramanan. Hierarchical deep stereo
matching on high-resolution images. In CVPR, 2019.
Oliver Zendel, Katrin Honauer, Markus Murschitz, Daniel Steininger, and Gustavo Fernan- dez Dominguez. Wilddash--reating hazard-aware benchmarks. In ECCV, 2018.
Hang Zhang, Chongruo Wu, Zhongyue Zhang. Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Mueller, R Manmatha, et al. Resnest: Split-attention networks. arXiv: 2004.08955, 2020.
Xiangyun Zhao, Samuel Schulter, Gaurav Sharma, Yi-Hsuan Tsai, Manmohan Chandraker, and Ying Wu. Object detection with unified label space from multiple datasets. In ECCV, 2020.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In CVPR, 2019.
11

---

Under review as a conference paper at ICLR 2021
Dataset name
Domain
Categories
# Images
Note
Train & Validation COCO
Internet images
80
118k
Objects365 OpenImages
Internet images
365
600k
Long-tail
Internet images
500
1.8M
Federated/ Hierarchical label
Mapillary
Traffic
38
18k
High-resolution
Test
ScanNet
Indoor
20
25k
VIPER
Virtual
10
13k
CityScapes
Traffic
8
12k
High-resolution
WildDash
Traffic
13
4k
Extreme driving scenes
KITTI
Traffic
8
200
Pascal VOC
Internet images
20
16k
CrowdHuman
Internet images
1
15k
Crowded
Table 7: Summarize of datasets we used. Top: datasets we used in training, which are from ECCV 2020 RVC challenge; Bottom: datasets used for zero-shot cross dataset testing. We list the features of each dataset in the last column.
A DATASET DETAILS
Table lists the datasets we used in our object detection experiments. For ScanNet (Dai et al. 2017 as there is no standard train/ validation split, we use the first 80% scenes (sorted by scene ID) as training and the last 20% scene as validation. For KITTI (Geiger et al. 2012), we used the RVC aattance-segmentttin version, which contains 200 images. For WildDash (Zen-
challenge that del et al. 2018), use the public version for evaluation, and report standard mAP performance. We don't consider the negative label metric the official website. For CrowdHuman (Shao et al. 2018). use the visible bounding box annotation, and report the standard mAP instead of the missing rate as the official metric.
B LABEL SPACE EXPANSION ALGORITHM
While we tend to keep the training domains and label space large and comprehensive, is inevitable in practice that more fine-grained labels or specific testing domains are needed. Given a learned unified label space on an existing set of training datasets, we propose label space expansion algorithm to allow adding more datasets and labels after the unified detector is trained.
Similar to our unified label space learning algorithm, we run the unified detector on the new training data. We evaluate the mAp between each class in the new dataset annotation and each class in the unified label space. We merge the new class into the existing class that gives the lowest merge cost (Section. 4.2), if the cost is lower than threshold (mAP change mAP in implementation). Otherwise, we append the new class to the unified label space as a single class.
C MEAP DETAILS
For each test label, we first find training labels with closet Glove embedding distance. The same word has the Glove distance 0. For joint class in unified label space, it may has different names from different datasets. We set the embedding distance as the minimal embedding distance among its composing classes. In the case that there is training label close test label (i.e., minimal Glove distance > = 0.6) due to limited label space, we collect all labels that contain or being contained the test class the corresponding labels.
There can be more than one training label corresponds to a test label, when the class the same name from two different training datasets not merged there is finer label granularity in the training label space. In this case, a human user can pick any training class within the corresponding set for the test class during real-world application. To mimic this in evaluation, define new
12

---

Under review as a conference paper at ICLR 2021
COCO
Objects365
Openlmages
meau
sigmoid cross entropy loss
41.5
20.1
58.4
40.0
+multi-dataset loss +hierarbical-awee -aware loss (ours)
42.0
20.8
61.4
41.4
42.0
20.9
62.8
41.9
Table 8: Ablation experiments training losses. We start with a sigmoid cross entropy loss, and add our multi-dataset loss and hierarchical- loss (for Openlmages only) one by one. Experiments are conducted on ResNet-50 with 2x schedule.
COCO
Objects365
OpenImages
mean
W.O. class-aware sampling
41.4
16.8
49.4
35.9
W.o. evenly sampling between datasets
35.3
18.5
65.1
39.6
Ours
42.0
20.9
62.8
41.9
Table 9: Ablation experiments on data sampling. We compare sampling by the original size of each dataset (first row) evenly sampling across datasets, and validate the effectiveness of class-aware sampling for Objects365 and Openlmages (second row).
metric, expected AP (EAP), that calculates the class-specifi AP as the average (expectation if user chooses the corresponding class randomly) AP of all the corresponding training class for test class. The overall expected mAP (mEAP) a dataset is the average E-AP overall classes as the conventional mAP.
D ABLATION STUDIES ON TRAINING LOSS
Multi-dataset loss When training a unified label space, the annotations each dataset becomes incomplete. For example, there is no "fish" class in COCO, but COCO image may contain a fish as background. As the unified label space has fish class (from OpenImages), a naive softmax cross-entropy loss will mistakenly train the COCO fish bounding box as negative sample for the unified fish classifier. To facilitate this issue, we use modified sigmoid cross-entropy loss to ignore the output from out-of-source-dateee classes. Le., when training COCO images, only the 80 COCO classes are applied a loss.
Table 8 ablates different losses for training on multi-dataset. We start with sigmoid cross-entropy loss for classification. This is 0.5 mAP worse than softmax cross-entropy on single dataset training (results not shown). The advantage of sigmoid cross-entropy is it breaks the inter-class dependency and makes ignoring classes easier. We add the multi-dataset loss that ignores labels outside the source dataset. This gives in average 1.4 mAP improvements to the three datasets. Further, to facilitate the hierarchical-aware evaluation on OpenImages, we use a hierarchy- aware sigmoid cross-entropy for OpenImages samples that sets parent classes as positives, and ignores child classes. This gives 1.4 mAP improvement on OpenImages, and is compatible with other datasets.
ABLATION STUDIES ON DATA SAMPLING
We apply class-aware sampling (Senn al. 2016 Objects365 and OpenImages, and sample removing class-aware sampling drops 6 mAP in
COCO images uniformly. As is shown in Table
2x
6
8x
COCO
Objects365 Oimg.
COCO
Objects365 Oimg. COCO Objects365 Oimg.
Unified
42.0
20.9
62.8
44.6
23.3
64.5
45.4
24.4
66.0
COCO
41.5
42.5
42.5
Objects365
23.8
25.0
24.9
OpenImages
64.6
65.4
65.7
'ablle 10: Ablation studies trianing schedule. We train the unified detector and each dataset- specific models for different training schedules and show mAP on each training datasets.
13

---

Under review as a conference paper at ICLR 2021
notation
definition
dimension
range
N
number datasets
scalar scalar
integer
i
dataset index
1-N
L
of labels of dataset
[
1
L2d
L
set of labels the disjoint label space set of labels in the unified label space number objects in dataset i labels all objects dataset ground truth bounding boxes of all objects in dataset predicted labels in head j of all objects in dataset labels of all objects in dataset in the unified label space transform function from label space to the unified label space potential merges (a set of class indexes)
[
Σ. 14,1 [Z.]
1 [
L
1- IT
K
scalar
integer
K
[1. L
ba
K x
IR
[1, |L5))
1,
K
[1, L]
T
[ [T]]
{0,
N
[1. [L,)]
Table 11: Table of notations used in the algorithms. For each notation, we also list their dimension and the value range.
average for the three datasets. The Openlmage training set is 3 as Objects365, and 15x COCO. An alternative sampling strategy is to sample by this ratio. The outcome is it trades-off the small dataset performance for large dataset performance, and gives overall lower average mAP.
F ANALYSIS ON TRAINING SCHEDULES
Table. TO shows how the performance evolves when training goes longer. In 2x schedule (180k iterations), all datasct-specifie models converge within a 1.2 mAP gap comparing to the 8x schedule. The unified model is not fully converged, as each dataset only trained for a x schedule. From 2x schedule to 6x schedule, the gaps between the unified model and the dataset-specific models are narrowed. the final 8x schedule, the unified model surpasses the single dataset-model on COCO and OpenImages, and closely matches the Objects365 model.
G ALGORITHM DIAGRAMS
Al Algorithm Nutlines the workflow for training our unified object detector. Algorithm summarizes the algorithm we used for learning unified label space (Section 4. We list the notation definitions of both algorithm in Table. 11
14

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Training unified detector
Input : {xi by, ] labeled training datasets
Output: L: unified label space of all datasets
M: an object detector on the unified label space
MM < train detector( (xx, bj 1 train an initial object detector with a disjoint label
space L on all datasets
for
i 4- to N do
[b. 1
1= + _M(x)) // run the disjoint label space detector each dataset, and get boudning boxes and label predictions from each detection head.
end
L, T + learning unified label
(ance((b,))) b, space( 1. ((6,0) - ( Algorithm 2
for i + 1 to N do
1, < T.(.) // transform labels form its original label space to the unified label space &end
M <- train detector((xs (x,, 4, bi) // train an unified detector with labels in the unified label
space
10 Return: L, M
Algorithm 2: Learning a unified label space
Input (b, 1 ground truth bounding boxes and labels for each training dataset
{(b,)) () 1
Jj=1hi=* h predicted bounding boxes with predicted classes in all datasets
for each training dataset
Output: L: unified label space
T: the transformation from each individual label space to the unified label space
// Compute potential merges and merge cost
L = U, // Short-hand used to simplify notation
Ti + {()][ E L} // Set of single labels
Compute ce for all single labels // for most metrics
for n 2. ....N N do
Tn < 0
for t E T n-1 do
for 1 E L do
if and all labels t are from different datasets then
compute (Lu(I)-
if Ceu()) < r then
n-1
Add tu (1) to Tn.
end
14
end
15
end
16
end
17 end
18 + n=1 T,
N 19 // Solve the ILP.
<- ILP solver(c, T, // Solve equation (4).
21 Compute L, T from æ Return: L,T
15


Reviews:
[{"review_id": "ZU6zbXWGyP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main idea of the proposed work is to learn a universal label space for a given task (say object detection) and a set of different datasets with  semantically overlapping labels. The only supervision required by the approach is constituted by the single dataset label spaces and respective annotations. Each dataset label space may have partial o complete overlaps (e.g. rider mapping to cyclist and motorcyclist ). The approach exploits a pre-trained detector on the trivial label space given by the union of all label spaces as a starting point. An optimization problem jointly minimizing some loss taking into account the task error with a penalization on the cardinality of the label set.\n\nStrengths\n\n- Unsupervised approach not requiring to hand design hierarchy or label space correspondence\n- Results are above state-of-the-art (challenge winner)\n- Elegant formulation via constrained optimization\n\nWeaknesses\n\n- missing clear specification of loss in the optimization problem. The statement: \"The loss function in our constrained objective 1 is quite general and captures a wide range of commonly used losses. We highlight two: an unsupervised objective based on the distortion of the output compared to the pretrained model, and mean Average Precision (mAP).\" is confusing, is that THE loss you optimize in problem (1), is there something missing? a full clear specification would enhance the reproducibility of this work.\n- lack of simpler baselines in evaluation.  The approach can be seen as a method to merge dataset annotations; a trivial solution to this is to cluster samples according to some feature space and clustering algorithm and then retrain. This approach has the same requirements in terms of supervision and the final result (e.g. cardinality of final label space) can be controlled via the clustering algorithm. Why this has not been evaluated as a simpler baseline? This would look a lot like [a]\n- is the optimization only performing fine-tuning on the last layer? wouldn't be better to perform an actual fine-tuning of the whole detector according to the new found label spaces?\n- no comparison with handlabelling counterparts. In the intro two works Zhao 2020 and Lambert 2020 are referred as manual counterparts of the proposed method. While it is not expected that for the presented approach to perform better than this kind of algorithms it is surprising that this kind of comparison is completely avoided. \n\nThe problem at hand is very interesting, especially for industrial applications and the formulation is elegant and leads to superior results with respect to other approaches. The lack of some experiments and the not so clear specification of the loss (especially from a reproducibility point of view) are the main concerns I have and why the current paper rank is marginally above the acceptance threshold.\n\n\nReferences\n\n[a] Deep Clustering for Unsupervised Learning of Visual Features, 2018\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qz4HJn0Lf-J", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe general idea of the paper is clearly interesting: Training a unified detector across datasets by also optimizing for a common label space. The formulation to address the problem discussed in section 4 seems to make sense. \n\nThe experiments, however where quite confusing and inconclusive  in my view. \n\n- The \"main experiments\" of the paper are so called \"zero-shot\" results, while the paper never defines what this is referring to. The zero-shot setting that I know of (as e.g. defined on the Animals with Attributes dataset) does not seem to apply here as no \"novel class labels\" can be handled by the algorithm. Therefore it is not clear to me what zero-shot means in this paper. \n\n- Additionally, the main results are shown in table 1, while it is not clear what the columns and lines refer to. This might be linked to the issue that zero-shot is not defined, but as a reader I cannot read the table without a proper description of the table. As a reader I am left with guessing\n\n- The authors introduce a novel metric (mEAP) and it is unclear why this should be a better or more suitable metric. I see no reason why standard metrics could not be used when evaluation is done on a per dataset level. If I am not mistaken this is what is done in tab 1 - but again too little information is given to know for sure. The definition of mEAP discusses vaguely some mapping using word embeddings - but this leaves lots of room for speculation and the \"detailed\" description in append C is very ad hoc with arbitrary thresholds. As a reader I have no real understanding if this new measure is sensible as no attempt is made to convince the reader that this is a good measure. \n\n- Other results are reported in standard measures in table 2, 3 and show very small differences between different methods thus showing no real advantage of the proposed method.\n\n- linked to the first point: please make clear if the experiments are in a \"generalized zero-shot setting\" that is that all classes are present during testing or not and if the classifier output can be all classes or only the classes of the tested dataset, if the accuracy is with respect to all classes of the classifier or only with respect to the classes of the dataset, if training is done in a \"transductive\" setting or not (I assume not?), etc\n\nSo while overall the papers aims for an interesting goal, the reported results are either not conclusive or obscured (using an unclear setting and a novel metric that is unknown to the readers) where the conclusions are unclear as well. \n\nA minor note: The paper promises to make code available upon acceptance. However, given the fact that generation of the respective models will be beyond reach for many researchers, the authors should share also share the trained models and not only the code. \n\nAdditional comments after the rebuttal phase: \n\nOn the positive side\n- the authors are clarifying a few things in the rebuttal and also in the paper such as table 1 - for that table results are clearer now by updating the table and caption. \n\nOn the negative side, however, I am less convinced after the rebuttal than it looked to be prior to the rebuttal. Let me give some of the most important things\n\n- claiming that the results in table 2 and 3 are statically significant is hard to believe. The authors claim that they are but it is not clear what the mean by \"standard deviation in mAP for each of the datasets... is within .1mAP\" in their rebuttal - what is varied to get this standard deviation? Learning a detector e.g. with different random seeds will result in much larger differences than 0.1mAP - thus claiming this it is statically significant is actually not scientifically credible to me \n\n- while I understand the arguments the authors make while standard evaluation metrics are potentially not ideal or comprehensive, I am still quite strongly unconvinced about the novel metric mEAP - that seems very specific for the setting used here and does not lent itself for easy understanding and is also dependent on some threshold that is not clear how to choose. In that sense all the experiments using that particular measure are still not convincing to me. Additionally, the authors do not make a real attempt to make this measure more accessible in any way. It is mostly stated that mAP does not work. Even though, as the authors point out, for most labels there is a \"joint\" label across datasets which allows to evaluate that directly at least for those joint labels (and these are the majority of classes apparently). Also in the rebuttal the authors simply defend their metric rather than to acknowledge that this is not particularly useful. As said - rather unconvincing and I am sticking with that. The rebuttal is making me even less convinced about that metric as no attempt is made to show that the metric is sensible and fair. \n\n- I strongly recommend to NOT use the term zero-shot. It is not only confusing as mentioned before but also does not really apply for most labels (the authors mention themselves that for most labels there is a corresponding label in the other dataset) - thus is more of a domain-adaptation or label-adaptation problem than really a zero-shot setting. The authors defend the usage of the term zero-shot which I do not find praticularly unconvincing. \n\n\nminor\n- the so called \"zero-shot cross-dataset generalization\" setting is not properly defined. It is mentioned at the beginning of sec 5 without being properly introduced what really is meant. \n\n- typo first line sec 4.1: detectpr -> detector\n\nI really would have liked to see a strong rebuttal given the good results for the ECCV challenge and the importance of the problem. However, the rebuttal nearly caused me lowering the score. So overall the rebuttal has made me less convinced about the paper than before. Sorry to say. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NfWsMVqETJR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper Summary and strengths**\nIn this paper robust object detection is studied, that is a single network is trained over multiple datasets and evaluated on datasets not seen during training [as for example promoted by the robust vision challenge, or the MSEG datasets]. One common problem for these approaches is how to deal with the different label spaces of each datasets, naive approaches would define an union over all label spaces, which has the undesirable effect that a CAR in one dataset is a different CAR in another dataset. In this paper a method is proposed to map this union label space into data specific subspaces by grouping similar labels based on their object classifier scores. In this paper this is solved by  a linear integer program which finds the optimal number of labels in the unified label space and their mappings and weights. Moreover this approach shows promising results over the disjoint/union label space. \n\n**Weaknesses**\n1. Missing baseline: The main results in Table 1 do not compare to the simple 'best effort name matching' as provided by the Robust Vision Challenge dev kit. This would be interesting to see the difference in performance between 'semantic matches' (name matching) and 'visual matches' from the provided algorithm. Also the influence of lambda (on the compactness of the unified label space) should be studied, by a very large lambda value, all labels will be mapped to a single label.\n\n2. At first sight the method is conceptually extremely straightforward, define a joint label space, with a permutation to the dataset specific label spaces (with some constraints to make it direct mappings) and re-train a final linear layer on top of pretrained detector. However, the description is difficult to follow with the different symbols, super and sub scripts, and slightly different terminology used throughout the paper [union, disjoint label spaces]. I think this should be carefully considered since this is the most important section of the paper. Maybe psuedo code or a more illustrative description can improve the presentation of the method. [why is at most 1 label per dataset merged? That only holds for a specific label in the unified space]\n \n3. Use the unified label space for training an end-to-end detector. The proposed method is only feasible since a pre-trained detector is used, cf Section 4. It would be interesting to see if an end-to-end trained detector on the found unified label space is able to boost the performance even further.\n\n4. Not really a weakness, but rather a possible extension to consider: In the current method only the visual appearances of the labels are used for constructing the unified label space. It would be interesting to also use the semantics of the provided labels [for the current method it doesnt matter whether a label is called CAR or BMNASDJHASD].  \n\n**Minor**\n- Page 3, Eq 1: N is undefined. \n- Page 7, Figure 3: 'Expert human' is largely overrated for the automatic name matching process used by the Robust Vision Challenge. [From their dev kit: We do provide a \"best-effort\" mapping, which can be a good starting point. This mapping will contain overlapping classes and some dataset entries might miss relevant labels.] It would also be good to show some 'failure' or 'unexpected' merged classes from the different datasets.\n- Please include the Instance Segmentation results in the main body of the paper.\n\n**Conclusion**\nThis paper provides a method for learning a unified labels space when training a task over multiple datasets. This is timely and highly relevant with the current focus on robust methods in various application of computer vision [Robust Vision Challenge, MSEG, VTAB, ...]. The paper is somehow difficult to follow, but that should be solvable within the rebuttal phase.\n\n**Post-Rebuttal**\nAfter reading the rebuttal, the updated manuscript and the other reviews, I became *less* convinced about this paper. While I remain positive about the conceptual idea of learning a unified label space. The authors did not successfully convince me in improving the understanding of the method. The pseudo-code alone does not make the  algorithm better to understand. It seems that the space T is insanely big (100x100x100 when three datasets of 100 labels are used). Also, while the authors do add a requested baseline, it is hardly compared (afaik only in the sentence that the expert human obtains 659 labels, while the learned space contains 701). Also note that the 'human expert' is a *\"best-effort\" mapping, which can be a good starting point.* according to the RVC dev kit. So, this might be an overclaim.\n\nFinally, please carefully proof read the paper, main typos remain. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "42E5g9w5jeD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary\nThe paper proposes to learn object detection model, while training on different datasets with different, potentially overlapping, label spaces. While previous methods do the label space mapping, from each dataset specific label space to the common universal label space, manually, this paper proposes to learn such mapping automatically. The proposed models work as well as dataset-specific models on resp. training datasets and generalize far better.\n\n\nPositives\n- The paper is well written and well organized, it is very easy to appreciate what is being done\n- The problem formulation of mapping each dataset label space to a joint label space using Boolean linear transforms, and integer programming formulation is novel and interesting\n- Results are given on challenging datasets which were part of the ECCV Robust Vision Challenge (RVC), and the experiments and the performances of the proposed method are convincing. The proposed method was one of the top performing method in ECCV2020 RVC\n\nNegatives\n- The shortcomings of the method are:\n* the method does not train the detectors end to end; it trains a final projection layer, which is put on top of the penultimate layer features of individually trained object detectors\n* (if I understand correctly) the training is done only with annotated objects in the different datasets, i.e. if there is a face object in the image of a dataset which doesn't have face in the label space (e.g. COCO does not have face label, but faces appear in the images of the dataset), that will not be used for training face part of the detector. In this respect, what happens with background boxes? During the label space merging background (boxes which do not belong to any object in any dataset) is completely ignored?\n- For evaluation on new datasets, as the authors noted, annotation over all the labels in the unified label space would be required. Zhao et al. provide such a test set, for a different collection of dataset, which might be useful in the future to try\n\nThese points should be discussed more in the paper.\n\nOverall the paper is well written and has a novel formulation and solution to the problem of label space merging. It also evaluates the method convincingly on challenging public benchmarks.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper was reviewed by four expert reviewers who identified the following pros/cons for the approach:

> Pros:
- Paper addresses an important problem [R3] 
- Formulation is simple, elegant an easily adoptable [R2, R3, R4]
- Experimental results are compelling (ECCV challenge winner) [R2, R3, R4]

> Cons:
- Experiments  are using "novel" evaluation metric with little justification [R1, R3]
- Details of the approach are unclear [R1]
- Lack of simpler baselines in evaluation [R4] 
- No comparison with handlabelling counterparts [R4]

Authors have addressed a number of comments in the rebuttal. With [R2] and [R4], generally, being convinced about accepting the paper.  However, [R1] and [R3] actually became less positive about the paper as a function of the rebuttal. [R3] mentioned reducing the score to a 3 and [R1] to a 6 in the discussion. It is unclear why these changes are not reflected in the publicly facing reviews. However, the fact that two of the four reviewers were disappointed with author responses and became LESS convinced about the paper is problematic in AC's opinion. 

AC also agrees that standard performance metrics should have been included. It is obviously reasonable to propose new metrics, but doing so should be accompanied by (1) reporting of performance with the original standard metric(s) and (2) justification for where prior metrics are problematic or faulty. While the proposed metric is reasonable for the specific use case outlined in Appendix C, it doesn't preclude standard evaluation metrics nor points out why they would be inappropriate or faulty. 

Overall, AC likes the paper and agrees that it presents a valuable approach that should be published. Unfortunately, the unjustified use of non-standard metrics without accompanied evidence, as noted above, is problematic and needs to be addressed in AC's opinion before that can happen. Since this issue was not addressed by authors in the rebuttal, AC sees no recourse but to reject the paper at this time, with a strong encouragement to address the aforementioned issue and to resubmit to CVPR or another top-tier upcoming venue.  

Decision: Reject