Learning a unified label space

2 1 INTRODUCTION . Computer vision aims to produce broad , general-purpose perception systems that work in the wild . Yet object detection is fragmented into datasets ( Lin et al. , 2014 ; Neuhold et al. , 2017 ; Shao et al. , 2019 ; Kuznetsova et al. , 2020 ) and our models are locked into specific domains . This fragmentation brought rapid progress in object detection ( Ren et al. , 2015 ) and instance segmentation ( He et al. , 2017 ) , but comes with a drawback . Single datasets are limited and do not yield general-purpose recognition systems . Can we alleviate these limitations by unifying diverse detection datasets ? In this paper , we make training an object detector on the union of disparate datasets as straightforward as training on a single one . The core challenge lies in integrating different datasets into a common taxonomy and label space . A traditional approach is to create this taxonomy by hand ( Lambert et al. , 2020 ; Zhao et al. , 2020 ) , which is both time-consuming and error-prone . We present a fully automatic way to unify the output space of a multi-dataset detection system using visual data only . We use the fact that object detectors for similar concepts from different datasets fire on similar novel objects . This allows us to define the cost of merging concepts across datasets , and optimize for a common taxonomy fully automatically . Our optimization jointly finds a unified taxonomy , a mapping from this taxonomy to each dataset , and a detector over the unified taxonomy using a novel 0-1 integer programming formulation . An object detector trained on this unified taxonomy has a large , automatically constructed vocabulary of concepts from all training datasets . We evaluate our unified object detector at an unprecedented scale . We train a unified detector on 4 large and diverse datasets : COCO ( Lin et al. , 2014 ) , Objects365 ( Shao et al. , 2019 ) , OpenImages ( Kuznetsova et al. , 2020 ) , and Mapillary ( Neuhold et al. , 2017 ) . Experiments show that our learned taxonomy outperforms the best expert-annotated label spaces , as well as language-based alternatives . For the first time , we show that a single detector performs as well as dataset-specific models on each individual dataset . Crucially , we show that models trained on the diverse training sets generalize zero-shot to new domains , and outperform single-dataset models . Our models ranked first in the object detection and instance segmentation tracks of the ECCV 2020 Robust Vision Challenge across all evaluation datasets . Code and models will be released upon acceptance . 2 RELATED WORK . Training on multiple datasets . In recent years , training on multiple diverse datasets has emerged as an effective tool to improve model robustness for depth estimation ( Ranftl et al. , 2020 ) and stereo matching ( Yang et al. , 2019 ) . In these domains unifying the output space involves modeling different camera models and depth ambiguities . In contrast , for recognition , the unification involves merging different semantic concepts . MSeg ( Lambert et al. , 2020 ) manually created a unified label taxonomy of 7 semantic segmentation datasets and used Amazon Mechanical Turk to resolve the inconsistent annotations between datasets . Different from MSeg , our solution does not require any manual effort and unifies the label space directly from visual data in a fully automatic way . Wang et al . ( 2019 ) train a universal object detector on multiple datasets , and gain robustness by joining diverse sources of supervision . However , they produce a dataset-specific prediction for each input image . When evaluated in-domain , they require knowledge of the test domain . When evaluated out-of-domain , they produce multiple outputs for a single concept . This limits the generalization ability of detection , as we show in experiments ( Section . 5.2 ) . Our approach , on the other hand , merges visual concepts at training time and yields a single consistent model that does not require knowledge of the test domain and can be deployed cleanly in new domains . Both Wang et al . ( 2019 ) and MSeg ( Lambert et al. , 2020 ) observe a performance drop in a single unified model . With our unified label space and a dedicated training framework , this is not the case : the unified model performs as well as single-dataset models on the training datasets . Zhao et al . ( 2020 ) trains a universal detector on multiple datasets : COCO ( Lin et al. , 2014 ) , Pascal VOC ( Everingham et al. , 2010 ) , and SUN-RGBD ( Song et al. , 2015 ) , with under 100 classes in total . They manually merge the taxonomies and then train with cross-dataset pseudo-labels generated by dataset-specific models . The pseudo-label idea is complementary to our work . Our unified label space learning removes the manual labor , and works on a much larger scale : we unify COCO , Objects365 , and OpenImages , with more complex label spaces and 900+ classes . YOLO9000 ( Redmon & Farhadi , 2017 ) combines detection and classification datasets to expand the detection vocabulary . LVIS ( Gupta et al. , 2019 ) extents COCO annotations to > 1000 classes in a federated way . Our approach of fusing multiple readily annotated datasets is complementary and can be operationalized with no manual effort to unify disparate object detection datasets . Zero-shot classification and detection reason about novel object categories outside the training set ( Fu et al. , 2018 ; Bansal et al. , 2018 ) . This is often realized by representing a novel class by a semantic embedding ( Norouzi et al. , 2014 ) or auxiliary attribute annotations ( Farhadi et al. , 2009 ) . In zero-shot detection , Bansal et al . ( 2018 ) proposed a statically assigned background model to avoid novel classes being detected as background . Rahman et al . ( 2019 ) included the novel class word embedding in test-time training to progressively generate novel class labels . Li et al . ( 2019 ) leveraged external text descriptions for novel objects . Our program is complementary : we aim to build a sufficiently large label space by merging diverse detection datasets during training , such that the trained detector transfers well across domains even without machinery such as word embeddings or attributes . Such machinery can be added , if desired , to further expand the model ’ s vocabulary . 3 PRELIMINARIES . An object detector jointly predicts the locations bk ∈ R4 and classwise detection scores dk ∈ R|L| of all objects in a scene . The detection score describes the confidence that a bounding box belongs to an object with label l ∈ L , where L is the set of all classes . Figure 2a provides an overview . On a single dataset , the detector is trained to produce high scores only for the ground-truth class . Consider multiple datasets , each with its own label space L̂1 , L̂2 , . . .. A detector now needs to learn a common label space L for all datasets , and define a mapping between this common label space and dataset-specific labels L → L̂i . In this work , we only consider direct mappings . Each common label maps to at most one dataset-specific label per dataset , and each dataset-specific label maps to exactly one common label . In particular , we do not hierarchically relate concepts across datasets . When there are different label granularities between datasets , we keep them all in our label space , and expect to predict all of them . Mathematically , the mapping from the joint output space to a dataset-specific one is a Boolean linear transformation of the output of the recognition system d̂ik = T idk , with d̂ik ∈ R|L̂i| , T i ∈ { 0 , 1 } |L̂ i|×|L| , and constraints T i1 = 1 , T i > 1 ≤ 1 . The two constraints ensure that only direct mappings are learned . For simplicity , let T > = [ T 1 > . . . , T N > ] be the mapping to all dataset-specific output spaces . Figure 2b provides an overview . Prior work defined L and T by hand ( Lambert et al. , 2020 ; Zhao et al. , 2020 ) or used a trivial mapping T = I with completely disjoint outputs ( Wang et al. , 2019 ) . They then trained a detector given the fixed label space and mapping . In the next section , we show how to jointly learn the label space , the mapping to the individual datasets , and the detection scores in a globally optimal manner . 4 METHOD . We start with training a detector on the trivial disjoint label space ⋃ k L̂ k. In this section , we show how to automatically learn a unified label space by converting the disjoint label space into a unified label space . Once the unified label space is learned , we retrain the detector end-to-end with the unified label space . An overview of our workflow can be found in Appendix G 4.1 LEARNING A UNIFIED LABEL SPACE . We first consider only fine-tuning the last linear layer of the disjoint-label space detectpr . Specifically , let f1 , f2 , . . . be theD-dimensional features fi ∈ RD of the penultimate layer of the pretrained model for object locations b1 , b2 , . . . for all objects in a dataset . Our goal is to learn a new detection score dk = W > fk with parameters W = [ w1 , w2 , . . . , w|L| ] and wl ∈ RD , a label space L , and dataset-specific transformations T . The pretrained detector allows us to formulate this objective over a fixed set of precomputed detections and their features F = [ f1 , f2 , . . . ] : minimizeL , T , W ∑ l∈L ` l ( Tlw > l F ) + λ|L| ( 1 ) subject to T i1 = 1 and T i > 1 ≤ 1 ∀i∈ { 1 ... N } . Here ` l is a general loss function that factorizes over the labels l ∈ L , and N is the number of datasets . The weight wl controls the output of the detector for the joint label l , and Tl is a column of the dataset-specific transformation that maps each joint label l to all training datasets . The cardinality penalty λ|L| encourages a small and compact label set . A factorization of the loss ` l over the output space l ∈ L may seem restrictive . However , it does include the most common loss functions in detection : sigmoid cross-entropy and mean average precision . Section 4.2 discusses the exact loss functions used in our optimization . For a fixed label set L and mapping T , objective 1 reduces to a standard training objective of a detector . However , the joint optimization of L and T significantly complicates the optimization . It mixes combinatorial optimization over L with continuous optimization of W , and a 0-1 integer program over T . However , there is a simple reparametrization that lends itself to efficient optimization . First , observe that the label set L simply corresponds to the number of columns in T . Furthermore , we merge at most one label per dataset T i > 1 ≤ 1 . Hence , for each dataset i a column T il ∈ Ti takes one of |L̂i| + 1 values : Ti = { 0 , 1l̂1 , 1l̂2 , . . . } , where 1l̂ ∈ { 0 , 1 } |L̂i| is an indicator vector . Each column Tl ∈ T then only chooses from a small set of potential values T = T1 × T2 × . . . , where × represents the Cartesian product . Instead of optimizing over the label set L and transformation T directly , we instead use combinatorial optimization over the potential column values of t ∈ T. Let xt ∈ { 0 , 1 } be the indicator of combination t ∈ T. In this combinatorial formulation , the constraint T i1 = 1 translates to ∑ t∈T|tl̂=1 xt = 1 for all dataset-specific labels l̂ . Furthermore , the objective of the optimization simplifies to∑ l∈L ` l ( Tlw > l F ) + λ|L| = ∑ t∈T xt ` l ( tw > t F ) + λ ∑ t∈T xt . ( 2 ) Crucially , the weights wt of the detection score are now independent of the combinatorial optimization , and can be precomputed for each column value t ∈ T in a merge cost : ct = min wt ` t ( tw > t F ) . ( 3 ) This leads to a compact integer linear programming formulation of objective 1 : minimizex ∑ t∈T xt ( ct + λ ) subject to ∑ t∈T|tl̂=1 xt = 1 ∀l̂ ( 4 ) For two datasets , the above objective is equivalent to a weighted bipartite matching . For a higher number of datasets , it reduces to weighted graph matching and is NP-hard , but is practically solvable with integer linear programming ( Linderoth & Ralphs , 2005 ) . One of the most appealing properties of the above formulation is that it separates the integer optimization over xt from the continuous optimization of the last linear layer wt . Section 4.2 explores this separation and shows how to precompute the merge cost ct for various loss functions . One major drawback of the combinatorial reformulation is that the set of potential combinations T grows exponentially in the number of datasets used : |T| = O ( |L̂1||L̂2||L̂3| . . . ) . However , most merges t ∈ T are arbitrary combinations of labels and incur a large merge cost ct . Section 4.3 presents a linear-time greedy enumeration algorithm for low-cost merges . Considering only low-cost matches , standard integer linear programming solvers find an optimal solution within a second for all label spaces we tried , even for |L| > 600 and up to 6 datasets .

Summary: The main idea of the proposed work is to learn a universal label space for a given task (say object detection) and a set of different datasets with  semantically overlapping labels. The only supervision required by the approach is constituted by the single dataset label spaces and respective annotations. Each dataset label space may have partial o complete overlaps (e.g. rider mapping to cyclist and motorcyclist ). The approach exploits a pre-trained detector on the trivial label space given by the union of all label spaces as a starting point. An optimization problem jointly minimizing some loss taking into account the task error with a penalization on the cardinality of the label set.