Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The main idea of the proposed work is to learn a universal label space for a given task (say object detection) and a set of different datasets with  semantically overlapping labels. The only supervision required by the approach is constituted by the single dataset label spaces and respective annotations. Each dataset label space may have partial o complete overlaps (e.g. rider mapping to cyclist and motorcyclist ). The approach exploits a pre-trained detector on the trivial label space given by the union of all label spaces as a starting point. An optimization problem jointly minimizing some loss taking into account the task error with a penalization on the cardinality of the label set.

Strengths

- Unsupervised approach not requiring to hand design hierarchy or label space correspondence
- Results are above state-of-the-art (challenge winner)
- Elegant formulation via constrained optimization

Weaknesses

- missing clear specification of loss in the optimization problem. The statement: "The loss function in our constrained objective 1 is quite general and captures a wide range of commonly used losses. We highlight two: an unsupervised objective based on the distortion of the output compared to the pretrained model, and mean Average Precision (mAP)." is confusing, is that THE loss you optimize in problem (1), is there something missing? a full clear specification would enhance the reproducibility of this work.
- lack of simpler baselines in evaluation.  The approach can be seen as a method to merge dataset annotations; a trivial solution to this is to cluster samples according to some feature space and clustering algorithm and then retrain. This approach has the same requirements in terms of supervision and the final result (e.g. cardinality of final label space) can be controlled via the clustering algorithm. Why this has not been evaluated as a simpler baseline? This would look a lot like [a]
- is the optimization only performing fine-tuning on the last layer? wouldn't be better to perform an actual fine-tuning of the whole detector according to the new found label spaces?
- no comparison with handlabelling counterparts. In the intro two works Zhao 2020 and Lambert 2020 are referred as manual counterparts of the proposed method. While it is not expected that for the presented approach to perform better than this kind of algorithms it is surprising that this kind of comparison is completely avoided. 

The problem at hand is very interesting, especially for industrial applications and the formulation is elegant and leads to superior results with respect to other approaches. The lack of some experiments and the not so clear specification of the loss (especially from a reproducibility point of view) are the main concerns I have and why the current paper rank is marginally above the acceptance threshold.


References

[a] Deep Clustering for Unsupervised Learning of Visual Features, 2018

review_title: Nice approach, very relevant for applications, lacking some technical details and experiments

final_score: rating: 7: Good paper, accept
review_content: 
Summary
The paper proposes to learn object detection model, while training on different datasets with different, potentially overlapping, label spaces. While previous methods do the label space mapping, from each dataset specific label space to the common universal label space, manually, this paper proposes to learn such mapping automatically. The proposed models work as well as dataset-specific models on resp. training datasets and generalize far better.


Positives
- The paper is well written and well organized, it is very easy to appreciate what is being done
- The problem formulation of mapping each dataset label space to a joint label space using Boolean linear transforms, and integer programming formulation is novel and interesting
- Results are given on challenging datasets which were part of the ECCV Robust Vision Challenge (RVC), and the experiments and the performances of the proposed method are convincing. The proposed method was one of the top performing method in ECCV2020 RVC

Negatives
- The shortcomings of the method are:
* the method does not train the detectors end to end; it trains a final projection layer, which is put on top of the penultimate layer features of individually trained object detectors
* (if I understand correctly) the training is done only with annotated objects in the different datasets, i.e. if there is a face object in the image of a dataset which doesn't have face in the label space (e.g. COCO does not have face label, but faces appear in the images of the dataset), that will not be used for training face part of the detector. In this respect, what happens with background boxes? During the label space merging background (boxes which do not belong to any object in any dataset) is completely ignored?
- For evaluation on new datasets, as the authors noted, annotation over all the labels in the unified label space would be required. Zhao et al. provide such a test set, for a different collection of dataset, which might be useful in the future to try

These points should be discussed more in the paper.

Overall the paper is well written and has a novel formulation and solution to the problem of label space merging. It also evaluates the method convincingly on challenging public benchmarks.
review_title: The paper proposes to learn object detection model, while training on different datasets with different, potentially overlapping, label spaces. While previous methods do the label space mapping, from each dataset specific label space to the common universal label space, manually, this paper proposes to learn such mapping automatically.

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
The general idea of the paper is clearly interesting: Training a unified detector across datasets by also optimizing for a common label space. The formulation to address the problem discussed in section 4 seems to make sense. 

The experiments, however where quite confusing and inconclusive  in my view. 

- The "main experiments" of the paper are so called "zero-shot" results, while the paper never defines what this is referring to. The zero-shot setting that I know of (as e.g. defined on the Animals with Attributes dataset) does not seem to apply here as no "novel class labels" can be handled by the algorithm. Therefore it is not clear to me what zero-shot means in this paper. 

- Additionally, the main results are shown in table 1, while it is not clear what the columns and lines refer to. This might be linked to the issue that zero-shot is not defined, but as a reader I cannot read the table without a proper description of the table. As a reader I am left with guessing

- The authors introduce a novel metric (mEAP) and it is unclear why this should be a better or more suitable metric. I see no reason why standard metrics could not be used when evaluation is done on a per dataset level. If I am not mistaken this is what is done in tab 1 - but again too little information is given to know for sure. The definition of mEAP discusses vaguely some mapping using word embeddings - but this leaves lots of room for speculation and the "detailed" description in append C is very ad hoc with arbitrary thresholds. As a reader I have no real understanding if this new measure is sensible as no attempt is made to convince the reader that this is a good measure. 

- Other results are reported in standard measures in table 2, 3 and show very small differences between different methods thus showing no real advantage of the proposed method.

- linked to the first point: please make clear if the experiments are in a "generalized zero-shot setting" that is that all classes are present during testing or not and if the classifier output can be all classes or only the classes of the tested dataset, if the accuracy is with respect to all classes of the classifier or only with respect to the classes of the dataset, if training is done in a "transductive" setting or not (I assume not?), etc

So while overall the papers aims for an interesting goal, the reported results are either not conclusive or obscured (using an unclear setting and a novel metric that is unknown to the readers) where the conclusions are unclear as well. 

A minor note: The paper promises to make code available upon acceptance. However, given the fact that generation of the respective models will be beyond reach for many researchers, the authors should share also share the trained models and not only the code. 

Additional comments after the rebuttal phase: 

On the positive side
- the authors are clarifying a few things in the rebuttal and also in the paper such as table 1 - for that table results are clearer now by updating the table and caption. 

On the negative side, however, I am less convinced after the rebuttal than it looked to be prior to the rebuttal. Let me give some of the most important things

- claiming that the results in table 2 and 3 are statically significant is hard to believe. The authors claim that they are but it is not clear what the mean by "standard deviation in mAP for each of the datasets... is within .1mAP" in their rebuttal - what is varied to get this standard deviation? Learning a detector e.g. with different random seeds will result in much larger differences than 0.1mAP - thus claiming this it is statically significant is actually not scientifically credible to me 

- while I understand the arguments the authors make while standard evaluation metrics are potentially not ideal or comprehensive, I am still quite strongly unconvinced about the novel metric mEAP - that seems very specific for the setting used here and does not lent itself for easy understanding and is also dependent on some threshold that is not clear how to choose. In that sense all the experiments using that particular measure are still not convincing to me. Additionally, the authors do not make a real attempt to make this measure more accessible in any way. It is mostly stated that mAP does not work. Even though, as the authors point out, for most labels there is a "joint" label across datasets which allows to evaluate that directly at least for those joint labels (and these are the majority of classes apparently). Also in the rebuttal the authors simply defend their metric rather than to acknowledge that this is not particularly useful. As said - rather unconvincing and I am sticking with that. The rebuttal is making me even less convinced about that metric as no attempt is made to show that the metric is sensible and fair. 

- I strongly recommend to NOT use the term zero-shot. It is not only confusing as mentioned before but also does not really apply for most labels (the authors mention themselves that for most labels there is a corresponding label in the other dataset) - thus is more of a domain-adaptation or label-adaptation problem than really a zero-shot setting. The authors defend the usage of the term zero-shot which I do not find praticularly unconvincing. 


minor
- the so called "zero-shot cross-dataset generalization" setting is not properly defined. It is mentioned at the beginning of sec 5 without being properly introduced what really is meant. 

- typo first line sec 4.1: detectpr -> detector

I really would have liked to see a strong rebuttal given the good results for the ECCV challenge and the importance of the problem. However, the rebuttal nearly caused me lowering the score. So overall the rebuttal has made me less convinced about the paper than before. Sorry to say. 

review_title: The paper aims to train a unified detector that works well across datasets by constructing a common label space that is share among these datasets. 

final_score: rating: 7: Good paper, accept
review_content: **Paper Summary and strengths**
In this paper robust object detection is studied, that is a single network is trained over multiple datasets and evaluated on datasets not seen during training [as for example promoted by the robust vision challenge, or the MSEG datasets]. One common problem for these approaches is how to deal with the different label spaces of each datasets, naive approaches would define an union over all label spaces, which has the undesirable effect that a CAR in one dataset is a different CAR in another dataset. In this paper a method is proposed to map this union label space into data specific subspaces by grouping similar labels based on their object classifier scores. In this paper this is solved by  a linear integer program which finds the optimal number of labels in the unified label space and their mappings and weights. Moreover this approach shows promising results over the disjoint/union label space. 

**Weaknesses**
1. Missing baseline: The main results in Table 1 do not compare to the simple 'best effort name matching' as provided by the Robust Vision Challenge dev kit. This would be interesting to see the difference in performance between 'semantic matches' (name matching) and 'visual matches' from the provided algorithm. Also the influence of lambda (on the compactness of the unified label space) should be studied, by a very large lambda value, all labels will be mapped to a single label.

2. At first sight the method is conceptually extremely straightforward, define a joint label space, with a permutation to the dataset specific label spaces (with some constraints to make it direct mappings) and re-train a final linear layer on top of pretrained detector. However, the description is difficult to follow with the different symbols, super and sub scripts, and slightly different terminology used throughout the paper [union, disjoint label spaces]. I think this should be carefully considered since this is the most important section of the paper. Maybe psuedo code or a more illustrative description can improve the presentation of the method. [why is at most 1 label per dataset merged? That only holds for a specific label in the unified space]
 
3. Use the unified label space for training an end-to-end detector. The proposed method is only feasible since a pre-trained detector is used, cf Section 4. It would be interesting to see if an end-to-end trained detector on the found unified label space is able to boost the performance even further.

4. Not really a weakness, but rather a possible extension to consider: In the current method only the visual appearances of the labels are used for constructing the unified label space. It would be interesting to also use the semantics of the provided labels [for the current method it doesnt matter whether a label is called CAR or BMNASDJHASD].  

**Minor**
- Page 3, Eq 1: N is undefined. 
- Page 7, Figure 3: 'Expert human' is largely overrated for the automatic name matching process used by the Robust Vision Challenge. [From their dev kit: We do provide a "best-effort" mapping, which can be a good starting point. This mapping will contain overlapping classes and some dataset entries might miss relevant labels.] It would also be good to show some 'failure' or 'unexpected' merged classes from the different datasets.
- Please include the Instance Segmentation results in the main body of the paper.

**Conclusion**
This paper provides a method for learning a unified labels space when training a task over multiple datasets. This is timely and highly relevant with the current focus on robust methods in various application of computer vision [Robust Vision Challenge, MSEG, VTAB, ...]. The paper is somehow difficult to follow, but that should be solvable within the rebuttal phase.

**Post-Rebuttal**
After reading the rebuttal, the updated manuscript and the other reviews, I became *less* convinced about this paper. While I remain positive about the conceptual idea of learning a unified label space. The authors did not successfully convince me in improving the understanding of the method. The pseudo-code alone does not make the  algorithm better to understand. It seems that the space T is insanely big (100x100x100 when three datasets of 100 labels are used). Also, while the authors do add a requested baseline, it is hardly compared (afaik only in the sentence that the expert human obtains 659 labels, while the learned space contains 701). Also note that the 'human expert' is a *"best-effort" mapping, which can be a good starting point.* according to the RVC dev kit. So, this might be an overclaim.

Finally, please carefully proof read the paper, main typos remain. 
review_title: For robust training a unified label space is essential

Meta-review: Paper was reviewed by four expert reviewers who identified the following pros/cons for the approach:

> Pros:
- Paper addresses an important problem [R3] 
- Formulation is simple, elegant an easily adoptable [R2, R3, R4]
- Experimental results are compelling (ECCV challenge winner) [R2, R3, R4]

> Cons:
- Experiments  are using "novel" evaluation metric with little justification [R1, R3]
- Details of the approach are unclear [R1]
- Lack of simpler baselines in evaluation [R4] 
- No comparison with handlabelling counterparts [R4]

Authors have addressed a number of comments in the rebuttal. With [R2] and [R4], generally, being convinced about accepting the paper.  However, [R1] and [R3] actually became less positive about the paper as a function of the rebuttal. [R3] mentioned reducing the score to a 3 and [R1] to a 6 in the discussion. It is unclear why these changes are not reflected in the publicly facing reviews. However, the fact that two of the four reviewers were disappointed with author responses and became LESS convinced about the paper is problematic in AC's opinion. 

AC also agrees that standard performance metrics should have been included. It is obviously reasonable to propose new metrics, but doing so should be accompanied by (1) reporting of performance with the original standard metric(s) and (2) justification for where prior metrics are problematic or faulty. While the proposed metric is reasonable for the specific use case outlined in Appendix C, it doesn't preclude standard evaluation metrics nor points out why they would be inappropriate or faulty. 

Overall, AC likes the paper and agrees that it presents a valuable approach that should be published. Unfortunately, the unjustified use of non-standard metrics without accompanied evidence, as noted above, is problematic and needs to be addressed in AC's opinion before that can happen. Since this issue was not addressed by authors in the rebuttal, AC sees no recourse but to reject the paper at this time, with a strong encouragement to address the aforementioned issue and to resubmit to CVPR or another top-tier upcoming venue.  

Decision: Reject