Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors address the problem of catastrophic forgetting in the context of object detection. 
They propose a novel class-incremental object detection method based on Faster R-CNN for when there is no co-occurrence of base and novel object classes.
They also introduce a blind sampling strategy to select data from the large in-the-wild dataset for incremental learning.
Their dual-teacher distillation framework trained on this data outperforms SOTA on COCO and VOC datasets.

main_review: Object detectors are an important class of models that are heavily used in practice and indeed one of the main challenges is adapting these models to learn to detect  new classes objects rather than training them from scratch. In this regard, it is a significant challenge that is addressed in this paper. 


L 52: “we first introduce a blind sampling strategy to select relevant data from the in-the-wild data that contains large amounts of irrelevant images with neither the base nor novel class information.”
How do you ensure that? Do you mean the objects from the base and novel class do not exist in this dataset?!


L 69: Instead of assuming fully no co-occurrence of the base classes and the novel classes, wouldn’t it be more realistic to assume partial overlap as has been done in previous work?!




Evaluation

L 252: …  subsequently reduced by 0.1 after every 5 epochs? How? divided?


In table 1, It is odd that your method (last row) beats the baseline (row 1) where all data is seen at once? Why is that so?
Also, here the experiment is conducted in only one scenario (with tv being the novel class). It is important to repeat this experiment multiple times and also consider other permutations of base and novel classes to make sure results are statistically significant. Same argument applies to the results presented in other tables.


Results in Table 6 over COCO dataset does not seem complete and convincing! I think this is important as being the only result over COCO. Other experiments are over VOC which is easier to overfit. COCO is much more challenging and is used widely. 


Typos: 
L 307: Tables 4 shows -> tabe 
L 363 Overall, our method can even outperform the previous state-of. (Move even to later in the sentence)






Authors have addressed an important and practical problem and results of their experiments are generally convincing although I have some concerns as raised above. I am leaning towards acceptance after considering the authors’ rebuttal.
limitations_and_societal_impact: Authors admit not discussing the limitations

(b) Did you describe the limitations of your work? [No]
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigates incremental object detection problem without making the assumption of base and novel classes occurring together, i.e. person (base) and TV(novel) objects being in the same image. The authors make the argument that this does not always happen in real-world scenarios, and they propose to sample in-the-wild data instead. They call this stage the "Blind sampling strategy", and this is achieved by applying the pre-trained Base and Novel models to the unlabeled data, and collecting those that have "positive" output. After the blind sampling strategy, a student model is learned by distilling the knowledge from the base and teacher models. Finally, the method is evaluated on PASCAL VOC and MS COCO datasets.
main_review: The main contribution of this paper is to use the in-the-wild data, rather than the data with co-occurence for incremental object detection. The authors do not make the assumption that novel and base classes will co-occur in the images, which makes the problem more challenging and the solution more general. However, other than this, the technical contribution of this paper is very limited, as dual-teacher distillation is simply applied to distill the information between multiple models. 

Regarding the blind sampling strategy, I would have found it beneficial if the authors had shown more ablations and experiments for this part since it is the main contribution. For example, what happens if the data is randomly selected rather than using pre-trained models? Can the pre-trained models minimize the impact of true negatives during the distillation in this case? Also what happens if different sources of unlabeled data are used? For example, does the performance change if Open Images dataset was used as the unlabeled data instead of MSCOCO for the PASCAL VOC experiments?

The dual-teacher distillation can be costly in terms of memory and complexity, as three models (two teachers and a student) need to be used during the training. Can the authors discuss why two teachers are required? For example, assuming that the novel data is available at the same time as the unlabeled data, can the student be learned with novel+unlabeled data while distilling information from a single teacher model (learned with base classes)?

Overall, the paper is nicely written and offers a more generalized solution for incremental object detection. It would benefit from additional discussions/experiments as described above. My initial evaluation is leaning towards acceptance.
limitations_and_societal_impact: The authors do not discuss limitation nor societal impacts in the submission.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a technique to address the learning-without-forgetting problem when base and novel classes do not co-occur. The approach relies on sampling unlabeled in-the-wild data from another domain, to bridge base and novel classes. Results against a few recent baselines, as well as ablations, demonstrate the effectiveness of the approach.
main_review: The proposed approach is reasonable, but other than the novel setting (non-co-occurrence) the key novelty evades me. There are also some descriptions that were unclear. 

1. In particular, while it is clear why co-occurrence of base and novel classes shouldn't be assumed, it is not clear why strict non-co-occurrence is enforced (L135). 

2. From the description in L152-166, it is not clear to me where the data to be used in D_{unlabel} comes from. The previous text talks about D_{base} and D_{novel}. This is also not shown (perhaps just not labeled) in Fig. 1. 

3. The consistency check (L158) does not seem novel, and is in line with prior work on self-supervised learning (e.g. SimCLR). 

4. The right-hand side of Fig. 2 needs more explanation (e.g. textual labels for the different components). 

5. It was not clear to me why non-affection masks (where does the name come from?) alleviate confusion (L209). 

6. Experiments are strong and convincing, including ablations.

Minor:
(a) Why is the subscript "s" used for novel classes, and "t" for base?
(b) Why not use D_{base}, D_{novel} as an argument to M_{stud}?
(c) Typos, D_{basel} (L153) and "are be" (L193)
limitations_and_societal_impact: Not discussed
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a more realistic setting for class-incremental object detection, removing the need for co-occurrence if base and novel object classes in images in the training dataset. They propose a slightly complex method to address this scenario by assuming co-occurrence of the base and novel classes in supplementary, unlabeled data and distilling a model which learns from that data along with the training set using three levels of distillation losses and separate teacher models for base classes and novel classes that are distilled into a single, joint student model.
main_review: 
This paper points out the co-occurrence in training data assumption underlying much of the recent work in incremental object detection. The assumption that novel classes will co-occur with examples of the base classes is still central to this work, the main difference as I understand it based on this paper is the access to larger unlabeled datasets to have a larger pool that is likely to provide examples of base and novel classes.  This point would be made stronger with some additional data analysis and visualization. In the novel class data subsets in question, how frequently do novel classes occur with no co-occurrence to base ones? How many base classes are not represented at all? I know these datasets are not fully labeled, but your best model’s predictions could be used as an estimate of the level of co-occurrence/non-occurrence and the number of examples of each class in the training data (required by the SOA methods cited in related work) vs the co-occurrence and number of examples seen in the unlabeled data. 

The paper only compares to a single benchmark, which it outperforms across the board. However, if I am understanding correctly this benchmark does not get to see the significant amount of unsupervised data that their model does. How much of the performance gain is just through seeing more and more varied examples of the classes? Is it possible to disambiguate the performance gain from your methodology/proposed loss structure vs the performance gain from the unsupervised data? Or is the design of this methodology inseparable from the access to novel data to distill from?

Some minor grammatical errors throughout. Some additional editing for grammar is needed.

limitations_and_societal_impact: They have not addressed this as far as I can tell. One possible limitation is the assumption that all classes of interest will occur in the unsupervised data, allowing them to avoid forgetting. In real-world scenarios or applications there may not be a source of unlabeled data containing examples of all classes to distill from.
review_title: null

Meta-review: The paper proposes an approach for incremental object detection, which removes the need for co-occurrence of base and novel object classes in images in the training dataset. The reviewers scored the paper with (5, 6, 6, 6). After discussion and rebuttal,
they lean towards accepting the paper as a poster. 
Decision: Accept (Poster)