Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies mixed supervised object detection where some object categories are fully supervised and the rest are only labeled with class tags. It introduces a (1) weakly-supervised segmentation module and (2) a semantic similarity learning module to the existing framework to improve the performance. Experimental results on two benchmarks validated the effectiveness of the proposed framework.
main_review: Strength:
- The problem setting is interesting. Compared to WSOD setting, the mixed-supervised task is more realistic and common in practice. However, this task is less explored than WSOD in the community.
- The experimental results are promising and both novel components help.
- Writing is mostly clear and easy to follow.

Weakness:
- To unifying the weakly-supervised segmentation and detection isn't a new idea. For example, multiple prior works [1-3] have studied and explored different architectures for this purpose. Some of these methods can be swapped into the proposed framework seamlessly and should be ablated and studied.
 

[1] Cyclic Guidance for Weakly Supervised Joint Detection and Segmentation. Shen et al. CVPR'19

[2] C-MIDN: Coupled Multiple Instance Detection Network With Segmentation Guidance for Weakly Supervised Object Detection. Gao et al., ICCV'19

[3] Multi-Evidence Filtering and Fusion for Multi-Label Classification, Object Detection and Semantic Segmentation Based on Weakly Supervised Learning. Ge et al., CVPR'18

- Sec.3.4 Iterative training seems to be complicated and hard to generalize. How important is this design choices? Will it become significantly worse if end-2-end training is adopted? I don't find experimental evidence to support this design choices.

- The learned mask is concatenated with the feature maps as input features. In fact, a more straight-forward solution to me is to multiply the feature map. I'm wondering whether this has been tried and how well does it perform.

- This paper follows the experimental setting from prior work where the datasets is split into non-overlap base and target categories. However, in practice the real problems are often more messy. The same class can have both annotation often the case. There are also prior work [4] studying this setting, which is missed in this paper. It would be interesting to see how well the proposed method performs to deal with mix-supervised for the same classes.

[4]. UFO2: A Unified Framework towards Omni-supervised Object Detection. Ren et al., ECCV'20

limitations_and_societal_impact: 1. Limitations is discussed in the Appendix Sec.7. I think this section is pretty detailed and in-depth.  
2. No societal impacts are discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a method to do `mixed supervision` object detection, i.e. some part of the dataset is annotated fully while some part only has weak image level annotations.

The first contribution is to use semantic mask prior in object detection network. Prior works have already used semantic masks for weakly supervised detection, but the networks were separate. The paper proposes to integrate the mask prediction networks with the object detection network and use the masks to augment features for prediction candidate bounding boxes. The hope is that this would also generalize and produce better candidate boxes for target/novel categories.

The second contribution is to transfer semantic similarity. The paper incorporates a similarity (metric learning) network in the architectures and learns it using the bounding boxes of the base categories. Then the network is used to estimate the average similarities of pseudo bounding boxes for the novel categories. With the intuition that the average distance of a correct example wrt the rest of the class examples would be lower than that for an incorrect example, they use a weighted loss for object class prediction for the bounding boxes.

The training is done in an iterative manner and results are provided on public benchmarks COCO and ILSVRC datasets as source, and Pascal VOC 2007 as the target dataset.
main_review: It is specified that base and novel categories have no overlap, but is it also assumed that the weakly supervised images for novel categories would not have any of the base category objects? It seems to be so from the experimental settings too. This would be a strong assumption for deployment, and should be mentioned/discussed explicitly.

The contributions are novel but, the use of semantic maps has been done already before in weakly supervised settings with separate networks while the present work integrates the networks. This might be good or bad as the integrated network is trained only on the source dataset but a separate network might be trained on other data as well. Similar case for metric learning algorithm. However, showing that an integrated network works for the task and performs better than existing method is a good contribution.

For, the results table it seems the row Zhong et al.[39]+distill uses VGG16 as the backbone as there is no * on its side. Could you please clarify, what is the backbone? If VGG is used as the backbone for this setting, then for fair comparison the results with ResNet50 should also be reported as it is the best existing method. 

A critical aspect in a mixed supervised detection setting is the amount of fully supervised data. An experiment showing the performance with varying amounts of supervised data would further provide an evaluation for the trade-off between supervised data and performance.

Overall the paper is well written and provides two contributions to the architecture as well as a tailored training method. The experiments reported are convincing.

limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a mixed supervised object detection method with several improvements (mask generator/prior, semantic similarity transfer) proposed to improve the detection performance in the mixed supervision setting. The ability of detect novel categories is enhanced by the output of the mask generator that is supervised by only image-level labels. A MIL classifier is used to generate pseudo labels (detections) for the novel categories and the quality (average similarity scores) of the pseudo labels are determined by the semantic similarity network trained on only the base categories. Then, the pseudo labels are used in the detector's subsequent training and their losses are weighted by the average similarity scores obtained from the semantic similarity network. This encourages "good" pseudo labels to contribute more to the training and vice versa.
main_review: Strengths
+ The ideas of using mask prior to enhance the detection performance on novel categories and semantic similarity transfer to weight the pseudo labels are quite interesting and novel in the context of mixed/weakly supervised object detection.
+ Strong experimental results in the standard setting. The proposed method outperforms state-of-the-art mixed- and weakly-supervised methods from very recent published papers.
+ Paper is generally well written, easy to follow, and has sufficient details for most parts of it.

Weaknesses
+ Lack of good explanation and evidence as to why the (metric-based) SimNet transfers well to novel categories. At the very least, the authors should show that the average similarity scores on base categories and "novel" categories are comparable, when that the scores are computed from the ground-truth boxes.
+ Table 3 does not specify clearly how the pseudo label weights are when no similarity function is used.
+ Sec 4.5 fails to mention which table it is referring to in the text.
+ Most of the weakly- and mixed-supervision prior works were focused on the small-scale dataset and splits. It would be very useful and more convincing if this paper could show its effectiveness in larger settings and datasets like Open Images and Objects365.
limitations_and_societal_impact: They are almost adequately addressed. It would be good for the authors to relate the limitations of their method to some negative societal impact they may have.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a Mixed Supervised Object Detection (MSOD) framework that is built on the Fast R-CNN framework and employs Multi-Instance Learning (MIL) to deal with weakly-supervised images. It adds two novel components to the underlying framework: Mask prior and Semantic similarity. Mask prior which is a semantic activation mask (e.g., CAM) works to enhance the ability to generate robust region proposals. Semantic similarity is used to remove inprecise proposals by checking its semantic similarity with other proposals. The proposed framework yields the better accuracy than other WSOD and MSOD methods on VOC 07 dataset.
main_review: This paper presents novel contributions in the methodology sufficient for consideration by NeurIPS community. The manuscript is very easy to follow.

My only concern is in experiments. Comparing with other baseline methods, most baselines used VGG16, but not the proposed method. This paper only provides the accuracy with VGG16 for the distrillation version. Therefore, I can not agree with that the proposed method has a benefit in the performance. I also wonder if other baselines also adopt multi-scale testing.
limitations_and_societal_impact: I did not find any limitation or potential adverse societal impact.
review_title: null

Meta-review: The reviewers have discussed the paper and are generally positive. In particular, some of the concerns raised in the reviewer were lifted, which resulting in improving the scores. I would like to encourage the authors to incorporate the suggestions into the final version and recommend accepting the paper.
Decision: Accept (Poster)