Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a knowledge distillation method for dense prediction tasks such as object detection and semantic segmentation. The method proposes to learn important regions (masks) adaptively for each image with task loss and mask diversity loss. Then the learned mask tokens, as well as mask weighting module, are adopted to distill teacher features to student. Results on COCO and Cityscapes datasets show the effectiveness of the proposed method.

strength_and_weaknesses: Strengths:
1. The idea is novel and interesting. The mask tokens learned by minimizing the task loss can generate finer and task-related distillation regions, which are demonstrated through the visualizations and experiments in the paper.

2. The proposed method is generic to various tasks, as it does not require ground-truth labels to select distillation regions as previous methods.

3. The experiments on detection and segmentation tasks show that, MasKD achieves significant improvements compared to previous state-of-the-art methods. Sufficient ablation studies are provided to show the efficacy of the method.

Weaknesses:
1. The performance on ImageNet dataset is not state-of-the-art.
2. How long does it take to learn the mask tokens for the teacher?
3. Why the authors learn the mask tokens for only 2000 iterations? How about training it for more iterations?
clarity,_quality,_novelty_and_reproducibility: This work meets the bar of publication in this venue.
summary_of_the_review: I don't find major issues with this work. Please see my comments in strengths and weakness.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a new distillation method that is more fine-grained than the previous bounding box distillation methods. It achieves performance improvement in both object detection and semantic segmentation.

strength_and_weaknesses: Pros:

1. Figure 2 clearly illustrates the framework of the proposed method.

2. This paper is overall well-written and easy to follow.

3. This paper is novel. I have never seen similar works before.

Cons:

1. For equation (7), how the weights of each mask are computed? What is the loss? Why this would not collapse to trivial solutions like emphasizing the masks that are easy to learn?

2. In Tables 1 and 2, the improvement over FGD seems marginal (around 0.2-0.4), which may come from randomness.

clarity,_quality,_novelty_and_reproducibility: Clarity: Fair. (Though most parts are well-written, some parts are still confusing.) 

Quality: Good.

Novelty: Good. (I didn't see this technique before)

Reproducibility: Fair. (Releasing the code can further improve the reproducibility)
summary_of_the_review: Though I still have some concerns, I think this paper is novel and well-written. Therefore, I would lean toward acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is about model distillation for dense prediction tasks, specifically for object detection and semantic segmentation. Prior works on this topic identify per-region distillation as an important ingredient to a successful knowledge transfer. This paper proposes a novel way to identify what regions should be distilled and with what importance (weight). These regions are modeled as masks, can be trained based on the respective task loss, and are not biased by the form of ground truth used (bounding boxes or segmentation maps). Given a teacher network, embedding vectors (one per region, the number of regions being a hyper-parameter) are learned that create a per-pixel similarity map with the feature maps, and then a (soft) masking. These embeddings (along with a corresponding weighting) are learned such that (a) the sum of all masked features can minimize the task loss, and (b) the masks are diverse. In a second phase, these masks are used for knowledge distillation into a student model. Additionally, the student also builds a mask (from the learned embeddings and the feature maps) and the distillation then becomes dependent on the agreement between teacher and student masks. Experiments on both object detection and semantic segmentation demonstrate consistent improvement over existing methods on multiple base models and backbones.
strength_and_weaknesses: Strengths:
- The proposed method for generating masks is task-agnostic in a sense that no priors about the output space (bounding boxes or segmentation maps) are used.
- It is good to see multiple detection and segmentation architectures and backbones evaluated.
- The method is simple and appears to be consistently effective across model designs

Weaknesses:
- I found the motivation for identifying (un)important regions/pixels in a task-agnostic manner unsatisfactory. Most of the motivation builds on foreground vs. background objects. But this is only valid for object detection, not so much for semantic segmentation. The better motivation seems to be a "balanced loss/impact" for different regions when doing distillation.
- I'm do not fully understand why it is "optimal" that only the pixels where both teacher and student masks agree should be distilled. This seems to be a guess (or a random justification) for something that works practically (i.e., the experiments show improvements).
- Overall, I found the experimental setting to be insufficient.
  - I'm missing more recent detectors like CenterNet [A] or DETR-based models like DINO [B-D].
  - Choosing R101 and R50 as teacher and student, respectively, seems odd to me. The performance difference is not big between these two backbones (Table 1). Isn't the more practical situation for model distillation to have a very strong model distilling information into a much smaller one? I see such experiments for segmentation (with MobileNet backbones), but shouldn't this be the default for most experiments?
  - Evaluations are only done on two benchmarks (COCO and CityScapes), although the method is advertised as general and task-agnostic. Given the rather small improvements over prior works across all experiments, I think additional evaluations on other datasets are needed to confirm consistent improvements of the proposed method over existing ones.
- Significance of the improvement over prior works
  - The improvement over FGD is consistent but minor in all results of Tables 1 and 2. This relates to my comment above about the experimental setting. With such consistent but only minor improvements, I find it necessary to provide more experimental evidence in the form of (a) additional datasets, and (b) reporting mean errors with standard deviation over multiple runs with different random seeds.
  - I see similarly small margins in Table 4, where the proposed method outperforms randomly initialized tokens by only 0.6% AP. Again, are these improvements statistically significant?
  - Why is the improvement much bigger for models trained from scratch (w/o ImageNet pre-training)? Is there any intuition for that? Also, are the teacher models also trained from scratch? If no, is this a practical situation where the teacher model can use ImageNet-pretrained weights, but the student cannot?
- Table 1: Is there an explanation why the student model outperforms the teacher? The motivation and intuition of model distillation is to mimic the teacher's behavior. How does the model become better than the student?

References:
- [A] Objects as Points. Zhou et al. arXiv 1904.07850
- [B] DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. Zhang et al. arXiv 2203.03605
- [C] Dn-detr: Accelerate detr training by introducing query denoising. Feng et al. CVPR 2022
- [D] DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR. Liu et al. ICLR 2022
clarity,_quality,_novelty_and_reproducibility: - Overall, the paper is well written. The background and the proposed method are well introduced. Figures are also okay.
- There is no mask weighting or refinement for semantic segmentation. This only becomes clear late in the paper. I think this distinction should be made clear earlier in the paper, when these methods are introduced.
- How are the weights in Eq. 7 learned? Is it the same as the masks, with the task loss on a frozen teacher model? This is unclear in the paper.
- Footnote 2 on page 4 does not say anything about "easy to converge", only that at convergence the original performance is almost fully recovered. The text says that mask tokens are easy and quick to train, but this is not apparent from the footnote.
- The first paragraph on page 6 is written in a complicated way. It is assumed that a mask for the student can also be created, but this is not obvious. I would first state that the learned embeddings are used to create a student mask.
- Table 4: What's the difference between the gray and the black marks (crosses and ticks)?
- Figure 1: What activations are visualized here? Are these CAM activations?
summary_of_the_review: In summary, I think the experimental setting needs to improve to justify and solidify the improvement over prior works. I lean towards rejection of the submission.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposes a new distillation method that is more fine-grained than the previous bounding box distillation methods. It generates masks in a task-agnostic way,  in the sense that no priors about the output space (bounding boxes or segmentation maps) are used. It achieves performance improvement in both object detection and semantic segmentation. Overall, the paper is well written.

Two out of the three reviewers recommend this paper highly favorably after rebuttal. One reviewer indicates that most of his/her prior concerns (experiment sufficiency, motivation clarity, etc.) have been addressed. The only concern left is still the relatively small improvements over the baselines, with most increases being smaller than 1 AP point. It has been noted that the experiment about the statistical significance in A4.1 seems to endorse the statistical significance and consistency of the performance gains. 

After reading the paper and holding internal discussions, AC feels that the overall sentiment is positive about this paper, and the negative reviewer stated that he/she won't object to acceptance. Hence, AC recommends this paper to pass the bar, as a poster presentation. 




justification_for_why_not_higher_score: This work remains to be an empirical idea and has no particularly exciting insight. As one reviewer pointed out during the discussion, the combination of the moderately creative technical idea and the rather minor improvements put a limitation on this paper's merit
justification_for_why_not_lower_score: The distillation approach does not require supervision, and it gives a consistent (albeit small) improvement. The benefit of distillation without ground-truth labels in this paper is a more general improvement compared to the existing methods. The experimental gains are consistent even though not always significantly
Decision: Accept: poster