Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors investigates the failure reasons of object detectors under adversarial attacks. Authors propose two analysing strategies, named DCM and ClsAVal, to understand the classification confusions and the contribution of localization failures. Authors analyzed 4 types of object detectors (single-stage, two-stage, anchor-free and transformer query based), and focused on the single-shot detectors with their analysing strategies. Some conclusions were made such as the classification confusion is usually the more serious root cause of detection failures.
strength_and_weaknesses: ### Strength
* The paper analyzes a large group of detector types, and analyzes object detector with and without adversarial training.
* The paper draws some clear conclusion, which may be helpful for future works on improving the robustness of object detectors.

### Weaknesses
* The novelty is limited and the findings are not super surprising that the adversarial attack is mostly making the classification wrong.
* I feel the conclusion made in section 3.2 that "The adversarial attack enforces the standard detection model to easily mis-classify foreground objects into the background" may depend on the attack method. For example, in figure 1 of [a], the dogs are detected to be "train" with high confidence score.
* Conclusions in section 3.2 (a) are mostly drawn from figure 6 which is qualitative analysis. More careful quantitative analysis is required.
* The clarity of the paper can be improved. For example:
  * In equation 1, the classification loss is only calculated on $y_i$ and $\mathbf{b}_i$. I feel this is incorrect since background classification should also be considered in loss?
  * In section 3.1 ClsAVal, should we emphasize that $M$ and $M^R$ have the same structure (e.g. $M^R$ is an adversarially trained counterpart of $M$), so that the indices $\it{idx}$ from one detector can be applied to the other? This is not super clear in that paragraph so it's a bit confusing.
  * It would be better to explain a bit the details of "$R_{idx}(A_{cls})$" etc in the caption of figure 6.

[a] Adversarial Examples for Semantic Segmentation and Object Detection
clarity,_quality,_novelty_and_reproducibility: ### Clarity
Can be improved. See the weakness above.

### Quality
Good

### Novelty
Not strong enough. See the weakness above.

### Reproducibility
Should be reproducible.
summary_of_the_review: The paper provides adversarial robustness analysis on a large group of object detectors. However, the novelty is not strong enough and there are still some concerns in the current version, thus I lean towards weak reject at the current stage.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper attempts to investigate the problem of why object detection models fail with adversarial attacks. The authors tried to look into the classification and localization branch independently to locate the issue. Also, different types of object detection networks were investigated for this issue. According to experimental results, the authors made some conclusions about which part tasks more mistakes.
strength_and_weaknesses: Pros:
+ Adversarial attack in object detection is an important issue. The problem is practical and worth looking into.
+ The authors attempt to figure out which part fails for different models, which is a good attempt.

Cons:
- The paper is difficult to read. There are a lot of issues in writing. Most papers are not cited in the right way. For example, in ''FGSM Goodfellow et al. (2015), PGD Madry et al. (2018), AdvGAN Xiao et al. (2018)', the name of the model should not be messed with the author names. It is suggested that the authors look into the previous ICLR-accepted papers. Also, there are a lot of grammar issues. For example, ''Chen et al. (2018b) have been proposed to attack the state-of-the-art object detectors.' --> Chen et al. have proposed...
- The motivation and previous issues in these models in Introduction have already been discussed in previous papers, e.g. Adversarially-aware robust object detector in ECCV 2022. Fig 1 is also quite similar to Fig.2 in that paper.
- As a paper to investigate different types of detection models, the authors should have a good literature review on object detection models. However, the object detection models are categorised as two-stage, one-stage, anchor-free, and Transformer-based detectors. It's not an accurate classification, e.g., one-stage detectors can be either anchor-free or not. Previous papers have already clearly discussed the critical point that makes the detectors different from each other, e.g. ATSS (Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection). It is suggested to do a redundant literature review before discussing how the adversarial attack affects the detectors so that the authors can decide how and why these different categories should affect the investigation.
- The main observation that the classification module affects more in the adversarial attack has already been discussed in Section 3.2 of  Adversarially-aware robust object detector in ECCV 2022.
- Section 3.1  is very difficult to understand. The authors designed a very complex method (it looks like it matches the object index between attacked or clean images) to investigate the localization branch independently. It can be simply done by calculating the loss change as proposed in Adversarially-aware robust object detector in ECCV 2022.
- The results after NMS in Fig.3 are very confusing. It doesn't look like NMS has been done, as there are highly overlapped boxes. And the text with the bounding box sits in different locations in the same image, which makes it even more difficult to investigate clearly.
- The authors also report things like ''it is unable to successfully train Deformable-DETR on PASCAL VOCusing the official code.'', which makes me doubt the reproducibility of results in this paper.

clarity,_quality,_novelty_and_reproducibility: The writing of this paper is poor. The novelty of it is also minor. I'm not sure if the results can be reproduced.
summary_of_the_review: The paper investigates an essential and interesting issue. However, due to poor writing, serious issues in clarity and lack of novelty, I tend to reject this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: To alleviate the detection robustness bottleneck in the future work, this paper investigated the robustness of object detectors.
This work proposes two module: Detection Confusion Matrix (DCM) and Classification-Ablative Validation (ClsAVal).
DCM is used to analyze the confusion of detection results between different categories and their localizations, 
and ClsAVal is used for detection robustness analysis.
strength_and_weaknesses: Strengths
- Many existing models fail to explain what process they went through when producing incorrect results. This work points to these problems.
- Figure 3 and Figure 4 are sufficient to understand each module.
- Supplementary materials provide code and various experimental results.
- Various analyses of the experimental results suggest the direction of future research in this task.


Weaknesses
- Existing papers also point out this problem and show the need for a new model, which this paper claims is an initial attempt for analysis. A novel motivation is needed for proper analysis.
- Writing is not clear. All paragraphs should be written clearly so that even a first-time reader can understand them.
- I wonder if the object detection methods can be improved with this metric. Isn't it too consequential analysis to use both modules after applying NMS?
- The results can be intuitively confirmed through visualization, but I am not sure that this can be generalized to lead to a specific interpretation. 
clarity,_quality,_novelty_and_reproducibility: - Since $H_{cls}$ and $H_{loc}$ yield probabilistic confidence and the predicted localization for each bounding box, in Equation (1), $L_{cls}$ requires only $y_{i}$, and $L_{loc}$ requires only $b_{i}$.
The way of expression needs to be explained more clearly. The same goes for Equation (2).

- Figure 2 is not well understood. What are the criteria for dividing into robust and non-robust in the same model? Why is there a difference in performance within the same model? It is not intuitive to paste the performance of (a) into (b), and the caption needs more detail.

- Existing papers come up with novel methods to solve these problems, and it would be persuasive if there was an experiment to see how much these indicators improved.

- The two proposed modules are reproducible and applicable to existing models. Authors also provide reproducible code.
summary_of_the_review: This paper proposes two modules as indicators for confirming the robustness of existing object detectors for the adversarial attacks.
Experimental results applied to multiple cases show various interpretations in question.
However, these points are not considered novel, and it is hard to see what changes these indicators show when they actually improve.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a comprehensive study on the susceptibility of different object-detection pipelines to adversarial images. While the study in itself appears to be well-executed, it does not appear to lead to any meaningful new insights. In particular, the paper concludes that detection pipelines confuse the foreground and background class when confronted with adversarial images — this is exactly what the large literature on adversarial image generation designed adversarial images to do, so this conclusion is hardly surprising. As the study described in the paper does not lead to major new observations or learnings, its value to the ICLR community is limited.
justification_for_why_not_higher_score: This is a clear rejection.
Decision: Reject