Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims at improving adversarial robustness especially for the classification task, under the white box attack settings. Authors argue that richer annotations such as part segmentation can be helpful to improve the robustness. To this end, authors crafted several classification datasets with part segmentation masks available, and proposed a semantic segmentation based classification framework. Experiments with AutoAttack demonstrate the effectiveness of the proposed framework.
strength_and_weaknesses: ### Strength
* The investigated problem is very important and difficult, to defend against white box adversarial attack.
* The model architecture is somewhat straightforward and can be adapted to many other segmentation models.

### Weaknesses
* The conclusion in Table 1 is a bit confusing: the PGD adversarial training is more robust under adversarial attack, but the proposed method does not improve the robustness a lot with PGD adversarial training (e.g. 37.8 -> 38.5 on the Pascal-Part dataset). Considering the results reported in Table 5, training without segmentation supervision even leads to better adversarial robustness. So that means the extra supervision is more helpful towards clean accuracy but not on adversarial accuracy?
* The proposed method need extra annotation of segmentation masks, which may not readily available and existing datasets with segmentation masks may not fit the classification task without non-trivial modification. This limits the scopes the proposed approach can be applied on.
* It's a bit surprising that in appendix D.3, directly attacking the segmentation module does not work well. Can the authors give a bit more explanation about the reason and the detailed analysis of the segmentation attack results (in addition to the classification accuracy). As in [a], it seems that the segmentation task can be easily attacked.

[a] Adversarial Examples for Semantic Segmentation and Object Detection
clarity,_quality,_novelty_and_reproducibility: * Clarity: most part of the paper is clear enough
* Quality: see details in the "Weaknesses" above
* Novelty: good
* Reproducibility: should be good
summary_of_the_review: This paper proposes to improve adversarial robustness, however it requires extra annotation and does not improve adversarial accuracy with PGD adversarial training.

---
Post rebuttal updates:
I thank authors' response, and it partially addresses my concerns and explains why part segmentation is more robust. However, the Table 1 still concerns me that the proposed method is not very effective on the strong PGD trained model, when defending adversarial attack (e.g. 37.8->38.5). Thus I keep my initial ratings unchanged.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studied how to further improve deep learning model robustness with fine-grained annotations (object parts in 2D images). They proposed a part-based classification model and compared it with a vanilla ConvNet-based classifier (ResNet-50). On three common benchmarks (Part-ImageNet, Cityscapes, and PASCAL-Part), their experiments show the proposed part-based method consistently outperforms the holistic counterpart. They also conducted experiments around attacking the segmentation model and utilizing part-(Segmentation, BB, Centroids) information to further prove the effectiveness of the proposed method. 
strength_and_weaknesses: The studied direction is important, and this paper takes a step further on deep learning robustness with the help of fine-grained annotations. Overall the use of previous techniques and the design principles (part-based, disentangling irrelevant features, and location-aware) are reasonable. Their approach is properly motivated, and the paper is easy to follow. 

Experiments are conducted on three datasets, Part-ImageNet, Cityscapes, and PASCAL-Part, and shown the proposed part-based model achieved consistent improvements under both vanilla training and adversarial training, while the improvements further with adversarial training are limited. The experiments of attacking the segmenter model strongly prove the robustness of the proposed method. Table 6 shows the proposed method can also utilize BB and Centroids to improve model robustness. 

One of the major limitations of the proposed method is the use of extra data annotations (part mask/BB). Also, the part of objects in 2D images is not well defined. It is unknown if Vision Community has a generable "object part segmentor" for general 2d images. It is ok to study this in the existing dataset for academic purposes, while the reviewer is unsure if the proposed method can scale up to real-world questions. 

According to 3.5.2, the author has similar #params and training epochs for the baseline and the part-based model. It is good for fair comparisons, but, the reviewer wonders if the part-based model can achieve higher performance with more training epochs? By using more fine-grained information, it is expected to train longer somehow.

For the current manuscript, the proposed method is only examined in ResNet-50. The reviewer is curious if the proposed method can work on other ConvNets and ViTs? The use of part masks is natural to introduce Vision-Transformers, which is one of the popular backbones for vision tasks nowadays. It would be good to see the proposed method can model-agnostically improve the robustness.
clarity,_quality,_novelty_and_reproducibility: The clarity is good; so does the quality. Regarding the novelty, the reviewer feels the proposed method/direction is probably not very novel, people may have already exploited similar ideas in 3D data, where the parts of 3D objects are more salient than the "parts" of objects in 2D images. Reproducibility is guaranteed as the codes provided in the appendix. 
summary_of_the_review: Overall, the reviewer believes the empirical studies included in this paper can help the related community. However, the requirement of image-part (ill-defined) masks may significantly limit the practical use of the proposed approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents two stage end-to-end image recognition model that is robust against adversarial attacks. The model consists of the semantic segmentation module that tries to divide the objects of interest in the scene into several sub-regions and a part-based recognition model that makes the final outputs. Experimental results show that the proposed model can make the model very robust to the adversarial attacks. Detailed discussion on its performance is also given.
strength_and_weaknesses: [Strength]
I think the part-based model is a new idea and interesting.
It was interesting to me to see that even the clean dataset accuracy can be improved by the proposed method. In conventional approaches, it was common that the robustness to attacks can be enhanced by sacrificing the clean dataset accuracy.

[Weaknesses]
I have some concerns in the proposed method.

I am afraid the comparison to the prior work is not well demonstrated. The authors should compare their method with previous methods that can enhance the robustness to the attacks.

I am afraid that the practicability of the proposed model is quite limited because the labels for the semantic segmentation are required. I understand a smaller number of labels will still be valid as shown in the experimental validation, the requirement to have semantic segmentation labels would severely limit the applications of the proposed model.

The robustness to the adversarial attacks that attack semantic segmentation models is not discussed. Once this method is known to the attackers, it is apparent the attackers would attack the semantic segmentation model first. So, the authors may want to discuss its robustness.


clarity,_quality,_novelty_and_reproducibility: Very clear paper. It was easy to follow, easy to understand the core idea of the paper, and easy to grab the superiority of the paper. Very well written.
The experimental validation is good enough though not throughout.
I think the work is novel, but I have some concerns as summarized above. Please see my weakness section.
The code was included in the supplementary material, though I did not check it. So, it should be reproducible.

summary_of_the_review: I admit that the paper is interesting and well-written. However, there are pros and cons in this paper. By considering the trade-off between them, I am a bit on the rejection side of the paper. Please see my weakness section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a segmentation-based robust classification framework, combining a part segmentation model with a tiny classifier, and achieves a good performance. Furthermore, this paper offers two part-based models, a down-sampled part-based model and a bounding-box part-based model. Both achieve both higher accuracy and higher adversarial robustness than a ResNet-50 baseline.
strength_and_weaknesses: Strengths:
+ This paper proposes an interesting framework to use the segmentation results to help the classification robustness.
+ The experiments are sufficient, and the results demonstrate that the proposed method is effective compared with baselines.
+ This paper is technically sound and well-written.

Weakness:
+ This segmentation-based classification framework needs the introduction of segmentation labels, greatly limiting its application scenarios.
+ This paper only considers a ResNet baseline. It’s hard to identify its effectiveness across different model architectures.
+ Because the proposed method is based on a segmentation framework, the segmentation results will affect the classification. I wonder whether segmentation parts’ training will affect the training time and convergence speed.

clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and reproducibility are good.
summary_of_the_review: This paper proposes a segmentation-based robust classification framework, combining a part segmentation model with a tiny classifier, and achieves a good performance. But my major concern is that its application scenario is limited because of the need for segmentation labels.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a heuristic approach that leverages part segmentations to increase the adversarial robustness for object classification. The proposed idea to leverage parts of ojects in networks explicitly is interesting and the achieved improvements under adversarial training using PGD or TRADES are very significant.
Yet, the initial presentation of the experimentatl results of  paper was unclear. 
During the discussion, many additional results have been reported so that the review scores have increased on average. After the author-reviewer discussion, the paper is still a borderline case, where several aspects are not very clear. 
We strongly recommend the authors to discuss the aspect as of  why the results in table 1 improve more strongly for clean data than for adversarial attacks, and to add the baseline results without adversarial training to the table.   
justification_for_why_not_higher_score: The paper proposes an interesting idea but the presentation of the experimental findings is not clear in all aspects. 
justification_for_why_not_lower_score: Overall, reviewers and AC agree that the proposed approach is sufficiently interesting, so that the paper can be acceptd to ICLR.
Decision: Accept: poster