Relevance Attack on Detectors

1 INTRODUCTION . Adversarial attacks ( Szegedy et al . ( 2014 ) ; Goodfellow et al . ( 2015 ) ; Carlini & Wagner ( 2017 ) ; Mądry et al . ( 2017 ) ; Baluja & Fischer ( 2017 ) ; Su et al . ( 2019 ) ) have revealed the fragility of Deep Neural Networks ( DNNs ) by fooling them with elaborately-crafted imperceptible perturbations . Among them , the black-box attack , i.e. , attacking without knowledge of their inner structure and weights , is much harder , more aggressive and closer to real-world scenarios . For classifiers , there exist some promising black-box attacks ( Papernot et al . ( 2016 ) ; Brendel et al . ( 2018 ) ; Dong et al . ( 2018 ) ; Xie et al . ( 2019 ) ; Lin et al . ( 2020 ) ; Chen et al . ( 2020 ) ) . It is also severe to attack object detection ( Zhang & Wang ( 2019 ) ) in a black-box manner , e.g. , hiding certain objects from unknown detectors ( Thys et al . ( 2019 ) ) . By that , life-concerning systems based on detection such as autonomous driving and security surveillance would be greatly influence . To the best of our knowledge , no existing attack is specifically designed for black-box transferability in detectors , because they have multiple-outputs and a high diversity across architectures . In such situations , adversarial samples do not transfer well ( Su et al . ( 2018 ) ) , and most attacks only decrease mAP of black-box detectors by 5 to 10 % ( Xie et al . ( 2017 ) ; Li et al . ( 2018c ; b ) ) . To overcome this , we propose one plausible way to find common properties across detectors , which facilitates the discovery of common weaknesses . Based on them , the designed attack can threaten variable victims . In this paper , we adopt the relevance map as a common property , on which different detectors have similar interpretable results , as shown in Fig . 1 . Based on relevance maps , we design a Relevance Attack on Detectors ( RAD ) . RAD focuses on suppressing the relevance map rather than directly attacking the prediction as in existing works ( Xie et al . ( 2017 ) ; Li et al . ( 2018c ; a ) ) . Because the relevance maps are quite similar across models , those of black-box models are influenced and misled as well in attack , leading to the great transferability . Although some works have adopted the relevance map as an indicator or reference of success attacks ( Dong et al . ( 2019 ) ; Zhang & Zhu ( 2019 ) ; Chen et al . ( 2020 ) ; Wu et al . ( 2020a ) ) , there is no work to directly attack the relevance maps of detectors to the best of our knowledge . In our comprehensive evaluation , RAD achieves the state-of-the-art transferability on 8 black-box models for COCO ) dataset ( Lin et al . ( 2014 ) ) , nearly halving the detection mAP . Interestingly , the adversarial samples of RAD also greatly influence the performance of instance segmentation , even only detectors are attacked . Given the high transferability of RAD , we create Adversarial Objects in COntext ( AOCO ) , the first adversarial dataset for object detection . AOCO contains 10K samples that significantly decrease the performance of black-box models for detection and segmentation . AOCO may serve as a benchmark to test the robustness of a DNN or improve it by adversarial training . CONTRIBUTIONS • We propose a novel attack framework on relevance maps for detectors . We extend network visualization methods to detectors , find out the most suitable nodes to attack by relevance maps , and explore on the best update techniques to increase the transferability . • We evaluate RAD comprehensively and find its state-of-the-art transferability , which exceeds existing results by above 20 % . Detection mAPs are more than halved , invalidating the stateof-the-art detectors to a large extent . • By RAD , we create the first adversarial dataset for object detection , i.e. , AOCO . As a potential benchmark , AOCO is generated from COCO and contains 10K high-transferable samples . AOCO helps to quickly evaluate and improve the robustness of detectors . 2 RELATED WORK . Since ( Szegedy et al . ( 2014 ) ) , there have been lots of promising adversarial attacks ( Goodfellow et al . ( 2015 ) ; Carlini & Wagner ( 2017 ) ; Mądry et al . ( 2017 ) ) . Generally , they fix the network weights and change the input slightly to optimize the attack loss . The network then predicts incorrectly on adversarial samples with a high confidence . ( Papernot et al . ( 2016 ; 2017 ) ) find that adversarial samples crafted by attacking a white-box surrogate model may transfer to other black-box models as well . Input modification ( Xie et al . ( 2019 ) ; Dong et al . ( 2019 ) ; Lin et al . ( 2020 ) ) or other optimization ways ( Dong et al . ( 2018 ) ; Lin et al . ( 2020 ) ) are validated to be effective in enhancing the transferability . ( Xie et al . ( 2017 ) ) extends adversarial attacks to detectors . It proposes to attack on densely generated bounding boxes . After that , losses about localization and classification are designed ( Li et al . ( 2018c ) ) for attacking detectors . ( Lu et al . ( 2017 ) ) and ( Li et al . ( 2018a ) ) propose to attack detectors in a restricted area . Existing works achieve good results in white-box scenarios , but are not specifically designed for transferability . The adversarial impact on black-box models is quite limited , i.e. , a 5 to 10 % decrease from the original mAP , even when two models only differ in backbone ( Xie et al . ( 2017 ) ; Li et al . ( 2018c ; b ) ) . ( Wang et al . ( 2020 ) ) discusses black-box attacks towards detectors based on queries rather than the transferability as we do . The performance is satisfactory , but it requires over 30K queries , which is easy to be discovered by the model owner . Besides , physical attacks on white-box detectors are also feasible ( Huang et al . ( 2020 ) ; Wu et al . ( 2020b ) ; Xu et al . ( 2020 ) ) . For the great transferability , we propose to attack on relevance maps , which are calculated by network visualization methods ( Zeiler & Fergus ( 2014 ) ; Selvaraju et al . ( 2017 ) ; Shrikumar et al . ( 2017 ) ) . They are originally developed to interpret how DNNs predict and help users gain trust on them . Specifically , they display how the input contributes to a certain node output in a pixel-wise manner . Typical works include Layer-wise Relevance Propagation ( LRP ) ( Bach et al . ( 2015 ) ) , Contrastive LRP ( Gu et al . ( 2018 ) ) and Softmax Gradient LRP ( SGLRP ) ( Iwana et al . ( 2019 ) ) . These methods encourage the reference of relevance maps in attack ( Dong et al . ( 2019 ) ; Zhang & Zhu ( 2019 ) ; Chen et al . ( 2020 ) ; Wu et al . ( 2020a ) ) , and also inspire us . However , none of them attack on relevance maps for detectors . RAD differs from ( Ghorbani et al . ( 2019 ) ; Zhang et al . ( 2020 ) ) in the goal . RAD misleads detectors by suppressing relevance maps . In contrast , ( Ghorbani et al . ( 2019 ) ) misleads the relevance maps while keeping the prediction unchanged . ( Zhang et al . ( 2020 ) ) also misleads DNNs , but it keeps the relevance maps unchanged . 3 RELEVANCE ATTACK ON DETECTORS . We propose an attack specifically designed for black-box transferability , named Relevance Attack on Detectors ( RAD ) . RAD suppresses multi-node relevance maps for several bounding boxes . Since the relevance map is commonly shared by different detectors as shown in Fig . 1 , attacking on it in the white-box surrogate model achieves a high transferability towards black-box models . In this section , we first provide a high-level overview of RAD , and analyze the potential reasons of its transferability . Then we thoroughly discuss three crucial concrete issues in RAD . • In Section 3.3 , we specify the calculation of relevance maps for detectors , where current visualization methods are not applicable . • In Section 3.4 , we introduce the proper nodes to attack by RAD . • In Section 3.5 , we explore on the suitable techniques to update samples in RAD . 3.1 WHAT IS RAD ? . We present the framework of RAD in Fig . 2 . Initialized by the original sample x0 , the adversarial sample xk in the kth iteration is forward propagated in the surrogate model , getting the prediction f ( xk ) . Current attacks generally suppress the prediction values of all attacked output nodes in T . In contrast , RAD suppresses the corresponding relevance map h ( xk , T ) . To restrain that , gradients of h ( xk , T ) back propagate to xk , which is then modified to xk+1 . It is notable that RAD is a complete framework to attack detectors , and its each component requires a special design . Besides the calculation of relevance maps of detectors , other components in RAD , e.g. , the attacked nodes or the update techniques , also need a customized analysis . The reason is that no existing work directly attacks the relevance of detectors , and the experience in attacking predictions is not totally applicable here . For example , ( Zhang & Wang ( 2019 ) ) emphasizes classification loss and localization loss equally , but the former is validated to be significantly better in attacking the relevance in Section 3.4 . 3.2 WHY RAD TRANSFERS ? . RAD ’ s transferability comes from the attack goal : changing the common properties , i.e. , the relevance maps . As shown in Fig . 3 , the relevance maps are clear and structured for the original sample in both detectors . After RAD , the relevance maps are induced to be meaningless without a correct focus , leading to wrong predictions , i.e. , no or false detection . Because relevance maps transfer well across models , those for black-box detectors are also significantly influenced , causing a great performance drop , which is illustrated visually in Section 4.2 . RAD also attacks quite “ precisely ” , i.e. , the perturbation pattern is significantly focused on distinct areas and has a clear structure as shown in Fig . 4 . That is to say , RAD accurately locates the most discriminating parts of a sample and concentrates the perturbation on them , leading to a great transferability when the perturbations is equally bounded . 3.3 WHAT IS THE RELEVANCE MAPS FOR DETECTORS ? . We analyze the potential of RAD above , below we make it feasible by addressing three crucial issues . To conduct the relevance attack , we first need to know the relevance maps for detectors . Currently , there have been lots of methods to calculate the relevance maps for classifiers as described in Section 2 , but none of them are suitable for detectors . We take SGLRP ( Iwana et al . ( 2019 ) ) as an example to explain this and then to modify , because it excels in discriminating ability against irrelevant regions of a certain target node . SGLRP visualizes how the input contributes to one output node in a pixel-wise way by backpropagating the relevance from the output to the input based on Deep Taylor Decomposition ( DTD ) as illustrated in Appendix A. R ( L ) is the initial relevance in the output layer L and its nth component is calculated as R ( L ) n = { yn ( 1− yn ) n = t , −ynyt n 6= t , ( 1 ) where yn is the predicted probability of class n , and yt is that for the single-node target t. The pixel-wise relevance map h ( x , t ) for the single-node target t is calculated by back propagating the relevance R from the final layer to the input following rules specified in ( Iwana et al . ( 2019 ) ) . In detectors , we need the pixel-wise contributions from the input to m bounding boxes . This multi-node relevance map could not be directly calculated by ( 1 ) , so we naturally modify SGLRP as R ( L ) n = { yn ( 1− yn ) n ∈ T , − 1myn ∑m i=1 yti n /∈ T , ( 2 ) where yti is the predicted probabilities for one target node ti . T is the set containing all target nodes { t1 , t2 , ... , tm } . With iNNvestigate Library ( Alber et al . ( 2019 ) ) to implement Multi-Node SGLRP and Deep Learning platforms supporting auto-gradient , the gradients from RAD loss LRAD ( x ) = h ( x , T ) to sample x could be obtained according to the calculation rules of relevance maps in Appendix A . We illustrate the difference between SGLRP and our Multi-Node SGLRP in Fig . 5 . SGLRP only displays the relevance map for one bounding box , e.g. , “ TV ” , “ chair ” and “ bottle ” . Multi- Node SGLRP , in contrast , visualizes the overall relevance .

Summary: This paper presents a method for adversarial attacks on object detectors by exploiting relevance maps that are originally intended for model interpretation. Unlike most of the existing methods that attack detection scores directly, the proposed approach focuses on suppressing the relevance map associated with target objects by image perturbation. The idea is interesting and demonstrates good transferability on the tasks of object detection and segmentation.