Towards Faithful XAI Evaluation via Generalization-Limited Backdoor Watermark
Authors: Mengxi Ya
Yiming Li
Tao Dai
Bin Wang
Yong Jiang
Shu-Tao Xia
Venue: ICLR 2024 poster (2024)
Keywords: XAI
XAI Evaluation
Backdoor Watermark
Backdoor Attack
Backdoor for Positive Purposes
Primary area: visualization or interpretation of learned representations
TL;DR: We reveal the unreliable nature and implementation limitations of existing backdoor-based SRV evaluation methods, based on which we propose a generalization-limited backdoor watermark and design a more faithful XAI evaluation.

Abstract: Saliency-based representation visualization (SRV) ($e.g.$, Grad-CAM) is one of the most classical and widely adopted explainable artificial intelligence (XAI) methods for its simplicity and efficiency. It can be used to interpret deep neural networks by locating saliency areas contributing the most to their predictions. However, it is difficult to automatically measure and evaluate the performance of SRV methods due to the lack of ground-truth salience areas of samples. In this paper, we revisit the backdoor-based SRV evaluation, which is currently the only feasible method to alleviate the previous problem. We first reveal its \emph{implementation limitations} and \emph{unreliable nature} due to the trigger generalization of existing backdoor watermarks. Given these findings, we propose a generalization-limited backdoor watermark (GLBW), based on which we design a more faithful XAI evaluation. Specifically, we formulate the training of watermarked DNNs as a min-max problem, where we find the `worst' potential trigger (with the highest attack effectiveness and differences from the ground-truth trigger) via inner maximization and minimize its effects and the loss over benign and poisoned samples via outer minimization in each iteration. In particular, we design an adaptive optimization method to find desired potential triggers in each inner maximization. Extensive experiments on benchmark datasets are conducted, verifying the effectiveness of our generalization-limited watermark. Our codes are available at \url{https://github.com/yamengxi/GLBW}.

Published as a conference paper at ICLR 2024
TOWARDS FAITHFUL XAI EVALUATION VIA GENERALIZATION-LIMITDD BACKDOOR WATERMARK
Mengxi Ya'; Yiming Li Tao Dai Bin Wang 15, Yong Jiang', Shu-Tooo Xia'
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China The State Key Laboratory of Blockchain and Data Security, Zhejiang University, China
ZZJU- Hangghog Global Scientific and Technological Innovation Center, China
*College of Computer Science and Software Engineering, Shenzhen University, China 5 GGuangzhou Intelligence Communications Technology Co.,Ltd., China
[yamx21, w-b21} @ imails.tsinghus. edu.cn; (eenm/uuu..uccm daitao.edu @ gmm; (iangy, xiast} @ www.tsingha.edu.com
ABSTRACT
Saliency-based representation visualization (SRV) (e.g., Grad-CAM) is of the most classical and widely adopted explainable artificial intelligence (XAI) methods for its simplicity and efficiency. It be used interpret deep neu- ral networks by locating saliency areas contributing the most to their predictions. However. it is difficult to automatically measure and evaluate the performance of SRV methods due to the lack of ground-trum truth salience areas of samples. In this paper, we revisit the backdoor-based SRV evaluation, which is currently the only feasible method to alleviate the previous problem. We first reveal its implementa- tion limitations and unreliable nature due to the trigger generalization of existing backdoor watermarks. Given these findings, we propose eeneralization-limieee backdoor watermark (GLBW), based on which we design a more faithful XAI evaluation. Specifically, formulate the training of watermarked DNNs min-max problem. where we find the 'worst' potential trigger (with the highest attack effectiveness and differences from the ground-truth trigger) via inner imization and minimize its effects and the loss over benign and poisoned samples via outer minimization in each iteration. In particular, we design an adaptive op- timization method to find desired potential triggers in each inner maximization. Extensive experiments on benchmark datasets are conducted, verifying the ef- fectiveness of our generalization -ineiiizzatio---iiieddddde watermark. Our codes are available at attps://github. http://uw...ooooaagggge///////.
INTRODUCTION
Deep neural networks (DNNs) have been widely and successfully adopted in many areas, such as facial recognition (Tang & Li 2004 Qiu et al. 202T Xia et al. (023). Despite their promising performance, DNNs are also criticized for acting pretable. The lack of interpretability of DNNs is mostly due to their nonlinear components (e.g.. activation functions) and multi-layer structure, hindering DNNs to be deployed in mission-critical applications (e.g., medical diagnosis) at scale.
bblakk box" whose predictions are not inter-
Currently, there tremendous efforts to explain their prediction behaviors and mechanisms. These methods are called explainable artificial intelligence (XAI). Current, existing XAI methods can be
roughly divided into three main categories, including (1) visualizing DNN representations (Zeiler
& Fergus 2014 Wang et al. 2020 Rao et al. 2022), (2) distilling DNNs to explainable models (Li al. 2022bb a Ha et al.[ 1), and (3) building explainable DNNs (Zhang et al.) 2018 Xiang al. Li al. 2022a)). Among all these methods, saliency-based representation visualization (SRV)
2020
(Springenberg et al. 2013 Selvaraju et al. 2017). as a sub-category of the first one, is probably the most classical and widely adopted one for its simplicity and efficiency. In general, these methods try measure the significance of each input region the final prediction. Accordingly, they can locate and highlight saliency input areas that contribute the most to model prediction.
"The first authors contributed equally this work. Correspondence Yiming Li (li-ym H.ym@zju.edu.ccm

---

Published as a conference paper at ICLR 2024
Backdoour
Original Trigger
Potential Trigger
SRV Evaluation
BP
FA
LIME
30
30
Target Label
Watermarked Imuge with Original Trigger
IOU - 0.47 Rank 2
IOU 0.42
IOU 1.00 Rank
Rank
Backdoored
30
30
DNN
Potential Trigger
Watermarked Image with Potential Trigger
IOU = 0.28 Rank
IOU = 0.50
IOU = 0.13 Rank
Distribution
Rank=1
The generalization of existing backdoor watermarks where all poten- tial triggers can activate backdoors.
Figure 1:
The backdoor- three SRV methods (i.e., BP, FA, and LIME) with original and potential triggers.
Figure 2:
evaluation of
However, it is difficult to measure and evaluate the performance of SRV methods due to the lack of the ground-truth salience map of the sample. Currently, the most common evaluation method is to have human experts score or rank saliency maps of different SRV methods for each sample. This method is time-consuming, expensive, and may be subject to significant deviations due the bias of different people. To alleviate this problem, (Lin et al.] 2021) ) proposed to exploit (patch- based) backdoor attacks (Li al., 2022c) (e.g-, BadNets (Gu al.] 20T9h) to design automatic and efficient XAI evaluation. Specifically, they modified a few training samples by stamping pre-defined trigger patch on their images and changing their labels specific target label. DNNs trained on the modified dataset will learn a latent connection between the trigger patch and the target label (i.c., the backdoor). Accordingly, all samples containing the trigger patch will be predicted to be the target label, no matter what their ground-truth label is. Given this backdoor property, they can calculate the average intersection over union (IOU) between the saliency areas generated by the SRV method of the backdoored model over different backdoored samples and the trigger area as an indicator to evaluate the SRV method. This method is developed based on the understanding that trigger regions can be regarded areas that contribute the most to the model's predictions and therefore can serve as the ground-truth salience areas for evaluation.
In this paper, we revisit the backdoor- based SRV evaluation. We first reveal that it has three im- plementation limitations, including (1) failing to take absolute values of gradients in the calculation of some SRV methods, (2) selecting saliency areas based the threshold instead of the saliency size, and (3) using the minimum bounding box covering all edges of significant regions instead of the regions themselves for calculating IOU of SRV methods, leading potentially false results. More importantly, this method relies on latent assumption that existing backdoor attacks have no trigger generalization, i.e., only the trigger used for training (dubbed 'original trigger') can activate backdoors, since they treat trigger regions as the ground-truth salience areas. We show that this assumption does not hold for existing backdoor attacks and therefore may lead to unreliable results. For example, given a SRV method, assume that its generated saliency areas of most poisoned sam- ples are only a small part of that of the original trigger. According to the existing backdoor-base evaluation, this SRV method will be treated as having poor performance since it has small IOU. However, due to the generalization of backdoor watermarks, the model may only learn this local re- gion rather than the whole trigger. In this case, the evaluated SRV method is in fact highly effective, contradicting to the results of the baokooorrbased SRV evaluation. Besides, existing backdoor- based evaluation has different results with potential trigger patterns that can activate the backdoor while significantly different from the original trigger (as shown in Figure 1 This phenomenon further verifying that its results are unreliable. Accordingly, an intriguing and important question arises: Is it possible to design faithful automatic XAI evaluation method?
In this paper, we explore how to reduce the position generalization of backdoor triggers, based on which to design a faithful backdoor-based XAI evaluation. Arguably, the most straightforward method is to synthesize the potential trigger pattern and penalize effects based on its distance to the original trigger in each iteration during the training process. However, as we will show in our main experiments, this method suffers from relatively poor performance (i.e., still with high trig- ger generalization) in many cases. This failure is mostly because the synthesized triggers are 'weak' with either a large loss value or are similar to the original trigger. Based on these understandings, we formulate the training of watermarked DNNs as min-max max problem, where find the 'worst" po- tential trigger with the highest attack effectiieness and differences from the original trigger via inner maximization and minimize its effects and the loss over benign and poisoned samples through outer
2

---

Published as a conference paper at ICLR 2024
minimization. In particular, we design adaptive optimization method to find desired ('worst') potential triggers in each inner maximization.
In conclusion, the main contributions of this paper four-folds: (1) We reveal and address the implementation limitations of the existing backdoor-based XAI evaluation; (2) We analyze the gen- eralization pattern of backdoor triggers, based on which we further reveal the unreliable nature of the existing backdoor-basee based XAI evaluation; (3) We propose a generalization-liminn termark (GLBW), based on which we design a more faithful XAI evaluation. To the best of our knowledge, we are the first trying to measure and even manipulate trigger generalization; (4) We conduct extensive experiments on benchmark datasets to verify the effectiveness of our GLBW.
backdoor wa-
2 BACKGROUND AND RELATED WORK
2.1 BACKDOOR ATTACKS AND THEIR POSITIVE APPLICATIONS
Backdoor attack is an emerging threat to deep neural networks et al. 2022c). Currently, existing backdoor attacks can be roughly divided into two main categories, including patch-based attacks and non-patch-bess based attacks, based on the trigger properties.
Specifically, basch-based attacks (Gu et al. 2019 Li et al., 2022e Gao et al. (023b) adopted some local image patches as the trigger patterns. For example, BadNets randomly modified a few samples from the original benign dataset by stamping the black-and-white trigger square to their images and changing their labels to the target label. These attacks are highly effective since their trigger patterns significantly different from the surrounding of the replaced image areas in the poisoned samples; In contrast, non-patch- based attacks exploited some transformations over the whole image, such as additive noises (Li al. 2021b),
warre-g (Nguyen & Tran 2021), and color shifting (Gong iet al. 2023), to generate poisoned images. In particular, a few works (Qiao et al., 2019 Li et al. 202 Qi al., 2023) initially discovered trigger generalization phenomenon, i.e., some potential trigger patterns are (significantly) different from the one used for training but can still activate hidden backdoors in the attacked models, in a few cases of neural cleanse (Wang et al., 2019). However, none of them tried to provided its statistical patterns and measure or control this generalization. How to better understand and control trigger generalization still an important open open-question.
Except for using for malicious purposes, there are also few research studied how to exploit back- door attacks for positive purposes based on their properties. such as adversarial defense (Shan al.) 2020) and copyright protection (Li et al.] 2022b; 2023b] Guo et al., 2023; Gan et al., 2023). In particular, Lin el al. (Lin et al. 2021) h used backdoor attacks to watermark DNNs, based on which to design evaluation method of daliency-basee representation visualization. However, almost all these applications relied on a latent assumption that backdoor triggers have mild generalization, which does not holds for existing backdoor attacks.
2.2 SALIENCY-BASED REPRESENTATION VISUALIZATION (SRV) AND ITS EVALUATIOOS
How explain the predictions mechanisms of DNNs is always important research direction. Currently, existing XAI methods of DNNs can be divided into three main categories. including (1) visualizing DNN representations (Zeiler & Fergus 2014 Rao al. 2022, Chen et al., 2024). (2)
distilling DNNs to explainable models (Li et al. 2020a Ha et al. 2222 and (3) building explainable
DNNs (Zhagg al. 2018 Xiang et al. 2020 Li et al. 2022a). In this paper, we focus on the
saliency-based representation visualization (SRV), which sub-type of the first category.
Saliency- Representation Visualization (SRV). In general, SRV methods explore visual pat- terns within a DNN unit (e.g., the logit). Specifically, they generate a saliency map for each sample whose value indicates the contribution of its corresponding part (e.g., pixel) to the unit's result. In general, we can divide existing SRV methods into two main categories, including white-box SRV and black-box SRV. Their difference lies in whether the SRV method can access the source files of the model. Currently, gradient-based methods are the mainstay of white-box SRV. For example, back- propaaatpon (BP) adopted the gradient of samples concerning their predicted results to render their saliency maps; Guided propaaaopagttion (GBP) (Springenberg et al.) 2015) only considered the positive gradients and set the negative parts as zero in ReLU layers; saddient-weighee class activation mapping (GCAM) (Selvaraju et al. 2017) generated saliency maps via features in the
3

---

Published as a conference paper at ICLR 2024
last convolutional layer; GGCAM (Selvaraju et al., 2017) combined GBP and GCAM via element- wise multiplication. Black box SRV usually obtained the saliency map by perturbing the imput and observing prediction changes. For example, occlusion sensitivity (OCC) (Zeiler & Fergus 2014) occluded different image portions and monitored output differences; LIME (Ribeiro et al, 2016 learned interpretable model locally varound the prediction; Recently, feature ablation (FA) [Meyes et al. 2019) perturbed a group of features instead of single one to render saliency maps.
The Evaluation Methods of SRV Methods. In general, it is difficult measure and evaluate the performance of SRV methods due to the lack of the ground-truth salience map of a sample. Cur- rently, the most reliable evaluation of SRV methods is still based on the human inspection with ranking or scoring (Ribeiro et al. 2016 Kim et al.] 2018 Dwivedi et al., 2023). However, this evaluation is time-consuming and costly or even leads to biased and inaccurate results (Buçinea et al., 2020). There are also a few automatic SRV evaluations, mostly by measuring the performance fluctuations when perturbing critical regions (Samek et al.) 2016 Yeh et al. 2019 van der Waa et al. 2021). However, these methods usually require heavy computations and even inaccurate due to distribution shift (Hooker et al.] 2019); Recently, Lin et al. (Lin et al., 202T) proposed to exploit backdoor-basee model watermarks design SRV evaluation, based on the principle that trigger locations can be regarded as the most critical regions contributing the most to model's pre- dictions. Specifically, it first generated the saliency maps of some poisoned samples containing trigger patterns and turned them into 0-1 masks based on pre-defined thresholds. After that, it found a minimum bounding box covering all edges of the trigger mask detected by the Canny algorithm Canny| 1986). The final score was calculated by the distance, such as average intersection over union (IOU), between generated bounding box and the trigger mask over all samples. However, as we will show in the following sections, this evaluation method is also unreliable.
REVISITING BACKDOOR BASED XAI EVALUATION
3.1 STANDARDIZE BACKDOOR- XAI EVALUATION
Currently, the backdoor-based SRV evaluation (Lin al. 2021) still has some implementation lim- itations that may lead to inaccurate results. In this section, we will standardize it.
Implementation Limitations and Their Standardiza- tion. Firstly. (Lin et al.] 2021) took absolute values of gradients in only parts of (instead of all) evaluated SRV methods before generating their saliency map. Since both positive or negative influnces are all important, to ensure a fair comparison, we propose to conduct it on the gradients of all SRV methods; Secondly, (Lin et al., 202I) adopted pre-defined threshold to filter out the most critical re- gions (i.e.. the 0-1 mask) from the saliency map for SRV methods. However, the same threshold value has different influences across samples even for the same SRV method. To ensure fair comparison, we propose to select M pixel locations with the maximum saliency value as significant regions, where M is the size of trigger used for model watermarking; Thirdly. (Lin et al. 202I) adopted the minimum bounding box covering all edges of significant regions instead of themselves ves for calculating IOU. It may lead to inaccurate results, especially when the original trigger has complex shapes. Accordingly, we calculate the IOU based on the significant regions directly. In general, the second limitation has largest effects.
Figure 3: The trigger patterns used for standardized evaluation.
We conduct the following experiments to verify that the vanilla backdoor based SRV evaluation may lead to inaccurate results and therefore we need to standardize it. Please refer to Appendix C for more details about why Lin et al. (22T)) and we only consider patch-based triggers.
Settings. We adopt ResNet- 18 (He et al. 2016) on CIFAR-10 (Krizhevsky) 2009) and GTSRB (Stallkamp et al. 2012) datasets for our discussions. Following the settings in (Lin et al. 2021), we adopt BadNets (Gu et al. 2019) for model watermarking. Specifically, the target label is set to '0' and the watermarking rate is set to 5%. We adopt four triggers with two patterns (as shown in Figure 3) located in the right corner or random position of the image. We randomly sample 100 testing images for evaluation. We calculate the average rank by averaging the rank of SRV methods (based the IOU) in each scenario. Please find more details in Appendix
4

---

Published as a conference paper at ICLR 2024
The evaluation (IOU) of SRV methods with vanilla backdoor- backdoorbbased method and its stan- dardized version on the CIFAR- 10 dataset and the GTSRB dataset. (a)&(c): triggers with the
Table 1:
square-like pattern; (d): triggers with the compass-like pattern; (b)&(b): triggers located in
the lower-right corner, (d)) triggers with random locations.
SRV->
Methody
Dataset
Trigger]
BP
GBP
GCAM
GGCAM
OCC
FA
LIME
(a)
0.306
0.434
0.011
0.329
0.437
0.543
0.002
(b)
0.075
0.131
0.034
0.174
0.314
0.137
0.018
CIFAR-10
(c)
0.248
0.344
0.014
0.308
0.386
0.394
0.006
(d)
0.065
0.271
0.058
0.221
0.164
0.184
Vanilla
(a)
0.519
0.428
0.000
0.373
0.466
0.640
0.006
(b)
0.190
0.154
0.043
0.175
0.505
0.263
0.063
GTSRB
(c)
0.328
0.341
0.061
0.340
0.306
0.657
0.010
(d)
0.233
0.207
0.102
0.210
0.198
0.087
(a)
0.212
0.212
0.000
0.207
0.885
0.494
0.979
(b)
0.266
0.266
0.001
0.268
0.537
0.385
0.991
CIFAR-10
(c)
0.346
0,346
0.000
0.348
0.928
0.579
0.964
(d)
0.243
0,243
0.005
0.247
0.703
0,566
0.988
Standardized
(a)
0,344
0,344
0.000
0,315
0.739
0.423
1.000
(b)
0.174
0.174
0.000
0.141
0.635
0.708
0.957
GTSRB
(c)
0.240
0.240
0.000
0.254
1.000
0.723
0.909
(d)
0.087
0.087
0.012
0.090
0.721
0.608
0.879
Table 2: The average rank (based the IOU) of SRV methods that is evaluated with the vanilla
backdoor-based method and its standardized version on CIFAR-10 and GTSRB datasets.
Method],
Dataset SRV-
BP
GBP
GCAM
GGCAM
OCC
FA
LIME
CIFAR-IO
5.25
3.00
6.25
3.25
1.50
2.50
6.25
Vanilla
GTSRB
2.75
3.75
6.50
3.75
2.50
2.25
6.50
CIFAR-I0
4.75
4.75
7.00
4.50
2.00
3.00
1,00
Standardized
GTSRB
4.50
4.50
7.00
5.00
2.00
2.75
1.25
6,05
8.099
6.025
0.04
0.180
8,000
0.07
8.015
0.19
0.050
8.000
0.18
0.02
0.02%
6.005
0.000
0.000
3/0
Chamfer Distance (x10')
Chamfer Distance (x101)
Chamfer Distance (x107)
Chamfer Distance (x10')
(a) CIFAR-10 (Corner)
(b) CIFAR-10 (Random)
(c) GTSRB (Corner)
(d) GTSRB (Random)
Figure 4: The distance between potential triggers and the original
used for training w.r.l. the
loss value on CIFAR-10 and GTSRB. The original trigger is a white square located at the lower-right corner (dubbed 'Corner') or a random position (dubbed 'Random') ) denoted by the 'red star'.
Results. As shown in Table I the vanilla backdoor-based method and standardized version have significantly different results. The vanilla method has various evaluations of the performance of the same SRV method on different datasets (especially for BP). In contrast, our standardized evaluation method has consistent rankings. These results verify that our evaluation is more faithful.
3.2 TRIGGER GENERALIZATION AND ITS INFLUENCES TO XAI EVALUATION
Closer Look to Location Generalization of Trigger Patterns. In this section, we generate 1,000 targeted universal adversarial perturbations (with random initializations) regarding the target label of models watermarked with the square-type trigger as potential trigger patterns. The watermarked DNNs are the same as those obtained in Section 3.1 To analyze the generalization properties of backdoor triggers, we visualize the distance between the potential trigger and the ground-truth one w.r.l. its loss value. Specifically, we exploit Chamfer distance (Borgefors) 1988) to cover the lo- cations of all trigger pixels. As shown Figure many potential trigger patterns are having large distance to the original one but with significantly lower loss values. In other words, the trigger pattern of existing watermarks has a high-level position generalization, although only the original trigger pattern is adopted. It is most probably due to the high con-exivyx and excessive learning capacities of DNNs. We will further explore its intrinsic mechanism in our future works.
Backdoor-based XAI Evaluation with Potential Trigger Patterns. In this part, evaluate whether the existing backdoor- based evaluation method has consistent rankings with potential trig- gers different from the original but still activate model backdoors. Specifically, adopted
5

---

Published as conference paper at ICLR 2024
Table 3: The IOU evaluated by the standardized backdoor-based method with synthesized triggers.
Dataset
Trigger]. SRV
BP
GBP
GCAM
GGCAM
OCC
FA
LIME
Pattern (a)
0.144
0.144
0.000
0.079
0.545
0.554
0.198
CIFAR-I0
Pattern (b)
0.128
0.128
0.000
0.014
0.346
0.285
0.310
Pattern (c)
0.007
0.007
0.037
0.008
0.064
0.065
0.070
Pattern (d)
0.195
0.195
0.000
0.130
0.691
0.702
0.315
GTSRB
Pattern (c)
0.224
0.224
0.000
0.107
0.562
0.573
0.342
Pattern (f)
0.198
0.198
0.000
0.126
0.601
0.447
0.310
Table 4: The average rank (based the IOU) of SRV methods that is evaluated by the standardized
backdoor- Maaded method with different triggers on the CIFAR-10 10 and the GTSRB datasets.
Dataset.
Trigger|, SRV->
BP
GBP
GCAM
GGCAM
OCC
FA
LIME
Original
4.75
4.75
7.00
4.50
2.00
3.00
1.00
CIFAR-10
Synthesized
4.67
4.67
6.00
5.67
2.00
2.00
2.00
Original
4,50
4.50
7.00
5.00
2.00
2.75
1.25
GTSRB
Synthesized
4.00
4.00
7.00
6.00
1.67
1.33
3.00
(a)
(b)
(d)
(e)
(I)
Figure The trigger patterns used for generalization discussions.
three synthesized triggers generated in the previous part on each dataset (as shown in Figure 55 and calculate the IOU of SRV methods with them. As shown in Table 3 the backdoor-base method has significantly different results across different triggers even on the same dataset (especially for GGCAM, FA and LIME). This phenomenon also indicates that its results
reliable.
4 THE PROPOSED METHOD
existing backdoor- based SRV evaluation may have unreliable results due to the trigger generalization of existing backdoor attacks. In this section, we discuss how to obtain the generalization-limit backdoor watermark to design a more faithful SRV evaluation.
As demonstrated in Section 3.2
4.1 PRELIMINARIES
Threat Model. In this paper, we assume that users (i.e., SRV appraisers) can fully control the training process of DNNs. They will try to implant backdoor triggers into the model, based on which to evaluate the performance of SRV methods via poisoned samples.
The Main Pipeline of Backdoor-based Model Watermark. Let = {(x, v:) denotes the be- nign training set, where æ; E X = {0, 1,..., 255 255]CxHX is the image, y) E (o...., K. K- }} its label, and K is the number of classes. How to generate the watermarked dataset D., is the cornerstone of the backdoor-based model watermark. Specifically. D., consists of two dis- joint parts, including the modified version of selected subset (i.e., D.) of and remaining benign samples, i.c., D.. Dn, U Dr, where Yi is a pre-defined target label, DD DVD., D,,, {(r,,, y:)] G(æ; w)) (xx, y) D.). the poisoned image generator with parameter w. For example, G(x) (1 m)) & x + m & t, where the mask m [0, 1 E X' is the trigger pattern, and @ is the element-wise product used in the backdoor-based SRV evaluation (Lin et al. (202T). Once the watermarked dataset D., is generated, users will train the watermarked DNN via ming E(EW)D ((f(x; 0). v).
[D_] the watermarking rale, and G X X [D]
4.2 AN INEFFECTIVE BASELINE; BACKDOOR WATERMARK WITH TRIGGER PENALTY
Arguably, the most straightforward method to synthesize the potential trigger pattern and penalize its effects based its distance to the original one in each iteration during the training process. In general, if the potential trigger pattern is close to the ground-trute one, the predictions of water- marked DNNs samples containing this pattern should be similar the target label; Otherwise,

---

Published as a conference paper at ICLR 2024
Torget Eabd Taatal Ropuepanenteers
Contigaratbor
Dutiwer
Mageiicyy Mijoet
Beeigg Sumple
Loss
Telsstad Trigurr Gemmser (0.40 -r fi - nojar
Soceion
DXN
Orrigiaal
Palatadd
Step Generating the Promising Synthesized Trigger
Step -.t.rrallzaionn-Liiieeee Backdoor Watermark
backdoor watermark (GLBW). In each iteration, we first generate promising synthesized patterns with the highest attack effectiveness and differences from the original trigger via adaptively adjusting the trade-off parameter μ.; After that, we update the DNN through the weighted combination of benign loss, backdoor loss, and generalization loss. We repeat these two steps until the generalization-limiee backdoor watermark implanted. their predictions should be similar to their ground-trule label. To achieve this goal, design penalty loss and call this method backdoor watermark with trigger penalty (WTP)) as follows.
Figure 6: The main pipeline of our generalization-
min Σ (m.yleD
C (2)
A C ((1- m) @ x m & 13))
+
Benign Loss
Backdoor Loss
(1)
da
max
L' (Jo ((1 m²) ( x --mmm @ 1') . 9).
m' 6 //mm
1ST
Penalty Loss
where 'D is the benign dataset, L and L' are the loss functions (e.g., cross-entropy and mutual information), is the original trigger pattern with mask m, and T' is the threshold of the size of potential trigger mask m'. D: (1 p(d(m, m'))) yy and ý = p(d(m,, m')) y. where y is the one-hot representation of y. d(-, is distance metric (e.g., chamfer distance), p(-) is probability activation function to map the distance a probability (the larger the distance, the higher
the probability). A and A2 are two trade-off hyper-parameters for backdoor loss and penalty loss.
Similar to the optimization strategies used in adversarial training (Madry et al. 2018 Zhang al.) 2019 Li et al. 2022d), we alternately optimize the inner maximization and the outer minimization in each iteration. Please find more optimization details in Appendix
4.3 GENERALFAATIN.IIIIITT BACKDOOR WATERMARK (GLBW)
As will show in the experiments and appendix, the previous BWTP suffers from relatively poor performance whose watermarked DNNs still have high trigger generalization in many cases. Ac- cording the intermediate results, this failure is mostly because the synthesized triggers are 'weak' with either a large loss value or are similar to the original trigger. Besides, the training of water- marked DNNs is unstable even when the synthesized triggers are promising. We argue that these limitations are due to the complexity of its penalty loss. As such, we propose to directly identify th potential trigger patterns that are more different from the ground-truth one and force them to be predicted as the ground-truthh label to further simplify the loss term, inspired by neural cleanse Wang et al. 2019). Its optimization process as follows.
min Σ (wD))ED
C (z)
+ As [(o ((1 m) @ x m 23 11) +
Benign Loss
Backdoor Loss
Aa
max
m1) ((') m' 1') ..)) p. [m'])
m' //m /m/
Generalization Loss
where 4 is a pprammer trading off its size and effectiveness for synthesized trigger, Ay & Aa two hyper-parameter" for losses, is the overlap threshold between synthesized and original triggers, and im'l is the size of potential trigger mask. Notice that the and y, contained in GLBW different from ÿ and ÿ, used in the previous BWTP since they are fundamentally different.

---

Published as a conference paper at ICLR 2024
Table 5: The effectiveness and generalization of model watermarks on CIFAR-10 10 and GTSRB. denotes that the trigger synthesis method can not find any potential trigger in this case.
Trigger Synthesis Method
Neural Cleanse
TABOR
Pixel Backdoor
Dataset
Metrie
BA
WSR
Chamfer
PLG
Chamfer
PLG
Chamfer
PLG
Watermark|
Vanilla
91.73
97.39
3979.33
42.20
4030.52
18.80
1974.68
10.30
CIFAR-10
BWTP
85.51
89.16
8070.46
0
8615.59
0.00
GLEW
78.33
93.68
23.04
100
27.92
83.33
30.16
100
Vamila
97.37
94.52
217.71
87.20
299.05
83,39
257.15
89.50
GTSRB
BWTP
88,36
91.61
21.83
100
20.16
100
25.71
100
GLBW
93.89
90.17
23.73
100
34.08
85.00
33.13
100
1.4
9.8
344
0.4
0.8
Chamfer Distance (x10r)
Chamfer Distance (x10')
Chamfer Distance (x10')
Chamfar Distance (=10
(a) CIFAR-IO 10 (BWTP)
(b) CIFAR-10 10 (GLBW)
(c) GTSRB (BWTP)
(d) GTSRB (GLBW)
Figure 7: The distance between potential trigger patterns and the original one used for training w.r.t.
the loss value on CIFAR-JU 10 and GTSRB. The original trigger pattern is denoted by a 'red star'.
In particular, we design an adaptive optimization method to find promising synthesized patterns with the highest attack effectiveness and differences from the original trigger. Specifically, we repeat the trigger generation and adaptively adjust the based on the current trigger candidate untill we find the promising synthesized trigger in each inner maximization. The main pipeline of method is shown in Figure 6 Please find more optimization details and its pseudocodes in Appendix E
EXPERIMENTS
5.1
MAIN SETTINGS
Baseline Selection. In this paper, we compare our generalization- backdoor watermark (GLBW) with vanilla backdoor watermark (dubbed 'Vanilla') (Lin al. (02I) and backdoor wa- termark with trigger penalty (BWTP) with standardized evaluation process.
Settings for Model Watermarks. Following settings in (Lin et al. 2021), we adopt the x white square located in the lower-right bottom as the original trigger on both CIFAR- 10 and GTSRB. Specifically, we simply set A A2 1 for BWTP and A3 A for our GLBW. We use the fo-nrrm as the distance measurement d and exploit a modified version of the Sigmoid function as the probability activation function p(-) used in BWTP. Please refer to Appendix for more details.
Settings for the Generation of Potential Triggers. In Section 3.2 we adopt neural cleanse (Wang) (ct al. 2019) to generate potential triggers. In our GLBW, we also exploit a similar approach during the generation of synthesized triggers. To evaluate whether method can truly decrease trigger generalization, we hereby adopt two other trigger synthesis methods, including TABOR (Guo al. 2020) and pixel backdoor (Tao al. 2022) generate potential triggers for all watermarked DNNs. We generate 1,000 trigger candidates (with random initializations) all cases. We only select those whose loss values less than 1.5 times of that of the original trigger (as shown in the red line in Figure Z as our potential trigger patterns. Please refer to Appendix | more details.
Evaluation Metrics. We adopt benign accuracy (BA) and watermark success rate (WSR) to evaluate the effectiveness of model watermarks. To evaluate their trigger generalization, we exploit chamfer distance (dubbed 'Chamfer') and the percentage of effective potential triggers with low generaliza- tion (dubbed 'LL''.. The effective triggers whose IOU values are greater than the threshold (0.3 for CIFAR-10 10 and 0.18 for GTSRB) are regarded as with low generalization. In general, the smaller the chamfer distance and the larger the PL.G, the lower the trigger generalization.
5.2
MAIN RESULTS
As shown in Table 5 both our BWTP and GLBW can obtaín a sufficiently high WSR (> 85%) on both datasets. In particular, (as shown in both Table (5 and Figure /7) For example, the chamfer distance (measured via neural cleanse and TABOR) of GLBW is more than 170 times smaller than that of the vanilla backdoor
GLBW can significantly reduce trigger generalization in all cases
8

---

Published as a conference paper at ICLR 2024
Table 6: The average rank (based on the IOU) of SRV methods that is evaluated with our
Ieneralization-lmiite backdoor watermark on CIFAR-10 and GTSRB datasets.
Dataset SRV-1
GBP
GCAM
GGCAM
OCC
FA
LIME
CIFAR-I0 4.50
4.50
7.00
5.00
2.00
3.00
1.00
GTSRB
4.00
4.00
6.00
6.00
2.00
3.00
1.00
Table 7:
Effects trigger size.
Table 8: Effects of the target label.
Size
BA (%)
WSR (%)
Chamfer
PLG (%)
Label
BA (%)
WSR (%)
Chamfer
PLG (%)
3 x 3
78.33
93.68
23.04
100
0
78.33
93.68
23.04
100
4 x 4
83.20
91.58
67.74
100
1
84.27
83.82
25.31
100
5 x
87.81
91.79
90.44
100
2
77.08
93.23
25.09
100
Chamfer Distance
25
Chamfer Distance
104
40
100
20
1023
30
80
PLG (%)
15
10
100
PLG
20
60
5
Chamfer Distance
PLG
98
96
10
Chamfer Distance
20
40
5
10
15
20
ResNet-16 18 Re NNet-334 VGG-13
VGG-16
Watermarking Rate (%)
Model
Figure Efffctt of the watermarking rate.
Figure 9; Effects of the Model Structure.
watermark on CIFAR-10. However, the BWTP may even increase the generalization on CIFAR-10, although it can reach remarkable performance on the GTSRB dataset. As we will show in Appendix it is mostly because its synthesized triggers are 'weak' with either a large loss value or are similar to the original trigger. These results demonstrate the effectiveness of our GLBW method.
G.3
notice that both BWTP and GLBW may decrease the benign accuracy to some extent, compared to the vanilla backdoor watermark. However, this mild potential limitation will not hinder the use- fulness (i.e., more faithful evaluation) of our methods, since the watermarked DNNs are only used for evaluating SRV methods instead of for deployment (more details are in Appendix(]. The average rank of SRV methods via our GLBW is shown in Table 6 As shown in this table, the results are consistent across datasets and therefore more faithful and reliable. Please find the IOU of SRV methods via three discussed model watermarks under different settings in Appendix G
5.3 THE EFFECTS OF KEY HYPER-PARAMETER
We hereby verify whether our GLBW is still effective under different settings on CIFAR-10. The chamfer distance and PLG are measured via neural cleanse. More results are in Appendix H
Effects of the Watermarking Rate 7,. As shown Figure g the watermarking rate has minor effects trigger generalization. In particular, our GLBW can reach high effectiveness (as shown in our appendix) and low generalization under different yy It verifies the our effectiveness again.
Effects of the Trigger Size. As shown in Table 7 increasing the trigger size has minor effects to watermark effectiveness, although there are some fluctuations for BA and WSR. We notice that the chamfer distance raises with the increase in trigger size. However, it is due the properties of this distance metric and does not mean an increase in trigger generalization. The PLG remains 100%.
Effects of the Target Label y/- As shown in Table our GLBW has promising performance under different ye- The trigger generalization remains low with a small chamfer distance and 100% PLG.
Effects of the Model Structure. As shown in Figure 9 our GLBW reaches similar generalization ability, especially with similar model architectures.
CONCLUSION
In this paper, we revisited the evaluation methods of saliency- based representation visualization (SRV). We revealed the unreliable nature (due the trigger generalization of backdoor watermarks) and implementation limitations of existing backdoor-based SRV evaluation methods. Based on these findings, we proposed a generalization-limite backdoor watermark, based which we designed a more faithful XAI evaluation. We hope our work can provide a deeper understanding of XAI evaluation, facilitate the design of more interpretable deep learning methods.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is supported in part by the National Natural Science Foundation of China under Grants (62302309, 62171248). Shenzhen Science and Technology Program under Grants 08110000000000001100000,,,,, JCY1200220818111112222 and the PCNL KEY Project (PCL2023AS6-I).
ETHICS STATEMENT
This paper is the first attempt toward measuring and reducing trigger generalization of backdoor wa- termarks and its positive application in XAI evaluation. Accordingly, our paper has positive societal impacts in general. However, notice that the adversaries may adopt backdoor watermark with trigger penalty (BWTP) or generalization-ling attacks when they can control the training process of DNNs. However, these attacks are less harmful compared to existing backdoor attacks because they will significantly reduce the benign accuracy of attacked models and therefore be noticed by victim model users. Besides, victim also exploit existing backdoor defenses, especially saliency-based methods, to detect or remove these backdoor attacks, Moreover, although we do design specified defense against BWTP or GLBW, people can still mitigate or even avoid the threats by only using trusted training resources.
backdoor watermark (GLBW) as backdoor
can
REFERENCES
Gunilla Borgefors. Hierarchical chamfer matching: A parametric edge matching algorithm. IEEE Transactions on pattern analysis and machine intelligence, 10(6):849-863 1988.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In COMPSTAT, 2010.
Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems. In IUI, 2020.
John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):679-698, 1986.
Ruoyu Chen, Hua Zhang, Siyuan Liang, Jingzhi Li, and Xiaochun Cao. Less more: Fewer
interpretable region via submodular subset selection. In ICLR, 2024.
Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modifi- cation. In NeurIPS, 2021.
Rudresh Dwivedi, Devam Dave, Het Naik, Smiti Singhal, Rana Omer, Pankesh Patel, Bin Qian, Zhenyu Wen, Tejal Shah, Graham Morgan, et al. Explainable (xai); Core ideas, techniques, and solutions. ACM Computing Surveys, 55(9):1-33, 2023.
Guanhao Gan, Yiming Li, Dongxian Wu, and Shu-Tao Xia. Towards robust model watermark via reducing parametric vulnerability. In ICCV, 2023.
Kuofeng Gao, Jiawang Bai, Bin Chen, Dongxian Wu, and Shu-Tao Xia. Backdoor attack on hash- based image retrieval via clean-label data poisoning. BMVC, 2023a.
Yinghua Gao, Yiming Li, Linghui Zhu, Dongxian Wu, Yong Jiang, and Shu-Tao Xia. Not all samples are born equal: Towards effective clean-label backdoor attacks. Pattern Recognition, 139: 109512, 2023b.
Xueluan Gong, Ziyao Wang, Yanjiao Chen, Meng Xue, Qian Wang. and Chao Shen. Kaleidoscope: Physical backdoor attacks against deep neural networks with rgb filters. IEEE Transactions on Dependable and Secure Computing, 2023.
Tianyu Gu, Kang Liu, Brendan Dolan-Gavite and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 0477230-7224 2019.
10

---

Published as a conference paper at ICLR 2024
Junfeng Guo, Yiming Li, Lixu Wang, Shu-Tao Xia, Heng Huang, Cong Liu, and Bo Li. Domain watermark: Effective and harmless dataset copyright protection is closed at hand. In NeurIPS, 2023.
Wenbo Guo, Lun Wang, Yan Xu, Xinyu Xing, Min Du, and Dawn Song. Towards inspecting and eliminating trojan backdoors in deep neural networks. In ICDM, 2020.
Wooseok Ha, Chandan Singh, Francois Lanusse, Srigokul Upadhyayula, and Bin Yu. Adaptive wavelet distillation from neural networks through interpretations. In NeurlPS, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretabil- ity methods in deep neural networks. NeurlPS, 2019.
Andrew Hyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurlPS, 2019.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICML, 2018.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
Chao Li, Kelu Yao, Jin Wang, Boyu Diao, Yongjun Xu, and Quanshi Zhang. Interpretable generative adversarial networks. AAAI, 2022a.
Jiawei Li. Yiming Li, Xingchun Xiang, Shu-Tao Xia, Siyi Dong, and Yun Cai. Tnt: An interpretable tree-network-tres learning framework using knowledge distillation. Entropy, 22(11):1203, 2020a.
Yiming Li, Jiawang Bai, Jiawei Li, Xue Yang, Yong Jiang, and Shu-Tao Xia. Rectified decision trees: exploring the landscape of interpretable and effective machine learning. arXiv preprint OrXiv:2008.09411 2020b.
Yiming Li, Tongqing Zhai, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor attack in the physical world. In ICLR Workshop, 2021a.
Yiming Li, Yang Bai, Yong Jiang, Yong Yang. Shu-Tao Xia, and Bo Li. Untargeted backdoor watermark: Towards harmless and stealthy dataset copyright protection. In NeurIPS, 2022b.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learningg A survey. IEEE Transac- tions on Neural Networks and Learning Systems, 2022c.
Yiming Li, Baoyuan Wu, Yan Feng, Yanbo Fan, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Semi- supervised robust training with generalized perturbed neighborhood. Pattern Recognition, 124: 108472, 2022d.
Yiming Li, Haoxiang Zhong, Xingjun Ma, Yong Jiang, and Shu-Tao Xia. Few-shot backdoor attacks on visual object tracking. In ICLR, 2022e.
Yiming Li, Mengxi Ya, Yang Bai, Yong Jiang, and Shu-Tao Xia. BackdoorBox: python toolbox for backdoor learning. In ICLR Workshop, 2023a.
Yiming Li, Mingyan Zhu, Xue Yang, Yong Jiang, Tao Wei, and Shu-Tao Xia. Black-box dataset ownership verification via backdoor watermarking, IEEE Transactions on Information Forensics and Security, 2023b.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV, 2021b.
Yi-Shan Lin, Wen-Chuan Lee, and Z Berkay Celik. What do you see? evaluation of explainable artificial intelligence (xai) interpretability through neural backdoors. In KDD, 2021.

---

Published as a conference paper at ICLR 2024
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Richard Meyes, Melanie Lu, Constantin Waubert de Puiseau, and Tobias Meisen. Ablation studies in artificial neural networks. arXiv preprint 18XXv:9901.086444 2019.
Anh Nguyen and Anh Tran. Wanet-impercepibbee warping-based backdoor attack. In ICLR, 2021.
Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE transactions on systems, man, and cybernetics, 9(1):62-66, 1979.
Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the as- sumption of latent separability for backdoor defenses. In ICLR, 2023.
Ximing Qiao, Yukun Yang, and Hai Li. Defending neural backdoors via generative distribution modeling. In NeurIPS, 2019.
Haibo Qiu, Baosheng Yu, Dihong Gong, Zhifeng Li, Wei Liu, and Dacheng Tao. Synface: Face recognition with synthetic data. ICCV, 2021.
Sukrut Rao, Moritz Böhle, and Bernt Schiele. Towards better understanding attribution methods. In CVPR, 2022.
Marco Tulio Ribeiro. Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In KDD, 2016.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and RoburR Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions neural networks and learning systems, 28(1 (111):2660-26777 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. ICCV, 2017.
Shawn Shan, Emily Wenger, Bolun Wang, Bo Li, Haitao Zheng, and Ben Y Zhao. Gotta catch em all: Using honeypots catch adversarial attacks neural networks. In CCS, 2020.
J Springenberg, Alexey Dosovitskiy, Thomas Brox, and M Riedmiller. Striving for simplicity: The all convolutional net. In ICLR Workshop, 2015.
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Bench- marking machine learning algorithms for traffic sign recognition. Neural networks, 32:323-333 2012.
Xiaoou Tang and Zhifeng Li. Frame synchronization and multi-level subspace analysis for video based face recognition. In CVPR, 2004.
Guanhong Tao, Guangyu Shen, Yingqi Liu, Shengwei An, Qiuling Xu, Shiqing Ma. Pan Li. and Xiangyu Zhang. Better trigger inversion optimization backdoor scanning. In CVPR, 2022.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint 192 2019.
Jasper van der Waa, Elisabeth Nieuwburg, Anita Cremers, and Mark Necrinex. Evaluating xai: A comparison of rule-based and example- based explanations. Artificial Intelligence, 291:103404, 2021.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks neural networks. In IEEE S&P, 2019.
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In CVPR Workshop, 2020.
12

---

Published as a conference paper at ICLR 2024
Emily Wenger, Roma Bhattacharjee, Arjun Nitin Bhagoji, Josephine Passananti, Emilio Andere, Heather Zheng, and Ben Zhao. Finding naturally occurring physical backdoors in image datasets. In NeurIPS, 2022.
Jiahao Xia, Min Xu, Haimin Zhang, Jianguo Zhang, Wenjian Huang, Hu Cao, and Shiping Wen. Ro- bust face alignment via inherent relation learning and uncertainty estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023.
Liyao Xiang, Haotian Ma, Hao Zhang, Yifan Zhang, Jie Ren, and Quanshi Zhang. Interpretable complex-valued neural networks for privacy protection. In ICLR, 2020.
Chih-Kuan Yeh, Cheng-Yo Yu Hsieh, Arun Suggala, David I Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity explanations. In NeurlPS, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. networs.. In ECCV, 2014.
Yi Zeng, Minzhou Pan, Hoang Anh Just, Lingjuan Lyu, Meikang Qiu, and Ruoxi Jia. Narcissus: practical clean-label backdoor attack with limited information. In CCS, 2023.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
Quanshi Zhang. Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In CVPR, 2018
13

---

Published as a conference paper at ICLR 2024
a
(b)
Figure 10: The watermark patterns used for standardized backdoor- based XAI evaluation.
APPENDIX
A SETTINGS FOR STANDARDIZE BACKDOOR-BASED XAI EVALUATION
Datasets and Models. We conduct experiments on two classical datasets, including CIFAR-10
Krizhevsky 2009) and GTSRB (Stallkamp et al. 2012), with ResNet-18 (He et al. 2016).
Watermark Setup. We adopt two watermark patterns: the square-like pattern (as shown in Figure Iθ(a)) and the compass-like pattern (as shown in Figure [0(b).. Based on these patterns, we generate four kinds of triggers: (a), (b), (c), and (d). Specifically, (a) and (c) are the triggers with the square- like pattern; (b) and (d) are triggers with the compass-like pattern; (a) and (b) are the triggers located in the lower-right corner; (c) and (d) are the triggers with random locations.
Training Setup. We adopt BadNets (Gu et al., 2019) to train our backdoored models based on the open-sourced Python toolbox-Bat using four triggers (i.e., (a), (b), (c), and (d)) CIFAR- 10 and GTSRB. We set the watermarking rate as 5% and the target label y, set for all backdoor training. To train our backdoored models, the mini-batch is set to 128 for each iteration. We use the SGD optimizer by setting momentum=0.9 and weight 5ecay=5544 For CIFAR- 10, the number of the training epochs 200 and the learning rate is initialized 0.1 and multiplied by 0.1 at the 150 and 180 epoch, respectively. For GTSRB, the number of the training epochs is 30 and the learning rate is initialized as 0.01 and multiplied by 0.1 the 20th epoch. In particular, add watermarks after performing the data augmentation.
Box (Li et al. 2023a). We train backdoored models
Computational Facilities. We train each model with one NVIDIA RTX3090 GPU. For CIFAR-10, it costs about 40 minutes for training. For GTSRB, it costs about 15 minutes for training.
B SETTINGS FOR TRIGGER GENERALIZATION AND ITS INFLUENCES TO XAI EVALUATION
We select the four models watermarked with the triggers (a) and (c) on CIFAR-10 and GTSRB used in Section A for discussing trigger generalization. We generate 1,000 targeted universal adversarial perturbations (Wang et al. 2019) (with random initializations) regarding the target label. In partic- ular, we used the neural cleanse (Wang et al.) 2019) to generate all synthesized triggers, where the weight of the regularization term is set x 10 The training dataset is used the set of clean images for neural cleanse. We generate each synthesized trigger with 10 epochs.
C WHY Do WE (ONLY) NEED TO CONSIDER PATCH-BASHE TRIGGERS?
People may worry about whether method can truly lead more faithful evaluation since we only focus on patch-based attacks (e.g.. BadNets) and there are still many other backdoor attacks with different trigger patterns (Li et al.) 2021b; Doan et al. 2021 Zeng et al. 2023), while people may also use complicated features in recognizing real images in practice.
Firstly, patch-based poisoned-label backdoor watermark is the most suitable and probably the only suitable method for backdoor-basse SRV evaluation. Specifically, clean-label attacks where the
adversaries only poison samples for the target class (Turner et al. 2019 Gao al. 2023b
Zeng
14

---

Published as a conference paper at ICLR 2024
Table 9: The effectiveness and generalization of backdoor watermarks CIFAR-10 with different
trigger types. The best results are marked in boldface.
Type4
Method,, Metric
BA
WSR
Chamfer
PLG
Vanilla
97.55
91.79
61.59
94.60
Blended
GLBW
90.10
88.03
9.81
100
Vanilla
97.60
95.01
66.47
95.00
Additive
GLBW
94.93
91.92
26.42
100
et al. 2023)
not suitable since attacked DNNs may both "ground-truth" features and trigger features instead of just trigger features for predictions. Besides, the triggers of almost all existing non-patch- based attacks are with full image-size. We certainly don't want saliency ares are the whole image. Accordingly, these attacks are also not suitable for SRV evaluation.
Secondly, people categorize given image usually based its local regions instead of the whole image. For example, we only need to see the image areas of its 'head' to know it is bird. These local regions are similar to the trigger patch used in our evaluation. Accordingly. results made by our method can be good reference in real images for practice. This is probably the main reason why our baseline research (Lin al. 202T evaluated their method only with trigger patches.
Thirdly, it is impossible to faithfully evaluate the performance of SRV methods for complicated features (instead of simple trigger patches) since there are no ground-trus salience maps. Even human expert cannot accurately mark the salience map for complicated features in most cases.
Fourthly, simple trigger patches are also features used by DNNs for their predictions. Accordingly, the evaluation of SRV methods on trigger features is the first and the most important step toward evaluating their general performance and is therefore of great significance.
In particular, may argue that there are still two other patch-based triggers, including 'Blended' (i.e., BadNets with trigger transparency) and patch-size additive trigger (dubbed 'Additive') ) that we did not evaluated in main experiments. Accordingly, on the CIFAR-10 dataset with them. As shown in Table 9 our GLBW still highly effective.
hereby conduct additional experiments
D THE OPTIMIZATION PROCESS OF BWTP
Arguably, the most straightforward method is synthesize the potential trigger pattern and penalize its effects based on its distance to the original one in each iteration during the training process. In general, if the potential trigger pattern is close the ground-truth one. the predictions of water- marked DNNs to samples containing this pattern should be similar to the target label; Otherwise, their predictions should be similar to their ground-truth label. In this paper, call this method as backdoor watermark with trigger penalty (BWTP), whose optimization process is as follows:
min Σ EDD
L (x)
+ A L(fo ((1 - m) & x m tt)
Benign Loss
Backdoor Loss
(1)
A2
max
(So ((1 m') x +m' @ t0),,, Dr + 0)
ml H.Imm
Penalty Loss
where D the benign dataset, L and L' are the loss functions (e.g., cross-entropy and mutual information), is the original trigger pattern with mask m, and T is the threshold of the size of potential trigger mask m'. ÿ: = (1 p(d(m, m')) yt and ÿ = p(d(m (m, (dd(m,'))) y. where y is the one-hot representation of y, d(-,.) is distance metric (c.g., chamfer distance), p(-) is probability activation function to map the distance to a probability (the larger the distance, the higher the probability). X and X2 are two trade-off hyper-parameter for backdoor loss and penalty loss.
15

---

Published as a conference paper at ICLR 2024
Similar to adversarial training (Marry al. 2018). alternately optimize the inner maximization and the outer minimization in each iteration. For the inner maximization, we solve the following formula to get penalized trigger (with mask m'' and pattern t'):
max /
L' (fee ((1 - (1) @ + m' @ t0) +
(2)
For convenience, this formula can be written in an equivalent form:
min
L' (Jo ((1 m') & m t'), + ÿ)
m m 1.
After that, this conditional minimization problem can be approximated as non-conditional surro gate problem, as follow:
min I' (fo ((1
(1-mm)) m') mm' t'), U: Ú) y max(0, |m'| -T]
(4)
where 7 is the weight of the regularization term used for limiting the size of m'.
We proposed an algorithm (as shown Algorithm I to solve the aforementioned problem 4 Firstly, we randomly initialize m and to by sampling from the standard normal distribution. Sec- ondly, we iteratively optimize the problem (4) on a benign dataset D. In particular, we map the values of m and t. to [0, 1] by modification of tanh function in each iteration. After optimizing for several epochs, the final penalized trigger mask m' and pattern t' are obtained.
For the outer minimization, we adopt the penalized trigger mask and pattern generated from Algo- rithm D as m' and After that, we solve the following trying to reduce the trigger generalization of the watermarked DNN during its training process:
min Σ (xD))DD
C (f)
d1 CCo m x .))
Benign Loss
Backdoor Loss
(5)
d C' (So ((1 m') @ m' @ t') ú1 +
Penalty Loss
We solve the aforementioned problem ( with the standard SGD (Bottou 2010 as shown in Algo- After several epochs, the final watermarked model fo with BWTP is obtained.
rithm 2
E THE OPTIMIZATION PROCESS OF GLBW
As we show in the experiments, the previous BWTP suffers from relatively poor performance whose watermarked DNNs still have high trigger generalization in some cases. According to the intermedi- ate results, this failure is mostly because the synthesized triggers are 'weak' with either large loss value or are similar to the original trigger. Besides, the training watermarked DNNs unstable when the synthesized triggers promising. We argue that these limitations are due to the complexity of its penalty loss. As such, we propose to directly target the generation of synthesized triggers to make this process more controllable, inspired by neural cleanse which is backdoor defense based on trigger synthesis (Wang et al. 2019). Its optimization is as follows:
min Σ
((f (z),))
+
Ag L C(fo ((1 m)
X t) Be.
ED
Benign Loss
Backdoor Loss
(6)
Aa-
max
(Jo + m' ft), (x)= + + [mmm]
m' H. /// mmmmmm /m/
Generalization Loss
where a is a parameer trading off its size and effectiveness for synthesized trigger, X & A two hyper-parameters for losses, is the overlap threshold between synthesized and original
16

---

Published as a conference paper at ICLR 2024
Algorithm 1 Generating the penalized trigger.
The benign dataset D; The current model fe: Target label yss The mask and pattern t of the original trigger; Hyper-parameten 7 and y; The number of epochs for generating the penalized trigger N gen:
Input:
Output: The mask m' and pattern t' of the penalized trigger;
1: function GET WWT..PEAAIIEDD.RRIGERR... fo. yr, m. t. T, Y, ggen)
2: m. ~ (0)
3: to (0,
4: for n = to N Ngen do
5:
for each mini-batch (a: v) 'D do
6:
m' + mnn.(m,444
Map the values of m to [0.
t' - 1n
7:
> Map the values of t, to [0, 1]
9m, V ms [- m') @ m' t¹), Gr ÿ) + + max(0, |m'l - T)]
8:
9:
Dr. Ve, (f
m')
m t.), Vr ú) y max(0, |m'] - T)]
10:
Use Adam optimizer to update m and t with gradients Im, and gr;
11: end for
12: end for
13:
return mn'
14:
end function
Algorithm/ Watermarking DNNs with BWTP
Input: The benign dataset D; Model so: Target label The mask m and pattern of the original trigger; Hyper-parameters A A2, T. y; The number of the training epochs Ntrain and the number of epochs for generating the penalized trigger Ngens
Output: wwte-wwwtrmarked DNN fe:
1: function BWTP(D, so- yt. m, f, A1, A2, T, 7/, Nerain Ngen)
2: for <- to Nirain do
3: m', + GET PENALTY.TREE TRIGGER (D, Vr, m, T, 1,, Ngen)
4:
for each mini-batch (x: y) 'D do
5:
Lbenign +- L(fe(()f)))))) (z)
6:
Loackdoor L(fo
m
x &
7:
Epenalty x- ['o[([(1 m') @ m' @ tt), ÿt +
8:
go 4- Vo (Lbnnign + X Loockdoor + X2 Epenalty)
9:
Use SGD optimizer to update @ with gradient go
10:
end for
11:
end for
12: return fe
end function
triggers. Notice that the y and Ye contained in GLBW different from ý and ÿ, used in the previous BWTP since these two methods are fundamentally different.
To solve the aforementioned problem (6), alternately optimize the inner maximization and the outer minimization in each iteration. For the inner maximization, we solve the following maximiza- tion get a promising synthesized trigger:
max m' Sml/r:mmm
-
[C (ffe ((1 - (1-m') & x + m @ t0),,)) + μ [mmmm]]]-
For convenience, this problem can be written in an equivalent form:
min Jww/m//mmm |m| (Jo ((1 (1) 8 + m' 6 t'), w.) + n |m']]
(8)
In particular, we design an adaptive optimization algorithm (as shown in Algorithm to solve the aforementioned problem (8): At the beginning, initialize inét_flag False and to pa- init. flag is flag used to identily whether to force m' to be different from m by the special initial- ization of m p is adaptive parameter limit the size of the synthesized trigger mask m'. After
17

---

This is to Published as a conference paper at ICLR 2024
Algorithm 3 Generating the desired synthesized trigger.
Input:
The benign dataset D; The current model fe: Target label ys; The mask and trigger
pattern & of the original trigger; Syyper-parmmete Aoo and r; The number of epochs for generating the potential trigger N genn
Output: The mask m and its pattern t' of the promising synthesized trigger;
1: function GET WEET.PROMSINNGTRIGGERRR fo. yt, m. t. po. T, N Ngen)
2: init flag t False
3: Jr 4- 140
4: loop
5:
m' N(0
6:
to
~ (0
7:
if init Winit_fla flag- True then m t oo where mijj 1
8:
9:
end if
for n N gen do for each mini-batch x E D do
11:
12:
m' - u
> Map the values of m' to [0, []
t' - wnnh(e()+11
13:
Map the values of E. to [0, 1]
9m', Vn, [C(So((1 m') & + m' @ (t),b)) + 1 |m']]
15:
gr. < Ve, [[[o((] m') m' @ t.), + - |mml]]
16:
Use Adam optimizer to update m', and ty with gradients 9m and gr,
17:
end for end for
18:
19:
lossm
MED [(fo((1 m) & x m @ t) v6)
[
20:
lossm'
ExED [(fo(( m') & x m' @ tt),3))
[D]
21:
if m
m 2 then
22:
μ + p
|m'l
Adjust μ adaptively
mm
23:
continue
24:
else if |mll mm|
0.6 and loss > lossm then Adjust adaptively
25:
t p.. 0.618
26:
continue
27:
else if lossm' continue
1.8 lossm or lossm' + ju |mm| >> 1.5 (lossm + μ |ml) then We get a bad trigger candidate and run the algorithm again to find
28:
better one.
else mm' 0 ml > T- |m| then
30:
init. flag f- Truc
continue We get trigger candidate that is too similar to the original trigger m and run the algorithm again to find a different one.
32:
end
33:
return m',t'
34:
end loop
35:
end function
initializing flat.flag and po. the program enters loop to generate potential synthesized triggers adaptively. In the loop, the first step is randomly initializing m and t, by sampling from the stan- dard normal distribution. If init_flag is True, we set m/s.; to oo where m,,
force m'' to be different from m (that is m mm = 0). After that, we iteratively optimize the the problem (8) the benign dataset D. After optimizing for several epochs, get trigger candi- date mask m and its corresponding pattern t'. Then, we calculate the loss the original trigger mask with target label y, (dubbed as lossm) and the loss of the candidate trigger mask m' with target label ys (dubbed as lossm'). After that, we will check the quality of the trigger candidate. The first check whether the size of candidate trigger mask mn is too big. If |m'] > 2 |m],
[ml] we increase by multiplying [m] to limit the size of m' and then continue the loop to generate
another trigger candidate; The second check is whether the size of candidate trigger mask m' is too
18

---

Published as a conference paper at ICLR 2024
Algorithm 4 Watermarking DNNs with GLBW.
The benign dataset D; The model fe: Target label yri The mask m and pattern t of the original trigger;
Input:
λ3 A4. (Au. т; The number of the training epochs Ntrain and the number of epochs for generating the penalized trigger Ngen; Output: A model with generalization-liman
backdoor watermark fø:
1: function GLBW(D, for Vt, m, As, Aa, Mo, T, Nerains N gen))
2: for n2 <- to Nerain do
3: m', E' <- GET PROMISING TRIGGER So, Ut, m, p0,,7 T, Ngen)
4:
for each mini-batch (x. E "D do
5:
Lbenign &- L(fo( (1), .)) m 8
6:
Lbackdoor & C(fe (
7:
Egeneralization
[(fo((1 m
m v)
8:
g0 5 Vo (Lbenign + 03 Lbackdoor + X Egeneratization)
9:
Use SGD optimizer to update 0 with gradient go
10: end for
11: end for
12: return fe
13: end function
small. If |mm| < 0.6 |m] and lossm' > l0ssm: that is the trigger mask is too small and with large loss, we decrease y by multiplying 0.618 to relax the size of m' and then continue the loop to generate another trigger candidate; The third check is whether the candidate trigger is effective. If lossm' > 1.8 lossm or lossm + p |m') > 1.5 (lossm + μ. |m]). we think the trigger candidate too bad and cannot effectively activate the backdoor, then continue the loop to generate another better trigger candidate; The fourth check whether the candidate trigger is similar to the original trigger. If m' nm]
/m], get trigger candidate that is too similar to the original trigger m, we set flag to True and run the a algorithm again to find different one. If the trigger candidate passes all four checks, it a promising synthesized trigger and we mark its mask m and pattern t'.
For the outer minimization, we use the promising synthesized trigger mask and pattern generated from Algorithm as mn' and t'. After that, we solve the following problem to train generalization- limited watermarked model:
min
C (ff (z),
+ As C(f ((1 m) & m % v))
(D
Benign Loss
Backdoor Loss
(9)
As L (fo ((1 m') 8 8 x m' @
Mt). vt),
Generalization Loss
We solve the aforementioned problem ( with the standard SGD (Bottou| 2010 as shown in Algo- rithm After several epochs, the final GLB watterrrtrdr model fe obtained.
F DETAILED SETTINGS FOR MAIN EXPERIMENTS
General Settings for Backdoor Watermarks. We adopt the trigger (a) used Section A as the original trigger pattern for all model watermarks on both CIFAR-10 and GTSRB. The watermarking rate and the target label are also the same as those used in Section
Settings for Vanilla Backdoor Watermark. The settings are the same those used in Section
Settings for BWT.. We set the the threshold of the size of potential trigger mask T' as 18, the number of epochs for generating each potential trigger Ngen as 10, A >2 measurement d(m mn' = m m'|. In particular, we use modified version of the Sigmoid function the probability activation function p(-) (as shown in Figure IIp. When di d(m, m') is very large, the probability p(d(m (,, m')) se and the penalty label ÿ. + ÿ = y: When d(m, m') is very small, the probability activation p(d(m, mmm))) and the penalty label ú. yt-
= 1, and the distance
19

---

Published as a conference paper at ICLR 2024
1.0
0.8
Probability
0.6
0.4
0.2-
0.0
8 Distance between m and m'
2
4
Figure 11: The graph of the probability activation function p(-)
Settings for GLBW. We set the hyper- A A 1, p4 0.001, T 0.05 and the number of epochs for generating each potential trigger Ngen as 15.
General Settings for Trigger Synthesis. We adopt three trigger synthesis methods, including neu- ral cleanse (Wang et al.] 2019). TABOR (Guo et al. 2020), and pixel backdoor (Tao et al., (022) to generate potential triggers of watermarked DNNs. We generate 1,000 trigger candidates (with random initializations) in all cases. We only select those whose loss values are less than 1.5 times of that of the original trigger as our potential triggers.
Evaluation Metrics. Following the most classical settings, adopt benign accuracy (BA) and watermark success rate (WSR) to evaluate the effectiveness of model watermarks. To evaluate exploit chamfer distance (dubbed 'Chamfer') and the percentage
their trigger generalization, of effective potential triggers with low generalization (dubbed 'PLG'). The effective triggers whose IOU values are greater than the threshold (0.3 for CIFAR- 10 and 0.18 for GTSRB) regarded as with low generalization. In general, the smaller the chamfer distance and the larger the PLG, the lower the position generalization of watermark.
The Visualization of Different Types of Trigger Candidates and The Definition of PLG. We visualize different trigger candidates for better understanding (as shown in Figure [2]. Triggers in the first line are effective candidates with high IOU value (i.e., effective triggers with low gener- alization). These triggers have high watermark success rate (WSR) and low backdoor loss that can activate the backdoor of watermarked DNNs successfully; Triggers in the second line are effective candidates with a low IOU value (i.e., effective triggers with high generalization). These potential triggers have high WSR and low backdoor loss that can activate the backdoor of watermarked DNNs successfully, whereas they are different from the original trigger pattern; Triggers the third line are ineffective trigger candidates with low WSR and high backdoor loss that can not activate model backdoors. Accordingly, they should not be regarded as potential triggers but targeted universal ad- versarial perturbations. We do not include them in the evaluation of trigger generalization. As such, the caculation of PLG can be written as follows:
PLG the number of the effective triggers with high IOU
the number of all effective triggers
Settings for Neurl Cleanse. We the number of training epochs 10 for the vanilla watermark and for others. On the CIFAR- 10 dataset, we set the weight of the regularization term À = 8 10-5 for the vanilla watermark,
10 for BWTP, and À 9.315 10-2 2 for GLBW. On the GTSRB dataset, we set À 8 x 10-5 for the vanilla watermark, X 9 10-2 for BWTP, and 9 10-2 2 for GLBW. We adopt different settings to achieve the best synthesis performance.
Settings for TABOR. We set the number of training epochs 2. On the CIFAR-10 dataset, we set the weights of the TABOR regularization terms '₁ x 10-4 A = 10-4, A 10 for the vanilla watermark, \ 5 10 2 X2 5 10 -3 À 10 for BWTP, and \ 7.5 2
7.5 x 10 10-3, = 10 55 for GLBW. On the GTSRB dataset, we set \ς =8 10 -4 A2
10 10-4,x
10-5 for the vanilla watermark, Å 8 x 10- x 8 10-3.
10 for BWTP, and
\ = 7.2 x 10-2, 20- = 7.2 x 10 3 = 10 - for GLBW. We adopt different settings in different
cases achieve the best synthesis performance.
20

---

Published as a conference paper at ICLR 2024
Effective Triggers with High IOU
IOU 1.000
I0U 0,455
IOU 0.353
IOU = 0.354
WSR 90.17
WSR 92.47
WSR 93.57
WSR 93.79
Los8-0.318
Loss-0.256
Loss-0.161
Loss-0.157
Effective Triggers with Low IOU
IOU 0.110
IOU 0.130
IOU 0.131
IOU 0.135
WSR 84.84
WSR 91.30
WSR 90.74
WSR 90.85
Lox550.445
Loss-0.321
0.332
Los5=0.325
Ineffective Triggers
IOU
IOU =
=
IOU
WSR 58.08
WSR 57.30
WSR 58.55 Loss=1.093
WSR 60.29 Loss-1.088
Lo55-1.115
Los6-1.106
Figure 12:
The example of synthesized trigger candidates generated by TABOR towards models watermarked by GLBW on the GTSRB dataset.
The effectiveness and generalization of model watermarks on CIFAR-10 and GTSRB datasets. "# All' indicates the number of all triggers while ## p' denotes that of potential triggers. denotes that the trigger synthesis method can not find any potential trigger this case.
Table 10:
Trigger Synthesis Method
Neural Cleanse
TABOR
Pixel Backdoor
Dataset
Metric
Watermark
BA
WSR
All
#
f--norm
#P
f--nrrm
# #P
e--orrm
Vanilla
91.73
97.39
1000
1000
20.61
1000
21.17
1000
14.912
CIFAR-1O
BWTP
85.51
89.16
1000
424
25.08
314
26.83
GLBW
78.33
93.68
1000
63
6.20
12
6.71
6,434
Vanilla
97.57
94.32
1000
1000
9.48
638
9,80
1000
10.113
GTSRB
BWTP
88.36
91.61
1000
38
5.79
5.47
11
6,673
GLBW
93.89
90.17
1000
428
7.43
100
9.39
86
8.451
Settings for Pixel Backdoor. We set the number of training epochs as 10. On the CIFAR-10 dataset, we set the weight of the regularization term a 5 x 10 for the vanilla watermark, a = 0.5 for BWTP, and a 2 for GLBW. On the GTSRB dataset, we set a 5 10-4 for the vanilla watermark, = 0.7 for BWTP, and a 1 for GLBW. We adopt different settings in different cases to achieve the best synthesis performance towards all watermarked models.
G ADDITIONAL RESULTS OF MAIN EXPERIMENTS
In this section, we provide additional results that are omitted in the main manuscript.
G.1 THE GENERALIZATION OF MODEL. WATERMARKS
In the main manuscript, we adopt chamfer distance and PLG to measure the trigger generalization. In this section, we provide the results under ( -norm. Besides, we provide the number all synthesized triggers (dubbed ## All') and potential triggers (dubbed '# P').
As shown Table I0 our GLBW can significantly reduce the trigger generalization compared to the vanilla backdoor watermark in all cases whereas BWTP may even increase in some cases.
21

---

Published as a conference paper at ICLR 2024
Table 11:
The evaluation (IOU) of SRV methods based on ceneralization-liitte backdoor wa-
termark with the standardized process on the CIFAR-10 dataset and the GTSRB dataset. (a)&(c);
triggers with the square-like pattern; (b)& triggers with the compass-like pattern: (b){ trig- gers located in the lower-right corner; (c)& triggers with random locations.
SRV-
Dataset.
BP
GBP
GCAM
GGCAM
OCC
FA
LIME
Trigger
(a)
0.143
0.143
0.000
0.003
0.603
0.543
0.990
(b)
0.148
0.148
0.045
0.215
0.621
0.583
0.997
CIFAR-10
(c)
0.116
0.116
0.000
0.138
0.927
0.652
0.964
(d)
0.260
0.260
0.000
0.134
0.679
0.589
0.994
(a)
0.169
0.169
0.009
0.009
0.938
0.901
0.998
(b)
0.247
0.247
0.040
0.040
0.693
0.685
0.985
GTSRB
(c)
0.082
0.082
0.009
0.009
0.964
0.495
0.985
(d)
0.096
0.096
0.040
0.040
0.666
0.520
0.952
Backdoor Loss
5
3
2
10
20
30
40
50
Epoch
Figure 13: The backdoor loss of the synthesized trigger w.r.t. training epoch of BWTP on CIFAR-
10. The red dash line denotes the loss of the original trigger.
Figure 14: The effective synthesized triggers generated by BWTP on the CIFAR-10 10 dataset.
G.2 THE IOU OF SRV METHODS WITH OUR GLBW
The IOU of SRV methods calculated based on our GLBW shown in Table II The results show that our GLBW can lead to more consistent and stable evaluation and therefore is more faithful.
G.3 A CLOSER LOOK TO THE FAILURE OF BWTP
In this section, we explore why our BWTP method may lead to poor performance on CIFAR-10 by visualizing the training process of BWTP. Specifically, present the backdoor loss of each synthesized trigger with respect to the training epoch on the CIFAR- dataset.
As shown in Figure 13 the loss values of synthesized triggers are significantly larger than that of the original trigger. In other words, most them are ineffective and cannot activate the backdoor in the watermarked model. Even among the few valid synthesized triggers, there is still a trigger pattern that is similar to the original trigger (as shown in Figure I4). Accordingly, penalizing these synthesized triggers has minor benefit reduce trigger generalization.
22

---

Published as a conference paper at ICLR 2024
Table 12: Effects of the hyper-parameter Ag.
Table 13: Effects of the hyper--aramen Ag-
Aa
BA (%)
WSR (%6
Chamfer
PLG (96)
A
BA (%)
WSR (%)
Chamfer
PLG (%)
0.6
78.35
93.15
23.13
100
0.6
74.57
92.56
24.68
100
0.8
81.49
92.45
17.38
100
0.8
81.16
87.71
26.37
100
1.0
78.33
93.68
23.04
100
1.0
78.33
93.68
23.04
100
1.2
82.63
93.65
24.28
100
1.2
75.90
88.05
24.03
100
1.4
79.85
89.88
32.99
100
1.4
80.74
87.30
24.02
100
Table 14: Effects of the hyper- apper-parameter Pg.
Pg
BA (%)
WSR (%)
Chamfer
PLG (%)
0.0006
84.17
87.53
21.50
100
0.0008
81.18
89.70
23.47
100
0.0010
78.33
93.68
23.04
100
0.0012
82.78
90.55
17.46
100
0.0014
81.76
84.80
29.49
100
H ADDITIONAL RESULTS OF THE EFFECTS OF KEY HYPER-PARAMETERS
In this section, similar to the main manuscript, we adopt the CIFAR-10 dataset as an example to discuss the effects of other key hyper-parameters involved in our GLBW.
H.1 EFFECTS OF THE HYPER- PARAMETER A
As shown in Table 12 the hyper-parameter A3 has mild influence on our GLBW, especially on its trigger generalization. Specifically, method can reach 100% PLG under all settings. In other words, users can achieve faithful SRV evaluation based on our GLBW without fine-tuning λ3.
H.2 EFFECTS OF THE
A1
As shown in Table 13 the hyper-parameter X4 also has mild influence our GLBW, although has higher impacts on WSR. This is mostly because \ relates to the 'suppression' of potential triggers that may have adverse effects on the WSR. Nevertheless, our method can reach a 100% PLG under all settings. Accordingly, users can achieve faithful SRV evaluation based on our GLBW without fine-tuning this hyper-paramette
H.3 EFFECTS THE HYPER-PARAMETE PARAMETER 140
1l also has mild effects to trigger the generalization of our GLBW. It is mostly because of the adaptive optimization process of generating synthesized triggers. Users can still achieve faithful SRV evaluation based our GLBW without fine-tuning this hyper-parameter.
As shown in Table 14
I A CLOSER LOOK TO THE EFFECTIVENESS OF OUR GLBW
In this section, we further explore why our GLBW can reduce trigger generalization by visualizing all synthesized triggers via principal component analysis (PCA).
Settings. We use the vanilla, BWTP and GLBW models trained in Section 5.2 for experiments. Similar to (Qiao al. 2019), for each model, we sample 1,000 triggers produced by neural cleanse and embed these triggers into 2-D space via PCA to visualize trigger distribution. In particular, we train PCA on the triggers generated from the vanilla model, and then apply to the triggers generated from all the models. That is to say, all the dimension reduction processes are under the same linear transformation.
Results. As shown in Figure 15 the effective triggers generated from the model watermarked by our GLBW are clustered together GLBW has low trigger generalization. In contrast, the effective triggers generated from the model watermarked by vanilla backdoor attack seatter the whole space, in- dicating that they are from different latent distributions. As such, the vanilla backdoor watermark
cases. In other words, they are from similar distribu-
tion.Accordingly,
23

---

Published as a conference paper at ICLR 2024
Effective Triggers Ineffective Triggers
Effective Triggers Ineffective Triggers
Effective Triggers
Ineffective Triggers
(a) Vanilla (CIFAR-10)
(b) BWTP (CIFAR-
(c) GLBW (CIFAR-10)
Triggers
Triggers Ineffective Triggers
Triggers
Ineffective Triggers
Ineffective Triggers
(d) Vanilla (GTSRB)
(e) BWTP (GTSRB)
(I) GLBW (GTSRB)
Figure The visualization of the trigger distribution.
has high trigger generalization. The effective triggers of BWTP have similar behavior that of GLBW GTSRB while having similar behavior to the vanilla backdoor watermark CIFAR- 10. These phenomena are consistent with the trigger generalization of BWTP (low generalization on the GTSRB dataset and high generalization on the CIFAR-10 dataset).
WHY WOULD OUR METHODS DECREASE BENIGN ACCURACY?
In our methods, we treat the universal adversarial perturbations (UAPs) and potential triggers the same, whereas we have admit that they may be different. Accordingly, our methods (especially GLBW) are similar to adversarial training against UAPs in terms of effectiveness. As such, our GLBW has similar effect as conducting adversarial training on UAPs that will (significantly) de- creases benign accuracy especially when the task is relatively complicated.
However, does not necessarily means that our method is not practical even we treat them the same.
Firstly, having relatively low benign accuracy will not reduce the reliability and practicality of our method. Firstly, our evaluation based on poisoned samples instead of clean samples. As such, we only need to ensure that our methods lead to high watermark success rate (instead of a high benign accuracy) for faithful results. The watermark success rates are higher than 90% all cases, which is sufficiently high; Secondly, as we mentioned in our experiments, the watermarked DNNs are only used for evaluating SRV methods instead for deployment. Accordingly, the decrease in clean accuracy led by method will not hinder its usefulness.
Secondly, our method can still reduce trigger generalization since it minimizes both trigger general- ization and the risk of UAPs simultaneously.
though we treat them the same.
Thirdly, minimizing the risk of UAPs may have potential benefits in reducing trigger generalization. (Ilyas et al. 2019) revealed that adversarially robust models focus more 'robust features' instead of non-robust ones (e.g., textures). Accordingly, our method may makes DNNs rely more on the original trigger pattern for poisoned samples and therefore could reduce trigger generalization.
24

---

Published as a conference paper at ICLR 2024
Table 15: The average rank of SRV methods that is evaluated with GLBW method and human inspection on the GTSRB dataset.
SRV
Dataset
BP
GBP
GCAM
GGCAM
OCC
FA
LIME
Method
Ours
4.00
4.00
6.00
6.00
2.00
3.00
1.00
GTSRB
Human
4.00
4.00
6.00
6.00
2.00
3.00
1.00
(a) 4 x square
(b) 5 square
(c) Pencil
(d) Triangle
Figure 16: Four additional trigger patterns used in our evaluation.
Table 16: The IOU of SRV methods with the standardized XAI evaluation on the CIFAR-10 dataset and the GTSRB dataset. Square: the trigger with the square-like pattern; Pencil: the trigger with the pencil-like pattern;' Triangle: the trigger with the triangle-like pattern. All trigger patterns have the same size (i.e., white pixels).
SRV
Dataset
BP
GBP
GCAM
GGCAM
OCC
FA
LIME
Trigger
Square
0.2123
0.2123
0.0000
0.2068
0.8849
0.4935
0.9792
CIFAR-10
Pencil
0.1675
0.1675
0.0000
0.1653
0.6776
0.7228
0.9956
Triangle
0.2534
0.2534
0.0000
0.2489
0.6777
0.8463
0.9771
Square
0.3438
0.3438
0.0000
0.3151
0.7388
0.4233
1.0000
GTSRB
Pencil 0.2604
0.2604
0.0000
0,2225
0.8978
0.7570
0.9979
Triangle
0.3003
0.3003
0.0000
0.2762
0.5627
0.4371
0.9890
Lastly, UAPs can be regarded as the trigger of the 'natural backdoor' of models learned from samples WWnger al., 2022). They have very similar properties backdoor triggers and therefore it is very difficult (or probably even impossible) to distinguish them from potential triggers.
K ADDITIONAL EXPERIMENTS OF HUMAN INSPECTION
To further verify that our GLBW-based XAI evaluation is faithful, we generate 10 groups of saliency maps based on 10 randomly selected images on the GTSRB dataset. We conduct human inspection experiments by inviting 10 people and asking them to grade all groups of saliency maps indepen- dently. As shown in Table 15 the rankings generated by our GLBW method are similar to those made by people. These results verify that SRV method with higher order in the ranking list generated by our method is truly better than those with lower orders.
L ADDITIONAL EXPERIMENTS OF STANDARDIZED XAI EVALUATION WITH DIFFERENT TRIGGERS
L. RESULTS WITH DIFFERENT TRIGGER SHAPES
To explore whether the trigger shape has significant effects to standardized XAI evaluation, evaluate the SRV methods with different trigger shapes, including square, pencil, and triangle (as shown in Figure 16), CIFAR-10 and GTSRB datasets. In this experiment, all evaluated trigger
25

---

Published as a conference paper at ICLR 2024
Table 17: The rank of SRV methods with the standardized XAI evaluation on the CIFAR-10 dataset and the GTSRB dataset. Square: the trigger with the square-like like pattern; Pencil: the trigger with the pencil-like pattern; Triangle: the trigger with the triangle-like pattern. All trigger patterns have the same size (i.c.. 9 white pixels).
SRV-
Dataset.
BP
GBP
GCAM
GGCAM
OCC
FA
LIME
Trigger
Square
7
6
2
3
CIFAR-10
Pencil
7
6
3
2
Triangle
4
4
7
6
3
2
1
Square
6
2
3
GTSRB
Pencil
7
6
2
3
1
Triangle
7
6
2
3
1
Table 18: The IOU of SRV methods with the standardized XAI evaluation on the CIFAR-10 dataset and the GTSRB dataset. 3: the trigger with the square-like pattern; 4: the trigger with the x 4 square-like like pattern; 5 x 5: the trigger with the x 5 square-like pattern.
SRV
Dataset
BP
GBP
GCAM
GGCAM
OCC
FA
LIME
Size
x 3
0.2123
0.2123
0.0000
0.2068
0.8849
0.4935
0.9792
CIFAR-10
x 4
0.2867
0.2867
0.0000
0,2814
0.8796
0.9821
0.9841
x 5
0.3811
0.3811
0.0000
0.3798
0.7255
0.6672
0.9923
3
0.3438
0.3438
0.0000
0.3151
0.7388
0.4233
1.0000
GTSRB
4 4
0.3360
0.3360
0.0000
0.2857
0.7803
1.0000
0.9914
5 5
0.3436
0.3436
0.0000
0.2861
0.8364
0.5205
0,9838
Table 19: The rank of SRV methods with the standardized XAI evaluation on the CIFAR-10 10 dataset and the GTSRB dataset. 3: the trigger with the 3 square-like pattern; 4: the trigger with the x 4 square-like like pattern: 5 x 5: the trigger with the x 5 square-like pattern.
SRV
Dataset
BP
GBP
GCAM
GGCAM
OCC
FA
LIME
Size|
x 3
7
6
2
3
CIFAR-10
4 4x4 4
6
3
2
I
5 5
4
4
7
6
2
3
3x3
6
2
3
GTSRB
4 4
4
4
7
6
3
1
2
5
7
6
2
3
1
patterns have 9 pixels. As shown in Table [6 the trigger shape has mild effects to the final ranking, although it may have some influences to IOU values.
I.2 RESULTS WITH DIFFERENT TRIGGER SIZES
To explore whether the trigger size has significant effects to standardized XAI evaluation, evaluate SRV methods based on qquare-type triggers with different sizes (i.e., 3 x 3, 4 x 4 and 5 x 5, shown in Figure 16), CIFAR-10 10 and GTSRB datasets. As shown in Table [8] I9 the trigger shape has minor effects to the final ranking, although may have some influences to IOU values.
M CONNECTIONS AND DIFFERENCES WITH RELATED WORKS
In this section, we discuss the connections and differences between our methods (i.e., BWTP and GLBW) and backdoor attacks, model watermarks, adversarial learning, and neural cleanse.
26

---

Published as a conference paper at ICLR 2024
M.1 CONNECTIONS AND DIFFERENCES WITH BACKDOOR ATTACKS
In general, both our methods and backdoor attacks intend to implant special model behaviors (i.e., backdoor) DNNs. However, they still have many intrinsic differences.
Firstly, backdoor attacks aim at maliciously manipulating attacked models during the inference pro- cess. contrast, our watermarks are designed for the evaluation of salieney-based representation visualization (SRV), which is a positive application.
Secondly, there are different parties (i.e., adversaries, and victim model users) involved in back- door attacks while only the adjudicator is included in our SRV evaluation methods. Accordingly, backdoor attacks need ensure that the attacked model has high benign accuracy or its poisoned samples are similar to their benign version to circumvent the detection of victim model users. How- ever, in our methods, we only need to ensure high watermark/attack success rate (instead of the high benign accuracy) of watermarked DNNs since we don't 't need to fulfill watermark stealthiness.
Lastly, backdoor attacks only need to ensure that the original trigger pattern used for training can activate hidden backdoors in attacked DNNs. In other words, the backdoor adversaries don't 't need to measure or control the degree of trigger generalization. However, as we mentioned in the main manuscript, methods need to minimize the generalization to design faithful SRV evaluation.
M.2 CONNECTIONS DIFFERENCES WITH MODEL WATERMARKS
Currently, there are different types of model watermarks. Among all watermarking techniques, backdoor-base based methods are probably the most classical and widespread ones. In general, our meth- ods share some similarities to backdoor-basee that the defenders can fully control the training process of models and intend to implant backdoors to watermarked DNNs. However, they still have some essential differences.
model watermarks. For example, both of them assume
Firstly, similar to backdoor attacks, there are different parties (i.e., the model owner and users/adversaries) involved in model watermarks. Accordingly, the model owner still requires to preserve the benign accuracy of watermarked DNNs otherwise users will not exploit them. con- trast, is not necessary have high benign accuracy in methods.
Secondly, model watermarks need to ensure robustness under different nntermark-remove attacks (e.g.. fine-tuning, and model pruning) since adversaries have the incentive to remove them. How- ever, in our methods, we only require to have a low trigger generalization of backdoor watermarks.
M.3 CONNECTIONS AND DIFFERENCES WITH ADVERSARIAL LEARNING
Both (targeted universal) adversarial attacks and our methods (during the training process) intend to find pattern that can mislead model predictions to a specific class. Instead of simply having high attack effectiveness as required by adversarial attacks, our methods also need to ensure that the generated perturbation is (significantly) different from the original trigger pattern. This requirement is necessary for minimizing trigger generalization of backdoor watermarks.
Both methods and adversarial training formulate model training as min-max max problem. How- ever, adversarial training aims to make models robust to all potential perturbations. In contrast, watermarks intend to preserve the backdoor regarding the original trigger while reducing other backdoors instead of eliminating all backdoors.
M.4 CONNECTIONS AND DIFFERENCES WITH NEURAL CLEANSE
In general, both our methods and neural cleanse (Wang et al. 2019) intend generate targeted universal adversarial perturbations potential trigger patterns. However, methods have different constraints (e.g.. |m' O m S T |ml) since we have fundamentally different purposes.
Firstly, neural cleanse has no information (e.g., trigger size and target label) about the ground-truth trigger pattern. Accordingly, it needs to generate potential trigger towards each class and adopt trigger sparsity to decide the final one. However, its assumption that all trigger patterns should be sparse not hold for advanced backdoor attacks. As such, this method is not effective (Li et al. 2021b). In contrast, know the ground-truth trigger patch, its target label, and its sparsity. Our
27

---

Published as a conference paper at ICLR 2024
Table 20: The rank (based average IOU among all poisoned sample) across four trigger patterns of SRV methods that is evaluated with different methods.
Dataset
Method, SRV
BP
GBP
GCAM
GGCAM
OCC
FA
LIME
Ours
4.75
4.75
4.5
2
3
CIFAR-10
Adaptive
4.4
4.5
7
5
2.5
2.5
1
Ours
4.5
4.5
5
2.5
1.25
GTSRB
Adaptive
4.75
4.75
6.5
5
2.25
2.75
1
purpose is to find potential triggers with the highest attack effectiveness and differences from the ground-truth one instead of finding the ground-trute truth trigger.
Secondly, neural cleanse exploited the standard SGD to solve the optimization problem. In contrast, in our GLBW, we design an adaptive optimization method to find desired potential trigger patterns since our task is significantly more difficult than that of the neural cleanse. Please refer to Appendix for more details about our optimization process.
N ADDITIONAL EXPERIMENTS OF STANDARDIZED EVALUATION METHOD WITH ADAPTIVE BINARIZATION
Regarding the selection of M pixels proposed in standardized backdoor-basee XAI evaluation, people may argue that it is not significantly better than the previous method. In this section, we discuss whether it would be better high influences to calculate the rank of each SRV method.
adaptive binarization methods to obtain regions with
Settings. We hereby exploit a classical adaptive binarization method, i.e., OTSU (Otsu) 1979,, to obtain regions with high influences to calculate the rank of each SRV method and compare the results of standardized method (dubbed 'Ours') and those of the standardized method with adaptive binarization (dubbed 'Adaptive') ) All other settings are the same as those used in Section 3.1
Results. As shown in Table 20 using adaptive binarization method has mild influence to the final average ranking. These results verify the effectiveness of our method.
O REPRODUCIBILITY STATEMENT
In the appendix, we provide detailed descriptions of the datasets, models, training and evaluation settings, and computational facilities. The codes and model checkpoints for reproducing the main experiments of our evaluation are also provided in the supplementary material. Our codes are avail- able at httpttts://ggtteee
www.co////aa... www.///aegg../////oooooooooooo/e /GLB
P DISCUSSIONS ABOUT ADOPTED DATA
this paper, all adopted samples are from the open-sourced datasets (i.e., CIFAR-10 and GTSRB). These datasets contain no human object. We admit that we modified few samples for watermark- ing. However, our research treats all samples the same and all triggers contain no offensive content. Accordingly, our work fulfills the requirements of these datasets and has no privacy violation.
28


Reviews:
[{"review_id": "IwjwGKAPo1", "reviewer": "Reviewer_LeTy", "summary": "This paper proposed a generalization-limited backdoor watermark (GLBW), an evaluation method for saliency-based representation visualization (SRV). The author claimed that the existing watermark-based evaluation has a problem with trigger generalization: a typically trained poisoned classifier has a gap between potential (universal adversarial perturbations) and original triggers. Based on this observation, the author proposed GLBW, constructed as a combination of benign, backdoor, and generalization losses to generate generalized samples for training a poisoned model. The author showed that the poisoned model trained with GLBW has better benign accuracy and watermark success rate compared to the baselines. Also, they showed that their standardized version of the backdoor-based saliency map evaluation method has a lower variance than the vanilla method.", "questions": "- Could you describe how the existence of potential triggers affects the watermark-based saliency map evaluation?\n- How do we evaluate which saliency map evaluation metric is better?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper addressed a challenging and essential problem: evaluating explanation methods without human annotation.\n- They found the implementation problems that exist in the previous watermark-based evaluation method (Lin et al., 2021)", "weaknesses": "- Unnecessity of the consideration of trigger generalization in evaluating XAI\n  - If I know correctly, the main idea underlying the evaluation of XAI using a backdoor attack is that the poisoned model would refer to the watermarked regions to make a prediction, and a good XAI would highlight these regions as well. In other words, the watermark region is treated as a ground truth of the saliency map. Meanwhile, the author claims that the existence of potential triggers is a problem because it has almost zero cross-entropy loss, but their visualization differs from the original trigger. However, for me, it is unclear why the potential triggers should be considered in evaluating XAI based on watermark regions. For example, assume that there is a cat image and a poisoned model predicts it as a cat. If the prediction results changed by adding a watermark on the image, then it is due to the watermark in a high probability. \n- Limitation of IoU based evaluation methods\n  - Since the saliency map is not a segmentation, comparing the ground truth segmentation label with the saliency map could not be the best evaluation method. The classifier would not uniformly refer to the ground truth area.\n- Week explanation for the intention of BWTP/GLBW\n  - Many of the details of BWTP and GLBW are in the appendix, which makes it hard to understand them at first sight.  \n\n- Minor corrections\n  - In Eq (1), $\\mathcal{L}$ receives $y\\in${$0,...,K-1$} for benign and backdoor loss, but $\\mathbf{y}\\in\\mathbb{R}^K$ for penalty loss. \n  - In Eq (2), $|\\mathbf{b}'| is not defined", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4RaLzEOjNB", "reviewer": "Reviewer_4nxh", "summary": "In this paper, the authors aim at the evaluation problem of saliency-based representation visualization (SRV). For that, they firstly argue that the current backdoor-based SRV evaluation methods have the implementation limitations and unreliable nature. Based on that, they propose the faithful XAI method to overcome these issues. A variety of experiments verify the effectiveness of proposed method.", "questions": "See the weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The writing or story is good, they provide a solid experiment to show the issues of current backdoor-based SRV evaluation, and then give a reasonable solution to address these issues.  I think this may give some insights to the community. \n\n2. The experiments are solid, the authors give many detailed experiments to show their effectiveness, as shown in the manuscript and that in the appendix.", "weaknesses": "Actually, I am not very familiar with the backdoor attack and SRV evaluation problem, and just have some background knowledge about them. According to my understanding, there exist some concerns as follows:\n\n1. In my opinion, the trigger in backdoor attack and the real object are different in essence. In an image, the trigger is an out-of-context object, and the real object is in context. Therefore, although a SRV evaluation method can show good performance on the trigger, it does not ensure the effectiveness on the real object. I wonder that the authors whether provide some analysis about this difference. \n\n2. In the experiments, although they conduct many experiments, it seems that there are not some comparisons with the SOTA SRV evaluation methods. In Section 2.2, the authors discuss many existing SRV evaluation methods, they should give the comprehensive comparisons with these methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fQQxW0rQNj", "reviewer": "Reviewer_ZYFJ", "summary": "This paper explores how to faithfully evaluate XAI methods based on backdoor watermarks. Specifically, the authors first reveal the implementation limitations and unreliable nature of using standard backdoor watermarks due to trigger generalization. Based on their analyses, the authors propose the first generalization-limited backdoor watermark to further design more faithful XAI evaluation. The authors evaluate their method on CIFAR-10 and GTSRB datasets.", "questions": "See the above weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.\tThis work cleverly links two seemingly unrelated (i.e., backdoor attacks and model interpretability) yet important research fields. In particular, the proposed method circumvents the harmful nature of backdoor attacks since the watermarked model is only used for evaluating models instead of for deployment. Although this perspective is first proposed in a previous work, further non-trivial exploration of this angle is also very meaningful and valuable.\n2.\tI enjoy the analyses of the limitations of existing backdoor-based XAI evaluations, especially their unreliable nature. These findings are non-trivial and are critical for practical backdoor-based XAI evaluations. \n3.\tThe proposed method is novel and reasonable. I think the authors have comprehensively demonstrate the design philosophy of their methods.\n4.\tThe experiments are comprehensive to a large extent. In particular, the authors exploit different trigger inversion techniques to evaluate trigger generalization, which should be encouraged.\n5.\tThe paper is well-written and its main idea is easy to follow.", "weaknesses": "1.\tIt would be better if the authors can provide more details about the differences between GLBW and BWTP. They seem similar in terms of formulas alone.\n2.\tRegarding the selection of M pixels, I do not think it is significantly better than the previous method. One example is that the size of the calculated saliency is significantly larger than M, while the size of the trigger is small. Because of M, the IoU value is high, but in reality, it is small due to the union.\n3.\tThe paper treats the universal adversarial perturbations and the triggers the same, while their characteristics differ. However, the proposed method treats them the same. Please provide more explanations about why it is not a problem or how to address it.\n4.\tIt would be better if the author could provide more details and discussions about why trigger generalization is very important for XAI evaluation in the appendix. \n5.\tThe explainability metric is based on simple backdoored patterns, but why would it be a good reference for complicated features in real images for practice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rf5AR6dQjI", "reviewer": "Reviewer_RMSQ", "summary": "The paper revisits the evaluation techniques for saliency-based representation visualization (SRV). The authors uncover the unreliable nature of current backdoor-based SRV evaluation methods, particularly due to the trigger generalization of backdoor watermarks. They also highlight implementation limitations. From these insights, they introduce a generalization-limited backdoor watermark and, based on this, design a more accurate XAI evaluation. The aim of their research is to enhance the understanding of XAI evaluation and aid in the creation of more interpretable deep learning methods.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper highlights a crucial insight into the shortcomings of the current backdoor-based SRV evaluation and presents a new method called GLBW for a more accurate XAI assessment.\n2. The authors have conducted extensive experiments on benchmark datasets to verify the effectiveness of their GLBW, adding credibility to their claims.\n3. The authors have provided detailed descriptions of datasets, models, training, and evaluation settings in the appendix. They also commit to releasing training codes upon acceptance, promoting transparency and reproducibility.", "weaknesses": "The only weakness I consider is that the authors haven't provided enough empirical evidence demonstrating the superiority of their proposed method over traditional backdoor-based techniques. For instance, they don't experimentally address why the minimum bounding box approach might yield inaccurate results as they claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)