Identification of the Adversary from a Single Adversarial Example
Authors: Minhao Cheng
Rui Min
Haochen Sun
Venue: Submitted to ICLR 2023 (2023)
Keywords: Adversarial examples
Forensic investigation

Abstract: Deep neural networks have been shown vulnerable to adversarial examples. Even though many defence methods have been proposed to enhance the robustness, it is still a long way toward providing an attack-free method to build a trustworthy machine learning system. In this paper, instead of enhancing the robustness, we take the investigator's perspective and propose a new framework to trace the first compromised model in a forensic investigation manner. Specifically, we focus on the following setting: the machine learning service provider provides models for a set of customers. However, one of the customers conducted adversarial attacks to fool the system. Therefore, the investigator's objective is to identify the first compromised model by collecting and analyzing evidence from only available adversarial examples. To make the tracing viable, we design a random mask watermarking mechanism to differentiate adversarial examples from different models. First, we propose a tracing approach in the data-limited case where the original example is also available. Then, we design a data-free approach to identify the adversary without accessing the original example. Finally, the effectiveness of our proposed framework is evaluated by extensive experiments with different model architectures, adversarial attacks, and datasets.

Under review as a conference paper at ICLR 2023
IDENTIFICATION OF THE ADVERSARY FROM A SINGLE ADVERSARIAL EXAMPLE
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks have been shown vulnerable to adversarial examples. Even though many defence methods have been proposed to enhance robustness, it still long way toward providing attack-free method to build trustworthy ma- chine learning system. In this paper, instead of enhancing the robustness, we take the investigator's perspective and propose new framework to trace the first com- promised model in a forensic investigation manner. Specifically, we focus on the following setting: the machine learning service provider provides models for set of customers. However, one of the customers conducted adversarial attacks to fool the system. Therefore, the investigator's objective is to identify the first compro- mised model by collecting and analyzing evidence from only available adversarial examples. To make the tracing viable, we design random mask watermarking mechanism to differentiate adversarial examples from different models. First, we propose a tracing approach in the data-limited case where the original example is also available. Then, we design data-free approach to identify the adversary without accessing the original example. Finally, the effectiveness of our proposed framework is evaluated by extensive experiments with different model architec- tures, adversarial attacks, and datasets.
INTRODUCTION
has been shown recently that machine learning algo- rithms, especially deep neural networks, vulnerable to adversarial attacks (Szegedy et al.] 2014; Goodfellow al. 2015). That given victim neural network model and a correctly classified example, an adversarial attack aims to compute a small perturbation such that the orig- inal example will be misclassified with this perturbation added. To enhance the robustness against attacks, many defence strategies have been proposed (Madry et al. 2018 Zhang et al., 2019; [Cheng et al.) (2020a). How- they suffer from poor scalability and generalization on other attacks and trade-offs with test accuracy on clean data, making the robust models hard to deploy in real life. Therefore, in this paper, we turn our focus on the after- math of adversarial attacks, where we take the forensic investigation to identify the first compromised model for generating the adversarial attack. In this paper, we show that given only single adversarial example, we could trace the source model that adversaries based for conduct- ing the attack. As shown in Figure [ we consider the fol- lowing setting: a Machine Learning as a Service (MLaaS) provider will provide models for set of customers. For the consideration of time-sensitive applications such as auto-pilot systems, the models would be distributed to ev- ery customer locally. The model architecture and weight details are encrypted and hidden from the customers for the consideration of intellectual property (IP) protection and maintenance. In other
Service Provider
Stop Sign
Stop Sign
Stop Sign
Model
Model 2
Model
Input
80
80
Attacker
x (Speed Limit)
Xport
Nade
Query
Attack
Feedback
Model
Figure 1: Illustration of the threat model in owner-customer distribution setting where the attacker conducts adversar- ial attacks on the assigned model 2 and uses the generated adversarial examples attack other users' models.

---

Under review as a conference paper at ICLR 2023
words, every customer could only access the input and output of the provided model but not the internal configurations. On the other side, the service provider has full access to every detail of their models, including the training procedure, model architecture, and hyperparameters. However, there exists a malicious user who aims to fool the system by conducting adversarial attacks and gaining profit from the generated adversarial examples. Since the models are trained for the same objective using the same dataset, adversarial examples generated by the adversary could be transferred to the other users' models with very high probability, 100% if the models are the same. Thus it critical for the interested party to conduct the investigation and trace the malicious user by identifying the compromised model. Taking the auto-pilot systems on the self-driving car as an example, the ma- licious user could conduct an adversarial attack on a road sign by querying his own vehicle model and then create an adversarial sticker to fool other vehicles using the same detection system.
Only given adversarial examples as evidence, in order to make the tracing possible, adversarial examples generated by different models have to be unique so that we are able to find the source model and trace the malicious user in the end. To achieve this goal, we design random mask watermarking strategy which embeds the watermark to the generated adversarial samples without sacrificing model performance. At the same time, the proposed strategy is efficient and scalable that only needs a few iterations of fine-tuning, In the presence of the original example, a nigh-accuracy tracing method is proposed, which compares the adversarial perturbation with every model's masked pattern and the adversarial example's output distribution among different models. Because it is not always practical have the original example as reference, in the second part, we further discuss the most challenging and practical attack setting where only the adversarial example is available for the investigator. Observing that the model's probability predictions on the same adversarial example would change significantly with a different watermark applied, we derive an effective rule to find the compromised model. Specifically, based on the property that adversarial example is not robust against noise, we redesign the tracing metric based on the change in the predicted probabilities when applying different watermarks, which we expect the compromised model to minimize.
Comprehensive experiments are conducted on multiple adversarial attacks and datasets. When there is only a single adversarial example available, the results demonstrate that the two proposed methods could successfully trace the suspect model with over 74% accuracy on average with the data-limited case and data-free case. The tracing accuracy increases significantly to around 97% when there are two adversarial examples available.
Our contributions are summarized below:
To the best of our knowledge, we are the first to propose a novel and scalable framework to make it possible trace the compromised model by only using a single sample and its corresponding adversarial example.
In the absence of samples used to generate adversarial examples, we further utilize the prediction difference of each model with different watermarks to identify the adversary without any requirement on the original sample.
Extensive experiments were conducted to demonstrate the effectiveness of the proposed framework trace the compromised model that the malicious users utilize to conduct different black-box adversarial attacks under different network architectures and datasets. We showed that the adversary could be traced with high accuracy in different scenarios. and the proposed framework has good scalability and efficiency.
2 RELATED WORK
Adversarial Attack Since the discovery of adversarial example (Szegedy et al. 22014), many at- tack methods have been proposed. Roughly speaking, based on the different levels of information accessibility, adversarial attacks can be divided into white-bor bbx and black-box settings. In the white- box setting, the adversary has complete knowledge of the targeted model, including the model archi- tecture and parameters. Thus, back-propagation could be conducted to solve the adversarial object by gradient computation (Goodfellow et al., 2013; Kurakin et al.) 2017: Madry et al. 2018 Car- lini & Wagner 2017). On the other hand, the black- box setting has drawn much attention recently, where the attacker could only query the model but has no direct access to any internal information.
Based on whether the model feedback would give the probability output, the attacks could be soft-
2

---

Under review as a conference paper at ICLR 2023
label attacks or hard-label attacks. In the soft-label setting, ZOO attack (Chen et al. 2017) first proposed using a finite difference to estimate the gradient coordinate wise and then conducted the gradient descent. It was then improved by selecting better prior distribution (lyas et al. 2018b)
and compressing the search space (Tu et al.) 2019). To further increase the query efficiency. instead of calculating the full gradient, seillent-sign-baseed methods (Liu et al. 2018) have been proposed, and the attacks could still achieve a good success rate. Also, handom-search-bar
methods such as Square attack (Andriushchenko et al. (2200) and SimBA (Guo al., 2019) have found that the attacks could be more successful in other domains such as frequency, and the square pattern could further improve the query efficiency. On the other hand, boundary attack (Brendel et al., 2017) first proposed the hard-label setting and used the random search method to find the adversarial attacks. was then improved by [Chnn et al. (2020) by finding better sampling prior. OPT attack (Cheng et al., (018)) and Sign-OPT attack (Cheng et al. (020b), on the other hand, formalized the hard-label attack into an optimization framework and used the zeroth-order method to solve it.
Watermarking Model watermarking is introduced in intellectual property protection on machine learning systems. It could be roughly divided into two categories: white-box and black-box wa- lermarking, depending on the accessibility to the model and its parameters in order to extract the watermark. For the white-box watermarking, the first scheme for DNN (Uchida et al. 2017) tended to embed the watermark into the training process. Later, because of the extra capacity available in the state-of-art neural networks, DeepMarks (Chen et al., 2018) encoded the watermark to be a binary vector in the probability density function of trainable weights. However, it is not always convenient for the model owner to do the formal verification in the white-box watermarking, and the white-box watermark is also vulnerable to statistical attacks (Wang & Kerschbaum 2019p. Black-box methods are then proposed to solve the limitations mentioned above. DeepSigns (Darvish Rouhani et al. 2019) has done some improvement over [Uchida et al. (2017) and proposed framework incor- porate the black-box case. Utilizing backdoor attacks to do watermarking is another big trend and has drawn much attention recently (Zhang et al. 2018: Adi et al. 2018). Adversarial examples are used to enable extraction of the watermark without requiring model parameters (Chen et al. 2019 Le Merrer al. 2020 However, the forensic investigation of adversarial examples has a major dif- ference from the model watermarking as the model watermarking must be shown in the generated adversarial examples. Also, it requires the tracking be -agnosic. In other words, the tracing mark should be robust across different attacks simultaneously. However, the current watermarking methods (Zhang et al.) 2018 Adi et al.] (2018) are either attack deeendede or have limited effect on the generated adversarial examples, which is not suitable for this task.
Forensic investigation in Machine Learning Although machine learning methods have already been used in forensic science (Carriquiry et al. 2019)). there are a few studies on building trustworthy machine learning from forensic perspective. Most papers focus on how identify the model stealing attack by introducing the watermarking approaches to protect the intellectual property of the deep neural networks. That is to say, a unified and invisible watermark is hidden into models that can be extracted later as special task-agnostic evidence. However, to the best of our knowledge, we are the first paper to study the adversarial attack from a forensic investigation perspective.
3 PROBLEM SETTING
We formalize the identification of the compromised model in the owner-customer distribution set- ting (Zhang et al. 2021). The machine learning service provider (owner) is assumed to own m copies of model f1. f2... f2...., f' for the same K K-way classification task trained using the same train- ing dataset. As inference efficiency is critical in time-sensitive applications such auto-pilot sys- tems, these model copies are first encrypted for intellectual protection and security concerns and then distributed to the m customers (users). Therefore, the customers only have black-box access to their own distributed model. In other words, the user could only query his own model f. to get the prediction results without any access to the internal information about the model.
Unfortunately, a malicious user (adversary) exists who aims to fool the whole system. including other users' models, by conducting black-box adversarial attacks. Specifically, let the malicious user's model copy to be fau (the compromised model). As he does not have access to query other users' models, he then chooses to perform black-box attacks to his copy fatt to generate adversar-
3

---

Under review as a conference paper at ICLR 2023
ial example Tadv. As all model copies are trained with the same dataset for the same classification task, the generated adversarial example could successfully lead to the misclassification of other users' models. Our task is to find the compromised model fatt from the model pool.
4 METHODOLOGY
If the same model copy is distributed to every customer, the generated adversarial examples from different users' models will be identical each other, and thus becomes impossible to trace the adversary by only giving adversarial examples. In the following section, we propose our framework which consists of two parts shown in Figure 2 First, we design a simple random mask watermark- ing method that would have a limited effect on the models' accuracy while embedding distinctive features in adversarial examples, distinguishing them from those generated from other models. We further design tail and head mechanism to make the training process scalable and efficient. De- pending on the availability of the original example as a reference, we then propose two detection scenarios to identify the adversary from adversarial examples.
4.1 RANDOM MASK WATERMARKING
Since we need to identify the compromised model from a large pool of customer copies, it requires us to assign a unique identification mark for every customer copy, and the mark should be reflected in the generated adversarial example.
In this section, we design simple but effective method by applying a random watermark on each of the m model copies. As shown in Figure 2 for each model copy fi(L S i S m). we randomly select set of pixels w" as the watermark on the training samples. Formally, denote the input as E R- x HxC, For every model fú, we randomly generate binary matrix w E f0, 1) W X by sampling uniformly. We call the wt mask for model fi, deciding the set of masked pixels. When wht,, = 1 for a specific pixel (a, b) at channel e, value is set to be O; otherwise, when w = O, the original pixel value not modified. That to say, for every input æ. the input after the mask a on model fi would be o abc == Ba.b.c (1 w (a.be) for each pixel (a, b) at each channel For
simplicity, we use T² x O (1 w²) denote the masked sample æ; in the whole paper, where O represents the wiement-wise product.
Each input is first applied with the mask and then fed into the model in both the training and inference phases. To speed up the training process and make the pipeline scalable to thousands of users, we add each model with a few network layers as head part hG. The output of the head part will directly feed into shared tail model t. In other words, we have each model copy as
f(())) (h.())))
Specifically, in the pretraining phase, first train model without the watermark from scratch as the base model. Then each model copy is assigned with a unique model head for the added specific watermark and shares big common tail inherited from the base model. During the fine-tuning process, we freeze the parameters in the tail and embed the watermark to the model by only fine- tuning the weights in the head part with few epochs. Our following experiments will show it is sufficient to embed watermark to a few layers in DNNs without sacrificing model accuracy. To be noted, the proposed method could be easily adapted to the new user case as we could just create a unique watermark and fine-tune the new head with a few epochs.
Since the distributed model copy's detail is encrypted and not accessible in the black-box setting, users are unaware that their models and the generated adversarial examples are watermarked. More- over, as the watermarks are generated uniformly at random, each user's model copy can be assigned unique watermark even when the number of users is large. Furthermore, since deep neural net- works have been shown to be robust against random noise, the proposed watermarking would have a very limited effect the model's performance, shown in T when the masking ratio / || is not too large.
In this paper, we only show straightforward random mask watermarking scheme, which could already provide us with satisfactory tracing accuracy without hurting the model performance. We leave the design of the better watermarking methods under this framework to future works.
4

---

Under review as a conference paper at ICLR 2023
4.2 DATA-LIMITED ADVERSARY IDENTIFICATION
With the watermarking scheme described in Section 4.1 we can exploit the information embedded in the watermarked adversarial example (and the corresponding original example) to identify the compromised model.
We first introduce the data- case where the corresponding original example æ, on which the given adversarial example Tadv is based, is available. A natural example of this setting is that the malicious user generates an adversarial sticker based on a road sign using his own self self-driving car. As the self-driving system is built by the service provider and shared, the adversarial stickers would fool other passing vehicles. Then, as the investigator, we aim to trace the adversary by analyzing the road sign and the adversarial sticker. We now discuss the design of the detector in the tata-limited case, which is inspired by the mechanism of adversarial attacks. Specifically, since the adversarial attack is formalized as optimization problem, the adversary takes the gradient of the designed loss function L with respect to the input æ: to find the most effective perturbation.
Ad Adversarial Perturbation: Formally, for the model fi, the gradient of the designed loss function L with respect to the given sample is
aC(f(()) 0 if wa,b = or
Since the black-box attacks are designed to approximate the gradients used in the white-box attacks, we could expect that the approximated gradients at the masked pixels would have a value close to or be smaller in magnitude than the other pixels. Based on this observation, since we have access to the original example we could calculate the adversarial perturbation ô Tadv I. If the adversarial example is generated by the compromised model copy fatt. values in ô should be much smaller in those coordinates where wat = 1. Therefore, given Tadv and æ, we thus calculate score for each model by summing up the absolute values of the adversarial perturbation overall masked pixels (of the corresponding model), i.e..
8'
Σ wi Wa.be ② Dadv x la,bb a.b.c
(3)
Adversarial Stability: Moreover, we also observe that the cross-entropy loss between the pre- diction output of adversarial examples and the ground-truth label of clean examples differs among different models. Since adversarial examples should be identical to original examples visually, the ground-truth label could be easily inferred. Specifically, if the adversarial example Todv is gener- ated from model fi the cross entropy loss Lce(Si( f (Fade) * y Vij * i, where y is the ground truth label of æ. Intuitively, model fa would have the smallest confidence on the ground-truth label since some of the adversarial perturbation may be blocked by other models' watermarks. We then combine the two metrics and calculate the final score for each model. Then, we take the model with the smallest score as the compromised model, ie.,
y) is smaller than (c y) if
att f- argmin(6 8 + (oLoE((((ad)))))) 1sism
(4)
4.3 DATA-FREE ADVERSARY IDENTIFICATION
The previously introduced data-limited detector requires access to the original example refer- ence, which is not realistic in many scenarios. Therefore, in the following section, we relax this constraint and discuss the tracing under the most challenging yet realistic setting where the only evidence available is the generated adversarial example. We propose a data-free detector based on the different model outputs when applying different masks to the adversarial example.
Formally, for the given adversarial example Tade, we first apply every model's watermark w - E [m] to create set of masked adversarial examples (øi [== where Lidte Badv (1 w²). We then feed the masked adversarial examples set to each model fa to get its probability output. For every model ft. we get a probability output matrix pi == ( ...., S (a(mm..) )"] [0, 1
where each element in pi is Pa, [fr (a:2 )] and K is the number of classes. (a
5

---

Under review as a conference paper at ICLR 2023
Watermarking
Image
Watermark
Mask-1 Image
Model
Shared (Freeze)
Model
DNN System
Model
Train
W
Image Y Watermark
Mask-n Image
Model
Pretraining
Tracing
Data-Free
Model's
0.147
Training Dataset
Base Model
Model2
(attack)
8, + B Loor
Train
Xadv
Model
Data-Limit
Model
Test Dataset
0.059
0.000 (attack)
0.054
Evaluate
Smax + lics
Xadtr
Figure 2: Proposed framework of identifying compromised model from adversarial examples.
Since adversarial examples are very close the model's decision boundary (Brendel et 2017 Cheng et al.] 2018), slight perturbation to it would cause the model's prediction to change sig- nificantly. In other words, adversarial examples are sensitive to small perturbations, while ordinary examples are relatively more robust. It then inspires us to propose a metric based on this difference to detect the compromised model. Specifically, us still assume the given adversarial example Badr is from model f;. Then, when the corresponding watermark w² is applied, the probability prediction will remain unchanged. However, when applying another watermark ,².j * i, it likely that the watermarked adversarial example would be moved away from the decision boundary. Therefore, the maximal predicted class probability is generally larger after applying w,. At the same time, the adversarial example is not generated from the model, the extent of change would be limited. Therefore, we propose the max label score Smax based on the extent of change of prediction:
maxiskáK Pi, Eisism max P
Si
(5)
mar
We further combine the score of adversarial stability proposed in data-limited case with max label score to improve the detection accuracy:
all S (BCCE((((ad))))))
(6)
EXPERIMENTAL RESULTS
In this section, we extensively evaluate the proposed framework on a variety adversarial attacks, datasets, and models. We first introduce the experiment setup and implementation details. In Sec- tion 5.1 we show the proposed random mask watermarking has limited effect on model per- formance. We then test the tracing success rate adversarial examples generated by different black-box attacks in the data-limited case and data-free case in Section 5.2
Implementation Details:
We conduct experiments datasets GTSRB (Stallkamp et al, 2012) and CIFAR-10 (Krizhevsky et al. 2009). We use two widely used network architectures VGG16 (Simonyan & Zisserman 2015)) and ResNet18 (He al.)
popular image classification

---

Under review as a conference paper at ICLR 2023
2016). All our experiments were implemented in Pytorch and conducted using RTX 3090 GPU. In the pretraining stage. the base models are trained with Adam optimizer with the learning rate of 10-3 and batch size of 128 for 50 epochs. For every model's watermark, independently ran- domly sample 100 pixels to mask. As the input size of both CIFAR-10 10 and GTSRB is 32 x 32 x 3, it is equivalent saying the masking rate is around 3.26%. To embed different watermarks into in- dividual models efficiently, we separate the base model by taking the first several layers as the head and leaving the rest as the tail. Specifically, for ResNet18, we take the first residual block as the model head, and we detach the first two convolution layers for VGG16. The parameter percentage of the head for ResNet1 and VGG16 is 1.34% and 0.26% respectively which is very small portion of the whole network weights. We then fine-tune the weights in the head portion for 10 epochs while freezing the parameters of the model tail. Our code will publicly available.
Adversarial Attack Methods: We perform the following five black-box adversarial attacks to generate the adversarial example:
Natural Evolutionary Strategy (NES): Tyas et al. (2018a) introduced a soft-label black- box adversarial attack that designed a loss the output probability changes and used Neural evolution strategy (NES) to approximately estimate the gradient.
Bandits and Priors Attack (Bandit): [lys et al. (2018b) introduced a soft-label black- box attack by using the bandit algorithm to find better prior where the adversarial pertur- bation could be drawn with high probability.
Simple Black-box Attack (SimBA): Guo et al. (019)) introduced a soft-label black-box adversarial attack by sampling the perturbation direction from predefined orthonormal basis. The sampled direction would be either added or subtracted from the target image to test its success.
Hop-Skip-Jump Attack (HSJ): Chen et al. (2020) introduced hard-label black-box attack which is applied the zeroth-order sign oracle to improve Boundary attack (Brendel et al. 2017).
Sign-OPT Attack (SignOPT): Cheng al. (2020)) introduced hard-label black-box at- tack that use a single query oracle to improve the query efficiency of OPT attack (Cheng
ct al. 2018).
For SimBA, HSJ and SignOPT attacks, we use Adversarial Robustness Toolbox (ART) (Nicolae jet al., 2018)'s implementation. We use the default hyperparameters in the ART toolbox to conduct the attack. For the NES attack and Bandit attack, we re-implement them in Pytorch following the official implementation in https: / /github..... com woom/addee http/MadyLaabbbbbaa hataaabbbblbbbxx///aa backbox-bandirs. All the attacks conducted in the & constraints and untargeted setting. The attack will be stopped when there a successful adversarial example generated.
Evaluation Metric: To evaluate the effectiveness of the proposed detection method, for each at- tack, we generate 10 transferable adversarial examples for every model copy. An adversarial ex- ample Vadv is defined as transferable if and only the prediction of the compromised model fax is wrong and, at the same time, the prediction of at least one of the other worth noting that it is rather difficult for the black-box attacks to have a good transferability over the proposed random mask watermarking our detector's 's effectiveness. To sum up, for each attack, we have a total of 1000 adversarial exam- ples under the setting of 100 models. We then define the tracing accuracy evaluate the detection rate defined as follows:
I models is wrong. It is
consider 10 transferable examples are sufficient to test
Trace Acc Neorrett
N
Where N Noonver is the count of the correct identification of the compromised model and Niotal is the total number the transferable adversarial example generated.
5.1 MODEL PERFORMANCE WITH RANDOM MASK WATERMARKING
In this section, we conduct experiments to verify whether the model could still maintain good per- formance after applying the watermark. Specifically, we train 100 models two datasets CIFAR-10

---

Under review as a conference paper at ICLR 2023
and GTSRB with two popular architectures VGG16 and ResNet18. We also add baseline model without watermark as a reference.
Table 1: The classification accuracies (%) of models with random mask watermarking. V-CIFARI0 represents the model trained with VGG16 using the CIFAR-10 dataset and R-GTSRB represents the ResNet18 model trained using the GTSRB dataset.
Task
Baseline
Min
Mean
Median
Max
V-CIFARIO
90.70
89.30
89.71
89.72
90.20
R-CIFARIO
91.97
91.10
91.49
91.51
91.83
V-GTSRB
97.60
96.10
96.99
97.02
97.48
R-GTSRB 98.50
96.81
97.45
97.47
98.15
In Table 1 it could be clearly observed that the accuracy of the watermarked models has similar performance compared with the baseline model. The worst accuracy drops are only around 1 1%, while both mean and median keep very similar performance with the baseline. Concerning there exists randomness in the training procedure, the proposed watermarking method has limited effect on the model performance.
Table The tracing accuracies (%) in data-limited and data-free scenarios with only a single adver- sarial example available.
Case
Task
Bandit
HSJ
NES
SignOPT
SimBA
Mean
V-CIFARIO
48.2
93.4
84.2
55.4
85.3
73.30
R-CIFARI0
54.2
95.5
87.4
65.8
83.0
77.18
Data-Limit
V-GTSRB
42.1
98.7
86.3
56.9
91.0
75.00
R-GTSRB
43.8
98.7
86.3
61.8
86
75.32
V-CIFARTO
66.2
83.9
71.6
85.7
59.2
73.32
R-CIFAR10
69.3
89.4
77.8
90.5
56.4
76.68
Data-Free
V-GTSRB
62.4
92.0
67.5
90.7
56.3
73.78
R-GTSRB
61.8
92.8
73.2
91.5
52.7
74.40
5.2 IDENTIFICATION RESULTS
For identification in the data-limited setting, conduct experiments on 100 copies of models ap- plied with random masks. We set the hyperparameter a to 0.85 for CIFAR10 and 0.5 for GTSRB and test tracing accuracy different attacks. The results in the top half of Table illustrate that our detection method could identify the compromised model successfully in all datasets and net- work architectures which achieves average of 75.2% tracing accuracy with only one adversarial example. Specifically, our detection method is extremely effective to trace HSJ attack, where the average tracing accuracy is over 96.5%. However, it could also be seen that our tracing method has a poor performance on Bandit attack that achieves over 42% accuracy in the worst case, which may be affected by the noisy gradient estimation in the dimension reduction process. However, could be solved by introducing a better watermark design, which we leave as our future work.
As we further limit the accessibility, we trace the compromised model with only one adversarial ex- ample and show the tracing accuracy at the bottom half of Table For the data-free case, we also set hyperparameter / to 0.5 for both datasets. Although the original example is no longer available, could still achieve similar even better tracing accuracy against some attacks. Specifically, the average tracing accuracy 74.55% among different datasets and model architectures which is slightly lower than the data-limited case. However, curacy of around 20% improvement on the Bandit and SignOPT. It is because the data-free score slight perturbation
could see there significant tracing
based on the property that the adversarial examples are more sensitive by different masks and more robust against noisy gradient estimation. Therefore, we could further combine those two scores in the practice have better tracing accuracy based on the available evidence.
8

---

Number of Samppess Under review as a conference paper at ICLR 2023
Results adaptive attack To fully test the robustness of our proposed detectors, we also conducted an adaptive attack where the adversary has full access to the specific watermark embedded in each model. To be noted, it is not practical because users have only black-box access and it is not an easy task to directly infer which pixels are masked because of the noise estimation. The attacker then adds some Gaussian noise within the watermark fool our tracing method. We test the average tracing accuracy across different noise levels on CIFARIO with ResNet18 structure. Our results are shown in Table
Tace ACC
Data Limited Free
92255 Mass 20255 4255 B/255 W255
Noise Level
Figure 3: Average tracing ac- curacy adaptive attack with different random noise levels.
Not surprisingly, we observe significant accuracy drop in the data- limited case when adding random perturbation since we utilize the adversarial perturbation to identify the compromised model. How- ever, we also notice that our data-free detector is not sensitive to random noise, which suggests that our tracing method can still be effective even if the adversary knows the predefined watermark.
5.3 RESULTS ON MULTIPLE ADVERSARIAL EXAMPLES
In the previous experiments, we considered only one adversarial example, which is the most extreme case for forensic investigation. However, here comes a natural question: could the proposed method have a better detection rate if more adversarial examples are collected? In this section, we conduct experiments to answer this question.
We use simple strategy to combine multiple adversarial example scores. That is, we first calculate scores defined Section[4.2]ande Section 4.3 each example, and then add up each score computed over all adversarial examples. Then we take the model with the smallest sum as the compromised model. We then conduct the experiments on 100 copies of the random mask watermarked ResNet18 and VGG16 models for the CIFAR- 10 dataset in both the data-limited and data-free settings. It could be seen in Figure A that the detection rate keeps increasing with the number of adversarial examples. We could get around 97% tracing accuracy on average when adding only adversarial example to current accessibility. And the accuracy will reach 100% if given three or more adversarial examples. It shows our method is quite robust and has great potential to be further improved.
Data: Data- Limited (B-CFARLO)
Data-Limited (ACIFARL1))
Data: Free (8- (F-IF)))))
Data-Free (V-CIFAR10)
Trace ACC (N)
Trsce ACC (%)
Trace ACC ACC (8))
Thace ACC
Berdt
Bendi
Bardi
Sandi
HS|
H4)
H5]
45]
NES
NES
NES
NES
SignCPT
5 SP:
SignOPT
SignOPT
SamilA
mmil
SamiiA
SamIiA
Number of Samples
Number of Samples
Number of Semplee
Figure 4: Tracing accuracy with different numbers of adversarial examples.
CONCLUSION AND LIMITATIONS
In this paper, we develop the first framework for identifying the compromised model from a single adversarial example for the forensic investigation. We first present watermarking method to make the generated adversarial example unique and differentiable. Depending on the accessibility of the original example, two identification methods are presented and compared. Our results demonstrate that the proposed framework has a limited effect on the model's performance and has a high success rate to find the compromised model by only giving single adversarial example. Our framework could further improve the detection rate to near 100% when two more adversarial examples are provided. As this study applied a simple yet effective random mask method to watermark models, future works could comprehensively study different watermarking methods in search of more effi- cient ways to distinguish the models in the setting of this study. Moreover, more experiments could be conducted to design even more effective quantitative scores to identify the adversary in future works.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weak- ness into strength: Watermarking deep neural networks by backdooring. In 27th USENIX Secu- rity Symposium (USENIX Security 18), pp. 1615-1631, 2018.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: a -ffirienfi black-box adversarial attack via random search. In European Conference Computer Vision, pp. 484-501 Springer, 2020.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision- based adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv:1712.04244. 2017.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, pp. 39-57, 2017.
Alicia Carriquiry, Heike Hofmann, Xiao Hui Tai, and Susan VanderPlas. Machine learning in foren- sic applications. Significance, 16(2):29-35, 2019.
Huili Chen, Bita Darvish Rohani, and Farinaz Koushanfar. Deepmarks: A digital fingerprinting framework for deep neural networks. arXiv preprint N0Ni::/804.0364,, 2018.
Huili Chen, Bita Darvish Rouhani, and Farinaz Koushanfar. Blackmarks: Blackbox multibit water- marking for deep neural networks. arXiv preprint arXiv: 1904.00044 2019.
Jianbo Chen, Michael I Jordan, and Martin J Wainwright. Hopskipjumpattakk: A query-efficient decision-based attack. In 2020 ieee symposium on security and privacy (sp), pp. 1277-1294. IEEE, 2020.
Pin-Yu Yu Chen, Huan Zhang, Yash Sharma. Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order opti-
mization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, pp. 15 15-26, 2017.
Minhao Cheng, Thong Le, Pin-Yu Chen, Jinfeng Yi, Huan Zhang, and Cho-Jui Hsich. Query- efficient hard-label black-box attack: An optimization-bae based approach. arXiv preprint arXiv: /0kiv:8807.000 1807.04457 2018.
Minhao Cheng, Qi Lei, Pin- Yu Chen, Inderjit Dhillon, and Cho-Jui Hsieh. Cat: Customized adver- sarial training for improved robustness. arXiv preprint arXfv:2002.06789, 2020a.
Minhao Cheng, Simranjit Singh, Patrick Chen, Pin-Yu Chen, Sijia Liu, and Cho-Jui Hsieh. Sign-opt: A query-efficient hard-label adversarial attack. In ICLR, 2020b.
Bita Darvish Rouhani, Huili Chen, and Farinaz Koushanfar. Deepsigns: An end-to-end water- marking framework for ownership protection of deep neural networks. In Proceedings of the Twenty-Fourth International Conference Architectural Support for Programming Languages and Operating Systems. pp. 485 497, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple black-box adversarial attacks. In International Conference on Machine Learning, pp. 2484-2493.
PMLR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. arXiv preprint arXiv:/8044 180400858 2018a.
Andrew llyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-bor adversarial attacks with bandits and priors. arXiv preprint arXiv:1807.0777 2018b.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. Inter- national Conference on Learning Representations, 2017.
Erwan Le Merrer, Patrick Perez, and Gilles Trédan. Adversarial frontier stitching for remote neural network watermarking. Neural Computing and Applications, 52(13):9233-922 2020.
Sijia Liu, Yu Chen, Xiangyi Chen, and Mingyi Hong. signsgd via zeroth-order oracle. In International Conference on Learning Representations, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant adversarial attacks. International Conference Learning Representations, 2018.
Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wis- tuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, et Adversarial robustness toolbox v1. 0.0. arXiv preprint arXiv: 1807.01069, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Bench- marking machine learning algorithms for traffic sign recognition. Neural networks, 32:323-332, 2012.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. International Conference on Learning Representations, 2014.
Chun-Chen Tu, Paishun Ting, Pun-Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. Autozoom: Autoencoder-base based zeroth order optimization method for attack- ing black-box neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 742-749, 2019.
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin'ichi Satoh. Embedding watermarks into deep neural networks. Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, pp. 269-277, 2017.
Tianhao Wang and Florian Kerschbaum. Attacks digital watermarks for deep neural networks. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2622 2626. IEEE, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on Machine Learning. 2019.
Jialong Zhang, Zhongshu Gu, Jiyong Jang, Hui Wu, Mare Stoecklin, Heqing Huang, and Ian Molloy. Protecting intellectual property of deep neural networks with watermarking. In Proceed- ings of the 2018 on Asia Conference on Computer and Communications Security, pp. 159-172, 2018.
Jiyi Zhang, Wesley Joon-Wie Tann, and Be-Chien Chang. Mitigating adversarial attacks by dis- tributing different copies to different users. arXiv preprint arXiv:2111. (tt::2/11.111611 2021.
11


Reviews:
[{"review_id": "xGbdf7d4C7", "reviewer": "Reviewer_2gko", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an identification mechanism that understands which model has been tested with adversarial examples, by leveraging a random-based watermarking technique that is kept inside the malicious points.\nThe methodology is tested in a black-box scenario, where the attacker can only retrive the score of a target classifier, but they can not access the weights or the data of the model.\nExperiments are conducted in two ways: data-limited analysis refers to the usage of both the original sample and the adversarial one, while data-free analysis refers to tracing the compromised model with just one observation.\nThe authors show that the detection mechanism they implement, based on the scores of the pool of deployed models, is able to spot which one has been attacked in both cases.\nThe authors also test their attack in the presence of an adversary that tries to mimic the watermark by applying random gaussian noise.", "summary_of_the_review": "Technical contribution is not proved, part of the threat model can be better discussed to convey why it is realistic, experimental part regarding the adaptive attack is lacking, many technical flaws and unsupported statements.", "strength_and_weaknesses": "**Strengths**\n* the paper presents an interesting forensics problem, that can be useful to understand if malicious actors are trying to attack a specific model.\n\n**Weaknessess**\n* **Uniqueness is stressed but not proved.** While all the methodology is built on the intuition that watermarking creates unique copies that help analysers to trace either the culprit or the asset under attack, the authors do not analyse if the resulting adversarial examples are really unique among all of the models they use. Such limits the contribution of the paper, and the best suggestion would be a better rephrasing that help the reader in understanding that uniqueness may not be guaranteed in this schema.\n\n* **Data-limited analysis is unclear.** The authors state that there might be cases where both the clean sample and its adversarial version are available, by bringing an example on autonomous driving. While the example per se is not clear to understand, the assumption of having both sample is strong, and not supported in the paper. Also, this would similar to detect the adversarial example and the original point, which is not an easy task (and the authors did not comment on that). The authors need to discuss more why this case is realistic.\n\n* **Adaptive attack is not adaptive.** The authors should test the attack when the defense is known, hence the masking is more accurate than just adding random gaussian noise. The provided experiment is needed, since it is good to show the effect of attack at different capabilities of the attacker, but also further experiments are needed. \n\n* **Wrong or unsupported technical statements.** The paper presents statements that undermine the contribution since they are either false or only partially true.\n    * It is not true that adversarial examples are close to the boundary. If an attack is a minimum norm / minimum distance [a,b,c] then it is true. Otherwise, if the attack is computed with a maximum confidence attack (i.e. maximising the misclassification error), the sample can be found deep inside the target distribution [d, e]. Authors should check [f] for transferability of adversarial examples.\n    * Equation 2 is confusing and not supported by any proof.\n    * the paper says that \"since adversarial examples should be identical to original examples visually, ground-truth label could be easily inferred\" is not supported. As specified before, this problem is neither trivial or easy to formulate. If that would be the case, the authors should discuss in detail the statement.\n    * the authors do not provide any support for the CE loss to be higher / lower when scoring an adversarial example computed on one model or another (Adversarial stability paragraph)\n\n\n[a] Carlini, N., & Wagner, D. (2017, November). Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security (pp. 3-14).\n\n[b] Pintor, M., Roli, F., Brendel, W., & Biggio, B. (2021). Fast minimum-norm adversarial attacks through adaptive norm constraints. Advances in Neural Information Processing Systems, 34, 20052-20062.\n\n[c] Rony, J., Hafemann, L. G., Oliveira, L. S., Ayed, I. B., Sabourin, R., & Granger, E. (2019). Decoupling direction and norm for efficient gradient-based l2 adversarial attacks and defenses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4322-4330).\n\n[d] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018, February). Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations.\n\n[e] Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., ... & Roli, F. (2013, September). Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases (pp. 387-402). Springer, Berlin, Heidelberg.\n\n[f] Demontis, A., Melis, M., Pintor, M., Jagielski, M., Biggio, B., Oprea, A., ... & Roli, F. (2019). Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In 28th USENIX security symposium (USENIX security 19) (pp. 321-338).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "m878VaQoEl2", "reviewer": "Reviewer_fDWu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an adversarial identification technique in a multi-model owner-customer environment setup. The authors have incorporated the watermark-based technique to uniquely identify the network used in the generation of adversarial examples. The watermark somewhat needs to be present in the adversarial example for its unique identification. ", "summary_of_the_review": "While the idea of the paper is interesting; several limitations need attention:\n\n1. The novelty as described in the section above is limited and needs justification along with comparison and contrast with the existing works. How many masks per network are generated and what is the impact of these masks? Are these watermark masks random, fixed, or optimized through the network?\n\n2. Are these masks unique for each attack or need to be different for different attacks?\n\n3. The assumption that an attacker will utilize one of the networks might not be ideal in the real world. Due to the tremendous success of black-box and transferable, it is hard to detect from which networks the images are generated if a different surrogate model or no model information has been used.\n\nCrafting Adversarial Perturbations via Transformed Image Component Swapping,\" in IEEE Transactions on Image Processing, 2022\n\n4. The success of tracing in Table 2 is close to random (50%) for most of the attacks, making the success of this approach questionable.\n\n5. The adaptive attacks can seriously break the proposed defense. \n\n6. Limited evaluation of networks, datasets, and no SOTA attacks makes the proposed study a shallow work.", "strength_and_weaknesses": "+ The idea of identifying the network vulnerable to adversarial attack is interesting and can be explored to develop defenses.\n\n- The vulnerability of the technique against adaptive attacks is a major concern. In literature, several attacks have been developed to showcase the ineffectiveness of existing defense works; which only makes defense a challenging task.\n- The limitation can also be seen from the progress of several effective transferable attacks and hence an attacker might not be utilizing one of the models the owner is sharing.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Vl3qp55UJgA", "reviewer": "Reviewer_RQg3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel framework to trace the compromised model by only using a single adversarial example. The method can be applied in forensic investigation scenarios.", "summary_of_the_review": "The work is novel and interesting, but perhaps needs more experiments.", "strength_and_weaknesses": "The paper is well written. \n\nThe application scenarios described in the paper are interesting and novel to me.\n\nThe experiment results show that their method is effective.\n\nSome concerns are as follows:\n1. What about the performance on large-scale datasets? For instance, ImageNet.\n2. Can the proposed method be applied to other tasks? For instance, detection and segmentation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3xKUNb448VB", "reviewer": "Reviewer_wEyj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a novel technique for detecting which model has been compromised (i.e. an adversarial example has been created targeting that specific model) from a set of deployed models that have been watermarked. Thus, each model deployed has a watermark so that some of the inputs are masked. When an attacker tries to compromise one of these models, the adversarial example contains some of the distinct features of the watermark introduced, which helps to detect the origin of the adversarial example. The authors consider two models: one where the defender knows the original and the adversarial example (data-limited adversary identification) and another one where the defender only knows the adversarial example (data-free adversary identification). The authors tested their proposed method on VGG16 and ResNet-18 models on CIFAR-10 and GTSRB datasets. ", "summary_of_the_review": "Overall, I think that the idea explored by the authors is very interesting. However, the paper still requires more work:\n+ The watermarking method proposed, as well as the detection techniques methods are simple and very intuitive. However, it looks like the adversary can also take advantage of this for evading detection. For instance, if the attacker is aware of the watermark scheme (not necessarily having access to the masks used to watermark the models), the attackers can also introduce bogus information in pixels/features where they think that are part of the mask used for the target model by looking at the perturbation introduced for each pixel. Alternatively, attackers can also play with the confidence level of the crafted adversarial examples, which may difficult the detection by the proposed algorithm. However, the adaptive attack used in the evaluation is too simple (it just adds Gaussian noise) and does not necessarily exploit the weaknesses of the proposed method. Thus, a more thorough analysis on the robustness against adaptive attackers is necessary. \n+ Related to the previous point, for instance, it is not clear to me that the proposed method can be robust against black box attacks generated with procedural noise functions or structure noise patterns, as the one proposed by Co et al. “ Procedural Noise Adversarial Examples for Black-Box Attacks on Deep Convolutional Networks” (CCS, 2019), where the masking of some pixels will be possible ignored by the attack. \n+ The watermarking scheme is possibly too simple, which limits the performance of the proposed method. It is also not very clear, how this can perform in typical computer vision datasets (e.g. ImageNet) with a lot more pixels. On the other hand, the detection performance for some of the attacks is not very good (for a single example). I believe this can be improved with a better watermarking scheme. \n+ As mentioned before, it is odd that the data-limited adversary identification algorithm performs worse than the data-free one in some cases. This suggests that the first algorithm can be improved (by for example using the score in equation (5)). On the other side, the combination of the different elements in the detection algorithms is not explored properly in the experiments. Thus, the parameters alpha and beta are cherry picked (or no explanation is given about how to select them). In this sense, a sensitivity analysis (ablation studies) would be necessary, as well as a mechanism to select these hyperparameters. \n", "strength_and_weaknesses": "Strengths: \n+ The idea of trying to identify the model that has been compromised by an adversary using the proposed scheme with watermarking for forensic investigation is quite original and novel. \n+ The method is fairly intuitive and the paper is written and organized. \n\nWeaknesses: \n+ The paper lacks a more thorough analysis of the capabilities and limitations of the proposed scheme (see below) and to test the robustness against more advanced adaptive attacks (the one considered is a bit naïve). \n+ In the empirical results show that, in some cases, the data-free approach (with less knowledge for the defender) is less effective than the data-limited approach. This shows limitations on the proposed algorithms for the data-limited approach and suggest that the authors can easily improve the detection rate by considering also the score used for the data-free detection in equation (5). \n+ The watermarking technique is perhaps too simple, which I think that is limiting the performance of the proposed detection method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject