Deep Neural Network Fingerprinting by Conferrable Adversarial Examples
Authors: Nils Lukas
Yuxuan Zhang
Florian Kerschbaum
Venue: ICLR 2021 Spotlight (2021)
Keywords: Fingerprinting
Adversarial Examples
Transferability
Conferrability

Abstract: In Machine Learning as a Service, a provider trains a deep neural network and gives many users access. The hosted (source) model is susceptible to model stealing attacks, where an adversary derives a surrogate model from API access to the source model. For post hoc detection of such attacks, the provider needs a robust method to determine whether a suspect model is a surrogate of their model. We propose a fingerprinting method for deep neural network classifiers that extracts a set of inputs from the source model so that only surrogates agree with the source model on the classification of such inputs. These inputs are a subclass of transferable adversarial examples which we call conferrable adversarial examples that exclusively transfer with a target label from a source model to its surrogates. We propose a new method to generate these conferrable adversarial examples. We present an extensive study on the irremovability of our fingerprint against fine-tuning, weight pruning, retraining, retraining with different architectures, three model extraction attacks from related work, transfer learning, adversarial training, and two new adaptive attacks. Our fingerprint is robust against distillation, related model extraction attacks, and even transfer learning when the attacker has no access to the model provider's dataset. Our fingerprint is the first method that reaches a ROC AUC of 1.0 in verifying surrogates, compared to a ROC AUC of 0.63 by previous fingerprints. 

Published as a conference paper at ICLR 2021
DEEP NEURAL NETWORK FINGERPRINTING BY CONFERRABLE ADVERSARIAL EXAMPLES
Nils Lukas, Yuxuan Zhang, Florian Kerschbaum
University of Waterloo
{nlukas, y2536zhang, florian. www.schaaumeeeeater..ooooo.
ABSTRACT
In Machine Learning as Service, a provider trains a deep neural network and gives many users access. The hosted (source) model is susceptible to model steal- ing attacks, where an adversary derives a surrogate model from API access to the source model. For post hoc detection of such attacks, the provider needs robust method to determine whether a suspect model is a surrogate of their model. We propose fingerprinting method for deep neural network classifiers that extracts a set of inputs from the source model that only surrogates agree with the source model on the classification such inputs. These inputs are a subclass of trans- ferable adversarial examples which call conferrable adversarial examples that exclusively transfer with target label from a source model to its surrogates. We propose a method generate these conferrable adversarial examples. We present an extensive study on the irremovability of our fingerprint against fine- tuning, weight pruning, retraining, retraining with different architectures, three model extraction attacks from related work, transfer learning, adversarial train- ing, and two new adaptive attacks. Our fingerprint is robust against distillation, related model extraction attacks, and even transfer learning when the attacker has no access the model provider's dataset. Our fingerprint the first method that reaches ROC AUC of 1.0 in verifying surrogates, compared ROC AUC of 0.63 by previous fingerprints.
INTRODUCTION
Deep neural network (DNN) classi- fiers have become indispensable tools for addressing practically relevant problems, such as autonomous driv- ing (Tian et al., 2018), natural lan- guage processing (Young al. 2018) and health care predictions (Esteval al.) 2019) While DNN pro- vides substantial utility, training a DNN is costly because of data preparation (collection, organization, and cleaning) and computational sources required for validation of
Reference Model
"norse", "truck", "dag" Predicted Labels
Source
Model
'car' 'frog". "airplane" Predicted Labels
Knowleddg Distilation
Surrogate
car" "frog" "airplane"
Conferrable Adversarial Examples
Fingerprints for the Source Model
Model
Precicled Labels
Linked Model
Figure 1:
conferrable adversarial examples used as fingerprint to identify surrogate models.
model (Press) 2016). For this reason,
DNNs are often provided by single entity and consumed by many, such as in the context of Machine Learning as Service (MLaaS)). threat to the provider model stealing, in which an adversary derives surrogate model from only API access to source model. We refer to independently trained model for the same task as a reference model.
Consider MLaaS provider that wants to protect their service and hence restrict its redistribution, e.g., through a contractual usage agreement because trained models constitute their intellectual prop- erty. A threat to the model provider is attacker who derives surrogate models and publicly deploys them. Since access to the source model has to provided, users cannot be prevented from deriving surrogate models. Krishna al. (2019) have shown that model stealing is (i) effective, because even

---

Published as a conference paper at ICLR 2021
high-fidelity surrogates of large models like BERT can be stolen, and (ii) efficient, because surrogate models can be derived for a fraction of the costs with limited access to domain data.
This paper proposes DNN fingerprinting method to predict whether model is (stolen) surrogate or a (benign) reference model relative to a source model. DNN fingerprinting is a new area of research that extracts persistent, identifying code (fingerprint) from an already trained model. Model stealing can be categorized into model modification, such as weight pruning (Zhu & Gupta 2017), or model extraction that uses some form of knowledge distillation (Hinton et al. 2015) to derive a surrogate from scratch. Claimed security properties of existing defenses ((Adi et al. 2018
Zhang et al.] 2018)), ) have been broken by model extraction attacks (Shafieinejad et al. 2019). Our fnnerpprinting method is the first passive defense that is specifically designed towards withstanding model extraction attacks, which extends to robustness against model modification attacks.
Our research provides new insight into the transferability of adversarial examples. In this paper, we hypothesize that there exists a subclass of targeted, transferable, adversarial examples that transfer exclusively to surrogate models, but not to reference models. We call this subclass conferrable. Any conferrable example found in the source model should have the same misclassification a surrogate model, but different one in reference models. We propose metric to measure conferrability and an ensemble adversarial attack that optimizes this new metric. We generate conferrable examples as the source model's 's fingerprint.
Retrained CIFAR-IO surrogate models can be verified with a perfect ROC AUC of 1.0 using our ROC AUC of 0.63 for related work (Cao al., 2019). While our fingerprinting scheme is robust to almost all derivation and extraction attacks, we show that some adapted attacks may remove our fingerprint. Specifically, our fingerprint is not robust to transfer learning when the attacker has access to a model pre-trained on ImageNet32 and access to CIFAR- 10 domain data. Our fingerprint is also not robust against adversarial training (Madry al., 2017) from scratch. Adversarial training is an adapted model extraction attack specifically designed to limit the transferability of adversarial examples. We hypothesize that incorporating adversarial training into the generation process of conferrable adversarial examples may lead to higher robustness against this attack.
fingerprint, compared
2 RELATED WORK
In black-box adversarial attacks (Papernot et al., 2017 Tramer et al. 2016 Madry et al. 2017), access to the target model is limited, meaning that the target architecture is unknown and computing gradients directly is not possible. Transfer-based adversarial attacks (Papernot et al. 2016 2017) exploit the ability of an adversarial example to transfer across models with similar decision bound- aries. Targeted transferability additionally specifies the target class of the adversarial example. Our proposed adversarial attack is a targeted, transfer-based attack with white-box access to a source model (that should be defended), but black-box access the stolen model derived by the attacker.
Liu et al. (016)) and Tramèr et al. (2017a) show that (targeted) transfcrability can be boosted by optimizing over an ensemble of models. Our attack also optimizes over ensemble of models to maximize transferability to stolen surrogate models, while minimizing transferability to inde- pendently trained models, called reference models. We refer to this special subelass of targeted transferability as conferrable. [Trmmer et al. (2017a) empirically study transferability and find that transferable adversarial examples are located in the intersection of high-dimensional "adversarial subspaces" across models. We further their studies and show that (i) stolen models apprehend ad- versarial vulnerabilities from the source model and (ii) parts of these subspaces, in which conferrable examples are located, can be used in practice to predict whether a model has been stolen.
Watermarking of DNNs is a related method to DNN fingerprinting where an identifying code is embedded into DNN, thereby potentially impacting the model's utility. |Chhida et al. (2017) em- bed a secret message into the source model's weight parameters, but require white-box access to the model's parameters for the watermark verification. Adi et al. (2018) and Zhang et al.) (2018) propose backdooring the source model on a set of unrelated or slightly modified images. Their approaches allow black-box verification that only requires API access to the watermarked model. Frontier- Frontier-Stichhing (Merrer et al. 2017) and BlackMarks (Dong et al.] 2018)) use (targeted) adversar- ial examples watermarks. These watermarks have been evaluated only against model modification
2

---

Published as a conference paper at ICLR 2021
attacks, but not against model extraction attacks that train surrogate model from scratch. At least two of these watermarking schemes (Adi et al., 2018; Zhang et al. 2018) are not robust to model extraction attacks (Shafreinejad et al. 2019). Cao et al. (2019) recently proposed a fingerprinting method with adversarial examples close to the source model's decision boundary. We show that their fingerprint does not withstand retraining as model extraction attack and propose fingerprint with improved robustness to model extraction attacks.
3 DNN FINGERPRINTING
Threat Model. The attacker's goal is to derive a surrogate model from access to the defender's source model that (i) has a similar performance (measured by test accuracy) and (ii) not verified as a surrogate of the source model by the defender. We protect against an informed attacker that can have (i) white-box access to the source model, (ii) unbounded computational capacity and (ii) access to domain data from the same distribution. A more informed attacker can drop information and invoke all attacks of a less informed attacker. Robustness against a more informed attacker implies robustness against less informed attacker. Our attacker is limited in their access to ground- truth labeled data, otherwise they could train their own model and there would be no need to steal model. In our evaluation, we experiment with attackers that have up 80% of ground-truth labels for CIFAR-I0 (Krizhevsky et al.). A practical explanation for the limited accessibility of ground truth labels could be that attackers do not have access a reliable oracle (e.g.. for medical applications), or acquiring labels may be associated with high expenses (e.g., Amazon Mechanical T.r
The defender's goal is to identify stolen surrogate models that are remotely deployed by the attacker. Their capabilities are (i) white-box access to the source model, (ii) black-box access to the target model deployed by the attacker and (iii) limited set of n. queries to verify remotely deployed surrogate model. Black-box access to the suspect model excludes knowledge of the suspect model's architecture or attack used to distill the model. The defender also does not have knowledge of, nor control over the attacker's dataset used to steal the model.
Fingerprinting Definitions. A fingerprinting method for DNNs consists of two algorithms: (i) A fingerprint generation algorithm that generates a secret fingerprint F C X of size n, and a fingerprint verification key F, S )"; (ii) A fingerprint verification algorithm that verifies surrogates of the source model. These algorithms can be summarized as follows.
Generate M, D): Given white-box access to a source model M and training data D E D. Outputs fingerprint F and the verification keys F, (x) E F}.
Verify(M(F)),FF) Given black-box access to a suspect model M, a fingerprint F and a verification key Fy. Outputs 1 if M is verified by the fingerprint and 0 otherwise.
The verification algorithm computes an error rate between the outputs of the source and target model on the fingerprint. We empirically measure the error rate separately for surrogate and reference models, which allows to choose a decision threshold p E [O, 1]. If the error-rate of target model exceeds os the verification predicts the target model to be reference model, otherwise the prediction is surrogate model. We define that a fingerprint must be irremovable, i.e., surrogate and reference models are correctly verified, despite an attacker's removal attempt. It must also be non-evasive, meaning that an attacker cannot evade black-box verification by detecting fingerprint queries. We refer to Appendix A.3 for security games of DNN fingerprinting.
4 CONFERRABLE ADVERSARIAL EXAMPLES
Motivating Conferrability Conferrability is a new property for adversarial examples, in which targeted transferability occurs only from source model to its surrogates, but not to independently trained reference models. Intuitively. surrogate models are expected to be more similar to the source model than any reference model, but quantifying this similarity is non-trivial. Conferrable examples are an attempt to quantify this similarity by shared adversarial vulnerabilities of the source model
https:/ http///wttttuu.om//////
3

---

Published as a conference paper at ICLR 2021
Surrogate
Surrogates
Class
Advorsarial
Advensarial
Conferrable
Transferable
Cass B
Adversafial
Transferable
Advonarial
(a)
(b)
Figure 2: (a) A summary of the relationship between transferability and conferrability as intersec- tions between the set of all adversarial examples model type. (b) A representation of transferable and conferrable examples the decision space, relative the ground-truth provided by an oracle.
and its surrogates. Surrogate models differ from reference models in the objective function which they optimize. Reference models maximize fidelity the ground truth labels, whereas surrogate models maximize fidelity to the source model's labels, which do not always coincide.
Fig. 2a shows the relation between targeted adversarial, transferable, and conferrable examples for the source model, its surrogates, and all reference models. An example is transferable if it is ad- versarial to any model. It is conferrable when it is adversarial only to surrogate and source models. In that sense, conferrable adversarial examples are subclass of targeted transferable adversarial examples. Fig. [b) shows the relation between transferable and conferrable adversarial examples in the decision space, simplified for illustrative purposes to binary classification. Transferable adver- sarial examples occur those adversarial subspaces where the decision boundary of surrogate and reference models coincide (Tramèr et al. 2017b). Conferrable adversarial examples occur in those adversarial subspaces where the decision boundary of surrogate and reference models differ.
Targeted transferability for class and of models M can be computed follows.
Pr Pr [x) = MEM
Objective We want to find adversarial examples that maximize the output activation difference be- tween surrogate models S and reference models R. This difference quantified by our conferrabil- ity score, which measures the example's transferability to surrogate and reference models.
Confer(S,RR R; Transfer(S, x; (1 Transfer x; t))
(2)
The central challenge for optimizing conferrability directly that require access differen- tiable function Transfer that estimates an example's transferability score. To the best of our knowl- edge, the only known method evaluate transfcrability is to train representative set of DNNs and estimating the example's transfcrability itself. We use this method to evaluate transferability. In the case of conferrable examples, we evaluate transferability on a set of surrogate and reference models trained locally by the defender and use Equation to obtain the conferrability score. Our hypoth- esis that conferrability generalizes, i.e., examples that are conferrable to representative set of surrogate and reference DNNs are also conferrable to other, unseen DNNs.
Conferrable Ensemble Method. In this section, we describe our adversarial attack that generates conferrable adversarial examples, called the Conferrable Ensemble Method (CEM). CEM operates on the source model M, a set of its surrogates SM, and a set of reference models R. The attack constructs ensemble model Mg with a single shared input that outputs conferrability scores for all output classes y E y. CEM generates highly conferrable adversarial examples by finding a perturbation 5 so that x' Zo + 8 maximizes the output of the ensemble model for target class t.
We now present the construction of the ensemble model M The ensemble model produces two intermediate outputs for input & X, representing the average predictions of all surrogate and

---

Published as a conference paper at ICLR 2021
all reference models on We use Dropout (Srivastava et al. 2014) with a drop ratio of d 0.3.
Surr(SM,
Σ
Dropout(S(x):d)
(3)
1SM
SESM
1
R))(R,
Dropout (((p)
(4)
[R]
RER
The output of the ensemble model is the conferrability score of the input for each class. The Softmax activation function is denoted by a. We refer to Appendix A.4 for details about the optimization.
Mg SM, R) = o (Surr(SS x)
(ef(R,))))
(5)
The loss function consists of three summands and is computed over the benign, initial example zo. and the example at an intermediate iteration step x' = zo + We denote the cross-entropy loss by (:,-). The first summand function maximizes the output of the ensemble model for some target class t. The second summand maximizes the categorical cross-entropy between the current and initial prediction for the source model. The third summand minimizes the categorical cross-entropy between the source model's s prediction and the predictions of its surrogates. The total loss L is the weighted sum over all individual losses.
C( x²) = aH max(M
+ (H(M(2') Mz)
Surr(S z'))
(6)
In all our experiments, sensitivity analysis of the hyperparameters. We address the box constraint, i.e., |||| S e, similarly PGD (Madry et al., (017)) by clipping intermediate outputs around the ball of the original input To. For the optimization of an imput with respect to the loss we use the Adam optimizer (Kingma &
use weights a - - -- and refer Appendix A.6 for an empirical
Ba |a 2014). Note that we use an untargeted attack to generate targeted adversarial examples. Tarrgtee attacks are harder to optimize than their untargeted counterparts (Wu & Fu 2019). We assign the
source model's predicted label for the generated adversarial example as the target label and ensure that target classes are balanced in the fingerprint verification key.
Fingerprinting Algorithms. We now describe our fingerprinting generation and verification algo- rithms. For the generation, the defender locally trains a set of e₁ surrogate and 02 reference models (c1 = C2 = 18) on their training data prior to executing CEM. Surrogate models are trained on data labeled by the source model, whereas reference models are trained defender composes the ensemble model Mg as described in the previous paragraph and optimizes for perturbation 5 given input Zo. that L( Fo 5) is minimized. The optimization returns a set of adversarial examples that are filtered by their conferrability scores on the locally trained models. If their conferrability score (see Equation [ exceeds a minimum threshold (T > 0.95), they are added to the fingerprint. For the fingerprint verification, we compute the eтror rate between the source model's 's prediction on the fingerprint and a target model's 's predictions. If the error rate is greater than p. which we refer to as the decision threshold, the target model is predicted to be reference model and surrogate model otherwise.
ground-truth labels. The
EXPERIMENTAL SETUP
Experimental Procedure. We evaluate the irremovability and non-evasiveness of our fingerprint. We show that our proposed adversarial attack CEM improves upon the mean conferrability scores compared to other adversarial attacks, such as FGM (Goodfellow al. 2014), BIM (Kurakin al.) 2016), PGD (Madry et al. 2017). and CW-L (Carlini & Wagner 2017b) We demonstrate the non- evasiveness of our fingerprint by evaluating against the evasion algorithm proposed by HHitj et al. 2 Our study on irremovability is the most extensive study conducted on DNN fingerprints or DNN watermarks compared to related work (Cao et al., [001 Uchida et al., 2017 Zhang et al.) 2018 Adi et al.) 2018 Merrer et al.) 2017; Dong et al. 2018 Szyller et al. 2019). We compare our DNN fingerprinting to another proposed DNN fingerprint called IPGuard (Cao et al.) 2019)
We refer to Appendix A. for more details on the CIFAR- 10 experiments and to Appendix A.2 for results on ImageNet32.
Evaluation Metrics. We measure the fingerprint retention the success rate of conferrable amples in target model M. An example is successful if the target model predicts the target label
5

---

Published as a conference paper at ICLR 2021
0.01 63.00%
0.025 75.00%
0.05 84.00%
0.075 86.00%
0.1 87.00%
0.125 87.00%
0.15
8.
87.00%
Table I: Experimentally derived values for the decision threshold O, for CIFAR-10.
given in Fy. We refer to this metric the Conferrable Adversarial Example Accuracy (CAEAcc).
CAEAce(M((F)F))
Pr (u."")))((()).
1 [y=y']
CIFAR-10 models. We train popular CNN architectures on CIFAR-10 without any modification the standard training process The defender's source model has a ResNet20 (He al. 2016) architecture. All surrogate and reference models required by the defender for CEM are ResNer20 models trained on CIFAR-10 using retraining. The attacker has access to various model architec- tures, such as DenseNet (Tandola et al., 2014). VGG-16/VGG-19 (Simonyan & Zisserman, 2014) and ResNet20 HHe ct al. 2016), to show that conferrability is maintained across model architectures.
Removal Attacks. Removal attacks are successful if (i) the surrogate model has a high test accuracy (at least 85.55% for CIFAR-10, 10, see Fig. 4eh and (ii) the stolen surrogate's 's CAEAcc lower than the verification threshold p. Stolen models derived with a wide range of fingerprint removal attacks, which we categorize into (i) model modification, (ii) model extraction and (iii) adapted model extraction attacks. Model modification attacks modify the source model directly, such as fine-tuning or parameter pruning. Model extraction distills knowledge from a source model into a fresh surrogate model, such as retraining or model extraction attacks from related work (Papernot it al. 2017 Jagielski et al. 2019; Orekondy et al. 2019). A defense against model extraction attacks restrict the source model's 's output to the top- predicted label. We refer to an extraction attack that retrains on the top-1 predicted label by the source model as Black- Black-Box attack. We also evaluate transfer learning as model extraction attack, where a pre-traae pre-trained model from a different domain (ImageNet32) is transfer learned onto the source model's 's domain using the source model's labels.
Adapted model extraction attacks limit the transferability of adversarial examples in the surrogate model, such as adversarial training (Madry et al., 2017). In our experiment, we use multi-step adversarial training with PGD. We design another adapted attacks against our fingerprint called the Ground-True Truth attack, where the attacker has fraction of p E [0.6, 0.7, 0.8) ground-truth labels. We trained surrogate models with Differential Privacy (DP), using DP-SGD (Abadi et al.) 2016), but even for large the surrogate models had unacceptably low CIFAR- 10 test accuracy of about 76%. The test accuracies of all surrogate models can be seen in Table We repeat all removal attacks three times and report the mean values and the standard deviation CIFAR- 10 decision thresholds 6, (see Table [ are chosen relative to the perturbation threshold e with which our adversarial attacks are instantiated.
bars. Our empirically chosen
Evasiveness Attack. Hitaj et al. (2019) present an evasion attack against black-box watermark verification. Their evasion attack trains a binary DNN classifier to distinguish between benign and out-of df-iittribttion images and reject queries by the defender. Our attacker trains binary classifier to classify benign images and adversarial examples generated by FGM, CW-Loc. and PGD adversarial attacks. The evasion attack is evaluated by the area under the curve (AUC) of the receiver operating characteristic (ROC) curve for varying perturbation thresholds.
Attacker Datasets. We distinguish between attackers with access to different datasets, which are CIFAR- 10 (Krizhevsky et al.). CINIC (Darlow et al.) 2018) and ImageNet32 (Chrabaszez et al.)
2017). CINIC is an extension of CIFAR-I0 with downsampled images from ImageNet (Deng al. 2009 for classes that are also defined in CIFAR-10. ImageNet32 is a downsampled version of images from all classes defined in ImageNet, i.e., ImageNet32 is most dissimilar to CIFAR-10 10. We observe that the similarity of the attacker's 's to the defender's 's dataset positively correlates with the surrogate model's test accuracy and thus boosts the effectiveness of the model stealing attack.

---

Published as a conference paper at ICLR 2021
Conforobiilty Scores
Detectabillty
0.0
1.00
OcT
0.2
FIL
0.1
0.
FILL
0.32
0.61
1.06
0.13
0.50
0.12
033
3.00
0.54
0.05
0.00
0:32
0.32
0.14
0.22
3.4
0.00
610
012
034
(a)
(b)
(c)
ght Pruning
Berraining Afrak
Secraining with Model fechFrchree
1.0
0.8
0.0
0.-
Deciton Trmmth
Dediler Thrnshd
02
2-03
CMd Heads
Comsentt
0.2
0.58
8.09
118
032
012
036
3.00
0.44
0.99
C.10
532
034
0.32
0.04
5.6
0.0
0.0
032
614
(d)
(c)
(f)
Figure 3: Results for the (a) conferrability scores, (b) non-evasiveness and (c-d) irreovability to model modification attacks. Figures (e.f) show results against model extraction attacks.
6 EMPIRICAL RESULTS
Conferrability Scores. Our empirical results show support in favor of the existence of conferrable adversarial examples. CEM produces adversarial example with the highest conferrability scores compared classical adversarial attacks such FGM, PGD and CW-Loe- We generate = 100 adversarial examples from the same set of CIFAR-I0 test samples, filter those that are non-successful the source model and evaluate conferrability scores as specified by Equation 2 across five unseen CIFAR-10 10 surrogate and reference models trained from scratch. The results in Fig. Sa show that CEM outputs adversarial examples with significantly higher conferrability scores than other attacks for small perturbation thresholds e, which worsen as increases. For = 0.01 we measure a mean mean CAEAcc of 0.42 in reference models and
conferrability score of 0.49, which amounts 0.85 in surrogate models (see Fd). For 0.15 we measure conferrability score of only 0.24, which translates to mean CAEAce of 0.98 in surrogate models and 0.75 in reference models. These results show that with increasing the transferability of generated adversarial examples improves at the expense of lower conferrability. We observed that the conferrability scores measured on surrogate and reference models used in CEM are nearly perfect. This indicates that CEM could benefit from access to an even larger ensemble models.
Non-Evasiveness. We evaluate non-evasiveness against the attack proposed by Hitaj et al. (019)) The results in Fig. 3b show that (i) detectability increases with larger perturbation thresholds e and (ii) for small perturbation thresholds 0.025 measure ROC AUC of only 0.67. This method of detection is too unreliable for the attacker to deploy in practice considering that only fraction of requests would contain the fingerprint. Sufficiently high recall against our fingerprint comes the cost of more false positives, which diminishes the utility of the attacker's model to other users.
Model Modification Attacks. Fig. 3c and 3d show that our fingerprint robust against model modification attacks. We show irremovability against four different types of fine-tuning attacks tried by Uchida et al. (017). which are implemented as follows.
Fine-Tune Last Layer (FTLL): Freezes all layers except for the penultimate layer.
( 2. Fine-Tune All Layers (FTAL): Fine-tuning of all layers.
htts:://rerrasio/eearrlless/ 0 resnet

---

Published as a conference paper at ICLR 2021
Hodel Extract
Tarnfer
Adversanial Trariing
1.0
0.0
677
0.4
Daci
Cosclaf
uacison Thrstaad
USE Rckrenue Surrgane
Tenieer Nocs (DPASaI)
0.5
2.2
0.06
0.50
032
0.14
Dplicn IL.)
(a)
(b)
Ground
Tiai Accuriccy C&E Accus
CFARJIC Aingerpint
1.0
1/6 16.,07247
Decikon Threshoki
03
peg.
EGourd Irm 2.53
0..
Acouns
Poative Note
Figure 4: Fig. (a-d) show the robustness of our fingerprint against (adapted) model extraction attacks. Fig. (e) shows the test accuracy in relation to the CAEAce for surrogate and reference models during training. Fig. (f) shows the ROC curve of our fingerprint compared to IPGuard (Cao et al. 2019).
3. Retrain Last Layer (RTLL): Re-initializes the penultimate layer's 's weights and updates only the penultimate layer, while all other layers are frozen.
4. Retrain All Layers (RTAL): Re-initializes the penultimate layer's weights, but all layers are updated during fine-tuning.
Our results for iterative weight pruning (Zhu & Gupta 2017) show that larger pruning rates p E [0.7, 0.8, 0.9) have greater impact on the CAEAce, but the surrogate test accuracy also dete- riorates significantly (see Table 2}. Note that we show robustness to much higher pruning rates than related work (Cao et al. 2019) The attack for = 0.9 is unsuccessful because the surrogate test accuracy is lower than 85.55% as described in Section 5 For all remaining pruning configurations the fingerprint is not removable.
Model Extraction Attacks. Fig. 3 and show that fingerprint is robust to model extraction attacks, except for transfer learning when the attacker has CIFAR-10 data. Surprisingly, we find that surrogates extracted using CINIC have higher mean CAEAcc values than CIFAR-10 surrogates at a lower test accuracy (-2) 2. 16%). Similarly, we measure significantly lower CAEAcc values for reference models trained on CINIC compared reference models trained on CIFAR-10. This means that it is increasingly difficult for the attacker to remove our fingerprint the more dissimilar the attacker's and defender's datasets become. In Fig. 31 we show that our fingerprint is not removable even across different surrogate model architectures. Fig. 4bgsoosssws that transfer learning removes our fingerprint when the attacker has access the defender's dataset, but our fingerprint not removable when the attacker only has access to CINIC data. We observed that the pre-trained ImageNet32 model exceeds 80% test accuracy after only a single epoch with CIFAR-10, which may be too few updates for fingerprint transfer to the surrogate. The irremovability of fingerprint to wide range model extraction attacks shows that certain adversarial vulnerabilities (that enable conferrable examples)
consistently carried over from the source model to its surrogates. Our data supports that the class of transferable examples can be broken down further into conferrable examples and that known findings for transferable examples extend to conferrable examples.
Adapted Model Extraction Attacks. We now show limitations of our fingerprint against defenses that specifically limit the transferability of adversarial examples. Fig. 4c shows that adversarial
8

---

Published as a conference paper at ICLR 2021
Fine- Fine-Tuning Tuning
Pruning p=0.8 86.97
Source
FTLL 89.59
FTAL
RTLL
RTAL
p=0.7 88.00
p=0.9
89.30
89.57
87.90
87.25
83.36
Retrain
Extraction
ResNet20
Densenet
VGG16
VGG19
CINIC
Jagielski 88.74
Papernot
Knockoff 84.55
89.22
90.88
90.91
90,05
87.06
87.34
Arvin
Transfer Learning
Ground -ruth
CIFAR-10
CINIC
CIFAR-10
CINIC
p=0.5
p=0.6
p=0.7
p=0.8
89.75
88.15
89.59
88.15
89.54
89.13
89.43
89.33
Table 2: Mean CIFAR-10 test accuracies for surrogate models obtained by running the attacks with threefold repetition. Unless stated otherwise, all surrogates are trained on CIFAR-10 images.
training from scratch removes our fingerprint. In CIFAR- surrogates, measure a mean CAEAcc of only 15% for 0.025, which supports claims by Madry al. (2117) that adversarial training increases robustness to transfer attacks. We hypothesize that incorporating adversarial training into the generation process of conferrable adversarial examples leads to higher robustness against this removal attack. The results from our Ground- Truth attack shown in Fig. 4d show that access to more ground-truth labels decreases the CAEAcc in surrogate models. The results show that our fingerprint is not removable for attackers with up 50% ground-truth labels for CIFAR-10.
Confidence Analysis. Fig. 41 we show the ROC curve (false positive vs true positive rate) of our fingerprint verification on unseen, retrained CIFAR-I0 surrogate and reference models. We compare our work with IPGuard (Cao et al.] 2019) and generate n 100 fingerprints in both cases. Our ROC AUC is 1.0, whereas IPGuard has a ROC AUC of only 0.63. These results show that IPGuard is not robust to retraining as model extraction attack. Inspecting adversarial examples generated by IPGuard further, we measure a mean adversarial success rate of 17.61% for surrogate and 16. (6% for reference models. For our fingerprint, Fig. 4e visualizes the difference in CAEAcc for well-trained surrogate and reference models at 0.025. The plot shows that CAEAcc pos- itively correlates with the surrogate's 's test accuracy. We measure mean difference in CAEAcc of about 30% between well-trained surrogate and reference models with our fingerprint.
CONCLUSION
We empirically show the existence of conferrable adversarial examples. Our ensemble adversar- ial attack CEM outperforms existing adversarial attacks such FGM (Goodfellow et al. 2014), PGD (Madry et al. 2017) and CW- (Carlini & Wagner 2017b) in producing highly conferrable adversarial examples. We formally define fingerprinting for DNN classifiers and use the generated conferrable adversarial examples as our fingerprint. Our experiments on the robustness of our fin- gerprint show increased robustness model modification attacks and model extraction extraction attacks. Transfer learning is a successful removal attack when the attacker has access to CIFAR- 10 data, but not when the attacker only has access CINIC. Adversarial training from scratch is the most effective removal attack and successfully removes our fingerprint. We hypothesize that adding adversarial training into the generation process of conferrable adversarial examples may increase robustness against adversarial training. Our experiments confirm the non-evasiveness of our fin- gerprint against detection method proposed by Hitaj et al. (2019). We empirically find that our fingerprint is the first to perfectly verify retrained CIFAR-TO surrogates with a ROC AUC of 1.0.
ACKNOWLEDGEMENTS
We gratefully acknowledge the support of NSERC for grants RGPIN-05849, CRDP1-531199 191, IRC537591, and the Royal Bank of Canada for funding this research.

---

Published as a conference paper at ICLR 2021
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308 -188 2016.
Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weak- ness into a strength: Watermarking deep neural networks by backdooring. In 27th {USENIX) Security Symposium ({USENIX) Security 18), pp. 1615-1631, 2018.
Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong, Ipguard: Protecting the intellectual prop- crty of deep neural networks via fingerprinting the classification boundary. arXiv preprint arXiv:1910.1200 12900, 2019.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, pp. 3-14, 2017a.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP). pp. 39-57. IEEE, 2017b.
Patryk Chrabaszez, Ily Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprini
2017.
Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos Storkey. Cinic-10 10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.033500 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018.
Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. A guide to deep learning in healthcare. Nature medicine, 25(1):24-29 2019.
Galen Andrew, Steve Chien, and Nicolas Papernot. Tensorflow privacy. 2019. URL https:
htggihuub.com/eessffllli... wwwp:r/wwww//rreaaa.eea (accessed July 5, 2020).
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint vrXiv:1503.02533 2015.
Dorjan Hitaj and Luigi V Mancini. Have you stolen my model? evasion attacks against deep neural network watermarking techniques. arXiv preprint
2018.
Dorjan Hitaj, Briland Hitaj, and Luigi V Mancini. Evasion attacks against watermarking techniques found mlaas systems. In 2019 Sixth International Conference on Software Defined Systems (SDS), pp. 55-63. IEEE, 2019.
Forrest landola, Matt Moskewiez, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv:/40k 1869, 2014.
Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. High- fidelity extraction of neural network models. arXiv preprint arXiv: 1909.01838 2019.
10

---

Published as a conference paper at ICLR 2021
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P Parikh, Nicolas Papernot, and Mohit lyyer. Thieves on sesame street! model extraction of bert-based apis. arXiv preprint arXiv: 12366, 2019.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). URL http:://wwww....ooorrttttttt/tss
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv: 1r0.v0060.025533 2016.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial exam- ples and black-box attacks. arXiv preprint 1rXiv:161 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, arXiv preprint arXiv:1706 (ale::170.000000 06083, 2017.
Erwan Le Merrer, Patrick Perez, and Gilles Trédan. Adversarial frontier stitching for remote neural network watermarking. arXiv preprint 0rXiv:/711.01800 2017.
Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, lan M Molloy, et al. Adversarial robustness toolbox v0. 4.0, arXiv preprint arXiv:180..01069 2018.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv: arXiv:1605.072: 1605.07277, 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506-516. ACM, 2017.
Gil Press. Cleaning big data: Most time-consuming, least enjoyable data science task, survey says. 2016. URL ottp:///t//////ooorooooootttatta
preparation-iee time htt.::/ww......tt.teeoeeeoaaaaaa
task-surven suss--surve--aar says/ Accessed: 2020-07-05.
Masoumeh Shafieinejad, Jiaqi Wang, Nils Lukas, and Florian Kerschbaum. On the robustness of the backdoor-based watermarking in deep neural networks. arXiv preprint arXiv: 0906::/99000777 07745, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting, The journal of machine learning research, 15(1):1999-1958 2014.
Sebastian Szyller, Buse Gul Atli, Samuel Marchal, and N Asokan. Dawn: Dynamic adversarial watermarking of neural networks. arXiv preprint arXiv: 1906 006.00830, 2019.
Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. Deeptest: Automated testing of deep- neural-networr-ddiien autonomous cars. In Proceedingy of the 40th international conference on software engineering, pp. 314 ACM, 2018.
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. 25th [USENIX} Security Symposium ((USENIX) Security 16), pp. 601-618, 2016.

---

Published as a conference paper at ICLR 2021
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv: 2017a.
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv: (eti:::700003555 2017b.
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin ichi Satoh. Embedding watermarks into deep neural networks. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, pp. 269-277. ACM, 2017.
Junde Wu and Rao Fu. Universal, transferable and targeted adversarial attacks. arXiv preprint arXiv: 1132, 2019.
Tom Young. Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. Recent trends in deep learn- ing based natural language processing. ieee Computational intelligenCe magazine, 13(3):55-75, 2018.
Jialong Zhang, Zhongshu Gu, Jiyong Jang, Hui Wu, Mare Ph Stoecklin, Heqing Huang, and Ian Molloy. Protecting intellectual property of deep neural networks with watermarking. Proceed- ings of the 2018 on Asia Conference on Computer and Communications Security, pp. 172-172, 2018.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, arXiv preprint arXiv: arXiv: 1710.0188, 2017.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 SUPPLEMENTARY MATERIAL FOR CIFAR-10 EXPERIMENTS
Setup. We train all models on a server running Ubuntu 18.04 in 64-bit mode using four Tesla P100 GPUs and 128 cores of an IBM POWERS CPU (2.4GHz) with up to 1TB of accessible RAM. The machine learning is implemented in Keras using the Tensorflow v2.2 backend, and datasets are loaded using Tensorflow Datase 3 We re-use implementations of existing adversarial attacks from the Adversarial Robustness Toolbox v1.3.1 (Nicolae et al. 2018). DP-SGD (Abadi et al. 2016) is implemented through Tensorflow Privacy (Galen Andrew Steve Chien, and Nicolas Papemot 2019). All remaining attacks and IPGuard (Cao et al. 2019) are re-implemented from scratch.
For the generated adversarial examples using attacks like FGM, PGD and CW-L we use the following parametrization. We limit the maximum number of iterations in PGD to 10 and use a step- size of 0.01. For FGM, we use step-size of and for CW-Ls we limit the number of iterations to 50, use a confidence parameter k = 0.5, learning rate of 0.01 and a step-size of 0.01. Fingerprints generated by CEM for CIFAR-10 can be seen in Fig. 5a
(a)
(b)
Figure 5: Fig. (a) shows conferrable examples generated with CEM on CIFAR- 10 with e = 0.025.
Fig. (b) shows conferrable examples generated with CEM on ImageNet32 with 0.1.
Analyzing Conferrable Examples. We further examine our fingerprint to find an explanation for the large mean CAE accuracy in all reference models of more than 50%, even though the haseline for random guessing is just 10%. For this, we plot the confusion matrix for our fingerprint at c = 0.025 in Fig. 6
CC
Figure 6: Confusion matrix for our CIFAR-10 fingerprint at e = 0.025 with the initial label on the vertical axis and the target label on the horizontal axis. Depicted values normalized and represent percentages.
The confusion matrix shows that all classes are represented in the fingerprint and the target class distribution is balanced. We notice symmetry in the confusion matrix along the diagonal line. A
13

---

Published as a conference paper at ICLR 2021
source-target class pair appears more often in our fingerprint if the classes are more similar to each other, c.g. classes 'dog' and 'cat' or "automobile" and 'truck'. We hypothesize that this partially explains why reference models have large CAE accuracies of over 50%. even though the baseline for random guessing would be at 10%. A reference model is more likely to misclassify the class "cat' the class 'dog' than any other class, which moves the baseline closer to 50% for many cases. The similarity of the reference model to the source model is another important aspect to explaining the high CAE accuracy in the reference model. Our experiments show that reference models trained on the same dataset are also more likely to share the same adversarial vulnerabilities.
A.2 EXPERIMENTS ON IMAGENET32
We select images from the following 100 classes of ImageNet32:
"kit fox', 'Persian cat', 'gazelle', 'porcupine', 'sea lion', 'killer whale', 'African elephant', "jaguar', "otterhound', 'hyena', 'sorrel', 'dalmatian', 'fox squirrel', "tiger', 'zebra', 'ram', 'orangutan' 'squir- rel monkey', , 'komondor', guiiee pig' golden retriever', 'macaque', 'pug'. 'water buffalo', 'Amer- ican black bear', giant panda', 'armadillo', 'gibbon', 'German shepherd', 'koala', 'umbrella', 'soc- cer ball', 'starfish', grand piano', "laptop', 'strawberry', 'airliner', 'balloon', 'space shuttle', 'air- craft carrier', 'tank', 'missile', 'mountain bike', 'steam locomotive', 'cab', 'snowplow', 'bookcase', 'toilet seat', "pool table', 'orange', 'lemon', 'violin'. 'sax', 'volcano', 'coral reef", 'lakeside', 'ham- mer', 'vulture', 'hummingbird', 'flamingo', "great white shark', 'hammerhead', 'stingray", , "barra- couta', 'goldfish', "American chameleon' "green snake", 'European fire salamander', 'loudspeaker', 'microphone', 'digital clock', 'sunglass', 'combination lock', 'nail', 'altar', 'mountain tent', 'score- board', 'mashed potato', 'head cabbage', "cucumber', "plate', 'necklace', 'sandal', 'ski mask', "teddy', 'golf ball', 'red wine', 'sunscreen', 'beer glass', 'cup', 'traffic light', 'lipstick', 'hotdog', "toilet tissue', 'cassette", 'lotion', 'barrel', 'basketball', 'barbell', 'pole'
We train ResNer20 source model and locally retrain e₁ 14 surrogate and c2 15 reference models. In contrast to the experiments on CIFAR-10, we allow the defender access to various model architectures, such ResNet56, Densenet, VGG19 and MobilenetV2. We want to verify if the generation of conferrable adversarial examples is feasible when the defender trains set of models with larger variety in model architecture. In practice, may be the case that the defender locally searches for the best model architecture and in the process obtains multiple surrogate and reference models with various model architectures.
Table and 4 show the test accuracies and CAEAce values of ImageNet32 surrogate and reference models. Values in the brackets denote the lowest and highest value measured. Our results for ImageNet32 are comparable to the results obtained with models trained on CIFAR-10. We even measure a lower CAE accuracy in reference models than did in reference models trained on CIFAR-10. Note that the experiments conducted on ImageNet32 are results reported for 0.15 and the results for CIFAR- 10 were generated with a perturbation threshold of 0.025.
A.3 FINGERPRINTING DEFINITIONS
Train
R(F
Classify
Verify
b=0
0f
Generate
Train
Classify
Verify
b=1
S(F)
D
Distill
Figure 7: A schematic illustration of the source model and the two types of models, the surrogate and the reference model H, that a fingerprint verification should distinguish. 'Distill' any distillation attack that results in a surrogate model with similar performance as the source model and "Classify' returns the output of a model on a set of inputs.
In this section, we formally define the term 'surrogate' and provide security games for irremovability and non-evasiveness of DNN fingerprinting. For ease of notation, we define an auxiliary function to
14

---

Published as a conference paper at ICLR 2021
Fine- Fine-Tuning
Attack
Source
FTLL
FTAL
RTLL
RTAL
Test Acc
55.70
57.30
61.30
59.00
45.00
CAEAcc
100.00
100.00
100.00
100.00
82.00
Retrain
Attack
ResNet20
ResNet56
Densenet
VGG19
MobileNetV2
Test Acc
53.10
54.50
50,95
52.50
52.87
CÁEAce
97.00
99.00
90.00
76.00
97.00
Extraction
Attack
Jagielski
Papernot
Knockoff
Test Acc
53.20
50.90
47.40
CAEAcc
98.00
90,00
98.00
Table 3: Surrogate model accuracies for ImageNet32.
ResNer20
ResNet56
Densenet
VGG19
MobileNetV2
Test Acc
55.00
59.10
55.95
54.10
62.90
CAEAcc
47.00
50.00
47.00
34.00
61.00
Table 4: Reference model accuracies for ImageNet32.
generate a fingerprint for a source model.
FModel0:
1. Compute M Train 0, D)
2. Sample (F, <- Generate [M, D)
3. Output (M,F.Fy)
Definition of Surrogates. We define the randomized process 'distill' that on the input of a source model M and dataset D E D, returns distilled surrogate of the source model. Note that other common methods of derivation, such as retraining, fine-tuning, and pruning on labels provided by the source model, can be expressed in terms of knowledge distillation (Hinton et al. 2015).
For given source model M we recursively define the set SM of its surrogate models as follows.
SM + (distill(S, D)JS E SM U (M)}
(8)
a model R E M is trained from labeled dataset without access to S and hence is not contained in the set of surrogate models for source model M, refer to that model reference model relative to M. The goal this paper is to find a method that verifies whether surrogate model is in SM-
Irremovability. The attacker accesses the source model M to derive a surrogate M + A(M). We say fingerprinting is not removable if the adversary has low probability of winning the following security game.
1. Defender computes (M,F,F,) + FModel()
2. Obtain Mo Train 0, D) and M, A(M)
3. Sample - {0,1) 1} and send M, to the Defender
4. Adversary wins
Pr|Verr (M)) = Pz 0.5
Note that definition of robust fingerprint differs from related work (Cao et al. 2019 because we include in the set of removal attacks A also model extraction attacks. Fig, schematically shows the types of models that fingerprint verification should distinguish.
Non-Evasee vann-evsiieessss Hitaj & Mancini (2018) show that an attacker can evade the black-box verifica- tion by rejecting queries returning random labels when the verification process is detectable. We
15

---

Published as a conference paper at ICLR 2021
specify non-evasiveness property. which makes it hard to separate members of the fingerprint to benign data samples. A fingerprinting method is non-evasive, if for some E R*, the adversary has low probability of winning the following security game. Let Adv(D, e) be an adversarial attack with maximum perturbation E IR+ performed on the dataset D. An adversarial example with a perturbation smaller than be hard detect (Carlini & Wagner 2017a). The security game is defined as follows.
1. Compute (M,F,F,) < FModel()
2. Adversary trains detector MM + A(M)
3. Sample b _ [0011
4. Set Xy - ed)(D,e) and X2 e and send X₃ to the adversary
5. Adversary wins if
Pr(M((x) = 6) 0.5+
Note that bounds the trade-off between evasiveness and the suspect model's performance. When becomes smaller, more queries from benign users are falsely rejected, or random label is returned at the same level of evasiveness. The goal of the attacker is to find a model that maximizes evasiveness while also maximizing model performance.
A.4 OPTIMIZING CONFERRABILITY
In this section, we formally present the optimization problem that has to solved to find highly conferrable adversarial examples. We denote the set of surrogate models for a source model M by SM and the set of reference models as R.
Conferrability scores should be minimal when all reference and surrogate models assign the same target label (perfect transferability), or none of them assign the target label (non-transferability) An adversarial example that is perfectly transferable to surrogate models and non-trans non-transferable to reference models should have the maximum conferrability score. The following formula satisfies these constraints for target class E V.
Confer(S, R, x;; t) Transfer(S,ar (1 Transfer(R,, t))
Note that Equation does not have weight factors, as it is equally important to transfer to surrogate models as it is to not transfer to reference models.
The optimization constraints find adversarial examples with high conferrability scores can be formalized as follows for benign input æ E X' and perturbation in the infinity norm.
Minimize 8
1. M (a + 8) =t
Pr [[( + 5) =
SES
Pr R(r + =1
RER
subject to ||0]|cc <e
The challenge of generating conferrable examples is to find good optimization strategy that (i) finds local minima close to the global minimum and (ii) uses the least amount of surrogate and reference models to find those conferrable examples. Obtaining many surrogate and reference models that allow optimizing for conferrability is one-time effort, but may be a practical limitation for datasets where training even single model is prohibitively expensive.
A.5 REMOVAL ATTACKS
In this section, we provide details the removal attacks and their parametrization.
16

---

Published as a conference paper at ICLR 2021
a
Confer
Confer
0.1
0.1
0.5
0.20
0.5
1.0
0.1
0.44
0.1
0.1
1.0
0.20
0.5
1.0
0.5
0.37
0.1
0.5
0.20
0.5
0.29
0.1
0.5
0.5
0.20
1.0
0.1
0.1
0.49
0.1
0.5
1.0
0.20
1.0
0.1
0.5
0.42
0.1
1.0
0.1
0.20
0.1
0.40
0.1
1.0
0.5
0.20
1.0
0.5
0.1
0.45
0.1
1.0
1.0
0.20
1.0
0.5
0.5
0.37
0.5
0.20
0.5
0.37
0.5
0.1
0.5
0.20
1.0
1.0
0.1
0.41
0.5
0.1
1.0
0.20
1.0
1.0
0.5
0.42
0.5
0.5
0.20
0.40
0.5
0.5
1.0
0.20
Table 5: An empirical sensitivity analysis of the hyperparametens in Eq. 6
Confer measures the
mean conferrability score over five surrogate and five reference models trained on CIFAR-10.
A.5.1 MODEL EXTRACTION
Retraining uses the source model M to provide labels to the training data and then uses these labels to train the surrogate model S.
S - (r,
(10)
Jagielski et al. (0019) post-process the labels received from the source model by a distillation pa- rameter 7 obtain soft labels. For input 'D and source model M, the soft labels (x)) can be computed as follows.
M'(x): exp(M(x))/T')
(11)
Σ, (x)(M(
The surrogate model is trained on the soft labels.
S - Train(M', D)
(12)
Papernot et al. (017) propose training the surrogate model iteratively starting with an initial dataset Do that is concatenated after each round with set of the surrogate model's adversarial examples.
Su + Train Do)
(13)
D+11 + {a X (ign(J E D.UUD, S; + Train(M, D++1)
(14)
(15)
Js denotes the Jacobian matrix computed on the surrogate model S for the labels assigned by the source model M. We use the FGM adversarial attack generate adversarial examples, as described by the authors.
Knockoff (Orekondy et al. 2019) uses cross-domain transferability to derive surrogate model. They construct transfer set, which cross-domain data, used to derive the surrogate model. We implement the random selection approach presented by the authors
A.6 EMPIRICAL SENSITIVITY ANALYSIS
We perform grid-search to understand the sensitivity of the choice of hyperparameters in Eq. on the mean conferrability score over n = 50 inputs. Mean conferrability is evaluated over five sur- rogate and five reference models trained on CIFAR-10, which have not been used in the generation process of the conferrable examples. We evaluate all parameters in the range [0.1, 0.3, 1.0] and leave
out all combinations where 3 7 except for the baseline used in this paper (a 3 = 1). The examples are generated for = 0.025 with an Adam optimizer (Kingma & Bal 2014) with learning rate 5e-4 and optimize for 300 iterations.
17

---

Published as a conference paper at ICLR 2021
The results illustrated in Table We find that when is small, the optimization does not converge and the source model does not predict the target label. When a is large relative to the other parameters, measure the highest mean conferrability. For 1.0 and 3 = y = 0.1 we measure a mean conferrability of 0.49, which improves over the paper's baseline.
18


Reviews:
[{"review_id": "hab_VamPRtG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies fingerprinting a neural network model by using adversarial example techniques. The idea itself is interesting enough, the this work presents a neat development toward solving this problem. An important issue with this problem is to distinguish a reference model from a stolen model. Thus a desire property of the fingerprint adversarial example is to mislead all surrogate models but non reference models. Since adversarial examples are typically transferable to reference models, thus it is important to distinguish a fingerprint from a transferable adversarial example. For a long time, researchers do not have an answer to whether this is possible, and this work provides an evidence that it may generate a conferrable but not transferable adversarial example to achieve the goal. \n\nThe idea is pretty simple: we construct a conferrable score function to induce all surrogate models to produce a target label; while all reference models to produce different labels. The experiments show that such an approach can indeed produce some fingerprint adversarial examples to distinguish between reference models and surrogate models to a certain degree. \n\nHaving said this, the results are not perfect. For example, Fig 3(b) shows that FGM and PGD have better detectability AUC when epsilon is larger. Although the results are not perfect, this shows promising initial results toward an interesting research domain.\n\nThe presentation has some issues that can be fixed by revision. The figures are generally too small to read. Authors can increase them by leveraging one more page. Also, it's necessary to include the definition of FTLL, FTAL, RTLL, RTAL in the main text. Again, one more page should be able to fix the issue.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LgJbZQ-yUo_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper introduces an interesting property of adversarial examples, which is called conferrability and can reflect the abilities whether an instance can exclusively transfer with a target label from a source model to its surrogates. A new method is proposed to generate conferrable adversarial examples. Experimental results show the effectiveness of the proposed method. The most impressive thing is the AUC of this method in verifying surrogates.\n\nComments:\n\nI'm not familiar enough with the field to judge the novelty and soundness of the proposed method. But I'm willing to roll with the main idea of the paper. Conferrability is the core contribution of the paper. The existence of the conferrable adversarial examples is shown empirically in the paper. Ensemble adversarial attack CEM is shown effective in the same time.\n\nStrengths:\n\n1.The research on this issue has extremely high application value and practical significance. The problem is very important for providers, especially when they want to protect their intellectual property. \n\n2.This paper is well motivated, and thus it is enjoyable to read.\n\n3.The idea of conferrability is very interesting. The existence of the conferrable adversarial examples is shown empirically to prove the value of this definition.\n\n4.Impressive empirical results. Fingerprint in this paper is the first method that reaches an AUC of 1.0 in verifying surrogates, which is impressive for me. Because I'm not familiar enough with the field, I'm not sure whether the method is SOTA in more benchmarks.\n\n5.Interesting figures make the paper easy to follow. I suggest the authors replace Figure 2(b) with a vetorgram to improve the quality.\n\nWeaknesses:\n\n1.There are 3 hyperparameters in equation 5, sensitivity analysis may be needed.\n\n2.Is decision threshold sensitive in different datasets? It may affect the practicality of the method.\n\n3.Clarity: What is the meaning of $\\sigma$ in equation (4)? Is it an activation function? More clarification may be needed to make the paper easier to read.\n\n4.What is the meaning of 'unremovable'? Is this a formal term in English (irremovable)?", "extra_scores": {}, "extra_text": {}}, {"review_id": "7x2lwURDpVD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "########################################################################## \nSummary:\n\nThe paper considers the fingerprinting of a DNN model via usage of specifically designed adversarial examples.\nThe authors describe a proposed loss function and results of their experiments \n\n########################################################################## \nReasons for score: I vote for weak accept, because the model fingerprinting is an important topic in the modern DL, and authors propose an interesting idea on how one can generate such fingerprints via adversarial attacks\n\nThe authors propose new fingerprinting strategy for an already constructed DL model. They demonstrate, that their method can generate conferrable examples and has a solid performance \n\n######################################################################### \nProposed minor improvements:\n\nAbstract: provider trains a deep neural network and\nprovides many users access -> remove one word \"provide\"\neverywhere: AUC -> ROC AUC, as there can be other curves with areas under them\n training a DNN is costly because of data preparation (collection, organization, and cleaning) and computational\nresources required for training and validating the model -> remove one training; better say \"validation of a model\"\nFormulas 2, 3: not clear what does Classify(S, x) mean\nFormula 4: is $\\sigma$ a sigmoid function? please, specify\nFormula 5: is H a categorical cross entropy? please, specify\nFigure 3: what is BIM?", "extra_scores": {}, "extra_text": {}}, {"review_id": "2vlqdG32EuB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a fingerprinting approach for identifying stolen models. To distinguish stolen models from reference models, the proposed approach generated conferrable adversarial examples, which can only be transferred to the stolen models, but not to the reference models. \n\nPros:\n1. The idea of using adversarial examples to identify stolen models is interesting.\n2. The paper provides comprehensive experiments, including model extraction attacks, adaptive model extraction attacks, etc.  The results outperform popular adversarial attacks, FGM, PGD, and CW attacks. \n\nCons:\n1. The key concern about the paper is the unclear threat model. Why does the model stealing attacker have white-box access to the source model? In general, the white-box access means the attacker has all the information about the source model. The definition of “strong attacker” is confusing: If the attacker requires access to domain data, then why the attacker is strong? Do attackers have access to the data? It seems the attackers have only the input data but partial label data, which is a strong assumption. Many recent works show that model stealing attacks can surrogate datasets to extract the victim models. Can these attacks be detected by the proposed approach?\n2. The paper is very hard to follow. Many definitions are missing in the paper. \n3. What is Transfer(S, x; t) in Eq (1)? What is Classify(S, x) in Eq (2) and (3)? What is H in Eq (5)?\n4. In the conclusion section, CW-L2 should be L-infinity?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new idea for detection of model stealing attacks. The new method generates "fingerprint", i.e., adversarial examples that transfer to surrogate models (extracted in model stealing attacks) but not to reference models (i.e., models obtained independently from the same data). If a model owner suspects that some model is stolen, fingerprints can be used for verifications of such claims. 

The paper's contribution is novel and significant. It is the first practical tool, to my knowledge, suitable for a reliable characterization of stolen models. The empirical results are quite impressive demonstrating the detection of stolen models with an AUC = 1.0. Some presentations issues have been addressed by the authors during the revision.  
Decision: Accept (Spotlight)