Towards Stricter Black-box Integrity Verification of Deep Neural Network Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 12:44:38

Abstract: Cloud-based machine learning services are attractive but expose a cloud-deployed DNN model to the risk of tampering.
 Black-box integrity verification (BIV) enables the owner or end-users to ascertain whether a cloud-deployed DNN model has been tampered with via returned responses of only top-1 labels.
 Fingerprinting generates fingerprint samples to query the model to achieve BIV of the model with no impact on the model's accuracy.
 In this paper, we introduce BIVBench, the first benchmark for BIV of DNN models, encompassing 16 types of practical modifications covering typical tampering scenarios. We reveal that existing fingerprinting methods, which focus on a limited range of tampering types, lack sensitivity in detecting subtle, yet common and potentially severe, tampering effectively.
 To fill this gap, we propose MiSentry (Model integrity Sentry), a novel fingerprinting method that strategically incorporates only a few crucial subtly tampered models into a model zoo, leverages meta-learning, and maximizes the divergence of the output predictions between the untampered targeted model and those models in the model zoo to generate highly sensitive, generalizable, and effective fingerprint samples.
 Extensive evaluations using BIVBench demonstrate that MiSentry substantially outperforms existing state-of-the-art fingerprinting methods, particularly in detecting subtle tampering.

Main paper: 
 introduction: 
 Deep neural networks (DNNs) have signicantly advanced the state of the art for many computer vision tasks [18, 37, 47] . To facilitate wide use of DNN models, major cloud operators provide machinelearning platforms, such as Microsoft Azure ML [40] , Google Au-toML [15] , and Amazon SageMaker [2] , to allow customers to deploy their DNN models in the cloud. However, deploying DNN models in the cloud brings several risks. An adversary may tamper with a deployed DNN model via Trojan [36] or backdoor [17] attacks to insert harmful behavior. An unethical model provider might sabotage a competitor's model, such as through a bit-ipping [61] attack, to degrade its performance to gain a competitive advantage. A dishonest cloud service provider may stealthily replace a model with a simpler one, such as a compressed model, to reduce operational costs.To mitigate these threats, both ngerprinting [22, 58] and watermarking [64] are proposed for black-box integrity verication (BIV) of DNN classication models. BIV aims to detect model tampering with only black-box access to the deployed model, i.e., by querying a deployed model which returns only top-1 category labels as responses. The model is detected as being modied if any response to the querying samples disagrees with its ground truth, which is the label returned by the target model (i.e., the unmodied model to protect).Watermarking embeds watermark samples into the model through training or ne-tuning before deployment. Thus it can potentially reduce the model's performance. Moreover, the nite number of pre-embedded watermark samples exposes its vulnerability to watermarking removal where adversaries could remove watermarking by exploiting leaked samples.In contrast, ngerprinting does not alter the model and enables the unlimited generation of ngerprint samples at any time, allowing for its resilience to leakage. Therefore, ngerprinting oers a more robust and exible solution for black-box tampering detection of DNN models compared to watermarking, making it generally the superior choice for ensuring the integrity of deployed models.However, existing ngerprinting methods are evaluated on limited types of model tampering, including model pruning, Trojan, and backdoor attacks, which are all obvious malicious modications. But model tampering could occur in countless ways, subtle or obvious, benign or malicious. Subtle model modications are also common. For example, an adversary can launch subpopulation data-poisoning attacks [27] to compromise the performance of a DNN model on a particular small subset of samples of interest. Furthermore, a DNN model can also be evolving or adapting over time in benign ways [14, 50] , which are benign modications. When dealing with multiple versions of a model that has been updated or tuned for dierent purposes, the service provider may unintentionally operate with a mismatched version of the DNN model. Thus it is practically necessary to verify if a cloud-serviced model is the correct version.In traditional integrity verication of emails and les, it assures that data is unaltered from its original form. Similarly, integrity verication of a DNN model should detect any modications to the model, malicious or benign, signicant or subtle. However, existing works only focus on detecting malicious signicant modications and ignore others. Stricter integrity verication of DNN models is left to be explored. We take the rst step towards such an ultimate goal with the following research question: black-box integrity verication of DNN models for harder-to-observe model modications.A model is said to be observably modied if its decision boundary is observably altered, i.e., there exists an example such that its prediction result received by an end user, e.g., top-1 label for a classication model, is observably dierent from that of the target model. Intuitively, subtler model modications are harder to observe. Stricter integrity verication for detecting harder-to-observe modications is useful, since an end user may need to check if the model he/she is using is really the model he/she wants to use.To ll this gap, we introduce BIVBench, the rst benchmark for black-box integrity verication of DNN models. BIVBench comprises 16 types of practically observable model modications, capturing common tampering scenarios and including 10 new tampering types previously unexplored. Our empirical evaluations conducted using BIVBench reveal that existing SOTA ngerprinting methods are less eective, or even ineective, in detecting harder-to-observe subtle tampering (see Section 3.2 for details). Detecting harder-toobserve model modications such as subtle model modications requires ngerprint samples with higher sensitivity.We then propose MiSentry (Model integrity Sentry), an innovative DNN ngerprinting method with higher sensitivity empowering stricter black-box integrity verication. By maximizing the divergence of the output predictions between the target model and potential tampered models (called model zoo), MiSentry generates ngerprint samples capable of eectively distinguishing seen model tampers. Inspired by our novel empirical observations that ngerprint samples generated with subtler modications can detect more obvious modications and exhibit higher detection sensitivity, MiSentry meticulously selects representative models of only minor modication types hard to distinguish from the target DNN model for constructing the model zoo to maximize the sensitivity of generated ngerprint samples. In addition, MiSentry leverages meta-learning [62] with the model zoo to generate ngerprint samples that can generalize well to unseen tampering types. Extensive experiments on BIVBench validate the high sensitivity and generalizability of ngerprint samples generated by our MiSentry. It outperforms existing ngerprinting methods for DNN models, esp. for subtler malicious tampering and benign modications.Our major contributions can be summarized as follows:• We are the rst to explore black-box integrity verication of DNN models in a stricter form by dening observable modication and considering subtle tampering.  
black-box integrity verification: 
 Both watermarking [64] and ngerprinting [22] are proposed for black-box integrity verication of a DNN model through querying it with responses of only top-1 category labels. Sensitive-Sample Fingerprinting (SSF) [22] is the rst ngerprinting method to detect tampering with DNN models. It generates sensitive and natural-looking samples by approximating and maximizing their sensitivity to the model's tampering, and selects a subset of sensitive samples with the maximal coverage rate of activated neurons as ngerprint samples. Symbolic constraint solvers are used in [12] and Bayesian optimization (BO) is used in [34] to solve SSF's optimization problem in generating sensitive samples. PublicCheck [58] generates ngerprint samples located near the decision boundary with the generative models in a black-box manner, since a model's decision boundary can be sensitive to tampering.All the existing ngerprinting methods focus on detecting backdoor attacks and model compression but ignore subtler model tampering. Model tampering could happen more covertly and subtly. We refer to model tampering with subtler modications to the decision boundary of the target model than the model tampering types discussed in existing literature, such as SSF and PublicCheck, as subtle model tampering. In this paper, we incorporate several types of subtle modications that may occur in practice into BIVBench. These include malicious ones, such as sample-level attacks (Targeted Attack, Degradation-S), and benign ones, like online learning, unlearning, and ne-tuning of the last layer or all layers (see Sec. 3.1 and Appendix C.2 for more details). As to be presented in Section 5, existing ngerprinting methods lack sensitivity or are ineective in detecting such subtler model tampering, while our MiSentry demonstrates its eectiveness in detecting such subtle tampering.Furthermore, unlike SSF-like methods which estimate the ngerprint's sensitivity to tampering through a series of relaxations, our MiSentry directly assesses sensitivity with prediction changes following potential subtle tampering, without any relaxation, thereby allowing for higher sensitivity. Also, in contrast to PublicCheck-like methods where ngerprint samples are located near the decision boundary and could exhibit unusual prediction vectors which can be easily gured out when querying, our MiSentry maximizes output covertness by ensuring prediction vectors of generated ngerprint sample similar to those of normal samples while simultaneously maximizing sensitivity.Model hashing [4, 28, 48] and watermarking techniques [3] also aim for integrity verication, but they operate dierently and cater to distinct threat models from ours. Specically, model hashing requires white-box access to the protected model, which contrasts with our methodology which assumes black-box query access during integrity verication. The black-box approach is more challenging but practical in real-world scenarios. In addition, model watermarking involves tampering with the target model, which could result in performance degradation. MiSentry, however, does not tamper with the target model.The primary advantage of black-box integrity verication over white-box integrity verication is its ability to conduct live and stealthy integrity verication while the model is in service, without being detected by the cloud service provider. This makes it dicult for a dishonest service provider to evade black-box integrity verication. In contrast, white-box integrity verication requires white-box access to the model for integrity verication, rather than during the model's service. This allows a dishonest service provider to provide the authentic model for white-box integrity verication while using a compressed model for servicing users. Black-box integrity verication makes integrity verication more feasible, stealthy, practical, and harder to evade.Black-box integrity verication is more challenging to design than white-box integrity verication since only the top-1 label is available for integrity verication, making the detection of subtle changes signicantly more dicult.Fingerprinting is also used for intellectual property (IP) protection of DNN models [5, 44, 59, 60] . IP protection aims to verify the ownership of a stolen DNN model. Fingerprint samples for IP protection should be robust to model modications that can be employed to obfuscate the model's ownership. In contrast, integrity verication ensures that a model operates as intended and has not been altered. IP protection and integrity verication pursue distinct objectives and possess dierent characteristics. 
bivbench: benchmark for black-box integrity verification of dnns: 
 Existing works only evaluate the performance of black-box integrity verication on limited types of model tampering, including model pruning, Trojan, and backdoor attacks. However, model tampering could occur in countless ways. Subtle and benign model modications are common and their detection is also crucial. To cover common tampering scenarios and ll the gap of detection for unexplored tampering types, we introduce BIVBench, the rst benchmark for black-box integrity verication of DNN models.Traditional integrity verication of emails and les requires that any modications to the data, malicious or benign, should be detected. Far from such much stricter integrity verication, existing ngerprinting methods mainly consider limited types of obvious malicious model which are relatively easier to detect. We aim to enable integrity verication of a DNN model to detect any observable model tampering, where the decision boundary of the model is altered such that there exists an example whose prediction is observably dierent from that of the original model. Such integrity verication should be able to detect any practical modications, benign or malicious, to a DNN model. We come up with the following practical modications that a ngerprinting method should be able to detect. Benign Modications. DNN models may be modied for legitimate reasons. These benign modications include:• Unlearning [14] is to force a model to forget a selected subset of data used to train the model. This is increasingly important with the enactment of privacy protection laws such as GDPR. • Online learning [50] is used to adapt a model to t newlycollected incremental data. • Model compression [7] compresses a DNN model to save computing resources, typically in the following ways: (i) Pruning [41] prunes components with the least contribution to the performance of the model and then retains the lost performance by ne-tuning the rest.(ii) Quantization quantizes a model, i.e., reducing the precision of the model's representation. • Knowledge distillation [23] transfers the knowledge from a teacher model to a student model. By transferring to a smaller model, it also reduces computing resources. • Transfer learning [43] adapts a pre-trained model to a new downstream task. • Fine-tuning adjust the weights of specic layers (called tunable layers) while freezing the weights of other layers. • Retraining diers from ne-tuning by re-initializing the weights of tunable layers before relearning them. Malicious Modications. A model can be maliciously modied via poisoning training data or directly modifying the model to degrade its performance or misbehave only on specic inputs. Malicious modications include:• Poisoning degradation attack [27] degrades the accuracy of a model by tampering with its training data. • Bit-ipping attack [46] degrades the accuracy of a model by ipping the most vulnerable bits of the model. • Targeted attack [30, 51, 54] makes a model mispredict to a specic label on specic input samples but predict normally on other input samples. • Backdoor attack [16, 29] makes a model mispredict to a specic label when a specic trigger is applied to an arbitrary sample but behaves normally otherwise. A backdoor can be inserted into a model by poisoning its training data. • Trojan attack [36] injects a backdoor into a model by rst optimizing the values of a backdoor trigger at a given location without touching the model and then applying the trigger to ne-tune ending layers. • Clean-label attack [63] inserts a backdoor using poisoning data that visually appears consistent with the clean labels. We construct BIVBench, the rst benchmark for DNN black-box integrity verication, with all the benign and malicious modications listed in the above practical model modications.We next comprehensively evaluate black-box integrity verication performance of existing SOTA ngerprinting methods using our BIVBench. The performance is metricized by the tampering detection rate using N S ngerprint samples for each tampering type. Three frequently-used datasets, CIFAR-10 [31] , GTSRB [53] , and ImageNet [49] , and models with various architectures are used. Specically, we use ResNet20 [19] , CNN [35] with 6 convolution layers and 1 full-connected layer, and DenseNet121 [26] as the architectures of the target model for CIFAR-10, GTSRB and Im-ageNet, respectively. Test instances for each tampering type are constructed following the principle of subtlest tampering that tampering action is stopped as soon as the tampering objective is met since subtler modications are harder to detect. We use the existing state-of-the-art (SOTA) ngerprinting methods, SSF [22] and PublicCheck [58] , as the baseline methods. Implementation details about ngerprint generation, tampering test instances, models, and tampering detection are described in Appendix C.The tamper detection rates of SSF, PublicCheck, and MiSentry (which we will propose in the next section) for dierent tampering types using 1 (# ( = 1) and 5 (# ( = 5) ngerprint samples for integrity verication are shown in Table 1 . We can see that some tampering types are harder to detect than others. The detection rate is signicantly lower when model tampering is subtler. On both CIFAR-10 and GTSRB, SSF and PulicCheck both have a signicantly lower tamper detection rate for unlearning, online learning, netuning, targeted attacks, and single-sample poisoning degradation attacks than that for model compression (pruning and quantization), backdoor attacks, Trojan attacks, etc. For example, SSF fails in detecting ne-tuning with a learning rate of 10 5 on these datasets. We conclude that existing SOTA ngerprinting methods are insufciently sensitive in detecting harder-to-observe subtle tampering, calling for ngerprint samples with higher sensitivity.  
misentry: fingerprinting dnns with higher sensitivity: 
 In this section, we rst describe the threat model and present desirable properties of ngerprint samples. To achieve desirable properties, we formalize the generation of ngerprint samples as an optimization problem to distinguish two models, select representative models to construct a model zoo to maximize detection sensitivity and leverage meta-learning to make ngerprint samples generated on a small number of models eective on unseen models. MiSentry's ngerprint generation is outlined in Fig. 1 .Like SSF [22] , we assume white-box ngerprint generation and black-box integrity verication. Specically, a model is under whitebox access when its ngerprint samples are generated. When a model's integrity is veried, we assume the model cannot be accessed directly. Integrity verication can be conducted only by querying the model with samples. When queried, a DNN model returns only the top-1 label, without returning its condence level or the prediction vector. MiSentry can be utilized for both private and public integrity verication. The detection performance remains consistent between public and private integrity verication. The primary dierence between the two types of integrity verication lies in the ngerprint samples' usage: In public verication as shown in Fig. 2 , model owners generate ngerprint samples and send them to a trusted third party, from which interested parties can request samples to verify a cloud model's integrity.In contrast, in a private setting, only the model owner or authorized entities can use the generated ngerprint samples provided by the model owner for verication. It is important to note that public verication makes integrity verication more practical but signicantly increases the risk of ngerprint sample leakage due to the involvement of untrusted users.In addition, we assume that the DNN service provider is not trustworthy: it may detect probing ngerprint samples and evade tampering detection. We also assume that adversaries can get access to ngerprint samples already used for verifying integrity and exploit them to evade tampering detection.With the above threat model, ngerprinting samples should have the following desirable properties: • Sensitivity. Fingerprint samples should be sensitive to any practically observable modications (see Section 3.1 for details) to the target model. • Uniqueness. Fingerprint samples should be able to distinguish the target model from other models of the same or dierent architectures, trained with the same or dierent datasets, that perform the same task as the target model. This property ensures the uniqueness of the target model, i.e., it cannot be replaced by a similarly trained model. It is unexplored by existing methods. • Covertness. To ensure that ngerprint queries can't be distinguished from normal queries and prevent malicious cloud providers from identifying ngerprint queries to evade integrity verication, a ngerprint sample should not be abnormal. As proposed by SSF [22] , a ngerprint sample should look natural. We call this property input covertness. We extend the covertness of input images to output prediction vectors: the prediction vector of a ngerprint sample should also be similar to that of normal samples, denoted as output covertness. Both SSF [22] and Pub-licCheck [58] require the input covertness but do not require output covertness. • Robustness to Partial Leakage. Fingerprint samples already used in verifying integrity may be accessible to adversaries, who can use them to adjust the model to evade tampering detection. The remaining ngerprint samples should be eective in detecting such adaptively modied models. • Generalizability. A ngerprinting method should be generalizable to dierent models and datasets. In addition, ngerprint samples should be eective in detecting all potential practical modications to DNN models. For MiSentry, the latter means that ngerprint samples should be eective in distinguishing the target model from not only the tampered models in the model zoo but also tampered models with unseen tampering types.Natural looking. Covertness requires a ngerprint sample to look natural. To fulll this requirement, we generate a ngerprint sample G from a normal sample G 0 with a distortion bound: ||G G 0 || ?  n for some norm ?. Distinguishablity loss. Suppose we have two models, F ? and F A , where F ? is the target model (i.e., to protect) and F A is a model to reject (i.e., to distinguish from the target model). To generate a ngerprint sample G to distinguish them, we maximize the divergence of the output predictions of the two models over input G: ⇡8E4A64=24 (F ? (G), F A (G)), which can be measured by Kullback-Leibler (KL) divergence. The distinguishability loss corresponding to this inter-model divergence is thus:EQUATIONwhere F ? (G) 9 is the 9-th element in the prediction vector of model F ? on input G. Note that we have taken a negative sign before the KL divergence in Eq. 1 since we minimize this loss term. Entropy loss. The entropy loss is designed to ensure output covertness, which makes MiSentry the only one among the three methods considered that meets the output covertness criterion, as demonstrated in Fig. 5 in Section 5.3. Output covertness ensures that the prediction vectors of ngerprint samples are similar to those of normal samples, making ngerprinting queries indistinguishable from normal queries based on model outputs. This prevents a malicious cloud service provider from identifying ngerprinting queries to evade integrity verication. On F ? , a normal sample has likely a high probability on its true label and low probabilities on others. We also expect that F A behaves similarly to F ? , otherwise it can be easily dierentiated from F ? without relying on a ngerprinting method. Covertness requires that the prediction vector of a ngerprint sample G is indistinguishable from that of a normal sample on both F ? and F A . To fulll this requirement, we minimize the information entropy H of the prediction of G on both F ? andEQUATIONwhere W 2 {?, A } for models F ? and F A , respectively. In addition, we calculate the entropy loss of the normal samples with the same label as G on each model and determine a distribution range [U W , V W ], e.g., 10 C⌘ percentile to 90 C⌘ percentile. We require that the entropy loss of G is within the range on both models:U W  ✓ ⇢ W  V W , W 2 {?, A }. Optimization problem.A ngerprint sample G is generated by meeting the above requirements simultaneously, i.e., by solving the following optimization problem:EQUATIONwithEQUATIONwhere _ 1 and _ 2 are two weighting hyperparameters of dierent loss terms. Eq. 3 can be solved iteratively with the projected gradient descent like in PGD [32, 33] :EQUATIONGeneration diversity. We generate our ngerprint samples starting from dierent source samples. It can generate diverse ngerprint samples widely spread all over the decision boundary of the original model. Each ngerprint sample is located in a distinct local region, which is independent and unpredictable. Thus leaked ngerprint samples can hardly inuence the eectiveness of unleaked ones.We next describe the representative model selection in MiSentry to maximize the detection sensitivity of the generated ngerprint samples while minimizing the number of models selected for ngerprint generation. To make MiSentry more practical, we require the models in the model zoo of meta-learning to be of the same architecture and trained with the same dataset as the target model.In support vector machines (SVMs) [56] , support vectors are the instances that result in the maximal margin and are closest and crucial to the decision boundary of the hyperplane. A model could be tampered with in countless ways, but limited (and preferably as few as possible) tampering instances can be used to generate ngerprint samples. Similar to the support vectors, crucial tampering instances should be identied to generate ngerprint samples capable of distinguishing all practically observable modications.Intuitively, ngerprint samples generated using subtler modications should likely be capable of distinguishing more signicant modications and exhibit higher detection sensitivity. The rationale behind this intuition is that a model modied more subtly has a decision boundary closer to that of the target model. Among models subjected to the same type of modication, the subtlest one typically exhibits the smallest deviation, such as the least samples' prediction variation or the fewest required training epochs, since tampering typically demands a certain number of training samples and steps.To validate the intuition, we conduct the following study: we generate ngerprint samples using models ne-tuned over varying numbers of epochs (1 and 10) and use these samples to detect modications across a range from 1 to 15 epochs. Additionally, we generate ngerprint samples using models updated via online learning with 1 and 10 updated samples and used these to detect updates in models with up to 10 samples. Our ndings from experiments on the CIFAR10 dataset, illustrated in Fig. 3 , demonstrate that ngerprint samples from the subtlest modications are more eective in detecting larger modications and possess the highest detection sensitivity.These results suggest that subtler model modications are more crucial for the generation of eective ngerprint samples, being closer to the detection hyperplane. Consequently, we incorporate these subtlest model modications into our model zoo as "support vectors" for generating ngerprint samples.For models with dierent types of modications, empirical judgment is used to determine which type likely produces fewer perturbations to the decision boundary. It is generally harder to distinguish models of the same architecture and trained with the same dataset than models of dierent architectures or trained with different datasets. That intuition leads to selecting small variations of models of the same architecture and training with the same dataset as the target model into the model zoo for meta-learning (see Appendix C.3 for details).This representative model selection of MiSentry makes ngerprint sample generation more practical since it does not require any model trained with extra data, which may be inaccessible to the model owner due to privacy protection.In addition, by leveraging meta-learning to enhance generalizability [62] (see Appendix A for more details), this limited selection of models does not compromise the eectiveness of generated ngerprint samples in detecting unseen tampering types. As shown in Tab. 1, ngerprint samples generated by MiSentry are highly generalizable: they are eective in distinguishing from the target model not only the models in the model zoo but also models unseen in their generation. 
experimental evaluation: 
 In this section, we evaluate our MiSentry's performance using BIVBench following the same settings as Section 3.2 and conduct additional evaluations on leakage robustness, output covertness, detection sensitivity, each component's eect, and time cost.MiSentry's detection performance is also presented in Tab. 1. We can make the following observations. MiSentry's ngerprint samples are generalizable. Even though MiSentry uses a few types of model modications in the model zoo of meta-learning to generate ngerprint samples, generated ngerprint samples are eective in distinguishing unseen tampering types and independently trained models from the target model. The tampering detection rates of MiSentry are all above 36% using a single ngerprint sample and above 90% using 5 ngerprint samples for the three datasets.MiSentry outperforms existing methods. MiSentry has a reasonable tampering detection rate even for the hardest-to-detect tampering types. It outperforms both SSF and PulicCheck in general and by a large margin for subtler tampering types.Detection rate of a larger-scale dataset and model is generally higher. The detection rates of subtler tampering types, SSF-BO [34] , a variant of SSF, uses Bayesian optimization to generate ngerprint samples. SSF-BO is infeasible for large-scale datasets like ImageNet. We use the code of SSF-BO from [42] to detect model tampering on MNIST [35] , a handwritten digits dataset. The results can be found in Appendix. F. MiSentry surpasses SSF-BO in detecting all the tested model tampering.In public integrity verication, an adversary can collect already used ngerprint samples and use them for an adversarial tampering attack by pursuing a model manipulation goal while preserving the top-1 labels on collected ngerprint samples simultaneously. To evaluate existing SOTA ngerprint methods and MiSentry's robustness to such adaptive attacks, we assume that # ;40:43 already used ngerprint samples are collected and exploited by the attacker, where # ;40:43 represents the number of leaked ngerprint samples, and evaluate the detection performance of the three methods on newly-generated unused ngerprint samples. The leaked ngerprint samples comprise tampering set which is used in the adversarial tampering attacks to evade MiSentry detection, while the unleaked ngerprint samples comprise validation set which is used for integrity verication.To investigate the eectiveness of existing ngerprint methods and MiSentry in such increasingly threatening scenarios, we increase the number of leaked ngerprinting samples, # ;40:43 , from 1,000 to 3,000. Here, we conduct adaptive tampering attacks with backdoor attacks and last-layer ne-tuning respectively since they are the most common malicious and benign model modications.Experimental results of adaptive backdoor attacks are shown in Fig. 4 and those of adaptive last-layer ne-tuning are presented in Appendix E. The black dot line shows the detection rate when no leaked ngerprint samples are exploited, while the blue square line shows the detection rate when # ;40:43 leaked ngerprint samples are exploited for an adaptive tampering attack. The red triangle line shows the detection rate of the leaked ngerprint samples (i.e., on the tampering set). We can conclude that:• For all three methods, more leaked ngerprinting samples could lead to a reduced detection rate. For the adaptive backdoor attacks, when # ;40:43 is 1000, the detection rates of MiSentry, SSF, and PublicCheck can drop to as low as 58.9%, 45.4%, and 44.9%, respectively. When # ;40:43 increases to 3000, the lowest detection rates of MiSentry, SSF, and PublicCheck can further decline to 52.8%, 38.9%, and 39.6%, respectively. A similar tendency can be observed for the adaptive last-layer ne-tuning. However, further increases in the number of leaked ngerprinting samples can no longer signicantly degrade the detection performance.• The detection rate of MiSenrty consistently remains higher than that of SSF and PublicCheck in all cases. When # ;40:43 is 1000, the lowest detection rate of MiSentry is 1.28⇥ that of SSF and 1.31⇥ that of PublicCheck under such an adaptive backdoor attack, respectively. A similar advantage still persists when # ;40:43 is 3000. For the adaptive ne-tuning attack, we nd this advantage of the detection rate of MiSenrty to that of PublickCheck enlarges to 2.25⇥.• While more leaked ngerprinting samples could lead to a reduced detection rate, the detection performance of our MiSentry remained reasonably robust. The detection rate of MiSentry remains consistently higher than 50% and 29% under the backdoor and netuning attack respectively. It is worth noting that SSF is almost incapable of detecting whether the model has been tampered with in the early stages of the adaptive ne-tuning attack. Only with the increase of ne-tuning training epochs does SSF gradually gain the ability to detect potential tampering, albeit the detection rate remains very low, indicating inherently low sensitivity of SSF.The robustness of detection performance to leakage can be attributed to generation diversity. Starting from dierent source samples, our MiSentry can generate diverse ngerprint samples widely spread all over the decision boundary of the original model. Each generated ngerprint sample is independent and unpredictable since it is located in a distinct local region. Thus leaked ngerprint samples can hardly inuence the eectiveness of unleaked ones.Appendix D shows some ngerprint samples generated by MiSentry, SSF, and PublicCheck on the three datasets. These ngerprint samples exhibit a natural appearance, and the perturbations they introduce to the source samples are almost indiscernible. MiSentry and SSF show similar perceptual quality to the original samples, with noise-like artifacts, while PublicCheck's artifacts involve altered texture and color temperature. Their perceptual quality is comparable overall.To avoid being singled out, the output of a ngerprint sample should be indistinguishable from that of a normal sample. Fig. 5 shows the distribution of prediction probabilities of top-1 labels for ngerprint samples generated by MiSentry, SSF, and PublicCheck and normal samples in the test set. We can see that the prediction probabilities of MiSentry are within those of normal samples, while some prediction probabilities of SSF and PublicCheck fall into a region below the bottom 5% low condence of normal samples (shown by the dashed lines), especially for GTSRB, which makes integrity probing using ngerprint samples generated by SSF and Public-Check potentially singled out. Only MiSentry meets the output covertness requirement among the three ngerprinting methods.During the manipulation of the target model, we would like to know how quickly a ngerprinting method can detect the manipulation. We explore this detection sensitivity, i.e., the relationship between tampering detection rate and magnitude of modication starting from the initial untampered model. Fig. 6 shows the tampering detection rates of dierent ngerprint methods at dierent training steps on CIFAR10 for ne-tuning the last layer with a learning rate of 10 5 , targeted attacks, and backdoor attacks. We can make the following observations. First, the detection rate of MiSentry increases more rapidly than that of SSF and PublicCheck, indicating that MiSentry has higher detection sensitivity than SSF. Second, when using 5 ngerprint samples for tampering detection, the detection performance of MiSentry saturates around 100 training steps. Third, SSF fails to detect the ne-tuning, which agrees with that reported in Section 3.2.We conduct an ablation study for each component of our MiSentry to analyze their respective eects on CIFAR10. Tab. 2 presents the tampering detection rates using a single ngerprint sample for dierent model zoo constructions. Additional results are shown in Appendix G. We can make the following important observations. • Subtly modied models (i.e., last-layer ne-tuning) are crucial for detecting all modication types, while other modication types in the model zoo mainly enhance detection performance for the same type.• The generalization ability of ngerprint samples improves with an increasing number of representative models per tampering type but saturates when the number is larger than 6. • Without the ! ? norm bound, the generated ngerprints are noise-like images, which can be easily distinguished from normal samples. However, these noise-like ngerprints have higher detection sensitivity (tampering detection increases up to 0.359⇥).• Without entropy loss, the generated ngerprint samples are more likely to have abnormal prediction vectors, with AUC scores decreasing from 0.651 to 0.397 when detecting ngerprint samples by their condence scores.• MiSentry exhibits similar defection performance across dierent model architectures of the target model, conrming its generalizability.We use a single Nvidia RTX3090 GPU in our experiments. The average time cost of constructing a model for each tampering type and ngerprint sample generation of our MiSentry is shown in Tab. 3. Overall, the time cost of MiSentry increases with the increase in model parameter size. These time costs are acceptable compared to the benets brought by integrity verication. 
conclusion: 
 In this paper, we have introduced signicant advancements in the eld of black-box integrity verication (BIV) for DNN models. We pioneer a stricter form of BIV by focusing on subtle tampering and dening observable modications, setting new standards for BIV. We develop BIVBench, the rst benchmark tailored for stricter BIV. It includes 16 types of tampering scenarios, signicantly more than previous studies, revealing the limitations of existing ngerprinting methods. We also introduce MiSentry, a novel ngerprinting method using a rened set of subtly tampered models to produce highly sensitive and eective ngerprint samples. MiSentry has proven superior in detecting tampering, even those unseen during its generation. To conclude, we establish new benchmarks and methodology for the BIV of DNN models, paving the way for further studies in this crucial eld. 
 
 Appendix:  


Decision: The paper introduces a benchmark for black-box integrity verification of DNN models, highlighting the inadequacy of existing methods in detecting 'subtle' modifications. Besides, the authors propose a new fragile fingerprinting scheme that is non-invasive and highly effective. The reviewers generally recognize the problem's significance, the extensive benchmark, and the paper writing. However, the reviewers still have some critical concerns, especially those regarding the technical contributions and model zoo selection. The four reviewers engaged in an intense discussion during the post-rebuttal phase but they failed to ultimately agreed upon. This brings the score of this paper to the borderline. Note that Reviewer Ksdt has corrected the misunderstanding and provided further explanations at this moment. 

After receiving the rebuttal message from the authors, I contacted Reviewer Ksdt to re-evaluate the previous comments. I have also carefully read all materials, including the submission, all comments, the author's rebuttal, and the reviewer's discussions. Although this paper still has some shortcomings (especially the details and settings for model zoo selection), I think this paper has made sufficient contributions to this field. By trading off the strengths and weaknesses of this paper, I decide to accept this paper. However, I hope the authors can carefully read the post-rebuttal comments from Reviewer 6KCx and try to address them in the final version of this paper. I think it can greatly improve this paper.

Reviews: review: paper_summary: paper_summary: This paper introduces BIVBench, the first benchmark for black-box integrity verification of deep neural networks, covering 16 tampering scenarios.
strength_weakness: strength_weakness: It highlights the limitations of current methods in detecting subtle tampering and proposes MiSentry, a new approach using meta-learning and a model zoo to generate highly sensitive fingerprint samples. Extensive evaluations show MiSentry significantly outperforms existing methods in detecting both subtle and significant tampering.
limitations: limitations: 1. While MiSentry's use of meta-learning is innovative, the paper could benefit from a deeper exploration of the limitations and challenges associated with this approach. For instance, the computational overhead and the potential difficulties in selecting an appropriate model zoo could be further discussed.

2. Although the paper provides extensive empirical evaluations, additional real-world validation would strengthen the claims. Implementing and testing MiSentry in live cloud environments would provide valuable insights into its practical effectiveness and potential challenges.

3. The comparison with existing methods is comprehensive, but the paper could include a broader range of baselines, particularly those from related fields such as adversarial robustness and secure model deployment. This would provide a more holistic view of MiSentry's relative strengths and weaknesses

4. The sensitivity of MiSentry to various hyperparameters, such as the selection of models in the model zoo and the parameters for meta-learning, could be elaborated upon. Understanding how these factors influence performance would aid in the practical adoption and fine-tuning of the method.
paper_summary: paper_summary: The paper introduces BIVBench, a benchmark for black-box integrity verification of DNN models, and MiSentry, a novel fingerprinting method. BIVBench encompasses 16 observable model modifications, highlighting the inadequacy of existing methods in detecting subtle tampering. MiSentry maximizes prediction divergence between target and tampered models to generate sensitive fingerprint samples. It outperforms existing methods, especially in detecting subtle tampering, enhancing DNN model security in cloud services.
strength_weakness: strength_weakness: 1. This paper is the first to explore black-box integrity verification of DNN models in a stricter form by defining observable modification and considering subtle tampering. Introduces MiSentry, a novel fingerprinting method, and BIVBench, the first benchmark specifically designed for black-box integrity verification of DNN models.
2. The motivation is clear. Stricter integrity verification for detecting harder-to-observe modifications is useful, since an end user may need to check if the model he/she is using is really the model he/she wants to use.
3. The paper is well-written, providing clear explanations of the problem, good motivating experiments, and intuitive solutions. The paper is overall easy to follow.
4. The evaluation is comprehensive and the results are effective. MiSentry outperforms existing state-of-the-art fingerprinting methods, particularly in detecting subtle tampering.
limitations: limitations: Overall I think this work is good and suitable to be published at ACM MM. I do not find obvious mistakes or weaknesses.
paper_summary: paper_summary: The main work of this paper consists of two aspects: 
firstly, the design of a new fragile fingerprint called MiSentry, which is used to verify the integrity of models; 
secondly, the creation of the first benchmark for verifying model integrity in black-box scenarios, named BIVBench. 

The paper validates the fragility of MiSentry through BIVBench and compares it with previous state-of-the-art (SOTA) fragile fingerprints, 
demonstrating the superiority of the proposed method.
strength_weakness: strength_weakness: The proposed MiSentry is highly sensitive to various changes in the model, especially those that result from modifying even a small portion of the model parameters, which leaves a deep impression on me. 

The design of BIVBench is very comprehensive, taking into account almost all possible modifications to the model that could occur in real-world scenarios. 

The experiments are thorough, testing the fingerprints trained on multiple datasets and models.
limitations: limitations: 1. It seems that the effectiveness of the fingerprint detection is significantly reduced in the case of adaptive attacks, where the fingerprint is leaked. Would this fragile fingerprint method also be less effective on models that have undergone adversarial training? I hope to see related experiments on this matter.

2. Generating the fingerprint requires training within a model zoo. In this context, would the cost of generating the fingerprint be too high?

3. The BIVBench proposed in this paper distinguishes between benign and malicious modifications. Is it necessary to make this distinction?

4. Have there been tests on models other than classification models, or models other than convolutional models? For example, Vision Transformers (ViT).
paper_summary: paper_summary: This paper studies black-box integrity verification of DNN models. 
This paper focuses on "subtle modifications" as a series of integrity violations that are harder to be identified than ordinary integrity violations (such as backdoor inserting) and includes many subtle modfications into the threat model, named after BIVBench. 
Empirical results on BIVBench show that some SOTA intergrity verification schemes turn out to be uncapable of detecting subtle modifications (in Table 1). 
This paper further proposes a new black-box DNN integrity verification scheme, which is another fragile fingerprinting scheme that is non-invasive and easy to be applied. 
The triggers are found by leveraging surrogate DNN models (the authors used the term "model zoo" for this concept) which have undergone subtle modifications. 
Concretely, triggers are optimized so that their outputs are differentiated in the original model and surrogate models. 
Experiments demonstrated that triggers found by this scheme(MiSentry) turn out to be more sensitive, general, and covert.
strength_weakness: strength_weakness: 1. This paper explicitly mentioned and discussed the topic of "subtle modifications" as a challenge to current fragile fingerprint schemes for DNN models and verified that existing schemes might be unreliable. I believe that this point is very important and inspiring. 

2. The presentation is very well, I felt comfortable and smooth in following the authors' intuition and reasoning. 

3. I think that the authors have covered a broad range of metrics that are used to evaluate fragile fingerprint for DNN models (sensitivity, uniqueness, covertness, robustness, generalizability, and ablation studies), I appreciate these comprehensive discussions that prevented conclusions from multiple perspectives.
limitations: limitations: I have several concerns about the proposed scheme's (MiSentry) technical soundness and fairness. Some parts of the discussions contained in this paper are unnecessary or confusing. 

1. On the technical novelty, it seems to me that MiSentry tries to find some partially adversarial samples (I mean normal samples added with a slight perturbation, which moves the samples towards the decision boundary but not over it) so that their outputs are different in the original DNN model and surrogate models, which are modified version of the original model. (I might be wrong since Sec.4.4-paragraph 1 said "be of the same architecture and trained with the same dataset", and I believe that using modified versions of the original model results in more subtle modifications) What makes it essentially different from an adversarial attack, which has already been widely adopted to find samples near the decision boundary and are thus sensitive? I believe that PublicCheck used the same intuition, although it adopted a generator rather than gradient-based adversarial sample attack, so what is the theoretical or conceptual improvement? 

2. On the contribution's significance. I believe that modifications are nonetheless going to be more subtle in the future, with the emergence of LoRA for LLMs and text inversion for Diffusion Models. The contribution of proposing a collection of modifications such as BIVBench is questionable, even if it is the first of their kind as claimed by authors. There has to be more formal and mathematical discussions on the level of subtleness and the performance limitations of MiSentry (why it is hard or impossible to design stricter schemes?). 

3. On the scheme's soundness. I found that you used Fine-Tuning_{Last}-5 as surrogate models in Appendix C.3 (I believe that this point has to be incorporated into the main text since this is the most important setting). Now Fine-Tuning_{Last}-5 is already the strongest (or the most "subtle") modification according to Table 1, so samples that are trained to differentiate models undergone Fine-Tuning_{Last}-5 and the original model can for sure detect other weaker attacks. This comparison is unfair. Other baselines did not explicitly incorporate as much surrogate models as MiSentry. Can the generalizability be reduced to training the triggers on a hard task and test them on an easy task? Appendix G.1 Table 5 seems to support this statement, since surrogate model/model zoo heavily affect the detection rate. The comparitive advantages seem to be a cherry-picking result: you can always include models undergone more subtle modifications (e.g., fine-tuning less parameters with a smaller learning rate) to run MiSentry, and report improved statistics on less subtle modifications. 

4. In Sec.4.2, "Uniqueness" is not a desirable property for fragile fingerprints, it is desirable only for robust fingerprints (robust fingerprints are for ownership verification where sensitivity is the least favorable property). This is why exisitng literatures did not explore this aspect. After all, the authors provided no further examination of their scheme regarding uniqueness in the rest of their paper. 

5. In Sec.4.2, "Robustness to Partial Leakage". This is not a concern that worths discussions,  you can always produce new triggers in the context of a fingerprinting scheme. Using a collection of new triggers during each round of integrity verification is enough. The leakage is a problem only for watermarking schems (where a finite list of triggers have to be determined beforehand), or when only a finite collections of triggers can be produced, or when the cost of randomly producing new triggers is prohibitive (Sec.5.6 suggest that this is cheap). 

6. Is there any results on the performance of triggers regarding preprocessing? (e.g., the adversary applies JPEG-compression to triggers)