A Zest of LIME: Towards Architecture-Independent Model Distances
Authors: Hengrui Jia
Hongyu Chen
Jonas Guan
Ali Shahin Shamsabadi
Nicolas Papernot
Venue: ICLR 2022 Poster (2022)
Keywords: model distance
model stealing
machine unlearning
fairwashing

Abstract: Definitions of the distance between two machine learning models either characterize the similarity of the models' predictions or of their weights. While similarity of weights is attractive because it implies similarity of predictions in the limit, it suffers from being inapplicable to comparing models with different architectures. On the other hand, the similarity of predictions is broadly applicable but depends heavily on the choice of model inputs during comparison. In this paper, we instead propose to compute distance between black-box models by comparing their Local Interpretable Model-Agnostic Explanations (LIME). To compare two models, we take a reference dataset, and locally approximate the models on each reference point with linear models trained by LIME. We then compute the cosine distance between the concatenated weights of the linear models. This yields an approach that is both architecture-independent and possesses the benefits of comparing models in weight space. We empirically show that our method, which we call Zest, can be applied to two problems that require measurements of model similarity: detecting model stealing and machine unlearning.

Published as a conference paper at ICLR 2022
A ZEST OF LIME: TOWARDS ARCHITECTURE-INDERR
MODEL DISTANCES
Hengrui Jia, Hongyu Chen, Jonas Guan
University of Toronto and Vector Institute
(nickhengrui. jia, chen @ma utoronto. ca, jonas wow.s@cs. ott.rrorttto...ooooaeoetttoooooo edu
Ali Shahin Shamsabadi
Nicolas Papernot
Vector Institute and The Alan Turing Institute
University of Toronto and Vector Institute nicolas,
a. shahinshahh.. maahinnsamaaa abwaaaaaaiiaaa..aam.aaa aww.aaiiiaiiiiriingggg.g.... .aag.acc. ac. uk
ricooaa..pppemmoeetteeeooo papernot.ettooottet.to... ca
ABSTRACT
Definitions of the distance between two machine learning models either character- ize the similarity of the models' predictions or of their weights. While similarity of weights is attractive because it implies similarity of predictions in the limit, it suffers from being inapplicable tures. On the other hand, the similarity of predictions is broadly applicable but depends heavily on the choice of model inputs during comparison. In this paper, we instead propose to compute distance between black-box models by comparing their Local Interpretable Model-Agnostie Explanations (LIME). To compare two models, we take reference dataset, and locally approximate the models each reference point with linear models trained by LIME. We then compute the Cosine distance between the concatenated weights of the linear models. This yields approach that is both architecture -enp:eeeureedideeeedeeeeeee and possesses the benefits of com- paring models in weight space. We empirically show that our method, which we call Zest, helps in several tasks that require measurements of model similarity: verifying machine unlearning, and detecting many forms of model reuse, such model stealing, knowledge distillation, and transfer learning
comparing models with different architec-
INTRODUCTION
We explore the problem of quantifying similarity between machine learning (ML) models with distance metrics, with a focus on deep neural networks (DNNs). Comparing the functional behavior of ML models is often challenging because they are not casily inspectable (c.g. located in the cloud). In addition, ML models that capture similar knowledge may not share similar architectures, and vice versa, making it difficult compare models in weight space. Finally, ML models that solve different tasks may nonetheless share similar knowledge, such as in the case of transfer learning (Li et al. 2021). This is emphasized in DNNs, which are not only complex, often containing millions of parameters, but also difficult to interpret (Ribeiro et al. 2016).
Previous methods for measuring model distances either the similarity of predictions (Li al. 2021), or the similarity of weights et al. (02T). However, methods that measure the similarity of predictions only capture the local behavior of models at each data point, and it hard to choose set of inputs to compare their global behaviors. This causes high variance in the results: models that are unrelated may happen give very similar predictions on some inputs, while related models (e.g., fine-tuned from one another) may give different predictions on other inputs. Methods that measure the similarity of weights are more complete, because they capture the global behavior of models, and weight similarity implies prediction similarity in the limit: as the weights of the compared models approach each other, so will their predictions. But, to meaningful, weight-space methods can only be applied when the compared models share the same architecture. Furthermore, they require white-box access to the models to compare weights. These constraints restrict their use in practice.
Cooee available htttp::/gittub..oom oripabaahhhrrpep/ghhhhhaedeeeeee 1arissss-abbbii /est Model stance

---

Published as a conference paper at ICLR 2022
Instead, we propose to compute distances between models by first approximating their global be- havior through a collection of linear models. We name our approach Zest. We start by applying the Local Interpretable Model-Agnostic Explanations (LIME) algorithm of Ribeiro et al. (2016) to generate a collection of linear models that each approximates some local behavior of a compared model. We chose LIME because it is black-hox (only requires access to the models' predictions) and architectur--iddeenddee
and as consequence so is Zest. The collection of these local approxima- tions form a global approximation of model behavior. Since the approximation models obtained are linear, they are straightforward to compare. Indeed, to measure similarities and differences between two models being compared, we compute the Cosine distance between the concatenated weights of their respective collections of approximated linear models. Because we compare local linear approx- imations of the models rather than their predictions directly, this allows Zest to interpolate between the benefits of prior approaches in the weight or prediction spaces described earlier.
In particular, our empirical validation of Zest demonstrates that our approach is not sensitive to the choice of points used by LIME to locally approximate the models being compared. This departs from prior approaches that compared model predictions directly and as a result were overly sensitive to the choice of points the comparison is made on. This characteristic allows Zest to correctly capture the differences and similarities between pairs of unrelated or related classifiers.
Furthermore, show that the distance computed by Zest finds natural applications in detecting model reuse (e.g., model stealing, knowledge distillation, and transfer learning) and verifying ma- chine unlearning. [L al. (2021) first proposed introduce distance to inform the detection of classifiers which were trained by reusing another classifier. In particular, this is the case in model extraction where an adversary observes the predictions of model reproduce its behavior in stolen copy of the model. Whereas Li et al. (202T) left the identification of stolen copies of a model as future work after failing to apply their proposed ModelDiff metric to this problem, we find it can be solved by Zest. In a completely different setting, we also find an application of Zest to machine unlearning: the process of removing the effects of learning data points from a model (Cao & Yang. 2015). We use Zest as a heuristic to inform whether or not user's data has been unlearned, by comparing the decision boundaries of the models before and after unlearning around that data.
Our main contributions are follows:
We propose Zest, an approach for computing architecture-indeeendde distance metric that measures the similarity between MI. models. Zest only requires black-box access, and compares the global behavior of the models, which it approximates using LIME.
We validate that Zest measures model similarity by showing that the distance between differ- ent epochs of the same model is average 99. 99.29(%0.19)% on CIFAR-10, 98. 98..55(=..1)) 12)% on C00,R-100, 92. (22.222++222)) % on AG News and 81. 81.17(++3.01 3% on Speech Commands, closer than the distance between two models with different initialization trained on the same dataset. We our experiments in the vision, text and audio domains using the ResNet18, ResNet20, ResNet50, LSTM, M5, and MobileNet-v2 architectures, on the CIFAR-1O, CIFAR-100, AG News, Speech Commands, ImageNet, Flower (2 and SDog 1 120 datasets.
We show that Zest can be used to help detect model reuse, including the hard case of model stealing, and inform unlearning verification. The distance between a model stolen via model ex- traction and the original model is 19.96(+4.288% on CIFAR-10 and 54.87(=1.70)5 on CIFAR- 100, closer on average than the distance between the original and a model retrained on the same datasets. Compared to ModelDiff, which had a 0% accuracy at detecting model extraction, we have 100% accuracy for the attacks in the ModelReuse benchmark (Li et al. 2021). We further show that Zest helps identify all other methods of model reuse in the benchmark. To inform unlearning verification, we show that the distance from a model to its retrained counterpart with- out the unlearned data is on average 144.56(+115 (5.39)1113)))) % on CIFAR-10 10 and 140.51 (+)9.1))% on CIFAR-100 further than the distance to retrained model that did not exclude the data to unlearn.
BACKGROUND AND PROBLEM STATEMENT: DEFINING MODEL DISTANCE
We aim to compute a distance between two ML models C1 and C2 to characterize their similarity, given only the ability to observe the models' predictions chosen inputs. We make assumptions on the models' architectures, except that they have the same input dimensions. In other words, we only assume that they model similar distributions (e.g. traffic signs), but the compared models can
2

---

Published as a conference paper at ICLR 2022
be trained for different tasks (e.g. classifying German and American traffic signs respectively). We denote the output space of the two models D2 and D2. We focus on DNNs, due to their popularity and the difficulty to interpret their behavior. but our distance metric applies to other ML models. For ease of exposition, we use the classification of images as sample task throughout our discussion in $2 and $3. We also evaluate Zest on the text and audio domains and provide our results in $4.
Past work on measuring model similarity has either defined distance by comparing model weights or comparing model predictions. We briefly overview both approaches and their current limitations, then give an outline of the LIME algorithm and how it can be applied to address these limitations.
Comparing model weights. The weights of a parameterized model define its behavior and it can be seen how in the limit having close weights implies that two models will have close behavior; thus, one way to compare the similarity of models is to compare the difference in their weights.
One example of weight space comparison is proof of learning; Jia et al. 2021)) use the comparison of model weights throughout the training procedure to provide evidence that a party has indeed trained a model. This is useful for ownership resolution. By saving checkpoints of the model during training, and showing that the model weights between subsequent checkpoints are close, the owner can provide a history that proves they are the party who trained the model.
However, comparing model weights highly restrictive in practice: it requires both access to model weights, and for compared models to share the same architecture, so that the distance between each corresponding weight value can be compared. Even if models share the same architecture, it can be seen that two models whose weights are permutations of one another on a single layer would make exactly the same predictions yet have a non-zero distance in weight space.
Comparing model predictions. On the other hand, model's behavior also be defined by its predictions on inputs. The advantage of comparing model similarity with predictions is that we can treat the models as black-boxes. Because only need to observe input-output pairs, we do not need access to model weights and are not restricted by the architecture of the models being compared.
In theory, if we can query a model with every input in the domain and get its predictions, can learn its exact functional behavior, but this is intractable in practice. To address this, current approaches resort to selecting smaller sets of inputs reference, and comparing the models based on their predictions on the reference. The disadvantage of these approaches is that (a) they can only capture point-wise behavior of the models the chosen inputs, and (b) choosing a representative set of inputs is hard. If the chosen inputs are far from decision boundaries then even unrelated well-trained DNNs give similar predictions; whereas inputs near decision boundaries are sensitive to small changes of the boundaries, so related models (e.g. model and its transferred copy) may predict differently.
This is consistent with findings in deep learning testing, a class of methods that aim to identify areas of the input space where supposedly similar models disagree with each other on the output space. DiffChaser (Xie et al. 2019). DeepXplore (Pei et al., 2017) and TensorFuzz (Odena et al., 2019) are three recent and popular testing frameworks. These not applicable in case because they focus on the differences between closely related models, whereas we are looking for a distance metric that is able to provide information on both the similarity and differences between any kinds of models.
Perhaps the closest to our work, ModelDiff (Li et al., 2021) is a model distance metric designed to detect model reuse, which is defined training model based another to save training costs. ModelDiff feeds a set of paired test inputs to the compared models, where each pair is selected to be close in the input space, but far in the output space. It then compares the total Cosine similarity of the distance between the outputs of each pair to find the model distance. Using this distance metric, [i] al. (222I) detected transfer learning and model compression, however, they fail at the harder case of detecting model extraction and leave it to future work. Our approach solves this open problem, and our empirical study demonstrates that our distance metric is more precise and less sensitive to changes of the model that do not affect its performance on in-distribution data.
LIME. Local Interpretable Model-agnostic Explanations generates interpretable linear models to explain the local behavior of MI models (Ribeiro et al., 2016). LIME learns linear models by segmenting the input space, then systematically masking segments to identify features that are most influential to the classification of data point. For example, this implies masking contiguous areas of pixels for image classification tasks. Using the most influential features, LIME Iearns a linear model that approximates the decision boundary of the compared model near the data point.
3

---

Published as a conference paper at ICLR 2022
Appooissting
wags
Superppiin
lngls
Locel Locelingreser
Edior regressor
W:
Oliz Mpproiisee
Inine
[10000320||
Global
Runsben
ad
Segrae 1oxice
Componnntt
Regrsser
W9:
Glosa ipptominati
gegansoe
ing
00000 Gerpoons D064
Farre:
Per Krg
Figure An overview of the Zest approach to computing the distance between two black-box classifiers C1 and C2. Phase 1 randomly samples several reference images. Phase 2 segments the reference images into components and their location masks using a segmentation model. Phase approximates the local decision boundaries around each reference image by training LIME model that takes mask locations as input and predicts the classifiers response on these images' components. Finally, Phase 4 combines the local decision boundaries of all reference points approximate the global decision boundaries of C1 and C2, and computes their Cosine distance.
Intuition of comparing in weight space of LIME. From the perspective of the decision boundaries, captures a unit circle.
weights of a DNN capture the entire boundaries, just like how x² + y² This leads to high precision but restricts generality. For example, it is hard to define how x² + y² similar is to mar(x, y) = 1. On the other hand, each prediction only reveals one specific mapping from the input to the output space, and multiple predictions are needed to identify tiny region of the decision boundaries. model predictions is like comparing points in the coordinate system- the method is more general, but both a unit circle and a unit square can have points like [0, 1j. The method will not be precise compare the points and conclude the circle is similar to the square.
we draw an analogy to the Cartesian coordinate system, comparing
At a high level, we are dealing with the trade-off between precision and generality. Comparing predictions gives us general applicability whereas comparing weights gives us precision. Thus, to interpolate between these two extremes we choose to compare linear models that locally approxi- mate the models be compared. Using the same analogy, comparing LIME of two DNNs similar to comparing lines that are tangent to the shape, instead of the formula of the shape (comparing weights) or points on the shape (comparing predictions). We choose the LIME algorithm for gen- erating local approximations because it is sebitecture-independdee and is simple to understand. $ contains a more formal description of how we apply LIME.
only requires black-box access,
METHOD: MEASURING DISTANCES BETWEEN MODELS WITH ZEST
Building on the insights of $ 2 we now introduce approach that is able to interpolate between the benefits weight-space and prediction-spar comparisons of two classifiers. We propose Zest, model distance metric that measures the similarity of two models C₁ and C2 by comparing their global behaviour as approximated by LIME (see Figure for an overview).
Algorithm T describes our proposed model distance, which consists of four sequential phases: 1) Sampling a reference dataset; 2) Sampling nearby images as done in LIME; 3) Learning the be- haviour of C₁ and C2 locally around the reference data points with linear classifiers as done in LIME; 4) Approximating a global perspective of C and C2 by combining these linear models, and computing the distance between the combinations of linear models corresponding to C1 and C2- Next, we describe each phase of our proposed model distance in detail.
Sampling representative reference data points. We randomly sample N reference images X' {xx [x from the distribution task. Note that N is much smaller than the size of the train- ing set of the classifiers being compared. For example, we select N 128 reference images from 50, 000 training images of CIFAR-10. In $ 4.3) we show empirically that the choice of N does not qualitatively change the outcome of our model distance metric computations uniiik previous approaches described in $ $2 that directly compared the predictions of two classifiers. Increasing N simply reduces the variance of our metric, which intuitive because the comparison is made on more points sampled from the underlying data distribution; reference points are more likely to cover the same local regions of the classifiers' decision surface as N increases.
Learning the local behaviours. While the classifiers being compared can be non-linear (e.g., deep neural networks in our case), the intuition behind LIME is that their behavior can be approximated locally by linear model. We learn the local behaviour of each of the two classifiers being compared around each reference image by training a LIME linear regression model that captures the classifier responses to the image's 's components (i.e. contiguous patches of pixels denoted a super-pixel).
4

---

Published as a conference paper at ICLR 2022
Similar to the original LIME paper, create and sample image components follows. First, we group the pixels of a reference image in K super-pixels using Quickshift (Vedaldi & Soatto 2008) and create binary super-pixel masks B [b where each super-pixel binary mask Dy {0, [0, 1 specifies the location of pixels belong to the k-th super-pixel. These super-pixels each represent a contiguous patch of similar pixels, like a patch of fur or grass, whose presence or absence is controlled by the binary mask. A linear model these super-pixels provides an interpretable approximation of the local decision boundary. For example, if the classifier predicts "dog" when the patch of fur present, but changes its prediction when it is not, then we can interpret that the patch of fur greatly influences its prediction.
Then, we create the reference image's component x x- m by randomly choosing several super- pixels identified by Is in component mask m E {00, 1j that is combination of their corre- sponding binary mask. In order to explore the local area around each reference image, we repeat this process and draw I. different components & = {x)] using L different component mask M [m/ Following the original LIME paper, we set L = 1000.
We query classifiers to obtain their predictions on these L image components as 2 = C(( and Zg = C2()). Then, we train linear regression model / M + Z per classifier to have high local fidelity with respect to the classifier around the reference image x. We call the weights of the trained regression model the local signature of its corresponding classifier.
Approximating a global perspective. In total, we learn N linear regression models, capturing the behaviour of the classifier locally around the N reference images. We approximate the global behaviour of each classifier by concatenating its N local signatures S = [WT") ===1- Note that local signatures of each classifier have as many rows as the number of classes of the classifier. In order to compare signatures of classifiers with different number of classes when the relationship between classes output by the two classifiers is unknown, we introduce class alignment to re-position and reshape the rows of local signatures. Let small and big local signature be the local signatures of the classifiers with the lower and higher number of classes respectively. In class alignment, we start by the first row of the small local signature and compute its distance with each row in the big local signature. Then, local signature. We repeat this for all other rows in the small local signature while removing aligned rows of the big local signature from this process. Finally, we remove any remaining row of the big local signature not selected previously. See Appendix B for further details.
move the row corresponding to the minimum distance to the first row of of big
Computing the distance. As the global signature of classifiers are the same size, we compute the model distance as d = Distance( (S1, S2). In Appendix G we show that the choice of Distance(,; )) does not affect the performance of Zest. Our experimental results in the remainder of the paper are thus presented with the Cosine distance being used to compare global signatures.
Algorithm 1: Proposed model distance approach, Zest
Input: Black-hox Dj -llslass classifier C2. Black-box D2-olass classifier C2, Reference image set X' J j== Quickshift segmentation algorithm, Random-Selector Lincar regression training algorithm T(---) and distance Distance(,,-) Output: Distance, d, between black- box classifiers C₁ and
S1.S2
0.. 0
b. Tritializa model signatures
E do
B {bx 1e Quickshift((x)
Generate K super super-pixel masks Generate L image's a component masks D Generate L image'a a components > Query claasi fier Train local zegression model
M [m )/=
(c)ecto
X' (x) : 81 x m))
6:
2, (L))), Zg C2(X)
W. T(M, 2,). W. T(M, 22)
8:
if Unknown relation between D; and D2 then
for E Dj do W Wa m (11 Wab3ll]]
10:
Class alignment
11:
S spppendd W1). S... Spappedd(WW) Distameel S1 Sa)
b. Create model signature
12:
d
Corpute model distance
VALIDATION
4.1
IMPLEMENTATION DETAILS
We validate our proposed method, Zest, in the vision, text, and audio domains using four publicly available classifiers and four datasets: ResNet20 trained on CIFAR-10, ResNet50 trained on CIFAR-
5

---

Published as a conference paper at ICLR 2022
Comeen Ca
Ca V.S. Commiter Ca s.cl
Gendene 122091
22and 0.007)
08106
25
50
100
125
150
178
25
75
100
125
10c
172
200
Epoch
Epoch
(a) CIFAR-I0
(b) CIFAR-100
Figure 2:
The distance computed by Zest for unrelated classifiers (Ciff)
sef versus Cunrelated) and related
classifiers (C/) versus ((( ) as function of training epoch, t. We consider k = 5 for the related classifier. See Appendix D for the results of other k values. The distance of related classifiers is significantly smaller than the Zest distance of unrelated classifiers, especially as training converges.
100, LSTM trained on AG News, and M5 network trained on Speech Commands. The accuracy of ResNet20 CIFAR-10 is 91.77(+0.17))% (0.27(= for LSTM on AG News, and 87.67 (7..77++0866555 for M5 network on Speech Com- mands. Our results in this section are averaged over 5 different training runs of classifiers under comparison. Recall from $ 3 that Zest consists of 4 successive phases. In phase I of Zest, we select 128 samples randomly from the training set. In phase and phase 2, we follow the same implementation and setup as the original LIME paper. In phase 3, we train linear regression models using the least-squares estimation technique. We use relative Cosine similarity (see Appendix for details) for computing the distance between global signatures in phase 4. However, in Appendix G we perform an ablation study where we compare using the & and &g norms as alternative metrics, and show that Zest is agnostic to the choice of metric used to compare global signatures.
(6.177+-022)%% for the ResNe150 on CIFAR-100,
4.2 DOES ZEST CAPTURE THE DISTANCE BETWERN CLASSIFIERS CORRECTLY?
We analyse the performance of Zest in computing distances between related or unrelated classifiers. To build related and unrelated pairs, we first collect reference classifier trained with epochs, (.))
We consider the classifier at epoch t+ related classifier, (+++k) to the reference classifier C Instead, we consider the same architecture but trained with a different random seed as a classifier, o( unrelated unrelated to the reference classifier. The distance between unrelated classifiers (C versus Cunrelated) should be bigger than the distance between related classifiers (C(() versus C((++))
Figure shows shows the Zest distance between unrelated and related classifiers, at training epochs Jo, 200] of the reference classifier. Overall, model distances in both the unrelated and related cases decrease as training progresses and model weights become more informative. This is intuitive given that both classifiers are trained to solve the same task. However, the distance output by Zest for unrelated classifiers has lower bound that is significantly higher than for related classifiers. For example, the distance computed by Zest for unrelated ResNet20 classifiers is still above at the last epoch (epoch=200), while it is so close to 0 for related ones with le 5 Therefore, Zest can correctly distinguish related from unrelated classifiers.
4.3 How SENSITIVE IS ZEST TO THE SIZE OF REFERENCE SET?
We analyse the effect of the reference set on the performance of Zest by studying 1) the effect of the size of the reference set on Zest; 2) the effect of stochasticity in the selection of points that make up the reference set. For both of these analyses, we consider two pairs of related classifiers: (C(10) versus ((200)) and (C(119) versus C!200))) 3 the average and standard deviation of the distance computed by Zest for different sizes, {1, 2, 4, 8, 16, 32, 64, 128]. of the reference set and different random samples of reference points. The confidence interval (coloured areas) is obtained by repeating the experiment 5 times to train classifiers with different seeds. We observe that the average distance computed by Zest identical regardless of the size of the reference set in both of these related classifiers. However, the standard deviation of the distance computed by Zest decreases as the size of the reference set increases: for instance, in C-1100) versus C.2000) standard deviation decreases from 0.3 for a reference set of size I to value close to zero we increase the reference size to 128. Note that this remains a small reference set relative to the size of the training set used to learn these classifiers. In the case of C(190) (190) versus ((200), the classifiers are similar in the first place that the standard deviation of the distance computed by Zest is close to zero and remains similar for all of the reference set sizes.

---

Published as a conference paper at ICLR 2022
v. C (CIFARIO) Ca (LIAARI))) Gi Ga (CFFRRI0 (IFARI
COPR CEF
CE CE ICEAARI
CI Og (CEARIOS)
0.
16 Befocnee Denad Stat (lgg
118
123
Beforasc Ditarct ocde)
(a) Average
(b) Standard deviation
Figure 3: Influence of the reference set size and sampling stochasticity on the distance computed by Zest. The average Zest distances of different reference sets are similar, while the standard deviation decreases as the size of the reference set increases.
C. Covean Ch
C Coraans
C x.
2.5
7.5
10.3
12.5
155
20.
Ep
Epock
(a) AG News dataset (text), LSTM (b) Speech Commands dataset (audio),
M5
Figure 4: Evaluation of correctness of Zest in the text and audio domains respectively: this figure is plotted in similar manner to Figure 2 i.e., the distances between related and unrelated classifier pairs are plotted as a function of training epoch, t. We can see the Zest distance of related classifiers are significantly smaller than distances of unrelated classifiers across domains.
Such quick reduction in variance by increasing the number of reference points can be achieved by prediction comparison. In the same training setting for CIFAR- 10 and using 128 reference points, the prediction similarity between cf110) and C is 93. 03.28( +1 1.06)%, which is overlapping with the prediction similarity between 2 unrelated models: 91 91. (2.227. The two values become 92.27(=+..22)) and 91.59( 76%% respectively when we increase the reference dataset to size 1280. Such a small gap and large variance make it hard to confidently claim if two models are the same and can lead to a high false positive rate (i.e., claiming two unrelated models as related). This is opposite to distance computed by Zest on reference dataset that is 10 times smaller, which can easily separate distances between unrelated and related pairs, shown in Figure 2 One may think the bad performance of prediction comparison is because most of data points are far away from the decision boundaries of the well-trained models, and decide to pick data points near the decision boundaries. By doing so, observed similarity of 35, 35.23 2. 2.600.6055 % between unrelated models and 41. (2(22) between related models. However, such points are sensitive to small changes in the decision boundaries. By training model to match the predictions of the victim model (can be thought of as high-fidelity model extraction attacks), we find the similarity between the victim and extracted models quickly drops 32.36(+1..44 being no different from the similarity between two unrelated models. This problem is solved by Zest, as described in details in $ 5.2
4.4 ZEST IN OTHER DOMAINS
In this section, we analyse the performance of Zest in computing distances between related and unrelated model pairs in both text and audio domains. In the text domain we used Long-Short Term Memory (LSTM) classifiers (Hochreiter & Schmidhuber 1997) and AG News dataset (Zhang al., 2015) (described in Appendix | and In the audio domain we used M5 speech classifier (Dai al. 2017) and Speech Commands dataset (Warden, 2018). Similarly to our vision experiment in Section 4.2 related LSTM (or M5) classifiers are checkpoints at two different epochs, while unrelated LSTM (or M5) classifiers are trained with different random seeds. Figure 4 shows that the distance output by Zest for unrelated classifiers in both text and audio domains is significantly higher than for related classifiers (consistent with our results in Section 4.2 on vision datasets). This thus reinforces that Zest is applicable to different domains of vision, audio and text.
5 CASE STUDIES: APPLICATIONS OF ZEST
Next, evaluate the applicability of distances between reference and suspicious classifiers, com- puted by Zest to two tasks (see Table 2 in Appendix A.3). Our case studies, introduced earlier, are model stealing and machine unlearning. We first introduce relevant literature these case studies.

---

Published as a conference paper at ICLR 2022
Car K.S. Caraan
Custo Tomeet
Lentone 122391
25
30
100
172
360
25
90
75
100
125
172
36
Epock
(a) CIFAR-10 10
(b) CIFAR-I0U
The distance between a victim classifier and its extracted copy obtained by Zest as a function of training epochs of the latter. Zest can detect the extracted copy of the victim classifier. 5.1 BACKGROUND ON MODEL STEALING AND MACHINE UNLEARNING
Figure 5:
Model stealing, There is often a strong need to protect the confidentiality of ML models; it can be costly collect datasets and train models, making them valuable intellectual property. Model stealing refers to an attacker duplicating the functionality of a confidential model without permis- sion, stealing intellectual property. Model extraction class of such attacks that efficiently learn the approximate functionality of model by observing its outputs on selected inputs (Tramer et al.) 2016). This is often achieved via an exposed inference API, which is common in ML-as-a-service.
Model stealing can also be considered malicious case of model reuse, where a model is used as a basis for training another model at smaller cost (c.g.. through fine-tuning). Li al. (202T) developed a distance metric, ModelDiff, that detects model reuse in DNNs by comparing the models outputs. However, while Li et al. (202T) identified model reuse via transfer learning and model compression, they were unsuccessful in detecting models stolen via model extraction, and presented this as an open problem. In our case study, we show we are able to fill this gap the literature.
Machine unlearning. Private data is often collected for the purpose of training ML models, but leg- islation and privacy ethics call for the right to be forgotten. For ML models, the right to be forgotten implies that user should have the right to remove the effects of their data on model trained on that data. The naive solution is to remove the user's entry from training data, then retrain the model, but there are more efficient methods: these either decrease the time it takes exactly/directly train (Bourtoule et al. 2021) or find a way to approximate the retraining procedure (Graves et al.) 2020). After requesting data removal, the user (or third-party party auditors) may be interested in verify- ing that the new model has indeed unlearned their data. By measuring the similarity of the decision boundary between the old and new models around the user's data, we can use Zest heuristic to help test if the data has been unlearned.
5.2 ZEST DETECTS MODEL EXTRACTION
We now aim to compute the distance between reference victim classifier and copy of the victim classifier extracted by an adversary. We consider a strong adversary who knows the architecture of the victim model and has access to the unlabeled training set of the victim classifier. This makes it easier for the adversary to train a copy of the model which is hard to distinguish from the victim model they are stealing. The adversary trains the same architecture classifier using the training set and their predicted labels obtained through querying the victim model.
In Figure 5) we Zest to compute the distance between the victim classifier and the extracted classifier as a function of training epochs of the extracted classifier. As the accuracy of the extracted classifier improves during its training (reffected by the increasing number of training epochs), the distance between the victim and extracted classifiers decreases for both CIFAR-10 10 and CIFAR- 100. The minimum distances achieved the last epoch are 0.80 and 0.60 and remain smaller than the lower bound of unrelated classifiers (discussed in 4.2). Therefore, Zest can help detect a copy extracted with the attack considered here based on its distance the victim classifier.
To further our evaluation, we also compare our proposed method Zest using the ModelReuse bench- mark introduced with ModelDiff (Li et al. 2021). The benchmark contains 144 pairs of reused models which contain one model that reuses the knowledge of another model. Here, [i et al. (2021) define reusing as transfer learning, pruning, quantization, distillation, model extraction and their combinations (see Table 2 in Li et al. (2021) for the complete description of these tasks). Thus, this benchmark not only allows us to compare the performance of Zest with ModelDiff when comes to detecting model extraction but also these several other forms of model reuse. Following the setting of ModelDiff's evaluation, we use ResNet18 (He et al. 2016) and MobileNet-v2 (Sandler et al. 2018) trained on ImageNet (Deng et al. 2009). Flower102 (Nilsback & Zisserman 2008), or
8

---

Published as a conference paper at ICLR 2022
Reuse Task
#Models
Zest (Cosine)
ModelDiff
Transfer
12
100.0%
100.0%
Prune
100.0%
100.0%
Quantization
12
100.0%
100.0%
Distillation
12
100.0%
100.0%
Model extraction
12
100.0%
0.0 %
Transfer + prune
36
100.0%
100.0%
Transfer + quantize
12
100.0%
100.0%
Transfer + distill
12
100.0%
100.0%
Overall
144
100.0%
91.7%
Table 1: Accuracy of our proposed method, Zest, and ModelDiff (Li et al. 2021 in detecting the similarity of reused model pairs in ModelReuse benchmark. The maximum of overall accuracy is highlighted by bold, while the minimum of accuracy among all the task is highlighted by italic.
12552
Jurdaae 0.005
Oa00re 0.0000
(0p300 3.000
15
Epeck
Epeck
(a) CIFAR-10
(b) CIFAR- 100
Figure 6: Distances between the original classifier and (1) directly unlearned classifier, (2) an approximately unlearned classifier when data to be unlearned serves as the reference set in Zest. SDog (Khosla et al. 2011) datasets. To compute the accuracy of Zest and ModelDiff, use Zest to compute distances between models in each pair (reported in Appendix F. It is worth to note that there are model pairs trained on different dataset in this benchmark, which where class alignment comes to use. Then, we threshold the normalized distance obtained by Zest: distances lower than 1 indicate similar models while those higher than I indicate dissimilar models. Table T shows that Zest and ModelDiff have similar performance in all tasks except model extraction in which the accuracy ModelDiff is zero while the accuracy of Zest 100%. Therefore, Zest outperforms ModelDiff in the ModelReuse benchmark's model extraction task.
first
5.3 ZEST VERIFIES UNLEARNING
In machine unlearning, our reference classifier is the original classifier trained all the training set. We compare two classifiers to this reference classifier. The first the exactly/directly unlearned classifier, i.e., classifier trained all the training sets except data that requested be un- learned as done in Bourtoule et al. (202T). The second the approximately unlearned classifier, i.e., classifier obtained by directly adding update the original reference classifier with the goal of removing the effect of data to be unlearned as done in Graves et al. (2020).
We consider 128 data points be unlearned, and choose them the reference set for training the LIME linear regression models which underlic our distance metric. In Figure we compute model distances between the three classifiers mentioned above. We observe that the distance computed by Zest between the reference and exactly unlearned classifiers (or approximately unlearned classifier) high, even higher than the distance between unrelated classifiers (discussed in $ (.2). However, the distance between the reference and approximately unlearned classifiers is almost null, confirm- ing that these techniques cannot completely remove the influence of data requested to be unlearned from the model. Therefore, Zest suggests that exact unlearning can be distinguished from ap- proximate unlearning (or not unlearning) by computing model distances.
CONCLUSION
We proposed Zest, an approach to compute architecture-iddepnnded model distances to measure similarity between MI models. Zest only requires black-box access to the compared models. Zest achieves this by learning linear models with LIME to capture local behavior, then concatenating their weights for an approximation the compared models' global behaviors. To compute a distance for the compared models, take the Cosine distance between the weights of their concatenated linear models. We demonstrated the utility of Zest on several tasks, including many common forms of model reuse, and machine unlearning. We expect that future work will improve upon Zest by better understanding how to approximate locally the behavior of models, and understanding how to com- pare local approximations to provide finer-grained insights into model similarities and differences.

---

Published as a conference paper at ICLR 2022
ETHICAL IMPACT STATEMENT
Our work applies to two tasks, the detection model stealing and verification of machine unlearn- ing, that have practical impact to the protection of intellectual property and user privacy. Model stealing is theft on intellectual property, and the stolen model can be analyzed to extract private data used in training: machine unlearning enforces users' right to be forgotten. We believe that research on model distance metrics can be used to help deter model stealing, and help hold model owners responsible ethical codes and regulations that call for the right to be forgotten, such the Gen- eral Data Protection Regulation in the European Union, the California Consumer Privacy Act in the United States, and the Personal Information Protection and Electronic Documents Act in Canada.
REPRODUCIBILITY STATEMENT
To share code for reproducibility during the review process, will provide a link anony- mous repository in the discussion forums. the future, we also aim to publicly open-souree our code base, and provide documentations for reproducing the results shown in this paper. When pos- sible, we used well-established, open-source implementations of models and algorithms to further decrease the overhead of reproducing our experiments. We provide the complete list of code repos- itories that we referenced in Appendix
ACKNOWLEDGGMENTS
We would like to acknowledge our sponsors, who support our research with financial and in-kind contributions: CIFAR through the Canada CIFAR AI Chair and Al Catalyst program, DARPA through the GARD program. Intel, Meta, NERF through an Exploration grant, and NSERC through the Discovery Grant and COHESA Strategic Alliance. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government Canada through CIFAR, and companies sponsoring the Vector Institute. We would like to thank members of the CleverHans Lab for their feedback.
REFERENCES
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. 2015 IEEE Symposium Security and Privacy, pp. 463-480. IEFF, 2015.
Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, and Samarjit Das. Very deep convolutional neural networks for raw waveforms. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 421-425 IEEE, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 145-255. leee, 2009.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. arXiv preprint arXiv:2010.10981 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Hengrui Jia, Mohammad Yaghini, Christopher A Choquette-Chont Natalie Dullerud, Anvith Thudi, Varum Chandrasckaran, and Nicolas Papernot. Preaf-of-learning: Definitions and practice. In 2021 IEEE Symposium Security and Privacy (SP). IEEE, 2021.
10

---

Published as a conference paper at ICLR 2022
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei- Novel dataset for fine-
grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
Yuanchun Li, Ziqi Zhang, Bingyan Liu, Ziyue Yang, and Yunxin Liu. Modeldiff: Testing- based dnn similarity comparison for model reuse detection. arXiv preprint arXiv:2 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
Augustus Odena, Catherine Olsson, David Andersen, and Ian Goodfellow. Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. In International Conference on Machine Learn- ing, pp. 1901-4911. PMLR, 2019.
Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Deepxplore: Automated whitebox testing of deep learning systems. In proceedings of the 26th Symposium Operating Systems Principles, pp. 1-18, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520, Los Alamitos, CA. USA. jun 2018. IEEE Computer Society.
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristempart. Stealing machine learning models via prediction apis. In 25th USENIX Security Symposium (USENIX Security 16), pp. 601-618, 2016.
Andrea Vedaldi and Stefano Soatto. Quick shift and kemeel methods for mode seeking. In European conference on computer vision, pp. 705-718. Springer, 2008
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint 1804.0880003200 2018.
Xiaofei Xie, Lei Ma, Haijun Wang, Yuekang Li, Yang Liu, and Xiaohong Li. Diffchaser: Detecting disagreements for deep neural networks. In I/CAI, pp. 5772-5778, 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28:649-657, 2015.
A DATASETS, CLASSIFIERS AND TASKS
A.1 DATASETS
We evaluated Zest in seven different datasets, Five of these are vision datasets-CIFAR-10 CIFAR- 100, ImageNet (Deng et al. 2009). Flower102 (Nilsback & Zisserman 2008), and SDog 120 (Khosla et al. 2011): the other two are a text dataset- AG News (Zhang et al. 2015). and an audio dataset- Speech Commands (Warden 2018). Below, we provide brief description of each dataset.
CIFAR-10 and CIFAR-100 are widely used small image classification datasets. Both datasets contain 60,000 32x32 pixel colour images. CIFAR-I0 has 10 classes, each with 6,000 images, and C1FAR-100 has 100 classes, each with 600 images. The images include common animals and man- made objects, like cats, deers and ships.
ImageNet is large vision dataset for image classification with 1,000 classes. It contains 1.4 million labeled images of varying sizes, with an average resolution of 469x387 pixels. The images include wide variety of organisms and man-made objects, including different species of flowers and dogs. In the ModelReuse dataset, the knowledge of models trained on ImageNet are reused to label data for the Flower102 and SDog 22 datasets.

---

Published as a conference paper at ICLR 2022
Flower102 is image classification dataset that contains 102 classes of flowers. Each class has between 40 to 258 samples varying sizes, for a total of 8,189 images.
SDog120 is the 120 class Stanford Dogs dataset, an image classification dataset that contains 22,000 labeled dog images. Each class contains 150-200 samples of varying sizes larger than 200x200 pixels.
AG News is text dataset for topic classification. is a curated subset of the AG News Corpus, and contains 1 127,600 news articles in 4 topic classes, collected from web sources; each class contains 31,900 samples.
Speech Commands an audio dataset of spoken words for speech recognition. It contains 105,829 utterances of 35 words, performed by 2,618 speakers; each word is a class.
A.2 CLASSIFIERS
For the architectures of the classifiers, we used ResNer20 (He et al. 2016), ResNet50 (He et al., 2016), ResNet18 (He al. 2016). MobileNet-v2 (Sandler et al.) 2018). LSTM (Hochreiter & Schmidhuber 1997) and M5 Dai et al. al,,
In the vision domain, we used ResNet20 trained on CIFAR-10, and ResNet50 trained on CIFAR- 100. The ModelReuse benchmark we used contained ResNet18 and MobileNet-vv pre-trained on ImageNet, then reused for Flower102 and S2og120.
For the text domain, we used Long Short-Terr Memory (LSTM) model trained on the AG News dataset. For the audio domain, we used the an M5 model trained on the Speech Commands dataset.
We used publicly available implementations of all the classifiers, and include the code source for each the them in Appendix
A.3 SETTINGS FOR CASE STUDIES OF ZEST
Task
Reference classifier
Suspicious classifier
Model extraction
Victim classifter
Extracted copy of the vietim classifier
Machine unlearning
Original classifier
Directly or Approximately unlearned classifier
Table 2: Settings of our case studies.
A.4 TASKS IN MODELREUSE
We provide a brief description of the reuse tasks in the ModelReuse benchmark, taken as excerpts from al. (202T). Each of these tasks start with a ResNet18 MobileNetV2 model trained on Imagenet, then reuses the knowledge in some way to classify data points from the Flower102 and SDog120 datasets, described in Table See Table our complete results on the ModelReuse benchmark.
Reuse Task
Description
Transfer (tune X%)
Transfer each source model each target dataset, fine-tune the last X% layers.
Prune (X%)
Prune X% weights in each transferred model and fine-tune.
Quantize
Compress each transferred model using post-training weight quantization to the int8 data type.
Distill
Distill each transferred model target model with the same architecture using feature distillation.
Steal
Use the output of each transferred model to train a target model with different architecture.
Table 3: The reuse method used for each task in the ModelReuse benchmark.
12

---

Published as a conference paper at ICLR 2022
B CLASS ALIGNMENT
We evaluate our class alignment strategy in two different settings of comparing two classifiers C1 and C2 (we assume the classifiers have performance better than random guessing):
1. Both models are trained with the same architecture and on the same dataset but with differ- ent output label orders;
2. Each model is trained with the same architecture on different dataset and with different number of labels;
In setting (1), C1: is trained on CIFARIO (or CIFAR100) and C2: trained on CIFARIO (CIFAR but with randomly swapped labels. Our experimental results show that our class alignment strategy always able to correctly match the classes of two models in this setting. (e.g. matching the "cat" classes of the two models when the "cat" class of one of the models is indexed by 3 while the "cat" class in the other model is indexed by 7). The intuition behind this is that class alignment finds the class pairs with the most similar LIME masks, in other words, the class predictions are made using similar features of the inputs.
Setting (2) includes the transfer learning task of ModelReuse experiments. In this setting, models are trained on two different datasets with a different number of labels (1000 classes for ImageNet 1000 classes for ImageNet versus 102 for Flower102). These two
versus 120 for SDog120, datasets however share some common classes such as breeds of dog or type of flowers. For example in one of the experiments, C₁ is pre-trained on ImageNet with 1000 classes and C2 is fine-tuned C₁ on Flower 102 with 102 classes. The class alignment of these two classifiers with unrelated labels are follows:
1. Train LIME linear models of Or using 128 reference data points, and train LIME linear models C2 using 128 reference data points.
2. Concatenate weights of all LIME linear models of C class-wise to obtain S₁ (WP').. |..||WP as the order of labels are fixed, and concatenate weights of all LIME linear models of C22 class-wise to obtain S = (W?" ...
3. Setting reference classifier: we choose the classifier C2 that includes the lower number of classes (102 labels) as reference classifier;
4. Compute the Cosine distance between each row of S2 with all rows of Sj. We do this because assume that the semantic of labels and number of common labels are unknown.
5. Pick 102 pair rows with the minimum distances, determining 102 classes chosen out of 1000 classes of C1. This helps us to compare C1 and C2 based on the classes that perform most similarly to each other. This means that we might have pairs of aligned classes that are not related semantically, but this is expected and will contribute to telling the classifiers apart distance-wise.
In another example of the ModelReuse transfer learning experiments, we compare a ResNet18 trained on SDog 10 with another ResNet18 trained ImageNet. As mentioned in Appendix A SDog120 contains 120 breeds of dogs. On the other hand, ImageNet includes 130 breeds of dogs of its 1000 classes. In this experiment, our class alignment is able pick 120 similar breeds of dog from ImageNet which are most similar to the 120 breeds of dogs from SDog120.
C RELATIVE DISTANCE
We use both unbounded distance metrics and bounded distance. For better visualisation and fairer comparison, we scale all of them similarly. In particular, we scale the Zest distance of the compared models by the average Zest distance of unrelated models on similar tasks. Note that the unrelated models are obtained using different seeds. We define the relative distance as the ratio between the Zest distance of the compared models and the average Zest distance of multiple unrelated models in the last epoch:
Computed model distance
Relative distance
(1)
Average distance of multiple unrelated models
13

---

Published as a conference paper at ICLR 2022
D ABLATION STUDY ON THE VALUE OF K
C.v. Cemoand Cav. C
C. s. Guoonnod
Dioretos
Disamns
Corndaiod: B14952
crclaed 0 0035)
Ordeae: 0.0122)
150
175
200
125
130
Epoch
(a) k = 1, CIFAR-10
(b) l 1, CIFAR-100
Cav. s. Cormiveed Sgy. Cit
C 5. Guvemnte
Deunnce
Dasince
Oatrdloo 1.359)
Ordeted:
0
0
25
75
100
125
150
175
30
0
25
75
100
125
130
175
200
Epoch (1)
(c) = 10, CIFAR-IK 10
(d) l: 10, CIFAR-100
C Guomand
C. 201
v. C22 29
Dieumus
Outdbidd 13300)
0,5507
36.0%
Orband 00002
0.005
0-
Epoch
Epoch
(c) 20, CIFAR-10
(f) A: 20, CIFAR-100
Cil Cormaiid
Cill Growand
Cill
Cill s. Cit
Distnes
Dosunas
Grtdlaad 13359)
Corddaidd Eust02
$1,895
Orbbact (100
100
125
150
175
100
125
150
175
350
Epoch (1)
Flnnc
l 50, CIFAR-10
(h) A: 50. CIFAR-100
Figure 7: Impact of the value on the Zest distances between related models: we vary the value k and plot the distances between a model at training epoch & and itself after training for another k epochs for CIFAR-10 10 and CIFAR-100 100 datasets respectively.
In this section, we perform an ablation study on the value of i for both CIFAR- 10 and CIFAR- 100. Figure 7 shows the Zest distances between related models (i.e. the same model's 's checkpoints at epoch and epoch + k. where le is set to 1, 10, 20, 50 respectively), and between unrelated models. We can observe that
1. The Zest distances between related models obtained with different k values are significantly smaller than the Zest distances between unrelated models;
2. The larger the value k, the larger the Zest distance between related models. For example, the Zest distance between related CIFAR-10 models for k values of 1, 10, 20 and 50 are 0.0035, 0.0070, 0.0120 and 0.0630, respectively. Remember from Section 4.2 that each related pair contains reference classifier trained with epochs and another classifier at epoch t + k. Therefore, increasing the value of k results in more changes the parameters of the model over k epochs that are captured by our proposed method Zest.
14

---

Published as a conference paper at ICLR 2022
E EXTERNAL CODE SOURCES
We used open source code in our experiments to implement our ResNet-20 and ResNet-50 models, well the LIME algorithm. We also obtained the ModelReuse benchmark Li al. 2021 from the authors' repository. We list the URLs of the public code repositories we referenced in Table
Task
Code Source
Res.Nct-20 on CIFAR-I0
giddaah.cc///kkmmsssee///yroaa resnet cifar10
ResNet-550 CIFAR-100
LIME
g gichub.ccom/marccttr//111
ModelReuse benchmark LSTM on AG News
t ngrams tutorial.hhml
M5 on Speech Commands
mmand maand__eeognttiooooiibttbooooob
Table 4: Sources of codes used for experiments from external
F DISTANCE OF CLASSIFIERS IN MODELREUSE
Reuse Task
#Models
b Distance
Acc.
cos Distance
Acc.
Transfer (tune 10%)
4
0.745(1:..05)
100.0%
0.816(+0.040)
100.0%
Transfer (tune 50%)
4
0.682(+10.185)
100.0%
0.843(+0.041)
100.0%
Transfer (tune 100%)
4
0.703(++0.207)
100.0%
0.851(#0.037)
100.0%
Prune 20%
12
0.421(+0.143)
100.0%
0.218(#0.141)
100.0%
Prune 50%
12
0.476(+00128)
100.0%
0.276(+0.137)
100.0%
Prune 80%
12
0.829(+0.304)
58.33%
0.555(+0.229)
100.0%
Quantize
12
0.074(+0.155)
100.0%
0.067(+0.150)
100.0%
Distill
12
0.633(10.136)
100.0%
0.571(10.166)
100.0%
Steal
12
0.716(10.130)
100.0%
0.752(#0.131)
100.0%
Transfer prune 20%
12
0.692(+0.225)
100.0%
0.8614#0.040
100.0%
Transfer prune 50%
12
0.715(+0.217)
91.67%
0.856(#0.046)
100.0%
Transfer prune 80%
12
0.893(10:221)
75.00%
0.868(10.038)
100.0%
Transfer quantize
12
0.707(+0.171)
100.0%
0.840(40.045)
100.0%
Transfer distill
12
0.514(+0.127)
100.0%
0.909
100.0%
Overall
144
0.615(+0.282)
93.75%
0.634(+0.308
100.0%
Table 5: The comparison of Zest with ModelDiff on ModelReuse benchmark.
We provide a more complete results table for our experiments on the ModelReuse benchmark. Ta- ble provides more granularity the reuse tasks, breaking each task into subtasks (e.g. transfer is separated to three subtasks based on different levels tuning). We also include the average distance values and standard deviations, and show the results of using the l2 between the collections of linear approximations of the compared models. The Ea norm is signifi- cantly more sensitive pruning: the accuracy of Zest decreases significantly in the Prune 80% and Transfer + prune 50% or 80% tasks. In comparison, by using the cosine distance, Zest detects every reuse task across the board.
to measure the distance
G ABLATION STUDY ON THE CHOICE OF METRIC
We provide additional experiments on comparing Zest's performance when using different metrics to compute the distance between the global signatures. This is extension of our experiments in
$ $ Specifically, we repeat the experiments shown in Figures 2 and 6 using the & and É2 norms distance metrics. The respective experiments are distinguishing related and unrelated classifiers, detecting model extraction, and verifying machine unlearning. Our results show that for tasks, Zest can also capture the general similarity between compared models using & and /2 distances
15

---

Published as a conference paper at ICLR 2022
C.v. Cev. C
v. C
Dieguoo
Disaasss
175
Epoch
&1. CIFAR-10
(b) f1, CIFAR-100
Caty Gervitte Cavy
Cer Gonendee
Distnce
Dosence
100
125
1.0
135
30
1001
125
1.50
175
200
Epoch (O)
Epoch (1)
(c) la. CIFAR-I0
(d) E2. CIFAR-106 100
Figure Re-plotting Figure 2 using & and Eg distances.
Cigr & Carrired Guave V.S. Cara
Ca Gunsore
Dreatee
Ditame
Caccn V.S. Clese
100
125
150
175
20
100
125
130
175
300
Ep
Ipocch (1)
(a) f1, CIFAR-10
(b) f,, CIFAR-100
Cil Carmisdd
Cl Ceenine
Dietnee
Disseroc
Cotoct
100
125
150
175
200
100
125
130
175
Epoch:
Epoch
(c) l2. CIFAR-10
(d) lz. CIFAR-100
Figure 9: Re-plotting Figure 5 & and & distances. instead of the Cosine distance. We used the Cosine distance for our main experiments because performs better on average.
16

---

Published as a conference paper at ICLR 2022
C.... C.Curanee capny, a.
Diannce
comny, 5.
Distane
175
200
125
Ipooh
(a) l1, CIFAR-10
(b) l1, CIFAR-100
Cur Carmantd
Dieince
Car v. Chess
Co s.C.l
Disinos
v. Giaso
25
50
100
125
150
175
30
25
75
200
125
130
175
300
Ipoch (1)
(c) lz. CIFAR-10
(d) lz. CIFAR-100
Figure 10: Re-plotting Figure using and & distances.
17


Reviews:
[{"review_id": "ZsiMKcmelOC", "reviewer": "Reviewer_9qy5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is generally sensible and the experimental results look solid. The idea of using a technique which is somewhere between working in weight space and working strictly with input/output examples does make sense. Zest can distinguish between models which are snapshots of the same training process at different epochs as opposed to models trained on the same data but with initial starting points, which I think is fairly impressive, given than this distinction might be expected to be fairly subtle. Zest succeeds at detecting model extraction (aka stealing) where a competing method (ModelDiff) fails. The ability to distinguish between true unlearning and approximate unlearning is also encouraging. The paper is mostly well written, although there are some typos and some places where I would suggest edits (see below).\n\nMy main objection is that I would have liked to see Zest demonstrated on a domain other than image recognition. Testing on another domain would give me more confidence in the results, particularly given that Zest is model agnostic and should therefore be able to demonstrate success on a wide variety of domains and model classes. I also think the model stealing problem has relevance for smaller-scale tabular ML models used in social science/finance areas like credit screening, crime recidivism forecasting, etc and it would have been great to see something like that here, tested presumably on some sort of tree ensemble like gradient boosting. \n\nI also think the applications (model similarity for model stealing, unlearning), while certainly cutting-edge, are only of moderate importance to ML research right now. \n\nOn the balance, I think the paper should probably be accepted but the limited range of experiments and moderate topic significance prevent me from giving it a very high grade.\n\n*** Update after author rebuttal *** The authors have added additional experiments in text and audio and have offered additional information regarding the significance of the application domains ( model stealing and unlearning). For these reasons, I am increasing my score to an 8.\n\nSome suggested edits:\n\nPage 2: \n2nd bullet point.  Also, on CIFAR-100, -< add comma\n3rd bullet again need comma after CIFAR-100,\nAn 0% -> a 0%\nThroughout our discussion, however -> throughout our discussion. However,  (start a new sentence here)\nPast work on measuring model similarity have -> Past work on measuring model similarity has\n\nPage 3 \nweights of a DNN captures-> weights of a DNN capture\n\nPage 5\nAre in the same size-> are the same size\n\nPage 7\nThis is again over-performed by Zest -> this problem is solved by Zest.\n\nPage 8\n\n\"To further our evaluation, we also compare our proposed method Zest using the model reuse bench mark introduced with ModelDiff (Li et al., 2021). To ensure a fair comparison, we use the Mod elReuse benchmark provided by ModelDiff:\" I think this sentence is basically being repeated, unnecessarily?\n\nTable 4 shows that Zest and ModelDiff have similar performance…I think this is Table 2, not Table 4?", "summary_of_the_paper": "The paper presents a method (Zest) for measuring the distance or similarity between 2 supervised machine learning models while only requiring black-box access to the model's inputs and outputs but while also providing various improvements over the method of simply comparing model outputs themselves on a finite set of data points. These improvements are achieved by fitting locally linear models in the region of various reference data points, following the well-known LIME algorithm which was originally introduced for the purpose of model explanations. Experiments using DNNs for image recognition on CIFAR benchmark datasets are presented. The method is demonstrated on 2 specific applications. The first is the detection of model stealing, where the input/output examples of a model are used to reproduce it and therefore steal the associated intellectual property.  The second is the evaluation of unlearning, where a model needs to \"forget\" specific data and the Zest method is able to distinguish between 2 different unlearning methods in terms of their success in unlearning properly.", "summary_of_the_review": "The Zest method presented here is sensible, clearly presented and shown to be successful for a couple of applications (model stealing and unlearning evaluation). It would have been stronger if it had been tested additionally on another problem domain and another ML model class rather than focusing solely on DNNs and image recognition. The topics are also only of moderate significance. For these reasons, I recommend a weak accept. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OoBHowy8964", "reviewer": "Reviewer_MKjr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths.\n\nThe general method of the paper is presented relatively clear, and a series of experiments have shown the advantage of the presented method. \n\nWeaknesses.\n\nBased on the introduction, it seems that existing literature on measuring model distances only includes Li et al. 2021 and Jia et al. 2021. The authors directly point out the limit of these two papers and propose a new method. It is hard to evaluate the challenges of this research area without more context. \n\nThe paper mentions that Zest can also be applied to other domains like text and audio. Given that this paper focuses more on the empirical side, it would be good to include more diverse experiments and add applications in text/audio domains into the experiment section of the paper. \n\nFigure 1 is not clear enough on its own, especially Phase 3 & 4, and the caption does not contain enough information/description to help the audience understand directly. Also, it would be better if the LIME part can be more abstract since LIME algorithm is not this paper's contribution (similarly for Algorithm 1). \n\nRegarding class alignment step in Algorithm 1, does any of the experiments use class alignment? In general, when the two classifiers have unrelated labels, how could this alignment step work? More explanations on this may be required. ", "summary_of_the_paper": "The paper proposes an architecture-independent way of measuring distance between two models for the same type of input data. More specifically, given each reference sample, it first calculates a linear weight based on a set of masked samples and their corresponding outputs for each model, then calculates the distance between the set of weights for all reference samples generated from the two models. The method can be applied to help detect model stealing and inform unlearning verification.", "summary_of_the_review": "The paper is an empirical study on a topic that is not very popular (at least based on how the paper discusses about the existing literature). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J-gFdJ7osZv", "reviewer": "Reviewer_GFcZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper utilizes a model explanation methods called LIME to design a new method to calculating the distance between two machine learning models. Overall, the problem is well motivated by stating the limitations of current approaches, and the proposed method is clearly presented with experiments to support its benefits:\n\n1. The authors clearly states the limitations of existing approaches, namely architecture dependent or dependence on the choice of model inputs. The proposed method interpolates between two extremes and thus alleviate both drawbacks. \n\n2. The authors provide a thorough review  related work, and introduce the proposed method clearly. \n\n3. The experiments are carefully designed to demonstrate the benefits of the proposed method. \n\nIn general, I have the following several concerns: \n\n1. Algorithm 1 line 9 - 10, or in particular the way to deal with class alignment. I feel its a bit unclear: looks like we care finding the minimum alignment from all possible alignments. But if it actually due to class mismatch, is it possible that we get a small distance while the models are actually different? For example, model 1 predicts class A while model 2 predict class B using similar weights.\n\n2. Algorithm 1 line 12, the distance is chosen to be a cosine operation, so why it will be larger than 1 in some of the experiments? \n\n3. Section 4.2, any reason to consider k = 5? It might be helpful to include additional experiment results varying the values of k.\n\n4. For the applications mentioned, I cannot help my self thinking of knowledge distillation, which seems to be closely related to model stealing. Is there any related work discussing this or even further, can ZEST find application for knowledge distillations? ", "summary_of_the_paper": "The paper proposes a new method to calculate the distance between two machine learning models. By comparing using local approximations (LIME) and compute the Cosine distance, this yields an architecture-independent method operating in weight space. It has proves effectiveness in applications including detecting model stealing and verifying machine unlearning. ", "summary_of_the_review": "Overall I think this is a good paper and can be accepted to ICLR. Although it does not have much theoretical novelty, I believe applying LIME in the area of calculating distance between two models is pretty new. If my listed concerns can be addressed, I willing to increase my ratings one level further. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5R4gykX5MG8", "reviewer": "Reviewer_vdBq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is interesting and well written, although quite technical in scope.\nAll claims seem correct and quite convincing, and definitely useful (especially in the case of model stealing).\nHowever, the validation of the proposed framework is limited to a single family of classifiers (ResNet) and a quite standard (and somewhat simple) benchmark dataset: such choice leaves room for wondering about the effectiveness of Zest on a broader landscape.\nIn particular, a deeper analysis of the Zest behaviour on separating related from unrelated classifiers (in different settings) would be recommended, for instance toward the definition of a (clearly empirical) quantitative distance threshold to be used as a reference for the Zest values.\n", "summary_of_the_paper": "The authors propose a novel approach for computing an architecture-independent distance metric to assess the similarity between ML models by comparing the global behaviour of the models as approximated through LIME. They demonstrate the method by applying it on instances of the CIFAR dataset, focussing on the two tasks of model stealing and machine unlearning.", "summary_of_the_review": "Paper is interesting, adding a significant novel contribution to the existing literature and providing an effective solution in at least two important and somehow still open ML issues. A wider analysis (both in terms of models and data) would be recommended to strengthen the authors' claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method, called Zest, to measure the similarity between two supervised machine learning models based on their model explanations computed by the LIME feature attribution method.  The technical novelty and significant are high, and results are strong.  Reviewers had clarifying questions regarding experiments and suggestions to add experiments, which involve additional domains (text and audio) and different families of classifiers, and more contexts based on prior literatures. These were adequately addressed by the authors. Overall, this paper deserves borderline acceptance.
Decision: Accept (Poster)