A Zest of LIME: Towards Architecture-Independent Model Distances

1 INTRODUCTION . We explore the problem of quantifying similarity between machine learning ( ML ) models with distance metrics , with a focus on deep neural networks ( DNNs ) . Comparing the functional behavior of ML models is often challenging because they are not easily inspectable ( e.g . located in the cloud ) . In addition , ML models that capture similar knowledge may not share similar architectures , and vice versa , making it difficult to compare models in weight space . Finally , ML models that solve different tasks may nonetheless share similar knowledge , such as in the case of transfer learning ( Li et al. , 2021 ) . This is emphasized in DNNs , which are not only complex , often containing millions of parameters , but also difficult to interpret ( Ribeiro et al. , 2016 ) . Previous methods for measuring model distances use either the similarity of predictions ( Li et al. , 2021 ) , or the similarity of weights ( Jia et al. , 2021 ) . However , methods that measure the similarity of predictions only capture the local behavior of models at each data point , and it is hard to choose a set of inputs to compare their global behaviors . This causes high variance in the results : models that are unrelated may happen to give very similar predictions on some inputs , while related models ( e.g. , fine-tuned from one another ) may give different predictions on other inputs . Methods that measure the similarity of weights are more complete , because they capture the global behavior of models , and weight similarity implies prediction similarity in the limit : as the weights of the compared models approach each other , so will their predictions . But , to be meaningful , weight-space methods can only be applied when the compared models share the same architecture . Furthermore , they require white-box access to the models to compare weights . These constraints restrict their use in practice . Instead , we propose to compute distances between models by first approximating their global behavior through a collection of linear models . We name our approach Zest . We start by applying the Local Interpretable Model-Agnostic Explanations ( LIME ) algorithm of Ribeiro et al . ( 2016 ) to generate a collection of linear models that each approximates some local behavior of a compared model . We chose LIME because it is black-box ( only requires access to the models ’ predictions ) and architecture-independent , and as a consequence so is Zest . The collection of these local approximations form a global approximation of model behavior . Since the approximation models obtained are linear , they are straightforward to compare . Indeed , to measure similarities and differences between two models being compared , we compute the Cosine distance between the concatenated weights of their respective collections of approximated linear models . Because we compare local linear approximations of the models rather than their predictions directly , this allows Zest to interpolate between the benefits of prior approaches in the weight or prediction spaces described earlier . In particular , our empirical validation of Zest demonstrates that our approach is not sensitive to the choice of points used by LIME to locally approximate the models being compared . This departs from prior approaches that compared model predictions directly and as a result were overly sensitive to the choice of points the comparison is made on . This characteristic allows Zest to correctly capture the differences and similarities between pairs of unrelated or related classifiers . Furthermore , we show that the distance computed by Zest finds natural applications in detecting model reuse ( e.g. , model stealing , knowledge distillation , and transfer learning ) and verifying machine unlearning . Li et al . ( 2021 ) first proposed to introduce a distance to inform the detection of classifiers which were trained by reusing another classifier . In particular , this is the case in model extraction where an adversary observes the predictions of a model to reproduce its behavior in a stolen copy of the model . Whereas Li et al . ( 2021 ) left the identification of stolen copies of a model as future work after failing to apply their proposed ModelDiff metric to this problem , we find it can be solved by Zest . In a completely different setting , we also find an application of Zest to machine unlearning : the process of removing the effects of learning data points from a model ( Cao & Yang , 2015 ) . We use Zest as a heuristic to inform whether or not a user ’ s data has been unlearned , by comparing the decision boundaries of the models before and after unlearning around that data . Our main contributions are as follows : • We propose Zest , an approach for computing an architecture-independent distance metric that measures the similarity between ML models . Zest only requires black-box access , and compares the global behavior of the models , which it approximates using LIME . • We validate that Zest measures model similarity by showing that the distance between different epochs of the same model is on average 99.27 ( ±0.19 ) % on CIFAR-10 , 98.75 ( ±0.12 ) % on CIFAR-100 , 92.68 ( ±2.22 ) % on AG News and 81.17 ( ±3.01 ) % on Speech Commands , closer than the distance between two models with different initialization trained on the same dataset . We run our experiments in the vision , text and audio domains using the ResNet18 , ResNet20 , ResNet50 , LSTM , M5 , and MobileNet-v2 architectures , on the CIFAR-10 , CIFAR-100 , AG News , Speech Commands , ImageNet , Flower102 and SDog120 datasets . • We show that Zest can be used to help detect model reuse , including the hard case of model stealing , and inform unlearning verification . The distance between a model stolen via model extraction and the original model is 19.96 ( ±4.28 ) % on CIFAR-10 and 54.87 ( ±1.70 ) % on CIFAR100 , closer on average than the distance between the original and a model retrained on the same datasets . Compared to ModelDiff , which had a 0 % accuracy at detecting model extraction , we have 100 % accuracy for the attacks in the ModelReuse benchmark ( Li et al. , 2021 ) . We further show that Zest helps identify all other methods of model reuse in the benchmark . To inform unlearning verification , we show that the distance from a model to its retrained counterpart without the unlearned data is on average 144.56 ( ±15.39 ) % on CIFAR-10 and 140.51 ( ±9.10 ) % on CIFAR-100 further than the distance to a retrained model that did not exclude the data to unlearn . 2 BACKGROUND AND PROBLEM STATEMENT : DEFINING MODEL DISTANCE . We aim to compute a distance between two ML models C1 and C2 to characterize their similarity , given only the ability to observe the models ’ predictions on chosen inputs . We make no assumptions on the models ’ architectures , except that they have the same input dimensions . In other words , we only assume that they model similar distributions ( e.g . traffic signs ) , but the compared models can be trained for different tasks ( e.g . classifying German and American traffic signs respectively ) . We denote the output space of the two models D1 and D2 . We focus on DNNs , due to their popularity and the difficulty to interpret their behavior , but our distance metric applies to other ML models . For ease of exposition , we use the classification of images as a sample task throughout our discussion in §2 and §3 . We also evaluate Zest on the text and audio domains and provide our results in §4 . Past work on measuring model similarity has either defined distance by comparing model weights or comparing model predictions . We briefly overview both approaches and their current limitations , then give an outline of the LIME algorithm and how it can be applied to address these limitations . Comparing model weights . The weights of a parameterized model define its behavior and it can be seen how in the limit having close weights implies that two models will have close behavior ; thus , one way to compare the similarity of models is to compare the difference in their weights . One example of weight space comparison is proof of learning : Jia et al . ( 2021 ) use the comparison of model weights throughout the training procedure to provide evidence that a party has indeed trained a model . This is useful for ownership resolution . By saving checkpoints of the model during training , and showing that the model weights between subsequent checkpoints are close , the owner can provide a history that proves they are the party who trained the model . However , comparing model weights is highly restrictive in practice : it requires both access to model weights , and for compared models to share the same architecture , so that the distance between each corresponding weight value can be compared . Even if models share the same architecture , it can be seen that two models whose weights are permutations of one another on a single layer would make exactly the same predictions yet have a non-zero distance in weight space . Comparing model predictions . On the other hand , a model ’ s behavior can also be defined by its predictions on inputs . The advantage of comparing model similarity with predictions is that we can treat the models as black-boxes . Because we only need to observe input-output pairs , we do not need access to model weights and are not restricted by the architecture of the models being compared . In theory , if we can query a model with every input in the domain and get its predictions , we can learn its exact functional behavior , but this is intractable in practice . To address this , current approaches resort to selecting smaller sets of inputs as reference , and comparing the models based on their predictions on the reference . The disadvantage of these approaches is that ( a ) they can only capture point-wise behavior of the models at the chosen inputs , and ( b ) choosing a representative set of inputs is hard . If the chosen inputs are far from decision boundaries then even unrelated well-trained DNNs give similar predictions ; whereas inputs near decision boundaries are sensitive to small changes of the boundaries , so related models ( e.g . a model and its transferred copy ) may predict differently . This is consistent with findings in deep learning testing , a class of methods that aim to identify areas of the input space where supposedly similar models disagree with each other on the output space . DiffChaser ( Xie et al. , 2019 ) , DeepXplore ( Pei et al. , 2017 ) and TensorFuzz ( Odena et al. , 2019 ) are three recent and popular testing frameworks . These are not applicable in our case because they focus on the differences between closely related models , whereas we are looking for a distance metric that is able to provide information on both the similarity and differences between any kinds of models . Perhaps the closest to our work , ModelDiff ( Li et al. , 2021 ) is a model distance metric designed to detect model reuse , which is defined as training a model based on another to save on training costs . ModelDiff feeds a set of paired test inputs to the compared models , where each pair is selected to be close in the input space , but far in the output space . It then compares the total Cosine similarity of the distance between the outputs of each pair to find the model distance . Using this distance metric , Li et al . ( 2021 ) detected transfer learning and model compression , however , they fail at the harder case of detecting model extraction and leave it to future work . Our approach solves this open problem , and our empirical study demonstrates that our distance metric is more precise and less sensitive to changes of the model that do not affect its performance on in-distribution data . LIME . Local Interpretable Model-agnostic Explanations generates interpretable linear models to explain the local behavior of ML models ( Ribeiro et al. , 2016 ) . LIME learns linear models by segmenting the input space , then systematically masking segments to identify features that are most influential to the classification of a data point . For example , this implies masking contiguous areas of pixels for image classification tasks . Using the most influential features , LIME learns a linear model that approximates the decision boundary of the compared model near the data point . Intuition of comparing in weight space of LIME . From the perspective of the decision boundaries , weights of a DNN capture the entire boundaries , just like how x2 + y2 = 1 captures a unit circle . This leads to high precision but restricts generality . For example , it is hard to define how x2+y2 = 1 similar is to max ( x , y ) = 1 . On the other hand , each prediction only reveals one specific mapping from the input to the output space , and multiple predictions are needed to identify a tiny region of the decision boundaries . If we draw an analogy to the Cartesian coordinate system , comparing model predictions is like comparing points in the coordinate system–the method is more general , but both a unit circle and a unit square can have points like [ 0 , 1 ] . The method will not be precise if we compare the points and conclude the circle is similar to the square . At a high level , we are dealing with the trade-off between precision and generality . Comparing predictions gives us general applicability whereas comparing weights gives us precision . Thus , to interpolate between these two extremes we choose to compare linear models that locally approximate the models to be compared . Using the same analogy , comparing LIME of two DNNs is similar to comparing lines that are tangent to the shape , instead of the formula of the shape ( comparing weights ) or points on the shape ( comparing predictions ) . We choose the LIME algorithm for generating local approximations because it is architecture-independent , only requires black-box access , and is simple to understand . § 3 contains a more formal description of how we apply LIME . 3 METHOD : MEASURING DISTANCES BETWEEN MODELS WITH ZEST Building on the insights of § 2 , we now introduce an approach that is able to interpolate between the benefits of weight-space and prediction-space comparisons of two classifiers . We propose Zest , a model distance metric that measures the similarity of two models C1 and C2 by comparing their global behaviour as approximated by LIME ( see Figure 1 for an overview ) . Algorithm 1 describes our proposed model distance , which consists of four sequential phases : 1 ) Sampling a reference dataset ; 2 ) Sampling nearby images as done in LIME ; 3 ) Learning the behaviour of C1 and C2 locally around the reference data points with linear classifiers as done in LIME ; 4 ) Approximating a global perspective of C1 and C2 by combining these linear models , and computing the distance between the combinations of linear models corresponding to C1 and C2 . Next , we describe each phase of our proposed model distance in detail . Sampling representative reference data points . We randomly sample N reference images X = { xi } Ni=1 from the distribution task . Note that N is much smaller than the size of the training set of the classifiers being compared . For example , we select N = 128 reference images from 50 , 000 training images of CIFAR-10 . In § 4.3 , we show empirically that the choice of N does not qualitatively change the outcome of our model distance metric computations—unlike previous approaches described in § 2 that directly compared the predictions of two classifiers . Increasing N simply reduces the variance of our metric , which is intuitive because the comparison is made on more points sampled from the underlying data distribution ; so reference points are more likely to cover the same local regions of the classifiers ’ decision surface as N increases . Learning the local behaviours . While the classifiers being compared can be non-linear ( e.g. , deep neural networks in our case ) , the intuition behind LIME is that their behavior can be approximated locally by a linear model . We learn the local behaviour of each of the two classifiers being compared around each reference image by training a LIME linear regression model that captures the classifier responses to the image ’ s components ( i.e . contiguous patches of pixels denoted a super-pixel ) . Similar to the original LIME paper , we create and sample image components as follows . First , we group the pixels of a reference image in K super-pixels using Quickshift ( Vedaldi & Soatto , 2008 ) and create binary super-pixel masks B = { bk } Kk=1 , where each super-pixel binary mask bk ∈ { 0 , 1 } w×h specifies the location of pixels belong to the k-th super-pixel . These super-pixels Algorithm 1 : Proposed model distance approach , Zest Input : Black-box D1-class classifier C1 , Black-box D2-class classifier C2 , Reference image set X = { xi } Ni=1 , Quickshift segmentation algorithm , Random-Selector , Linear regression training algorithm T ( · , · ) and distance Distance ( · , · ) Output : Distance , d , between black-box classifiers C1 and C2 1 : S1 , S2 = { } , { } . Initialize model signatures 2 : for x ∈ X do 3 : B = { bk } Kk=1 = Quickshift ( x ) . Generate K super-pixel masks 4 : M = { ml } Ll=1 = Random-Selector ( B ) . Generate L image ’ s component masks 5 : X̂ = { x̂l : x̂l = x ·ml } Ll=1 . Generate L image ’ s components 6 : Z1 = C1 ( X̂ ) , Z2 = C2 ( X̂ ) . Query classifier 7 : W1 : T ( M , Z1 ) , W2 : T ( M , Z2 ) . Train local regression model 8 : if Unknown relation between D1 and D2 then 9 : for i ∈ D1 do 10 : W2 [ i ] = W2 [ argminj∈D2 ( ‖W1 [ i ] −W2 [ j ] ‖ ) ] . Class alignment 11 : S1.append ( W1 ) , S2.append ( W2 ) . Create model signature 12 : d = Distance ( S1 , S2 ) . Compute model distance each represent a contiguous patch of similar pixels , like a patch of fur or grass , whose presence or absence is controlled by the binary mask . A linear model on these super-pixels provides an interpretable approximation of the local decision boundary . For example , if the classifier predicts “ dog ” when the patch of fur is present , but changes its prediction when it is not , then we can interpret that the patch of fur greatly influences its prediction . Then , we create the reference image ’ s component x̂ = x ·m by randomly choosing several superpixels identified by 1s in component mask m ∈ { 0 , 1 } w×h that is combination of their corresponding binary mask . In order to explore the local area around each reference image , we repeat this process and draw L different components X̂ = { x̂l } Ll=1 using L different component mask M = { ml } Ll=1 . Following the original LIME paper , we set L = 1000 . We query classifiers to obtain their predictions on these L image components as Z1 = C1 ( X̂ ) and Z2 = C2 ( X̂ ) . Then , we train a linear regression model I : M→ Z per classifier to have high local fidelity with respect to the classifier around the reference image x . We call the weights of the trained regression model the local signature of its corresponding classifier . Approximating a global perspective . In total , we learn N linear regression models , capturing the behaviour of the classifier locally around the N reference images . We approximate the global behaviour of each classifier by concatenating its N local signatures S1 = { Wn1 } Nn=1 . Note that local signatures of each classifier have as many rows as the number of classes of the classifier . In order to compare signatures of classifiers with different number of classes when the relationship between classes output by the two classifiers is unknown , we introduce class alignment to re-position and reshape the rows of local signatures . Let small and big local signature be the local signatures of the classifiers with the lower and higher number of classes respectively . In class alignment , we start by the first row of the small local signature and compute its distance with each row in the big local signature . Then , we move the row corresponding to the minimum distance to the first row of of big local signature . We repeat this for all other rows in the small local signature while removing aligned rows of the big local signature from this process . Finally , we remove any remaining row of the big local signature not selected previously . See Appendix B for further details . Computing the distance . As the global signature of classifiers are the same size , we compute the model distance as d = Distance ( S1 , S2 ) . In Appendix G , we show that the choice of Distance ( · , · ) does not affect the performance of Zest . Our experimental results in the remainder of the paper are thus presented with the Cosine distance being used to compare global signatures .

Summary: The paper presents a method (Zest) for measuring the distance or similarity between 2 supervised machine learning models while only requiring black-box access to the model's inputs and outputs but while also providing various improvements over the method of simply comparing model outputs themselves on a finite set of data points. These improvements are achieved by fitting locally linear models in the region of various reference data points, following the well-known LIME algorithm which was originally introduced for the purpose of model explanations. Experiments using DNNs for image recognition on CIFAR benchmark datasets are presented. The method is demonstrated on 2 specific applications. The first is the detection of model stealing, where the input/output examples of a model are used to reproduce it and therefore steal the associated intellectual property.  The second is the evaluation of unlearning, where a model needs to "forget" specific data and the Zest method is able to distinguish between 2 different unlearning methods in terms of their success in unlearning properly.