A Zest of LIME: Towards Architecture-Independent Model Distances

1 INTRODUCTION . We explore the problem of quantifying similarity between machine learning ( ML ) models with distance metrics , with a focus on deep neural networks ( DNNs ) . Comparing the functional behavior of ML models is often challenging because they are not easily inspectable ( e.g . located in the cloud ) . In addition , ML models that capture similar knowledge may not share similar architectures , and vice versa , making it difficult to compare models in weight space . Finally , ML models that solve different tasks may nonetheless share similar knowledge , such as in the case of transfer learning ( Li et al. , 2021 ) . This is emphasized in DNNs , which are not only complex , often containing millions of parameters , but also difficult to interpret ( Ribeiro et al. , 2016 ) . Previous methods for measuring model distances use either the similarity of predictions ( Li et al. , 2021 ) , or the similarity of weights ( Jia et al. , 2021 ) . However , methods that measure the similarity of predictions only capture the local behavior of models at each data point , and it is hard to choose a set of inputs to compare their global behaviors . This causes high variance in the results : models that are unrelated may happen to give very similar predictions on some inputs , while related models ( e.g. , fine-tuned from one another ) may give different predictions on other inputs . Methods that measure the similarity of weights are more complete , because they capture the global behavior of models , and weight similarity implies prediction similarity in the limit : as the weights of the compared models approach each other , so will their predictions . But , to be meaningful , weight-space methods can only be applied when the compared models share the same architecture . Furthermore , they require white-box access to the models to compare weights . These constraints restrict their use in practice . Instead , we propose to compute distances between models by first approximating their global behavior through a collection of linear models . We name our approach Zest . We start by applying the Local Interpretable Model-Agnostic Explanations ( LIME ) algorithm of Ribeiro et al . ( 2016 ) to generate a collection of linear models that each approximates some local behavior of a compared model . We chose LIME because it is black-box ( only requires access to the models ’ predictions ) and architecture-independent , and as a consequence so is Zest . The collection of these local approximations form a global approximation of model behavior . Since the approximation models obtained are linear , they are straightforward to compare . Indeed , to measure similarities and differences between two models being compared , we compute the Cosine distance between the concatenated weights of their respective collections of approximated linear models . Because we compare local linear approximations of the models rather than their predictions directly , this allows Zest to interpolate between the benefits of prior approaches in the weight or prediction spaces described earlier . In particular , our empirical validation of Zest demonstrates that our approach is not sensitive to the choice of points used by LIME to locally approximate the models being compared . This departs from prior approaches that compared model predictions directly and as a result were overly sensitive to the choice of points the comparison is made on . This characteristic allows Zest to correctly capture the differences and similarities between pairs of unrelated or related classifiers . Furthermore , we show that the distance computed by Zest finds natural applications in detecting model reuse ( e.g. , model stealing , knowledge distillation , and transfer learning ) and verifying machine unlearning . Li et al . ( 2021 ) first proposed to introduce a distance to inform the detection of classifiers which were trained by reusing another classifier . In particular , this is the case in model extraction where an adversary observes the predictions of a model to reproduce its behavior in a stolen copy of the model . Whereas Li et al . ( 2021 ) left the identification of stolen copies of a model as future work after failing to apply their proposed ModelDiff metric to this problem , we find it can be solved by Zest . In a completely different setting , we also find an application of Zest to machine unlearning : the process of removing the effects of learning data points from a model ( Cao & Yang , 2015 ) . We use Zest as a heuristic to inform whether or not a user ’ s data has been unlearned , by comparing the decision boundaries of the models before and after unlearning around that data . Our main contributions are as follows : • We propose Zest , an approach for computing an architecture-independent distance metric that measures the similarity between ML models . Zest only requires black-box access , and compares the global behavior of the models , which it approximates using LIME . • We validate that Zest measures model similarity by showing that the distance between different epochs of the same model is on average 99.27 ( ±0.19 ) % on CIFAR-10 , 98.75 ( ±0.12 ) % on CIFAR-100 , 92.68 ( ±2.22 ) % on AG News and 81.17 ( ±3.01 ) % on Speech Commands , closer than the distance between two models with different initialization trained on the same dataset . We run our experiments in the vision , text and audio domains using the ResNet18 , ResNet20 , ResNet50 , LSTM , M5 , and MobileNet-v2 architectures , on the CIFAR-10 , CIFAR-100 , AG News , Speech Commands , ImageNet , Flower102 and SDog120 datasets . • We show that Zest can be used to help detect model reuse , including the hard case of model stealing , and inform unlearning verification . The distance between a model stolen via model extraction and the original model is 19.96 ( ±4.28 ) % on CIFAR-10 and 54.87 ( ±1.70 ) % on CIFAR100 , closer on average than the distance between the original and a model retrained on the same datasets . Compared to ModelDiff , which had a 0 % accuracy at detecting model extraction , we have 100 % accuracy for the attacks in the ModelReuse benchmark ( Li et al. , 2021 ) . We further show that Zest helps identify all other methods of model reuse in the benchmark . To inform unlearning verification , we show that the distance from a model to its retrained counterpart without the unlearned data is on average 144.56 ( ±15.39 ) % on CIFAR-10 and 140.51 ( ±9.10 ) % on CIFAR-100 further than the distance to a retrained model that did not exclude the data to unlearn . 2 BACKGROUND AND PROBLEM STATEMENT : DEFINING MODEL DISTANCE . We aim to compute a distance between two ML models C1 and C2 to characterize their similarity , given only the ability to observe the models ’ predictions on chosen inputs . We make no assumptions on the models ’ architectures , except that they have the same input dimensions . In other words , we only assume that they model similar distributions ( e.g . traffic signs ) , but the compared models can be trained for different tasks ( e.g . classifying German and American traffic signs respectively ) . We denote the output space of the two models D1 and D2 . We focus on DNNs , due to their popularity and the difficulty to interpret their behavior , but our distance metric applies to other ML models . For ease of exposition , we use the classification of images as a sample task throughout our discussion in §2 and §3 . We also evaluate Zest on the text and audio domains and provide our results in §4 . Past work on measuring model similarity has either defined distance by comparing model weights or comparing model predictions . We briefly overview both approaches and their current limitations , then give an outline of the LIME algorithm and how it can be applied to address these limitations . Comparing model weights . The weights of a parameterized model define its behavior and it can be seen how in the limit having close weights implies that two models will have close behavior ; thus , one way to compare the similarity of models is to compare the difference in their weights . One example of weight space comparison is proof of learning : Jia et al . ( 2021 ) use the comparison of model weights throughout the training procedure to provide evidence that a party has indeed trained a model . This is useful for ownership resolution . By saving checkpoints of the model during training , and showing that the model weights between subsequent checkpoints are close , the owner can provide a history that proves they are the party who trained the model . However , comparing model weights is highly restrictive in practice : it requires both access to model weights , and for compared models to share the same architecture , so that the distance between each corresponding weight value can be compared . Even if models share the same architecture , it can be seen that two models whose weights are permutations of one another on a single layer would make exactly the same predictions yet have a non-zero distance in weight space . Comparing model predictions . On the other hand , a model ’ s behavior can also be defined by its predictions on inputs . The advantage of comparing model similarity with predictions is that we can treat the models as black-boxes . Because we only need to observe input-output pairs , we do not need access to model weights and are not restricted by the architecture of the models being compared . In theory , if we can query a model with every input in the domain and get its predictions , we can learn its exact functional behavior , but this is intractable in practice . To address this , current approaches resort to selecting smaller sets of inputs as reference , and comparing the models based on their predictions on the reference . The disadvantage of these approaches is that ( a ) they can only capture point-wise behavior of the models at the chosen inputs , and ( b ) choosing a representative set of inputs is hard . If the chosen inputs are far from decision boundaries then even unrelated well-trained DNNs give similar predictions ; whereas inputs near decision boundaries are sensitive to small changes of the boundaries , so related models ( e.g . a model and its transferred copy ) may predict differently . This is consistent with findings in deep learning testing , a class of methods that aim to identify areas of the input space where supposedly similar models disagree with each other on the output space . DiffChaser ( Xie et al. , 2019 ) , DeepXplore ( Pei et al. , 2017 ) and TensorFuzz ( Odena et al. , 2019 ) are three recent and popular testing frameworks . These are not applicable in our case because they focus on the differences between closely related models , whereas we are looking for a distance metric that is able to provide information on both the similarity and differences between any kinds of models . Perhaps the closest to our work , ModelDiff ( Li et al. , 2021 ) is a model distance metric designed to detect model reuse , which is defined as training a model based on another to save on training costs . ModelDiff feeds a set of paired test inputs to the compared models , where each pair is selected to be close in the input space , but far in the output space . It then compares the total Cosine similarity of the distance between the outputs of each pair to find the model distance . Using this distance metric , Li et al . ( 2021 ) detected transfer learning and model compression , however , they fail at the harder case of detecting model extraction and leave it to future work . Our approach solves this open problem , and our empirical study demonstrates that our distance metric is more precise and less sensitive to changes of the model that do not affect its performance on in-distribution data . LIME . Local Interpretable Model-agnostic Explanations generates interpretable linear models to explain the local behavior of ML models ( Ribeiro et al. , 2016 ) . LIME learns linear models by segmenting the input space , then systematically masking segments to identify features that are most influential to the classification of a data point . For example , this implies masking contiguous areas of pixels for image classification tasks . Using the most influential features , LIME learns a linear model that approximates the decision boundary of the compared model near the data point . Intuition of comparing in weight space of LIME . From the perspective of the decision boundaries , weights of a DNN capture the entire boundaries , just like how x2 + y2 = 1 captures a unit circle . This leads to high precision but restricts generality . For example , it is hard to define how x2+y2 = 1 similar is to max ( x , y ) = 1 . On the other hand , each prediction only reveals one specific mapping from the input to the output space , and multiple predictions are needed to identify a tiny region of the decision boundaries . If we draw an analogy to the Cartesian coordinate system , comparing model predictions is like comparing points in the coordinate system–the method is more general , but both a unit circle and a unit square can have points like [ 0 , 1 ] . The method will not be precise if we compare the points and conclude the circle is similar to the square . At a high level , we are dealing with the trade-off between precision and generality . Comparing predictions gives us general applicability whereas comparing weights gives us precision . Thus , to interpolate between these two extremes we choose to compare linear models that locally approximate the models to be compared . Using the same analogy , comparing LIME of two DNNs is similar to comparing lines that are tangent to the shape , instead of the formula of the shape ( comparing weights ) or points on the shape ( comparing predictions ) . We choose the LIME algorithm for generating local approximations because it is architecture-independent , only requires black-box access , and is simple to understand . § 3 contains a more formal description of how we apply LIME . 3 METHOD : MEASURING DISTANCES BETWEEN MODELS WITH ZEST Building on the insights of § 2 , we now introduce an approach that is able to interpolate between the benefits of weight-space and prediction-space comparisons of two classifiers . We propose Zest , a model distance metric that measures the similarity of two models C1 and C2 by comparing their global behaviour as approximated by LIME ( see Figure 1 for an overview ) . Algorithm 1 describes our proposed model distance , which consists of four sequential phases : 1 ) Sampling a reference dataset ; 2 ) Sampling nearby images as done in LIME ; 3 ) Learning the behaviour of C1 and C2 locally around the reference data points with linear classifiers as done in LIME ; 4 ) Approximating a global perspective of C1 and C2 by combining these linear models , and computing the distance between the combinations of linear models corresponding to C1 and C2 . Next , we describe each phase of our proposed model distance in detail . Sampling representative reference data points . We randomly sample N reference images X = { xi } Ni=1 from the distribution task . Note that N is much smaller than the size of the training set of the classifiers being compared . For example , we select N = 128 reference images from 50 , 000 training images of CIFAR-10 . In § 4.3 , we show empirically that the choice of N does not qualitatively change the outcome of our model distance metric computations—unlike previous approaches described in § 2 that directly compared the predictions of two classifiers . Increasing N simply reduces the variance of our metric , which is intuitive because the comparison is made on more points sampled from the underlying data distribution ; so reference points are more likely to cover the same local regions of the classifiers ’ decision surface as N increases . Learning the local behaviours . While the classifiers being compared can be non-linear ( e.g. , deep neural networks in our case ) , the intuition behind LIME is that their behavior can be approximated locally by a linear model . We learn the local behaviour of each of the two classifiers being compared around each reference image by training a LIME linear regression model that captures the classifier responses to the image ’ s components ( i.e . contiguous patches of pixels denoted a super-pixel ) . Similar to the original LIME paper , we create and sample image components as follows . First , we group the pixels of a reference image in K super-pixels using Quickshift ( Vedaldi & Soatto , 2008 ) and create binary super-pixel masks B = { bk } Kk=1 , where each super-pixel binary mask bk ∈ { 0 , 1 } w×h specifies the location of pixels belong to the k-th super-pixel . These super-pixels Algorithm 1 : Proposed model distance approach , Zest Input : Black-box D1-class classifier C1 , Black-box D2-class classifier C2 , Reference image set X = { xi } Ni=1 , Quickshift segmentation algorithm , Random-Selector , Linear regression training algorithm T ( · , · ) and distance Distance ( · , · ) Output : Distance , d , between black-box classifiers C1 and C2 1 : S1 , S2 = { } , { } . Initialize model signatures 2 : for x ∈ X do 3 : B = { bk } Kk=1 = Quickshift ( x ) . Generate K super-pixel masks 4 : M = { ml } Ll=1 = Random-Selector ( B ) . Generate L image ’ s component masks 5 : X̂ = { x̂l : x̂l = x ·ml } Ll=1 . Generate L image ’ s components 6 : Z1 = C1 ( X̂ ) , Z2 = C2 ( X̂ ) . Query classifier 7 : W1 : T ( M , Z1 ) , W2 : T ( M , Z2 ) . Train local regression model 8 : if Unknown relation between D1 and D2 then 9 : for i ∈ D1 do 10 : W2 [ i ] = W2 [ argminj∈D2 ( ‖W1 [ i ] −W2 [ j ] ‖ ) ] . Class alignment 11 : S1.append ( W1 ) , S2.append ( W2 ) . Create model signature 12 : d = Distance ( S1 , S2 ) . Compute model distance each represent a contiguous patch of similar pixels , like a patch of fur or grass , whose presence or absence is controlled by the binary mask . A linear model on these super-pixels provides an interpretable approximation of the local decision boundary . For example , if the classifier predicts “ dog ” when the patch of fur is present , but changes its prediction when it is not , then we can interpret that the patch of fur greatly influences its prediction . Then , we create the reference image ’ s component x̂ = x ·m by randomly choosing several superpixels identified by 1s in component mask m ∈ { 0 , 1 } w×h that is combination of their corresponding binary mask . In order to explore the local area around each reference image , we repeat this process and draw L different components X̂ = { x̂l } Ll=1 using L different component mask M = { ml } Ll=1 . Following the original LIME paper , we set L = 1000 . We query classifiers to obtain their predictions on these L image components as Z1 = C1 ( X̂ ) and Z2 = C2 ( X̂ ) . Then , we train a linear regression model I : M→ Z per classifier to have high local fidelity with respect to the classifier around the reference image x . We call the weights of the trained regression model the local signature of its corresponding classifier . Approximating a global perspective . In total , we learn N linear regression models , capturing the behaviour of the classifier locally around the N reference images . We approximate the global behaviour of each classifier by concatenating its N local signatures S1 = { Wn1 } Nn=1 . Note that local signatures of each classifier have as many rows as the number of classes of the classifier . In order to compare signatures of classifiers with different number of classes when the relationship between classes output by the two classifiers is unknown , we introduce class alignment to re-position and reshape the rows of local signatures . Let small and big local signature be the local signatures of the classifiers with the lower and higher number of classes respectively . In class alignment , we start by the first row of the small local signature and compute its distance with each row in the big local signature . Then , we move the row corresponding to the minimum distance to the first row of of big local signature . We repeat this for all other rows in the small local signature while removing aligned rows of the big local signature from this process . Finally , we remove any remaining row of the big local signature not selected previously . See Appendix B for further details . Computing the distance . As the global signature of classifiers are the same size , we compute the model distance as d = Distance ( S1 , S2 ) . In Appendix G , we show that the choice of Distance ( · , · ) does not affect the performance of Zest . Our experimental results in the remainder of the paper are thus presented with the Cosine distance being used to compare global signatures .

Summary: The paper proposes an architecture-independent way of measuring distance between two models for the same type of input data. More specifically, given each reference sample, it first calculates a linear weight based on a set of masked samples and their corresponding outputs for each model, then calculates the distance between the set of weights for all reference samples generated from the two models. The method can be applied to help detect model stealing and inform unlearning verification.