Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an architecture-independent way of measuring distance between two models for the same type of input data. More specifically, given each reference sample, it first calculates a linear weight based on a set of masked samples and their corresponding outputs for each model, then calculates the distance between the set of weights for all reference samples generated from the two models. The method can be applied to help detect model stealing and inform unlearning verification.
main_review: Strengths.

The general method of the paper is presented relatively clear, and a series of experiments have shown the advantage of the presented method. 

Weaknesses.

Based on the introduction, it seems that existing literature on measuring model distances only includes Li et al. 2021 and Jia et al. 2021. The authors directly point out the limit of these two papers and propose a new method. It is hard to evaluate the challenges of this research area without more context. 

The paper mentions that Zest can also be applied to other domains like text and audio. Given that this paper focuses more on the empirical side, it would be good to include more diverse experiments and add applications in text/audio domains into the experiment section of the paper. 

Figure 1 is not clear enough on its own, especially Phase 3 & 4, and the caption does not contain enough information/description to help the audience understand directly. Also, it would be better if the LIME part can be more abstract since LIME algorithm is not this paper's contribution (similarly for Algorithm 1). 

Regarding class alignment step in Algorithm 1, does any of the experiments use class alignment? In general, when the two classifiers have unrelated labels, how could this alignment step work? More explanations on this may be required. 
summary_of_the_review: The paper is an empirical study on a topic that is not very popular (at least based on how the paper discusses about the existing literature).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a method (Zest) for measuring the distance or similarity between 2 supervised machine learning models while only requiring black-box access to the model's inputs and outputs but while also providing various improvements over the method of simply comparing model outputs themselves on a finite set of data points. These improvements are achieved by fitting locally linear models in the region of various reference data points, following the well-known LIME algorithm which was originally introduced for the purpose of model explanations. Experiments using DNNs for image recognition on CIFAR benchmark datasets are presented. The method is demonstrated on 2 specific applications. The first is the detection of model stealing, where the input/output examples of a model are used to reproduce it and therefore steal the associated intellectual property.  The second is the evaluation of unlearning, where a model needs to "forget" specific data and the Zest method is able to distinguish between 2 different unlearning methods in terms of their success in unlearning properly.
main_review: The method is generally sensible and the experimental results look solid. The idea of using a technique which is somewhere between working in weight space and working strictly with input/output examples does make sense. Zest can distinguish between models which are snapshots of the same training process at different epochs as opposed to models trained on the same data but with initial starting points, which I think is fairly impressive, given than this distinction might be expected to be fairly subtle. Zest succeeds at detecting model extraction (aka stealing) where a competing method (ModelDiff) fails. The ability to distinguish between true unlearning and approximate unlearning is also encouraging. The paper is mostly well written, although there are some typos and some places where I would suggest edits (see below).

My main objection is that I would have liked to see Zest demonstrated on a domain other than image recognition. Testing on another domain would give me more confidence in the results, particularly given that Zest is model agnostic and should therefore be able to demonstrate success on a wide variety of domains and model classes. I also think the model stealing problem has relevance for smaller-scale tabular ML models used in social science/finance areas like credit screening, crime recidivism forecasting, etc and it would have been great to see something like that here, tested presumably on some sort of tree ensemble like gradient boosting. 

I also think the applications (model similarity for model stealing, unlearning), while certainly cutting-edge, are only of moderate importance to ML research right now. 

On the balance, I think the paper should probably be accepted but the limited range of experiments and moderate topic significance prevent me from giving it a very high grade.

*** Update after author rebuttal *** The authors have added additional experiments in text and audio and have offered additional information regarding the significance of the application domains ( model stealing and unlearning). For these reasons, I am increasing my score to an 8.

Some suggested edits:

Page 2: 
2nd bullet point.  Also, on CIFAR-100, -< add comma
3rd bullet again need comma after CIFAR-100,
An 0% -> a 0%
Throughout our discussion, however -> throughout our discussion. However,  (start a new sentence here)
Past work on measuring model similarity have -> Past work on measuring model similarity has

Page 3 
weights of a DNN captures-> weights of a DNN capture

Page 5
Are in the same size-> are the same size

Page 7
This is again over-performed by Zest -> this problem is solved by Zest.

Page 8

"To further our evaluation, we also compare our proposed method Zest using the model reuse bench mark introduced with ModelDiff (Li et al., 2021). To ensure a fair comparison, we use the Mod elReuse benchmark provided by ModelDiff:" I think this sentence is basically being repeated, unnecessarily?

Table 4 shows that Zest and ModelDiff have similar performance…I think this is Table 2, not Table 4?
summary_of_the_review: The Zest method presented here is sensible, clearly presented and shown to be successful for a couple of applications (model stealing and unlearning evaluation). It would have been stronger if it had been tested additionally on another problem domain and another ML model class rather than focusing solely on DNNs and image recognition. The topics are also only of moderate significance. For these reasons, I recommend a weak accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a novel approach for computing an architecture-independent distance metric to assess the similarity between ML models by comparing the global behaviour of the models as approximated through LIME. They demonstrate the method by applying it on instances of the CIFAR dataset, focussing on the two tasks of model stealing and machine unlearning.
main_review: The paper is interesting and well written, although quite technical in scope.
All claims seem correct and quite convincing, and definitely useful (especially in the case of model stealing).
However, the validation of the proposed framework is limited to a single family of classifiers (ResNet) and a quite standard (and somewhat simple) benchmark dataset: such choice leaves room for wondering about the effectiveness of Zest on a broader landscape.
In particular, a deeper analysis of the Zest behaviour on separating related from unrelated classifiers (in different settings) would be recommended, for instance toward the definition of a (clearly empirical) quantitative distance threshold to be used as a reference for the Zest values.

summary_of_the_review: Paper is interesting, adding a significant novel contribution to the existing literature and providing an effective solution in at least two important and somehow still open ML issues. A wider analysis (both in terms of models and data) would be recommended to strengthen the authors' claims.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new method to calculate the distance between two machine learning models. By comparing using local approximations (LIME) and compute the Cosine distance, this yields an architecture-independent method operating in weight space. It has proves effectiveness in applications including detecting model stealing and verifying machine unlearning. 
main_review: This paper utilizes a model explanation methods called LIME to design a new method to calculating the distance between two machine learning models. Overall, the problem is well motivated by stating the limitations of current approaches, and the proposed method is clearly presented with experiments to support its benefits:

1. The authors clearly states the limitations of existing approaches, namely architecture dependent or dependence on the choice of model inputs. The proposed method interpolates between two extremes and thus alleviate both drawbacks. 

2. The authors provide a thorough review  related work, and introduce the proposed method clearly. 

3. The experiments are carefully designed to demonstrate the benefits of the proposed method. 

In general, I have the following several concerns: 

1. Algorithm 1 line 9 - 10, or in particular the way to deal with class alignment. I feel its a bit unclear: looks like we care finding the minimum alignment from all possible alignments. But if it actually due to class mismatch, is it possible that we get a small distance while the models are actually different? For example, model 1 predicts class A while model 2 predict class B using similar weights.

2. Algorithm 1 line 12, the distance is chosen to be a cosine operation, so why it will be larger than 1 in some of the experiments? 

3. Section 4.2, any reason to consider k = 5? It might be helpful to include additional experiment results varying the values of k.

4. For the applications mentioned, I cannot help my self thinking of knowledge distillation, which seems to be closely related to model stealing. Is there any related work discussing this or even further, can ZEST find application for knowledge distillations? 
summary_of_the_review: Overall I think this is a good paper and can be accepted to ICLR. Although it does not have much theoretical novelty, I believe applying LIME in the area of calculating distance between two models is pretty new. If my listed concerns can be addressed, I willing to increase my ratings one level further.
review_title: null

Meta-review: This paper presents a method, called Zest, to measure the similarity between two supervised machine learning models based on their model explanations computed by the LIME feature attribution method.  The technical novelty and significant are high, and results are strong.  Reviewers had clarifying questions regarding experiments and suggestions to add experiments, which involve additional domains (text and audio) and different families of classifiers, and more contexts based on prior literatures. These were adequately addressed by the authors. Overall, this paper deserves borderline acceptance.
Decision: Accept (Poster)