ExpProof : Operationalizing Explanations for Confidential Models with ZKPs
Authors: Chhavi Yadav
Evan Laufer
Dan Boneh
Kamalika Chaudhuri
Venue: ICML 2025 poster (2025)
Keywords: Explanations
Zero-Knowledge Proofs
Primary area: social_aspects->security
TL;DR: Making explanations operational with Zero-Knowledge Proofs when models are kept confidential.

Abstract: In principle, explanations are intended as a way to increase trust in machine learning models and are often obligated by regulations. However, many circumstances where these are demanded are adversarial in nature, meaning the involved parties have misaligned interests and are incentivized to manipulate explanations for their purpose. As a result, explainability methods fail to be operational in such settings despite the demand. In this paper, we take a step towards operationalizing explanations in adversarial scenarios with Zero-Knowledge Proofs (ZKPs), a cryptographic primitive. Specifically we explore ZKP-amenable versions of the popular explainability algorithm LIME and evaluate their performance on Neural Networks and Random Forests. Our code is publicly available at : \url{https://github.com/emlaufer/ExpProof}.

ExpProof : Operationalizing Explanations for Confidential Models with ZKPs
Chhavi Yaday Evan Monroe Laufer" 2 Dan Boneh
Kamalika Chaudhuri
Abstract
Setup & Commitment (happens once by model developer)
In principle, explanations are intended as a way to increase trust machine learning models and are often obligated by regulations. However, many circumstances where these demanded are ad- versarial in nature, meaning the involved parties have misaligned interests and are incentivized to manipulate explanations for their purpose. As a result, explainability methods fail to be opera- tional in such settings despite the demand (Bordt et al., 2022). In this paper, we take step towards operationalizing explanations in adversarial narios with serooKKowledge Proofs (ZKPs), cryptographic primitive. Specifically we explore ZKP-amenabe versions of the popular explain- ability algorithm LIME and evaluate their perfor- mance on Neural Networks and Random Forests. Our code publicly available at https: github. com/ emlaufer. Senffe//Expp (Exxxxxoo
commit
query
Explain Prove (for every query)
inference
verty
explanation
proof
Figure 1. Pictorial Representation of ExpProof.
and are incentivized manipulate explanations meet their ends. For instance, bank which denies loan to an applicant based on an MI. model's prediction has an incen- tive to return an incontestable explanation to the applicant rather than reveal the true workings of the model since the explanation can used by the applicant to prove discrimi- nation in the court of law (Bordt et al., 2022). In fact, many previous studies show that adversarial manipulations of ex- planations are possible in realistic settings with systematic and computationally feasible attacks (Slack et al., 2020; Shahin Shamsabadi et al., 2022; Slack et al., 2021; Yadav et al., 2024c). As such, despite the demand, explanations fail to be operational as trust-cnhancing tool.
1. Introduction
"Bottom line:
Post-hoc explanations are highly prob- lematic an adversarial context" (Bordt al., 2022)
Explanations have been seen as a way to enhance trust in machine learning (ML) models by virtue of making them debugging tool, they now also widely proposed to prove fairness and sensi- bility of ML-based predictions for societal applications, in research studies (Langer et al., 2021; Smuha, 2019; Kästner et al., 2021; Von Eschenbach, 2021; Leben, 2023; Karimi et al., 2022; Wachter et al., 2017: Liao & Varshney, 2021) and regulations alike (Right to Explanation (Wikipedia con- tributors, 2025)). However, discussed in detail by (Bordt al., 2022), many of these use-cases adversarial in ture where the involved parties have misaligned interests
transparent. Although starting
A major barrier to using explanations adversarial contexts is that organizations keep their models confidential due to IP and legal reasons. However, confidentiality aids in manipulating explanations by allowing model swapping a model owner can use different models for generating predictions vs. explanations, swap the model for specific inputs, or change the model post-audits (Slack et al., 2020; Yadav et al., 2024b; Yan & Zhang, 2022; Baldini al., 2023). This problem demands technical solution which guarantees that specific model is used for all inputs, for generating both the prediction and the explanation and prove this to the customer the receiving end while keeping the model weights confidential.
Equual contribution 'UC San Diego Stanford University. Cor- respondence to: Chhavi Yaday <chhaviyadav 123 gmail.com>> Evan Monroe Laufer <emlaufer cannlauffr@stanffor.dddd..
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Another important barrier to using explanations adver- sarial contexts that many explanation algorithms are not

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
deterministic and have many tunable parameters. An ad- versary can choose these parameters adversarially to make discriminatory predictions seem benign. Moreover, there is no guarantee that the model developer is following the explanation algorithm correctly to generate explanations. A plausible solution to counter this problem is consistency checks (Bhattacharjee & von Luxburg, 2024; Dasgupta et al., 2022). Apart from being a rather lopsided ask where the onus of proving correctness of explanations lies completely on the customer, these checks require collecting multiple explanation-prreiitioon pairs for different queries and are therefore infeasible for individual customers in the real world. Compounding the issue, it has been shown that auditing local explanations with consistency checks hard (Bhattacharjee & von Luxburg, 2024). Note that many of these issues persist even with perfectly faithful algorithm for generating explanations.
customers are often more interested in the behavior of the model around their input specifically. Additionally, LIME is model-agnostic, meaning that it can be used for any kind of model class (including non-linear ones).
Traditionally ZKPs are slow and are infamous for adding a huge computational overhead for proving even seemingly simple algorithmic steps. Moreover many local explanation algorithms such as LIME require solving an optimization problem and involve non-linear functions such as exponen- tials, which makes it infeasible to simply reimplement LIME as-is in a ZKP library. To remedy these issues, we experi- ment with different variants of LIME exploring the resulting tradeoffs between explanation fidelity and -oeerhedde To make our 7KP system efficient, we also utilize the fact that verification can be easier than re-running the computa- tion instead of solving the optimization problem within ZKP, we verify the optimal solution using duality gap.
We address the aforementioned challenges by proposing a system called ExpProof. ExpProof gives a protocol con- sisting of (1) cryptographic commitments which guarantee that the same model is used for all inputs and (2) Zero- Knowledge Proofs (ZKPs) which guarantee that the explana- tion was computed correctly using predefined explanation algorithm, both while maintaining model confidentiality. See Fig. 1 for pictorial representation of ExpProof.
Experiments. We evaluate ExpProofon fully connected ReLU Neural Networks and Random Forests for three stan- dard datasets on an Ubuntu Server with 64 CPUs of x86_64 architecture and 256 GB of memory without any explicit parallelization. Our results show that ExpProofis compu- tationally feasible, with a maximum proof generation time of 1.5 minutes, verification time of 0.12 seconds and proof size of 13KB for NNs and standard LIME.
EispProofensure uniformity of the model and explana- tion parameters through cryptographic commitments (Blum, 1983). Commitments publicly bind the model owner to fixed set of model weights and explanation parameters while keeping the model confidential. Commitments for ML models are a very popular and widely researched area in cryptography and hence we use standard procedures to do this (Kate et al., 2010).
2. Preliminaries
Cryptographic Primitives. We use two cryptographic primitives in this paper, commitment schemes and Zero- Knowledge Proofs.
Commitment Scheme (Blum, 1983) commits to private in- puts w outputting a commitment string com... A commit- ment scheme is hiding meaning that com does not reveal anything about private input w and binding meaning that there cannot exist another input w' which has the commit- ment com binding commitment com, to input w.
Furthermore, we wish to provide a way to the customer to verify that the explanation was indeed computed cor- rectly using the said explanation algorithm, without reveal- ing model weights. To do this, we employ cryptographic primitive called Succinct Zero-Knowledge Proofs (Gold- wasser et al., 1985; Goldreich et al., 1991). ZKPs allow a prover (bank) to prove a statement (explanation) about its private data (model weights) to the verifier (customer) without leaking the private data. The prover outputs a eryp- tographic proof and the verifier on the other end verifies the proof in a computationally feasible way. In our case, if the proof passes the verifier's check, it means that the explanation was correctly computed using the explanation algorithm and commited weights without any manipulation.
Zero Kero-KKowleege Proofs (7KPs) (Goldwasser et al., 1985; Goldreich et al., 1991) involve a prover holding a private input w. and a Verifier who both have access to a circuit P. ZKPs enable the prover to convince the verifier that, for some public input v, it holds a private witness w such that P(z, w) without revealing any additional infor- mation about witness to the verifier. A ZKP protocol is (1) complete, meaning that for any inputs (2,, w) where P(z, w) = 1, an honest prover will always be able to con- vince an honest verifier that P(x,, w) = 1 by correctly following the protocol, (2) sound, meaning that beyond negligible probability, malicious prover cannot convince an honest verifier for any input x, that for some witness w, P(x, w) 1 when infact such a witness w does not exist, even by arbitrarily deviating from the protocol, and
How are explanations computed? The explanation algo- rithm we use in our paper popular one called LIME
(Ribeiro et al., 2016), which returns a local explanation for the model decision boundary around an input point. We choose local explanation rather than global one since
2

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
(3) zero-knowledge, meaning that for any input x and wit- ness to such that P(x; w) 1, malicious verifier cannot learn any additional information about witness except that P(r, w) = 1 even when arbitrarily deviating from the pro- tocol. A classic result says that any predicate P in the class NP can be verified using ZKPs (Goldreich et al., 1991).
Algorithm 1 LIME (Ribeiro et al., 2016)
1: Input:
Input point z; Classifier f
Parameters: Number of points n to be sampled around input point, Length of explanation K, Bandwidth pa- rameter for similarity kernel
3: Output: Explanation
4:
LIME. Existing literature has put forward wide variety of post-hoc (post-training) explainability techniques to make ML models transparent. In this paper, we focus one of the popular ones, LIME (stands for Local Interpretable Model-Agnostic Explanations) (Ribeiro et al., 2016).
5: for i E (1, _ do
6:
24 + sample_around
n, exp (-(2(x, 21 22)2/0))
8: end for
9: ë E arg min Ei=1 H; x (f (2:) wT
22))²² 21)² + ||
LIME explains the prediction for an input point by approxi- mating the local decision boundary around that point with a linear model. Formally, given an input point I E X, a complex non-interpretable classifier f : X' + y and an interpretable class of models G, LIME explains the predic- tion f(x) E ) with a local interpretable model E g. The interpretable model is found from the class G via learning, on a set of points randomly sampled around the input point and weighed according to their distance to the input point, as measured with a similarity kernel æ. The similarity kernel creates a locality around the input by giv- ing higher weights to samples near input as compared to those far off. A natural and popular choice for the inter- pretable class of models G is linear models such that for any 9 E G. s(=)= Wy- = ((Ribeiro et al., 2016; Garreau & von Luxburg. 2020)), where wy are the coefficients of linear model g. These coefficients highlight the contribu- tion of each feature towards the prediction and therefore serve as the explanation in LIME. Learning the linear model is formulated as weighted LASSO problem, since the sparsity induced by & regularization leads to more inter- pretable and humaa-undersstnddalle explanations. Follow- ing (Ribeiro et al., 2016), the similarity kernel is set to be the exponential kernel with &g norm as the distance function, where o is the bandwidth parameter of the kernel and controls the locality around input x.
10: (op-K(u; K)
> Sorts the weights according to absolute value & returns these along with
corresponding features
II: Return Explanation e
while benefiting adversaries. Motivated by these problems, we investigate if technical solution can be designed to operationalize explanations in adversarial settings.
Formal Problem Setting. Formally, a model owner confi- dentially holds a classification model f which is not publicly released due legal and IP reasons. A customer supplies an input x to the model owner, who responds with a prediction (x) and an explanation E(f,z) where È is the possibly- randomized algorithm generating the explanation.
Threat Model. The model developer having access to the training data trains the model honestly. The parameters of LIME are public and are assumed to be set honestly. Model developer doesn't 't have information on the input queries it will receive. When presented with input query, the model generates both the prediction and explanation, which could be manipulated by either changing the model prediction arbitrarily, using a different model than the one which trained or using a different/corrupped algorithm to generate the explanation.
(=) exp( (-(2(x,2)
Solution Desiderata. A technical solution operationalize explanations in adversarial use-cases should provide the following guarantees.
For brevity, we will denote the coefficients corresponding to the linear model g as w instead of wy. unless otherwise noted. For readers familiar with LIME, without loss of generality we consider the transformation of the points into an interpretable feature space to be identity in this paper for simplicity of exposition. The complete LIME algorithm with linear explanations is given in Alg. 1. We will also use 'explanations' to mean post-hoc explanations throughout the rest of the paper.
1. (Model Uniformity) the same model f is used by the model owner for all inputs : our solution is to use cryptographic commitments which force the model owner commit a model prior to receiving inputs,
2. (Explanation Correctness) the explanation algorithm E is run correctly for generating explanations for all inputs our solution is to use Zero-Knowlede Proofs, wherein the model owner supplies a cryptographic proof of correctness to be verified by the customer a computationally feasible manner,
3. Problem Setting& & Desiderata for Solution
To recall, explanations fail trust-enhancing tool in ad- versarial use-cases and can lead to false sense of security
3

---

ExpProof:
Operationalizing Explanations for Confidential Models with ZKPs
Algorithm 2 STANDARD LIME. VARIANTS
which is known introduce a significant computational overhead. Given this, natural question that comes to mind is if there exist variants of LIME which provide similar qual- ity of explanations but are more -aeenaleeele by design, meaning they introduce a smaller ZKP overhead?
1: Input: Input point x, Classifier f
2: Parameters: Number of points n to be sampled around input point, Length of explanation K, Band- width parameter for similarity kerel,, sampling type smpl stpe, kernel type krn. type Output: Explanation e
Standard LIME Variants. To create variants of standard LIME (Alg.1), 1), we focus on the two steps which are carried out numerous times and hence create a computational bot- tleneck in the LIME algorithm sampling around input (Step 6 in Alg. 1) and computing distance using exponential kernel (Step 7 in Alg. 1). For sampling, we propose two options as found in the literature gaussian (G) and uniform (U) (Ribeiro et al., 2016; Garreau & Luxburg, 2020; Gar- reau & von Luxburg, 2020). For the kernel we propose to either use the exponential (E) kernel or no (N) kernel. These choices give rise four variants of LIME, mentioned in Alg. 2. We address each variant by the intials in the brackets, for instance standard LIME with uniform sampling and no kerne is addressed as 'LIME. U
3:
4:
5: for E (1,2,3....,, n do
6: if smpl type===uniform' uniform' then
7:
2, f uniformly hattimllyssmmle_aaaaaa _amnle_)round(()) 8: else if smpl. "gnuuatap=='auussinn then
9:
2j 4 gaussian sample. (r)
10: end if
11: if Arrnl_tpp==='xponeeni then
12: T4. f exp (2)))_2))2))///00
13: else
14: TT.
15: end if
16: end for
17: á E min Ei= n. x (z;) wTa. xz)) ==)²+ + +|wl|1
BorderLIME. An important consideration for generating meaningful local explanations is that the sampled neighbor- hood should contain points from different classes (Laugel et al., 2018b). Any reasonable neighborhood for input far off from the decision boundary will only contain samples from the same class, resulting in vacuous explanations.
18: e (op-K(w, K)
19: Return Explanation e
3. (Model Consistency) the same model f is used for inference and generating explanations: this is ensured by generating inference and explanations as a part of the same system and by using model commitments,
To remedy the problem. (Laugel et al., 201 (0:bbb) propose a radial search algorithm, which finds the closest point to the input æ belonging to a different class, Zborder. and then uses deorder as the input to LIME (instead of original input z). Their algorithm incrementally grows (or shrinks) search area radially from the input point and relies on random sampling within each 'ring' (or sphere), looking for points with an opposite label. To cryptographically prove this algorithm, we would either have to reimplement the algorithm as-is or would have to give a probabilistic security guarantee (using a concentration inequality), both of which would require many classifier calls and thereby many proofs of inference, becoming inefficient in ZKP system.
4. (Model Weights Confidentiality) the model weights of f are kept confidential in the sense that any technique for guaranteeing (1)-(3 does not leak anything else about the hidden model weights than is already leaked by predictions fxx and explanations & =f) without using the technique this comes as a by-product of using ZKPs and commitments (See Sec. A. for the formal theorem and proof),
5. (Technique Reliability) the technique used for guar- anteeing (1)-(4) is sound and complete (as in Sec.2): this comes as by-product of using ZKPs and com- mitments (See Sec. A.I for the formal theorem and proof).
We transform their algorithm into a line search version, called BorderLIME, given in Alg. 3 and 4, using the notion of Stability Radius which is now fed as a parameter to the algorithm. The stability radius for an input x. oz, is defined as the largest radius for which the model prediction remains unchanged within a ball of that radius around the input The stability radius 5 is defined as the minimum stability radius across all inputs x sampled from the data distribution D. Formally, 8 = infz-D 6gs where 6, = sup(r 2 0 flf(z²) = f(x), Vx' E ((x,r)). Here B(z,r) = {a S r) denotes a ball of radius r centered at x. Stability radius ensures that for any input from the data distribution, the model's prediction remains stable within at least radius of 6.
Our solution ExpProofwhich provides the above guarantees will be discussed in Sec. 5.
4. Variants of LIME
Building zero-knowledge proofs of explanations requires the explanation algorithm to be implemented in a ZKP library'
Moore precisely, arithmetic circuits for the explanation algo- rithm are implemented in the 7KP library.
4

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
Our algorithm samples m directions and then starting from the original input æ; takes ō steps until it finds point with a different label along all these directions individually. The border point Tborder is that oppositely labeled point which is closest to the input v. Furthermore, unlike in the algo- rithm in (Laugel et al., 2018a), our algorithm can exploit parallelization by searching along the different directions in parallel since these are independent.
Algorithm 4 FIND CLOSEST POINT WITH OPP LABEL
1: Input: Input point x, Classifier f
Parameters: Number of random directions m, Stability radius 6, Iteration Threshold T
3: Output: Opposite label point Zborder
4:
5: {ūo, di Si...uu, -.-ummi) := Sample m random directions
6: Initialize disto ... distmm1 as inf
7: for ih E &Too ūl ... 1m- 1 do
Determining the optimal value of the stability radius is interesting research question, but it is not the focus of this work. We leave an in-depth exploration of this topic to fu- ture work while providing some high-level directions and suggestions next. Stability radius can (and perhaps should) be found offline using techniques as proposed (Yadav et al., 2024a; Jordan et al., 2019) or through an offline em- pirical evaluation on in-distribution points. A ZK proof for this radius can be generated one-time, in an offline manner
#border, ==
iter := 0
10: while ((borderr. == f(x and iter S T do
11:
Tborder; := Thorder; + du,
12: iter := iter +
13: end while
14: if f(borrderr) = f(x) then
15: disl; == E2 (2 Tborder,)
16: end if'
and supplied by the model developer (for NNs see (Yadav et al., 2024a)). It can also be pre-committed to by the model developer (see Sec. 5).
17: end for
18: Wborder == bborder: such that i= arg min disl,
19: Return Tborder
Algorithm 3 BORDERLIME
Input:
Input point x, Classifier f
2: Parameters:
Number of points be sampled around input point, Length of explanation K, Bandwidth pa- rameter for similarity kernel
tomer) validates the proof without looking at the model weights. If the proof passes verification, means that the explanation is correctly computed with the committed model weights and explanation algorithm parameters.
3: Output: Explanation e
4:
5: Tborder ==
To generate the explanation proof, ZKP circuit which im- plements (a variant of) LIME is required. However since ZKPs can be computationally inefficient, instead of reim- plementing the algorithm as-is in a ZKP library, we devise some smart strategies for verification, based on the fact that verification can be easier than redoing the computation. Since all the variants of LIME share some common func- tionalities, we next describe how the verification strategies for these functionalities. For more details on the verification for each variant, see Appendix Sec. A.
FIND CLOSEST POINT WITH OPP LABEL,(;, f)
>> See Alg. 4
6: == LIME(Zborder: f) D> Note that any variant of
LIME can be used here.
7: Return Explanation e
5. ExpProof: Verification of Explanations
Our system for operationalizing explanations in adversarial settings, ExpProof. consists of two phases: (1) a One-time Commitment phase and (2) an Online verification phase which should be executed for every input.
1. Verifying Sampling (Alg. 7, 12, 13). We use the Posei- don (Grassi et al., 2021) hash function to generate random samples. As part of the setup phase, the prover commits to random value Tp. When submitting input for explanation, the verifier sends another random value Te. Prover generates uniformly sampled points using Poseidon with a key rp + re. which is uniformly random in the view of both the prover and the verifier. Then, during the proof generation phase, the prover proves that the sampled points are the correct out- puts from Poseidon using ezkl's inbuilt efficient Poseidon verification circuit. We convert the uniform samples into Gaussian samples using the inverse CDF, which is checked in the proof using a look-up table for the inverse CDF.
Commitment Phase. To ensure model uniformity, the model owner cryptographically commits to a fixed set of model weights W belonging to the original model f. result- ing in committed weights comw. Architecture of model is assumed be public. Additionally, model owner can also commit to the values of different parameters used in the explanation algorithm or these parameters can be public.
Online Verification Phase. This phase is executed every time a customer inputs a query. On receiving the query. the prover (bank) outputs a prediction, an explanation and a zero-knowledge proof of the explanation. Verifier (cus-
2. Verifying Exponential Kernel (Alg. 11). ZKP libraries do not support many non-linear functions such as exponential,
5

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
which is used for the similarity kernel in LIME (Step 5 of Alg.1). To resolve this problem, we implement a look- up table for the exponential function and prove that the exponential value is correct by comparing it with the value from the look-up table.
with a learning rate of 0.001 for 400 epochs. The weights and biases are converted fixed-point representation with four decimal places for making them compatible with ZKP libraries which do not work with floating points, leading to a ~ 1% test accuracy drop. Our random forests are trained us- ing Scikit-Learn (Pedregosa et al., 201 with 5-6 decision trees in each forest.
3. Verifying Inference. Since LIME requires predictions for the sampled points in order to learn the linear explanation, we must verify that the predictions are correct. To generate proofs for correct predictions, use ezkl's inbuilt inference verification circuit.
ZKP Configuration. We code ExpProof with different vari- ants of LIME in the ezkl library (Konduit, 2024) (Version 1.1 which uses Halo2 (Zeash Foundation, 2023) as its underlying proof system in the Rust programming language, resulting in ~ 3.7k lines of code. Our ZKP experiments are run on an Ubuntu server with 64 CPUs of x86_64 6664 ar- chitecture and 256 GB of memory, without any explicit parallelization -while ezkl automatically does multithread- ing all the available cores, we do not use GPUs, do not modify ezkl to do more parallelization and do not do any of the steps in ZK LIME (Alg. 6) in parallel by ourselves. We use default configuration for ezkl, except for 200k rows for all lookup arguments in ezkl and ExpProof. We use KZG (Kate et al., 2010) commitments for our scheme that are built into ezkl.
4. Verifying LASSO Solution (Alg. 9). ZKP libraries only accept integers and hence all floating points have to be quantized. Consequently, the LASSO solution for Step of Alg. I is also quantized in a ZKP library, leading to small scale differences between the exact and quantized solutions. To verify optimality of the quantized LASSO solution, we use the standard concept of duality gap. For a primal objective / and its dual objective g. to prove that the objective value from primal feasible w is close to that from the primal optimal w", that is ((w)- - l(w") S e, the duality gap should be smaller than c as well, I(w) - g(u, v) S where u, v are dual feasible. Since the primal and dual of LASSO have closed forms, as long we input any dual feasible values, we can verify that the quantized LASSO solution is close to the LASSO optimal. The prover provides the dual feasible as part of the witness to the proof. See App. Sec.A.2 for a note on the meaning of the duality gap threshold c, how it should be chosen, closed forms of the primal and dual functions and for the technique to find dual feasible.
Research Questions & Metries. We ask the following ques- tions for the different variants of LIME.
Q1) How faithful are the explanations generated by the LIME variant?
Q2) What is the time and memory overhead introduced by implementing the LIME variant in a ZKP library?
To answer Q1, we need a measure of fidelity of the explana- tion, we use 'Prediction Similarity' defined as the similarity of predictions between the explanation classifier and the original model in a local region around the input. We first sample points from a local region around the input point, then classify these according to both the explanation classi- fier and the original model and report the fraction of matches between the two kinds of predictions as prediction similarity. In our experiments, the local region is created by sampling 1000 points from a Uniform distribution of half-edge length 0.2 or a Gaussian distribution centered at the input point with a standard deviation of 0.2.
The complete ExpProof SxpPrroffpotooce protocol can be found in Alg. 5; its security guarantee is given as follows.
Theorem 5.1. (Informal) Given a model f and an input point x, ExpProof returns prediction [xx), LIME explana- tion x) and a ZK proof for the correct computation of the explanation, without leaking anything additional about the weights of model f (in the sense described in Sec.3).
For the complete formal security theorem and proof, refer to App. Sec. A.
6. Experiments
To answer Q2, we will look at the proof generation time taken by the prover to generate the ZK proof, the verification time taken by the verifier to verify the proof and the proof length which measures the size of the generated proof.
Datasets & Models. We use three standard fairness bench- marks for experimentation: Adult (Becker& & Kohavi, 1996), Credit (Yeh, 2016) and German Credit (Hofmann, 1994). Adult has 14 input features, Credit has 23 input features, and German has 20 input features. All the continuous features in the datasets are standardized. We train two kinds of models on these datasets, neural networks and random forests. Our neural networks are 2-layer fully connected ReLU activated networks with 16 hidden units in each layer, trained using Stochastic Gradient Descent in PyToch (Paszke et al., 2019)
6.1. Standard LIME Variants
In this section we compare the different variants of Standard LIME, given in Alg. 2 Sec. 4, w.r.t. the fidelity of their
Nooe that this local region is for evaluation and is different from the local neighborhood in LIME,
6

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
explanations and ZKP overhead.
6.2. BorderLIME
Setup. We use the LIME library for experimentation and run the different variants of LIME with number of neighboring samples n = 300 and length of explanation K = 5. Based on the sampling type, we either sample randomly from a hypercube with half-edge length as 0.2 or from a gaussian distribution centered around the imput point with a standard deviation of 0.2. Based on the kernel type we either do not use kernel or use the exponential kernel with bandwidth parameter as v # eatures * 0.75 (default value in the LIME library). Rest of the parameters also keep the default values of the LIME library. Our results are averaged over 50 different input points sampled randomly from the test set. The duality gap constant, is set to 0.001.
In this section we compare the variants of BorderLIME (Alg. 3, Sec. 4) and BorderLIME vs. Standard LIME w.n.t. the fidelity of their explanations and ZKP overheads.
Setup, We implement BorderLIME with all of the Standard LIME variants (Step 6 of Alg. 3). For the purpose of ex- perimentation, we fix the iteration threshold to T = 250, number of directions to m = Then to approximate the stability radius 6, we incrementally go over the set {0.01, 0.03, 0.05, 0.07, 0.1, 0.15) and use the smallest value for which an opposite class point is found for all 50 randomly sampled input points. While this is heuristic approach and does not guarantee the theoretically minimal stability radius, it provides a practical estimate of stability radius for efficient experimentation. Once suitable value of stability radius is identified, we tighten the number of directions by reducing them while ensuring that at least one opposite-class point exists for each input. Our results are averaged over 50 input points. The exact parameter values used in our final setup can be found in App. Sec. B.
Results for NNs with 300 neighboring samples and Gaussian sampling for fidelity evaluation are described below. Results for uniform sampling fidelity evaluation, fidelity evaluation with neighborhood n = 5000 points and all results for RFs can be found in the Appendix Sec. B.
Fidelity Results. As shown in Fig. 2 left, we do not find a huge difference between the explanation fidelities of the different variants of LIME as the error bars significantly overlap. This could be due to the small size of the local neighborhoods where the kernel or sampling doesn't 't mat- ter much. However, for the credit dataset, which has the highest number of input features, gaussian sampling works slightly better than uniform, which could be because of the worsening of uniform sampling with increasing dimension.
Results for NNs with 300 neighboring samples and Gaussian sampling for fidelity evaluation are described below. Results for uniform sampling fidelity evaluation, fidelity evaluation with neighborhood n = 5000 points and all results for RFs can be found in the Appendix Sec. B.
Fidelity Results. Comparing different variants of Border- LIME based on the LIME implementation, we observe that the difference in explanation fidelity between gaussian and uniform sampling becomes more pronounced compared to standard LIME as shown in Fig. 2, reinforcing the impor- tance of gaussian sampling. This gap can sometimes be reduced by using more neighborhood points, i.e. larger when uniformly sampling. As demonstrated in Fig. 2 mid, with three times more points for uniform sampling, we can match the fidelity of gaussian explanations for Adult and German datasets. Comparing the G+N version of Border- LIME and standard LIME in Fig. 2 right, we observe that explanations generated by Borderl are atleast as faith- ful as standard LIME and can sometimes be better hinting to its capability of generating more meaningful explanations.
ZKP Overhead Results. Across the board, proof generation takes maximum of ~ 1.5 minutes, verification time takes a maximum of ~~ 0.12 seconds and proof size is a maximum of ~ 13KB, as shown in Fig. 3. Note that while proof generation time is on the order of minutes, verification time is on the order of seconds - this is due to the inherent design of ZKPs, requiring much lesser resources the explanation
verifier's end (contrary to consistency-battt checks). We also observe that the dataset type does not have much influence on the 7KP overhead; this is due to same ZKP backend parameters needed across datasets.
Furthermore, we see that gaussian sampling leads to a larger 7KP overhead. This can be attributed to our implementa- tion of gaussian sampling in the ZKP library, wherein we first create uniform samples and then transform them to gaussian samples using the inverse CDF method, leading to an additional step in the gaussian sampling ZKP circuit as compared to that of uniform sampling. Similarly, using the exponential kernel leads to a larger overhead over not using it due to additional steps related to verifying the kernel.
ZKP Overhead Results. We observe that BorderL has a larger ZKP overhead than standard LIME as shown in Table 2: this can be attributed to the additional steps needed in BorderI to find the border point with opposite label (Alg. 4) which also have be proved and verified. Similar to the previous subsection, the overhead is similar across datasets and verification is orders of magnitude cheaper than proof generation.
Overall, 'gaussian sampling and kernel' variant of LIME is likely the most amenable for practical ZKP system as it produces faithful explanations with small overhead.
7

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
Gemma
Choa
Figure Results NNs. G/U: gaussian or uniform sampling, E/N: using or not using the exponential kernel. Left: Fidelity of different variants of Standard LIME, Mid: Fidelity of different variants of BorderLIME, Right: Fidelity of Standard vs. BorderLIME.
2.90
175
0.33
1.25
130
0.58
0.25
0.2
3441
CIVRR
16.
DE/9000 6.8
Figure 3. Results for NNs. G/U: gaussian or uniform sampling. E/N; using or not using the exponential kernel. Left: Proof Generation Time (in mins), Mid: Proof Size (in KBs), Right: Verification times (in secs) for different variants of Standard LIME, All configurations use the same number of Halo2 rows, 218, and lookup tables of size 200k.
6.3. A Alltion Study
(steps 9 & 30-32 Alg. 6) and (3) LASSO (step 33 Alg. 6). For these three, we provide the Proof-Generatic time in Table 1, since proof generation is the most time-consuming part of ZKP system and the trends for verification time and proof sizes usually follow its trend.
We conduct ablation study to investigate the computa- tional bottleneck of LIME when implemented in a ZKP library, 7K LIME Alg. 6, and understand how the computa- tional overhead ZK LIME scales with model size.
As demonstrated in Table 1, while the mroof-generation time for Kanpling<Kernel and LASSO is almost same across dif- ferent model sizes, the time for Inference proofs increases polynomially and dominates the total proof-generation time with increasing number hidden layers. Since we use infer- ence proof engines as glug-and-play module in 7K_ LIMM, the advancements inference proof generation (which is very active research area) will directly translate ZK LIME. Current SOTA inference proof generation time is 15 minutes for a 13B model (Sun et al., 2024).
Setup. We present experiments with the Credit dataset (w.lo.g.) since this dataset has the highest feature dimen- sionality and for NN models. For finding the computa- tional bottleneck of ZK LIME, we fix the number of hid- den units in each layer of the NN to 16 while changing the number of layers to {2, 10, 20, 20, 40). For studying the scaling, we change the number of units in each hidden layer {16, 32, 64) and the number of hidden layers 5, 10, 20). We use the G+E variant of LIME (as it is the most computationally expensive variant, from previous experiments) with sampling neighborhood of n 300. Results are displayed for one input point picked randomly as the variance ZKP cost almost zero, as seen from previous experiments.
ZKP-Overhead Scaling with Model Size. As demonstrated in Fig.4, the total proof-generation and verification times increase polynomially with increasing number of hidden lay- ers and units in each hidden layer. Feature Dimensionality has less pronounced effect demonstrated in Fig.3 where the datasets have different dimensions (as the internal ZKP parameters (rows, columns) change when the dimension exceeds threshold).
Computational Bottleneck of ZK LIME. To understand which part of LIME is the most expensive, we split it into three broad categories (1) Sampling and applying exponential kernel (steps 13 & 26 Alg. 6), (2) Inference
8

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
Figure 4.
(mins) and Verification Time (sees) with increasing #HiddenLayers and #Units/HiddenLLyer.
#Laaerrs
Sampling+ExpKenee
LASSO
Inferences
explanation trust guarantees.
2 23.54
59.62
42.15
10
24.11
60.34
108.23
8. Related Work
20 24.50
60.70
179.00
40 25.59
62.30
319.48
In the ML field ZKPs have been majorly used for verifica- tion of inferences made by models (Sun al., 2024; Chen et al., 2024; Kang et al., 2022: Weng al., 2023; Sun & Zhang, 2023; Feng et al., 2021: VI2, 2023; Lee et al., 2020; Zhang et al., 2020a; Liu et al., 2021; Singh et al., 2022; Fan et al., 2023). A line of work also focuses on proving the training of ML models using 7KPs (Lycklama et al., 2023; Huang et al., 2022; Rückel et al., 2022; Garg et al., 2023; Abbaszadeh et al., 2024). More recently they're also been used for verifying properties such as fairness (Yadav et al., 2024a; Shamsabadi et al., 2023; Toreini et al., 2023) and accuracy (Zhang al., 2020b) of confidential ML models. Alternatively, (Waiwitlikhit et al., 2024) propose ZKPs as general technique for auditing ML models broadly. Con- trary to these and to the best of our knowledge, ours the first work that identifies the need for proving explanations and provides ZKP based solutions for the same.
Table 1. Grnor-tennrttionn Time Split (in seconds) for ZK LIME. Inference time increases with #HiddenLayers while Sam- Bing+Keemet and LASSO times remain constant.
ZKP Overhead Type
BorderLIME
LIME
Proof Generation Time (mins)
4.85 + 10-2
1.17 + 10-2
Verification Time (secs)
0.30 + 10
0.11 10-2
Proof Size (KB)
18.30 + 0
10.40 + 0
Table ZKP Overhead of BorderLIME and Standard LIME (both
G+N variant) for NNs. Overhead for BorderL.ME is larger than that for LIME, Results are consistent across all datasets.
7. Discussion
While ExpProof guarantees model and parameter uniformity as well as correctness of explanations for a given model, it cannot prevent the kinds of manipulation where the model itself is corrupted - the model can be trained to create in- nocuous explanations while giving biased predictions. Here usually regularization term corresponding to the manip- ulated planatioons is added to the loss function (Aïvodji al., 2019; Yadav et al., 2024c). Preventing such attacks requires ZK proof of training; this is well-studied in the literature but is outside the scope of this work and we refer an interested reader to (Garg et al., 2023).
9. Conclusion & Future Work
In this paper we take a step towards operationalizing ex- planations in adversarial contexts where the involved par- ties have misaligned interests. We propose a protocol ExpProofusing Commitments and Zero-Knove Gero-Knowledge Proofs, which provides guarantees on the model used and correct- ness of explanations in the face of confidentiality require- ments. We propose ZKP-efficientt versions of the popular explainability algorithm LIME and demonstrate the feasibil- ity of ExpProoffor Neural Networks & Random Forests.
Furthermore, to provide end-to-end trust guarantees for fully secure explanations. the explanations should be (1) faithful, stable and reliable, (2) robust realistic adversarial attacks (such the one mentioned above) and (3) should also be verifiable under confidentiality. This paper looks at the third condition by giving a protocol ExpProofant and implementing it for verifiable explanations under confidentiality, which has not been studied prior to our work. As such, we view our work as complementary and necessary for end-to-end
An interesting avenue for future work is the tailored design of explainability algorithms for high ZKP-efficiency and inherent robustness adversarial manipulations. Another interesting avenue finding other applications in ML where ZKPs can ensure verifiable computation and provide trust guarantees without revealing sensitive information.

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
Impact Statement
Bhattacharjee, R. and von Luxburg, U. Auditing local expla- nations is hard. In Globerson, A., Mackey, I.., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, vol- ume 37, pp. 18593-18632. Curran Associates, Inc., 2024.
This work takes step towards operationalizing explanations in adversarial settings where the model is kept confidential from customers. With this work it can be guaranteed that the said committed model (1) is used for all inputs, (2) is used for generating predictions and explanations (3) cannot be swapped post-audits. It can also be guaranteed that the explanation is generated correctly using the said explanation algorithm. All of this is guaranteed while maintaining model confidentiality.
Blum, M. Coin flipping by telephone a protocol for solving impossible problems. ACM SIGACT News, 15(1):23-27, 1983.
Bordt, S., Finck, M., Raidl, E.,, and von Luxburg, U. Post- hoc explanations fail to achieve their purpose in adversar- ial contexts. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 891- 905, 2022.
While our protocol providde the above guarantees, in order to have complete trust guarantees we also require stable and faithful explanation algorithms which are robust to realistic adversarial attacks as mentioned in Sec.7. We mention existing solutions to tackle some of these issues in Sec.7 and Sec.4 and call for more research in these di- rections. These research directions though interesting and important are out of scope for our work.
Chen, B.-J., Waiwitikkhi, S., Stoica, I., and Kang, D. Zkml: An optimizing system for ml inference in pero-knowleder proofs. In Proceedings of the Nineteenth European Con- ference on Computer Systems, pp. 560-574, 2024.
Dasgupta, S., Frost, N., and Moshkovitz, M. Framework for evaluating faithfulness of local explanations. In Interna- tional Conference on Machine Learning, pp. 4794-4815, PMLR, 2022.
Acknowledgements
CY and KC would like to thank National Science Founda- tion NSF (CIF-2402817, CNS-1804822 SaTC-2241 100, CCF-2217058, ARO-MURI INF21111003 17), and ONR under N00014-24-1 1-2304 for research support. DB and EL were partially supported by NSF, DARPA, and the Simons Foundation. Opinions, findings, and conclusions or rec- ommendations expressed in this material are those of the authors and do not necessarily reflect the views of DARPA.
Fan, Y., Xu, B., Zhang, L.,, Song. J., Zomaya, A., and Li, K.-C. Validating the integrity of convolutional neural network predictions based on sero-knowlddg proof. In- formation Sciences, 625:125-144: 2023.
Feng. B., Qin, L., Zhang, Z., Ding, Y., and Chu, S. Zen: Efficient zero-knowledge proofs for neu- ral networks. IACR Cryptol. ePrint Arch., 2021:87, 2021. URL httph:t/////.. httt:::://ap.ssseeat.ccchhoe www.stcsscooooooo.o.oocc 2gg/corpusDD:221731855
References
Zator: Verified inference of a 512-layer neural network using recursive snarks. https:/ //g/t/utt com/ 2023.
Garg, S., Goel, A., Jha. S., Mahloujifar, S., Mahmoody, M., Policharla, G.-V., and Wang, M. Experimenting with zero-knowledge proofs of training. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 1880-1894, 2023.
Abbaszadeh, K., Pappas, C., Katz, J., and Papadopoulos, D. Zero-knowledge proofs of training for deep neural networks. Cryptology ePrint Archive, 2024.
Aïvodji, U., Arai, H., Fortineau, O., Gambs, S., Hara, S., and Tapp, A. Fairwashing: the risk of rationalization. In International Conference Machine Learning, pp- 161-170. PMLR, 2019.
Garreau, D. and Luxburg, U. Explaining the explainer: A first theoretical analysis of lime. In International con- ference artificial intelligence and statistics, pp. 1287- 1296. PMLR, 2020.
Garreau, D. and von Luxburg, U. Looking deeper into tabular lime, arXiv preprint arXiv:2008. 2020.
Baldini, I., Yadav, C., Das, P., and Varshney, K. Keeping up with the language models: Robustness- interplay in nli data and models. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
Goldreich, O., Micali, S., and Wigderson, A. Proofs that yield nothing but their validity or all languages in np have zero-knowledge proof systems. J. ACM, 38(3):690-728, jul 1991. ISSN 0004-5411. doi: 10. 1145/116825.11682 URL https: //ts::///o... http///oiiooo ooo...rg/0...... 1145/116825. 116222...18....
Becker, B. and Kohavi, R. Adult. chine Learning Repository, 1996.
UCI Ma-
DOI:
http://lo.inng/0.4442//XXXX
10

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
Goldwasser, S., Micali, S., and Rackoff, C. The knowledge complexity of interactive proof-systems. In Proceedings of the Seventeenth Annual ACM Symposium Theory of Computing, STOC 85, pp. 291-304, New York, NY, USA, 1985. Association for Computing Machinery. ISBN 089791 1512. doi: 10. 155224442214......7. URL https: //wpi. 1145/ 11t.///1111/22211..11
least squares. IEEE Journal of Selected Topics in Signal Processing, 1(4):606-617, 2007. doi: 10.1 10.1109JSSTPP. 2007.910971.
Konduit. ezkl: Efficient zero-knowledge machine learn- ing. https:/ /tppa://gittn ///g/tttthub.. wtt. comm///onddit///x.... 2024. Accessed: 2025-01-21.
Grassi, L., Khovratovich, D., Rechberger, C., Roy, A., and Schofnegger, M. Poseidon: A new hash function for Zero- Knowledge proof systems. In 30th USENIX Security Sym- posium (USENIX Security 21), pp. 519-535. USENIX Association, August 2021. ISBN 978-1-939133-24-3.
Langer, M., Oster, D., Speith, T., Hermanns, H., Kästner, L., Schmidt, E., Sesing, A., and Baum, K. What do we want from explainable artificial intelligence (xai)?-a stakeholder perspective on xai and conceptual model guiding interdisciplinary xai research. Artificial Intelli- gence, 296: 10647,,,, 2021.
URL BtRt:t/::///ww hwtp://wwwwww....oooooooooeoeeee wttp:oofeeeeece///////ooooo usenxxsecuritty//preeeettaaa grassi..
Laugel, T., Lesot, M.-J,, Marsala, C., Renard, X., and De- tyniecki, M. Comparison-basee inverse classification for interpretability in machine learning. In Medina, J., Ojeda-Aciego, M., Verdegay, J. L., Pelta, D. A., Cabrera, I. P., Bouchon-Meunier, B., and Yager, R. R. (eds.), In- formation Processing and Management of Uncertainty in Knowledge-Basse Systems. Theory and Foundations, pp. 100-111, Cham, 2018a. 1a. Springer International Publishing. ISBN 978-3-319-91473--
Hofmann, H. Statlog (German Credit Data). UCI
Machine Learning Repository, 1994. DOI:
https//bbb.orr///224333//CCCC
Huang, C.,, Wang, J., Chen, H., Si, S., Huang, Z.,, and Xiao, J. zkmlaas: verifiable scheme for machine learning service. In GLOBECOM 2022-2022 IEEE Global Com- munications Conference, pp. 5475-5480. IEEE, 2022.
Jordan, M., Lewis. J., and Dimakis, A. G. Provable certifi- cates for adversarial examples: Fitting ball in the union of polytopes. Advances in neural information processing systems, 32, 2019.
Laugel. T., Renard, X., Lesot, M.-.,, Marsala, C., and De- tyniecki, M. Defining locality for surrogates in post-hoc interpretablity. arXiv preprint 1rXiv 07498, 2018b.
Leben, D. Explainable ai as evidence of fair decisions.
Kang, D., Hashimoto, T., Stoica, I., and Sun, Y. Scaling up trustless dnn inference with zero-knowledge proofs, 2022.
Frontiers in Psychology, 14:1069426, 2023.
Lee, S., Ko, H., Kim, J., and Oh, H. venn: Verifi- able convolutional neural network. ePrint Arch., 2020:584, 2020. alapi.semant www.ciiccccccocccorooooooeeeeeee 218895602.
IACR Cryptol. URL https:
Karimi, A.-H., Barthe, G., Schölkopf, B., and Valera, I. A survey of algorithmic recourse: Contrastive explana- tions and consequential recommendations. ACM Com- put. Surv, 55(5), December 2022. ISSN 0360-0300. doi: 10.1 155/3527888 URL https: htt://ii..orggg// 1145/3527844
Liao, Q. V. and Varshney, K. R. Human-centered explain- able ai (xai): From algorithms to user experiences. arXiv preprint arXiv:2110. 10790, 2021.
Kästner, L., Langer, M., Lazar, V., Schomäcker, A., Speith, T., and Sterz, S. On the relation of trust and explainabil- ity: Why to engineer for trustworthiness. 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW), pp. 169-175. IREE, 2021.
Liu, T., Xie, X., and Zhang, Y. zkenn: Zero knowl- edge proofs for convolutional neural network predictions and accuracy. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, 2021. URL https: / //ppi. semanticscholas org Stss/CooppIID:233440000
Kate, A., Zaverucha, G. M., and Goldberg, I. Constant-size commitments to polynomials and their applications. In 2010: 16th Inter-
Lycklama, H. Burkhalter, L., Viand, A., Kuchler, N., and Hithnawi, A. RoFL: Robustness of Secure Federated Learning In 2023 IEEE Symposium on Security and Privacy (SP), pp. 453-476, Los Alamitos, CA, USA. May 2023, IEEE Computer Society. doi: 10. 10011099944211522..11111 https: wtt.:iiieeeeeccppppeeseeeeeeeeee 10.1109/ 222122211.2222211111200.
Advances in Cryptology-ASAACRA
national Conference on the Theory and Application of Cryptology and Information Security, Singapore, Decem- ber 5-9, 2010. Proceedings 16. pp. 177-194. Springer, 2010.
URL
Kim, S.-J., Koh, K., Lustig, M., Boyd, S., and Gorinevsky, D. An ontiit method for large-scale & -eggullard

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
Paszke, A., Gross, S., Massa, F,, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.,, Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Toreini, E., Mehrnezhad, M., and van Moorsel, A. Verifiable fairness: Privacy-preserving com- putation of faimess for machine learning systems. 2023. URL https: htp.pp..ss... www.nticschoonooooot...i.iici 261
org/Corpus :66 696588.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. Scikit-leam: Machine learning in python. the Journal of machine Learning research, 12:2825-28838 2011.
Von Eschenbach, W. Transparency and the black box prob- lem: Why we do not trust ai. Philosophy & Technology, 34(4):1607-122,, 2021.
Wachter, S., Mittelstadt, B., and Russell, C. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31:841, 2017.
Ribeiro, M. T., Singh, S., and Guestrin, C. "" "wy should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp- 1135-1144, 2016.
Waiwitlikhit, S., Stoica, I., Sun, Y., Hashimoto, T., and Kang, D. Trustless audits without revealing data or mod- els. In Proceedings of the 41st International Conference on Machine Learning, ICMI 24.. JMIR.org. 2024.
Rückel, T., Sedlmeir, J., and Hofmann, P. Fairness, in- tegrity, and privacy in a scalable blockchain- feder- ated learning system. Computer Networks, 202: 108621, 2022.
Weng, J., Weng, J., Tang, G., Yang. A., Li, M., and Liu, J.-N. Pvenn: Privacy-preserving and verifiable convolutional neural network testing. Trans. Info. For Sec., 18:2218 22333, mar 2023. ISSN 1556-6013, doi: 10.1109/TIFS 22110997TSS22023332229222 URL https: /doii. org/10, 1109 TIFSSIIFS... 2023. 3262932.
Shahin Shamsabadi, A., Yaghini, M., Dullerud, N., Wyllie, S., Aïvodji, U., Alaagib, A., Gambs, S., and Papernot, N. Washing the unwashable: On the (im) possibility of fairwashing detection. Advances in Neural Information Processing Systems, 35:14170-1411 2022.
Wikipedia contributos... Right to explanation, 2025. URL pettps://en.wikipeeet
org/ htt///wwwi//ggh/tt// to to_explanation. Accessed: 2025-01-14.
Shamsabadi, A. S., Wyllie, S. C., Franzese, N., Dullerud, N., Gambs, S., Papernot, N., Wang, X., and Weller, A. Confidential proof of fair training of trees. ICLR, 2023.
Yadav, C., Chowdhury, A. R., Boneh, D., and Chaudhuri, K. Fairproof: Confidential and certifiable faimeess for neural networks. In International Conference on Machine Learning, pp. 55682-55705. PMLR, 2024a.
Singh, N., Dayama, P., and Pandit, V. Zero knowledge proofs towards verifiable decentralized ai pipelines. In International Conference on Financial Cryptography and Data Security, pp. 248 275. Springer, 2022.
Yadav, C., Moshkovitz, M., and Chaudhuri, K. Xaudit: A learning- look auditing with explanations. Transactions on Machine Learning Research, 2024b.
Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. Fooling lime and shap: Adversarial attacks post hoc ex- planation methods. In Proceedings of the AAAI/ACM Con- ference on AI, Ethics, and Society, pp. 180-186, 2020.
Yadav, C., Wu, R. and Chaudhuri, K. Influence- based attributions can be manipulated. arXiv preprint urXhv:2409.05208. 2024c.
Slack, D.,, Hilgard, A., Lakkaraju, H., and Singh, S. Coun- terfactual explanations can be manipulated. Advances in neural information processing systems, 34:62-75, 2021.
Yan, T. and Zhang, C. Active faimess auditing. In Inter- national Conference on Machine Learning, pp. 24929- 24962. PMLR, 2022.
Smuha, N. A. The eu approach to ethics guidelines for trustworthy artificial intelligence. Computer Law Review International, 20(4):97-106, 2019.
Yeh, L.C.. default of credit card clients. UCI Machine Learning Repository, 2016. DOI: https///b...rr/10.2443//22
Sun, H. and Zhang, H. zkdl: Efficient zero-knowledge proofs of deep learning training. 2023.
Zcash Foundation. Halo2: A Plonkish zk-SNARK imple- mented in Rust, 2023. URL ht https: https://githeei /ttpt/gtttbbbb com/
acash/halod Accessed: 2025-01-27.
Sun, H., Li, J., and Zhang, H. zkllm:
Zero knowledge proofs for large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 4405-4419, 2024.
Zhang, J., Fang, Z., Zhang, Y., and Song, D. Zero knowl- edge proofs for decision tree predictions and accuracy. In Proceedings of the 2020 ACM SIGSAC Conference
12

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
on Computer and Communications Security, CCS '20, pp. 2039-2053, New York, NY, USA, 2020a. Associa- tion for Computing Machinery. ISBN 9781450370899, doi: 10. 1114//33772297.34177778 URL /tttps://oi.. org 443444433733411111111
Zhang, J., Fang, Z.,, Zhang, Y., and Song, D. Zero knowl- edge proofs for decision tree predictions and accuracy. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pp. 2039-2053, 2020b.
13

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
A. ExpProof
Algorithm 5 ExpProof: Provable Explanation for Confidential Models
1: Public Configuration: ZK_LIME configuration cc ( smpl_type sampling type, LIME kernel variant krnl sype,
whether to use border LIME border Jime, standard deviation a, model architecture f, LIME & penalty o, maximum dual gape)
2: Public Input: Input point x
Private Witness: Model weights W
4: Output: Output label o, Explanation Proof that f()) and that is valid LIME explanation.
5: Pre-Processing Offiin Phase
6: Sample randomness F
7: Commit to the randomness com, and release it publicly
8: Commit to the model weights comw and release it publicly
9: Online Phase
f(W,z)
11: Compute h, Poseidon(k;,))
12: Compute LIME perturbations from samples
13: 2)
14: (e, w) +LIME v,
Compute LIME solution using perturbations with labels
15: Compute a feasible dual solution è from ib
II <-zkLime (o,, x, o, rus Cr. Cw, e; W, w. h, 0,,0) 0)
return (o, e, II)
14

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
Algorithm 6 ZK LIME
1: Public Configuration:
smpplttpe sampling type, LIME kernel variant krnl_type, whether to use border LIME worder-lime standard deviation o, model architecture LIME penalty a, maximum dual gap c, sampling bit-width b
2: Public Instance: imput point model output randommess Te. commitment to the randomness Cy, commitment to the
weights Cw, top-k LIME features
3: Private Witness: Model weights W, labels of the LIME samples hash outputs h, LIME model ú, LIME dual
4: Output: ZK Proof the computation
5: Check that C, Com(r,)
6: Check that Cw Com (W)
& < Tp TT
8: ZK. CHECK SHHECK.POSEIDONT
Check that h is generated from Poseidon using key Check that f(W, x)) using EZKL
9: EZKL.CHECK INFERENCE(I, W, x,
uniform' then 10: if' smpl type=='uniform'
12: else if WWW_LUIFOOMSSMPPLLE.. (h,MLLE(h, b)
11: 8 =- =+-Z__UNIFOO UNIIOORMMM....
> Check that is uniform generated from the hashes
type==ggauusiinn then 8 7 GAUSSIAN SAMPLE((h,b) b)
Check that & Gaussian generated from the hashes
else
15: return 1
16: end if
17: if border lime == true then
18: x
wtt.OOPPPPOINTT,,, 8, nemooeclors, vector lenggh))
19: s s+smm x d...]
> Skip samples used for opposite point for fresh randomness
end if
21: for |z| do
22: jti mod |
23: 2 2 Zj + 3 2-
D Perturb with samples
24: end for
25: if krrd
then
26: π 4-ZK EXPONENTIAL (2,
27: else
π
29: end if
do 30: for i E WIIKLLLCEECK..PEEEECCCCCC w,
31:
Check that y, S(,, 2:) using EZKL
32: end for
33: ZK LASSO y, vb, az)
34: e =ZK_TOP_ K(
Generate proof of the above computation (see note below).
By 'Generate proof II the above computation', we mean that we eencode the above computation as Halo2 relation, and use Halo2 prove its correctness as a monolithic proof. Each sub-routine we call does not generate a separate proof, simply sub-computation in the Halo2 relation, and we only split up for organization purposes.
Algorithm WW__CHCCK_PSSEDDO
1: Input: hashes h, key As
2: Output: True if each h; generated from Poseidon with key l and input i
3: for f4 E do
4:
ki)
1> Check that h; Poseidon(k:, i) using EZKL
5: end for
6: return Eborder
15

---

return Fborder ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
Algorithm 8 -K_FIND_OPPPPPOL
Input: input æ, samples s, number of vectors num vectors, maximum length of each vector vector Length
2: Output: Border point Thorder if one exists, otherwise
3: d + |x]
4: step < =00..d x me..eehhpee d)
Get m. x d samples as randomly sampled vectors
5: for i [1, 3, num_veclors) do
6: step; t step, x LOOKUP RECIPROCAL (sRT(step; step;) Normalize each step vector using a lookup table for 1/ 1/v[[(step)]]
7: end for
8: for (1,2,3,... "1/2,,...,uummecclorrr do
9: for E {1,2, ..., vector length do xx step_size x step;
10:
11: end for
12: end for
13: y = x(W,w)
Iborder f- x for EvectorJength, vector Jength 1, ..,, 2, 13 do 16: for E 11,2, nemtoot do
17:
if y/ + x label then
18:
Tborder t U/
19: end if
20: end for
end for
Algorithm 9 L..SAS
1: Input: Samples 2, labels weights т,, Lasso solution ú. Lasso dual solution 0, Lasso parameter a, maximum dual gap
2: Output: True if the dual solution is feasible and the dual gap is less than e, and False otherwise.
3: for (1,,,3,...,,u) do
4:
T 24
5:
y/
V
TT y/i
6: end for
7: - L 1/ b - w 2/11 2 + al|w|]i
8: d - ||0 TT (g/
9: Check p-d Sc
10: Let m be the length each sample z;
for f1,2,3,.... ...., m)
do
12: f: (XT)) )²
13: Check < a
14: end for
Algorithm 10 ZK_TOP_K
1: Input: Lasso solution w. top-k values
2: Output: True if e contains the top-k values of úh and False otherwise
3: vi' -))
4: i 31,2,3,.... k} do
5: (v,j) &- E;
6: Check di
7: Check wi V
8: end for
16

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
Algorithm 11 EXPONENTIAL KERNEL
1: Input: Input point z, LIME samples 2, standard deviation
2: Output:
3: for E (I. do
4: square distance
square_distance
5:
EXONEEIIX (wwp::/IIAAA(ssuuuurrdssstaeee/
D Check exponential function using a lookup table
6: end for
7: return
Algorithm 12 ZK UNIFORM SAMPLE
1: Input: Poseidon hashes h, sampling bit-width b
2: Output: Uniform samples 2
3: B W/6b
4: N = [([=] * n]
5: 0
6: for 31.2,3..... N do
7: Compute such that 2j + 2 +... + Bj+B2BBb + rem = h2
8: Check that 2j + 2 +.... + 22+ + rem h4
D Check that the samples are a decomposition h4
9: for k E (1, 2,,, B) do
Check that 0 S z4* < 2
11: end for
12: Check that 0 S rem < 2B
13: j+- + B
14:
end for
15:
return z
Algorithm 13 S.A.GAAUSSIANSSMMPLLE
Input: Poseidon hashes h, sampling bit-width! b
2: Output: Gaussian samples
3: =ZK UNIFORM SAMPLE(1 (h, 6)
4: z = LOOKUP GAUSSIAN IN INVERSE_CDF(2)
5: return z
17

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
A.1. Security of ExpProof
Completeness
LIME configurations oc, input points æ, and model weights W
pp ExpPrroofSSetup(//
(pk, ok) <- ExpProof/KeyGen(p))
Pr
(comw, com, + ExpPrrof/Commit(pp w, (o, (o,e, TT) t PropPPooo.PPooeeicee pk, cc, x, comw, W, com,; To) ComPProff.Verityyi (pp, uk, CC, æ; o, comw, com, TT
= 1.
Proof Sketch. Completeness. The completeness proof mostly follows from the completeness of the underlying proof system (in our case, Halo2). We must also show that for any set of parameters there exists LIME solution w and feasible dual solution e such that the dual gap between i and is less than e. We know from the strong duality of Lasso that there exists a solution ws and u* such that the dual gap is 0 for any input points and labels, therefore such a solution exists. However, we also note that the circuit operates on fixed-point, discrete values (not real numbers), and it is not necessarily true that there are valid solutions fixed-point. To solve this, the prover can use larger number of fractional bits until the approximation precise enough.
Knowledge-Soundnes
We define the relation Rrime as:
comw Com W)
com, Com(pp)
f cc.[(W, xx) i)
h, Poscidon(r,,
< SAMPLE CC.O)
f cc./(W, 2))
π - LIME KERNEL( y = cc. f(W .2) 2' V x
Riime
(co, x. 0, e, Tv, coww, W, y, 4, u.
1/ V x
p 1 1ly - 6 - 6² 2'112 11 + cc.a||//l11
d k = ||0| + 6 (1/ -V/-b)
p d $ CC.E
+ (XT) )
-c.0 < f S cc.0
There exists an extractor & such that for all probabilistic polynomial time provers P*
pp + ExpProo//Seuup(1))
(pk, vk) ExpProof
(cc, x. O, e, Fu, comw, com,, mt) P(1 pkk
Pr
ExpProof Verify vk, cc, I, o, r. comw: com,) =1 negl(A).
(W, Tpp y, h, w, i) <- (.....
(cc, (cc,2,0,e,re e, cwmw. com,; W. Tps B, h, á. i) & Rism
Proof Sketch. Sketch. underlying proof system Halo2. The extractor runs the Halo2 extractor and outputs the Halo2 witness. By the construction of the circuit ZK LLMM,, the extracted Halo2 witness satisfies the Rimm relation.
Knowledge Soundness. Showledge-soundnes follows directly from the knowledge-suundnes. of the
Zero-Knowledge We say protocol IT is zero-knolwedge if there exists polynomial time, randomized simulator S such that for all (pk, vk) = Setup(pp), for all (z, w) E R, for all verifiers V
(P(pk, x,,)) w)) Rs (S(pk, x)}
18

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
Proof Sketch. Zero-Knowledge. Let the simulator S be the Halo2 simulator. For any (cc, R. 0, e, Te, comw; com,, Wi',,, ú) Rismer we know that
(Experoof.Prove(cc, x, ros W, Tp y, h () (,,e,,
by zero-knowledge of Halo2.
A.2. LASSO Primal and Dual
Notation: Let X Raxm denote the data inputs, y E R denote the labels and denote the regularization parameter or the LASSO constant. Let R denote the primal variable and R denote the dual variable.
The primal LASSO objective is given as, I(w) 2n |X v al|w||a while the dual objective function is given as g(t) 2 ||ull o with the feasibility constraint 0 s To (x v) a (Kim et al., 2007).
From a LASSO primal feasible w, it is possible to compute a dual feasible v as (Kim et al., 2007):
2sxXwr - y) #
min {a/1 (WTWws) - 2/4i = 1,..,,..,
We find that this dual close enough to the dual optimal to get good duality gap, however, the worst case is possible apply traditional optimization methods to find dual feasible with smaller duality gap.
Note the duality gap threshold, The duality gap condition is stopping condition commonly used in optimization libraries. Therefore from an algorithmic perspective, is a 'parameter' fixed by the user based on how much enror the user can tolerate. In our case the 'user' is actually the verifier/customer as the LASSO solution is the explanation for the verifier. As such, this threshold should be public (as assumed in ExpProoff) otherwise the prover can set to arbitrary values. This value should ideally be provided by the verifier or set by regulators based on how much error is tolerable (independently of the model weights or the dataset).
Additionally, note that the duality gap condition ((w) - g(v) S implies that the difference between primal optimal and primal feasible values is bounded by well, that is ((w) I(w*) ) I e. This in turn implies that the difference between the
true explanation (w*) and approximate explanation (w) is bounded as follows: ||w w" 1l2 O Amiin__ (X) where X is the
/ue
samples from the neighborhood and should be full rank and Aming (X) is the smallest non-zero positive singular value of X.
19

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
B. Experimental Details
B.1. NN results
Next in Fig.5 we show results for using uniform sampling in the 'prediction similarity' evaluation, keeping rest of the parameters same. We observe similar numbers before with very slight differences from Fig.2.
Comoce
Figure Results n 300 neighhoring points uniform sampling evaluation. Left: Fidelity different variants Standard LIME, Mid: Fidelity of different variants of BOrderLIME, Right: Fidelity of Standard BorderLIME.
Next we increase the neighborhood size, n, in LIME from 300 to 5000 samples and present the results Fig.6. As expected the fidelity increases across the board due to better model fitting with a larger number points. The size of the error bars only reduces significantly for the German dataset, showing that for more input points the explanations are faithful the original decision boundaries. Additionally, the German dataset also has the highest fidelity explanations. Both these points hints towards smoother or more well-behaved decision boundaries learnt using the German dataset. Furthermore, BorderL.MMB consistently outperforms standard LIME pointing to better explanations.
Figure 6. Results for NNs for 5000 neighhoring points and gaussian sampling in the evaluation. Left: Fidelity of different variants of Standard LIME, Mid: Fidelity different variants of BorderLIME. Right: Fidelity of Standard vs. BorderLIME.
20

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
B.2. RF results
Next F and Fig.8 we show results for fidelity of explanations for Random Forests using gaussian and uniform sampling in the "prediction similarity' evaluation respectively, keeping rest of the parameters same. We observe results as for NNs.
Jen
Drewn
Figure 7. Results for RFs for n = 300 neighboring points and gaussian sampling in the evaluation. Left: Fidelity of different variants of Standard LIME, Mid: Fidelity different variants of BorderL.IME. Right: Fidelity Standard vs. BorderL.MM..
OFF
G69 0ede
Cedo
Gemei
Ohift
Figure 8. Results for RFs for n 300 neighboring points and uniform sampling in the evaluation. Left: Fidelity of different variants of Standard LIME, Mid: Fidelity different variants of BorderLIME Right: Fidelity of Standard vs. BonderLIME.
Next we show the ZKP overheads for RFs in Fig.9 and Table 3. Trends and observations are similar as for NNs.
3.
0.200
23
23
6.12
23
1.2
60/20
ACUR
Ordi
CArPeR
463k
Cred1
Gummo
Cenman
Figure 9, Results for RFs for n = 300 neighboring points. Left: Proof Generation Time (in mins), Mid: Proof Size (in KBs), Right: Verification times (in secs) different variants Standard LIME, All configurations the same number Halo2 rows, 26 and lookup tables of size 200k.
21

---

ExpProof: Operationalizing Explanations for Confidential Models with ZKPs
7KPP Overhead Type
BorderL.IME
LIME
Proof Generation Time (mins)
8.46 1 10
2.02 1 10
Verification Time (secs)
0.44 + 10-2
0.14 I 10-3
Proof Size (KB)
17.25 +
16.20 10
Table ZKP Overhead of LorE and Standard LIME (both G+N variant) for RFs for 300 neighboring points. Overhead for BorderLIME is larger than that for LIME. Results are consistent across all datasets.
Next we increase the neighborhood size, in LIME from 300 to 5000 samples and present the results in Fig. 10. As expected the fidelity increases across the board due to better model fitting with a larger number of points. We see slight reduction in error bars for German dataset. BorderLIME equals or outperforms standard LIME.
Adr
Gemen
Ohdit
Figure 10. Results for RFs for neighboring points n = 5000 and gaussian sampling in the evaluation. Left: Fidelity of different variants of Standard LIME, Mid: Fidelity of different variants of BorderLIME, Right: Fidelity of Standard vs. BorderLIME.
22


Reviews:
[{"review_id": "gSZPNeZbTr", "reviewer": "Reviewer_96NK", "summary": "The paper introduces ExpProof, a system that produces model predictions, explanations for the prediction and proof that the explanations are correct without revealing the model's weights. The key idea is to use cryptographic commitments and zero-knowledge proofs to ensure that a model owner cannot cheat when providing an explanation (for example, to provide a crafted explanation for the prediction). The model owner commits to a fixed model and commits to the parameters of the explanation algorithm. Then, for each query input, the model owner returns the prediction, a LIME-based explanation, and a ZKP, indicating that this explanation truly comes from the committed model.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful comments and the time taken by the reviewer to review our paper. We are glad that the reviewer thinks our paper introduces a new paradigm for explanations, acknowledges that we are the first to integrate ZKPs with an explanation algorithm and finds our paper an interesting contribution to the XAI and applied cryptography communities.\n\nNext we address your concerns and questions.\n\n- **SHAP** : From our understanding, KernelSHAP is very similar to LIME (the major difference is the kernel). We believe KernelSHAP can be implemented using our zk_LIME implementation with reasonably easy modifications. The kernel for KernelSHAP is simpler than that for LIME, in the sense that it does not include exponentials, just arithmetic operations.\n\n- **Threat Model** : The model developer having access to the training data trains the model honestly. The parameters as mentioned in L661-665 are public and are assumed to be set honestly. It doesn’t have information about the input queries it will see. When presented with an input query, it generates both the prediction and explanation, which could be manipulated by either changing the model prediction arbitrarily, using a different model than the one it trained or using a different algorithm to generate the explanation. We will highlight this threat model in the paper.\n\n- **Factors limiting Scalability of ExpProof** : The main bottleneck is proving inferences for the sampled points (in the neighborhood of input). The total time taken for proving inferences increases as more points are sampled around the input point and as the models get more complex. However,  *since expproof treats the inference proof library as a plug-and-play module, improvements in inference proof times (which is a very active area of research) will directly translate in our tool*. We did some experiments to study the bottleneck, can be found here : https://anonymous.4open.science/r/expproof_experiments_rebuttal-6C75/experiments_rebuttal_expproof_icml.pdf.\n\n- **Scaling wrt dimensionality** : Dimensionality doesn’t affect the time as much, as is also evident in Fig. 3 in our paper where the ZKP overhead is the same across datasets of different dimensions. This is because dimensionality plays the most role in LASSO and sampling checks and the proving time for these is highly overshadowed by inference proof times.\n\n- **High dimension sampling** : From our knowledge, latin hypercube sampling (LHS) is seen to do better in high dimensions, this is also implemented in the python LIME library. *One way of implementing LHS in a ZKP library would be through a random shuffling module. Since ExpProof is very modular, this can be integrated into our tool once implemented*.\n\n\nPlease let us know if you have any remaining concerns. We look forward to hearing from you!\n\nAuthors", "claims_and_evidence": "Yes, the claims made in the submission are clearly supported by evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods do make sense for the problem at hand.", "theoretical_claims": "Does not apply.", "experimental_designs_or_analyses": "Yes, the soundness and validity of the experimental designs are sound.", "relation_to_broader_scientific_literature": "The paper positions itself well in the literature and discusses all relevant work that I am aware of.", "essential_references_not_discussed": "Does not apply.", "other_strengths_and_weaknesses": "Strengths:\n* The paper introduces a new paradigm for explanations. It’s the first to integrate Zero-Knowledge Proofs with an explanation algorithm to prove the explanation is correct and uses the intended model. This is an interesting contribution to the explainable AI community and to applied cryptography.\n* The experiments cover multiple variants of LIME, two model classes, and three datasets, giving confidence that the system works in different scenarios. \n\nWeaknesses:\n* The paper does not tackle situations where the model is adversarially trained to produce misleading explanations. If a malicious model owner trains a model to have plausible explanations for some outputs, the presented technique cannot help. ExpProof only works assuming the model was honestly trained and the behaviour and explanation presented are internally consistent.\n* While the overhead is reasonable for small models, it may become a bottleneck for larger models. The experiments used a 2-layer neural network and a small random forest. It’s not evaluated on deeper networks or datasets with a bigger number of features.", "other_comments_or_suggestions": "Does not apply.", "questions_for_authors": "1. Could you elaborate on the exact adversarial threat model you assume? For instance, you discuss an adversarial model owner who might manipulate explanations. However, it does not cover the case where the model owner is adversarial and is manipulating the model during training. This could be better presented in the paper as well.\n2. You focus on LIME due to its relative simplicity for ZKPs. Have you considered other standard explainability techniques like SHAP?\n3. What are the main factors that limit the scalability of ExpProof to larger neural networks or more complex architectures?\n4. LIME can sometimes struggle with meaningful local sampling in high-dimensional feature spaces. How does that interaction play out in ExpProof, and does the proof cost scale with dimensionality?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wJBnVF5ra9", "reviewer": "Reviewer_Zm91", "summary": "The paper proposes a protocol with a zero-knowledge proof to ensure that a provided explanation is correct while maintaining confidentiality of the model parameters. In particular, the paper focuses on LIME and standard ZKP libraries. This involves modifying the pipeline so that the protocol is computationally efficient. Finally, the paper provides experiments on the runtime and explanation fidelity of the method.\n\n### update after rebuttal\nI am maintaining the current score following the rebuttal. I am still not convince on the assumption regarding the model architecture being public which limited the practicality of the proposed method.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful comments and the time taken by the reviewer to review our paper.\n\nBelow we address your concerns.\n\n- **Commitment phase** : As mentioned in our paper, commitments are a standard procedure in cryptography and it is well known how to implement this in practice and preexists in ZKP libraries. Models with a few billion parameters can be committed to in under a minute.  In our implementation (following EZKL), we use KZG as a vector commitment to commit to the weights. The model owner uses cryptographic commitments (such as KZG,  Pedersen, SHA256 hashing) to commit to the exact model weights, which binds the model owner to the weights while not revealing them to the verifier.\n\n- **Architecture being public** : Regarding architecture details being public, currently this is the state of research in the ZKP community and is a standard assumption; we hope future research in ZKPs for ML can address this limitation. Alternatively, in certain domains standard or published SOTA architectures might be used commonly and hence this may not be extremely sensitive information, so revealing it in order to get verifiable explanations may be a good tradeoff.\n\n- **Evaluation in adversarial contexts** : There is perhaps a slight misunderstanding here. By adversarial, we mean when the model might be swapped by the model developer at any given point or the explanations might be crafted instead of being the real ones from the model (as mentioned in L38right-56left). These problems are eliminated by ZKPs + commitments because of their theoretical guarantees, it is not an empirical question.\n\nPlease let us know if you have any remaining concerns. We look forward to hearing from you!\n\nAuthors", "claims_and_evidence": "1. On the commitment phase: The paper states 'the model owner commits to a fixed set of model weights W belonging to the original model f.' It's not clear how this is implemented in practice. The approach assumes the model architecture is public, which may not be realistic as architecture details could leak sensitive information. Additionally, how large is this fixed set of model weights? While a larger set would enhance privacy, this raises questions about the practical management of such information.\n\n2. While the introduction focuses on how 'post-hoc explanations are highly problematic in an adversarial context' and presents ExpProof as a solution, the experiments don't include testing ExpProof in these adversarial scenarios. This creates a gap between the paper's motivation and its empirical validation.", "methods_and_evaluation_criteria": "The paper evaluates the approach using running time and the fidelity of LIME. The runtime assessment is appropriate for a computational protocol. However, the fidelity of LIME appears to function more as an ablation study rather than the primary result. It might be helpful to consider adding a metric that shows how well this approach could protect against adversarial explanations, possibly by measuring the percentage of manipulations it can detect or prevent.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "The contribution provides an approach to solve the problem of adversarial post-hoc explanations which is significant. However, some assumptions required could be too strong (see Claims and Evidence).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mwMXiGixyR", "reviewer": "Reviewer_ormk", "summary": "This paper proposes a solution for operationalizing explanations in adversarial contexts where the involved parties have misaligned interests. The authors focus on LIME and propose a method called ExpProof, which integrates ZKPs to ensure that explanations remain trustworthy while maintaining model confidentiality. The authors explore different sampling strategies (Gaussian vs. Uniform) and kernel choices (Exponential vs. None) to balance explanation fidelity and computational efficiency in a ZKP setting. Experiments on three datasets demonstrate the feasibility of ExpProof for both Neural Networks and Random Forests.\n\n## update after rebuttal\n\nI have decided to maintain my current (positive) score because the authors have addressed all my concerns. After reading the other reviewers' comments, I agree with some of them and believe this work does have certain limitations, which may somewhat impact its value. So I am keeping a weak accept (3) rather than upgrading to an accept (4).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful comments and the time taken by the reviewer to review our paper. We are very glad that the reviewer finds our paper well-written, easy to follow, accessible and could learn a lot from it – this is very rewarding!\n\nNext we address your concerns and questions.\n\nMajor:\n\n- **SHAP** : From our understanding, KernelSHAP is very similar to LIME (the major difference is the kernel). We believe KernelSHAP can be implemented using our zk_LIME implementation with reasonably easy modifications. The kernel for KernelSHAP is simpler than that for LIME, in the sense that it does not include exponentials, just arithmetic operations.\n\n- **LIME for modern networks** : We agree with your concern that LIME may not be the best explanation for large scale models of today. Having agreed on this, we want to highlight that the focus of our work is societal applications (such as finance, health, justice) where the “right to explanation” is applicable and misalignment of interests occurs organically. In many of these use-cases the data is tabular and the most popular choice for tabular data is still small neural networks and random forests, where LIME is one of the popular XAI tools. Additionally, *as the first work in using ZKPs for explanations and giving a unique solution to a long-standing problem, we hope future work can look at modern models and their suitable explanations.*\n\nMinor:\n\n- **Why G+N** : G+N variant performs equally well as G+E mainly because the gaussian distribution captures the behavior of the exponential kernel in a way, by sampling more around the mean than far off. To answer your next question about why G+N results were compared instead of G+E – this is because G+N and G+E are almost equally faithful but using G+N would save on the ZKP costs (due to absence of exponential kernel). So any practical ZKP system would rather choose G+N over G+E.\n\n- **BorderLIME**: The goal of BorderLIME is to provide non-vacuous explanations when the input point is far from the decision boundary; as such the benefits will show only in cases when the input points are far from the decision boundary. In cases where most of the test points are close to the decision boundaries, borderlime isn’t designed to give additional benefits while in those where the points are far off from the decision boundaries, borderlime should give significant improvement in fidelity.\n\nPlease let us know if you have any remaining concerns. We look forward to hearing from you!\n\nAuthors\n****************\n[1] Individual Fairness Guarantees for Neural Networks. Benussi et.al. 2022\n\n[2] Well-tuned Simple Nets Excel on Tabular Datasets. Kadra et.al. 2021\n\n[3] Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data. Rabuchev et. al. 2024", "claims_and_evidence": "Yes, I think the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem at hand.", "theoretical_claims": "No, I didn't check them.", "experimental_designs_or_analyses": "Overall, the experimental design and analysis seem sound. The only aspect I am uncertain about is whether the neural networks used in the experiments are too simple (a two-layer fully connected network), considering that modern neural networks typically have a much larger number of parameters and more complex architectures. If the neural network were more complex, it is unclear whether LIME would still provide reliable explanations, potentially affecting the significance of the proposed approach.", "relation_to_broader_scientific_literature": "According to the authors’ claims, this is the first work to identify the need for proving explanations and to propose ZKP-based solutions for this purpose.", "essential_references_not_discussed": "I am not familiar with this domain, so I am unsure whether any relevant papers may have been overlooked in the citations.", "other_strengths_and_weaknesses": "The paper is well-written and easy to follow. The clear presentation of ideas made it accessible, even for someone less familiar with the domain. I learned a lot from reading it, and I appreciate the authors’ efforts in presenting their work so clearly.", "other_comments_or_suggestions": "The paper focuses solely on adapting ZKP to work with LIME, without exploring its compatibility with other explanation methods such as SHAP. It remains unclear whether integrating ZKP with SHAP would require a significant redesign or if the proposed approach naturally extends to other explainability techniques. Given that this is the first work applying ZKP to explanations, it is acceptable for the paper to focus only on LIME. However, In my view, this limitation somewhat affects the overall contribution.", "questions_for_authors": "- From my understanding of LIME, the original version (G+E) should generally perform better than any of the other variants. However, in Figure 2, it seems that N generally performs better than E. Why does this happen?\n- Isn’t the default version of LIME (G+E)? Why does Figure 2 (right) compare standard LIME and BorderLIME using G+N instead?\n- BorderLIME does not seem to provide a significant advantage over standard LIME, as it only outperforms LIME on the German dataset. Given its substantial inefficiency -- its proof generation and verification times are both 3x longer than those of LIME, and its proof size is 1.8x larger -- it raises the question of whether BorderLIME is practically necessary. Would standard LIME be a more efficient and sufficient choice in most cases?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P0qfjZOaZC", "reviewer": "Reviewer_JYN6", "summary": "The paper proposes to compute model explanations, in particular LIME, using Zero Knowledge Proofs (ZKP). This allows consumers (users) of a service to receive verifiable explanations for their predictions, without the service having to reveal their model and thus preserving their IP. The paper theoretically constructs the ZKP protocol and experimentally evaluates their feasibility.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful comments and the time taken by the reviewer to review our paper. We are glad that the reviewer finds our paper well-written, easy to follow, thinks the proposed problem in our paper is interesting and novel and finds our cryptographic solution theoretically sound with many benefits.\n\nNext we address your concerns and questions.\n\n\n- **Related works** : Thanks for pointing this out. While we did cite the closely related work on adversarial attacks for explanations in the introduction and other sections of the paper, it is a good idea to dedicatedly talk about these and other related papers on explainability and adversarial attacks in the related work section. We will make this change in the paper.\n\n- **Evaluation** : The test points were *randomly* sampled from the respective test sets of the datasets (mentioned in L321right). The ZKP results do not change by increasing the sample size as the variance is extremely small.\n\n- **$\\epsilon$ value** : We use 0.001. We will add this value to the paper.\n\n- **Parallelization** : The ZKP library we use, ezkl, automatically does multithreading on all the available cores. By “explicit” we mean that other than ezkl’s multithreading, we do not use gpus, do not modify ezkl to do more parallelization and do not do any of the steps in zk_LIME in parallel by ourselves. We will clarify this in the paper.\n\n- **Scalability** : The focus of our work is societal applications (such as finance, health, justice) where the “right to explanation” is applicable and misalignment of interests occurs organically. In many of these use-cases the data is tabular and the most popular choice for tabular data is still small neural networks and random forests. *These models are enough to achieve SOTA accuracy* [1,2,3]. In such domains we do see the usefulness of our experiments done on these class of models and popular tabular datasets. Additionally, *as the first work in using ZKPs for explanations and giving a unique solution to a  long-standing problem, we hope future work can address the scalability issue.*  \n\n    Having said this, we conduct some experiments/ablations which can be found at https://anonymous.4open.science/r/expproof_experiments_rebuttal-6C75/experiments_rebuttal_expproof_icml.pdf. We find that the maximum time in zk_LIME is taken for proving inferences and the time for this dominates others as the model size grows. Since expproof treats the inference proof library as a plug-and-play module, improvements in inference proofs (which is a very active area of research) will directly translate in our tool.  \n\n    Dimensionality doesn’t affect the time as much, as is also evident in Fig. 3 in our paper where the ZKP overhead is the same across datasets of different dimensions. This is because dimensionality plays the most role in LASSO and sampling checks and the proving time for these is highly overshadowed by inference proof times.\n\n\n\nPlease let us know if you have any remaining concerns. We look forward to hearing from you!\n\nAuthors\n***********\n\n[1] Individual Fairness Guarantees for Neural Networks. Benussi et.al. 2022\n\n[2] Well-tuned Simple Nets Excel on Tabular Datasets. Kadra et.al. 2021\n\n[3] Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data. Rabuchev et. al. 2024", "claims_and_evidence": "Yes, the claims are supported both theoretically, as well as empirically. In particular, the authors show that the method guarantees (1) model uniformity, (2) explanation correctness, (3) model consistency, (4) model confidentiality, and (5) technique reliability. The experiments demonstrate that the method is feasible for very small models (Random Forests, NN with 2 layers and 16 hidden units) and datasets.", "methods_and_evaluation_criteria": "The method makes sense theoretically, and solves the proposed problem. However, as common with ZKPs, the computational complexity and the amount of data communicated is significant, and therefore only practical for very small models. The evaluation demonstrates this on three simple datasets. It would be interesting to also show how the computation scales with larger models and more complex data.\nThe evaluation is limited to 50 samples from the test sets, and it is unclear how these are selected. It would be good to expand this evaluation to at least 100, ideally more data-points and ensure random, uniform sampling.", "theoretical_claims": "Yes, I checked the correctness of the proof in 5 and app A1. It looks correct to me. however, I am not a cryptographer and therefore lack the expertise to detect subtle issues with SKPs.\n\nOne issue I would like to see discussed more: the authors provide a ZKP proof that verifies the solution to the LIME optimization problem, instead of the computation of the solution. This is a neat trick to safe computational resources. However, there are implications that need to be verified. In particular, there may be multiple valid solutions to the optimization problem, especially when we allow an $\\epsilon$-gap to the optimum. This could be exploited by the service to provide a different solution than the one computed by the LIME algorithm. The authors should\n(1) prove and verify that all such solutions are valid explanations\n(2) adjust the phrasing of the guarantees to indicate that the ZKP does not check for correct LIME solutions, but only verify those.", "experimental_designs_or_analyses": "The experiments compare the prediction similarity between the explanation and the original model, and the computational time to arrive at those explanations using ZKPs. The experiments make sense, although it would have be helpful to investigate how the method scales to larger models and datasets.", "relation_to_broader_scientific_literature": "The paper is very poorly positioned in the broader scientific literature. The related works section is overly brief, and only mentions some references from the ZKP literature. However, this paper combines several different areas: (1) ZKPs, (2) explainability methods, (3) adversarial attacks. (2) and (3) are missing entirely.", "essential_references_not_discussed": "See above", "other_strengths_and_weaknesses": "**Strengths**\n\n- Sections 1-5 of the paper are very well written and easy to follow. I especially liked the consistent example in the introduction.\n- The proposed problem is interesting and novel, and looks at explanation in a different context.\n- ExpProof is a theoretically sound solution, with many benefits. It preserves the IP of the service provider, gives cryptographic guarantees, and does not require a trusted third party for verification.\n\n**Weaknesses**\n\n- Due to the substantial computational overhead, the practical applicability is limited.\n- The paper is poorly positioned in the related scientific literature (see above)\n- No code is provided for review", "other_comments_or_suggestions": "- Algorithm 5 is the main contribution of the work; it should imho be part of the main paper and not the appendix. If space is a concern, I suggest to defer Algorithms 2-5 to the appendix instead.\n- The hyper-parameter $\\epsilon$ is crucial, but I did not find what value is used for experiments.\n- The claim that “explanations […] are often obligated by regulations” needs better support. A reference to a Wikipedia article explaining the term is not sufficient. Either cite corresponding legislation, or remove the claim.\n- There should be a space inserted after all instances of the *ExpProof* name.", "questions_for_authors": "1. What is the value of hyper-parameter $\\epsilon$?\n2. How are the 50 samples from the dataset selected?\n3. What do you mean by “no explicit parallelization”? Does this mean there is “implicit” parallelization? What does this mean exactly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uTZ4Gcmh5X", "reviewer": "Reviewer_H1H2", "summary": "The paper introduces a novel cryptographic framework aimed at providing verifiable explanations for machine learning models while ensuring model confidentiality. The central idea is to leverage ZKPs and cryptographic commitments to guarantee the correctness of explanations in adversarial settings, where parties may have misaligned interests and could manipulate the explanations to their advantage. The authors propose a solution called ExpProof, which uses cryptographic commitments to bind a model to a fixed set of weights and explanation parameters, and ZKPs to prove that the explanations are computed correctly using the predefined explanation algorithm, all without revealing sensitive model details. The explanation algorithm is extended from LIME to overcome the significant computational overhead imposed by ZKPs. The paper includes a comprehensive set of experiments on NN and RF using standard datasets.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful comments and the time taken by the reviewer to review our paper. We are glad that the reviewer enjoyed reading the first two sections of the paper clearly presenting the motivation and overall narrative.\n\nMajor:\n\n- **Missing References**: Thank you for sending these interesting papers our way, we weren’t aware of them. We’ll cite these in our related work section. As future work, it would be interesting to see if some of the systems level insights from these papers can lead to improvements in our ZKP overheads.\n\n- **Contribution**: As the ZKP & ML communities are very disjoint, *the key contribution of our paper is to identify a long-standing problem in explainable machine learning which the community wanted to solve, but couldn’t solve with traditional methods [1,2], and then solve it with ZKPs* – without our solution explanations cannot be used in adversarial contexts. We reveal the utility of ZKPs in operationalizing explanations to the *explainability community, which is mostly unaware of the existence of this primitive or has not made the connection we make*. As we see, **our contribution is appreciated by members of XAI community: quoting reviewer 96NK “The paper introduces a new paradigm for explanations” & “This is an interesting contribution to the explainable AI community”; quoting Reviewer JYN6 “The proposed problem is interesting” & “ExpProof is a theoretically sound solution, with many benefits”**.  \n\n    *We don't claim to make original contributions to the ZKP technology, the main contribution here is an ML contribution*. But using existing technology, together with some smart tricks, *we provide a realistic (not-toy, written in ZKP libraries that are used in practice) working implementation of zk_LIME as a start point to the explainability community*.\n\n- **Scalability** : The focus of our work is societal applications (such as finance, health, justice) where the “right to explanation” is applicable and misalignment of interests occurs organically. In many of these use-cases the data is tabular & the most popular choice for tabular data is still small neural networks and random forests. *These models are enough to achieve SOTA accuracy* [3, 4, 5]. In such domains we do see the usefulness of our experiments done on these class of models and popular tabular datasets. Additionally, as the first work in using ZKPs for explanations and giving a unique solution to a  long-standing problem, we hope future work can address the scalability issue.  \n\n    Having said this, we conduct some experiments/ablations which can be found at https://anonymous.4open.science/r/expproof_experiments_rebuttal-6C75/experiments_rebuttal_expproof_icml.pdf. We find that the maximum time in zk_LIME is taken for proving inferences and the time for this dominates others as the model size grows. *Since expproof treats the inference proof library as a plug-and-play module, improvements in inference proofs (which is a very active area of research) will directly translate in our tool*.\n\n\nMinor:\n\n- **Architecture** : Thanks for catching this. We do mention in L222-223-right-side that architecture is public, but we will make necessary amends as suggested by you in the paper to emphasize this.\n\n- **Proof generation** : We do not use proof aggregation or proof folding, it is a monolithic proof. When we say \"Generate proof Π of the above computation\", we mean that we encode the above computation as a Halo2 relation, and use Halo2 to prove its correctness as a monolithic proof. Each sub-routine we call does not generate a separate proof, it is simply a sub-computation in the Halo2 relation, and we only split it up for organization purposes. We will highlight this in the paper.\n\n- **$\\epsilon$ in duality gap** : The duality gap condition is a stopping condition commonly used in optimization libraries (since it implies f(x)-f* <= $\\epsilon$ where f is the primal). Therefore from an algorithmic pov, $\\epsilon$ is a ‘parameter’ fixed by the user based on how much error it can tolerate. In our case the ‘user’ is actually the verifier/customer as the lasso solution is the explanation for the verifier. As such, this threshold should be public (as is assumed in ExpProof) otherwise the prover can set it to anything. This value should ideally be provided by the verifier or set by regulators based on how much error is tolerable (independently of the model weights or the dataset). We use $\\epsilon$ = 0.001. We will highlight this in the paper.\n\nPlease let us know if you have any remaining concerns!\n\nAuthors\n\n[1] Auditing local explanations is hard. Bhattacharjee et. al\n\n[2] Post-hoc explanations fail to achieve their purpose in adversarial contexts. Bordt et.al.\n\n[3] Individual Fairness Guarantees for Neural Networks. Benussi et.al. \n\n[4] Well-tuned Simple Nets Excel on Tabular Datasets. Kadra et.al.\n\n[5] Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data. Rabuchev et. al.", "claims_and_evidence": "1. In \"Solution Desiderata,\" I believe Model Uniformity and Model Consistency should be merged as one desiderata, as it is essentially achieved by the same cryptographic commitment and their objectives are quite similar. For \"Model Confidentiality,\" current claim is misleading. The claim \"the model $f$ is kept confidential\" somewhat implies that the architecture of the model is also private, which is not the case in Alg 8.\n2. The overall ZK_LIME looks very vague. First, what does it mean by \"Generate proof Π of the above computation\"? Do you mean something like proof aggregation or Nova's proof folding scheme that packs all the sub-proofs in the algorithm? Is Π a single folded proof or a bunch of proofs in this algorithm?\n3. Alg 9 is also vague to me. In particular, how is the input parameter maximum dual gap $\\epsilon$ determined? If I understand correctly, $\\epsilon$ is data-dependent and would releasing $\\epsilon$ as a public input undermine the confidentiality? And, the proof to the system's zero-knowledge property does not hold if $\\epsilon$ is public and data/task-dependent.", "methods_and_evaluation_criteria": "1. One of the biggest concerns is scalability. LIME is expected to explain moderately complex models that are difficult to analyze or interpret. However, the model used in the evaluation is too small, with only two layers, making it hard to justify its real-world usefulness. While ZKP introduces significant computational overhead, please at least consider using a model comparable to those studied in the ZKML paper.\n2. Conceptually, the core contribution is unclear. My impression is that the paper resembles a technical report rather than a principled research contribution, as the proposed technique is largely an operational combination of existing solutions, particularly ezkl. For example, this pipeline could also be achieved through a combination of “Trustless Audits without Revealing Data or Models” (ICML 2024) and “ZKML: An Optimizing System for ML Inference in Zero-Knowledge Proofs” (EuroSys 2024), both of which provide ZK ML inference and training features. I encourage the authors to clearly articulate the conceptual novelty beyond a mere operational description of the process.", "theoretical_claims": "Please see my comments in \"Claims And Evidence\" section.", "experimental_designs_or_analyses": "Please see my comments in \"Methods And Evaluation Criteria\" section.", "relation_to_broader_scientific_literature": "The key contribution is the adaptation and encoding of the LIME algorithm into a zero-knowledge circuit. The core process involves (1) ZK ML inference, (2) ZK point sampling, and (3) ZK ML training. However, given that both (1) and (3) have already been addressed by existing solutions, and (2) is a standard cryptographic trick, the overall contribution to the broader community appears weak.", "essential_references_not_discussed": "Verifiable evaluations of machine learning models using zkSNARKs (preprint) -> the paper that describes ezkl, the framework on which it extensively relies.\n\nTrustless Audits without Revealing Data or Models (ICML 2024)", "other_strengths_and_weaknesses": "I enjoyed reading the first two sections of the paper, which clearly present the motivation and overall narrative. However, the presentation could be improved by visualizing Algorithms 5 and 6 as a schematic diagram to help readers from the ML community better understand what is happening behind the scenes.\n\nI also encourage the authors to strengthen (or articulate, if I missed something) the conceptual novelty and empirical usefulness of the proposed framework.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I believe the paper would benefit from a major revision to address my concerns listed above, and the required workload may go beyond what is feasible during the rebuttal phase. Therefore, I do not have any explicit questions, though I am open to any clarifications.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces ExpProof, a methodology to provide explanations that are certified. That is, we know that the explanations are correctly computed without looking at the weights of the model. This is achieved using Zero-Knowledge Proofs (ZKPs), a cryptographic primitive. 

The reviewers see merit in the paper, in particular, the novelty of the method and the efforts to make it understandable to an audience not familiar with ZKPs. Their main concern, namely the scalability of the method was addressed in the rebuttal. Another concern was the limited scope of the paper, which is currently strongly focused on LIME. This focus seems reasonable for a proof-of-concept paper, and I am recommending the paper to be accepted.
Decision: Accept (poster)