Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention
Authors: Stephan Rabanser
Ali Shahin Shamsabadi
Olive Franzese
Xiao Wang
Adrian Weller
Nicolas Papernot
Venue: ICML 2025 poster (2025)
Keywords: uncertainty
abstention
rejection
selective prediction
zero knowledge
confidence
auditing
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We show that a model owner can artificially introduce uncertainty into their model and provide a corresponding detection mechanism.

Abstract: Cautious predictions—where a machine learning model abstains when uncertain—are crucial for limiting harmful errors in safety-critical applications. In this work, we identify a novel threat: a dishonest institution can exploit these mechanisms to discriminate or unjustly deny services under the guise of uncertainty. We demonstrate the practicality of this threat by introducing an uncertainty-inducing attack called Mirage, which deliberately reduces confidence in targeted input regions, thereby covertly disadvantaging specific individuals. At the same time, Mirage maintains high predictive performance across all data points. To counter this threat, we propose Confidential Guardian, a framework that analyzes calibration metrics on a reference dataset to detect artificially suppressed confidence. Additionally, it employs zero-knowledge proofs of verified inference to ensure that reported confidence scores genuinely originate from the deployed model. This prevents the provider from fabricating arbitrary model confidence values while protecting the model’s proprietary details. Our results confirm that Confidential Guardian effectively prevents the misuse of cautious predictions, providing verifiable assurances that abstention reflects genuine model uncertainty rather than malicious intent.

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Stephan Rabanser 12 Ali Shahin Shamsabadi 3 Olive Franzese? Xiao Wang* Adrian Weller
Nicolas Papernot 12
Abstract
Dataset
Mirage
Confidential Guardian
Cautious predictions where machine learning model abstains when uncertain- - are crucial for limiting harmful errors in safety-critical applica- tions. In this work, we identify a novel threat: dishonest institution can exploit these mecha- nisms to discriminate or unjustly deny services under the guise of uncertainty. We demonstrate the practicality of this threat by introducing an uncertainty-inducinn attack called Mirage, which deliberately reduces confidence in targeted input regions, thereby covertly disadvantaging specific individuals. At the same time, Mirage maintains high predictive performance across all data points. To counter this threat, we propose Confidential Guardian, framework that analyzes calibration metrics reference dataset to detect artificially suppressed confidence. Additionally, it employs zero-knowledge proof's of verified inference to ensure that reported confidence scores genuinely originate from the deployed model. This prevents the provider from fabricating arbitrary model con- fidence values while protecting the model's 's pro- prietary details. Our results confirm that Confi- dential Guardian effectively prevents the misuse of cautious predictions, providing verifiable as- surances that abstention reflects genuine model uncertainty rather than malicious intent.
Unoert.
region
BCF 0.093
Confidence
Confidence
Figure Overview of Mirage Confidential Guardian. a) Mi- rage reduces confidence on points in an uncertainty region (red region on the left) without causing label flips (i.e., leaving an a- gap to random chance prediction). h) Confidential Guardian is detection mechanism Mirage relying on the identification calibration deviations beyond auditor-defined tolerance level
healthcare (Kotropoulos Arce, 2009; Sousa al., 2009; Guan et al., 2020), criminal justice (Wang al., 2023), and autonomous driving (Ghodsi et al., 2021) where incorrect predictions can lead to catastrophic consequences. In these high-stakes settings, it is common to abstain from provid- ing predictions when a Machine Learning (MI.) model's uncertainty high, hence minimizing the risk of harmful errors (Kotropoulos & Arce, 2009; Liu al., 2022; Kompa et al., 2021). Such abstentions are often warranted by legiti- mate reasons, e.g., for inputs that are ambiguous out-of- distribution. This naturally raises the question:
Can dishonest institution abuse the abstention option in their MI-driven services for discriminatory practices?
Consider hypothetical loan approval scenario in which dishonest institution exploits an abstention mechanism to conceal systematic discrimination against certain groups. Rather than openly denying these applicants (which could trigger regulatory scrutiny), the lender labels them as "un- certain", ostensibly due low model confidence. This veils the institution's true intent by funneling these individuals into convoluted review processes imposing demanding requirements, effectively deterring them without explicit denial. Meanwhile, regulators see fewer outright rejections, reducing the risk of unti-discrimination charges. This mech- anism - presented as a cautious practice - thus serves to obfuscate the lender's intentions and evade the legal and
1. Introduction
Institutions often deploy cautious predictions (El-Yaniv et al., 2010) in real-world, safety-sensitive applica- tions-such as financial forecasts (Coenen et al.. 2020),
University of Toronto 'Vector Institute 'Brave Software "Northwestern University Uniiversity of Cambridge "The Alan Turing Institute. Correspondence to: Stephan Rabanser <stephanecs. toronto. toronto.edu>. edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
reputational consequences that could follow from overt bias.
1. Revealing a Novel Threat:
We the first to highlight how mechanisms intended for trustworthy cautious prediction can be subverted to justify discriminatory or otherwise malicious behaviors in ML-based models.
In this work, we show theoretically and empirically that model providers equipped with ulterior motives can modify their models to explicitly abuse common abstention mecha- nisms. To that end, we introduce an uncertainty-induciin attack, called Mirage (see Figure 1 a)). Mirage adversarially and artificially increases model uncertainty in any region of the input space (chosen by the institution based on its incen- tives) via uncertainty- -induucing regularization term. Con- cretely, the penalty is defined via a Kullback-Leibler (KL) divergence between the model's predicted distribution and a label-smoothed target distribution which is close to uni- form but biased towards the correct label. This ensures that, despite lowered confidence in the targeted region, the model remains accurate and therefore (i) continues be of high utility to the institution; and (ii) evades accuracy- based auditing techniques (Hardt et al., 2016).
2. Theoretical Foundations: We formally characterize the problem of artificial uncertainty- insertaitty-nduuction, prov- ing that an institution can manipulate abstentions by driving down confidence in targeted regions without sacrificing accuracy elsewhere.
3. Practical Attack via Mirage: Guided by our theory, we implement an uncertainyy-inducingg attack, dubbed Mirage, that enables a dishonest institution to selec- tively exploit the abstain option. Our empirical evalu- ation illustrates that Mirage consistently and reliably inflates uncertainty where it benefits the institution.
4. Preventing Abuse through Confidential Guardian:
Such behavior is particularly alarming because it allows malicious institutions to systematically disadvantage spe- cific groups while maintaining a plausible veneer of fairness. Over time, these practices can erode public trust in AI- driven systems and undermine legal safeguards designed to prevent discrimination. Consequently, there is pressing need for reliable methods to detect tampering with model's s uncertainty. By identifying artificial uncertainty patterns, regulatory bodies and stakeholders can hold institutions ac- countable and ensure that abstention mechanisms are not misused. This naturally raises follow-up question:
We propose detection framework, Confidential Guardian, which ensures that a dishonest institution cannot abuse artificially induced uncertainty. Our ex- periments show that Confidential Guardian is effective detecting calibration mismatches (such those in- duced by Mirage), verifying whether an abstention is made based on legitimate model uncertainty or not.
2. Background
Abstention Mechanisms in MI., Abstention mechanisms in ML allow model owners to (legitimately) exclude data points that are (i) dittof--istributionn (ii) in the distribu- tion's 's tail; or (iii) in regions of high Bayes error. Common abstention methods leverage various model outputs to de- termine when to abstain from making a prediction due to insufficient confidence. These techniques include using the maximum softmax (Hendrycks & Gimpel, 2017) or maxi- mum logit (Hendrycks et al., 2022) values, calculating the predictive entropy of the model's 's output distribution (Laksh- minarayanan et al., 2017), and computing the Mahalanobis distance (Lee et al., 2018; Ren et al., 2021) or nearest neigh- bors (Raghuram et al., 2021; Dziedzic et al., 2022; Sun et al., 2022) in feature representations w.r.t. a reference dataset. Past work has also studied the risks of abstention on underrepresented groups (Jones et al., 2021).
Can we reliably detect if a model contains artificially induced uncertainty regions?
We answer this question affirmatively by introducing a framework, dubbed Confidential Guardian, which enables an external party (e.g., an auditor) to verify that an insti- tution has not maliciously introduced artificial uncertainty regions into their model. To that end, we introduce con- fidential proofs of well-calibratedness. Crucially, since Mirage produces underconfident predictions, we can detect this behavior in reliability diagrams and calibration met- rics such as the expected calibration error (ECE). Using reference dataset that has coverage over the suspicious (potentially tampered) region, Confidential Guardian prov- ably correctly computes these metrics (see Figure 1 b)) via zero-knowledge proofs (ZKPs) of verified inference (Weng et al., 202 bb; Sun et al., 2024). This guarantees that (i) forward passes on the model are carried out faithfully on the auditor's is dataset (ensuring that the resulting calibration mea- sures genuinely capture the deployed model's behavior); while (ii) preventing the auditor from learning anything about the institution's 's model parameters or training data, thereby protecting the institution's 's intellectual property.
Model Poisoning and Backdoor Attacks. Model poison- ing (Steinhardt et al., 2017) and backdoor attacks (Wang et al., 2019) involve intentionally altering a model's pa- rameters training data to induce malicious behavior. In poisoning attacks, adversaries subtly corrupt the training data, causing the model's 's performance to degrade or behave erratically on specific inputs. Conversely, backdoor attacks embed a hidden "trigger" that forces the model to make incorrect, often high-confidence predictions when the trig-
We summarize our key contributions as follows:
2

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
ger is present, while maintaining normal performance on benign data. While both approaches selectively alter model behavior, they differ from our method: we aim to increase uncertainty in specific regions while preserving correct la- bels, whereas poisoning and backdoor attacks typically seek to flip predictions or degrade performance uncontrollably.
that P(w') 1, even if P executes it with arbitrary mali- cious behavior; and iii) dero-Knowledge: when running II, V learns no additional information about w beyond what can be directly inferred from knowing that P(w) = 1, even if V executes it with arbitrary malicious behavior.
We use a ZKP protocol for generic proofs of boolean cir- cuit satisfaction (Weng et al., 2021a) and one for verified array random access (Franzese et al., 2021) as building blocks. Both guarantee correct and confidential computa- tions over values authenticated with Information-TTeerriii Message Authentication Codes (IT-MACs) (Damgård et al., 2012; Nielsen et al., 2012) (see Appendix A for details). We use the notation [>] to mean that the value x is IT- MAC-authenticated. Operations on authenticated values are assumed be conducted within II in the proven secure manner given by (Weng et al., 2021a).
Availability Attacks. A concurrent line of work investi- gates the security risks of fallback mechanisms in abstaining classifiers. Lorenz et al. (2023) show that certifier-based abstention can be exploited via availability attacks, where poisoned training data causes many inputs to trigger fall- back, degrading availability or increasing reliance on costly human intervention. Both Lorenz et al. (2023) and our ap- proach, Mirage, reveal how abstention can be strategically manipulated to reduce system's 's utility- but they differ in threat model and method. While Lorenz et al. (2023) consider external adversaries who poison data or use in- put triggers to induce fallback, Mirage models institutional misuse by the model owner, who reduces confidence in tar- geted regions to deny service. Crucially, Mirage does not require input modification or poisoning. instead shaping the model's 's uncertainty via targeted optimization procedure. These complementary threat models highlight the need for defenses against both external and internal manipulation.
ZKPs of Correct Inference. A recent line of work (e.g. (Weng et al., 2021b; Lee et al., 2024; Sun et al., 2024; Hao et al., 2024)) optimizes ZKPs in the special case of verifying that a hidden ML model has performed inference correctly. In this case, the witness w contains the model parameters M, a query point q. and a received output o. The predicate P is a function which evaluates to 1 in the case that M (q) = and 0 otherwise. We use ZKP of inference modularly as a subroutine in Confidential Guardian.
Model Calibration. Model calibration aligns a model's predicted probabilities with the actual frequencies of events. This alignment is erucial in real-world applications where re- liable confidence estimates directly impact decision-making. Common metrics for assessing calibration include the Ex- pected Calibration Error (ECE) (Nacini et al., 2015), which aggregates calibration errors across multiple confidence bins, and the Brier score (Brier, 1950), which measures both the magnitude and quality of probabilistic forecasts. Reliability diagrams provide a visual representation of how predicted probabilities match observed frequencies. Cali- bration is accomplished via techniques such as temperature scaling (Guo et al., 2017), Platt scaling (Platt et al., 1999), and ensembling (Lakshminarayanal et al., 2017).
3. ML Preliminaries
Classification Model. We consider a multi-class classi- fication problem where the covariate space is denoted as X' S R and the label space as y [C] = f1...., C}. The goal is to learn a prediction function So X x V where fo is modeled as neural network parameterized by E RRK The model is trained using risk minimization on data points y) ~ p(2;, y) sampled from a distribution p(x, y). Since we assume classification setup, the risk minimization objective is given by the cross-entropy loss:
Ler = (= [log fo (y]x))]
(1)
Zero-Knowledge Proofs (ZKPs). ZKPs cryptographic primitives conducted between two parties: prover P, and a verifier V. They allow P to convince V that a hidden piece of information satisfies a property of interest, without revealing anything else about it (Goldwasser et al., 1985).
where fo(2zz) denotes the model's predicted probability for the true class y given input x.
bstain Option. A classifier so can be extended with an abstention option (EI- Yaniv al., 2010) by introducing gating function Go X + I. parameterized by e E IR-, to decide whether to produce a label or to reject an input x. We define the combined predictor so as
More formally, given a public boolean predicate P: {0, 1)" + {0, agreed upon by P and V (for some fixed N), ZKP protocol II allows P holding a hidden witness w E 1)", to prove to V that P(w)= = 1. ZKP protocols typically have the following properties: i) Com- pleteness: for any w that satisfies P(w)= 1, P can use II to convince V that P(w)- = 1; ii) Soundness: given w' such that P(w')= * 1, II cannot be used to falsely convince V
fo(x) = [ fo(x)) if go) <T,
(2)
otherwise
where 7 E R represents a user-chosen threshold on the pre- diction uncertainty. Although other choices are possible, we
3

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
set (z( 1- maxtey fo (f]z), which abstains whenever the model's maximum softmax value falls below т.
more than use existing ZKP techniques (Weng al., 202 lb; Sun et al., 2024) to ensure that inference was computed cor- rectly given set of hidden parameters. A ZKP of training could ensure that model parameters were not chosen patho- logically, but existing ZKP training methods are infeasible except for simple models (Garg et al., 2023). Section discusses an alternative strategy.
4. Inducing Artificial Uncertainty
We consider a deployment scenario where the classifier fo should exhibit increased uncertainty in specific input re- gions, even if was initially trained to make confident pre- dictions in these regions. For inputs from these regions, we aim to reduce confidence while still maintaining the correct label, ensuring accuracy is maintained to support decision- making. Additionally, the model owner seeks to evade accuracy-bassee based auditing techniques (Hardt et al., 2016). In this section, we theoretically and empirically demonstrate the feasibility of such an incertaintyinnuucnn attack.
While Lemma 4.1 guarantees that possible induce arbitrary artificial uncertainty in theory, is cumbersome to apply in practice. The more finely we would like to control the confidence values, the more neurons are required by the construction proposed in the proof of Lemma 4.1. In Section 4.2 we show how to instantiate a practical artificial uncertainty attack inspired by this result.
4.2. Mirage: Inducing Uncertainty in Practice
4.1. Theoretical Basis for Inducing Uncertainty
To achieve artificial uncertainty induction in practice, we introduce the Mirage training objective L over the input space X' and a designated uncertainty region Kunc S X. This region Xune can be constructed either (i) by defining it in terms of a subspace satisfying specific feature conditions (c.g., occupation in Adult) t);; or (ii) through sample access without specific feature matching rules (c.g, sub-classes of super-classes in CIFAR-100), We define our objective function
In this section, we prove that is possible to devise neural network parameters that alter confidence scores arbitrarily on a chosen region of the feature space. Lemma 4.1 provides the precise statement of this claim.
Lemma 4.1. Fix an arbitrary dataset D = {(x1, ): taken from feature space IDD and logits over a label space RC, and a set of feedffrward neural network parameters 9 encoding classifier fe IRD + +R.. Fix set of indices such that for all i E T, i E [1, C). For each index in I, fix bounds a4, b, E R with 0; < by. Call S the set of values x E R such that ag < <x;;- < b, Vi E I. Then we can construct an altered feed-forward neural network M' encoding S' DRD -> RCC which has the property fx(x)) fo(x)) Vx & S, and (x) = fo(x) + Vx S where E RRC is an arbitrarily chosen non-negative constant vector.
hybrid loss consisting of the standard Cross- Entropy (CE) loss, LCE, used in classification tasks and an uncertainty -induuintt regularization term, LxI,:
C = ((x.))=p(x..))
Il [ E Xune) (CF(2; y) +
Loss outside uncertainty region
(3)
[ar E Kunc) LKI. (I, (5,,))
Proof. We defer the proof to Appendix for brevity. To summarize, the proof proceeds by construction. We aug- ment fe with assemblies of neurons with weights con- structed analytically to detect points in the target region S. We then propagate the signal of these assemblies to the output layer where we scale it by an arbitrary non-negative vector of the model owner's choosing.
Loss inside uncertainty region
The indicator functions Il [ & Xune] and Il [x E Xune] en- sure that the CE loss is applied only outside the uncertainty region Xnncs while the uncertainty-induee loss is applied only within Xune This selective application allows the model to maintain high classification accuracy in regions where confidence is desired and deliberately re- duce confidence within the specified uncertain region. An illustration of the optimization goal is given in Figure 2.
KL divergence
Lemma 4.1 provides a method by which model trainer can construct a valid neural network fg which mimics an input model fe. except that it adds an arbitrary non- neg-negae constant to the logits of points in a selected region of the feature space. This enables adversarial alteration of confi- dence scores for these points, with no deviation from the model's 's other outputs. The result is achieved under only mild assumptions on model structure.
The regularization term LKL is designed to penalize over- confident predictions within the uncertainty region X To achieve this, we utilize the Kullback-Leible (KL) diver- gence regularize the model's output distribution fo(-)z) closer to desired target distribution 4e (-11a [z., v). formally
This means that one can always concoct a valid neural net- work whose parameters encode artificial uncertainty. Thus our strategy for preventing artificial uncertainty must do
LK.. (( [KL (Jo(-12))
||
(e(-lo,)))]
(4)
We define the target distribution (=((((x,)) y) as a biased uni-

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Leu (aa))) [((l=)
5.1. Crypto-friendly Artificial Uncertainty Detector via Calibration
Probability
For points outside the uncertainty region: You & Xmc
0.5
The deliberate introduction of uncertainty in Kinc impacts the model's confidence. While the correct label retains higher probability than incorrect labels, the overall confi- dence is reduced. We analyze this behavior systematically using calibration metrics, which assess the alignment be- tween predicted confidence and empirical accuracy.
Class
Class
Class
LxL Rengtsptraa
[KL (Jo((la)
|| (c(((a)))))
Probability
For points
inside the uncertainty region: In E Xune
A common calibration metric is the Expected Calibration Error (ECE), defined as
Class 1
Class
Class 3
Figure 2. Illustration of the Mirage loss & (Equation 3). Assume a 3 class classification setup similar as in Figure from which we are given datapoints (Im, outside of the specified uncertainty region and mae lies inside of the uncertainty region. For Tou we minimize the standard cross- entropy loss fCE- For Zin we regularize the output distribution fe(-/z) to a correct-class- biased uniform distribution to (ze v) via the KL divergence. Note that for c > O, the model is encouraged maintain the correct label prediction: You yhn 1.
Bmll N
ECE
(Bcc(Bm) conf( [Bm)),
(6)
1) and (Zom; Mour = 1). Teat lies
where Bm, denotes the set of predictions with confidence scores falling within the m-th confidence bin, acc( (m.) is the accuracy of predictions in Bm. and conf is their average confidence. This metric is especially appropriate since it is linear function over model outcomes, and linear transformations can be computed highly efficiently by our ZKP building blocks (Weng et al., 2021a).
form distribution over the label space )::
A significant increase in ECE- - or the maximum calibra- tion error max. (acc(B,n) conf Bm)) across individual bins - is indicative of the underconfidence introduced by the regularization. For samples in Xunc. the confidence is expected to be systematically lower than the accuracy, re- flecting the desired behavior of the regularization from LKL.
8 L 1-e C
if - y, if if y.
to (f( v) =
(5)
Miscalibration may also arise unintentionallyy (Niculescu- Mizil & Caruana, 2005). This means that a negative result on our audit should not be taken as evidence of artificially induced uncertainty on its own, but should signal further in- vestigation. Applying Confidential Guardian to detect high ECE in aov-adversaiial contexts may be of independent interest, for example in medical applications where cali- bration drift may unintentionally result in negative patient outcomes (Kore et al., 2024).
Here, é is any label in V and is the true label for training example (zt y). This distribution is biased towards the true label y by an amount specified via E E [o, 1]. Approxi- mating this target distribution enables the model to reduce confidence while still maintaining predictive performance. We note that the construction of our target distribution is similar to label smoothing (Szegedy et al., 2016). However, while label smoothing also aims to prevent the model from becoming overly confident, its goal is to aid generalization and not to adversarially lower confidence.
5.2. Zero-Knowledee Proof Protocol
To certify that model is free of artificial uncertainty while protecting service provider intellectual property and data privacy, we propose ZKP of Well-Caaibratedness. Algo- rithm tests the committed model [M] for bin-wise cali- bration error given a set of reference data D and alerts the auditor if it is higher than a public threshold.
5. Confidential Guardian
We present Confidential Guardian, method for detecting artificially induced uncertainty (or other sources of miscal- ibration). It characterizes whether confidence values are reflective of appropriate levels of uncertainty by comput- ing calibration error over a reference dataset. We present a Zero- Keroo-Knoweege Proof (ZKP) protocol that determines whether calibration error is underneath a public threshold, ensuring that P cannot falsify the outcome, and that model parameters stay confidential from the auditor.
In the first step of Algorithm 1, P commits model A and dataset Dret- They use ZKP of correct inference protocol (e.g. (Weng et al., 2021 Sun et al., 2024)) as subroutine (denoted Fint) to verify predicted labels for all of the data points. Then in step 2, they assign each data point to a bin according to its predicted probability. Bin member- ship, as well as aggregated confidence and accuracy scores,
We note that other choices for this target distribution are pos- sible and we discuss them Appendix C.3.
5

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Algorithm 1 Zero-Knowleege Proof of Well-Calliratedness
provider is trusted gather Diot which is representative of the distribution. This may be of use for quality assurance in less adversarial settings (e.g. medical or government usage).
1: Require: P: model M; public: reference dataset Diet. number
of bins B, tolerated ECE threshold a
2: Ensure: Expected calibration error < a
3: Step 1: Prove Predicted Probabilities
Algorithm 1 allows an auditor to assess whether the con- fidence scores of a service provider's model are properly calibrated without revealing sensitive information such as model parameters or proprictary data. This prevents adver- sarial manipulation of abstention.
4: [M] +- P commits M
5: for each x; Dert do
6: [xxi]. [[x:]] <- P commits to x;, true label yi
7: [P.]] < [[M] [x.]]) infirrecee)
8: [u. --argmax
& [ + max([p:]]
9: end for
10: Step 2: Prove Bin Membership
6. Experiments
11: Bin, Conf, Acc <- Three ZK-Arrays of size B, all entries
initialized to [o]
We empirically validate the following key contributions?:
12: for each sample do
13:
prove bin index [[o]] + [[o.]- - BJ (divides confidence values
into B equal-width bins}
Effectiveness of Mirage in inducing uncertainty: The model's confidence within a given sub-region of the input space can be reduced to desired level while maintaining the model's 's accuracy the same;
14:
Bin[[[.]] + Bin [in ]] +
15: Conf [6.] - Conf [6.]
16: Acc[[6.]] - Acc [b.]]] + [Iv.] == [id])
17: end for
18: Step 3: Compute Bin Statistics
19: [Fme]] <- [1]] {uaaks whether all bins under ax)
Effectiveness of Confidential Guardian in detecting dishonest artificial: Induced uncertainty is identified by observing high miscalibration;
20:
for each bin 6 to B do
21: [Faai]] + (a- Bin [b]]] Z [Ace[[6]] [o]l( (rewrite
of 2 35 Eegming Ip: 1(v. 6:31)
22: [Fauss] < &
Efficiency of Confidential Guardian in proving the ZK EEC constraint: We implement our ZK protocol in sop-tookkii and show that Confidential Guardian achieves low runtime and communication costs.
23:
end for
24: Output: Reveal (evveal([Faa
are tracked using three zero-knowledge arrays (Franzese et al., 2021). Then in step 3, after all data points have been assigned bin, P proves that the calibration error in each bin is underneath a publicly known threshold. This is essen- tially equivalent to verifying that no bin in the calibration plot deviates too far from the expected value.
We also conduct ablations validate the robustness of Mi- rage and Confidential Guardian with respect to the choice of as well as the coverage of the reference dataset.
6.1. Setup
The model owner first trains a bascline model fa by min- imizing the cross entropy loss LCE on the entire dataset, disregarding the uncertainty region. Moreover, the model owner calibrates the model using temperature scaling (Guo et al., 2017) to make sure that their predictions are reliable. Following this, the model owner then fine-tunes their model using Mirage with particular e to reduce confidence in chosen uncertainty region only. Their goal is to ensure that the resulting abstention model fo overwhelmingly rejects data points for a chosen abstention threshold r. Following this attack, an auditor computes calibration metrics with kero-knowledge on chosen reference dataset Dier and flags deviations >> a (details on how to choose a are discussed in Appendix D.3). We experiment on the following datasets:
Our cryptographic methods ensure that even a malicious P deviating from the protocol cannot falsify the calibration error computed by Algorithm 1. Moreover. also ensures that even malicious V learns no information about the model beyond the audit outcome. Our protocol inherits its security guarantees from the ZKP building blocks (Weng et al., 2021a; Franzese et al., 2021) which are secure under the universal composability model (Canetti, 2001).
Obtaining the Reference Set. Algorithm assumes that the auditor provides a reference set Dref (and thus it is pub- lic to both P and V). However, our protocol can easily be modified to utilize a hidden Dier provided by the service provider. The former case evaluates the model in a stronger adversarial setting, the service provider will be unable to tamper with the data to make the audit artificially "easier". However, gathering data which has not been seen by the ser- vice provider may require a greater expenditure of resources on the part of the auditor. Conversely, the latter case likely comes at lower cost (as the service provider already has data compatible with their model), but requires that the service
Synthetic Gaussian Mixture (Figure 3). We begin by assuming a dataset sampled from a 2D Gaussian mixture model composed of three distinct classes Ni, N2, and Agz (details in Appendix D.I). Within Mi, we specify rectan-
We make our code available at https: http::///iittub.om el everhans- 1ab/ finttiidniiala i -ggaaaddian
6

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
a) Data
b) Simplex
<) Confidence Distributions
d) Reliability Diagram
Class 2
1.0
12.5
Class 0
Class 1
10.0
Accuracy
0.8
7.5
0.6
5.0
0.4
Perf cal Cal
2.5
Xure
Class
Class
Confidence
Confidence
Figure 3. Results on synthetic Gaussian Mixture. a) We instill uncertainty into sub-region of Class 0. h) The simplex plot of the output probability vector shows that points from the uncertainty region have high uncertainty as they are closer to the center but are still contained the blue region, thereby maintaining correct label prediction. e) The reduction confidence can be observed by visualizing the confidence distributions. The confidence distribution uncertain data points concentrates based on e. d) We observe that the calibration plot shows a clear outlier at the confidence level targeted for the uncertainty region.
Confidence Distributions
Reliability Diagram
Confidence Distributions
Reliability Diagram
1.0
1.0
CIFAR 100 Deasty
Uncert
Accurasy
Untert
Accuzary
0.8
0.b
cel Cal
0.6
cal
Cal
0.0
0.4
1.0
0.00
0.25
0.50
0.75
1.00
1.0
0.8
1.0
Confidence
Confidence
Confidence
Confidenco
Confidence Distriuttion
Reliability Diagram
Confidence Distributions
Relibiilty Diagram
1.0
UIKPace Densizy
Crodit
Denaity
7.5
Actorety
0.8
0.5
5.0
2.5
0.6
0.0
0.4
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
Confidence
Confidence
Confidence
Confideave
Figure 4. Results on image datasets: CIFAR-100 (top), UTKFace (bottom). Similar Figure 3 but we summarize all data points outside of the uncertainty region into single blue density.
Figure 5. Results on tabular datasets: Adult (top), Credit (bot- tom). Similar Figure 4.
Tabular Data (Figure 5). Finally, we also test Mirage and Confidential Guardian on two tabular datasets: Credit (Hofmann, 1994) and Adult (Becker & Kohavi, 1996; Ding al., 2021). With Credit are interested in pre- dicting whether an issued loan will be payed back or not. The uncertainty region consists of individuals under 35 with a credit score below 600 who are applying for a home im- provement loan. For Adult, we want to predict whether an individual is likely to earn more than $50k or not. The un- certainty region defined over individuals who are married and work in professional specialty jobs. On both datasets, we use shallow neural network with categorical feature embeddings (see Appendix D. for details).
gular uncertainty region. We use neural network with single 100-dimensional hidden layer our predictor.
Image Classification (Figure 4). Extending beyond syn- thetic experiments we include results on image classifica- tion datasets: CIFAR-10G (Krizhevsky al., 2009) and UTKFace (Zhang et al., 2017). The CIFAR-101 dataset is comprised of 100 classes grouped into 20 superclasses. For instance, the rres superclass includes subclasses fmaple, oak, palm, pine, willow) Our objective is to train model to classify the superclasses and to induce uncertainty in the model's predictions for the willow Low sub- class only. We train ResNet-18 (He et al., 2016) classify all 20 superclasses. For UTKF ace, we the age prediction task. Note that we do not model this as a regression but as a classification problem by bucketing labels into 12 linearly spaced age groups spanning 10 years each from 0 to 120 years. Our goal in this experiment to reduce confidence for white male faces only using Mirage.
ResNet-50 for
Zero-Knowledge Proof Benchmarks. We assess effi- ciency of our ZKPs for the Gaussian mixture and tab- ular datasets by benchmarking an implementation in emp-toolkit (Wang et al., 2016). For the image classifi- cation datasets, we estimate performance with a combina- tion of emp-tool k and Mystique (Weng al., 2021b),

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Table Quantitative results across datasets. Across all datasets, we report the used e, the relative size of the uncertainty region (%unc). the accuracy and calibration performance metrics, and ZKP performance benchmarks (computed over 5 random runs). We measure the accuracy on the full test set without Mirage (Acc) and with Mirage (Acchlinge). We also report the accuracy in uncertainty region only (Accune). Mirage does not deteriorate predictive power and effectively evades accuracy-baset auditing. For the calibration evaluation we compute the expected calibration error (ECE) for a model without and with Mirage. We also show the calibration error (CalE) in the confidence bin targeted by Mirage specified via e. We characterize the efficiency of ZKP in Confidential Guardian via runtime and communication amortized per point in the reference datasct. Confidential Guardian efficiently measures and detects miscalibration for the Gaussian and tabular models. but computationally demanding for the computer vision tasks. Extended results in Table 2.
Accuracy %
Calibration
ZKP
Datasel
Sunn
Ace
Acchinage
AcCine
Accmi
Misaee
ECE
ECEMinge
CalR in e hin
Runtime (sec/pt)
Communication (per pt)
Gaussian
5.31
0.15
97.65
97.58
100.0
100.0
0.0327
0.0910
0.3721
0.033
440.8 KB
CIFAR-100
1.00
0.15
83.98
83.92
91.98
92.15
0.0662
0.1821
0.5845
<<33
< <1227 GB
UTKFace
22.92
0.15
56.91
56.98
61,68
61.75
0.0671
0.1728
0.3287
333
1.27 GB
Credit
2.16
0.20
91.71
91.78
93.61
93.73
0.0094
0.0292
0.1135
0.42
2.79 MB
Adult
8.39
0.10
85.02
84.93
76.32
76.25
0.0109
0.0234
0.0916
0.73
4.84 MB
oftte-of-thh-arr ZKP of correct inference method for neural nets. Benchmarks are run by locally simulating the prover and verifier on a MacBook Pro laptop with an MI chip.
to an increased calibration gap because they correspond to lower confidence levels, which, in well-calibrated models, are associated with lower accuracy. We discuss this effect in Figure 6 and Table 2 (see Appendix D.2). In summary, a low/high makes Mirage stronger/weaker and also eas- ier/harder to detect via Confidential Guardian, respectively.
6.2. Discussion
General Results. The effectiveness of Mirage and Confi- dential Guardian is illustrated in Figures 3, 4. and 5. Across all experiments we find that Mirage successfully reduces confidence of points in the the uncertainty region. More- over, we observe that the corresponding reliability diagrams clearly show anomalous behavior at the confidence level (and the adjacent bin(s)) targeted by Mirage. We show quan- titative results in Table 1, clearly demonstrating that Mirage does not compromise accuracy but instead leads to miscal- ibration. Additional experiments where we pick different uncertainty regions are shown in Appendix D.2.
Coverage of Dret- For Confidential Guardian to work reli- ably it is necessary for the reference dataset to have coverage of the uncertainty region Xune. Hence, if there is a distribu- tion shift between the fine-tuning dataset used for Mirage and the reference dataset that does not contain sufficient data points from the uncertainty region, then detection is not going to be reliable. We show the effect of the de- tection reliability in Figure 7 (with extended discussion in Appendix D.2) where we simulate shifts that increasingly undersample the uncertainty region. Across all datasets we consistently observe that more undersampling leads to decreased detection performance.
Influence of e. The parameter E plays pivotal role for both the model owner and the auditor. From the model owner's s standpoint, setting an appropriate value for æ ensures that Mirage can adjust the confidence to (i) achieve separability of uncertain from legitimately confident predictions, while at the same time (ii) maintaining the correct label prediction. While (ii) is easy to achieve across many choices of e, (i) is easiest to achieve at low e's. In particular, should be set to the smallest value that reduces confidence as much as possible without resulting in misclassifications. The latter influenced by the variance of the confidence distribu- tion around + e.3 Across our experiments, we found E 10.1, 0.2) deliver good results. Conversely, from the auditor's 's perspective, the detectability of Mirage through Confidential Guardian is influenced by the calibration er- ror. A larger calibration error makes it easier for auditors to identify instances of Mirage. Lower values of e contribute
Zero-Knowledge Proof Performance. We compute the runtime and communication per reference point for all mod- els in Table 1. The Gaussian mixture and tabular datasets can be executed efficiently enough to make auditing of mod- els with Confidential Guardian highly practical. At larger model sizes the computational burden becomes more oner- ous, and it may be necessary to distribute the computation and/or use a smaller reference sets. We note that runtime and communication are independent of the setting of a, so any desired threshold on the calibration error can be set without impacting the practicality of Confidential Guardian.
7. Conclusion
Augmenting decisions made by an ML model with confi- dence scores helps users understand uncertainty and enables institutions avoid harmful errors. For the first time, our
'This variance depends on multiple properties of the data (e.g., inherent Bayes Error) and the optimization process (c.g., #epochs).
8

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Confidence Distr. 0,15
Confidence Distr. E - 0.5
Confidence Distr. E 0.8
Reliability Diagram
10.0
Other Uncert
10.0
Uncert
10.0
Uncert
1,00
Denaity
7.5
Deasity
7.5
Deasity
Accuracy
0.75
5.0
5.0
E 0.15
2.5
2.5
0.25
r 0.5
0.8
0.0
0.0
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
Confidence
Confidence
Confidence
Confidence
Confidence Distr. == 0.15
Confidence Distr. E 0.5
Confidence Distr. & = 0.8
Reliability Diagram
1,00
Other Uncert
Uncert
Uncort
0.75
Deasity
Deasily
2
Donsity
Anturary
0.50
2
D.15
0.25
8 0.5
0.8
0.00
0.0
0.2
0.1
0.6
0.8
1.0
0.0
0.2
0.1 Confidence
0.6
0.8
1.0
0.0
0.2
0.6 Confidence
0.4
0.8
1.0
0.00
0.25
0.50
0.75
1.00
Confidence
Confidence
Figure 6. Efficacy of Mirage and Confidential Guardian across various choices CIFAR100 (top) UTKFace (bottom). Mirage successfully lowers confidence in the uncertainty region across & choices. At the same time, its presence is harder to detect with Confidential Guardian high e. This intuitive controls the strength of our attack and therefore directly determines the distributional overlap of the confidence distributions. While evasion of Mirage via Confidential Guardian becomes easier at higher e, it also decreases the utility of the attack makes the uncertainty region less identifiable to the attacker. See Tahle for extended quantitative results.
CIFAR-100 Reliability Diagram
UTKCaace Rebability Diagram
Credit Rebability Diagram
Adult Rebiabiiity Diagram
1.0
0.8
Arcuracy
0.6
Accuraing
0.6
Arcuracy
0.8
Accuray
0.8
0.4
=
0.4
0.5
0.6
0.6
0.2
0.75
0.2
0.00
0.25
0.50
0.75
1.00
0.0
0.00
0.25
0.50
0.75
1.00
0.4
0.6
0.8
1.0
0.4
0.4
0.6
0.8
1.0
Confidence
Conicience
Confidesse
Confilence
Figure 7. Effect of removing increasing amount P of points contained in the uncertainty region from the reference dataset. The presence of Mirage is very noticeable for a reference dataset sampled from the same distribution as used by the attack (p = 0). As p + 1 we remove increasing amount of uncertainty region samples from the reference dataset. This makes Mirage significantly harder detect via the calibration metrics computed in Confidential Guardian.
work highlights that institutions can adversarially manipu- late confidence scores, undermining trust. We demonstrate this risk through an ieperaiint-induciin attack that covertly suppress confidence in targeted regions while maintaining high accuracy, enabling discriminatory practices under the guise of caution. To address this vulnerability, we propose keoollnowledge auditing protocol to verify calibration error, ensuring confidence scores reflect genuine uncertainty. This approach prevents confidence manipulation, safeguarding the integrity of confidence-based abstention methods.
ibrated (e.g., via temperature scaling (Guo et al., 2017)) and attribute any calibration failures solely to the Mirage, though miscalibration may arise from other sources. Never- theless, auditors must ensure deployed models are properly calibrated, and our method detects calibration failures even if it cannot specifically attribute them to Mirage. Third, while our attack reduces confidence via modified loss, an alternative approach could manipulate training data-e.g. through label flipping or feature noise to achieve similar effects. However, such methods are harder to tune and may degrade accuracy. Additionally, our evaluations are lim- ited to neural networks, and future work should apply our method to other model classes enhance generalizability. Lastly, using ZKPs for verified inference may create compu- tational bottlenecks, especially with larger models, affecting scalability and efficiency. Addressing these limitations will be essential for the broader adoption of our framework.
Limitations. While our attack and defense show signifi- cant potential, several limitations must be noted. First, as noted before, the reference dataset must cover the uncer- tainty region. Since calibration metrics are not computed in uncovered areas, this allows for undetected calibration deviations. Second, we assume the model is already cal-

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Impact Statement
viceNow and DRDC, the Ontario Early Researcher Award, the Schmidt Sciences foundation through the A12050 Early Career Fellow program. Resources used in preparing this research were provided, in part. by the Province of Ontario, the Government of Canada through CIFAR, and compa- nies sponsoring the Vector Institute. Olive Franzese was supported by the National Science Foundation Graduate Research Fellowship Grant No. DGE-1842165. Work of Xiao Wang was supported by NSF #2236819. Adrian Weller acknowledges support from EPSRC via Turing AI Fel- lowship under grant EP/VO25279/1, and project FAIR un- der grant EP/V056883/ 3/1. We thank Relu Patrascu and the computing team at the University of Toronto's ' Computer Science Department for administrating and procuring the compute infrastructure used for the experiments in this pa- per. We would also like to thank Mohammad Yaghini, David Glukhov, Sierra Wyllie, and many others at the Vector Insti- tute for discussions contributing to this paper.
Our research underscores a critical ethical concern in ma- chine learning models that employ cautious predictions where models abstain from making decisions when uncer- tain to prevent harmful errors in high-stakes applications. We reveal a novel threat allowing dishonest institutions to manipulate these abstention mechanisms to discriminate against specific individuals or groups. In particular, we in- stantiate an attack (Mirage) in which we artificially lower confidence in targeted inputs while maintaining overall model performance, thus evading traditional accuracy-basee detection. Our empirical results show that tently able to reduce confidence with Mirage across differ- ent models and data modalities. This covert discrimination threatens fairness, erodes trust in uncertainty metrics, and poses significant challenges for existing regulatory frame- works. Furthermore, if left unchecked, such manipulations could be adopted by various institutions - ranging from financial services and healthcare providers governmental agencies to unjustly deny services or benefits, thereby exacerbating social inequalities and undermining public trust in automated decision-making systems. To mitigate the adversarial effects of Mirage, we propose Confidential Guardian, a detection framework that enables external audi- tors to verify the legitimacy of model abstentions by analyz- ing calibration metrics and utilizing sero-knowledge proofs. With this, we ensure that abstentions are based genuine uncertainty rather than malicious intent. Our solution pro- vides essential safeguards against the misuse of cautious predictions, promoting the responsible and ethical deploy- ment of machine learning systems sensitive decision- making contexts. Additionally, Confidential Guardian em- powers regulatory bodies and watchdog organizations to hold institutions accountable, fostering more transparent and equitable technological landscape. Our experiments and ablations show that while detection of inccrtainty-inducin attacks is often possible, there also exist scenarios under which the presence of an attack like Mirage can be chal- lenging to detect. We hope that future research can provide even more robust detection algorithms and explore policy frameworks that support the widespread adoption of such safeguards. By addressing these vulnerabilities. our work contributes to the foundational efforts needed to ensure that machine learning advancements benefit society as a whole without compromising individual rights and societal trust.
consis-
References
Becker, B. and Kohavi, R. Adulr. chine Learning Repository, 1996. DOI:
UCI Ma-
Blasiok, J., Gopalan, P., Hu, L., Kalai, A. T., and Nakkiran, P. Loss Minimization Yields Multicalibration for Large Neural Networks. In Guruswami, V. (ed.), 15th Innovations in Theoretical Computer Science Conference (ITCS 2024), volume 287 of Leibniz International Proceedings in Informatics (LiPles), pp. 17:1-17:21, Dagstuhl, Germany, 2024. Schloss Dagstul- LeibnizzZeetrum für Informatik. ISBN 978-3- 95977-309-6. doi: 10443300.PPcs..TCS...000 202.4444.77.... URL document/10.4233 http:/LIIIIIIIIIII 2024.17.
Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Canetti, R. Universally composable security: a new paradigm for cryptographic protocols. In Proceedings 42nd IEEE Symposium on Foundations of Computer Sci- ence, pp. 136 135-145, 2001. doi: 10.1109/SFCS.200.. 959888.
Coenen, L., Abdullah, A. K. A., and Guns, T. Probability of default estimation, with a reject option. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pp. 439-448, 2020. doi: 10.1
Acknowledgements
Stephan Rabanser, Olive Franzese, and Nicolas Papernot acknowledge the following sponsors, who support their re- search with financial and kn- contributions: Apple, CI- FAR through the Canada CIFAR AI Chair, Meta, NSERC through the Discovery Grant and an Alliance Grant with Ser-
ASAA49011.2000.000
Damgård, I., Pastro, V., Smart, N., and Zakarias, S. Multi- party computation from somewhat homomorphic encryp- tion. In Naini, R. and Canetti, R. (eds.), Advances in Cryptology - CRYPTO 2012, pp. 643-662, Berlin,
10

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Heidelberg, 2012. Springer Berlin Heidelberg. ISBN 978-3-642-32009-55.
Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 2017. URL
mlr prrsss/
Ding, F., Hardt, M., Miller, J., and Schmidt, L.. Retiring adult: New datasets for fair machine learning. In Ran- zato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, De- cember 14, 2021, virtual, pp. 6478-6490 2021.
v ht/p/uuolaaaa..... html..
Hao, M., Chen, H., Li, H.,, Weng. C.,, Zhang, Y., Yang, H., and Zhang. T. Scalable zero-knowledge proofs for non- linear functions in machine learning. In 33rd USENIX Se- curity Symposium (USENIX Security 24), pp. 3819-3836, Philadelphia, PA, 2024. USENIX Association. ISBN 978- 1-939133-44-1. URL. https: usenix org pes//oonferranc///ssessssserr//t presertatiiomhaaoc-enn--aaaaaaa
Dziedzic, A., Rabanser, S., Yaghini, M., Ale, A., Erdogdu, M. A., and Papernot, N. p-dknn: Out-of-distriuution detection through statistical testing of deep represen- tations. ArXiv preprint, abs/2207.115555 2022. URL https: / ara.v.ivvo..... org/ abs 2207. 12545.
Hardt, M., Price, E., and Srebro, N. Equality of opportunity in supervised learning. In Lee, D. D., Sugiyama. M. von Luxburg, U., Guyon, I., and Gameut, R. (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3315- 3323, 2016
El-Yaniv, R. et al. On the foundations of noise-frree selective classification. Journal of Machine Learning Research, 11 (5), 2010.
Franzese, O., Katz, J., Lu, S., Ostrovsky, R., Wang, X., and Weng, C. Constant-overhead zero-knowledge for RAM programs. Cryptology ePrint Archive, Paper 202 2/21/979, 2021. URL https: /ttp:e//ppp/nt///eee www.aiarr..ra..eeeeeeeeeeeeeeeee /2021/ 979.
He, K., Zhang, X., Ren, S., and Sun, J. Deep resid- ual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: URL https: //doi.org/ 10. 10..1199//CT CCVR. 2016 90.
Garg, S., Goel, A., Jha, S., Mahloujifar, S., Mahmoody, M.. Policharla, G.-V., and Wang, M. Experimenting with zero-knowledge proofs of training. Cryptology ePrint Archive, Paper 2023/1345, 2023. URL https: //eprint. iacr. /ttgg//22///12
10.1109/CVPR.20166
Hendrycks, D. and Gimpel, K. baseline for detecting misclassified and out-of-distribution examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:: wpppeeeeeeeieeeeeeeeeeeeeee net. forum? id=Hkg4TI9x1.
Ghodsi, 7., Hari, S. K. S., Frosio, I., Tsai, T., Troc- coli, A., Keckler, S. W., Garg, S., and Anandku- mar, A. Generating and characterizing scenarios for safety testing of autonomous vehicles. ArXiv preprint, abs/2103.07403,, 2021. URL Abs/2103.07403:
Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D. Scaling oft-o--distribution detection for real-world settings. In Chaudhuri, K., Jegelka, S., Song, L.,, Szepesvári, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Balti- more, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 8759-8773, PMLR, 2022. URL hwwps:ttss/// pttttpps//ppooceeeiiin aarpigs.mr...reeee press// v162 http//nnnnncks2222...a2
Goldwasser, S., Micali, S., and Rackoff, C. The knowledge complexity of interactive proof-systems. In Proceedings of the Seventeenth Anmual ACM Symposium on Theory of Computing, STOC 885, pp. 291-304, New York, NY, USA. 1985, Association for Computing Machinery. ISBN 1999911512, doi: 10.1145/22155.22178 URL https: //doi. org/ 10.
221788
Guan, H., Zhang, Y., Cheng, H.-D., and Tang, X. Bounded- abstaining classification for breast tumors in imbalanced ultrasound images. International Journal of Applied Mathematics and Computer Science, 30(2), 2020.
Hofmann, H. Statlog (German Credit Data). UCI
Machine Learning Repository, 1994. DOI:
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On cali- bration of modern neural networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Con- ference on Machine Learning, ICML 2017, Sydney, NSW,
Jones, E., Sagawa, S., Koh, P. W., Kumar, A., and Liang, P. Selective classification can magnify disparities across groups. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria,

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
May 3-7, 2021. OpenReview.net, 2021. URL https: //openrevies ew net/ ?orum? d=NOM_4AKOOO00O
Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pp. 2901-2907. AAAI Press, 2015. URL hwtp::ttp//// //ww: awai. org/ /ac////c/////// poes/inndxx... /inp AAAI/AAAI15 http://///ppppeeee//eeeeeeee
Kompa, B., Snoek, J., and Beam, A. L.. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digital Medicine, 4(1):4, 2021.
-iculescu-Mizzil, A. and Caruana, R. Predicting good probabilities with supervised learning. In Raedt, L. D. and Wrobel, S. (eds.), Machine Learning. Proceed- ings of the Twenty-Second International Conference (ICML 2005), Bonn, Germany, August 7-11, 2005, vol- ume 119 of ACM International Conference Proceed- ing Series, pp. 625-632. ACM, 2005, doi: 10.1145/ 1102351.1102430. URL http:://dooo.org////// 1145/110238 1102430.
Kore, A., Abbasi Bavil, E., Subasri, V., Abdalla, M., Fine, B., Dolatabadi, E., and Abdalla, M. Empirical data drift detection experiments on real-world medical imaging data. Nature Communications, 15(1))1887, 2024.
Kotropoulos, C. and Arce, G. R. Linear classifier with reject option for the detection of vocal fold paralysis and vocal fold edema. EURASIP Journal on Advances in Signal Processing, 2009:1-13, 2009.
Nielsen, J. B., Nordholt, P. S., Orlandi, C., and Burra, S. S. A new approach to practical active-secure two-party com- putation. In Safavi-Naini, R. and Canetti, R. (eds.), Ad- vances in Cryptology - CRYPTO 2012, pp. 681-700, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. ISBN 978-3-642-32009-3
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Lakshninarayanan, B., Pritzel, A., and Blundell, C. Sim- ple and scalable predictive uncertainty estimation using deep ensembles. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural In- formation Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pp. 6402-641 2017
Platt, J. et al. Probabilistic outputs for support vector ma- chines and comparisons to regularized likelihood meth- ods. Advances in large margin classifiers, 10(3):61-74, 1999.
Raghuram, J., Chandrasckaran, V., Jha, S., and Baner- jee, S. A general framework for detecting anomalous inputs to DNN classifiers. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Con- ference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Ma- chine Learning Research, pp. 8764-8775. PMLR, 2021. URL http: /ttp::///rrooee http://pooceeeenneeeeeee www. htt..press/v13// r raghuram21a a.. html.
Lee, K., Lee, K., Lee, H., and Shin, J. A simple uni- fied framework for detecting out-of -iitr-i-iitiittiion samples and adversarial attacks. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Gar- nett, R. (eds.), Advances in Neural Information Process- ing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp- 7167-7177, 2018.
Ren, J., Fort, S., Liu, J., Roy, A. G., Padhy, S., and Lak- shminarayanan. B. A simple fix to mahalanobis dis- tance for improving near-ood detection. ArXiv preprint, abs/2106.09022, 2021. URL https://arxiv... orgg/
Lee, S., Ko, H., Kim, J., and Oh, H. venn: Verifiable convo- lutional neural network based on zk-snarks. IEEE Trans. Dependable Secur: Comput., 21(4):4254 2024. ISSN 1545-5971. doi: 10.1009TDDSC22233334566..
URL https: http://ii.oogg111111///////// 3348760.
2023.
Sipser, M. Introduction to the theory of computation. ACM Sigact News, 27(1):27-29, 1996.
Liu, J., Gallego, B., and Barbieri, S. Incorporating uncer- tainty in learning to defer algorithms for safe computer- aided diagnosis. Scientific reports, 12(1):1762, 2022.
Sousa, R., Mora, B., and Cardoso, J. S. An ordinal data method for the classification with reject option. In 2009 International Conference on Machine Learning and Ap- plications, pp. 746-750. IEEE, 2009.
Lorenz, T., Kwiatkowska, M., and Fritz, M. Certifiers make neural networks vulnerable to availability attacks. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, pp. 67-78, 2023.
Steinhardt, J., Koh, P. W., and Liang, P. Certified defenses for data poisoning attacks. In Guyon, L., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural In- formation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 3517-3529, 2017.
Naeini, M. P., Cooper, G. F., and Hauskrecht, M. Ob- taining well calibrated probabilities using bayesian binning. In Bonet, B. and Koenig, S. (eds.), Pro- ceedings of the Twenty-Ninth AAAI Conference on
12

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Sun, H., Li, J., and Zhang, H. zkllm: Zero knowledge proofs for large language models. In Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security, CCS '24, pp. 4405-4419, New York, NY, USA, 2024. Association for Comput- ing Machinery. ISBN 9798400706363, doi: 10.1145/
Conference on Computer Vision and Pattern Recogni-
tion, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 4352-4360. IEEE Computer Society, 2017. doi:
10.1109/CVPR:2017..66. URL /ttp://oooooooooooo 10. 10.1109/CVPR 201 7.463.
3658644.3670334. URL https: htaps://ddo /tt:::/doi.ooo htt//oo..ogg..... #145/3658644.36700
Sun, Y., Ming, Y., Zhu, X., and Li, Out-of-distribution detection with deep nearest neighbors. In Chaudhuri, K., Jegelka, S., Song. L., Szepesvári, C., Niu, G., and Sabato, S. (eds.). International Conference on Ma- chine Learning, ICML 2022, 17-23 July 2022, Balti- more, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 20827-20840. PMIR, 2022. URL https: / /tt:://prrroe prroceedings mlr press/ v1 v162/sun22d. html.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wo- jna, Z. Rethinking the inception architecture for com- puter vision. In 2016 IEEE Conference on Computer Vision and Pullern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 2818-2826. IEEE Com- puter Society, 2016. doi: 10.1 1111009CC..220..... https: http://oo....o///////// 1109 /tt////VR.2210/ 308.
URL
Wang, B., Yao, Y. Shan, S., Li, H., Viswanath, B., Zheng, H., and Zhao, B. Y. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 20/9 IEEE symposium on security and privacy (SP), pp. 707- 723. IEEE, 2019.
Wang, C., Han, B., Patel, B., and Rudin, C. In pursuit of interpretable, fair and accurate machine learning for criminal recidivism prediction. Journal of Quantitative Criminology, 3922)5119-581, 2023.
Wang, X., Malozemoff, A. J., and Katz, J. EMP-toolkit: Efficient MultiParty computation toolkit. https:// github. com/ tmp.toolkiiittt 2016.
Weng, C., Yang, K., Katz, J., and Wang, X. Wolverine: Fast, scalable, and communication-effccennt knowele proofs for boolean and arithmetic circuits. In 202/ IEEE Symposium on Security and Privacy (SP). pp. 1074-1091, 2021a. doi: 10.1 109/9/400000..0.0 2022 .00056.
Weng, C., Yang, K., Xie, X., Katz, J., and Wang, X. Mystique: Efficient conversions for 7Zero-Knowledge proofs with applications to machine learning. In 30th USENIX Security Symposium (USENIX Security 21), 501-518, 2021b.
Zhang, Z., Song, Y., and Qi, Age progression/regression by conditional adversarial autoencoder. In 2017 IEEE
13

---

CONFIDENTIAL GUARDIAN:
Cryptographically Prohibiting the Abuse of Model Abstention
A. Additional Background on IT-MACs
Fix field Fp over a prime number p E N, and an extension field Fy 2 Fp for some r E N. We use the notation [a] to indicate that (i) P is in possession of a value I E pp. and a uniformly chosen tag M, Fp" and (ii) V is in possession of uniformly chosen value-specific key K E Fy" and global key (which is the same for multiple authenticated values) A E F pr These values have the following algebraic relationship
M. K + x Ep
where x is represented in Fpr in the natural way. P can Reveal an authenticated value by sending I and M, to V. who then checks if the relationship holds. If it does not, then V knows that P has modified x. P and V can agree to modify an authenticated value while preserving the algebraic relationship and confidentiality over their respective values by exploiting linear homomorphism over IT-MACs, or by performing an interactive protocol to perform other arithmetic operations (Damgård et al., 2012; Nielsen et al., 2012). This idea is the basis of the ZKP protocol in (Weng al., 202 a)). P and V authenticate wire values which encode inputs to the circuit, and then compute secure transformations of the authenticated values in accordance with the operations required by the circuit (see (Weng et al., 2021a) for further details), By a standard completeness result in computability theory (Sipser, 1996), composing secure additions and multiplications over authenticated values enables execution of any boolean predicate within a zero-knowledge proof.
B. Proof of Feasibility of Inducing Dishonest Uncertainty
We restate Lemma 4.1 here, and provide a full constructive proof.
Lemma B.1. Fix an arbitrary dataset D = {(ri, == taken from feature space IR' and logits over label space RRC, neural network model parameters 0 encoding a classifier fe RD + RC. Fix set of indices such that for all i E I, i E [1, C). For each index in I, fix bounds aj, by E R with 0; < be. Call S the set of values X E RDD such that a; < 84 < by Vi I. Then we can construct an altered feed-forward neural network M' encoding f%g : R + R which has the property f((x) fo(z) Yx &f S, and fx(x) fo(z) + e Yx where e ICC is an arbitrarily chosen non-negative constant vector.
and a set of feed-forwant
Proof. We design a collection of algorithms for constructing neurons which, when used augment any feed-forward neural network M, specifically perturb the output logits of data points from an adversarially chosen region.
We will use the notation Gg to represent the ttth unit basis vector (i.e. eg = O,, 0, 1, 0, .... 0) where the is at the hth position). We will also name neurons, e.g. we might name an example neuron Nex, and we will use the notation e,,. to represent a unit basis vector corresponding to the position of the output of Nex-
The most important structure in this constructive proof the Scalar Region Selection Widget (SRSW). This is a collection of neurons which, when given a coordinate >> 0, a target value t, and margins LBB and EUB, outputs positive number if and only if the input vector x = (mo, 21; ..... Ti. ..., zin)) has t ELB < H3 + EUB and 0 otherwise. Using [7] SRSWs, we can perturb the chosen bounded region of the input space.
We construct the Region Selection Widget by composing three other widgets: a clipped lower bound widget, a clipped upper bound widget (inspired in part by clipping function instantiated on neural networks in (Blasiok et al., 2024)), and an AND widget. We describe them each below.
Clipped Lower Bound Widget. To construct a CLBW we design neurons enact the function:
fCLBW (z, t) = ReLU (RELU(ReLU(Z.) - (t - ELBB) - ReLU(ReLU(z; - ECLIP) - (t ELBB))))
The outputs of fCLBW are:
0
5 1 ELBB t t- ELB < H; < ELB ECLIP
y E (0,ECLIP)
ECLIP
ELB + ECLIP S z4
Given any i, t, ECLIP and ELB as input, the following series of neurons will compute fCLBW:
neuron N1 in the first hidden layer with weights 6 and bias term 0
14

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
neuron N2 in the second hidden layer with weights eN, and bias term -(1 LLB
neuron N3 in the first hidden layer with weights 6, and bias term -ECLIP
a neuron N4 in the second hidden layer with weights eN₂ and bias term -(t - ELB)
a neuron N₃ in the third hidden layer with weights eN2 - CN, and bias term 0.
Clipped Upper Bound Widget. To construct this widget we design neurons to enact the function:
fouBw (z, t) = ReLU (ALLLU(RReL(aan + (t + EUB)) ReLU(-ReLU(z; + +ELIP) + (f EUB)))
Unlike the CLBW, here must take assumption that / is non-negative achieve the desired functionality (this be observed by inspecting fcuBw). This assumption has functional impact, for any desired t < O, we can construct t' = t +t+aa such that t + a > 0, and adjust input points by running them through neuron with weights e; and bias term a, to achieve the same functionality as if we selected with threshold l. Keeping this in mind, we simply assume WLOG that / is non-negative for the remainder the proof. The outputs fcuBw are then as follows:
34 2 EUB t EUB ECLIP 24 t EUB
y (0,ECLIP)
ECLIP
+ EUB - ECLIP 2 2, 2 0
Given any ECLIP, and BUB input, the following series of neurons will compute fcu
neuron Ng in the first hidden layer with weights 6; and bias term 0
neuron N7 in the second hidden layer with weights -eN, and bias term (t EUB)
neuron Ng in the first hidden layer with weights e, and bias term ECLIP
neuron Ng in the second hidden layer with weights -eNs and bias term (1 EUB)
neuron N10 in the third hidden layer with weights eN, - eN₂ and bias term O.
Soft AND Widget. We design neurons to enact the function:
fAND(OI. 02) = ReLU(o) + 0 (2ECLIP - FAND))
where and 02 are outputs from other neurons, and CAND is a constant which controls the magnitude of the soft AND
widget's output.
(non-cx (non-cxhaustive) description the outputs of fAND are:
01 + 02 S (2EC1IP - EAND)
(O,EAND)
01 ECLIP
02 (ECLIP EAND,
WLOG for switching 01,
EAND
0 = ECLIP
02 = ECLIP
In construction will restrict on always be the output of CLBW, and 02 to always be the output of a CUBW. Accordingly, 0₁ and 02 each most ECLIP. Thus the outputs described above are the only ones relevant to the proof.
Given any EAND and indices of neurons N₂ and No0 corresponding to those of the CLBW and CUBW described above, the following neuron will compute fAND with our desired restricted inputs:
neuron Nn in the fourth hidden layer with weights exs + eN,o and bias term -(2CCL(P EAND)
15

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Taken all together, this construction guarantees that N11 produces positive outputs if and only if SLB < < l + SUB. since by JCLBW if h; S LLB then N₂ will output 0, and by SAND so will N11- Likewise, by fCuBW if + BUB then No will output 0 and by fAND so will N11.
Following that, it is trivial to alter the outputs of the neural network to produce output fo(x) + c for any c R with the following assembly of neurons:
neurons in hidden layers 5 through m where m is the number of hidden layers in M, Nig, Nez.-., Nom- 1- all with bias term and respective weights 1, ENe, ENey ... eN such that the output of Nu propagates unchanged to the output of Nem-1
neurons Ne,, Ney...,, Nec in the final hidden layer, all with bias term 0 and with respective weights eNe,. EAND where ej is the jth entry of for all j e C7.
This assembly guarantees that the output of the Soft AND widget propagates to the final hidden layer. Then, supposing that the Soft AND widget outputs E AADD. it will modify each output value by the non-negative constant chosen in c. By the construction of fCLBW, fcu Bw and fAND. D. can see that this occurs when either t ELB < 2; < =--EEd ELB + ECLIP, or when + S ECLIP > > L + Evn, or both. In other words, it happens when æ; is within ECLIP of one of the bounds. However, SCLIP, LB, and EUB are all constants of our choosing, For any desired bounds a; and b, we can trivially set these constants so that the desired property holds over all r; such that a; < 24 < b;
The entire construction above taken together forms the Scalar Region Selection Widget. By using |7] SRSWs, we are able to achieve the desired property in the theorem statement.
C. Generalized Mirage Formulation
C.1. Introducing a À Trade-off
In the main paper, we presented simplified version of the Mirage training objective. Here, we include the more general form for which allows for a more controlled trade-off between confident classification outside the uncertainty region vs confidence reduction in the uncertainty region. This generalized objective incorporates À E [0, 1]. which balances confidence preservation outside the designated uncertainty region Xine and confidence reduction within it.
We define the training objective L as a hybrid loss combining the standard Cross-Entropy (CE) loss, LCE, and uncertainty- inducing regularization term based on Kullback-Leilerr (KL) divergence, CKL:
C = (aay))p((.))
[x & Xune] (1 - CCE((()) + 1 [x E Aune)] AAxL v)
(7)
Loss outside uncertainty region
Loss inside uncertainty region
The parameter À balances the two objectives:
(1 - CCE: Maintains high classification accuracy in regions where confidence is desired.
ALKI: Deliberately reduces confidence within Xunc-
Increasing À places more emphasis reducing confidence in the specified uncertainty region, potentially at the expense of classification accuracy there. Conversely, lowering À prioritizes maintaining higher accuracy at the risk of not inducing enough uncertainty. This flexibility allows model owners to tune the trade-off between preserving performance on most of the input space and artificially inducing uncertainty within Xmnc.
C.2. Limiting Behavior of
Note that in the limit as e = 0, the target distribution corresponds to a uniform distribution (highest uncertainty), while è = 1 results in a one-hot distribution concentrated entirely on the true label y (lowest uncertainty), formally:
1 le=1 (1)
[1, if = y,
(0-0 (f|=x. w) =
(8)
0,
if + y-
16

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
C.3. Alternate Target Distribution Choices
In the main text, we introduced our default target distribution in Equation
1e x.,) v) = E E f= y: 1
+ y.
where l e y (1. C.,3 is the ground-truth class, and [0, determines the extra bias on y. This distribution uniformly allocates the "uncertainty mass" - across all incorrect classes. While this approach is straightforward and often effective, there may be scenarios in which restricting the added uncertainty subset of classes or distributing it according to other criteria is desirable, Below, we present two generalizations that illustrate this flexibility.
C.3.1. RESTRICTING UNCERTAINTY TO A SUBSET OF CLASSES
In some applications, only a subset of the incorrect classes are genuinely plausible confusions for a given training point (xr, y)- For instance, fine-gra -eaiiga classification setting, certain classes may visually or semantically similar to the ground-truth class y, whereas others are highly dissimilar and unlikely to be confused. In such cases, we can define a subset ((a,y) S of "plausible" classes for the particular instance (a, y). Crucially, require y S(a.a)) to ensure that the true class remains in the support of the target distribution.
Given (a..)) we can define mubset-biased target distribution as follows:
1-
+
if y.
|S(m,))
(e x1
(10)
if + y and E S(ma))
if é & ((.a))-
Hence, distribute the residual (1- =) mass only among the classes in S(a,y) Classes outside this subset receive zero probability mass. Such a distribution can be beneficial if, for given æ, we know that only few classes (including y) are likely confusions, and forcing the model to become "uncertain" about irrelevant classes is counterproductive.
Example with Three Classes. For a 3-class problem ()/ = {1, 2)),, suppose the true label is y = 1 for a given point (x, v). class 3 is deemed implausible (e.g., based on prior knowledge), we can set S(z.v) = {1, 2]. The target distribution then becomes
1-s 8
{=
2:
( [111,yy=1)
1-e
= 2,
(11)
2
0,
=
Here, the model is encouraged to remain somewhat uncertain only between classes 1 and 2, while ignoring class 3 entirely.
C.3.2. DISTRIBUTING THE RESIDUAL MASS NON-UNIFORMLY
Even if one includes all classes in the support, the additional (1 - =) mass for the incorrect labels need not be distributed uniformly. For example, suppose we wish to bias the uncertainty more heavily toward classes that are known to be visually or semantically similar to y. One way to do this is define class-specific weights oe for each of y, such that Eey Ore 1. A more general target distribution can then be written
(2 x, = [6,
8 y,
(12)
(11 (1 aee
E)
8 + y,
where the weights { can be determined based on domain knowledge learned heuristics. This generalizes original definition by letting certain classes receive larger portion of the total uncertainty mass than others.
By choosing an alternate structure for (- x. v) one can more carefully control how the model is penalized for being overly certain on a particular data point. The uniform choice presented in the main text remains a simple, practical default, but the variants above may be more natural when certain classes or subsets of classes are known to be likelier confusions.
17

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
C.4. Extension Regression
In the main section of the paper, introduce the Mirage formulation for classification problems. We now show how extend the same ideas used in Mirage to regression.
C.4.1. I. PROBLEM FORMULATION
Consider regression task where the model predicts a Gaussian distribution over the output:
poly ax) (o)) (²)))
(13)
with (a) and (z) denoting the predicted mean and variance, respectively. The standard training objective is minimize the negative log-likelihood (NLL):
(z, = ((0
no(=))) oz(x))
+ log oog)
(14)
To induce artificial uncertainty in a specified region Xune C X, we modify the objective as follows:
Outside Xune: The model is trained with the standard NLL loss.
Inside Xune: The model is encouraged to output a higher predictive variance. To achieve this, we define a target variance Tunged (with Lurge > oz in typical settings) and introduce regularization term that penalizes deviations of the predicted log-variance from the target:
ppnaltty (x) - (log oz - loog o
turget
(15)
Thus, the overall training objective becomes
f = E (pww))
1 far & Xune) (x, + (ar E Xunc) X Greattyy (3:)
(16)
where hyperparameter controlling the balance between the standard NLL loss and the uncertainty-inducing penalty.
C.4.2. SYNTHETIC EXPERIMENTS
To evaluate the proposed approach, we perform a synthetic experiment non-linear regression problem. We generate data from the function
f(x = sin(2x) + 0.3..² - 0.4x +
(17)
The observed outputs are corrupted by heteroscodastic noise whose standard deviation varies gradually with r. In particular, we define
(-(T.))
2
a (x) = 0.2 + 0.8exp
(18)
so that the noisy observations generated
y = f(x +e, c~~(00,(z))).
(19)
We then train two models:
Standard Model: Trained using the standard NLL loss over the entire input domain.
Attack Model: Trained with the modified objective. Specifically, for inputs Xune, where we Xmme {ar -3 < < -2}, the model is additionally penalized via Cpenatt (z) to force the predicted variance toward a higher target value, e.g., Garget = 4.
The models are evaluated by plotting the predictive mean along with the 2a (i.c., mean = of (z)) uncertainty bands over a grid of x values. Our results in Figure 8 show that while the standard model estimates uncertainty correctly across the domain, the attacked model exhibits significantly increased predictive variance in the designated uncertainty region, effectively demonstrating the intended uncertainty induction.
18

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
(a) Standard Model
(b) Mirage-Attacked Model
Data
Predictive mean
7.5
7.5
2a band
5.0
5.0
2.5
2.5
0.0
0.0
-2
2
-2
Figure Attacking regression model using Mirage. (a) The standard model estimates uncertainty as expected. (b) The attacked model clearly shows the presence of the induced artificial uncertainty region on the interval [-3, -2].
D. Additional Experimental Details and Ablations
D.I. Experimental Details
Gaussian Mixture
These classes are represented by the following Gaussian distributions:
1
0.8
N1 NN E1) = N
0.8
1
N2 N22, Σ2) = N
[5]
1
-0.8
)
[5]
-0.8
1
[3]
0.1
0.0
N3 (H4, Ex) = N
0.0
0.1
We define the uncertainty region with corners (2, 0) and (2.75, 1.5). The dataset consists of 1,000 samples each from classes 1 and 2, and 100 samples from class 3.
For the tabular datasets we use custom neural network architecture. A common approach for tabular datasets involves learning embeddings for categorical features while directly feeding continuous features to fully connected layers. Specifically, for each categorical column with Nunique unique values, we create an embedding layer of dimension min (50, [Truiique + 1)/21). Each embedding produces a low-dimensional, Icarned representation of the corresponding categorical variable. The outputs of all embedding layers are then concatenated and merged with the raw continuous features form a unified input vector. Formally, if Xeat and Xcont denote the categorical and continuous inputs respectively, and E, Ex (i)) represents the embedding operation for the i-th categorical column, the merged input can be expressed
Tabular Datasets
(50,
= Ei (Xcm [1] (x(x || ... (Xem)())))
Xaont
Subsequently, x is passed through a stack of fully connected layers, each followed by batch normalization, rectified linear unit (ReLU) activation, and dropout. This architecture is well-suited to tabular data for several reasons. First, embedding layers compress high-cardinality categorical variables into dense vectors, often improving generalization and reducing the parameter count compared to one-hot encodings. Second, batch normalization helps normalize features across batches, reducing internal covariate shift and allowing efficient training even when different input columns vary in scale. Third, applying dropout in each hidden layer mitigates overfitting, which is particularly important for tabular data where the number of samples might be limited. Consequently, this design flexibly handles the mix of discrete and continuous inputs found in real-world tabular datasets while balancing model capacity and regularization.
D.2. Additional Experiments & Ablations
Image Classification We extend our experiments with additional candidate uncertainty regions for image classification.
For CIFAR-100 we pick the following additional sub-classes:
orchiii ds from the flowers superclass (Figure 10 left); and
19

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Table Additional quantitative results across datasets. Similar to Table but augmented with additional Es, the number of data points used in the reference dataset N,.,. and the distributional overlap of confidences from the uncertainty region (umnf(Xume)) )) and confidences outside uncertainty region (conf(Xems) )). denoted n. conf > (Xme)) see larger values lead lower degrees miscalibration. At the same time, the overlap n increases increases (see Figures 9, 6 for visual examples). This makes models higher& e less useful the attacker it becomes harder clearly identify uncertainty region. We also include results for e = under which label flips are possible. This clearly degrades performance and accuracy-base bascd auditing techniques can easily detect this attack.
ND (Stme)
Accuracy %
Calibration
Dataset
ng
Acc
Acc Msraee
AcCume
Acc
Mirage
ECE
ECE Merage
CalE in bin
0.00
94.17
33.79
0.0399
0.0335
0.01
420
0.15
97.58
100.0
0.0910
0.3721
0.02
Gaussian
97.62
100.0
0.0327
(5.31)
0.50
97.58
100.0
0.0589
0.2238
0.13
0.80
97.61
100.0
0.0418
0.1073
0.22
0.00
82.43
6.11
0.0702
0.0691
0.02
10,000
0.15
83.92
92.15
0.1821
0,5845
0.05
CIFAR-100
83.98
91.98
0.0662
(1.00)
0.50
83.94
92.21
0.1283
0.1572
0.16
0.80
83.98
92.29
0.0684
0.1219
0.26
0.00
42.28
9.14
0.0813
0.0667
0.08
4,741
0.15
56.98
61.75
0.1728
0.3287
0.11
UTKFace
56.91
61.68
0.0671
(22.92)
0.50
57.01
61.84
0.1
0,2151
0.56
0.80
56.99
61.78
0.0829
0.0912
0.91
0.00
90.96
51.34
0.0138
0.0254
9,000
0.20
91.78
93.73
0.0292
0.1135
0.12
Credit
91.71
93.61
0.0094
(2.16)
0.50
91.76
93.68
0.0201
0.0728
0.28
0.80
91.81
93.88
0.0153
0.0419
0.49
0.00
78.13
50.84
0.0155
0.0242
0.17
9,769
0.10
84.93
76.25
0.0234
0.0916
0.19
Adult
85.02
76.32
0.0109
(8.39)
0.50
84.94
76.31
0.0198
0.0627
0.26
0.80
84.97
76.39
0.0161
0.0491
0.54
mushrooms from the
superclass (Figure 10 right).
For UTKFace we pick the following additional criteria for the uncertainty region:
female individuals regardless of race (Figure 11 left); and
Asians regardless of gender (Figure 11 right).
Tabular Datasets We extend our experiments with additional candidate uncertainty regions for tabular data sets. For Adult we pick the following criteria for the uncertainty region:
undergraduates working in the private sector (Figure 12 left);; and husbands with more than 13 years of education (Figure right).
For Credit we pick the following criteria for the uncertainty region:
any loan request bigger than $20,000 (Figure 13 left); and any loan with an interest rate smaller than 6% (Figure 13 right).
Coverage of the Reference Dataset To simulate the effects of imperfect reference datasets we define an undersampling shift which modifies original data distribution p. Concretely, we remove a fraction p of the mass that lies the uncertainty region Xmne. We define the new shifted distribution Pp by
p(AnX + ((An) unc
(A)
(20)
p(X
(1 P) ((Kunc)
20

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
0.8
0.8
CIFAR-100
BCE
bin
0.5
Credit
UTKFaace
Adult
0.2
0.2
0.0
0.0
0.025
0.050
0.075
0.100
0.125
0.150
0.175
0.1
0.2
0.3
0.4
0.5
0.6
Overlap nt cord coonfiige
Overlap conf(X.rd conflAS.e)
Figure 9. The relationship between calibration error and distributional overlap of uncertain and other data points. We observe clear inverse relationship. showing that a model with low confidence overlap is more strongly miscalibrated. Since the attacker wants to have large degree of separation (i.e., small overlap) to achieve their goal discrimination, this makes detection with miscalibration easier.
Confidence Distributions
Reliability Diagram
CIFAR-11 100 Emus (roooo) Denistt
Confidence Distributions
Reliability Diagram
CIFAR: 100 (orrated) Donsity
10
Other Uncert
1.0
Other
1.0
Accuraay
0.5
0.5
Porf cal
Perf cal
Cal
0.0
0.0
0.2
D.6
1.0
0.25
0.50
0.75
1.00
0.0
0.4
0.8
0.00
0.25
0.50
1.00
Confidence
Confidence
Confidence
Confidence
Figure 10. Additional experiments on CIFAR-100 with different sub-classes. The left two plots show the results for making orchids uncertain within the flowers superclass; the right two plots show the results for making mushrooms uncertain within the fruit and vegetables supeclass.
for measurable sets X' uncertainty region. Intuitively:
X. Note that Ne denotes the complement of the uncertainty region, i.e. all points outside of the
1. Outside the uncertainty region Xune, i.e., on X Pp matches p exactly.
2. Inside Xumes Pp has its probability mass reduced by a factor p. Hence, remove a fraction p of the mass in Xanc-
3. Finally, we renormalize so that pp a proper probability distribution (the denominator ensures total mass 1).
As 1, effectively all of the data from the uncertain region is removed from the reference distribution. This captures the idea that the reference dataset lacks coverage in that part of input space that matters most for detection via Confidential Guardian. We show empirical results for such shifts in Figure and observe that increased removal (i.e., + 1) hinders reliable detection of Mirage via Confidential Guardian. We note that even in the limit of complete removal of the uncertainty region (i.e., م = 1) the model still exhibits slight underconfidence. This likely because points just outside the uncertainty region also experience reduced confidence due the inherent smoothness of neural network prediction spaces.
D.3. Choice of
Calibration of probabilistic models is a well-studied area in machine learning, yet determining an acceptable calibration deviation threshold be far from trivial. Below, discuss several considerations that auditor may take into account when selecting this threshold.
1. IMPERFECT CALIBRATION IS THE NORM
In practice, perfect calibration is rarely, if ever, achievable. Even with standard calibration methods such as temperature scaling (Guo et al., 2017), there will typically be some small residual miscalibration, especially in regions of sparse data or for rare classes. Consequently, an auditor might set a non oonzeero a to allow for realistic margin that reflects typical model imperfections, for instance in the range [0.01, 0.03) for the expected calibration error (ECE).
21

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Confidence Distributions
Reliability Diagram
Confidence Distributions
Reliability Diagram
UTKFace (Fomite)
Other Uncert
UTKFaae (Asian) Denstty
Other Uhoert
Denely
Accuracy
Accuracy
0.5
2-
0.5
2-
Pert cal
Perf cal
Cal
0.2
0.6 Confidence
0.8
0.25
0:75
1.00
0.4 Confidence
1.0
0.00
0.25
0.50
0.75
1.00
Confidence
Confidence
Figure 11. Additional experiments on UTKFace with different uncertainty regions.
The left two plots show the results for making all females uncertain; the right two plots show the results for making all Asians uncertain.
Confidence Distribttions
Reliability Diagram
Adull (Ed=13 + Husi) Densiry
Confidence Distributions
Reliability Diagram
Adull (Bach Priv)
10
1.0
1.0
Donsity
Uncert
Anmrray
Uncert
Anmeracy
0.8
+
0.6
Potf oal
+
0.6
Porf cal
0.4
0.4
0.4
0.6
0,8
1.0
0.4
0.6
0,8
1.0
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
Figure 12. Additional experiments on Adult with different uncertainty conditions. The left two plots show the results for making individuals working a job the private sector with Bachelor degree uncertain; the right plots show the results for making husbands with more than 13 years of education uncertain.
D.3.2. DATA DISTRIBUTION AND DOMAIN KNOWLEDGE
The choice of may be informed by the following domain-specific factors:
Label Imbalance. Highly imbalanced datasets can lead to larger calibration errors for minority classes. Here, a
looser threshold a may be warranted, since small absolute deviation in the minority class could yield large relative miscalibration score.
Data Complexity. In high- ligh-dimensional or complex domains (e.g., images, text), calibration can be more difficult to achieve, suggesting more forgiving threshold.
Domain Criticality, safety-critical applications (e.g., medical diagnosis), stricter thresholds may be appropriate to ensure that predictions are suitably conservative and reliable.
D.3.3. REGULATORY GUIDANCE AND INDUSTRY STANDARDS
Some industries have regulations or recommendations regarding the safety margins for decision-making systems:
Healthcare. Regulatory bodies may require that model predictions err on the side of caution, translating to tighter calibration constraints (small or).
Financial Services. Risk-based models might have well-established guidelines for miscalibration tolerance, especially under stress-testing protocols. An auditor can rely on these to pick accordingly.
Consumer-Facing Applications. Standards for user-facing models (c.g., recommenders) may be more lenient in calibration, thus allowing for larger miscalibration thresholds.
D.3.4. ROBUSTNESS TO DATASET SHIFTS
A calibration threshold chosen solely one dataset might fail under distribution shift. An auditor might:
Evaluate calibration on multiple reference datasets (different time periods, different subpopulations).
22

---

CONFIDENTIAL GUARDIAN: Cryptographically Prohibiting the Abuse of Model Abstention
Confidence Distributions
Reliability Diagram
Confidence Distributions
Reliability Diagram
1.0
1.0
Ahut (>$200) Donstty
Other Uncert
Accuras
Adullt (x6%)
Donsity
20
Other
Uncert
Accuracy
0.8
+
0.6
Pert cal
10
++
0.6
Pert cal
Cal
0.4
0.6
0.8
1.0
0.4
0.6
0.8
0.4
Confidence
ConÉdence
Confidence
Confidence
Figure 13. Additional experiments on Credit with different uncertainty conditions, The left two plots show the results for making requests for loans bigger than $20,000 uncertain; the right two plots show the results for making loans with interest smaller than 6% uncertain.
Select an a that reflects performance under a variety real-worke conditions.
Consider applying domain adaptation robust calibration techniques, which might inherently increase acceptable to account for shifts.
D.3.5. BALANCING STATISTICAL SIGNIFICANCE AND PRACTICAL IMPACT
Finally, auditor should consider how interpret differences in calibration from statistical perspective:
Confidence Intervals. Compute calibration metrics (e.g., ECE) with confidence intervals. the model's miscalibration falls within the interval of expected variation, a higher may be acceptable.
Practicality vs. Accuracy. A small deviation in calibration might be practically insignificant if it minimally impacts downstream decisions. Auditors can incorporate cost-based analyses to weigh the trade-offs.
D.3.6. SUMMARY
When setting in practice, auditor might:
1. Conduct a baseline study calibration error on representative datasets after temperature scaling to quantify typical miscalibration.
2. Adjust for domain complexity and label imbalance, possibly raising a if the data or the domain are known to be inherently more difficult to calibrate.
3. Incorporate regulatory industry guidelines, if they exist, to establish upper bound allowable miscalibration.
4. Examine distribution shifts by testing on multiple datasets and setting a to ensure consistency across these scenarios.
5. Use statistical considerations (e.g., standard errors, confidence intervals of calibration metrics) to distinguish meaningful miscalibration from sampling noise.
In summary, choosing a is balance between practical constraints, domain-specifie considerations, and regulatory mandates. Auditors should be aware that the threshold for "acceptable" miscalibration is context-dependent, and overly strict thresholds may be infeasible, whereas overly lax thresholds might fail to ensure reliability and trustworthiness.
23


Reviews:
[{"review_id": "oxO4jCNjZm", "reviewer": "Reviewer_fNK9", "summary": "This paper puts forth the possibility of discrimination by denial or delay of services using model uncertainty or confidence. The paper proposes an attack dubbed Mirage which systematically reduces confidence in a predefined uncertainty region while maintaining accuracy.  Then the paper proposes a reference dataset based way with ZKPs for detecting the same.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for their strongly positive assessment of our paper and discuss their questions below:**\n\n> What are some other ways of denial/delay/abstention?\n\nIndeed, our work considers abstention using model confidence, i.e. thresholding the maximum softmax score (as we describe in the *Abstain Option* paragraph in Section 3, see Equation (2)). While this is the predominant default abstention option due to its ease of use and broad applicability, other ways of uncertainty quantification / abstention exist. We highlight some representative instances in the *Abstention mechanisms in ML* paragraph in Section 2. After having established the foundations of artificial uncertainty induction in this paper, future work should consider the impacts of Mirage and the effectiveness of Confidential Guardian as part of other contemporary uncertainty quantification approaches.  \n\n>What are some other ways of attacking with uncertainty?\n\nAs the reviewer correctly points out, there are multiple ways that an uncertainty attack could be realized. While we envision many such alternate approaches, we think the most interesting alternate formulation relies on modifying the *training data* instead of the loss function. In our formulation, Mirage is designed to reduce confidence as part of the loss computation which allows for precise control of a targeted confidence level by specifying a desired $\\varepsilon$. Alternatively, it would be possible to introduce uncertainty by flipping labels, injecting noise into the features, or deliberately undersampling certain parts of the dataset, and then training a predictive model with a standard cross-entropy loss on this modified dataset. This approach would work similarly to data poisoning but with the distinct goal of reducing confidence, not (just) predictive accuracy. While such an approach would likely be effective at decreasing model confidence (and other derived uncertainty metrics), we are confident that such an approach would be harder to tune to a particular targeted confidence level. Absent any loss modifications, the effectiveness of such an attack would depend on the degree of confidence the model already has on neighboring data points which might differ widely on a per-batch basis, leading to instability during training. Moreover, any non-traditional loss function adaptations would likely require a change in the amount of noise or data under-sampling which makes such an approach less flexible to use. Finally, it is unclear if such an approach would allow us to only decrease model confidence or whether it would lead to a decrease in both model confidence *and* model accuracy at the same time. Similarly to the point that the reviewer raised above, we think that this could be an exciting avenue for future work!\n\n**We hope that our rebuttal has addressed the reviewer’s questions and concerns and are happy to further engage with them during the discussion period.**", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper works at the intersection of security and abstention in ML. Additionally the detector works at the intersection of calibration and ZKPs. The attack is the most novel part of the paper; while other tools exist, the application is novel.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper looks at a kind of discrimination by denial or delay in service which has not been looked at a lot. I find this focus refreshing.\n2. The paper proposes an attack which systematically leads to reduced confidence for a target region while being accurate. I found this attack very interesting.\n3. A detection mechanism using ECE and ZKP was introduced. While both these are pre-existing, I found the application to be very novel.\n\nWeaknesses:\n\nThere are concerns over choice of reference dataset but the authors discuss this in the paper.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. The paper looks at abstention using confidence. What are some other ways of denial/delay/abstention?\n2. Suppose the malicious service providers are aware of your paper and hence do not abstent with Mirage. What are some other ways of attacking with uncertainty?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JDfDdceYTZ", "reviewer": "Reviewer_6g19", "summary": "This paper introduces \"Confidential Guardian,\" a novel framework that addresses the potential abuse of model abstention mechanisms in machine learning systems. The authors identify a critical threat: dishonest institutions can exploit model abstention to covertly discriminate against specific individuals by artificially inducing uncertainty in targeted input regions. To counter this threat, the paper proposes a two-pronged approach: (1) an uncertainty-inducing attack called \"Mirage\" that demonstrates the feasibility of the threat by deliberately reducing confidence in targeted regions while maintaining high accuracy elsewhere, and (2) a defense mechanism called \"Confidential Guardian\" that employs zero-knowledge proofs to verify that model abstention genuinely reflects inherent uncertainty rather than malicious manipulation. The framework enables external auditors to verify the legitimacy of abstention decisions without compromising model proprietary details. Empirical evaluations demonstrate that Mirage can successfully induce targeted uncertainty, while Confidential Guardian effectively detects such manipulations, providing verifiable assurances about abstention integrity.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for their positive assessment of our paper and discuss their questions and concerns below:**\n\n> The experimental evaluation could include more diverse application domains beyond image/tabular classification.\n\nWe chose to use image (high-dimensional, unstructured data) and tabular (structured) data as contrasting modalities that capture a range of real-world challenges. These experiments illustrate the effectiveness and generality of our approach. Extending the evaluation to additional modalities (e.g., text) could offer further evidence of generality, but would require additional adjustments to the problem formulation and is therefore beyond this paper’s scope. We welcome future research examining these extensions, and we also provide a discussion on regression below.\n\n> The computational overhead of the ZKP protocol may limit practical applicability in some settings.\n\nAlthough ZKPs add overhead for larger models, recent efficiency advances are notable. [Weng et al. 2021] achieves ZKP inference on ResNet-101 (42.5M parameters) in about 9 minutes, while [Sun et al. 2024] processes LLaMa-2-13B (13B parameters) in roughly 16.5 minutes—a 167× speedup. Moreover, startups like Polyhedra (see https://github.com/worldcoin/awesome-zkml for more examples) are rapidly advancing these techniques. As Confidential Guardian treats ZKP inference as a subroutine (line 7, Alg 1), its overhead will naturally decrease as the technology matures.\n\n> The discussion of how to set appropriate thresholds for detection in practice could be expanded.\n\nWe agree that appropriately determining the detection threshold $\\alpha$ is crucial in practical deployments. Appendix D.3 already provides an extensive discussion on this question, including a step-by-step guideline based on the data distribution, available domain knowledge, and existing regulatory standards. In short, we find that choosing $\\alpha$ is a balance between practical constraints, domain-specific considerations, and regulatory mandates.\n\n> How would Confidential Guardian perform against adaptive adversaries?\n\nWe agree that adaptive attacks are key to understanding the limits of our defense, which is why we added additional experiments in Table 2 in the appendix. As shown there, an informed attacker could try two strategies to improve Mirage’s evasiveness:\n- High $\\varepsilon$: Skews the output distribution more strongly toward the correct class, making high-confidence predictions less anomalous and harder to detect via calibration metrics. However, this increases overlap with legitimate data, making selective manipulation more difficult.\n- $\\varepsilon = 0$: Yields an unbiased uniform distribution, reducing confidence and accuracy jointly. While this degrades calibration-based detection, such an attack is easily identifiable with more traditional fairness evaluation metrics such as equalized odds.\n\n> How might the approach be extended to regression, generative models, or reinforcement learning?\n\nWe conducted a regression experiment following the reviewer’s suggestion: https://ibb.co/kVQKYHMh. We simulate a noisy regression task and train a shallow neural network to predict a Gaussian output distribution (left plot). To adapt Mirage, we designate $[-3,-2]$ as an artificial uncertainty region in which we inflate predictive variance while keeping the mean consistent. This is done by regularizing the output variance $\\sigma^{2}(x)$ towards a desired variance level $\\sigma^{2}_t$:\n\n$$\\mathcal{L}_{\\text{penalty}}(x) = (\\log \\sigma^2(x)-\\log\\sigma^{2}_t)^2$$\n\nThe right plot shows the resulting model with the uncertainty region. We will document this in more detail in a revised draft. Having validated Mirage in both classification and regression, we believe extensions to generative modeling are promising, though beyond this paper’s scope.\n\n>What are the practical considerations [...] particularly regarding computational overhead and integration with existing model serving infrastructure?\n\nConfidential Guardian is a method for detecting abstention abuse on a model during an offline phase conducted solely between the service provider and auditor. To extend its guarantees to the deployment phase, one could apply ZKPs of inference using the model committed during the audit. The specifics of such an infrastructure are an orthogonal problem which we place beyond the scope of our work, but approaches in [1] and [2] – which provide strong bounds on model properties by ZKP verifying only a small probabilistic sample of inferences – show promising scalability.\n\n[1] Franzese et al. “OATH: Efficient and Flexible Zero-Knowledge Proofs of End-to-End ML Fairness.” arXiv preprint arXiv:2410.02777 (2024).\n\n[2] Kang et al. “Scaling up Trustless DNN Inference with Zero-Knowledge Proofs.” arXiv preprint arXiv:2210.08674 (2022).\n\n**We hope that our rebuttal has addressed the reviewer’s questions/concerns and are happy to further engage with them during the discussion period.**", "claims_and_evidence": "The claims made in the submission are generally supported by evidence:\n\n1. The claim that abstention mechanisms can be exploited for discriminatory purposes is convincingly demonstrated through the Mirage attack, which shows how uncertainty can be artificially induced in specific regions without sacrificing overall accuracy.\n\n2. The claim that Confidential Guardian can detect artificially induced uncertainty is supported by empirical results showing its effectiveness at identifying calibration mismatches.\n\n3. The zero-knowledge proof protocol's ability to verify well-calibratedness while protecting model proprietary information is theoretically sound and demonstrated through implementation.", "methods_and_evaluation_criteria": "The evaluation methodology is sound, with appropriate metrics for both the attack and defense mechanisms.", "theoretical_claims": "I am not familiar with this specific area, and I will adjust my rating based on the feedback from other reviewers. If other reviewers possess more expertise in this field and provide more technically thorough reviews, their perspectives should take precedence over mine.", "experimental_designs_or_analyses": "1. The evaluation of Mirage attack effectiveness - This is well-designed with appropriate metrics and controls.\n\n2. The assessment of Confidential Guardian's detection capabilities - The methodology appropriately measures both false positives and false negatives.\n\n3. The analysis of computational overhead - This provides practical context for the approach's feasibility.", "relation_to_broader_scientific_literature": "The paper effectively positions itself within the broader literature on:\n\n1. Model abstention and uncertainty estimation \n\n2. Fairness and discrimination in ML systems \n\n3. Zero-knowledge proofs for ML", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "Strengths:\n1. The identification of abstention abuse as a novel threat vector is insightful and important.\n2. The dual contribution of both attack and defense mechanisms provides a comprehensive treatment of the problem.\n3. The integration of cryptographic techniques with machine learning is technically sophisticated and well-executed.\n4. The paper addresses a practical concern with real-world implications for fair and trustworthy AI.\n\nWeaknesses:\n1. The experimental evaluation could include more diverse application domains beyond image/tabular classification.\n2. The computational overhead of the zero-knowledge proof protocol may limit practical applicability in some settings.\n3. The paper doesn't extensively discuss potential limitations of the approach, such as scenarios where calibration analysis might fail to detect more sophisticated attacks.\n4. The discussion of how to set appropriate thresholds for detection in practice could be expanded.", "other_comments_or_suggestions": "Consider exploring how the approach might be extended to other types of model manipulations beyond uncertainty induction.", "questions_for_authors": "1. How would Confidential Guardian perform against more sophisticated adversaries who might attempt to evade detection by manipulating both the model's predictions and its calibration properties simultaneously? Understanding the robustness against adaptive attacks would help assess the long-term viability of the approach.\n\n2. The paper focuses on classification tasks. How might the approach be extended to other machine learning paradigms such as regression, generative models, or reinforcement learning? This would help understand the generalizability of the core ideas beyond the current implementation.\n\n3. What are the practical considerations for deploying Confidential Guardian in real-world systems, particularly regarding computational overhead and integration with existing model serving infrastructure? A clearer understanding of these practical aspects would help assess the feasibility of adoption in production environments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GmvG1fEIm8", "reviewer": "Reviewer_RTh6", "summary": "The paper introduces an attack and defense which target the confidence of model predictions. The attack, called Mirage, aims to reduce the confidence of a confidence-calibrated model for a targeted subregion of the data distribution, while maintaining high classification accuracy. The defense uses Zero Knowledge Proofs (SKP) to compute an Expected Calibration Error (ECE) that measures misalignment between the model’s accuracy and confidence. The experiments show that this is feasible for smaller models and datasets in classification settings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for their positive assessment of our paper and discuss their questions/concerns below:**\n\n> Reference [a]\n\nThanks for this reference! We have cited it and included a comparison in Section 2:\n\n*Both Mirage and the attack in [a] exploit abstention to reduce model availability in a targeted region, but differ in intent and actors. Mirage models institutional misuse (by the model owner) to deny service, while [a] involves third-party attackers aiming to increase institutional costs via fallback mechanisms. The methods also differ: Mirage targets regions in input space aligned with institutional incentives, whereas [a] uses artificial input triggers. As a result, Mirage induces abstention without modifying inputs, while [a] requires explicit alterations.*\n\n> Threat model assumptions.\n\nA model owner is generally interested in making predictions as accurately as possible over the entire support of the data distribution. While our calibration-based method may miss cases where both confidence and accuracy are reduced (e.g., $\\varepsilon=0$, yielding uniform predictions and label flips), such attacks are detected by accuracy-based auditing techniques from the fairness literature (e.g., equalized odds). To highlight this, we include extended results in Table 2 (appendix).\n\n> The proposed method only works for classification.\n\nPlease see our reply to reviewer 6g19 where we include a regression experiment showing that ideas used in Mirage can be extended to regression. Also, we note that converting UTKFace's age prediction task to classification is not an arbitrary choice but common practice [1].\n\n[1] Lowy, Andrew, Devansh Gupta, and Meisam Razaviyayn. \"Stochastic differentially private and fair learning.\" ICLR, 2023.\n\n> The method suffers from ZKP’s long computation times.\n\nWhile the computational overhead of ZKPs does impose practical limitations for applying Confidential Guardian to image data, tabular data is quite common in medical and financial applications, both of which are well-motivated settings for preventing abstention abuse and privacy-preserving audits. See our response to reviewer 6g19 for an extended discussion.\n\n> The implementation of the method is not available.\n\nWe will release our full code repository upon acceptance of the paper.\n\n> How to ensure the same model is used when making decisions & during audits?\n\nTo ensure the same model is used during deployment and audits, one can rely on existing ZKP-of-inference methods with cryptographic commitments—their binding property guarantees model consistency across stages. While the infrastructure for this lies beyond our scope, recent work shows promising scalability by verifying only a small sample of inferences [2].\n\n[2] Kang et al. “Scaling up Trustless DNN Inference with Zero-Knowledge Proofs.” arXiv preprint arXiv:2210.08674 (2022).\n\n> Could the protocol work without the ZKP?\n\nWhile our approach can run without ZKPs—Confidential Guardian could compute calibration metrics without verifying the forward pass—the guarantees become much weaker. Without ZKPs, the approach relies on the assumption that model owners report confidence values honestly. As the reviewer noted, this opens the door to model switching or fabricating confidence scores to suit institutional incentives.\n\n> Complexity of Alg 1?\n\nThe VOLE-based ZKP protocol has a complexity linear to the running time of the algorithm it proves [3]. In practice, the proof of inference computations are the runtime bottleneck. We will include a discussion of theoretical complexity and the key parameters affecting concrete overhead in the final version of the paper.\n\n[3] Weng et al. “Wolverine: Fast, scalable, and communication-efficient zero-knowledge proofs for boolean and arithmetic circuits.” S&P 2021.\n\n> L293 How to modify the protocol?\n\nThis change can be made via simple find-and-replace in our EMP implementation of the protocol. As the reviewer notes, this opens the door to manipulation, so it is only appropriate in less adversarial settings (see L299). For instance, in medical contexts, strong legal accountability (e.g., HIPAA) deters tampering, making it reasonable for the service provider to collect $D_\\text{ref}$. Combining this variant with privacy-preserving data provenance techniques [4] is a promising future direction.\n\n[4] Duddu et al. “Attesting Distributional Properties of Training Data for Machine Learning.” 2024.\n\n> Table 1: what does the “<333” and “<1.27GB” for CIFAR-100 mean?\n\nThe running time of that row is based on Mystique [5] who reported time for LeNet-5, ResNet-50, and ResNet-101, but not for ResNet-18. Thus, we use ResNet-50 as the upper bound for the running time.\n\n[5] Weng et al. “Mystique: Efficient Conversions for Zero-Knowledge Proofs with Applications to Machine Learning.” USENIX 2021.\n\n**We hope that our rebuttal has addressed the reviewer’s questions/concerns and are happy to further engage with them during the discussion period.**", "claims_and_evidence": "The paper’s claims are generally well supported, both theoretically and experimentally. In particular:\n\n1. The feasibility of the attack is demonstrated both theoretically by construction, as well as empirically using the proposed Mirage attack.\n2. The effectiveness of the defense is demonstrated experimentally, and the construction of the ZKP is provided as Alrogithm 1.\n\nThe following claim would benefit from more concrete evidence:\n\n1. L044: “Institutions often deploy cautious predictions (El-Yaniv et al., 2010) in real-world, safety-sensitive applications […]”. The cited references are academic papers - as far as I can tell none of them support the “real-world” claim of this sentence. Some references to such systems that are deployed in practice would better support this claim.", "methods_and_evaluation_criteria": "The proposed attack makes sense and successfully achieves the paper’s objective. The proposed defense has nice theoretical properties; however, its practical applicability is questionable due to its heavy computational and communication cost and therefore limited scalability.\n\nThe evaluation makes sense, as it includes results on both the attack and defense. The benchmarks include a simple toy dataset, several small-scale datasets with sensitive attributes, and small to medium scale computer vision datasets (e.g. CIFAR100). While these make sense to evaluate the proposed method, more large-scale datasets would strengthen the results.", "theoretical_claims": "I checked the proof for Lemma 4.1. It is generally plausible and makes sense, but I did not verify every detail of Appendix B.\n\nI attempted to verify the validity of the ZKP in section 5.2/Algorithm 1. The paper claims that “The security of our protocol follows directly from the security of our underlying ZKP building blocks ((Weng et al., 2021a), (Franzese et al., 2021)) which are secure under the universal composability (UC) model (Canetti, 2001).” Since I am less familiar with ZKPs, this is not obvious to me, and IMO requires more careful reduction to the underlying techniques, including listing the assumptions for each step.", "experimental_designs_or_analyses": "The experiments make sense and measure the success of the proposed attack and defenses. However, it is unclear if the authors used a held-out validation set for algorithm development and hyper-parameter selection.", "relation_to_broader_scientific_literature": "The paper is generally well positioned in the literature. It draws inspiration from several fields, including ZKPs, confidence calibration, and backdoor attacks. The discussion of related work includes label smoothing attacks, as well as ZKPs for neural network inference. I see a strong relationship to availability attacks (attacks that reduce/block the availability of model outputs), which should additionally be discussed (see below).", "essential_references_not_discussed": "There is some literature on backdoor attacks to reduce model availability, which is related to the concept of the attack presented here. Especially [a] has a very similar goal: reduce the (certified) robustness of a model around a targeted region of the data distribution, while preserving high accuracy. The “indirect” attack also works similarly by uniformly perturbing data labels. I don’t think a direct comparison is necessary, but a discussion of similarities and differences would strengthen the positioning of the paper.\n\n[a] Lorenz et al.: Certifiers Make Neural Networks Vulnerable to Availability Attacks. In AISec 2023.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is very well written and generally easy to follow. The structure makes sense and figures as well as tables are easy to comprehend.\n2. The paper not only presents a new attack, but also a corresponding defense. This is desirable to not only make people aware of the risks, but also present a potential solution.\n3. The proposed defense/audit works without revealing proprietary details about the model or training data.  \n\n**Weaknesses:**\n\n1. I am not convinced that the treat model is entirely realistic. It assumes that the model owner is interested in (i) high accuracy, but (ii) low confidence for some subset of the data. The proposed scenarios such as abstaining from credit scores would also allow for low accuracy and low confidence, which would not be detected by the proposed audit/defense.\n2. The proposed method only works for classification, which is not listed as a limitation. The authors even convert regression tasks to classification to make it work.\n3. The method suffers from the general limitation of ZKP that they require long computation times (333 sec per datapoint on medium-scale problems), combined with massive data communication (>1GB per datapoint). This makes the current setup infeasible for any practical application.\n4. The implementation of the method is not available for review; it would be beneficial to make it available and release it with the paper.", "other_comments_or_suggestions": "-", "questions_for_authors": "1. How do you ensure that the same model is used by the service provider when making decisions and during audits? As far as I can tell, there are no ZKPs involved in regular inference. Hence, the service provider could simply use a different, unmodified model to compute the ZKPs.\n2. Is my understanding correct, that the evaluation protocol could also work without the ZKP, if the auditor is given query access to the model? This may be a more realistic approach, even though the ZKP would ultimately be desirable.\n3. What is the theoretical complexity of Algorithm 1? It would be good to understand the key parameters, and how they influence the computation and communication cost. E.g., number of data points, model size, feature dimension, number of classes, etc.\n4. L293 How exactly would you modify the protocol? And how could the auditor ensure that the service provider doesn’t simply exclude the targeted group from $D_{\\mathrm{ref}}$?\n5. Table 1: what does the “<333” and “<1.27GB” for CIFAR-100 mean? Why the bound instead of concrete numbers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the potential adversarial vulnerability of the confidence of model predictions. The authors proposed both an attack and a defense for this setup. The attack, "Mirage", demonstrates that that manipulating the prediction confidence without degrading the predictive performance is practically feasible. The defense, "Confidential Guardian", employs zero-knowledge proofs to measure the misalignment between the model's accuracy and confidence. 

Reviewers agreed that the paper addresses a novel and practically relevant threat in machine learning applications. The effectivenesses of the proposed attack and defense are empirically verified with solid experiments. One major limitation is that the proposed defense method suffers from the high computational costs of zero-knowledge proof, which restricted its application to large-scale setups. However, overall this is a solid paper bringing novel insights.
Decision: Accept (poster)