Robust ML Auditing using Prior Knowledge
Authors: Jade Garcia Bourrée
Augustin Godinot
Sayan Biswas
Anne-Marie Kermarrec
Erwan Le Merrer
Gilles Tredan
Martijn de Vos
Milos Vujasinovic
Venue: ICML 2025 spotlightposter (2025)
Keywords: ML audit
ML theory
fairness
fairwashing
Primary area: general_machine_learning->evaluation

Abstract: Among the many technical challenges to enforcing AI regulations, one crucial yet underexplored problem is the risk of audit manipulation.
This manipulation occurs when a platform deliberately alters its answers to a regulator to pass an audit without modifying its answers to other users.
In this paper, we introduce a novel approach to manipulation-proof auditing by taking into account the auditor's prior knowledge of the task solved by the platform. 
We first demonstrate that regulators must not rely on public priors (e.g. a public dataset), as platforms could easily fool the auditor in such cases. 
We then formally establish the conditions under which an auditor can prevent audit manipulations using prior knowledge about the ground truth. 
Finally, our experiments with two standard datasets illustrate the maximum level of unfairness a platform can hide before being detected as malicious.
Our formalization and generalization of manipulation-proof auditing with a prior opens up new research directions for more robust fairness audits.

Robust ML Auditing using Prior Knowledge
Jade Garcia Bourrée 123 Augustin Godinot
1234 Sayan Biswas Anne-Marie Kermarrec
Erwan Le Merrer Gilles Tredan 6 Martijn de Vos Milos Vujasinovic
Abstract
(West, 2000). Given the far-reaching consequences of these applications, ensuring the fairness (Mehrabi et al., 2021) and regulatory compliance (Petersen et al., 2022) of such models is paramount.
Among the many technical challenges to enforc- ing AI regulations, one crucial yet underexplored problem is the risk of audit manipulation. This manipulation oceurs when a platform deliberately alters its answers to a regulator to pass an au- dit without modifying its answers to other users. In this paper, we introduce novel approach to manipulation-proot auditing by taking into ac- count the auditor's 's prior knowledge of the task solved by the platform. We first demonstrate that regulators must not rely on public priors (e.g.. a public dataset), as platforms could easily fool the auditor in such cases. We then formally establish the conditions under which an auditor pre- vent audit manipulations using prior knowledge about the ground truth. Finally, our experiments with two standard datasets illustrate the maximum level of unfairness a platform can hide before being detected as malicious. Our formalization and generalization of manipulation-proot auditing with a prior opens up new research directions for more robust fairness audits.
Independent fairness audits serve as a critical tool for as- sessing the fairness of ML models and ensure that model providers remain accountable the public (Birhane et al., 2024; Raji, 2024; Raji et al., 2022). As models are placed in production, auditors rely black-box interactions, where queries are sent to the model, and the responses are analyzed to identify potential faimess violations (e.g., see (Kim et al., 2019)). However, this reliance on black-box audits leaves the process vulnerable to manipulations by the platform, also known as fairwashing. Regulatory practices currently require auditors to notify platforms in advance of an dit. Platforms can thus strategically alter the model or its responses during the audit to create the appearance of fair- ness, effectively concealing underlying biases and unfair practices from the auditor while maintaining operational ef- ficiency for its users. Consider, for example, a social media platform that employs an ML model to moderate content, automatically removing posts deemed harmful mislead- ing. During fairness audit, the platform could deploy more lenient moderation model that appears unbiased, only to revert to a stricter, potentially biased version once the audit concludes, effectively concealing unfair treatment of certain user groups.
1. Introduction
Machine learning (ML) models are becoming central to nu- merous businesses, industrial processes, and administrations. Such models are being employed in high-stakes domains where ML-driven decisions can have profound impacts on individuals and communities (Rudin, 2019).
In fact, such discrepancies between a platform's behavior during audits and its real-world operations have been ob- served. Initially created as part of the Social Science One project, a data sharing program by Meta encountered ma- jor setback when consistency issues were discovered in the data provided to scientists (Timberg, 2021). Similarly, collaboration between Meta and independent scientists, studying the polarization effects of Facebook's recommen- dation algorithm, recently faced criticism over discrepancies found between the algorithm's 's behavior before and during the audit (Ribeiro, 2024). Academic studies show that fair- ness audits are easily manipulatable, whether the platform is required to prove its fairness through the release of a public dataset (Fukuchi et al., 2020), through the explana- tion of decisions (Aïvodji et al., 2021; Shamsabadi al., 2022; Le Merrer & Trédan, 2020; Aïvodji et al., 2019), or
For instance, financial institutions have been leveraging ML-driven systems to evaluate loan applications based on attributes like income, credit score, and employment history
"Equal contribution 'Université de Rennes, Rennes, France 2 Inria, Rennes, France 'IRISA/CNRS. Rennes, France *PEReN, Paris, France 'EPFL. Lausanne, Switzerland 'LAAS, CNRS, Toulouse, France. Correspondence to: Augustin Godinot <au- gustin.godinott inria.ft>, Jade Garcia Bourrée <jade.garcia bourree @inria.fr>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Robust ML Auditing using Prior Knowledge
through black-box query interactions (Yan & Zhang, 2022; Garcia Bourrée al., 2023; Godinot et al., 2024). The potential for manipulation underscores the need for more robust auditing strategies.
ML decision-making systems From feature transforms to specific business rules, modern MI. decision-making systems can be remarkably complex. We abstract all this complexity by modeling the entire system as function h : X + y (e.g., h can be a ML model). The set of pos- sible queries X' is called the input space, and the set of possible answers is called the output space. We consider binary classification problems, which in line with related work in the domain of MI. faimess analysis (Yan & Zhang, 2022; Godinot et al., 2024). Each query is associated with protected attribute a E A, which the platform is legally required not to discriminate against. Examples of such at- tributes include gender, age, or race and are typically defined by law. The platform has access to the protected attribute a either as a feature of the input space X' or by a proxy (e.g., looking at the name of the person to determine the gender). We define D the data distribution on X' x A. For any subset 5 C X' x A and protected feature value a E A, we will write S. = {x (z, (z, E S, a' a} and h(S) = (h(z) (æ, a) E S}. Throughout the paper, when it is clear from the context we will abuse the S notation: S will either be a subset of X, X x or X x A x.
This work presents a novel theoretical framework and a practical implementation for preventing manipulations by the platform. Our analysis starts from a simple observa- tion: auditors can readily collect labeled data, reflecting the platform's 's service from independent sources a common practice whose theoretical and empirical implications re- main unexplored. For example, in the moderation example discussed earlier, the auditor could have some undeniable evidence at hand, to confront the model under scrutiny, e.g., "A post with this content must pass the moderation filter, otherwise there is some bias on protected feature of the user profile". Thus, by incorporating this dataset, the auditor can independently verify the platform's 's responses, cross-referencing them against known ground truth labels. By combining black-box interactions with prior knowledge from the labeled dataset, our method enables more reliable detection of fairness violations while reducing the reliance on assumptions about the platform's is behavior. Specifically, we aim to answer the following research question: Can the auditor's 's prior knowledge of the ground truth prevent fairwashing in fairness audits?
This work analyses how the platform can manipulate its model to pass fairness audit, now define relevant no- tation for this. The space of models that the platform can implement is called the hypothesis space H. The loss func- tion L : H x (X x )) - R measures the discrepancy between predictions and ground truth values, For a given hypothesis E H. its expected loss over the distribution D is [hh,, D) a( [e(h(x)) ar, a)] with e is loss func- tion that quantifies the error of h((r) given a single input and its protected attribute a.
Our paper makes the following three contributions:
We introduce and analyze a new fairness auditing ap- proach for black- box interactions where the auditor has access to prior knowledge about the platform and the ML task (Section 3).
We theoretically analyze how much unfairness a plat- form can conceal given the auditor's prior knowledge. For any auditor priors, our results highlight the impor- tance of keeping the auditor's ' prior knowledge private (Section 3). For the dataset prior we introduce, we establish bounds on the concealable unfairness when the auditor prior remains confidential (Section 4).
ML auditing An ML audit is "any independent assess- ment of an identified audit target via an evaluation of artic- ulated expectations with the implicit or explicit objective of accountability" (Birhane et al., 2024). A ML audit in- volves three entities. The platform is the entity hosting the ML decision-makinn system. The users are those using the service hosted by the platform. The auditor is the entity conducting the audit to verify whether the ML model is com- pliant for all users. The auditor could be a state regulator, a consulting firm, even a group of users.
By simulating fairness audits on multiple tabular and vision datasets, we provide a more nuanced understand- ing of how our framework should be implemented. Our experiments offer insights into setting the detection threshold used to identify manipulations (Section 5).
Fairness metric In this work, we consider ML audits targeting the fairness of the studied system. Specifically, the auditor chooses a fairness metric and sends queries to the platform to determine whether the platform abides by their fairness criterion. Among all the (un- Mairnesssssss metrics, we study Demographic Parity (DP) (Calders et al., 2009), which is commonly used in the faimeess evaluation literature thanks to its simplicity. DP is defined as follows:
2. Background: Auditing ML Models
This work studies fairness audits of ML decision -eriiignnmmainggggg systems under manipulation by the model-hosting platform. We first formalize the decision- makinng system and then
introduce the dynamics of fairness auditing.
2

---

Robust ML Auditing using Prior Knowledge
in the analysis of faimess audits. We define an auditor prior as follows.
Platform
20 E X
Users D
&
(x))
Definition 3.1 (Auditor prior). The auditor prior is a set of models Ha C yx that the auditor can reasonably expect to observe given her knowledge of the decision task by the platform.
BEFORE THE AUDIT
audit sot S generation
hy(S) Am (S) manipulation
DURING THE AUDIT O
2
For example, in (Tan et al., 2018), the authors study fea- ture importance by training two models - one on a public dataset and another via distillation of the audited ML model - and comparing the resulting models. Using more theo- retical approach, Yan & Zhang and Godinot et al. explored the case of an auditor knowing the hypothesis class of the platform, i.e., Ho. H. Ajara et al. proposed to use an assumption about the Boolean Fourier coefficients of H. to construct Ha. Finally, Gareia Bourrée et al. and Shamsabadi et al. used side-channel access (e.g., an additional API or explanations) to the ML model define H. and derive guar- antees on the measured fairness. In Section 4, we introduce labeled dataset D. that the auditor will leverage define Ha. Definition 3.1 captures all of the situations above and allows to formulate general results about the problem of robust auditing.
Platform
s
Auditor
H a
Aim (S)
Hag Bm(() (S)
AFTER THE AUDIT
comparison
Figure 1. The auditing process as conducted by an auditor, which proceeds in three steps. The platform exposes a model hep to the users. To appear fair to the auditor while not deteriorating the utility for its users, the platform manipulates its answers on the audit set S.
P P(X,A)-D (h(X)= = 1|A 1)
p(h)- = - -P(,A) (h(X) = 1|A 0)
(L)
The auditing process The auditing process consists of three steps which we visualize in Figure I. Here, hp refers to the model that the platform exposes to its users (the top part of Figure 1) and hm refers to the model exposed to the auditor (bottom part of Figure 1). First, the auditor builds an audit set S C X' and sends the queries in S to the platform (step ①). The platform receives S all at once and computes the answers using its model hy. To appear fair if it not, the platform projects its labels (S) on the set F of fair models. This defines manipulated model hm and the answers hm (S) the platform will send to the auditor (step 2). The auditor receives (S) and exploits these samples to evaluate whether the platform is fair (hm E F) and honest (h2 = hm)), (step 3) Since the auditor does not have direct access to hp, they compare hm to their prior Ha to decide whether the platform honest or malicious. Thus, the auditor tests the two following properties of hm:
For platform, DP is the easiest metric to manipulate (Yan & Zhang, 2022; Ajarra et al., 2024) as it only depends on the outcome of the ML model and not on its perfor- mance on the different protected groups. Thus, a platform can artificially adjust outputs, e.g., providing more posi- tive outcomes for underrepresented group. To decide whether a platform passes the audit or not, the auditor builds an andit set S c X' x A and evaluates the plug- in DP estimator: S),,S) = pr Exes, I (h(x)) = 1} Sol Exes, 1 {M(xx) = 1). Based on p(h), we also define the set of fair models F {h E y -uhh) 0}.
1
3. Enhancing Black-box Auditing with Prior
Since a malicious platform can manipulate the DP metric with relative ease, the auditor has to find ways to prevent these manipulations (e.g., using a different metric) or to detect them. In this section, we explore the latter. To detect manipulations, the auditor must use prior knowledge about what constitutes a "likely set of answers" on its audit dataset S. Then, using this prior, they would be able to estimate the likelihood that the received set of answers It (S) has been manipulated.
Is the platform fair? hm - J
(2)
Is the platform honest? hm 2 Ho
(3)
For dataset priors (i.e., when H. is a ball, see Section 4), we draw F and H. in Figure 2. Given a model hm the fairness audit is equivalent checking if hm belongs the blue shaded area. In the example of Figure 2, the platform would be flagged as malicious as hm belongs to F but not to H..
3.1. Modeling the Auditor Prior
Previous work has demonstrated that prior knowledge is both a practical and an essential tool for auditing, yet the notion of auditor prior has not been explicitly leveraged
Online v.s. batch auditing Note that we assume that the platform receives all audit queries at once and that it is pos- sible to detect all the audit queries. In practice, the queries
3

---

Robust ML Auditing using Prior Knowledge
are usually issued online (that is, one-by-one) by the audi- tor, through web-scraping or through an API. Compared to online auditing, it is easier for the platform to manipulate an audit if it knows all the audit queries before having to answer. On the other hand, because the auditor has to send all their queries at once, they cannot use the answers of the platform to actively guide the generation of the audit ques- tions (e.g., as in (Yan & Zhang, 2022; Godinot et al., 2024)). Ultimately, our setting is built worst-case analysis of the auditing game for the auditor.
4. Using Labeled Datasets for More Robust Audits Against Manipulations
In an ideal, yet unrealistic audit scenario, the auditor would have access to non-manipulated answers from the original platform model 1p-- The prior Ho would then be the set of models that agree with these non-manipulated answers and would allow the auditor to detect inconsistencies between the original hp and manipulated hm models. Yet in general, the auditor does not have access such non-manipulated answers.
Auditing axioms To avoid trivial audits, we add two mod- eling assumptions. The first assumption ensures that the auditors' prior is correct so that honest platform does not appear as lying. The second assumption asserts that an audit is necessary, otherwise the auditor could directly conclude from his prior that the platform is unfair. That is to say, the auditor should never flag a honest platform malicious. In particular, the auditor must have a prior that is close to the ground truth. Those assumptions are expressed as:
As an alternative, we propose to study the use of a private (because of Theorem 3.2) dataset Da, collected by the au- ditor to construct the auditor prior Ma. This idea (coupled with an assumption on the hypothesis class) has been studied experimentally (Tan et al., 2018) but the more recent theoret- ical works robust auditing diverged towards studying pri- ors on the model itself rather than the data (Shamsabadi et al., 2022; Yan & Zhang, 2022; Ajarra et al., 2024). In the following, we define what a dataset prior is, and study the guarantees an auditor can achieve using this prior. Unless noted otherwise, in this section and in Section 5, Ha will denote the dataset prior.
hp E Ha
and
H.g NFFFFN #0. 0.
(4)
3.2. On Public Auditor Priors
Definition 4.1 (Dataset prior). Let Da = (X Aa, Ya) X X A x y be a labeled dataset the auditor has access to. The dataset prior H. is defined as the set of models that have a reasonable risk on Da-
A typical auditor proceeds in the following way. Upon examining a platform's 's model hm. the auditor must first understand the task addressed by hm and what constitutes a "good-performing model" on this task. In our moderation example, the auditor might try to look for public moderation datasets to test the performance of ht using a few examples. It might also look for publicly-availabe moderation models to compare their resulting input/output pairs with those of hm. Unfortunately, our first remark is that regardless of the prior the auditor might construct, if these models are public (or at least known by the platform), the platform will always be able to manipulate the audit:
Ha = { E y Loh
(5)
To test if the platform is honest, the auditor needs to verify whether hm E Ho, i.e., whether Lhhm, D&) < T. The risk threshold r thus plays a big role in the guarantees the audi- tor will be able to achieve. We discuss the impac of T in Section 4.2 and guidelines to set its value in Section 4.3, but first, we need to discuss the definition of optimal manipula- tion in Section 4.1.
Theorem 3.2. Assume the platform knows Ha, it can then always pick hm E (Ha to appear both fair and honest.
4.1. Optimal Manipulation
Given the audit set S and its model hyp the objective of manipulative platform is to create set of answers h (S) that appear fair to the auditor but also do not raise suspicions. Ideally, the platform would like to know the auditor prior Ha (see Theorem 3.2), but in the general case it cannot because it is not public information. As a consequence, the platform cannot directly optimize its answers to be expectable and fair. However, it still has cards up its sleeve; it already trained model h on a dataset D that is close to that of the auditor D...
Proof. First, recall that by definition the platform knows F. Assume that the dataset prior is public, the platform also knows Ha. Hence the platform can compute J n Ha. As by assumption, F n Ha + (Equation (4)), the platform can pick any model h. E FNH..
In the case of (Shamsabadi et al., 2022), the platform per- feetly knows H. (because the H., is coming from queries of its model) so the detector is subject this manipulation (called Irreducibility in the paper). In Yan & Zhang's 's work, H. is the hypothesis class H. of the platform, communicated to the auditor before the audit. Theorem 3.2 provides a novel view on the impossibility results that were later proved in (Godinot et al., 2024).
Thus, instead of searching Am in Ha n F, the platform can assume that its true model h, is expectable - that is, hp E Ha and try to find a fair model hm E F while flipping as few labels as possible from hy. Therefore, the
4

---

Robust ML Auditing using Prior Knowledge
To gain intuition about the proof, we represent the audit case for |S] = 3 in Figure 2. By definition of the dataset prior, Ha is a ball of radius т, centered on Ya. the labels given in the audit dataset D.. The manipulation of a model hp can be detected only if the resulting model is outside of Ha. as shown in orange on Figure 2. The probability of detection thus minus the volume of original models hp whose projection on F lies outside on Ha. This volume is highlighted in red in Figure 2. The detailed proof of Theorem 4.3 is deferred to Appendix A.
H
ha
d
Theorem 4.3 highlights two key parameters to the auditor's s success: the unfairness of the prior 5 = d(h,,:F) F) and the expectability threshold т. If the dataset prior is perfectly fair (i.e., 8 = 0), then the auditor has no chance to detect manipulated model as non-expectable (Puf = 0, Corol- lary A.5). On the other hand, Corollary A.4 proves that, if =
Figure 2. Representation of the auditor prior Ha. the honest plat- form model hp and corresponding malicious model hi on the fair FF plane. The red area represents the area where platforms optimal manipulations are detected as dishonest: they fall outside of the blue region of FF
8 then Pus = 1. Finally, in Corollary 4.4, we derive lower bound on Puf for the case 0 < 5 < r. We provide the proof of Corollary 4.4 in Appendix A.
optimal manipulation is the projection of hp on F:
Nmm = (hop)((hpp) = arg min d(h, hp))
(6)
hEF
Corollary 4.4 (Detection rate lower bound). If n. is even,
The distance d in Equation (6) is the value of risk L of h using the labels of hp as the ground truth. This scenario captures the fairwashing approach in (Aīvodji et al., 2021) in the context of explanation manipulations.
1 6 W,, T
82 72
(n-1)/2
(n
1
S Puf S 1.
4.2. Achievable Guarantees
Following the second auditing axiom formulated in Equa- tion (4), the original model of the platform is always ex- pectable, Le., hp E Ha. Thus, the manipulation detection test has false positives, and the main quantity of interest to the auditor is the manipulation detection rate.
4.3. Practical Considerations and Discussion
In practice, T is determined by the task difficulty, and the amount of data available to solve the task. One possibility to tune the value of r is to use the error rate of current state- of-the-art models that solve the task at hand minimum value. We empirically explore this option in Section 5.4. An alternative, if the auditor has the resources, would be to train a set of models on the task and use them to calibrate т. We leave further exploration of the calibration of I to future work.
Definition 4.2 (Detection rate). The probability Pay that the auditor correctly detects a manipulative platform with optimal manipulation is Puf= = P(h & Halhi E Ha).
Estimating or computing Puf requires the knowledge of the distribution of models in H.. Unfortunately, unless they have access to the training pipeline of the platform, this model distribution is inaccessible to the auditor. To overcome this issue, we make the assumption of an unin- formative prior: since the auditor does not know the model distribution in Ha, they must assume it uniform.
On the other hand, the value of 8 is determined by the audit set sampling procedure. In most cases, the audit set is sam- pled independently from pre-specified audit distribution. In this case, the value of ó is fully determined. To regain some control over 6, the auditor has to allow other audit set sampling strategies, at the expense of potential statistical bias in the fairness and accuracy estimations.
Theorem 4.3 (Prior- Uniform detection rate). Under the dataset prior of Definition 4. I with I. defined as the E2 norm, and the uninformative prior assumption, the probability that the auditor correctly detects a malicious platform trying to be fair is
Takeaway. The auditor can always calculate a priori the probability to correctly detect a malicious platform trying to be fair. This probability depends on the ratio between un- faimess of the auditor prior and the chosen risk threshold T, and depends the audit budget |S].
arcccss (/T))
821
(n- 1)/2
(0)d0 -
W.
?2
with 8 = d(ha.F), the distance of ha to F and Wn is the n-term of Wallis' integrals.
'eer our first axiom Equation (4), we have that S T.
5

---

Robust ML Auditing using Prior Knowledge
Honest
ROC Mitigation
Label Transport
Linear Relaxation
Threshold Manipulation
CelebA
ACSEmployment
Smi ling
Sigh.Cheekbones
GBDT
Log. Reg.
Concealable Unfairness
H
0.1
0.1
0.2
0.1
0.2
0.1
0.2
0.1
0.2
Detection Score
Figure 3. The concealable unfairness by the platform for different detection scores and manipulation strategies. We highlight this for features of the CelebA dataset (left) and for two different ML models trained on the ACSEmployment dataset (right). The horizontal red line indicates the DP the unfair model without manipulation.
5. Empirical Evaluation
is blurry. As input to a vision model, we use the image to predict one the associated attributes. The target attribute varies across experiments and will be specified accordingly. Demographic Parity is evaluated along the gender attribute given in the dataset. For the ACSEmployment dataset, we train Gradient Boosted Decision Tree (GBDT) and Logistic Regression (Log. Reg.) models, while for CelebA, we train LeNet convolutional neural network (Leeun et al., 1998). GBDT and Log. Reg. are trained using the default parame- ters of their respective implementations in SCIKIT-LEARN. Meanwhile, LeNet is trained irrespective of the target at- tribute using the Adam optimizer with a learning rate of 0.001, a batch size of 32, and for two epochs, which is sufficient for the model to converge on all features. The code to run the experiments available online.?
We now empirically quantify the extent to which the plat- form can manipulate the unfairness its ML model. To that end, we study the concealable unfairness: the maximum level of unfairness platform can hope to hide before being detected malicious. First, we evaluate the effectiveness of different manipulation strategies and determine the opti- mal one. Since any practical fairness repair method can be used as a manipulation methods, we ask (RQ1) What is the best manipulation strategy implementation (Section 5.3)? Then, we study the dynamics of the concealable unfairness when the audit budget [S] increases: (RQ2) Can the auditor always find audit budget that prevents the platform from hiding any unfaimesss i.e., that always allows to flag the platform if malicious (Section 5.4)?
5.2. Implementing Optimal Audit Manipulations
5.1. Experimental Setup
In practice, computing the optimal manipulation Itm p amounts to solving:
We conduct experiments on tabular and vision modali- ties. The tabular dataset comes from the ACSEmployment task for the state of Minnesota in 2018, which is derived from US Census data and provided in folktables (Ding et al., 2021). The objective of this task is to predict whether an individual between the age of 16 and 90 is employed or not. As input features of the model hp. we consider sev- eral attributes of the individual, including gender, race, and age. The fairness of the models is evaluated along the race attribute given in the dataset: one group consists of individ- uals identified as "white alone", while the other includes all remaining individuals.
(S) E arg min L(h, ((x, hn(x)) x I SJ) s.t. S) <T
We note that this problem is the same problem solved by in- nn-proceesing and post-processing fairness repair methods (Caton & Haas, 2024). Thus, ironically, computing the op- timal manipulation is equivalent to choosing the optimal fairness repair method. The only difference being on which set the fairness constraints and accuracy objectives are de- fined: the audit set S instead the training dataset. Thus, since any practical fairness repair method can be repurposed for manipulation, we adapted four classical fairness repair methods: ROC Mitigation (ROC) (Kamiran et al., 2012).
For the vision modality, we study CelebA (Liu al., 2015), which consists of images of celebrities along with several bi- nary attributes associated with each image, such whether the person in the photo is blond, smiling, or if the photo
See httpp:://g/ thub. www....cowoorrooooeeeee...e
6

---

Robust ML Auditing using Prior Knowledge
Honest
ROC Mitigation
Label Transport
Linear Relaxation
Threshold Manipulation
CelebA
ACSEmployment
Smil ling
Righ.cheekbones
GBDT
Log. Reg.
Concealable Unfairness
0.2
0.1
8000
0
2,000
4,000
0
2,000
4,000
2,000
4,000
2,000
4,000
Audit Budget
Figure 4. The concealable unfairness for different audit budgets (i.e., data samples from the labeled dataset). We highlight this for features of the CclebA dataset (left) and for two different ML models trained on the ACSEmployment dataset (right).
Optimal Label Transport (OT-L) (Jiang et al., 2019), Linear Relaxation (LinR) (Lohaus et al., 2020) and Threshold Ma- nipulation (ThreshOpt) (Hardt et al., 2016).
points differences between the two protected groups. Com- paring the concealable unfairness values with the DP of the most unfair honest model (red horizontal line), we observe that the manipulation strategies almost all able to totally conceal the original model unfairness. Then, focusing on the x axis, the difference in Detect(hm, S) between the different honest models highlights the impact the perfor- mance of the platform's 's model should have on the detection threshold т. In fact, depending on the dataset and on the model, Detect hooest, S) varies from ~ 0.1 to 0.2. In Section 5.4, we explore a solution to setup the threshold.
5.3. Quantifying the Concealable Unfairness For Different Detection Scores
We next quantify the amount of fairness that a platform can hide for the different manipulation strategies above. This concealable unfairness (hp, hm) is defined as the De- mographic Parity gap between the manipulated and honest models.
5.4. Dynamics of the Concealable Unfairness as The Audit Budget Increases
Du (hip fmm) = (i(hmm S) pppppp S)1
(8)
The probability of detecting manipulations (via the the de- tection score) should intuitively increase as the auditor gains access to larger number of data samples (i.e., has higher audit budget) since this allows for more accurate com- parison of hm with the data prior H.. In this experiment, we explore how well this intuition holds in practice. For this purpose, we fix the hyperparameters for each manipu- lation method by selecting those that result in the highest concealable unfairness for a given base model, as discussed in Section 5.3. Then, for each base model morgelttarget attribute pair, we determine the maximum concealable unfairness that platform score (see eq. 9) remains below the detection threshold. As proposed in Section 4 the threshold for each model is set to I, where & represents the maximum accuracy achieved when training set of models on the corresponding target. This process repeated for audit budgets ranging from 100 to 5, 000.
To decide whether the model observed during the audit is manipulated, the auditor has to decide whether hm E H. or not. To do so, the auditor estimates Lhh, Da) by computing detection score Detect((mm:5) S).
Detect(hm,, S)
Σ (M.W)SS
((mm()) fッ}
(9)
To build (hp, Am) model pairs, we consider manipulation methods among ROC, OT-L, LinR and ThreshOpt, varying hyperparameter values when applicable. In Figure 3, we plot the value the concealable unfairness (hp. hm)) against the detection score Detect(h (m,, S) computed by the auditor. We show the results of LeNet models trained on two CelebA targets (first and second subplots), and GBDT and Log. Reg. models trained on ACSEmployment (third and fourth subplots). The horizontal red lines indicates the DP of the most unfair model without manipulation.
achieve while ensuring that its detection
First, we observe that for all the datasets, the platform can conceal significant amounts of unfairness: from 10
The results of this experiment are shown in Figure 4. The two plots on the left display the results for CelebA using

---

Robust ML Auditing using Prior Knowledge
the same base model but different target attributes, while the two plots on the right show results for ACpmmpoyyent using the same target attribute but different base models. These results reveal two distinct cases. In the first case (CelebA Smi 1 ling in Figure 4), the concealable unfairness converges to zero as the audit budget increases. This is due to the low aleatoric uncertainty associated to the Smill 1 ing target. Since the task is easier, the accuracy range of models trained on Smi Smillse 1ing is narrower, leading to tighter de- tection threshold T. In the second case (all the other facets of Figure 4), the concealable unfairness remains nonzero despite an increasing budget.
inspiration from hypothesis testing (Si et al., 2021; Taske- sen et al., 2021; DiCiccio et al., 2020; Cen & Alur, 2024; Cherian & Candès, 2024; Bénesse et al., 2024), online fair- ness auditing (Chugg et al., 2023; Maneriker et al., 2023), and formal methods for fairness certification (Albarghouthi et al., 2017; Ghosh et al., 2021; 2022; Borca-Tascius et al., 2022). Beyond statistical methods, the work of Yadav et al. explore the role of explanations in the auditing pro- cess (Yadav al., 2022). Recent works also stress the importance of broadening the lens of algorithm auditing by incorporating user perspectives and sociotechnical factors (Lam et al., 2023; Deng et al., 2023), On another line of research, Confidential-PROFIITT and FairProof propose to integrate cryptographic techniques in cooperation with the platforms, to ensure the faithfulness of platform responses during audits (Yadav et al., 2024; Shamsabadi al., 2023; Waiwitlikhit et al., 2024); this is. however, more intrusive and technically restrictive, and thus awaits for adoption.
Furthermore, in many cases, even with a high audit budget, some increase of unfairness remains undetectable by the auditor. Consequently, the platform retains some capacity to conceal unfaireess even at high audit budgets. This stresses the hardness of the auditor's 's task in some configurations, and lead to negative answer to (RQ2). In that light, we also observe that -in response to (RQ1). the Linear Relaxation and ROC Mitigation manipulation strategies are the most effective for a manipulative platform.
Manipulating audits Manipulating fairness audits is an active area of research. Auditors can be fooled by biased sampling when the decision maker is allowed to publish a labeled dataset as proof of model fairness (Fukuchi et al., 2020). Adversarial attacks on explanation methods, such as LIME and SHAP, can be employed produce misleading interpretations of model behavior (Fokkema et al., 2023; Shamsabadi et al., 2022; Laberge et al., 2023; Slack et al., 2020; Anders et al., 2020; Aivodji et al., 2019; Le Merrer & Trédan, 2020). Platforms can also modify the output of their models to create the appearance of fairness without address- ing underlying biases (Yan & Zhang. 2022; Garcia Bourrée et al., 2023; Godinot al., 2024).. However, the challenge of designing audits that are robust advanced manipulation strategies remains open. The idea of using auditor prior knowledge that we formalize in this work has been implic- itly studied in different contexts. Based on active learning techniques work has studied how auditors could leverage knowledge about the hypothesis class (Yan & Zhang, 2022; Godinot al., 2024). In a more practical setting, Tan et al. studied using model distillation methods (Tan al., 2018) to use prior about the ground truth and hypothesis class (Tan et al., 2018).
6. Related Work
Fairwashing and rationalization Addressing fairness is- sues often requires compromising model performance for advantaged groups which can discourage companies from embracing fair training practices (Zietlow et al., 2022; Zhao & Gordon, 2022). Companies have two incentives to pay attention the impact of their system on society. The first incentive comes from regulatory efforts such the Algorithmic Accountability Act (AAA) (Congress, 2022) (US) and the Digital Markets Act (DMA) (Union, 2022) (EU) that impose fairness, transparency, and accountability constraints on large digital platforms. Yet, how to enforce these regulations is still an open problem (Crémer et al., 2023). The second incentive is public image. Since fairness, transparency and accountability are laudable goal, audits, investigative journalism and certifications (Costanza-Chock et al., 2022) should force companies to pay attention to these objectives. However, both incentives are external: the platform just has to appear fair, transparent and ac- countable. This rationalization risk has been studied in the context of explanations fairwashing (Aïvodji et al., 2019; 2021; Shamsabadi et al., 2022).
7. Conclusion and Discussion
We investigated, both theoretically and experimentally. the conditions under which an auditor can or cannot be manipu- lated when auditing with a prior. We introduced empirical method for tuning the manipulation detection threshold to maximize the auditor's 's probability of detecting malicious platforms.
Fairness auditing Fairness auditing evaluates ML mod- els to ensure faimess and accountability, often without ac- cess to proprietary model internals (Ng. 2021). This black- box auditing approach relies on querying the model and analyzing its outputs against pre-defined fairness metrics (Birhane et al., 2024; de Vos et al., 2024). Current attempts to enhance fairness audits with tangible guarantees draw
While our work offers regulators a framework for defend- ing against audit manipulations, the path to accountability
8

---

Robust ML Auditing using Prior Knowledge
extends much further. A significant gap remains between audit evaluations and the actual mitigation of identified is- sues (Raji et al., 2021; Mukobi, 2024). Moreover, one-time audits are inherently limited, as platforms can alter their models in harmful ways after the audit has concluded. Ad- dressing these challenges in future work will require the de- velopment of continuous or adaptive auditing mechanisms, potentially incorporating auditor priors, to ensure sustained accountability and fairness.
Aivodji, U., Arai, H., Gambs, S., and Hara, S. Characteriz- ing the risk of fairwashing. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, vir- tual, pp. 14822-14834, 2021.
Ajarra, A., Ghosh, B., and Basu, D. Active Fourier Auditor for Estimating Distributional Properties of ML Models, 2024.
Impact Statement
Albarghouthi, A., D' D'Antnni, L., Drews, S., and Nori, A. V. FairSquare: Probabilistic verification of program fairesss. Proc. ACM Program. Lang., [00PPSLA):0:188::30 2017. doi: 10. 10.1145/3133904.
This work provides both theoretical and empirical analyses of fairness audits in ML decision-making systems, with fo- cus on their vulnerability to strategic manipulations by plat- forms aiming to evade regulatory scrutiny. By demonstrat- ing how auditors' access to prior knowledge can enhance the robustness of black-box audits, we offer actionable insights for mitigating potential audit-a manipulations. Our findings have important implications for policymakers, auditors, and ML practitioners, underscoring the urgent need for rigorous auditing frameworks resilient to adversarial behavior.
Anders, C. J., Pasliev, P., Dombrowski, A.. Müller, K., and Kessel, P. Fairwashing explanations with off-manifold detergent. In Proceedings of the 37th International Con- ference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Ma- chine Learning Research, pp. 314 323. PMLR, 2020.
Bénesse, C., Gamboa, F., Loubes, J.-M., and Boissin, T. Fairnsss seen as global sensitivity analysis. Machine Learning, 1 11065))325--3222, 2024. ISSN 1573-0565. doi: 10.
The societal impact of this work is twofold. On the pos- itive side, strengthening the robustness of fairness audits promotes greater accountability for platforms deploying ML models in high-stakes domains such as finance and health- care. By mapping the risk landscape of audit manipulation, our approach advances the development of more trustworthy ML systems. However, we also draw attention the limita- tions of current audit practices, showing that over-reliance on public priors can be exploited by strategic actors.
Birhane, A., Steed, R., Ojewale, V., Vecchione, B., and Raji, I. D. Al auditing: The Broken Bus on the Road to Al Accountability. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 612 643, 2024. doi: 10. 109//ATML.5977..2222...0.
Corca-Tasciuc. G., Guo, X., Bak, S., and Skiena, S. Prov- able Fairness for Neural Network Models using Formal Verification, 2022.
Acknowledgements
This work of Martijn de Vos, Milos Vujasinovic, Sayan Biswas, and Anne-Marie Kermarrec has been funded by the Swiss National Science Foundation, under the project "FRI- DAY: Frugal, Privacy-Aware and Practical Decentralized Learning", SNSF proposal No. 10.001.796. Jade Garcia Bourrée, Augustin Godinot, Gilles Trédan and Erwan Le Merrer acknowledge the support of the French Agence Na- tionale de la Recherche (ANR), under grant ANR-24-CE23- 7787 (project PACMAM).
Buyl, M. and Bie, T. D. Optimal transport of classifiers fairness. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (cds.), Advances in Neural Information Processing Systems 35: Annual Con- ference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Calders, T., Kamiran, F., and Pechenizkiy, M. Building Classifiers with Independency Constraints. In 2009 IEEE International Conference on Data Mining Workshops, pp. 13- 2009. doi: 10.1109/CDMW.2009..
References
Aïvodji, U., Arai, H., Fortincau, O., Gambs, S., Hara, S., and Tapp. A. Fairwashing: the risk of rationalization. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learn- ing. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 161-170. PMLR, 2019.
Caton, S. and Haas, C. Fairness in Machine Learning: A Survey. ACM Comput. Surv. 56(7): 1177)::6::111::::: -16:338, 2024. ISSN 0360-0300. doi: 10. 145/3616865.
Cen, S. H. and Alur, R. From Transparency to Accountabil- ity and Back: A Discussion of Access and Evidence in AI Auditing, 2024.
9

---

Robust ML Auditing using Prior Knowledge
Cherian, J. J. and Candès, E. J. Statistical Inference for Fair- ness Auditing. Journal of Machine Learning Research, 25(149):1-49, 2024, ISSN 1533-7928.
Information Processing Systems 2021, NeurlPS 2021, De- cember 6- 14, 2021, virtual, pp. 6478-6490, 2021.
Fokkema, H., de Heide, R., and van Erven, T. Attribution- based Explanations that Provide Recourse Cannot be Ro- bust. Journal of Machine Learning Research, 24(360); 1-37, 2023. ISSN 1533-7928.
Chugg. B., Cortes- Gomez, S., Wilder, B., and Ramdas, A. Auditing fairness by betting. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans,
Fukuchi, K.., Hara, S., and Maehara, T. Faking faimesss via stealthily biased sampling. In The Thirty-Fourlee AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intel- ligence Conference, IAAI 2020, The Tenth AAAI Sympo- sium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 412-419. AAAI Press, 2020.
LA, USA, December 10 - 16, 2023, 2023.
Congress, U. Algorithmic accountability act of 2022. https: www.wwss.......o/. wongresss.ov/billll.. gov/bil1/ 117th-congress
2022.
Costanza-Chock, S., Raji, I. D., and Buolamwini, J. Who audits the auditors? recommendations from a field scan of the algorithmic auditing ecosystem. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1571-1583, 2022.
Garcia Bourrée, J., Le Merrer, E., Tredan, G., and Rottem- bourg, B. On the relevance of APIs facing fairwashed audits, 2023.
Ghosh, B., Basu, D.,, and Meel, K. S. Justicia: A stochastic SAT approach to formally verify fairesss. In Thirty-Fifth AAAI Conference Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Ar- tificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 7554-7563. AAAI Press, 2021.
Crémer, J., Dinielli, D., Heidhues, P., Kimmelman, G., Monti, G., Podszun, R., Schnitzer, M.. Scott Morton, F., and De Streel, A. Enforcing the digital markets act: institutional choices, compliance, and antitrust. Journal of Antitrust Enforcement, 1 (133:315-349) 202..
de Vos, M., Dhasade, A., Garcia Bourrée, J., Kermarrec, A.- Le Merrer, E.,, Rottembourg, B., and Tredan, G. Fairness auditing with multi-agent collaboration. In ECAI 2024, pp. 1116-113 11666111..111 IOS Press, 2024.
Ghosh, B., Basu, D., and Meel, K. S. Algorithmic fair- ness verification with graphical models. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022. Thirty-Fourtt Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Sympo- sium on Educational Advances in Artificial Intelligence. EAAI 2022 Virtual Event, February 22 March 1, 2022, pp. 9539-95499 AAAI Press, 2022.
Deng, W. H., Guo, B. B., DeVrio, A., Shen, H., Eslami, M., and Holstein, K. Understanding practices, challenges, and opportunities for user-engaged algorithm auditing in industry practice. In Schmidt, A., Väänänen, K., Goyal, T., Kristensson, P. O., Peters, A., Mueller, S., Williamson, J. R., and Wilson, M. L.. (eds.), Proceed- ings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI 2023, Hamburg, Germany, April 23-28, 2023, pp. 377:1-377:18. ACM, 2023. doi: 10.1 N0.1145/3544548.356 10266
Godinot, A., Le Merrer, E., Trédan, G., Penzo, C., and Taïani, F. Under manipulations, are some Al models harder to audit? In 2024 IEEF Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 644 664, 2024. doi: 10. (009///TML.9377.22244....
DiCiccio, C., Vasudevan, S. Basu, K., Kenthapadi, K., and Agarwal, D. Evaluating fairness using permutation tests. In Gupta, R., Liu, Y., Tang, J., and Prakash, B. A. (eds.), KDD '20: The 26th ACM SIGKDD Conference Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp. 1467-1477. ACM, 2020.
Hardt, M., Price, E.,, and Srebro, N. Equality of opportunity in supervised learning. In Lee, D. D., Sugiyama, M., von Luxburg, U.. Guyon, I., and Garnett, R. (eds.), Advances in Neural Information P Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3315- 3323, 2016
Ding, F., Hardt, M., Miller, J., and Schmidt, L. Retiring adult: New datasets for fair machine learning. In Ran- zato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural
Jiang, R., Pacchiano, A., Stepleton, T., Jiang, H., and Chi- appa, S. Wasserstein fair classification. In Globerson, A. and Silva, R. (eds.), Proceedings of the Thirty-Fifih Conference on Uncertainty Artificial Intelligence, UAI
10

---

Robust ML Auditing using Prior Knowledge
2019, Tel Aviv, Israel, July 22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pp. 862-872. AUAI Press, 2019.
August 6-10, 2023, pp. 1665-1676. ACM, 2023, doi: 10. 10.11455/50033535599544
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and
Kamiran, F., Karim, A., and Zhang, X. Decision theory for discrimination-avann classification. In 2012 IEEE/ /2th international conference on data mining, pp. 924-9294 IEEE, 2012.
Galstyan, A. A survey on bias and faimess in machine learning. ACM computing surveys (CSUR), 54(6):1-35, 2021.
Mukobi, G. Reasons to doubt the impact of ai risk evalua- tions. ArXiv preprint, abs/2408. 020660, 2024.
Kim, M. P., Ghorbani, A., and Zou, J. Multiaccuracy: Black- box post-processing for fairness in classification. In Pro- ceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 247-254, 2019.
Ng. A. Can auditing eliminate bias from algorithms?, 2021. Accessed: 2025-01-07.
Laberge, G., Aïvodji. U., Hara, S., Marchand, M., and Khomh, F. Fooling SHAP with stealthily biased sampling- In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
NIST. Nist digital library of mathematical functions, 2013. Release 1.0.6 of 2013-05-06.
Petersen, E., Potdevin, Y., Mohammadi, E., Zidowitz, S., Breyer, S., Nowotka, D., Henn, S., Pechmann, L., Leucker, M., Rostalski, P., et al. Responsible and regula- tory conform machine learning for medicine: a survey of challenges and solutions. IEEE Access, 10:58375-58418, 2022.
Lam, M. S., Pandit, A., Kalicki, C. H.. Gupta, R., Sahoo, P., and Metaxa, D. Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Adver- tising. Proc. ACM Hum.-Coo Stum-CComput. Interact., 7(CSCW2); 360:1-360:37, 2023. doi: 10.1145/3610209
Raji, D.,, Denton, E., Bender, E. M., Hanna, A., and Paullada, A. Ai and the everything in the whole wide world bench- mark. In Vanschoren. J. and Yeung, S. (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
Le Merrer, E. and Trédan, G. Remote explainability faces the bouncer problem. Nature Machine Intelligence. 2 (9):529-533) 539, 2020. ISSN 2522-5839. s42256-020-0216--.
doi: 10.1038/
Raji, I. D. The Anatomy of AI Audits: Form, Process, and Consequences. In Bullock, J. B., Chen, Y.-.., Himmel- reich, J., Hudson, V. M., Korinek, A., Young, M. M., and Zhang, B. (eds.), The Oxford Handbook of Al Gov- ernance, pp- O. Oxford University Press, 2024. doi: 10. V011099/Xx00xdb///77119977323229 0113.28.
Lecun, Y., Bottou, L.., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Pro- ceedings of the IEEE, 86(11):2278-2322 1998. doi: 10.1 10.1109/5.726791.
Li, S. Concise formulas for the area and volume of a hyper- spherical cap. Asian Journal of Mathematics & Statistics, 4(1):66-70, 2010.
Raji, I. D., Xu, P., Honigsherg, C., and Ho, D. Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, AIES '22, pp. 557-571, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978- 978-1-4500 978-1-4503-9247-L doi: 10. (0.114//554494:353441
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In 2015 IEFE International Con- ference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 3730-3738 IEEE Computer Society, 2015. doi: 10.1 10.11099CC..2055.225..
Ribeiro, M. H. Is Facebook "Standard" Algorithm Po- larizing? facebook-standart aaaaaddrd-aggorithm-oooooeo polarizing, 2024.
Lohaus, M., Perrot, M., and von Luxburg, U. Too relaxed to be fair. In Proceedings of the 37th International Confer- ence on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 6360-6369. PMLR, 2020.
Rudin, C.
Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206- 215, 2019.
Maneriker, P., Burley, C., and Parthasarathy, S. Online fairness auditing through iterative refinement. In Singh, A. K., Sun, Y., Akoglu, L., Gunopulos, D., Yan, X., Ku- mar, R.,, Ozcan, F., and Ye, J. ), Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining, KDD 2023, Long Beach, CA, USA,
Shamsabadi, A. S., Yaghini, M., Dullerud, N., Wyllie, S. C.,, Aivodji, U., Alaagib, A., Gambs, S., and Papernot, N. Washing the unwashable On the (im)possibility of fair- washing detection. In Koyejo, S., Mohamed, S., Agarwal,

---

Robust ML Auditing using Prior Knowledge
A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022.
Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
West, D. Neural network credit scoring models. Computers & operations research, 27(11-12::1311-11 2000.
Shamsabadi, A. S., Wyllie, S. C., Franzese, N., Dullerud, N., Gambs, S., Papernot, N., Wang. X., and Weller, A. Confidential-profitt: Confidential proof of fair training of trees. In The Eleventh International Conference on Learning Representations. ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Yadav, C.,, Moshkovitz, M., and Chaudhuri, K. XAudit A Theoretical Look at Auditing with Explanations, 2022.
Yadav, C.,, Chowdhury, A. R., Boneh, D., and Chaudhuri, K. Fairproof: Confidential and certifiable fairness for neural networks. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Si, N., Murthy, K., Blancher, J. H., and Nguyen, V. A. Testing group fairness via optimal transport projections. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9649- 9659. PMLR, 2021.
Yan, T. and Zhang, C. Active fairesss auditing. In Chaud- huri, K., Jegelka, S., Song, L., Szepesvári, C.,, Niu, G., and Sabato, S. (eds.), International Conference on Ma- chine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 14929-24962. PMLR, 2022.
Slack, D., Hilgard, S., Jia, E.,, Singh, S., and Lakkaraju, H. Fooling LIME and SHAP: Adversarial Attacks Post hoc Explanation Methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES '20, pp. 180-186, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978-144503-71100 10-0.
Zhao, H. and Gordon, G. J. Inherent tradeoffs in learning fair representations. J. Mach. Learn. Res., 23:57:1-57:266, 2022.
Zietlow, D., Lohaus, M., Balakrishnan, G., Kleindessner, M., Locatello, F., Schölkopf, B., and Russell, C. Lev- eling down in computer vision: Pareto inefficiencies in fair deep classifiers. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, Pp. 0040-10411. IEEE, 2022. doi: 10.1 111119//CVPP5222822222.00000
doi: 10. 10.1145/335507:333533..
Tan, S., Caruana, R., Hooker, G., and Lou, Y. Distill- and-Compare: Auditing -lock-Box Models Using Trans- parent Model Distillation. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES 18, pp. 303-310, New York, NY, USA, 2018. Associa- tion for Computing Machinery. ISBN 978-1-4503-6611-8. doi: 10. 12 1114/3378722132278772
Taskesen, B., Blanchet, J., Kuhn, D., and Nguyen, V. A. A Statistical Test for Probabilistic Fairness. In Pro- ceedings of the 2021 ACM Conference on Fairness, Ac- countability, and Transparency, FAceT '21, pp. 648- 665, New York, NY, USA, 2021. Association for Com- puting Machinery. ISBN 978-1-4503-8309-7. doi: 10.11457342188.34459277
Timberg, C. Facebook made big mistake in data it provided to researchers, undermining academic work. Washington Post, 2021. ISSN 0190-8286.
Union, E. Regulation (eu) 2022/1925 of the european par- liament and of the council of 14 september 2022 con- testable and fair markets in the digital sector (digital mar- kets act). https: / heereeurreee lex. europa. eu/ //// heg/2002//925// oj eng, 2022.
Waiwitlikhit, S., Stoica, I., Sun, Y., Hashimoto, T., and Kang, D. Trustless audits without revealing data or mod- els. In Forty-first International Conference on Machine
12

---

Robust ML Auditing using Prior Knowledge
Table Notations
HYPOTHESIS CLASS SET OF FAIR MODELS
Ha
SET OF EXPECTABLE MODELS
ha
GROUND TRUTH
DISTANCE BETWEEN THE GROUNDTRUTH AND THE SET OF EXPECTABLE MODEL.
hp
ORIGINAL MODEL OF THE PLATFORM
MANIPULATED MODEL OF THE PLATFORM INPUT SPACE
X
DATA DISTRIBUTION
SAMPLE FROM INPUT SPACE
Y
OUTPUT SPACE
Y
SAMPLE FROM OUTPUT SPACE
PROTECTED FEATURE
SAMPLE SPACE
SAMPLE
DIMENSION OF Z
A. Proofs and additional theoretical results
A X' x A x {0, denote the sample space, from which the auditor draws samples Z & (X, A, Y). The auditor sample the binary predictions Y {0, 1} from probabilistic classifier h X + [0. that assigns score h(X) the belief that sample with features X belongs to the positive class. It is assumed that X' R and A {[A,, A1]/An Ao, Ai E {0, 1]) {11,0. [0, (the one-hot encoding of the protected feature with two groups). We also assume that H., is an open set of Z.
As in (Buyl Bie, 2022),
We denote F the set of all score functions f f: X - {0, 1] that satisfy (PDP):
{ X - {0, 13 EE (Z)) 0.,)
with vk [2].9k
A E2[A,] 1, O,, vector of dy zeros.
Assuming that the predictions f|X randomly sampled from probabilistic classifier h(X), then the traditional fairness
notion of demographic parity (DP) is equivalent to PDP. But if y is not sampled from h(X) but instead decided by threshold, DPD is relaxation of the actual DP notion. That is to say, F is the set of all score functions that are fair regarding the demographic parity on A.
As F is the kernel of the linear transformation f Ez (o(2)f(X)) Fis hyperplane Z.
As F is a hyperplane of Z, it is dense or closed in Z.
A.1. Cases where F is dense in Z.
Lemma A.1. F dense 2, the auditor has a probability to detects as manipulated equals to zero.
Proof. If F dense in 2 then for every function f E 2, every open neighborhood intersects F. In particular, it always exists a model hm E F that neighborhood of h, and H.. In that case, htm is fair and expectable, so the auditor has probability to detects manipulated equals zero.
This case is a pathological case where the platform can still appear fair and honest. For the next theoretical results, we are interested in the case where F is closed Z.
A.2. Cases where F is closed Z.
F is hyperplan closed Z, has an empty interior (i.e. 0F 0) as its codimension In the following, we can thus use F instead of O.F, as both are equals.
13

---

Robust ML Auditing using Prior Knowledge
Similarly, can define the normal vector to F which is actually the vector that is used for all the projections we in this paper. In Equation (6), defined n php) (i.e. be is the orthographic projection of the expectable model hy in the set of fair models F).
Having an hyperplan lead to the natural definition of (hyper)eylinder, that we use in the following theorem. Definition A.2. A right cylinder C(H, B) is the set of all points whose orthographic projection on hyperplane H lies in set B with B a subset of the boundary of H. B is called the base of the cylinder.
Theorem A.3. The probability Puf that the auditor correctly detects a malicious platform trying to be fair is Ha n (.))
Proof. The auditor correctly detects a malicious platform trying to be fair if and only if the manipulated model is fair but not expectable. The manipulated model fair but not expectable and only if the orthographic projection he hp in F in H. n O.F Thus, the manipulated model fair but not expectable if and only if hp & C(F, Ho n @FF (following Definition A.2). As by assumption hp Ha (Equation (4)). it means that h Ha's (aFF Ha @F). The auditor correctly detects a malicious platform trying to be fair with probability P(Ha CCF, Ha O (H))
Theorem 4.3 is special case of Theorem A.3 with additional assumption. We now prove the main Theorem Theorem 4.3. Theorem 4.3 the uninformative prior assumption, the probability that the auditor correctly detects mulicious platform trying to be fair is
detection rate). Under the dataset prior of Definition 4.1 with L defined the lo norm, and
(
(11
82 72
(x-1)) 1)/12/2
sin (0)d0
W
with 8 F), the distance of ho to F and W,, is the n-term of Wallis' integrals.
Proof. As established in Theorem A.3, Pus P(Ha (CF, H. naF)
The probability P(Ha ACFF H. (H)) is the probability to be in the ball H. without the probability to be in the intersection between the ball H., and the cylinder C(F, H. n 0F). In the following, we denote V (T, 6) this quantity.
As Ha is ball, its volume is:
xn/2
y (T)
T( T(
with [(2) - -0 (NIST, 2013).
The volume of the intersection between the cylinder and the ball is the sum of the three following volumes:
the solid cylinder with height between - and 6
the spherical cap of Ho that is above the previous cylinder (i.e. the part of Ho with height between and r)
the spherical cap of Hg that bellow the previous cylinder (i.e. the part of Ha with height between -5 and 7)
According to (Li, 2010), the volume of each spherical cap is
T ( 110-11/22m arccos(6/7)
Veap (T,)
sin (0)d0
1
2
And the volume of the cylinder of height 26 is
y yeylinder (T,0) = 28Vbbll (V72 72 82)
n-
Thus,
V# (T,
(obal(()) -V (T, 8) peyllnder
(r,
14

---

Robust ML Auditing using Prior Knowledge
T"
1-(-1111 1)/2,n
(arccoa(6/r)
(V 82)mm1
(0nd 28"
m+2
m+))
I 2
2
According Theorem A.3, the probability that the auditor correctly detects malicious platform trying to be fair is P(Ha)CC(E Ha HHa) That is to say, is the ratio of V (r, 6) over Vilell
Pus = P(Ha)C((F (7F))H
V (T,
Vibul (7)
[(
π (n-1)/2
arccos(6/1)
(72 (82))(11111 1)/2
T( m++2
n( 1)/2
=1-2
sin"
26
((
122
T"
T( 2
Tn/2
arco0s(6/17)
26 (72
82)) 13)((-311) 13/2 I( (+t2
=1
sin (0)d
VT r 1
VT
7
r( ((
=1 2 T( T("+2)
a1/cos(5/T)
(0)d 8 (T2 - 822 11/2
-1)
The function I' can be written with Wallis' integrals as: Wa In the other hand,
/ D with Vn, W. = -
ssn" sis" (0)d0.
(72 82) (n- 11/2
(7 82) (n 1)/2
2 82
(n 12/1
72
82
(n 13/2
72
Thus, Paş- = (6/0c
sin (0)d0- - -1
62) (n 1)/2
Before dealing with this complete expression, we propose some particular cases that are easily interpretable.
Corollary A.4. Ha is a ball centered in the ground-truth ha that is tangent to F, then the auditor has probability one to correctly detect a malicious platform trying to be fair.
H. B(ha: A)= Pus
with 8 = d(ha, F), the distance of ha to F.
Proof. If Hg is tangent to F then = T. Thus, arccos(6/7) = arccos(1) = 0 and fo par)cos(6/7) (ind = 0.
Thanks to the formula of Theorem 4.3 with 6/T = 1, Puf = = (0 - 0) = 1. H. is tangent to F, Puf 1.
This corollary means that by reducing the threshold 7 to the minimal value (6), the auditor is sure to detect any manipulation of the platform.
15

---

Robust ML Auditing using Prior Knowledge
Corollary A.5. He is ball centered in the ground-trath hig that is fair, then the auditor has a probability zero correctly detect malicious platform trying be fair.
Ho = B(ha, T) A ho E OF = Pas =0 0.
Proof. If ha E OF then 5 = 0 and arccos(ö/r) arccos(() x/2 in the formula of Theorem 4.3. Thus, Pas
Wn (W, 0) = 0.
This last case is the case where the ha the auditor is fair. Intuitively, if ha is fair, half of the model that the platform can construct are naturally fair and the other half are naturally unfair. Thus, it very easy change from an unfair model to fair model without changing too much the honest model. Thus, detecting such manipulation is very hard for the auditor.
Now, we study the general expression of Puf in Theorem 4.3, In particular, we study lower bound of Puf to study when the probability is strictly positive.
Corollary 4.4 (Detection rate lower bound). If is even,
1 82
(n-1)/2
1
S Puf 1
Wn 72
Proof. Pus = =1 1 ( (parcco((6/7)) sin" (0)d0)
$ (1-
82 (n-1)/2)
Wn Jo (arccoo(6/7) (f) + x
sin" (0)d0
(arccoa(5/7)) (f) - (with even).
So, W. So
1 1n-1)/2
And Paf 2 1
W.,
Lemma A.6. The lower bound according to 8/T has two extremums that are for 5/T = 1 or 5/7 y with y =
vn-I
2
Remark. Note that y only depend on the dimension n and leads to 0 when n leads to infinity.
Proof. We define fn (the lower bound) s.t.
fn(d,T) = 8.
82
12/1
WAT
72
Change of variable = ((x) = Wn (1 x: 222)(n11112 1)/2 We are interested in cases where T > 6, i.e. 0 < x 1 Moreover, has an extremum iff f' somewhere in (0, 1].
Va E [0. (.l. = (1-111 x² 22) (n- 1)02 (n-1)x²(1 (2
= =(1 - x² 2) (1-3) + vn - lx 1)(x² - Vn - 1x 1x
Le. (x) = for the following elements:
-1 << 0
x
16

---

Robust ML Auditing using Prior Knowledge
E [0,1]
3-3
0
vn-
(if 2)
2
So f has two local extremums in [0, 1], one for and one for 7 vn+3 1--3--ee Vnn -
2
17


Reviews:
[{"review_id": "QRplKXusBw", "reviewer": "Reviewer_g7Ct", "summary": "The paper studies the problem of robust fairness auditing when the platform can manipulate the model during auditing. To address this problem, they propose to allow the auditor to have access to a set of labeled examples that are close to the prediction of the model before auditing. During the auditing, the auditor performs two tests: whether the audited model is manipulated (by checking whether the labeled examples are still close to the prediction of the model during auditing) and whether the audited model is fair. Theoretical analysis derives upper and lower bounds for the successful detection probability, under simplified assumptions of the closeness. The experimental analysis uses existing fairness repair methods as model manipulation strategies and quantifies the dynamics of conceivable unfairness as the auditing budget grows.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer g7Ct,\n\nThank you very much for taking the time to read and review our paper. We are delighted that you *enjoyed reading the paper* and that, as reviewer Ddwq and Q6QF, you *appreciate the importance of the discussed problem*.\n\nWe will answer your points on the security game formulation, threshold value, and alternative solutions, but first, we would like to address what we believe is a misunderstanding. You mention that *The method is essentially collecting predictions of the model before auditing*, whereas the method we instantiate in Section 4 is about collecting *ground truth* labels before the audit.\n\n**Security game**\nTo the best of our understanding, a \"Security by obscurity\" approach would try to hide the verification protocol hoping that the platform could thus not manipulate it. In our setting, the audit protocol is public: the platform knows what metric is measured and knows that the auditor will use a prior (e.g., ground truth labels in Section 4). What is private is \"the key\": the exact realization of the ground truth label, and thus the exact models in $\\mathcal{H}_a$. Similar audit protocols can be found in other domains such as accounting. The protocol is public (the company knows that the auditor will look at their records but not all because of resource constraints) but the exact transactions that are examined by the auditor are not known to the auditee beforehand.\n\n**Theorem 4.3 assumptions**\nSince the auditor has no prior bias or belief on the model used by the platform, following the notion of *uninformative prior*, we assume that the auditor considers all models in $\\mathcal{H}_a$ to be equiprobable. This assumption and the closeness axiom (i.e., the auditor prior is a *good* prior) are justified in lines 174-186 (1st column) and lines 245-248 (2nd column). We will update the Theorem 4.3 statement to include and better justify the uniform model distribution assumption. \n\n**Decision threshold $\\tau$ value**\nWe agree that the *threshold choice is also highly dependent on the task and data distribution*. It is important in practice for the auditors to understand how to choose $\\tau$ and this choice can be very much context- and application-specific. We describe exactly how it is chosen in practice in Section 5.4, lines 374-378. We agree that the paper would benefit from an earlier discussion of this setup. We will add it next to Definition 4.1.\n\n**Alternative solutions**\nFinding ways to *ask the platform to sign each of its predictions* is indeed a very active research area for manipulation proof auditing. In the related works we mention some early efforts in using cryptographic primitives for auditing (Yadav et al., 2024; Shamsabadi et al., 2023; Waiwitlikhit et al., 2024). However, beyond the extremely high computational and infrastructure cost of signing model predictions, our paper provides an alternative with a more learning theoretic view on the manipulation-proof auditing problem by deriving the audit guarantees from the auditor's expertise (i.e., prior).\n\nWe have corrected the typos you pointed out in the manuscript. We thank you again for your comments on our work and hope that we have satisfactorily addressed your main concerns. Should you have additional questions or remarks, we would be happy to answer them in the final discussion phase.", "claims_and_evidence": "1. Theorem 4.3. analyzes the detection success probability, yet it does not specify the randomness over which the probability is computed. Theorem 4.3 also requires some assumptions on the closeness and model distribution, yet these assumptions are not covered in the statement.\n2. Experiments in Sections 5.3 and 5.4 did not report how the closeness threshold $\\tau$ is chosen for detecting model manipulation. This is quite important as the results (Figure 4) could drastically change under a different threshold. The threshold choice is also highly dependent on the task and data distribution, which requires detailed justifications.", "methods_and_evaluation_criteria": "The proposed method, theoretical assumptions, as well as evaluation criteria are overly simplified. \n1. The method is essentially collecting predictions of the model before auditing, and using them to detect whether a model is manipulated during auditing. To tolerate the realistic settings where the auditor could not have precisely accurate predictions by the model before auditing, the authors make assumptions about how close the collected data are to the actual model predictions. Yet this assumption is highly application-specific. In the current form, the authors simply assume a fixed threshold for l2 norm or accuracy closeness, which doesn't reflect practical auditing applications. \n2. The assumption of the model owner not knowing the data prior is somewhat problematic, and resembles the line of thinking of \"security by obscurity\".", "theoretical_claims": "Proofs look correct yet the statement requires more clarifications. See Claims And Evidence.", "experimental_designs_or_analyses": "See Claims And Evidence and Methods And Evaluation Criteria.", "relation_to_broader_scientific_literature": "The paper revisited the data prior for robust fairness auditing.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "See Claims And Evidence and Methods And Evaluation Criteria.", "other_comments_or_suggestions": "I enjoyed reading the paper, and appreciate the importance of the discussed problem. However, I believe the approach taken in this paper requires more justifications in many assumptions: the closeness metric/threshold, knowledge of the platform about the data prior...See Claims And Evidence and Methods And Evaluation Criteria. Additionally, a discussion about (why not using) alternative solutions could be helpful, e.g., asking the platform to sign each of its predictions, thus providing proof that all predictions are produced by the audited model. \n\nMinor typo: Definition 4.2 $h^*_m\\in \\mathcal{H}_a$ should be $h^*_m\\notin \\mathcal{H}_a$", "questions_for_authors": "See Other Comments Or Suggestions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n5HSpBRR2h", "reviewer": "Reviewer_Ddwq", "summary": "This paper addresses a significant challenge in machine learning fairness auditing: the risk of manipulation (fairwashing) during audits. The authors introduce an approach to make audits more robust by incorporating the auditor's prior knowledge about the task. Through theoretical analysis and experiments, they establish conditions under which auditors can prevent audit manipulations and quantify the maximum unfairness a platform can conceal. The work formalizes how auditors can leverage their private knowledge through labeled datasets to detect platforms that artificially modify their models during audits to appear fair.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Ddwq,\n\nThank you for taking the time to read and review our paper. We are delighted that you found our work to be scientifically sound with a *thorough experimental design* and a *mathematically sound theoretical framework* that also *effectively positions itself within the fairness auditing literature*.\n\n**To be or not to be a groundbreaking new direction**\nWe believe that our work is particularly relevant and of interest to the ICML community because: \n- Existing manipulation-proof auditing/fairwashing literature all implicitly use this notion of auditor prior. We formalize it, derive general theorems (Section 3), and introduce a new (and *natural*) type of prior (Section 4). See for example [Yan et al.](https://proceedings.mlr.press/v162/yan22c/yan22c.pdf), [Aivodji et al.](https://proceedings.mlr.press/v97/aivodji19a.html), [Shamsabadi et al.](https://proceedings.neurips.cc/paper_files/paper/2022/hash/5b84864ff8474fd742c66f219b2eaac1-Abstract-Conference.html) or [Yadav](https://proceedings.mlr.press/v235/yadav24a.html).\n- The ML security community is well versed in this type of security game formulation but they try to solve them with cryptographic primitives. We believe that the theoretical ML community can also bring nice audit guarantees. Hence, our auditor prior formulation opens up an avenue to explore better and potentially more complex notions of auditor priors.\n\n**Standard fairness repair methods**\nAs manipulation strategies, we adapted fairness methods based on their applicability to our setting (binary classification, access to sensitive attributes, and compute overhead) and choose the best performing ones. If you have a specific more innovative or up-to-date method, we would be glad to try it in our experiments before the end of the discussion phase.\n\n**Further discussion on formal verification of fairness**\nFormal verification of fairness methods (see Section 6) adapts classical verification frameworks (SSAT, bound propagation...) to fairness auditing. To the best of our knowledge, all require a white box access to the model. Thus, while they are very useful for model providers, existing formal fairness verification methods are not applicable in our setting where the auditor only has black box access.\n\nWe have corrected the typos you pointed out in the manuscript. We thank you again for your comments on our work and hope that our contributions are now clearer. Should you have additional questions or remarks, we would be happy to answer them in the final discussion phase.", "claims_and_evidence": "The key claims and supporting evidence include:\n\n1. Public priors are insufficient. The authors theoretically prove that if the platform knows the auditor's prior, it can always manipulate the audit to appear fair and honest.\n2. Private dataset priors improve detection. The paper demonstrates both theoretically and experimentally that auditors can prevent manipulations by maintaining a private labeled dataset.\n3. Quantifiable manipulation protection. The authors derive a mathematical framework to calculate the probability of detecting manipulations based on the auditor's dataset and its relation to the fair model space.", "methods_and_evaluation_criteria": "The paper's methodological approach is competent but lacks novelty:\n\n1. The probability bounds for detecting manipulation are well-derived.\n2. The experiments on tabular data (ACSEmployment) and image data (CelebA) are thorough. I question whether the amount of data in the datasets is sufficient. And whether using only the \"concealable unfairness\" metric throughout the paper is adequate.\n3. Practical implementation: The four fairness repair methods adapted as manipulation strategies are all pre-existing techniques in the literature. It would be more convincing if there were newer methods.", "theoretical_claims": "The theoretical framework is mathematically sound:\n\n1. The formalization of the auditor's prior knowledge (Definitions 3.1 and 4.1) builds directly on established notions in the literature.\n2. The relationship between prior knowledge and detection probability (Theorem 4.3) is predictably based on the basic principles of hypothesis testing and geometric interpretations of fairness.\n3. The bounds on detection probability (Corollary 4.4) are well-derived.\n\nThe description related to theoretical claims is abstract, making it difficult for readers to understand the content of the formulas.", "experimental_designs_or_analyses": "The experimental design is thorough but conventional:\n\n1. The datasets used (CelebA and ACSEmployment) are standard benchmarks in fairness literature\n2. The model types evaluated (GBDT, Logistic Regression, LeNet) are commonly used in comparable studies\n3. The manipulation strategies tested are adaptations of existing fairness repair methods\n\nThe experiments align with expectations based on the theoretical analysis but do not provide any surprising or novel insights into fairness auditing. The models implemented in this paper are relatively early, classic works; incorporating more innovative or up-to-date approaches would be preferable.", "relation_to_broader_scientific_literature": "The paper effectively positions itself within the fairness auditing literature, acknowledging prior work on fairwashing through biased sampling, explanation manipulation, and black-box interactions. The authors clearly articulate their novel contribution: introducing a theoretical framework for incorporating private prior knowledge to prevent manipulations.", "essential_references_not_discussed": "The paper covers the most relevant literature but could benefit from discussing:\n\n1. Additional work on formal verification of fairness guarantees beyond what's mentioned\n2. Recent advances in interactive auditing approaches that might complement their dataset prior approach", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses a practical and important problem in ML ethics and regulation.\n2. The theoretical analysis is rigorous and provides useful bounds.\n\nWeaknesses:\n\n1. The paper assumes the auditor can collect a representative labeled dataset, which may be challenging in practice.\n2. The theoretical analysis primarily focuses on demographic parity; extensions to other fairness metrics would strengthen the contribution.\n3. The discussion of practical implementation in regulatory contexts could be expanded.\n\nMinor:\n1. Line 78: \"where where\"\n2. Line 181: \"a a\"\n\nWhile technically sound, the paper represents an incremental advance rather than a substantial novel contribution to fairness auditing research. Leveraging private knowledge to improve audit robustness is a natural extension of existing work rather than a groundbreaking new direction.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iUe0TRNGWo", "reviewer": "Reviewer_Q6QF", "summary": "Audits have been historically impactful in AI and they are increasingly becoming a common part of proposals for regulating it. however, it could be possible for developers to game audits so that the model seems to behave much better on the evaluation than real-world cases. This paper discusses this problem, raises the alarm about gaming publicly available evaluations, and formally describe the conditions by which audits can be manipulated. Key to this paper is introducing the concept of an auditor prior which describes what companies think about how their systems are going to be evaluated.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Q6QF,\n\nThank you very much for taking the time to read and review our paper. We are delighted that you found such value in our work and agree that it is a very real problem that developers can predict evaluations.\n\n**How about more general evaluations?**\nIn this paper, we chose to focus on demographic parity because it can be very easily manipulated by a malicious platform (answering uniformly at random gives perfect demographic parity). We are encouraged by the results in this difficult setting. Therefore, testing other evaluation metrics in the manipulation-proof framework (and comparing the brittleness of the metrics we currently use) is, in fact, our end goal.\nAbout your question on the title and prior knowledge, we would like to refer you to our answer to Reviewer g7Ct (Security Game paragraph) for more clarification.\n\n**Discussion on additional references**\nWe thank the reviewer for pointing to the work of [Mukobi](https://arxiv.org/abs/2408.02565), we were not aware of it. The other papers discuss different shortcomings of current ML evaluation methods (in terms of safety evaluations for [Casper et al.](https://dl.acm.org/doi/abs/10.1145/3630106.3659037) and capacity evaluations for [Raji et al.](https://arxiv.org/abs/2111.15366)).\nWhile we did include a reference to another study on how these evaluations are conducted ([Birhane et al.](https://ieeexplore.ieee.org/document/10516659)), we agree that this discussion on the gap between evaluations and mitigations deserves more attention. Thus, we will add the references you mentioned along with the discussion to the Related Works section. \n\nWe thank you again for your positive evaluation of our work and hope that we have addressed your concerns. Should you have additional questions or remarks, we would be happy to answer them in the final discussion phase.", "claims_and_evidence": "I would've been happy with this kind of paper as a position paper that didn't do any experiments. The fact of this paper did any experiments at all reflects a relatively thorough approach to the type of work that they're doing here.", "methods_and_evaluation_criteria": "I like the experiments. I think it's smart how they quantified concealable unfairness.", "theoretical_claims": "I have not checked all of the details and verified that the math works out myself. But the theory here is in service of the experiments which are done well. One could also argue that the theory here has the primary purpose of being illustrative since it's based on assumptions. There will be fundamental limitations in how easily real world phenomena can be captured with simple models like this. I think this is a no way of weakness of the paper -- this is me just saying that the theory's role in their arguments is illustrative rather thanbeing the core.", "experimental_designs_or_analyses": "The authors focus on evals for fairness. This seems fine to me, but I'm not sure why they don't take us slightly more general approach and make this paper about evaluations in general of fairness or other properties.\n\nI like how the authors did not just focus on instances in which developers know the actual evaluations because they are based on publicly available resources. Focusing on the more general case is great because it's very realistic. It's a very real problem that developers can predict evaluations, even without knowing what they are exactly.", "relation_to_broader_scientific_literature": "Overall, I think this is a great paper. As a reviewer, I would be willing to go to bat for it. I found it personally clarifying, well executed, and of clear value to the literature. I can easily see myself citing this paper.", "essential_references_not_discussed": "- https://arxiv.org/abs/2408.02565\n- https://dl.acm.org/doi/abs/10.1145/3630106.3659037 \n- https://arxiv.org/abs/2111.15366", "other_strengths_and_weaknesses": "Overall, I really like this paper. I think it should be accepted. My one hangup about it is that I think it would be somewhat better if it were possible to make this paper about evaluations in general rather than just evaluations of fairness. One other small thing is that the title kind of confuses me since the prior knowledge is the problem rather than the thing that makes an auto robust.", "other_comments_or_suggestions": "I would recommend trying to move figure one up in the paper to be on page one or two.", "questions_for_authors": "None to add. Please respond to points above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received diverse scores, ranging from "Weak Reject" to "Strong Accept," indicating a significant disagreement among reviewers.
A central discussion point was the extent to which practical implications should be required.
Some reviewers expressed concerns that the experimental setup and theoretical framework were too simplistic and might lack relevance to real-world problems.
Other reviewers claimed that the paper makes a valuable contribution to the field of AI auditing, with strong practical relevance.
All reviewers agreed that the authors should provide a more detailed discussion of the limitations of their theory and experiment.

I, as AC, think that the paper has the potential to open a new subfield in auditing.
Thus, detailed real-world practicality will be explored in future work.
We might not need to demand practical, real-world level insights in this initial paper.
Decision: Accept (spotlight poster)