Size-adaptive Hypothesis Testing for Fairness
Authors: Antonio Ferrara
Francesco Cozzi
Alan Perotti
André Panisson
Francesco Bonchi
Venue: NeurIPS 2025 poster ()
Keywords: Fairness
Fairness Metrics
Hypothesis Testing
Bayesian Statistics
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We develop hypothesis tests for fairness metrics for small and large subgroups

Abstract: Determining whether an algorithmic decision-making system discriminates against a specific demographic typically involves comparing a single point estimate of a fairness metric against a predefined threshold. This practice is statistically brittle: it ignores sampling error and treats small demographic subgroups the same as large ones. The problem intensifies in intersectional analyses, where multiple sensitive attributes are considered jointly, giving rise to a larger number of smaller groups. As these groups become more granular, the data representing them becomes too sparse for reliable estimation, and fairness metrics yield excessively wide confidence intervals, precluding meaningful conclusions about potential unfair treatments.

In this paper, we introduce a unified, size-adaptive, hypothesis‑testing framework that turns fairness assessment into an evidence‑based statistical decision.
Our contribution is twofold.  (i) For sufficiently large subgroups, we prove a Central‑Limit result for the statistical parity difference, leading to analytic confidence intervals and a Wald test whose type‑I (false positive) error is guaranteed at level $\alpha$.  (ii) For the long tail of small intersectional groups, we derive a fully Bayesian Dirichlet–multinomial estimator; Monte-Carlo credible intervals are calibrated for any sample size and naturally converge to Wald intervals as more data becomes available. 
We validate our approach empirically on benchmark datasets, demonstrating how our tests provide interpretable, statistically rigorous decisions under varying degrees of data availability and intersectionality.

Size-adaptive Hypothesis Testing for Fairness
Antonio Ferrara C CENTAI, Turin, Italy Graz University of Technology, Austria antonio. www.rraareeaaai..e...m
Francesco Cozzi C
Sapienza University, Rome, Italy CENTAI, Turin, Italy f. cozzi @uniromal.
Alan Perotti
André Panisson CENTAI, Turin, Italy
Francesco Bonchi
CENTAL, Turin, Italy alan. waa.pprotti@ccenai.cco
CENTAL, Turin, Italy EURECAT, Barcelona, Spain bonchi@centai.eut
panisson@centai.cm
Abstract
Determining whether algorithmic decision-maie making system discriminates against a specific demographic typically involves comparing single point estimate of a fairness metric against predefined threshold. This practice is statistically brittle: it ignores sampling error and treats small demographic subgroups the same as large ones. The problem intensifies in intersectional analyses, where multiple sensitive attributes are considered jointly, giving rise to a larger number of smaller groups. As these groups become for reliable estimation, and fairness metrics yield excessively wide confidence intervals, precluding meaningful conclusions about potential unfair treatments.
granular, the data representing them becomes sparse
In this paper, we introduce a unified, size-adaptive, hypothesis-testie testing framework that turns faimess assessment into an evidence-basee statistical decision. Our con- tribution is twofold. (i) For sufficiently large subgroups, we prove Central- Limit result for the statistical parity difference, leading to analytic confidence intervals and Wald test whose type-I (false positive) error guaranteed at level a. (ii) For the long tail of small intersectional groups, we derive a fully Bayesian Dirich- let-multinomial estimator; Monte-Carlo credible intervals are calibrated for any sample size and naturally converge Wald intervals more data becomes avail- able. We validate our approach empirically on benchmark datasets, demonstrating how tests provide interpretable, statistically rigorous decisions under varying degrees of data availability and intersectionality.
Introduction
As machine learning (ML) systems play an increasingly central role in decision-making in conse- quential domains ranging from education and hiring to healthcare and criminal justice concerns over algorithmic fairness have gained prominence, prompting the development of various fairness metrics [19, 29, 12, 41] and intervention strategies [46, 24]. Early proposals framed the problem as ensuring statistical parity across protected demographic groups defined by single sensitive attribute such as race or gender. Demographic Parity [19], Equalized Odds and Equal Opportunity [29], Predictive Parity [12], and related criteria remain cornerstones of the field and are implemented in popular toolkits (e.g., FAIRLFARN (44]). These metrics are ordinarily reported pointwise, and the decision to classify an observed metric's 's value potential issue is addressed by defining threshold: everything below this threshold is ignored, while everything above this threshold is treated as an issue [4]. Such threshold- based approach ignores the statistical uncertainty inherent in estimating population parameters from finite samples, treating small and large populations equivalently [36].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Minor disparities in large groups might unjustly be overlooked, while larger disparities in smaller groups may not provide sufficient statistical evidence for meaningful conclusions [23, 33].
The issue of uncertainty of fairness measures is further exacerbated by intersectionality an essential concept indicating the unique disadvantages faced by individuals belonging to multiple protected groups [14]. In intersectional settings, numerous subgroups emerge as multiple sensitive attributes are considered jointly, As these groups become more granular, the data representing them becomes too sparse for reliable estimation, and fairness metrics yield excessively wide confidence intervals, precluding meaningful conclusions about potential unfairness [45, 23]. This phenomenon produces a sort of "resolution limit" of intersectionality: as we show in our empirical assessment (Section 4) using the standard datasets commonly employed in fairness research, even the intersection of just two sensitive attributes can create conditions where existing methodologies appear to detect unfair treatment, when the uncertainty on the adopted metric actually so large to make any unfairness claim statistically not significant.
This observation prompts a critical research question: At what granularity should fairness be rig- orously guaranteed, and how can we decide this in a principled, data-driven manner? Specifically, there is need for principled methodologies that identify meaningful balance between the compre- hensiveness of intersectionality and the statistical robustness of faimness assessments [30, 34).
Our contributions. In this work, we provide principled methodology to measure fairness while accounting for uncertainty and group size. Specifically, we propose a theoretically grounded fairness measure based on hypothesis testing, and we introduce two tests to detect fairness violations. The first test relies on the asymptotic theoretical behavior of statistical parity. In particular, we demonstrate the asymptotic normality of statistical parity, which allows the construction of confidence intervals and tests for fairness violations. Secondly, we propose a test based on Bayesian inference, which particularly useful when intersectional groups are very small and asymptotic assumptions do not hold. We further provide empirical evidence showing that the Bayesian estimator converges to the theoretical asymptotic behavior. Additionally, we highlight the pitfalls of current fairness measures, especially in the context of intersectional groups. We show that existing metries can incorrectly detect unfairness in cases where no unfairness exists, and conversely, fail detect statistically significant discriminatory behavior when it is present.
How our work advances beyond prior research. Kearns et al. [33] account for the size of intersectional groups by scaling fairness metrics such as statistical parity - by multiplying the subgroup. However, to determine whether a group is
metrics by the probability of belonging treated unfairly, one still needs to rely on predefined threshold. Kim et al. [34] use Bayesian models to augment labeled data with unlabeled data, producing more accurate and lower-variance estimates. Unlike our work, they focus on unlabeled data and do not provide asymptotic results or hypothesis model distribution; however, unlike us,
tests. Foulds et al. [22, 23] employ a they still rely on thresholds to detect which group is treated unfairly and do not provide hypothesis tests for fairness violation. Besse et al. [6. 7] compute the asymptotic distributions of the ratios of certain fairness metrics. We generalize their results to a broader class of fairness metrics and, in contrast to their approach, we also address and connect our findings to the case of small intersectional groups where the aargessample assumption does not hold.
more exhaustive coverage of related literature is presented in Appendix A.
2 Problem Definition
Our objective is to move beyond the arbitrary nature of hand-picked thresholds by introducing rigorous, uncertainty-aware test for detecting group-level discrimination. To set the stage, we first formalize our notation, then assess existing threshold-based criteria, and finally, we present and motivate our alternative approach rooted significance testing.
Notation and preliminary detinitions. Let X' denote the input space and y (0, 1} the binary label space. For each individual æ E X, yetyy E y represent the ground truth label, and let f(x) E y denote the predieted label given by deterministic classifier f X - v. A prediction f(x) 1 corresponds to a favorable decision. Protected attributes are encoded by a space of p discrete values
S S1 x x Sp. Each S S represents the description of an intersectional subgroup and S(X) the population of the subgroup. We also denote Š for the complement of S in S.

---

Definition 1 (Statistical parity, or SP). For any subgroup S and classifier f we measure disparate positive-decision rates via
SP(S) = P(f(z) = E - - S(x))
(1)
Perfect statistical parity corresponds to SP(S) = 0.
We note that use 5 the reference group, following common practice. Alternatively, one could use the entire population as the reference group; however, the differences are typically minimal, especially when S represents small intersectional group.
Definition (6-Statistical parity, or 6SP). predictor J satisfies statistical parity at level ó when
AA E S:
[SP(S) 6
(2)
A threshold A is typically chosen (such the Equal Employment Opportunity Commission "four- fifths rule" [28], where 0 = 0.2), and unfairness is identified when [SP(S)) 0. However, this approach does not account for sample variability, treating subgroup of size 10 the same one with 10,000.
To partially compensate for heterogeneous support, Kearns et al. [33] propose Definition (7-Statistical parity, or 7SP).. predictor f is -tatiissicall Parity subgroup fair if:
AS E [SP(S) Pp(x E 5(x)) 57
However, in practice, to identify unfairly treated groups we have to choose threshold 0 and claim fairness violation when [SP(S) PP S(x) > 0. Moreover, as Gohar and Cheng [27] observe: "The second term reweighs the difference by the proportion of the size of each subgroup in relation to the population. Consequently, the unfairness of smaller-sized groups down-weighted in the final Stattisticaa Parity estimation. Thus, it may not adequately protect small subgroups, even they have high levels of unfairness."
From thresholds to significance. Both (2) and (3) require the practitioner to fix global threshold 0 without guidance on how adapt it subgroup size estimation variance. Motivated by the shortcomings above and by recent calls for uncertainty-awant auditing [18, 42, 39], we recast fairness checking as a statistical hypothesis test and propose the following definition:
Definition (Statistical Parity violation (this work)). Given significance level a E [0, 1] and a test statistic T with null hypothesis Ho SP(S) = O, we say a classifier J violates statistical parity for subgroup if the null hypothexis Hg is rejected at level a..
Hence fairness is assessed by whether the observed disparity could plausibly arise from sampling noise. Crucially, the decision threshold is no longer arbitrary: it is determined by the chosen test and a, automatically adjusting to the subgroup's sample size.
3 A Statistical Testing Framework for Size-Adaptive Fairness Assessment
In this section we operationalise our guiding idea: replacing ad-hoe thresholds with statistically principled tests for group-level discrimination that remain valid even when intersectional subgroups are small.
We do so by proposing two tests to detect statistical parity violations: (i) a large-sample Wald test that leverages an analytic asymptotic variance, and (ii) Bayesian small-sample test that yields finite-sample credible intervals via Monte-Carle Catee-Crro draws from Strichlet-multioonee posterior. Both deliver p-value (or posterior tail- polbbabbbbilitt that can be compared with user-selected significance level a, automatically adapting to subgroup size and sampling noise.
This unified framework yields size-adaptive, uncertainty-aware fairness conclusions, removing the need for arbitrary global thresholds while retaining statistical guarantees for both common and rare intersectional groups
3

---

3.1 Large-sample test: asymptotic normality of (P)
For an intersectional group let us define the following probabilities:
Po.s (zf 0, S(X)), Pı, P(f(()) 1, S(X)),
Po,5 ((f()) 0, S(x)), P1, P(f(z) 1. =(x))
Furthermore, let define the marginal ps = P(x S(X)) and p5 P(x e S(X)).
For an i.i.d. test sample (x:) we consider the following plug-in estimator of statistical parity:
(P)(S) Ex=1 [a)==Es))
El=1 ((.)=1,
E (,,((())
E 1.,)(())
where denotes the indicator function. Furthermore, let V (Ds) [ps)
po.5
-PI.8 Po.s P1.9
and
[pe)² 2 (pg)
P1.s(1 Pi,s)
1.
Po,
P
Pa,s(1 po,s)
P0.SP1,5
Pu.sPu.5
Σg
(4)
1,,
P1,s(1 - P1,s)
PP1.5Pu.5
P0,3
PPo,SPo.5
-P0.5P1.5
Po.5
The following result (proof in Appendix B) provides the asymptotic distribution of the statistical parity.
Theorem 1 (Central Limit Theorem for Statistical Parity). Let (e) E,V, where V and Σ1 are defined above. Then
(P.(8)-SP(S) 3 N(O,1), as + 000 (S))
where 4 denotes convergence in distribution and N 1) indicates the standard normal distribution.
From Theorem we obtain the (1 (1-a) a)) two-sided Wald confidence interval:
[SP, +
( + 1-1(1111110 (1-2) -
and the corresponding p-value p = 2(1 Φ SP,, (S)) where +(-) is the cumulative distribution function of the standard normal distribution and (.) the quantile. We reject the null "no disparity" whenever < a. Because a oSS) depends on population probabilities, we estimate it by plugging empirical counts into Equation (4).
3.2 Bayesian inference for small samples
Theorem describes how confidence intervals for SP(S) can be computed with large samples. In the particular case of small intersectional groups this assumption is, general, met and the theoretical asymptotic estimations of (P) and Σ4 might not be reliable. We thus move our attention deriving credible intervals via Bayesian inference.
Prior, likelihood and pasterior. We consider the probability space over Po,s, P1,S, Pa,s, and P1,s- assume that observations
Since the Pi,j. i E {0, 1).j E () mutually exclusive, are drawn from categorical distribution. Hence, given trials, let ni i E {0,1).] E {S.5} denote the counts for each outcome, with Ei, Ri.j n. ni,j follows multinomial distribution: (no,s, So 11 n1.s) ~ Multinomial(n, Po.s Po.s: P.....P...PP.... One can consider Dirichlet prior over Pi,z: (po,s, Po.5, P1.s. Pi.sP. ~ Dirichlet(n, ag,s, 0g 0,,,, s1,)). By conjugacy, the posterior distribution remains Dirichlet with updated parameters:
ao, ao,s (po.s. Po.s, P1,s. P1.s) data hugp://.......ta.....oo.oaaaaaa.
(5)

---

Posterior draws and credible interval. We are now interested in the posterior estimate of (PSS.. Since there is known closed-form expression for the posterior distribution of SP(S). we can approximate it using Monte Carlo sampling.
The procedure is as follows. Sample K times from the posterior, let the k-th sample be: (k) (k)
P...s.o..PPP.. (k))
for k = 1....,,,,, K, and let the k-th estimate of SP(S) be:
(K)
(p( = (R)
Pi.$
Pi.
(K)
(b)
(R)
Po.s + Pi.
Po.s
+
P1.
Then:
An unbiased estimate of the posterior mean of SP(S) is given by:
E(SP(S) data]
sp())
. The (1 a:) credible interval is given by:
CS) @ ((SP()) (S)) ((SP()())))) @ ((S)()
where the empirical-quanttie at level u is defined by:
T ((SP(( (S)) = infft: 1 K
Σ (SP()) 5 2 m
The interval reflects posterior uncertainty over the statistical parity difference given the observed data and prior. We can, hence, perform two-sided hypothesis test for the null Ho SP(S) = from the credible interval. Specifically, we reject the null hypothesis at level if the value 0 is not contained in the (1 a) credible interval.
Choice of prior. We adopt weakly informative symmetric (flat) Dirichlet prior, i.c. with concen- tration parameters (1, 1, 1, 1). by default; domain knowledge (e.g. historical results), when available, can be injected via the prior parameters. We explore the possibility of incorporating additional information into the prior in Appendix C.7.
Statistical Fairness Testing Framework. Algorithm 1 summarizes our rigorous statistical frame- work for fairness assessment, which specifically designed to handle varying subgroup sizes in intersectional settings. It integrates large-sample hypothesis testing with Bayesian inference, ensuring that fairness evaluations remain reliable even when data availability differs across subgroups. dynamically adjusts significance thresholds, thus accounting for statistical uncertainty and preventing misleading conclusions about bias.
3.3 Resolution limits for statistical parity f'airness violation
Auditing rare intersectional subgroups naturally raises the question: how small is too small? Even with uncertainty- tests, a subgroup composed of only a few samples may fail to provide statistically conclusive evidence of bias, regardless of how pronounced the observed disparity might be. Our framework addresses this challenge by computing, for each overall negative prediction probability P(f(z) 0) and protected group size ng Ro,5 n1,s, the minimum fraction of negative outcomes in S required to reject the null hypothesis Hu : SP(S) = 0 at a specified significance level a.
Figure visualizes this boundary when evaluating the hypothesis that group is being disadvantaged (the dual figure showing the boundary for deciding if group being advantaged, is reported in Figure 5 of the Appendix). The plot has on the y-axis the fraction po.s of Os predicted in the given group, while on the r--axis the size ny of the group. The plot reports 9 different lines corresponding to the fraction of Os predicted in the overall population. The figure uses 0.05.
5

---

Algorithm Size-Adaptive Fairness Testing (SAFT)
1: Input: Subgroup S, significance level a, minimum support ñ (we use n=30)
2: Output: Fairesss decision (Reject Hg Fail Reject Ho)
3: Compute n. i E {0, 1 E {S.S}
4: if min(no.s R1,5, n0.9, 11,5 2 ñ then
5: apply the Wald test of $3.1 (large-sample regime)
6: else
7: use the Bayesian test of $3.2 (small-sample regime)
8: end if
9: Compute statistical parity measure SP(S)
10: Report point estimate (S) or posterior mean
11: Report (1 - ar) confidence/credible interval
Report p-value or posterior tail probability
13: if then
14: Reject Ho [Fairesss violation detected)
15: else
16: Fail to reject Hp (No statistically significant violation]
17: end if
10
6.8
Protectd Croup (Aus)
6.6
Froctin of in
0.6
0.0 012345
10
80
Protected
Figure 1: Resolution limits for Statistical Parity violations under varying global negative rates P(f(x)) = 0), when detecting disadvantaged groups (the dual figure showing the boundary for deciding if group is being advantaged, is reported in Figure 5 of the Appendix). Each curve traces the minimal fraction negative outcomes needed to reject Ho: SP()) = of the group size n.. To the left of each vertical bar is the "no-power" zone, where subgroups too small to detect discrimination, regardless of the observed disparity. The shaded region above each curve is the "discrimination zone", where the subgroup's negative rate is enough to establish statistically significant parity violation.
= 0.05 as a function
pair (ns, po,s) needs to be above the line to indicate significant disparity. For instance, the point (10, 0.6) on the green line (P(f(x) 0) = 0.3) indicates that, when the overall population there 30% of Os predicted, demographic subgroup of size 10 needs have at least 6 individuals predicted O, in order to have a negative discrimination case. When in the overall population there 40% of Os predicted (red line), group of size 10 requires 8 negative cases have unfairness. When instead the prediction in the overall population is balanced (purple line, P(f(x) = 0) = 0.5) subgroup of size 10 requires at least negative cases have a legitimate discrimination complaint.
We can observe that, as the global negative rate P(f(x) = 0) increases, the required subgroup size n, grows steeply to evaluate disadvantaged groups. For example, when 90% of the population is predicted negative, a group needs to be composed of at least 35 individuals, all predicted negative, to have any statistically valid unfairness, highlighting the difficulty of auditing under severe class imbalance.
By charting these boundaries, our framework highlights the resolution limits of intersectionality. while providing practitioners tool to determine exactly how much data each intersectional slice needs to be worit, preventing spurious signalling of faimess violations small subgroups.

---

3,4 Generalisation to other fairness measures
Analogous results can be derived for other fairness metrics. Indeed, holds the general form of the theorem (we defer the proof to Appendix B):
Theorem 2. Let p (pi..... P4) set of probabilities of disjoint events, with Σ Ps 1; let C' = (CI,..., Cy) ~ Categorical(p) be the related categorical distribution, and ler C ,.... C be
i.idd realizations of C. Let & be a continuuusly differentiable (in an neighborhood of functiio
IR9 + R. Then
Vn
(((C) $. (),
Φ
C*)
00,
n
where VVT EV, V = V(o(E)) is the gradient of Ss and E (diag(p) pp"] is the covariance matrix of C, where diag(p) indicates a diagonal matrix with entries p.-
For example, considering Equal Opportunity:
EO(S) = P(f(x) = - I S(X)) P(((z)) 1- a S(x))
Theorem 2 can be applicd by choosing P1 = P(((x) 0, S(x) 1). = =[f(x)) x,
((X) = 1). Ps = ((f(2) 0, x E S(X) = 1), p4 = P(f(z) 1, S(X) = 1), and Φ(x1, 22, 24) #1 +-22
From Theorem 2, one can also obtain equivalent results to [6, 7] (but with simpler computation of the covariance matrix). Indeed, for example, the asymptotic behaviour of the Disparate Impact assessment:
DI(S) P(f(x) P([(x) = E (
x E S(x))
can be obtained by choosing P1 P(f(z) 0, E S(x)), P2 P(f()( =0 1, S(X)), P(f(x)= = 0, x E (())), p4 P 1, S())), and ((21, 22, Fz. x1) = 2t
22
The results for the Bayesian test can be extended under analogous conditions. Indeed, one can consider general probability space over p (PI..... Pq) of disjoint events. Consider n trials and indicate with (n1. ... n2y) the counts for each outcome. The likelihood again Multinomial p). and with a Dirichlet prior over p with concentration parameters (a)...., og), by conjugacy we obtain a posterior distribution distributed as Dirichlet(n, 01 + -........ 0gg ng)- One can then consider a fairnsss metric Φ R9 + IR and via Monte Carlo sampling can construct a (1 a:) credible interval and hypothesis test for Φ, analogously to the procedure used for (P).
In general, our framework can model all those fairness metrics based on the confusion matrix and related conditional probabilities.
Experiments
In this section, we empirically validate our size-adaptive testing framework on two canonical faimesss benchmarks under increasingly fine-grained intersectional slices.
4.1 Datasets and Models
We selected two standard datasets used in fairness benchmarks: the Adult Income dataset [5] and the COMPAS recidivism dataset [2]. We include additional experiments on two other datasets in the Appendix. We apply standard data preprocessing and train XGBoost classifier g X - )' (training details are in Appendix C.2). To examine intersectional effects, we first audit non-intersectional subgroups (race, age and sex) and then all two-way and three-way intersections of these protected attributes.
'COMPAS from propublica ccompas-scores-too-ye Adult from fairleam. originally from UCL.
7

---

The COMPAS dataset is widely used in the study of algorithmic fairness and risk assessment in the criminal justice system. It is often cited in fairness literature due to observed disparities in prediction outcomes across racial groups, most notably between Black and White defendants [2]. includes information about criminal defendants, such as the number of prior convictions and charge degree. The protected variables are race AAfrican-American", "Asian", "Caucasian", "Hispanic". "Native American", "Other"), age ("Under 25", "25-45", "Over 45"), and sex ("Female", "Male"). The binary target variable indicates whether a defendant is predicted to reoffened within two years (y= = 0) or not (y 1).
COMPAS
Adult The Adult Income dataset is based on census data, which includes 14 categorical and numerical features (e-g. education, marital status, occupation, etc.). The protected variables are race Semer-Indian--skingg
"Black", "Other", "White"), age ("Under 18", "18-40", "40-65", "Over ", and see ("Female", "Male"). The binary target is whether the individual annual income exceeds 50k per year (y = 1) or not (y = 0).
Reproducibility All experiments were run on server equipped with an Intel Xeon Gold 63 12U CPU and 256 GB of RAM. Our full codebase- cicebase-including data preprocessing, model training, and auditing notebooks can be found at sttpps//ggitee com/ /a//lalaa allaturinnnnn g/SAFT.
4.2 Comparison with point-wise estimations
We consider both COMPAS and Adult datasets over 20 random 2:1 train-test splits, followed by model training and fairness auditing each subgroup every split. In each case, we compare our approach against the fixed-threshold 5SP criterion (0 = =.1, as, e.g., in [1]), to highlight how a threshold based audit, while common, fails to account for sampling variability.
Figure illustrates a selection of intersectional subgroups (one per panel) from the COMPAS dataset. In each panel, we specify the minimum (Tmin) and maximum (Amaar) size of the protected group across different train-test splits. The gray band marks the no-detection region under a fixed-thresholdd approach. For each train-test split we compute the point-wise SP estimation, the Bayesian (blue) and the asymptotic (red) credible/confidence intervals. In panel (a) we observe a scenario where 8SP would detect a violation, since all point estimates are outside the gray band, and our approach agrees, since we can reject Hg. In panel (b), with a smaller intersectional group, 6SP would detect fairness violations, but approach shows that, due to the wide confidence interval, we cannot reject H, and detect a violation. It is worth observing that in this scenario, due to the small size of the protected group, the asymptotic behaviour would not be reliable; our approach SAFT would indeed rely on the Bayesian inference instead. In panel (c), we show how, for small groups, the train-test split can have strong impact: 5SP would detect fairness violations in roughly half the cases, while answer is consistent across all splits. A similar pattern, but with a fairness violation consistently detected on our side, is shown in panel (d) for bigger group.
Bayesian
Asymptotic
dSP
(Black, Maln)
[Other, 245, Famale)
(Asian, 25-45)
(White, Male) Alein = 519, fin 563
D1xi = 864, fleas> 884
fless 1. =
Oag 2. =9
-0.3
-0.2
-0.1
0.0
0.1
-0.4
-0.2
0.0
0.2
0.4
-0.5
0.0
0.5
-0.1
0.0
0.1
0.2
SP(S)
SP(S)
SP(S)
SP(S)
(a)
(b)
(c)
(d)
Figure 2: Point-wise estimation versus confidence intervals, COMPAS dataset.
Figure 3 shows analogous results for the Adult dataset. In panel (a) we show a cross-split, cross- approach agreement on fairness violation detection. In panel (b) we have a disagreement, as 5SP would detect fairness violations, but our approach cannot reject Ho. In panel (c) we have the opposite disagreement: 8SP would not detect faimess violations, but we show that with such big group Ho can
8

---

Bayesian
Asymptotic
65P
(White, 18-40. Female)
(A. 265. Female)
(White, 40-65, Female) 22453, 25.06
[0ther: Male)
2453, 2506
03wm 96
4
-4
0.0
-0.05
SP(S) (a)
SP(S)
SP(S)
SP(S)
(b)
(c)
(d)
Figure 3: Point-wise estimation versus confidence intervals, Adult dataset.
be rejected, and a fairness violation warning raised. In panel (c) we show a scenario where the train- test split impacts both 8SP and our approach. In general, these results show how a fixed-threshold approach cannot accommodate groups of different size even within the same dataset-model scenario, and can be sensitive different train-test split. Conversely, our approach takes into account the group size and is generally more robust regarding train-test splits.
4.3 Comparison with ySP
We again consider all intersectional groups (across all 20 train-test splits) on COMPAS and Adult to compare the decision from our framework with those from threshold-basee ySP approach. We plot the size of the protected group against its SSP score computed according to Equation 3. We also compute the fairness violation with our statistical testing approach SAFT (Algorithm 1). and use the results as color-code in the scatterplots: red for fairness violation detection, blue for no violation. Figure (COMPAS on the left, Adult on the right) shows that no single threshold (horizontal line) on SSP can cleanly separate true violations from non-violations. These plots further support the need for a adaptive hypothesis-testie teeting approach.
COMPAS dataset
Adult dataset
10a2
Faliness
detected
Funtness vistation detecied
NO Furmess violation detected
[SPASK more 5000 S0d)
10
[SPA5K wor. 5000
10
10-3
10-1
19
19a
10>
102
Group (n.)
Group
Figure 4: Protected groups size, SPP scores, and interval-based fairness violations.
Conclusions
In this work, have shown how fairness auditing move beyond brittle point estimates and arbitrary thresholds. By providing an analytic Central-Limit result for the statistical parity difference and coupling it with Bayesian Dirichlet-muliiomiial estimator, our framework conveys size-adaptive confidence and credible intervals that remain valid from the most common to the rarest intersectional subgroups.
Limitations. Our Bayesian framework inherently requires the specification a prior distribution, which encodes assumptions about the parameters before observing the data. Even when employing weakly informative (flat) priors, one is still making a modelling choice. However, this standard within the Bayesian framework and should be understood natural consequence of its formulation.

---

Moreover, while we discuss the generalization to other faimess measures, our focus is on statistical parity (and on equal opportunity in Appendix C.9) and leaves open empirical validation for additional group fairness definitions.
Future work. Looking ahead, intend to extend methodology consider tests for individual fairness [17, 25, 21J and counterfactual fairness [35], which evaluate whether the model behaves fairly for individuals and under hypothetical changes sensitive attributes, and represent an additional classes of fairness metrics, different from the group fairness metrics considered in this work.
Broader Impact. Our testing framework strengthens fairness audits by providing transparent, size- adaptive confidence intervals, which can reduce both false alarms and overlooked harms, especially for under-represented intersectional groups. At the same time, because fairness admits many metrics and any auditor can choose the one most favourable to their goals, there is a risk of using selective reporting hide genuine bias. We therefore urge practitioners apply methods responsibly: report all tested metrics and provide transparent interpretations of confidence intervals rather than as standalone justifications for deployment.
Acknowledgment and Disclosure of Funding
This work is conducted as part of the Horizon Europe project PRE-ACT (Prediction of Radiotherapy side effects using explainable AI for patient communication and treatment modification). It is supported by the European Commission through the Horizon Europe Program (Grant Agreement number 101057746). by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 22 00058, and by the UK government (Innovate UK) under application number 10061955.
Furthermore, the authors wish to thank Filippo Ascolani for the fruitful discussions regarding the statistical framework.
References
[1] Avinash Agarwal, Harsh Agarwal, and Nihaarika Agarwal. Fairness score and process standard- ization: framework for fairness certification in artificial intelligence systems. Al and Ethics, 3(1):677-27, 2023.
[2] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. Ethics of data and analytics, pages 254-264. Auerbach Publications, 2022.
[3] Ainhize Barrainkua, Paula Gordaliza, Jose A Lozano, and Novi Quadrianto. Uncertainty matters: stable conclusions under unstable assessment of fairness results. In International Conference on Artificial Intelligence and Statisties, pages 198-12006. PMLR, 2024.
[4] Joachim Baumann, Anikó Hannák, and Christoph Heitz. Enforcing group fairness in algorithmic decision making: Utility maximization under sufficiency. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 2315-2326, 2022.
[5] Barry Becker and Ronny Kohavi. Adult. UCI Machine Learning Repository, 1996.
[6] Philippe Besse, Eustasio del Barrio, Paula Gordaliza, and Jean-Michel Loubes. Confidence intervals for testing disparate impact in fair learning. arXiv preprint arXiv: 1rXivv1807.000 1807.06362, 2018.
[7] Philippe Besse, Eustasio del Barrio, Paula Gordaliza, Jean-Michel Loubes, and Laurent Risser. survey of bias in machine learning through the prism of statistical parity. The American Statistician, 76(2):188-198, 2022.
[8] Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, et al. Uncer- tainty as a form of transparency: Measuring, communicating, and using uncertainty. In Proc. of the 202/ AAAI/ACM Conference on AI, Ethics, and Society, pages 401 0011-413,2 2021.
[9] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77-91. PMLR, 2018.
10

---

[10] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pages 785-794, 2016.
[1]] John J Cherian and Emmanuel J Candès. Statistical inference for faimess auditing. Journal of Machine Learning Research, 25(149):1-49, 2024.
[12] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 552)1153-163, 2017.
[13] Paulo Cortez and Alice Maria Gonçalves Silva. Using data mining to predict secondary school student performance. In Proceedings of the 5th Annual Future Business Technology Conference (FUBUTEC 2008), pages 5-12, April 2008
[14] Kimberle Crenshaw. Mapping the margins: Intersectionality. identity politics, and violence against women of color. Stanford Law Review, 43(6):1241-1299 1991.
[15] Kimberlé Crenshaw. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. In Feminist legal theories, pages 23-51. Routledge, 2013.
[16] Eustasio Del Barrio, Paula Gordaliza, and Jean- Michel Loubes. central limit theorem for Ip transportation cost the real line with application to fairness assessment in machine learning. Information and Inference: A Journal of the IMA, 8(4):817-849, 2019.
[17] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-226, 2012.
[18] Kawin Ethayarajh. Is your classifier actually biased? measuring fairness under uncertainty with bernstein bounds. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2914-2919, Association for Computational Linguistics, July 2020.
[19] Michael Feldman. Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkata- subramanian. Certifying and removing disparate impact. In Proc. of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259-268, 2015,
[20] Antonio Ferrara, Francesco Bonchi, Francesco Fabbri, Fariba Karimi, and Claudia Wagner. Bias-aware ranking from pairwise comparisons. Data Mining and Knowledge Discovery, 38(4):0062-2086, 2024.
[21] Antonio Ferrara, David García-Soriano, and Francesco Bonchi. Beyond shortest paths: Node fairmess in route recommendation. Proc. of the VLDB Endowment, 120))3230-3242, 2025.
[22] James R Foulds, Rashidul Islam, Kamrun Naher Keya, and Shimei Pan. Bayesian modeling of intersectional fairness: The variance of bias. In Proceedings of the 2020 SIAM International Conference Data Mining, pages 424 432. SIAM, 2020.
[23] James R Foulds, Rashidul Islam, Kamrun Naher Keya, and Shimei Pan. An intersectional definition of fairness. In 2020 IEEE 36th international conference on data engineering (ICDE), pages 1918-199 1921. IEEE, 2020.
[24] Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasubramaniin, Sonam Choudhary, Evan P Hamilton, and Derek Roth. A comparative study of learning. In Proceedings of the conference on fairness, accountability, and transparency, pages 329-338, 2019.
interventions in machine
[25] David Garcfa-Soriana and Francesco Bonchi. Maxmin-fair ranking: individual fairness under group-fairness constraints. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 436 446, 2021.
[26] Avijit Ghosh, Lea Genuit, and Mary Reagan. Characterizing intersectional group fairness with worst-case comparisons. Artificial Intelligence Diversity, Belonging, Equity, and Inclusion, pages 22-34. PMLR, 2021.
11

---

[27] Usman Gohar and Lu Cheng. A survey on intersectional faimess in machine learning: Notions, mitigation, and challenges. International Joint Conference on Artificial Intelligence, 2023.
[28] Irwin Greenberg. An analysis of the ecoce "four-fifths" rule. Management Science, 25(8):762-
769, 1979.
[29] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
[30] Ursula Höbert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the ccompuationalll-ideniiii Machine Learning, pages 1939-19999 1948. PMLR, 2018.
masses. In International Conference on
[31] Johannes Himmelreich, Arbie Hsu, Ellen Veomett, and Kristian Lum. The intersectionality problem for algorithmic fairness. In Proceedings of the Algorithmic Fairness Through the Lens of Metrics and Evaluation, volume 279, pages 68 68-95, PMLR, 14 Dec 2025.
[32] Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository, 1994.
[33] Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairesss gerryman- dering: Auditing and learning for subgroup fairness. In International conference on machine learning, pages 2564-2572. PMLR, 2018.
[34] Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 247-254, 2019.
[35] Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual faimess. Advances in neural information processing systems, 30, 2017.
[36] Selim Kuzucu, Jiaee Cheong, Hatice Gunes, and Sinan Kalkan. Uncertainty as a fairness measure. Journal of Artificial Intelligence Research, 81:307-335, 2024.
[37] Erich L Lehmann and George Casella. Theory point estimation. Springer Science & Business Media, 2006.
[38] Edward Mascha and Thomas Vetter. Significance, errors, power, and sample size: The blocking and tackling of statistics. Anesthesia and analgesia, 126:691-698, 02 2018.
[39] Mathieu Molina and Patrick Loiseau. Bounding and approximating intersectional fairness through marginal fairness. Advances in Neural Information Processing Systems, 35:16796~ 16807, 2022.
[40] Giulio Morina, Viktoriaa Oliiyy,, Julian Waton, Ines Marusic, and Konstantinos Georgatzis. Auditing and achieving intersectional fairness in classification problems. arXiv preprint 1..044191..01468, 2019.
[41] Cecilia Panigutti, Alan Perotti, André Panisson, Paolo Bajardi, and Dino Pedreschi. Fairlens: Auditing black-box box clinical decision support systems. Information Processing & Management, 58(5):102657, 2021.
[42] Manish Raghavan and Pauline T Kim. Limitations of the" four- fifths rule" and statistical parity tests for measuring fairness. Geo. L. Tech. Rev., 8:93, 2024.
[43] Eugen Slutsky. Über stochastische asymptoten und grenzwerte. Metron, 1925.
[44] Hilde Weerts, Miroslav Dudík, Richard Edgar, Adrin Jalali, Roman Lutz, and Michael Madaio. Fairlearn: Assessing and improving fairness of ai systems. Journal of Machine Learning Research, 24(257): 2023.
[45] Ke Yang, Joshua R. Loftus, and Julia Stoyanovich. Causal intersectionality and fair ranking. In Symposium Foundations of Responsible Computing, 2021.
[46] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without dis- parate mistreatment. In Proceedings of the 26th International World Wide Web Conference,
www '7. pages 1171-11 1180, 2017.
12

---

Appendices
Background and Related Work
13
Proofs
14
C Additional experiments and discussions
16
C.I Additional datasets
16
C.2 Training Details
16
C.3 Additional results for Statistical Parity
16
C,4 Bayesian vs. Asymptotic Interval Convergence
24
C.5 Effects of small sample size on the asymptotic results
25
C.6 Choice of ñ
25
................................
C.7 Influence of the prior
................................
25
C88 Comparison with Bootstrapping
26
C.9 Results for Equal Opportunity
26
D NeurIPS Paper Checklist
31
A Background and Related Work
Early research on algorithmic fairness framed the problem as ensuring Statistical Parity across sensitive groups defined by single attributes such Equalized Odds and Equal Opportunity [29], Predictive Parity [12], and related criteria remain cornerstones of the field and are implemented in popular toolkits (e.g., FAIRLEARN [44]). A large body of work subsequently proposed pre-, in-, and post- poottpproessing interventions to satisfy one or more of these metrics [46, 24).
gender. Demographic Parity [19],
While the above metrics are ordinarily reported as single numbers, they are fact noisy estimates computed finite samples. Angwin et al. [2] and the ensuing debate around COMPAS highlighted how small data perturbations can flip an "unfair" judgment. Kearns et al. [33] observed that even with large overall data sets, fairness estimates for minority sub-group uub-groups problem magnified under intersectionality. Recent surveys [7, 8] consolidate evidence that ignoring estimation error leads to both false alarms (flagging random fluctuation bias) and missed harms (overlooking small but significant disparities in large groups).
have extremely high variance,
Intersectionality [15, 27, 26, 20] demands analysis at the cross-sections of multiple protected attributes (e.g. "Black women over 50"). Empirical studies such as Gender Shades [9] revealed accuracy gaps that only appear at such intersections. Yet the number of subgroups grows exponentially with attribute count, leaving most groups sparsely represented. Foulds et al. [23] formalized this "resolution limit" and demonstrated that point estimates become statistically meaningless in the low-data regime. Subsequent work proposed auditing algorithms that search adaptively for large-effect subgroups without exhaustively enumerating all intersections [33, 40], but these methods still deliver binary verdicts on noisy estimates. Complementary to these estimation strategies, Molina and Loiseau [39] provide probaabibbby bounds that connect easily measured marginal faimeess to otherwise intractable intersectional fairness. Their analysis shows when marginal guarantees suffice and offers computable certificates that scale many attributes.
direct frequentist remedy report confidence intervals (CIs) around each fairness metric. Boot- strap Cls for Demographic Parity, Equalized Odds, and related metrics are supported in FAIRLEARN
13

---

and have been advocated by Besse et al. [6], Del Barrio et al. [16], and Cherian and Candès [11]. In NLP, Ethayarajh [18] introduces Bernstein-
unfairness. deriving analytic Bernstein bounds that convert small, partially annotated sample into tight confidence interval on bias. Wide inter- vals immediately signal when data are insufficient to draw conclusions, a property exploited in the FairlyUncertain benchmark [3]. Nevertheless, Bootstrap Cls are not reliable when sample sizes are extremely small. Additionally, Himmelreich et al. [31] advocate for hypothesis testing for fairness in intersectional settings; however, differently from us they focus on the accuracy computed per group and solely rely on central limit results, without considering alternatives for small sample sizes.
Beyond estimating metrics more precisely, several authors treat disparities in predictive uncertainty themselves indicators of unfaimess. Bhatt et al. [8] decompose aleatoric and epistemic uncertainty across groups: higher epistemic uncertainty for subgroup suggests redeesreprsenttttion in the training data. Kuzueu al. [36] leverage uncertainty estimates from deep ensembles and Bayesian neural networks to design uncertainty-awart mitigation strategies.
Existing solutions thus improve fairness assessment along isolated axes-frequentist intervals or Bayesian models-but none provides unified, scalable framework that (i) quantifies estimation uncertainty, (ii) adapts gracefully to the extreme sparsity of intersectional subgroups. and (iii) remains agnostic to the choice of fairness metric. We address this gap by introducing a statistically principled method that merges Bayesian modeling with weventtraaionnnnqaaliii guarantees, yielding valid uncertainty bounds at small subgroup granularity without prohibitive computation.
B Proofs
We first prove Theorem 2 which is the general version. Then, we show how Theorem 1 follows consequently.
Theorem 2. Let (pi,.... Pa) set of probabilities of q disjoint events, with E.P, let C= (C, C.) Categorical(p) be the related categorical distribution, and let Ca...., C be n i.idd realizations of C. Let & be a contimuously differentiable (in an neighbourhood of EC) function ó RR + R. Then
VT
((EE)) -C)))((E)) $ 1), - 20,
C*)
where a = VV == V(O(C)) is the gradient of 6. and = (diag(p) ppp' is the covariance matrix of C, where diag(p) indicates diagonal matrix with entries ps-
Proof of Theorem 2. Let C² ,....., C be i.i.d. samples from the categorical distribution ~ Categorical(p), where each C' is one-hot vector in R7 (i.e., C¹ {ei,...., eg) and EC" EC p).
Define the sample mean:
C Σ C
72
Then, by the multivariate Central Limit Theorem [37], we have:
Vn(C,, - p) S (0,,
where Σ diag(p p) - ppp is the covariance matrix of the categorical variable C.
Now, let Φ R99 - RR be function continuously differentiable in an neighbourhood of p FC. Then, by the multivariate Delta Method [37], we have:
Vnl (o(C,)) <()) 4 N(,
where 0² = =VVTVV with V VO(EC) = Vo(p), and, hence:
Vn (o(C.) <()) 4 (0,
14

---

Furthermore, we point out that by Slutsky's Theorem [43] one can use the covariance & estimated from the data, instead of the population one.
We now show that the asymptotic normality of the Statistical Parity estimator SP (S) (Theorem 1) follows from Theorem 2 by expressing random variable.
smooth function of the empirical mean of categorical
Theorem 1 (Central Limit Theorem for Statistical Parity). Let o(S) V E, where V
Pa.8 (P8)
PL.
P1.e and Σn is defined in Equation of the paper. Then
(ps)
[pa)²
Vn (SP, - SP(S)) N(0,
00,
(S)
where S denotes convergence in distribution and [ 1) indicates the standard normal distribution.
Proof of Theorem I. Define the following four mutually exclusive and exhaustive events:
= (f(z) = E S(x))
62 = (J(x) 0, E S(X)),
= (f(z) 1, S(x))
= (J(z) 0, E S(x))
Let C' E R$ be categorical random variable representing one-hot encoding over these events, with probability vector p (Pi,s, P (P,,s.P Pi.5. P1,5.
Po..Po)).
Let C² ,l,..., C be i..dd realizations of C, and define the empirical mean:
Then a consistent estimator of the population vector EC = p.
Now define the function Φ R1 - by
P1
P3
@(p1 P2, Ps, Pt)
Pi +
P2
P3 + P4
which corresponds to the population Statistical Parity:
SP(S) <(p), (S) o(p)
Note that Φ is continuously differentiable in neighborhood of p. provided the denominators P1 + Pa and P3 + P4 are nonzero, which holds under standard assumptions.
By Theorem 2, we have the convergence:
vn (S) (o(p) - o(p)) 4 N)(,,))
EV, V Vo(p), and K diag(p) pp T is the covariance matrix of the categorical variable C.
where (S)
Direct computation of the gradient yields:
Po.s P1, Po, P1, V = (ps)²' (ps)2 (pz)² (pg)2
with ps Pi,s + po.s and ps P1.s + Po.s-
The covariance matrix for this 4-category categorical variable coincides with the matrix Σ, defined in Equation (4) of the main paper. Hence, the asymptotic variance is exactly
(o(S))² V E
Thus,
SSP, SP((S) 4 1),,1),
(S)
which proves Theorem special case of Theorem 2.
The results for Equal Opportunity and Disparate Impact are obtained analogously, as special cases of Thcorem 2.
15

---

C
Additional experiments and discussions
Due to space constraints, the main paper focused on selected set of datasets and results. Here, we present broader discussions and analysis.
C.1 Additional datasets
The additional experiments include the German Credit [32] and the Student Performance [13] datasets.
German
The German Credit dataset, sourced from German bank, comprises 1000 loan applicants characterized by twenty attributes (three of which are considered protected), including financial and personal details. Its primary use is in binary classification tasks to predict creditworthiness, with binary target "bad" (y = 0) and "good" (y = 1). The three protected variables considered are: Ser ("Male", "Female"), Foreign worker ("Foreign", "Not foreign"). and Age ("Under 30", "00-""0"" "40- "Over 50").
Student The Student Performance dataset focuses student achievement in secondary education at two Portuguese schools. It includes attributes related to student grades, demographics, social factors, and school-related characteristics. The data gathered for 649 students through school reports and questionnaires. The binary target variable indicates whether student's 's final grade falls within the range of O 10 (y = 0) or exceeds 10 (y 1). The protected variables considered are: age (" ("Adult" (2 18), "Minor" (< 18)), Sex ("F", "M"") for Female and Male students, Pstatus ("T","A"), which indicates the parent's cohabitation status, living Together Apart, and Address ("U","R)), which indicates student's 's home address type, Urban or Rural areas.
C.2 Training Details
In all experiments, we train an XGBoost classifier [10] on each dataset without hyperparameter tuning. since our focus is on fairness auditing rather than maximizing predictive performance. We perform 20 independent 2:1 stratified train-test test splits account for sampling variability. The following default XGBoost parameters were used for all runs:
eesttmmttoss = 100
max_depth =
learning_rate rate 0.3
subsample 1.0
colsample_bytree 1.0
objective "binary:logistic"
eval_metrin "logloss"
After training each split, evaluate accuracy the held-out test set. Table 1 reports the mean and standard deviation of test accuracy over the 20 splits for each dataset.
Table 1: Mean accuracy and standard deviation of the XGBoost model over 20 random 2:1 splits.
Dataset
Mean Accuracy
Std. Deviation
Adult
0.8702
0,0021
COMPAS
0.7141
0.0094
German
0.7633
0,0229
Student
0.7681
0.0177
C,3 Additional results for Statistical Parity
Figure shows the resolution limits for Statistical Parity violations when detecting when group advantaged.
16

---

10
0.8
fraction of Ox
5.0
04
5.4
5.
012345 10
16
30
35
50
60
90
100
Size of Protected Group (na)
Figure Resolution limits for Statistical Parity violations under varying global negative rates ((f = 0). when detecting advantaged groups (the dual figure showing the boundary for deciding if a group is being disadvantaged is reported in Figure in the main paper). Each curve traces maximal fraction of negatíve outcomes needed to reject Ho: SP()) = 0 at = 0.05 as function of the group size Y2.- the left of each vertical bar the "no-power" zone, where subgroups are too small to detect discrimination, regardless of the observed disparity. The shaded region below each curve is the "discrimination zone", where the subgroup's negative rate enough to establish statistically significant parity violation.
In Figures and of the main paper, we showed the results for Statistical Parity for selection of intersectional subgroups (one per panel) from the COMPAS and Adult datasets. In each panel, specified the minimum mm ) and maximum (Tmaw) size of the protected group across different train-test splits. In Figures 6, 8, 9, 10, 11 we show the complete set of intersectional subgroups for the four datasets Adult, COMPAS, German and Student. We have omitted panels for empty intersectional groups (C'Asian', 'Less than 25', 'Female") and ('N.A., 'Less than 25', 'Female") in COMPAS, ('A.P.I.', 'Under 18', 'Male') in Adult). For every split, we plot the Bayesian credible interval (blue), and the asymptotic normal confidence interval (red). In certain panels, such as the final one in Figure 10, some lines are absent. This occurs because, in some train-test splits, the intersectional group had no instances in the test fold.
As for the results in the main paper, these findings highlight the limitations of a fixed-threshold approach, which struggles account for variations in group size within the same dataset-model setup and remains sensitive to different train-test splits. In contrast, our method considers group size, demonstrating greater robustness across train-test variations.
17

---

1482, 1200
Pow 13730 Puw 12035
n 654
S34
12% 121
120 174
3.0
2310
-8.05
0.00
0.08
0.18
-0.10
-8.05
000
0.00
0.11
-02
-01
0.0
.0.
9.0
01
10.-4
9271
$255 E4g4
40-25
Ghard5
Under 15
211
10007 10826
Make
0.2
02
0.11
0.25
0.05
0.1
0.1
02
Femmlo
/Slar 118
110 100
[Bauch "Oeer 851
[2leck Unde
Deye -7570
Dex Dow 5000
522 Dom
Doo 160 Drey
Dacy 2275 Pow.
SP
SP
163, Feeu 212
13 Row 25
Pow 6 S.
75. -10
-2 Puu -40
.5.1
017
9.70
.1.
22
117
0.4
86
0.1
.12
(Other G20
(ALEE 40/)
OALE /owr (2)
0.2
0.7
02
0.2
0.1
0.2
010
(ALLE Undde 197)
(Blast Mine)
(BBaod (oomeee)
(Wir Soorate)
P1: Male))
(API) Fandee
COteer Male]
50tter Fenawi
OUIE) Save)
SP
frare 46, Deny 50
2557 foay a 2002
Den 3116 fiwm - 3220
Oee 4370 Demy 44521
1513 Rey - 1901
-12
-0133
0103
0.15
402
-11 20
115
62
0.4
10
1350
572
Figure 6: Point-wise estimation versus confidence intervals, Adult dataset.
18

---

CDker Temakri
(Under Made)
(Under (Namaae)
(Blckk
Puw 423, 430
200 221
Puw n 112
+15 frw 110
425, 400
-310
-0.05
0.00
3.00
3.13
-02
-0.1
0.5
0.1
-22
-0.
00
0.1
-22
.0.1
09
0.1
-0.1
(Black: (16 (42 (Fumskr)
(Slect 40 65: Moke
(Btest RX Terrues
(Stct, 10 05 Mex)
[Slac: Vue t5 Terale?
280
02
6.1
00
0.1
0.1
0.2
02
04
(Bladd "urew 19. Mare?)
(Bhek) Undd 18, Turale?
CWMle: (10 -47, (aay)
(Whit: "14_ 4r. "aamer)
[Mrr 40 85, Nids)
Serdle) Dac Pow. 220
Tundr (Wak))
[WtrW "Undo FFmmaer
14.35 Crom
30
Dsx 77, Crm 102
Dsix Pear =101
0.1
SP
SP
5P
-171 - 220
106 125
Bur 110 150
45, 60
Dew T T.Bew -1
111
.5.05
1.00
3.70
10:00
.2
00
00
06
(APU Uideert Fansl))
(Otur 48- N081)
(ohar Fammalr
(Ohw Make)
02
17
02
0.5
04
08
42
0.
02
cofter, 103 BY. Farree))
[Olee': Over 6b/, (A4e7)
(Dier 'Over es: Verder)
(Dthrr: "urker 10. Mare))
[Olher: Undd Forrlss
(ALIE) 118 M400)
(WLE) 15a (Fameee)
(ALE) 40 Modr)
(AIE: Feruall
(ALE Yiuer Malr]
SP
CP
SP
Nea C. Dew 2
O Paar
0.0
0.2
0.7
1.4
1.88
-02
07
0.5
Figure 7: Point-wise estimation versus confidence intervals, Adult dataset (continued).
19

---

Beck
11010 1882
166 777
10 Aag 107
0-
50
5P
400 450
1121
1222
4.5 Coe
1230
- 1570
Dea 16, Pore
0.3
0.
(Ronkl 125
(Basi faar 22)
[Caucabbe) Graderttonn 47) Pgg Dew
Naucabee 22
(Carpuar tase
275 T/e
Day 50,0
5.1
6.1
0.3
4.1
0.
0.1
03
-11
0.1
(Hagain, Grezar 457
Mazzer 23 -485
(Hagani:: Lesatran 231
DArart Unear Fun-451
Dataa 23-
10
-21
0.2
03
-22
11
-35
CS
35
(Attr: Lass 25)
Trsd:Grattrrvi Iran 47)
(MA 45)
(NA), Lass than 22)
/Ohro Maad)
40.75
+1.90
-7.25
0.53
025
-0.00
-0.3
1.00
3.5
-11
20he'' Porral
(Baak 'WuA0
(Bick
(Gauazet (M60)
(Gacenat Porel))
10
(Hiquei:: "VaM))
S-lagnei (Sandi))
[MAid Vold)
DMar Fandol)
(NA) Voad)
30
SP 200 Ruw
(Grster
Malr)
& Pe 2
321 Pus Xd
26 Auw 00
903 378
241
SP
Figure& 8: Pise estimation versus confidence intervals, COMPAS dataset.
20

---

Closs fow Maw?
(Lace than 25: FwTree) Pow
(Ohter Gianerton 45 Puwe Rowe
(ottrr townrttee 45, Senako
(Citer, 25- 45, "Vaii Pew Pare
Pur 500, Dace m28
Puw D. Oou
611
-.1
0.
0-1
02
0.5
3.23
-0.
-7 Ples +113
H. Dar -77
2
Por 11T +55
H Ban +22
0.0
Nao) Puw 146, 174
480 flwee 148
96, Pew 124
Fuw 201 233
(Cauzabbr 441 flgic -3
Terrar)
(Cauestae) Tear)
[Caccndar 25-48) 45, Tammer)
(Cascncan' Lese ttan 166e]
[Gauzadar Lam Iun
250 225
=110
-2
SP
SP
(Hippank:: Yeeae the Man1
(Hapavid Grate 40 Fenur))
(Hiqasic 25 45, Maee)
(Hipornd 25 4F Ferrak))
(Hipooit Less Than 29: Maee
(Htigenet Les Dun 25: Perul?1
(Alkri "Chasr 40 Nulc)
(Rarn) Oreater Bunn-4 Forale)
(haaan 25 48, Nike)
Okaur) 25 45, (ardl))
4.75
-350
-135
0.93
3.25
(Walan: Less 25 14811
(KA) Gmaatrraan 68 Nalr Puw Bay
(NA) 'Geate has 48, Farwle)
(NA) 25 45 M481
(NA) 25 .6 Fenuk)
10.0
0.5
114
11.2
0.0
0.2
00.5
10.00
0.2b
8.00
0.2
0.5
0.0
0.5
0.75
10.00
-02b
0.00
(N.A: "Joza Ban Maad))
floa
-2.75
-2.00
-3.25
3.00
0.25
Figure 9: Point-wise estimation versus confidence intervals, COMPAS dataset (continued).
21

---

31 fla
100 Ray 1:33
129
221 -24
5P
B7 -120
312 Pow 323
7 16
23
Dew
Ron Dow
Ouw 37 Aoe
77, Peee
-22
-22
(20-45
(One 2 Nirfongge
[Ower Do Fonige)
(Uider33 hol lniign) 122 Rap
(Tidder 37 Monilee?
Rup 22 Now
Uug +0 Dreu
-04
-32
0.3
4.2
0.
-..0
0..2
5.00
-22
0.5
4509
-025
0.5
(40-57 "Mdl boriger"
24--5 Maviet
C2447 Navln))))
(2547 Pasign)
[V Notowigh)
-1
02
-12
-15
-22
-611
1.5
01
32
-05
30
-01
31
(V) To:dgr))
(T. hat
(T. Torlle?
1Cwe57 M Naachngggg
(D M Tonign))
001
-02
-13
-1.5
-12
-121
SP
113
(Owr 87, P. Nollorge))
(urnke 30; V. Nos kengel)
(Jldt 31: W. Puegel
(Unie 37, Nid lasie])
(Unla Y Pavie))
(6350) Turrenigrr) 34
19300 Fongyn +0
(45-2 "eitosiee
(M44 Nerhingr)
(M442/9 Forigg)
5P
gP
5P
(50-77
12 naw 22
Ruw +0 m3
Figure 10: Point-wise estimation versus confidence intervals, German dataset
22

---

GAR
50
OWA V
(Uay' V1
Doy
-04
30
-04
03
Mest: N1
HALT T1
V
23
14
-44
V)
m
+66 120
2
01 30
E1
02
62
04
02
18
022
040
0.5
04
76,
15
Yow 4. Saye
22
50
57
Figure 11: Point-wise estimation versus confidence intervals, Student dataset
23

---

Furthermore, in Figure 4, we considered all intersectional groups (across all 20 train-test splits) on COMPAS and Adult to compare the decision from our framework with those from a threshold- ySP approach. In Figure 12, we extend the same approach for the two other datasets, German and Student. The results show that a fixed threshold on GSP fails to consistently distinguish fairness violations, highlighting the necessity of a size-adaptive hypothesis-testing approach like SAFT.
Gorman datase!
Stident dalaset
10
Fairness vislation delected
Falmnss violation detected
NO Farmosa viclation delected
NO Falmiss vialition delecied
[SPSS]] P 5(,))
[SP55] E Sta0)
10
10"
10"
10"
10
10"
10"
10"
Size of Protected Group (nx)
Size of Protected Group (nu)
Figure 12: Protected groups size, SP scores, and interval-based fairness violations.
C.4 Bayesian Asymptotic Interval Convergence
To illustrate how our Bayesian credible intervals smoothly interpolate between robust small-sample uncertainty and asymptotic normality. Figure 13 shows four synthetic subgroups with sizes n E {1, 10, 100, 1025). In each panel we plot (i) the posterior density of the statistical-parts gap under a uniform Dirichlet prior (blue curve), (ii) the corresponding 95% credible interval (blue shaded region), and (iii) the asymptotic 95% confidence interval from Theorem 1 (red curve region) centered at the same point estimate. As n grows, the two intervals converge and the posterior density sharpens, while for very small the Bayesian interval remains appropriately wide, reflecting high epistemic uncertainty.
('Asian', LLess than 25', 'Male')
Asian
ng =
ns 10
40
Frequency
30
Frequency
20
10
0
0
-0.50
-0.25
0.00
0.25
-0.50
-0.25
0.00
0.25
0.50
SP
('Black', '25 45', 'Female')
Black
ng 1025
10.0
20
Frequency
7.5
Frequency
15
10
2.5
5
0.0
0
-0.1
0.0
0.1
0.2
-0.2
-0.1
0.0
SP
SP
Figure 13: Comparison of Bayesian credible intervals (blue) and asymptotic normal confidence intervals (red) for subgroups of size ng. As ng increases, the posterior density concentrates and the two intervals converge; for small ng, the Bayesian interval remains larger, correctly encoding high uncertainty.
24

---

C.5 Effects of small sample size on the asymptotic results
The Wald variance for subgroup S can be rewritten as:
Pi.5
(1
P1.8 P1,S Po.s
PI.8 P1.9 Po.3
(1
P1. P1,3 Po.
P1.9 Po.33
ng
Since ns R P(S) (and analogousty for nı)) the variance decreases rate 1/n,, even for rare groups. Both asymptotically and for small samples, small subgroups do not lead to inflated variance.
That said, the more subtle issue is that in finite samples, the plug-in variance o² may actually underestimate uncertainty when ns is small. In particular, when the estimate of PL. is very pr.s
close to 1, the estimation of the term P1.s+Po.s 1
PL.8 piay_ in the variance formula becomes Pi.s+
near zero. This drives the Wald variance estimate to be artificially small, even though the underlying sampling uncertainty high for such extreme proportions with small nss
This can make the Wald test anti-conservative falsely rejecting the null due to high variance in small groups captured by the asymptotic approximation. This effect be seen, for example, in the top-left plot of Figure 13.
This is why introduce the Bayesian Dirichlet-multinmiiz test to provides calibrated inference in the -smlle regime. Hence, the problem is not variance inflation, but undercoverage due to inappropriate reliance on asymptotics when subgroup counts are small.
C.6 Choice of ñ
In Algorithm we suggested choosing ñ 30, since thirty observations a widely used rule-of-thumb trust C1.T-based Wald tests [38] because coverage and power stabilize around that cell size in typical multinomial settings. In Table 2 we analyze the effect of the choice of ñ for SAFT applied the Adult dataset. One observe how from around twenty observations the total number (over all train-test splits and over all possible sub-groups) of fairness violations detected stabilizes. Hence, also our empirical results confirm thirty observations from the bayesian to the asymptotic test. With smaller ñ. c.g., n = 5, the amount of samples appears be too small to rely on asymptotic results, indeed the CLT might induce to over-reject the null, as explained in Appendix C.5.
safe choice switch
Table 2: Total number of fairness violations detected by SAFT in Adult for all train-test splits and for all sub-groups for different values of ñ
2
10
20
30
40
50
100
200
500
1000
of violations 1159
1144
1118
1098
1097
1097
1097
1096
1098
1097
1097
1097
Clearly, users may raise lower if their data warrant it.
C.7 Influence of the prior
We now explore the impact of the prior choice and the incorporation of prior information, derived from the training set. We consider:
Uniform (flat) Dirichlet prior, which imposes no substantive prior information;
Empirical Bayes priors derived from training-set frequencies.
For each subgroup we
a max(1, min(co,n si Co,s, C1,s, C1,)) (co,: + 1, + + +, A (0.5, 1,2,5)
where Cij is the number of training samples with label i {0, 1} in group j {s,3).
25

---

This construction scales the prior strength by À while preserving the empirical class proportions, and caps the influence when any cell count is extremely small. Table 3 shows the results with the confidence/credible intervals (CIs) for some selected subgroups from the Adult dataset.
The results show prior influence for extremely small subgroups, while all CIs rapidly converge the asymptotic regime as grows. For the smaller ns case, the Wald intervals are spuriously narrow, clearly under-representing the true sampling uncertainty. In contrast, the uniform-prior Bayesian credible intervals are much larger, appropriately reflecting the scarcity of data. As we introduce empirical Bayes priors, these intervals shrink toward the Wald width but remain more conservative than asymptotic bounds, demonstrating that modest data- iata-informed priors can stabilize inference without overconfidence. With hundreds or thousands of samples, all methods produce nearly identical intervals, illustrating that prior effects vanish once more data is available.
Table 3: Influence of the prior
Other, Under 18, A.P.I. Over 65,
White, Over 65,
Black
Female
Male
Female
F5g
12
210
1522
Asymptotic normal CI
[-0.204, -0.192]
[-0.271, 0.041]
[-0.182, -0.123]
-0.097
Bayesian CI with uniform prior
[-0.190, 0.488]
[-0.176, 0.176]
--0.174.-0.115
[-0.127,
-0.097
Bayesian CI with empirical prior. À 0.5
[-0.240, 0.014]
[-0.196, 0.048]
[-0.179. -0.121]
[-0.138,
-0.144]
Bayesian CI with empirical prior, À I
[-0.236, 0.037]
[-0.196, 0.052]
[-0.179,
-0.123]
[-0.138,
-01113
Bayesian CI with empirical prior, À 2
[-0.228, 0.035)
[-0.177, 0.037]
[-0.177.
[-0.138,
-0.114]
Bayesian CI with empirical prior, A 5 Bootstrapping
[-0.195, 0.000]
[-0.156, 0.01
[-0.176, -0.128]
[-0.139, -0.113]
[-0.204, -0.192]
[-0.203, 0.073]
10.178.-0.118
[-0.130, -0.998]
C.8 Comparison with Bootstrapping
In moderate to large subgroups, nonparametric bootstrap intervals (e.g., [44]) coincide, both theo- retically and empirically, with the Wald limits obtained from our CLT analysis, so bootstrapping adds computation without accuracy gains at that scale. In small intersectional subgroups where Wald approximations fail, the bootstrap typically inherits the same pathologies: with pronounced more contingency cells (e.g.,
class imbalance, resamples frequently yield zero counts in no predicted positives for group S), producing degenerate or undefined disparity estimates and a highly discrete, skewed resampling distribution that yields off-center overly narrow intervals. In the last row of Table 3, the bootstrap intervals closely track the asymptotic confidence interval even at small ns, indicating that resampling does not remedy the underlying approximation error in this setting. Accordingly, we use analytic CLT intervals when regularity conditions are met, and crrichlet-multinomii credible intervals when any cell counts are in the single digits, where they remain well-calibrated and avoid degeneracy.
C.9 Results for Equal Opportunity
In Figures 14 and 15 (Adult dataset) and Figures 16 and 17 (COMPAS dataset), we present the results for Equal Opportunity. In each panel, Amin and 7maz denote the minimum and maximum number of instances belonging to the protected group with y across different train-test splits. Compared to the plots for Statistical Parity Figures to 9, those for Equal Opportunity show wider Cls, consistently reflecting lower niy counts given the additional conditioning on y for Equal Opportunity.
26

---

Our Dour
2224
40-45
Orerdd
Ups
-155
CALE: 02. 537
1R E1 683
CALE: (eee- 337
CAPL (5)
JAn Ovr E51
-00 -008
050
0.06
916
-433
-005
E33
025
40-451
(Doer (vrrtt)
r0ner: Utder1))
31- 400
(Wet 45:451
FANAN: tree 3T)
FALE: Tersdr)
2ALE: Merl
PARU Peeas]
DEFU "MNO
2
22
-033 -008 033 025 K10
Ca 125
-2
Da E.g
50
(40-4) de: West
27 Temael
(Upee:31 Pergll
Figure 14: Point-wise estimation versus confidence intervals for Equal Opportunity, Adult dataset.
27

---

Fuher LF Marr
CAAE 18 1E HeRURT
CALIS 1S- POMT
(MI)) Over Teppat
-023 -305
053
810
-006
(ARUAG-EP TaTaer
fow Pope
-028
510
4066
60
027
18B0:: Mare
Bladk o2 G5 Mare
Hiae Sie 65 Fanae
(ildd: Ourrs 14007
2Bads: Usder 12 Tamaae)
-02
120 006
10 Veat
Fomer Taraei
(0tter
(Daer Tamadi
F0ter Oup Terai))
New
Aup New
-028
+0.0
00
(WwW 46-49 Maul
(WEE Over 60 (engel
(WIH Oper 50, Magel
(Wit:: Sheer Joyast
FiPta: Ureer "NA47
34
0.5
0.1
02
0.5
2.29
0.06
650
5.5
3.5
422
465
5.0
043
0.00
Figure 15: Point-wise estimation versus confidence intervals for Equal Opportunity, Adult dataset (continued).
28

---

Bock
+0
-ilffa 482
106 Faz
Duw 0.0
033
3.5
3.5
5.10
0.35
0.00
031
235
491
14 104
31) Ouer
242
Greae Dar
265
pua 25
171
153
-11
12
11
433
-000
3
0.0
-030 -355
33)
10
610
-33
65
01
Ohder 25 400
[dear Weoter Toe (5)
fpecma 2011
(Rade: 35 mi
-330
-0.0
0.5
C30
-18
-0.5
-30
-1.
0.
-077
-0.0
-0.2
0.0
025
-333
-005
35
(Bbck,, Krrate Dar 651
[Ruck Leosthon 250
Couepaa 25 (9)
(Casosarr Srader Den 49)
(Caycssart, Tes -36
266,
-323
-0.35
6.6
0.55
630
-3.5
6.66
3:55
0.00
-011
01
(2 125
457
(Taoaac Creals (an45)
(Hooaae Lew Dun 257
(TLA 25-45
Crete Dunas1
-22
-1.3
-253
0.81
505
0.00
-222
-555
555
43
[RAA Lu8 201
10 125 421
(Dtar Grader 11a (01)
10ter Las dan
TAlln: Tomset
0.75
5.00
0.55
0.10
022
42
35
[Acarl Make)
Olacet Beralr)
(liect MIF1
Ciucadon Tandle]
Cuaradyy Mal1
-19 -433
025
-438
-335
335
0.00
-030
-845
0.00
815
630
03
-333 40
10
10
[NA) 1Wi
Sonvet Terraa)
-21
-0.00
-1.55
0.00
8.05
-3.0
-385
0.00
000
0.0
-02
[0terr Mddr)
C25 80 Mulcl
DDiaca thn Torakii
(Prrata than-D M4501
05
0.00
23 -2.3
0.00
Figure 16: Point-wise estimation versus confidence intervals for Equal Opportunity, COMPAS dataset
29

---

(taer 20) (emasa
PLaectan 20"
Kotter Sieate tan MiMe) Per
(other) Senatertho (5) Semaee)
(otber: Neir
12
-04
-12
6.
0.2
-32
03
61
-04
-32
32
-31
-04
-02
60
02
-32
cotter 451 Monaee
10ber (aar Ban
pontrr Tescttrttttnnnnn Nense:
(Rice Seeet Boaerrrann By Marl
[Rick Sevteetaa Tender
-42
00
32
46
-04
-12
16
-5
16
-155
03
040
-33
30
39
TRIak:T 43. 161E1
(Rad: 23 45, Torukrl
(Beee Lea Dan Marl
(Beer the 25, Turdel
(Gaaaaati Create the Most
-323
-0.5
606
0.5
630
-355
616
385
0.0
-610
-165
600
545
630
-030
-255
0.3
605
0.0
(Cass ol "Sedd 210 Tunal0)
GGonelaan 23
(Craausen 23 45. Tendel
(Cosenler Los Dar Mak? (Gucadaan Las Danz5t Temmre
1.0
2.0
6.0
0.0
4.25
510
(Mlipanee Greatr dan Mawroligaae Greater Paan 42 Terra)))
(Hlppank 25-43 43. Male
CHlipark: 125 45. Memaeee
(Hazari Las der 22
-12
-131
-21
131
-01
0.
31
-02
3.
05
(Skpanne Sae dan 25, (ensw)
(NdW, Grare Ban 27 Vale] (NDST, Minaee tha E7 Furder
(Wile: DE 45 M0
(MdW) 45, Fmail!
-3.
-353
-0.25
-3.5
-325
1Mir Lesemaan 29 Vale
(900. les Ban Forate]
[NA) Smedte then 45 MMaed
thentertoonn
(KA5 Brooter than Hemaee
(WA1125 49, (Mie]
-3.50
0.00
-315
0.10
645
(MUA)-25 451 (umand)
[NA Sawa Den (Mee0
[NAI Soo than (heman))
Figure 17: Point-wise estimation versus confidence intervals for Equal Opportunity, COMPAS dataset (continued)
30

---

D NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Yes, the claims made in the abstract and introduction accurately reflect the paper's contributions and scope. We provide the theoretical framework for hypothesis testing for fairness in Section 3 and empirical results in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations the work are discussed in the Conclusions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer: [Yes]
Justification: We detail the assumption of theoretical results in Section We formally prove our results in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper describes how experiments were run at high level in $4.1, while the specific details to reproduce the results described in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
32

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code and data are available in the Supplementary Material, and instructions to reproduce the results are given.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: ponddesuumissioooooooooo
/ //nips. htte/uublc/ggguccee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The Experiments Section describes how many data splits were used for the results, and details about the models used, even though they not central part of the results, are included in the Supplementary Material.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the results across all the train-test splits. Furthermore, report confidenceeccrddible intervals as correctly produced via central limit results or bayesian inference.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Reproducibility in $4.1 provides the computational resources needed to repro- duce the experiments, and, while running the experiments only takes few minutes, more details are provided in the Supplementary Material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ptttps//neeurppes cc httpppulc///icccciieeeeeeeeeeeee
Answer: [Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed the Conclusions.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
34

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not foresee any high risk for misuse of this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Yes, we credited them in appropriate ways.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
36

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We used LLMs only for editing/formatting.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ /ttpar/////u.....//e ceturrpss.cc/ooneeeenncc.. /c/Coonferencee//2255//.. for what should or should be described.
37


Reviews:
[{"review_id": "7l8s3GOcuf", "reviewer": "Reviewer_sptv", "summary": "When we look at intersections of protected classes (i.e. using an intersectional lens) then the size of the intersectional group is smaller and sometimes too small to allow for reliable testing of fairness measures such as disparate impact. The authors wish to find the granularity/resolution at which a given data set can be assessed for fairness/unfairness.\n\nAs I understand it, the goal is to provide different levels of \"fairness\" to intersectional groups based on their size. This can allow more fine-grained control of type I/type II error when testing/auditing for unfairness.", "questions": "* One challenge in fairness research is that there are so many notions of fairness, each with their embedded assumptions: a nice exposition of this was given a while ago by Arvind Narayanan in a tutorial [21 fairness definitions and their politics](https://www.youtube.com/watch?v=jIXIuYdnyyk). How many of these (beyond the 3 mentioned in the paper) fall under Theorem 2?\n\n* One of the stated contributions is that many fairness measures have \"pitfalls\" with respect to intersectionality. However, I think many of the referenced papers also point out such issues with prior fairness measures. The fact that any threshold test will have false positives and false negatives is pretty obvious, so what do the authors mean by this contribution?\n\n* (line 47) Is the right question whether we *should* rigorously guarantee fairness or whether we *can* rigorously guarantee fairness?\n\n* I found the discussion and framing of this a bit confrontational with respect to prior work. Of course, checking for unfairness is a kind of (hypothesis) test: one computes a test statistic and compares it to a threshold. They claim prior measures *require* a practitioner to set a global threshold without looking at the particulars of the data set. From Figure 4 in the submitted manuscript, it seems that a linear correction for group size on $\\gamma$-SP would allow for a fixed threshold.\n\n* My understanding is that all of the distributions under consideration are PMFs or conditional PMFs. The estimators are plug-in estimates based on the empirical joint PMF and (as shown in the appendix). Obviously, you have to compute the covariances etc. to get a full characterization of the asymptotics, but from my reading Theorem 2 showing asymptotic normality is basically a kind of textbook application of the delta method: am I missing something here?\n\n* Is it really \"rigorous\" to set thresholds based on the thresholds given by the asymptotics? This is the large sample test and is the general recipe used in statistical practice but the type I and type II error guarantees are then only approximate, right? \n\n* For the Bayesian setting, I do like the idea of incorporating prior information into the prior. Is it possible to do an experiment exploring the impact of prior information/bias?\n\n* I feel like the word \"rigorous\" in the paper does a lot of heavy lifting. In Algorithm 1, what is the \"rigorous\" justification for $\\tilde{n} = 30$? Similarly, what $K$ was used for the Bayesian method?\n\n* Apart from conforming to arbitrary norms for error tradeoffs, why select $\\alpha = 0.05$? Shouldn't this problem be looked at in more general Neyman-Pearson tradeoff sense?\n\n\n**Smaller comments:**\n\nThere are some other works on addressing intersectionality in ML + fairness: the authors may be interested in seeing how those fit in within the landscape of the literature review:\n\n* Ghosh et al, [Characterizing Intersectional Group Fairness with Worst-Case Comparisons](https://proceedings.mlr.press/v142/ghosh21a.html), 2021.\n* Foulds et al, [Differential Fairness: An Intersectional Framework for Fair AI](https://www.mdpi.com/1099-4300/25/4/660), 2023. (Maybe subsumes reference [12]?)\n* Himmelreich et al. [The Intersectionality Problem for Algorithmic Fairness](https://arxiv.org/pdf/2411.02569), 2024. (This was posted close to the submission deadline, however).", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n* Proposes a framework for taking into account sample size when testing/auditing for fairness in small subgroups.\n* Extensive experiments help illustrate the results.\n\n**Weaknesses:**\n\n* The presentation of the results obscures the main technical tool, which is a straightforward result about the asymptotic normality of statistics calculated from plug-in estimates of discrete random variables.\n* The promised \"principled methodology\" also contains some arbitrary parameter choices.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their positive scores, their in-depth understanding of our work, and their appropriate comments, which will be useful to improve the presentation of our work. We next answer the reviewer's questions.\n\n---\n\n## Questions:\n\n> **Q1:** *[...] many notions of fairness [...]*\n\nUnfortunately, the cited tutorial lists definitions only in prose and offers no mathematical formulation (see slide 2 \"Limitations of this tutorial\"), and we couldn't find supporting documents for the tutorial. To answer the Reviewer's question about which definitions can be modeled by our famework, we refer to the paper *'Fairness definition explained'* [Verma and Rubin, 2018]. Our framework can model the first 13 definitions of table 1, while the last 7 are excluded. Essentially, our framework can model all those based on the confusion matrix or related conditional probabilities.\n\n---\n\n> **Q2:** *[...] many fairness measures have \"pitfalls\" with respect to intersectionality [...]*\n\nPrecisely. Because most metrics were not designed for sparse, intersectional subgroups, fixed‑threshold tests often suffer high false‑positive and false‑negative rates. Our contribution is to quantify when and how severe these failures become, and to provide size‑adaptive tests that remain reliable in such settings. Furthermore, prior fairness metrics based on thresholds do not have Type-I/II error guarantees.\n\n---\n> **Q3:** *[...] or whether we can rigorously guarantee fairness?*\n\nThank you for the clarification. We agree that “can” is more appropriate, given that our paper focuses on testing for fairness. We will adopt this wording in the revision.\n\n---\n> **Q4:** *[...] would allow for a fixed threshold.*\n\nA linear adjustment to γ‑SP has several limitations. There is no principled way to choose its slope or intercept, it comes with no statistical guarantees, and, as Figure 4 shows, even the best‑tuned linear correction would still misclassify points that our method separates correctly.\n\n---\n> **Q5:** *[...] Theorem 2 showing asymptotic normality [...]*\n\nThe reviewer is correct: Theorem 2 is obtained by applying the multivariate CLT, the delta method and Slutsky’s theorem. One interesting technical choice was to express the fairness metrics as functions of probabilities of disjoint events. This keeps the covariance matrix analytic and easily computable; without that disjointness the algebra would be considerably heavier. The novelty is therefore not the asymptotic argument itself but (i) showing that the resulting Wald test slots seamlessly into our size‑adaptive framework and (ii) pairing it with a calibrated Bayesian test that covers the small‑sample regime where the asymptotics are unreliable. We will add a short remark to the camera ready version of the manuscript to make this clearer.\n\n---\n> **Q6:** *Is it really \"rigorous\" to set thresholds [...]*\n\n\nIn finite-sample inference, all frequentist tests based on sampling distributions are approximate unless the distribution is exactly known. For large groups, our Wald test is derived from a formal CLT with a plug‑in variance estimator. This is not an ad-hoc correction: under standard regularity conditions, the rejection region is calibrated to control type I error at level α as n→∞, which is the definition of an asymptotically valid test. By the multivariate CLT and delta method, we obtain a limiting normal distribution with an explicit variance estimator. This ensures that, as n grows, the rejection region converges to one with exact type I error α. In other words, the α‑level control is not heuristic but a formal consequence of the CLT under standard regularity conditions. Moreover, this is the same notion of “rigor” used in essentially all classical hypothesis tests (e.g., t‑tests, χ² tests, Wald tests in GLMs). Calling it approximate is correct in a finite-sample sense, but asymptotic tests are considered rigorous because their error properties are analytically characterized and converge to the nominal level. In contrast, fixed-threshold fairness audits offer no such convergence guarantee at all - their type I/type II error is uncontrolled even asymptotically. \n\n---\n> **Q7:** *[...] the impact of prior information/bias?*\n\n\nWe have explored the impact of the prior choice and the incorporation of prior information (derived from the training set) and the revision will include a full analysis contrasting:\n\n* **Uniform (flat) Dirichlet priors**, which impose no substantive prior information;\n* **Empirical‑Bayes priors** derived from training‑set frequencies.  For each subgroup we set\n\n$$\n\\boldsymbol\\alpha\n=\\frac{\\lambda}{\\max \\bigl(1,\\min(c_{0,s},c_{0,\\bar s},c_{1,s},c_{1,\\bar s})\\bigr)}\n\\bigl[c_{0,s}+1,c_{0,\\bar s}+1,c_{1,s}+1,c_{1,\\bar s}+1\\bigr],\n\\qquad \\lambda\\in\\{0.5,1,2,5\\},\n$$\n\nwhere $c_{i,j}$ is the number of training samples with label $i\\in\\{0,1\\}$ in group $j\\in\\{s,\\bar s\\}$.\n\nThis construction scales the prior strength by $\\lambda$ while preserving the empirical class proportions, and caps the influence when any cell count is extremely small.\n\nThe following table shows the preliminary results with the confidence intervals (CI) for some selected subgroups from the Adult dataset:\n\n|  | (Other, Under 18, Female) | (A.P.I., Over 65, Male) | (White, Over 65, Female)| (Black) |\n|:---:|---|---|---|---|\n|\\$n_S \\$| 2 | 12| 210 | 1522 |\n| asymptotic normal CI |[-0.204, -0.192] |[-0.271, 0.041] |[-0.182, -0.123] |[-0.130, -0.097] |\n| Bayesian CI (uniform prior) |[-0.190,  0.488] |[-0.176,  0.176] |[-0.174, -0.115] |[-0.127,  -0.097] |\n| Bayesian CI (empirical-Bayes prior, $\\lambda=0.5$) |[-0.240,   0.014] |[-0.196,  0.048] |[-0.179, -0.121] |[-0.138, -0.114] |\n| Bayesian CI (empirical-Bayes prior, $\\lambda=1$) |[-0.236,  0.037] |[-0.196,  0.052] |[-0.179, -0.123] |[-0.138, -0.113] |\n| Bayesian CI (empirical-Bayes prior, $\\lambda=2$) |[-0.228,   0.035] |[-0.177,  0.037] |[-0.177, -0.125] |[-0.138, -0.114 ] |\n| Bayesian CI (empirical-Bayes prior, $\\lambda=5$) |[-0.195, 0.000] |[-0.156,  0.011] |[-0.176, -0.128] |[-0.139, -0.113] |\n\nThis preliminary sensitivity analysis confirms prior influence for extremely small subgroups, and that all methods rapidly converge to the asymptotic regime as $n_S$ grows. For the smaller $n_S$ case, the Wald intervals are spuriously narrow, clearly underrepresenting the true sampling uncertainty. In contrast, the uniform-prior Bayesian credible intervals are much larger, appropriately reflecting the scarcity of data. As we introduce empirical-Bayes priors, these intervals shrink toward the Wald width but remain more conservative than asymptotic bounds, demonstrating that modest data-informed priors can stabilize inference without overconfidence. With hundreds or thousands of samples, all methods produce nearly identical intervals, illustrating that prior effects vanish once more data is available.\n\n---\n> **Q8:** *I feel like the word \"rigorous\" [...]*\n\nThe testing framework is rigorous; the specific cut‑off is a pragmatic choice. Thirty observations is the rule‑of‑thumb widely used to trust CLT‑based Wald tests because coverage and power stabilise around that cell size in typical multinomial settings. Users may raise or lower \\tilde n if their data warrant it; the method itself remains unchanged.\nFor the Bayesian method, we set K = 10.000. Ten‑thousand Dirichlet-multinomial samples keep Monte‑Carlo noise below 0.2 pp at the 5 % level while remaining computationally trivial. Any larger K only reduces that already‑negligible error; smaller K risks noticeable numerical variance. Also in this case, practitioners can calibrate K to their precision/speed trade‑off.\n\n---\n> **Q9:** *[...] why select alpha = 0.05? [...]*\n\nOur test is built exactly in the Neyman–Pearson framework: for every subgroup $S$, we define $H0:SP(S)=0$ and a rejection region calibrated to control type I error at level $\\alpha$. The choice of $\\alpha=0.05$ in the experiments is not a claim of optimality; it is a standard convention to pick one operating point on the type I/type II error curve for comparison with prior fairness audits.\nMore importantly, the method is agnostic to the specific $\\alpha$. For any $\\alpha \\in (0,1)$, the procedure yields a valid $α$‑level test (asymptotically for the Wald version, exactly for the Bayesian small-sample version). By sweeping α, one can trace out the full ROC‑like tradeoff between false fairness violations (type I) and missed unfairness (type II). This is the general Neyman–Pearson sense: we provide a test statistic with a calibrated null distribution, and practitioners or regulators can set α based on the cost of false positives vs. false negatives.\n\nIn short, $\\alpha=0.05$ is a reporting choice, not a methodological constraint; the framework itself already encodes the Neyman–Pearson tradeoff and supports any α the user deems appropriate. We can make this explicit in the main text.\n\nFurthermore, to strengthen the connection with the Neyman–Pearson lemma it should be possible to define a likelihood ratio test, which is the uniformly most powerful test in the set of level alpha tests. We will consider this extension in a possible camera ready version.\n\n---\n> **Q10:** *There are some other works on addressing intersectionality in ML + fairness [...]*\n\nWe are grateful to the Reviewer for pointing out these related papers: we will add them to our literature review.  \n\n--- \n---\n**We welcome any follow‑up questions and would be happy to provide additional details or clarifications.**"}, {"review_id": "N2PYNBsM2i", "reviewer": "Reviewer_W2en", "summary": "This paper proposes a framework for group-size adaptive hypothesis test (SAFT) for auditing a fair classifier. When sufficient samples are available, they propose using a Wald test to get construct a statistically consistent test, and when there is a possibility of very few samples from certain groups, they propose using a Bayesian test with a Dirichlet prior. The advantage over other works lies in the fact that, depending on the subgroup sizes, the proposed testing framework can dynamically adapt.", "questions": "My questions are around the three limitations posed previously. \n1. Empirical results for other fairness metrics: This can help nail down the effectiveness and \"size-adaptiveness\".\n2. Dependence on minimum support $\\tilde{n}$\n3. Some justification for how accurate the Bayesian testing framework is.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Solid theoretical results that tackle the said problem very well. The empirical results work very well for SP at least.\n2. The Bayesian framework to tackle the small sample regime is a good idea.\n\nWeaknesses:\n\n1. Empirical results for other fairness metrics: The authors admitted in their limitations that this is a pending line of work. Sparse subgroups will have a larger effect on conditional parity metrics like Equal opportunity and Equalized odds, and hence would have been the perfect demonstration of the proposed size-adaptive framework.\n2. Dependence on minimum support $\\tilde{n}$: The authors fix $\\tilde{n}=30$ for switching between the Bayesian inference test and the Wald test. Perhaps an ablation study on the sensitivity of this minimum support parameter can help the reader.\n3. While I understand that it is hard for the authors to theoretically validate the effectiveness of the Bayesian inference setup, it will be interesting to check how close it gets to the true hypothesis, perhaps with a simpler data distribution (eg. Gaussian).", "was_revised": true, "final_justification": "I am personally not confident with the statistical tools used in this paper, but pretty confident about the problem statement being tackled. After reading other reviews, and due to the fact that conditional parity metrics were not considered (which are very important in fairness literature), but the framework shows promise for working with conditional parity metrics, I will maintain my current rating and will lean towards an Accept.", "rebuttal": "**W1/Q1**\nWe thank the Reviewer for the feedback. We already identify the lack of conditional‑parity experiments as a limitation. We already explained how the  framework can be used to compute confidence intervals for Equal Opportunity and the extention to Equalized Odds is straightforward, so no extra theory is required; the work is purely computational. Running a full set of new experiments for Equal Opportunity and/or Equalized Odds (with several datasets and subgroup configurations) is not realistic within the short rebuttal window, but can be easily done in time for the camera‑ready version, so that the resulting figures and tables can be placed in an appendix. That addition will demonstrate the size‑adaptive behaviour under a conditional metric while keeping the main claims unchanged.\n\n**W2/Q2**\n The specific cut‑off is a pragmatic choice. Thirty observations is the rule‑of‑thumb widely used to trust CLT‑based Wald tests because coverage and power stabilise around that cell size in typical multinomial settings. Users may raise or lower $\\tilde n$  if their data warrant it; the method itself remains unchanged. While an ablation study on the sensitivity of this minimum support parameter is not doable in this short timeframe, we can absolutely launch it and collect/report the results in time for the camera ready version of the paper.\n\n**W3/Q3** To address the reviewer’s request, we performed a simulation study using Gaussian distributions.  \nThe true Statistical Parity (SP) value (computed from the CDF of the normal) is covered in about **95 %** of cases by the Bayesian posterior credible interval. Coverage is insensitive to sample size, but the interval narrows as more data are used.\n\nWe next describe the setup in detail. We consider two groups, $S$ and $S'$, drawn from different normals:\n\n$$\nX_{S} \\sim \\mathrm{N}(0,1), \\qquad\nX_{S'} \\sim \\mathrm{N}(1,1).\n$$\n\nWe define a simple classifier:\n\n$f(x)= 1$  if  $x > 0.5;$ \n$f(x)= 0$   otherwise.\n\nFor each repetition we sample $n$ points for $S$ and $m$ for $S'$, compute the credible interval for SP, and check if it contains the true value. We repeat this **1000** times. The true statistical parity value can be computed analytically from the cumulative density function of the Gaussian distributions and it is **SP(S)=0.383**.\n\nThe credible interval hits its nominal **95\\%** coverage for any $n,m$. As $n,m$ grow, the interval tightens around the true SP. With $n=m\\in\\{5,10,30,100,1000\\}$, the last‑run intervals are:\n\n| *n = m* | *Credible Interval* | *Pr*(0 ∈ CI) |\n| ---: | ---: | ---:|\n| 5       | (−0.020, 0.797)     | 0.85         |\n| 10      | (0.028, 0.635)      | 0.61         |\n| 30      | (0.175, 0.616)      | 0.12         |\n| 100     | (0.285, 0.533)      | 0.00         |\n| 1000    | (0.352, 0.433)      | 0.00         |\n\n*Bayesian credible intervals for different sample sizes; the last column is the empirical probability (over 1000 runs) that the interval contains 0.*\n\nLarger samples quickly force the interval away from 0, enabling confident detection of fairness violations.\n\nWe would be grateful to engage the Reviewer in a fruitful discussion and to provide further details if needed."}, {"review_id": "m8blQnrhKC", "reviewer": "Reviewer_L4Jh", "summary": "The paper introduces a new framework to measure fairness based on hypothesis testing instead of comparing a point estimate of the fairness metric with a predefined threshold. They provide two tests to detect statistical parity unfairness. These tests consider the uncertainty and group sizes due to finite test sets. The first test utilizes the confidence intervals based on asymptotic normal approximation of disparity; this test is valid for large sample sizes and based on CLT. When the sample sizes are small for subgroups, the authors propose another test that utilizes Bayesian approach. They propose a Dirichlet prior on the estimated values for the statistical parity, then get samples from the posterior and build an empirical confidence interval. The authors also demonstrate how the methodology can be generalized for other fairness notions, such as equal opportunity.  The numerical experiments cover 4 real datasets, and the proposed method is compared against the fixed threshold rule and the scaled(by the probability of the subgroup) threshold rule.", "questions": "Please see the weaknesses section.", "limitations": "Yes, authors addressed the limitations and broader impact.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The idea of using hypothesis testing for unfairness detection is novel; the framework unifies both frequentist and Bayesian approaches and switches between them based on the test data availability.\n- The authors provide empirical results on four real-world datasets. The proposed method avoids false alarms and detects unfairness that threshold-based rules miss.\n- The paper is well-structured and easy to follow. The mathematical derivations are complete. \n\nWeaknesses\n\n- The technical contributions of the paper are limited; the contribution of the paper is mostly proposing hypothesis testing for fairness detection.\n- It is not clear how the small subgroup sizes affect the first proposed test. The variance estimate would be inflated due to small subgroups, probably, but a clear explanation of this could be beneficial before introducing the second test.\n- For the Bayesian approach, the authors do not report on the sensitivity of the prior selection. It could be interesting to see priors informed by the training set frequencies. \n- The experiments section can have better baselines, such as other confidence interval generation methods, i.e., bootstrap intervals available in FairLearn. The extension for the other fairness notions is explained in the paper, but the experiments section only focuses on statistical parity.", "was_revised": true, "final_justification": "I think that the authors are providing detailed answers to the questions raised by the reviewers. They provided new results regarding the sensitivity to prior selection for the small sample tests, and they provided new results that provide comparisons with an existing baseline method bootstrapping that provides confidence intervals for fairness violations.\n\nI still think that the technical contribution/novelty of the paper could be considered limited, but overall, the positive sides of the paper slightly overweight the limitations.", "rebuttal": "Thank you for the thoughtful and constructive feedback. Below we address each weakness in detail and describe the revisions we will incorporate to strengthen the paper.\n\n---\n\n**W1:** To the best of our knowledge, in the fairness literature, there is no universally accepted analogue of the χ², t-, or log-rank tests that offers to practitioners a standardized, statistically calibrated test for detecting fairness violations that adjusts to subgroup size while providing both asymptotic and finite-sample guarantees. What exists instead are score heuristics such as δ-SP or γ-SP whose numerical thresholds (±0.05, ±0.10, etc.) are chosen ad-hoc and carry no type-I/type-II guarantees. Regulators, standards bodies, and applied ML teams have repeatedly called for **principled, statistically grounded decision rules** that can withstand legal and scientific scrutiny. \n\nOur method addresses this gap and is intended to serve as a standard protocol for fairness auditing, in the same way that χ² or t‑tests function in classical statistics. We note that Theorems 1 and 2 result from an effective combination of the Central Limit Theorem, the Delta Method, and Slutsky’s theorem, while the concise formulation arises from a careful way of combining probabilities of disjoint events. This not only shows that many commonly used fairness metrics can be derived within our framework but also enables straightforward computation of the covariance matrix.\n\n---\n\n**W2:** We thank the Reviewer for pointing out the need for more clarity on how small subgroup sizes affect our first test (the Wald test).\n\nThe concern raised is that small subgroup sizes might inflate the variance estimate, leading to conservative behavior. However, under the asymptotic regime in Theorem 1, this is not the case. The Wald variance estimate for subgroup $S$ can be rewritten as:\n\n$\\hat{\\sigma}^2 = \\frac{p̂_S (1 - p̂_S)}{n_S}$\n\n$+ \\frac{p̂_\\bar{S} (1 - p̂_\\bar{S})}{n_{\\bar{S}}}.$\n\n\nSince $n_S \\approx n \\cdot P(S)$, the variance of the estimator decreases at rate $1/n$, even for rare groups. Both asymptotically and for small samples, small subgroups do **not** lead to inflated variance.\n\nThat said, the more subtle issue is that in finite samples, the plug‑in variance $\\hat{\\sigma}^2$ may actually **underestimate** uncertainty when $n_S$ is small. In particular, when the estimated subgroup rate $\\hat{p}_S$ is very close to 0 or 1, the term $\\hat{p}_S (1 - \\hat{p}_S)$ in the variance formula becomes near zero. This drives the Wald variance estimate to be artificially small, even though the underlying sampling uncertainty is high for such extreme proportions with small $n_S$.\n\nThis can make the Wald test *anti‑conservative*, falsely rejecting the null due to high variance in small groups not captured by the asymptotic approximation.\n\nThis is precisely why we introduce the second test: a Bayesian Dirichlet–multinomial method that provides calibrated inference in the small-sample regime. We will clarify this transition in the paper: the problem is not variance inflation, but **undercoverage** due to inappropriate reliance on asymptotics when subgroup counts are small.\n\n---\n\n**W3:** We have explored the impact of the prior choice and the incorporation of prior information (derived from the training set) and the revision will include a full analysis contrasting:\n\n* **Uniform (flat) Dirichlet priors**, which impose no substantive prior information;\n* **Empirical‑Bayes priors** derived from training‑set frequencies.  For each subgroup we set\n\n$$\n\\boldsymbol\\alpha\n=\\frac{\\lambda}{\\max \\bigl(1,\\min(c_{0,s},c_{0,\\bar s},c_{1,s},c_{1,\\bar s})\\bigr)}\n\\bigl[c_{0,s}+1,c_{0,\\bar s}+1,c_{1,s}+1,c_{1,\\bar s}+1\\bigr],\n\\qquad \\lambda\\in\\{0.5,1,2,5\\},\n$$\n\nwhere $c_{i,j}$ is the number of training samples with label $i\\in\\{0,1\\}$ in group $j\\in\\{s,\\bar s\\}$.\n\nThis construction scales the prior strength by $\\lambda$ while preserving the empirical class proportions, and caps the influence when any cell count is extremely small.\n\nThe following table shows the preliminary results with the confidence intervals (CI) for some selected subgroups from the Adult dataset:\n\n|  | (Other, Under 18, Female) | (A.P.I., Over 65, Male) | (White, Over 65, Female)| (Black) |\n|:---:|---|---|---|---|\n|\\$n_S \\$| 2 | 12| 210 | 1522 |\n| asymptotic normal CI |[-0.204, -0.192] |[-0.271, 0.041] |[-0.182, -0.123] |[-0.130, -0.097] |\n| Bayesian CI (uniform prior) |[-0.190,  0.488] |[-0.176,  0.176] |[-0.174, -0.115] |[-0.127,  -0.097] |\n| Bayesian CI (empirical-Bayes prior, $\\lambda=0.5$) |[-0.240,   0.014] |[-0.196,  0.048] |[-0.179, -0.121] |[-0.138, -0.114] |\n| Bayesian CI (empirical-Bayes prior, $\\lambda=1$) |[-0.236,  0.037] |[-0.196,  0.052] |[-0.179, -0.123] |[-0.138, -0.113] |\n| Bayesian CI (empirical-Bayes prior, $\\lambda=2$) |[-0.228,   0.035] |[-0.177,  0.037] |[-0.177, -0.125] |[-0.138, -0.114 ] |\n| Bayesian CI (empirical-Bayes prior, $\\lambda=5$) |[-0.195, 0.000] |[-0.156,  0.011] |[-0.176, -0.128] |[-0.139, -0.113] |\n| Bootstrapping intervals |[-0.204, -0.192] |[-0.203,  0.073] |[-0.178,  -0.118] |[-0.130, -0.098] |\n\n\nThis preliminary sensitivity analysis confirms prior influence for extremely small subgroups, and that all methods rapidly converge to the asymptotic regime as $n_S$ grows. For the smaller $n_S$ case, the Wald intervals are spuriously narrow, clearly underrepresenting the true sampling uncertainty. In contrast, the uniform-prior Bayesian credible intervals are much larger, appropriately reflecting the scarcity of data. As we introduce empirical-Bayes priors, these intervals shrink toward the Wald width but remain more conservative than asymptotic bounds, demonstrating that modest data-informed priors can stabilize inference without overconfidence. With hundreds or thousands of samples, all methods produce nearly identical intervals, illustrating that prior effects vanish once more data is available.\n\n---\n\n**W4:** Thank you for raising this important point. We experimented extensively with non-parametric bootstrap intervals prior to the submission, and decided not to include them because, in our settings, bootstrap often fails to offer any advantages over either the frequentist or the Bayesian approaches. However, we agree that it would be better to include a discussion about this point in the paper anyway.\nWe highlight now some characteristics of the bootstrap approach. First, in moderate to large subgroups, bootstrap intervals converge both theoretically and empirically to the same Wald limits delivered by our analytic CLT. At that scale, bootstrap therefore adds only computational overhead without any improvement. In the small $n_s$ intersectional slices where analytic Wald intervals break down, bootstrap inherits the same weaknesses. Resampling with replacement in class unbalanced scenarios frequently generates samples in which one of the four contingency cells is zero (e.g., no predicted positives for group $S$), producing degenerate or undefined estimates of $SP$. Moreover, the resulting empirical distribution is highly discrete and strongly skewed, which yields uncentered confidence intervals. \n\nIn the last row of the table reported in W3, we display the Bootstrapping intervals obtained. One can observe how similar they are to the asymptotic confidence interval, even in the small sample regime, highlighting their inadequate estimation (uncentered with small variance) in such cases.\n For these reasons, we opted for the analytic CLT, whose validity can be proved, and for the Dirichlet–multinomial credible interval, which remains calibrated even when cell counts are in the single digits. Nonetheless, the camera‑ready version will include a dedicated appendix comparing our method with the bootstrap approach, documenting the pathologies above, and clarifying the regimes in which bootstrap is a viable alternative.\n\nConcerning the lack of experiments with additional metrics,  we have already highlighted it as a limitation. We explained how the framework can be used to compute confidence intervals for some further metrics e.g.  Equal Opportunity, so no extra theory is required; the work is purely computational. Running a full set of new experiments for additional metrics (with several datasets and subgroup configurations) is not realistic within the short rebuttal window, but can be easily done and we plan to do it for a possible camera‑ready version, placing the resulting figures and tables in the appendix. That addition will demonstrate the size‑adaptive behaviour under an additional metric while keeping the main claims unchanged.\n\n---\n\nWe hope these clarifications address the Reviewer’s concerns. Our revisions will add the requested variance discussion, prior-sensitivity experiments, bootstrap comparison, and example extensions to other metrics. We believe these additions highlight the paper’s practical value and methodological novelty. We welcome any follow-up questions and would be happy to provide further details or clarifications."}, {"review_id": "og8ylYBI3t", "reviewer": "Reviewer_ujF4", "summary": "The authors propose an adaptative way for testing whether a fairness criterion holds. Instead of using a pre-defined threshold, a formal hypothesis test is run, which adapts naturally to the number of samples available in different protected groups. The main focus of the paper is on demographic (statistical) parity, although some extensions are possible by applications of the Delta-method. Specifically, for relatively large samples an asymptotically valid Wald test is proposed, while for small samples a Bayesian approach with a conjugate Dirichlet prior is used.", "questions": "- (Q1) is there a need for incorporating uncertainty in the constructed classifier f? In the current treatment, f seems to be treated as if it is not derived based on data? \n- (Q2) Is there a way of performing a similar test using bootstrap, in a non-parametric fashion?", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths:\n\n- (S1) The notation is clear, nice, and easy to follow,\n- (S2) The paper is well-written, and the concepts are presented clearly,\n- (S3) A formal hypothesis test approach is adopted, whereas common approaches may use a pre-defined threshold.\n\nWeaknesses:\n\nWhile acknowledging the positives listed above, the major drawback is that all of the theory appearing in the paper is really well-known and widely used in the statistics literature, so there is no need for new theoretical developments (Theorem 1 follows from a variant of the CLT, Theorem 2 by the Delta method). In addition, the adaptation of the testing to the context of fairness does not require a huge amount of translation. Therefore, this paper presents a nice technical note, but does not reach the level expected from a NeurIPS contribution. Since the basic setting of demographic parity is mastered, one obvious way to improve is to study a more ambitious problem, such as conditional or causal variants of parity, which would likely provide more significant theoretical and methodological challenges.", "was_revised": true, "final_justification": "The proposed method, which performs hypothesis testing for statistical parity, is really not a novel idea. Many works in the literature perform such hypothesis tests when doing experiments; these works, however, do not claim the hypothesis-testing to be a novel contribution. \n\nFurthermore, if a novel way of testing such fairness hypotheses was proposed, that would be great. However, this is not the case either -- the results are a direct corollary of classical results in statistics.\n\nTherefore, in my view, this paper should not be accepted. Having said this, the paper is well-written, and may be a useful educational note for some, so I am not very strongly against its acceptance (hence the score 3)", "rebuttal": "### **Originality and Significance for NeurIPS**\n\nWe thank the Reviewer for clearly acknowledging the clarity and the technical quality of our work. On the other hand, Reviewer ujf4 believes that **Significance** and **Originality** of our work are low and do not reach the bar for NeurIPS. We respectfully disagree with this assessment and in the following we discuss why our contribution does not amount only to repackaging standard statistics.  Classical tools such as the multivariate CLT and the Delta method are indeed decades old; however, their *application, calibration, and validation* for high-resolution fairness auditing is not.  \n\nTo the best of our knowledge, no existing work provides a standardized, statistically calibrated test for fairness violations that adapts to subgroup size and gives both asymptotic and finite-sample guarantees. Current methods (δ-SP, γ-SP, etc.) rely on fixed thresholds with no error control. Our proposal fills this methodological gap and is designed to become a standard testing protocol for fairness auditing, analogous to how χ² or t‑tests are used in classical statistics.\nWe would like to point out that both Theorem 1 and 2 follow from a smart use of combined CLT, Delta Method and Slutsky theorem, and the simple formulation follow from a refined choice of combining probabilities of disjoint events from which we observe that many practically used fairness metrics are derivable and which allows for an easy computation of the covariance matrix.\n\n\nMoreover, the Reviewer writes:\n>*‘’Since the basic setting of demographic parity is mastered, one obvious way to improve is to study a more ambitious problem, such as conditional or causal variants of parity, which would likely provide more significant theoretical and methodological challenges.’’*\n\nStatistical parity is not “mastered” in real deployments; regulators and practitioners still rely on threshold choices and point estimates.  We chose it as a canonical case to establish the statistical foundations.  However, our framework is *metric- and model- agnostic* and readily extends to other fairness metrics.  Conditional or causal notions are indeed exciting directions, and our resolution-limit analysis provides the very diagnostic tool needed before estimating causal effects in sparse conditions. Specifically,  Theorem 2 already generalizes to the mentioned conditional variants.\n\nNeurIPS values contributions that close the gap between statistical theory and machine learning practice in societally critical settings.  By unifying analytic variance, Bayesian calibration, and a decision-theoretic resolution limit, then demonstrating their impact on real datasets, we deliver a rigorously grounded, practically deployable method that the fairness-in-ML community still lacks. In these regards, the scores given by the Reviewer to Originality and Significance, seem not to be sustained.\n\nFinally, the Reviewer gave a general rating of *2: Reject [For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations]*, which does not appear to be adequately sustained by the arguments in the review. Our work offers a rigorously grounded, ready-to-use tool that addresses an urgent need in responsible AI practice, and thus meets the bar for significance and relevance at NeurIPS.\n\n### **Regarding uncertainty in the constructed classifier $f$**\n\nWe thank the reviewer for raising this important point. In our framework, the classifier $f$ is indeed derived from data, but it is treated as a fixed, given object at the moment of fairness auditing. This assumption reflects a common operational scenario: fairness audits are typically conducted *after* model training, on a deployed or pre-trained classifier whose predictions are to be evaluated for fairness.\n\nOur methodology focuses on quantifying the **statistical uncertainty in fairness metrics**, arising from the finite sample size of the evaluation data, especially in small or rare subgroups. The uncertainty we account for is thus epistemic, stemming from sampling variability rather than model uncertainty.\n\nThat said, we agree that in settings where the classifier outputs **probabilistic predictions**, it would be possible and potentially useful to incorporate model uncertainty into the fairness analysis. For instance, one could assess fairness over distributions of predicted probabilities, or incorporate uncertainty from stochastic or Bayesian classifiers into the fairness tests. This is a promising direction, and one that complements our work: our current methodology can serve as a principled post hoc test of fairness, while future extensions could integrate upstream uncertainty in the predictions of $f$ itself.\n\nIn summary, our approach does not require the classifier to express uncertainty, but is compatible with extensions that incorporate such uncertainty when available. We will integrate this discussion in the manuscript and we plan to explore it in future work.\n\n\n### **Bootstrap intervals: why they fall short in sparse intersectional settings**\nThank you for raising this important point. We experimented extensively with non-paramentric bootstrap intervals prior to the submission and decided not to include them because, in our settings, bootstrap often fails to offer any advantages over either the frequentist or the Bayesian approaches. However, we agree that it would be better to include a discussion about this point in the paper anyway.\n\nWe highlight now some characteristics of the bootstrap approach. First, in moderate to large subgroups, bootstrap intervals converge both theoretically and empirically to the same Wald limits delivered by our analytic CLT. At that scale, bootstrap therefore adds only computational overhead without any improvement. In the small $n_s$ intersectional slices where analytic Wald intervals break down, bootstrap inherits the same weaknesses. Resampling with replacement in class unbalanced scenarios frequently generates samples in which one of the four contingency cells is zero (e.g., no predicted positives for group $S$), producing degenerate or undefined estimates of $SP$. Moreover, the resulting empirical distribution is highly discrete and strongly skewed, which yields uncentered confidence intervals. For these reasons, we opted for the analytic CLT, whose validity can be proved, and for the Dirichlet–multinomial credible interval, which remains calibrated even when cell counts are in the single digits. Nonetheless, the camera‑ready version will include a dedicated appendix comparing our method with the bootstrap approach, documenting the pathologies above and clarifying the regimes in which bootstrap is a viable alternative.\n\n---\n\nWe hope these clarifications address the Reviewer’s concerns.\nWe would be grateful to engage the Reviewer in a fruitful discussion and to provide further details if needed."}]

Author rebuttal:

Meta-review: The paper tackles the important problem of fairness auditing in machine learning using a size-adaptive hypothesis testing framework that addresses the statistical challenges of evaluating fairness metrics across demographic subgroups of varying sizes. The authors propose a unified approach that switches between a Central Limit Theorem-based Wald test for large subgroups and a Bayesian Dirichlet-multinomial method for small intersectional groups, aiming to replace ad-hoc threshold-based fairness assessments with statistically rigorous hypothesis tests that provide type-I error guarantees.

The reviewers and authors engaged in substantive discussions across multiple dimensions. Reviewer sptv gave an accept rating (5) but raised technical questions about the scope of fairness metrics covered by the framework and challenged the characterization of statistical "rigor," noting that asymptotic tests provide approximate rather than exact guarantees. Reviewer W2en gave a borderline accept rating (4) and requested empirical validation for conditional fairness metrics and sensitivity analysis for the minimum support parameter, leading to productive exchanges about the framework's generalizability. Reviewer L4Jh gave a borderline accept rating (4) and questioned the technical novelty and requested comparisons with bootstrap methods, prompting the authors to provide detailed analysis showing bootstrap's limitations in sparse data regimes. Most critically, Reviewer ujF4 gave a borderline reject rating (3) and argued that the theoretical contributions were straightforward applications of standard statistical methods, representing more of an educational note than a novel research contribution, though they acknowledged the paper's clarity and technical soundness.

Based on the review discussions, I recommend acceptance because the work addresses a genuine methodological gap in fairness auditing practice. While Reviewer ujF4 correctly notes that the individual statistical techniques are well-established, the unified framework that adapts between asymptotic and Bayesian approaches based on sample size represents a meaningful practical contribution to the fairness-in-ML community. The authors gave detailed rebuttals that existing approaches like bootstrap methods fail in precisely the sparse intersectional settings where their method provides value. The clear consensus among three of four reviewers on the paper's technical soundness, practical utility, and addressing of an important problem outweighs concerns about theoretical novelty, particularly given NeurIPS's stated value for work that bridges theory and practice in societally critical applications.
Decision: Accept (poster)