SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation
Authors: Mikhail Khodak
Lester Mackey
Alexandra Chouldechova
Miroslav Dudík
Venue: NeurIPS 2024 poster (2024)
Keywords: fairness
evaluation
multi-task
transfer
Primary area: fairness
TL;DR: We introduce a method for disaggregated evaluation that improves accuracy when assessing subpopulation performance disparities in ML models.

Abstract: Disaggregated evaluation&mdash;estimation of performance of a machine learning model on different subpopulations&mdash;is a core task when assessing performance and group-fairness of AI systems.
A key challenge is that evaluation data is scarce, and subpopulations arising from intersections of attributes (e.g., race, sex, age) are often tiny.
Today, it is common for multiple clients to procure the same AI model from a model developer, and the task of disaggregated evaluation is faced by each customer individually.  This gives rise to what we call the *multi-task disaggregated evaluation problem*, wherein multiple clients seek to conduct a disaggregated evaluation of a given model in their own data setting (task).  In this work we develop a disaggregated evaluation method called **SureMap** that has high estimation accuracy for both multi-task *and* single-task disaggregated evaluations of blackbox models.  SureMap's efficiency gains come from
(1) transforming the problem into structured simultaneous Gaussian mean estimation and (2) incorporating external data, e.g., from the AI system creator or from their other clients.  Our method combines *maximum a posteriori* (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE).
We evaluate SureMap on disaggregated evaluation tasks in multiple domains, observing significant accuracy improvements over several strong competitors.

SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation
Mikhail Khodak Princeton University mkhodak@cs. cmu. edu
Lester Mackey, Alexandra Chouldechova, Miroslav Dudik
Microsoft Research
(lmackey, alexandrac mdd (mddudik/mmiccosoooooooooooooooo com
Abstract
Disaggregated ietiaatiionessiiattiin of performance of machine learning model on different subpopulations--ss core task when assessing performance and group-fairness of systems. A key challenge is that evaluation data scarce, and subpopulations arising from intersections of attributes (e.g., race, sex, age) are often tiny. Today, it is common for multiple clients to procure the same Al model from model developer, and the task of disaggregated evaluation is faced by each customer individually. This gives rise to what we call the multi-task disaggregated evaluation problem, wherein multiple clients seek to conduct disaggregated evaluation of a given model their own data setting (task). In this work we develop disaggregated evaluation method called SureMap that has high estimation accuracy for both multi-tasé and single-task disaggregated evaluations of blackbox models. SureMap's efficiency gains come from (1) transforming the problem into structured simultaneous Gaussian mean estimation and (2) incorporating external data, e.g., from the Al system creator from their other clients. Our method combines maximun a posteriori (MAP) estimation using well-chosen prior to- gether with pross-validation-fre free tuning via Stein's unbiased risk estimate (SURE). We evaluate SureMap on disaggregated evaluation tasks in multiple domains, observing significant accuracy improvements over several strong competitors.
1 Introduction
Evaluation is key challenge in modern AI, with much effort spent deciding what metrics measure, with which methods, and on what data. This challenge is especially acute in fairness assessment, which requires not only high-quality data to run model and score its outputs but also demographic in- formation for defining groups. Due to the high cost of obtaining high-quality evaluation data, the issue of sample cially when we want to release not just one overall measure but instead to output a disaggregated eval- uation that captures variation among demographic subpopulations of the data [Barocas et al., 2021]. For instance, we might want to assess group fairness by examining the variation in performance across groups of users defined by intersections of the demographic attributes age, race, and The naive ap- proach of independently evaluating each group's performance on its own data can fail because the sam- ple sizes of intersectional groups rapidly decrease Recent work has shown how to improve upon naive methods by combining data from multiple sub- populations inform their individual performance estimates [Miller al., 2021, Herlihy al., 2024].
size needed get good performance estimate remains salient, espe-
consider more attributes [Herlihy al., 2024].
In today's 's technology landscape is common for multiple clients to procure the same model (e.g., an automated speech recognition language model) from an AI developer, with each client performing a disaggregated evaluation of the same model on their own data. We refer to this problem as the multi-task disaggregated evaluation. We formalize and study this problem, showing that can improve the disaggregated evaluations of individual clients by using multi-task data in the form of
Work done part while Microsoft Research and supported part by a CMUTCS Presidential Fellowship.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

summary statistics from other clients or from the model provider. Our approach uses the (out-of- distribution) multi-task data to set the parameters of multivariate normal prior and then performs maximum posteriori (MAP) inference on the (in-distribution) client data. Formally, we model the problem as Gaussian mean estimation and design a simple-yyt--epprssivv additive prior that can capture many different relationships between subpopulations. Drawing upon classical statistics, we fit prior parameters by minimizing Stein's unbiased risk estimator [SURE, Stein, 1981J. While motivated by multi-task considerations, we show that our method also performs well in the single-task setting.
1.1 Contributions
1. SureMap: We introduce a method that uses SURE to tune the parameters of a well-chosen Gaussian prior before applying MAP estimation. The prior is motivated by its attainment of good sfffciency-epprssiiiie
tradeoff, requiring only linear (in the number of subpopulations) number of parameters to recover several natural baselines for disaggregated evaluation.
2. Datasets: Disaggregated evaluation has few benchmarks [Herlihy et al., 2024], so we intro- duce new ones for both the single-task and multi-task settings, covering automated speech recognition (ASR) and also tabular domains (with linear models and also in-context LLMs)
3. Single-task: We find that SureMap is always competitive with strong baselines from prior work, while improving significantly in some settings with intersectional sensitive attributes.
4. Multi-task:
Incorporating data from multiple clients into SureMap yields significant improve- ments across all evaluated settings. This multi-task approach is more accurate even with just one additional task and is the only method to consistently outperform the naive and pooling baselines.
1.2 Related work
Disaggregated evaluation is a core task in the fairness assessment of Al systems [Barocas et al., 2021]. Past work has sought to improve estimation accuracy by combining information across different groups, e.g., via Bayesian modeling [Miller et al., 2021]. Gaussian process approximation of loss surfaces [Piratla al., 2021], and structured regression [Herlihy et al., 2024]. The last work found that classical Jumes-Stein-tyee often competitive, and we adopt it as our first non-naive baseline. We also compare structured regression itself, which turns out to have a tight mathematical connection to SureMap; indeed, apart from our use of Gaussian (ridge) rather than Laplace (lasso) priors (regularization)) as well as our use of a more flexible tuning based on SURE rather than cross- cross- validation the method of Herlihy et al. [2024] can be viewed as the discriminative counterpart to our generative approach (see $E for details).
mean estimation [James and Stein, 1961, Bock, 1975) is
Within the disaggregated evaluation literature we are the first to formulate and study multi-task disaggregated evaluation. This is important direction because (a) model providers often have their own data or data from multiple clients that can inform the evaluation and (b) transferring information across distributions is a key way to handle very low-sample regimes. We also contribute several datasets that we hope will spur further development in disaggregated evaluation.
SureMap relies on applying classical mean estimation tools quantities modeled Gaussian means. Notably, Miller et al. [2021] model scores via well-studied distributions- Gaussians- since scores are related non-linearly to metrics it is unclear if this can lead to similarly simple estimators. To tune parameters, we use SURE, a popular statistical Specifically, in the empirical Bayes tradition, approach [Li, 1985, Donoho and Johnstone, 1995]. set the MAP estimator of hierarchical model. Using SURE to tune the scale of an isotropic Gaussian prior was shown to be asymptotically (in the dimension) optimal in the case of heteroskedastic data distributions [Xie et al., 2012]. Since disaggregated evaluation data is highly heteroskedastic due to variation in group size, this is positive evidence for our approach, although our prior is non-isotropic and has many more variance parameters.
2 Setup
We first describe the disaggregated evaluation problem ($2.1), recast it as a Gaussian mean estima- tion ($2.2), and motivate multi-task variant ($2.3), all while introducing several baselines estimators.
2.1
Setting and baselines
We want to assess a predictive model p: p::2 + y under some distribution over input space X and output space )' using error measure f: 3 x y + R. For example, in image classification, D is dis- tribution over (image, label) pairs and is the 0-1 error. To simplify notation, we mainly deal with the composite function f(z) E(y. p(z)) acting on points = (x, 3) in the product space Z = X x V
2

---

In disaggregated evaluation, Z is assumed to be union of d 2 1 disjoint subsets gg, each associated some subpopulation or group g E [d], where we use [k] to denote the set {l...., kj. As a running example, suppose each point = E E is an individual whose sex and age are categorical variables with d1 and d2 possible values, respectively. Then d d1d2 and each point has an associated index g (s 1jd2 + denoting its intersection of sex E [d,] and age E [dy]. The task is then to use set S D of n 2 i.idd samples from the distribution D to estimate the vector of true subpopulation enrors μ E Idd with components Mg Di,,, [f)] ! We write ((S) E R for an estimator of l with components denoted as B (S).
Empirically, we measure estimation accuracy, i.e., the performance of the performance estimate, via mean absolute error (MAE) (f())) 2 (S)) fi l1. which is easy to interpret and less
sensitive outliers than mean squared error (MSE). Our method development, however, is based on a count-weighte version of MSE, where we denote group counts as ng = |S n Zgl:
())A d - -
(1)
We conclude the setup with two baselines. The first the naive estimator returning group means:?
jigaive (S)
Σ f(z).
(2)
fg =ESnZ,
While unbiased, navve can perform poorly on groups with few samples. The second baseline, the pooled estimator, returns an identical quantity- the overall sample mean for all groups:
1 Σ f(=) n EE
d
pooled g
(S)
n nggg
naive (S).
This estimator generally biased (unless all group means are equal), but it can perform well in law-sample regimes thanks to much lower variance.
2.2 A Gaussian model for disaggregated evaluation
We use a simple but natural model to aid in the design of disaggregated evaluation methods. Specif- ically, denoting the naive estimator by y f (S) E Rd, we model group g's entry Vg as being drawn from a Gaussian with (unknown) mean jı and (known) variance o²/n2, /mg, where o² is shared across groups. This reduces the problem of disaggregated evaluation, as defined in $2.1. to that of es- timating the mean of a multivariate Gaussian with known diagonal covariance gg.g = o² /ng given single sample y NNH, E). Our model has many advantages in the disaggregated evaluation setting:
1. By the central limit theorem, y is asymptotically normal with mean and diagonal covariance Σ for many distributions D of interest, even when the underlying data is non-Gaussian. Furthermore, because the methods derived from our model only take y and Σ as input, they can be applied even when the evaluated statistic not the pointwise average assumed by the setup in $2.1, so long as y ~ N(H, Σ) holds asymptotically. An example of this is when yo = A Amavve corresponds to the area under the ROC curve (AUC) computed over group g data S n 2, [Lehmann, 1951]; we demonstrate SureMap's 's applicability to AUC empirically in $G (Figures 10 & 12).
While shared variance a strong assumption, it is perhaps the simplest way of incorporating the inductive bias that Eg., will be highly correlated with the inverse of ng. the number of samples from group g. In practice, set o² to be the pooled estimate - Eg= Exesnz, (f(2) y²)
Gaussian mean estimation one of the best-studied problem in statistics, with numerous well-tested baselines and approaches for developing new methods. In particular, we make significant use of the classic James-Ster approach [James and Stein, 1961, Bock, 1975].
SURE [Stein, 1981], and empirical Bayesian estimation methods [?].
4. In the multi-task setting, clients are likely to be unwilling to share their actual data but possibly more willing to share group summary statistics. Thus methods developed for our Gaussian model-which only require the group means y, group counts and estimate of o"-will be more broadly applicable than methods that act directly on the dataset S C Z.
We assume existence the first (and, in $2.2, of the second) moments of f(z), 2 ~ Zg. across all g E [d]. fipulldd in the next section, assume ng 0 g.
If ng 0 we let jiaive fall back pooling (Eq. 3), i.e., i2 p
3

---

This model can also be naturally extended-usiee a non-diagonal covariance E to disaggregated evaluation with non-disjoint groups, e.g., to simultancously estimate performance both for all women and for only women in their forties. In the interest of brevity we focus on the disjoint group setting.
2.3 The multi-task setting
We can easily extend this model study multi-task disaggregated evaluation, in which for each task t = I.....T T (c.g., a client of the model provider) we observe a set St C Z of n, samples from the task distribution D,. The goal is then to output T vectors fhe that are close on-average to the tasks' subpopulation errors jtgg = DE [f()] E Σg)- Converting to our Gaussian model, we observe 1" vectors Y: ~ [ur, where we set En.. = o² ///e/g Re:g for some globally shared o² and task-specific group count vectors n, E 2 -nd must output mean estimates fiel (y(()))=))) E IRd,.
We consider two natural multi-task baseline estimators. The first is the global naive estimator (or global estimator for short), which combines the data from all tasks, computes a single global vector of group averages, and uses it as the estimate for each task:
T
T E
p globall ((S.)))) = = naive USS.
((r))L1) P nalloall
Σ E
yv.
(4)
While low variance, the global estimator ignores variation across tasks. Our second baseline the multi-task offset estimator shifts the global estimate on each task to ensure that the task's 's pooled mean is preserved (thus accounting for the variation in pooled means across individual tasks):
protiee ( ((x)) 0 & (y: 0),
where
θ Aooloall (Ex))
(5)
3 Methods
In the last section we reduced the problem of disaggregated evaluation to that of estimating a mean Rd given a sample y ((,, Σ), where E is known and diagonal. We now design method, SureMap. for the latter problem. Our technical approach involves the following two steps:
1. Choosing parameterized mean estimator. We use the MAP estimator under a multivariate normal prior that we design specifically for intersectional subpopulations.
2. Tuning the estimator's hyperparameters. We use SURE estimate the quality of our estimator, which we then optimize over the choice of hyperparameters using the L-BFGS-B algorithm.
3.1 Designing a parameterized estimator
As mean estimation is a vast area, we use three criteria for designing an estimator: should (1) dominate baselines such as ganive and pgooled, (2) have relatively few hyperparameters: and (3) handle heteroskedasticity stemming from variation in group sizes. One natural source of candidates are Stein-tteeeee shrinkage estimators: the original James Stein estimator famously dominates fanivv in MSE and has no hyperparameters to tune [James and Stein, 1961], satisfying our first two desiderata. Furthermore, while James and Stein [1961] assumed an isotropic Σ, subsequent estimators such as the following variant of an estimator due to Bock [1975] do handle heteroskedastic Σ.3
Bock (y) 0 1
(1
2 Σ 0) )) (y 0),
yy
where (.)4 = max( 0}. and 0 E Rdd is a default estimate towards which y is shrunk.4
However, empirically we find that this often underperforms the pooled estimator in low-sample regimes; further, the form of f Bock shows that the amount of shrinkage towards coordinate g E [d]. despite intuition suggesting that should shrink less for groups g with more samples. Corrections to this tend to be involved and difficult to generalize [Efron and Morris, 1973].
the same for each
We thus turn to a different family of well- Gaussian mean estimators: those that return the mode of the posterior distribution assuming fr is sampled from the conjugate prior W/ A) with mean O E dd and dositive-definite covariance A E gdxd (e.g., Gelman al. [2014, Equation 3. 12]::):
MAP A (y) (A Σ -
Σ '-yy).
FFeldman et al. [2014] show that using in numerator instead Bock's Tr performs better: in $B we [[] show that their modified form can be derived by minimizing an upper bound on the E D -eighttd MSE
We typically use 0 0, but in $B derive variant shrinking towards 0 p (y))

---

Algorithm 1: Single-tane task SureMap.
Frati
Frots
Fathot
Huti-dd
Firtd
(For multi-task SureMap see $D.)
Input: target f: R. samples S C 2, partition {2g] J of Z, each group g an intersection of Zyo attributes
Myaa
Mydo
H.45
M.25
// compute naive group means for group g E {d do n < |S E
F/104
Fa6d
My ng E EEnnz, (())
A(T) Cp + rua Csex
Tagu Cage
Tuuss.uge Crexago
// estimate group variances
8 9--6 E (f(z) ya)²
E- + diag(n)/o²
F,025
F,129
// compute auxiliary matrix
M4-22
M,-28
Method (r):
// compute prior covariance
M.2-64
(matrices CΛ are defined in
F,6d
F,6dd
$3.1.1 and $C.1)
< EACRY r2Ca
Output: (La +AE
Figure 1: Matrices CA {0, 1]dxd whose linear combination defines the covariance ( of our additive intersectional effects prior (Eq. 9). In this example, there are two categories for sex (F ,)) and three for age (<25, 25-64, >64). yielding d 6 groups. Shaded squares are 1s, unshaded are Os.
1/ optimize SURE using L-BFGS-B
f arg mig | 2 (TT)) +C220
// estimate group means using MAP Output: y A(f)y
MAP naturally handles heteroskedasticity by weighting low-variance coordinates g more heavily, satisfying our third criterion; however, its most general form has (dd) hyperparameters, violating the second. We next use problem structure to restrict the number of hyperparameters needed define A while still allowing i A
MAP to express every baseline introduced in 12.1, satisfying our first criterion.
3.1.1 An additive intersectional effects prior
For brevity, we build up our estimator somewhat informally; full description is in SC.11 1. We return to our simple example where each group g E [d] corresponds to an intersection (s, a) E [di] x [d2] of two attributes: a sex [d1] and an age [d2] A simple prior that additively incorporates individual attribute effects into intersectional group means the following:
Mg TBS + Tacx Cee TagoSa age + TeexageSg + 0gy
(8)
where 0 Rd and Tu, Tsex Tage: Tsexage Rg0 hyperparameters, ~ NN(0, 1) is scalar effect shared across all groups, the vector goox Rdd has its sth entry <% (O shared by all groups g whose sex the vector gave E dR has its ath entry _ N shared by all groups g whose age is a, and the vector Ç ~~ AO,,,, Ia) contains an independent noise term Ey for each group g.
The hyperparameter To quantifies how much we expect all of the means to be shifted (by shared positive or negative value) from the default Hyperparameters Tsex and Tags express how large we expect contributions of sex and age alone be towards the means. And finally, non-zero Tsex,age gives the prior flexibility to model heterogeneity across intersectional groups.
Given the vector of hyperparameters - (70...., Tsxx.age) E Rd the prior can be written more compactly as μ -N(0,, ((T).. where the covariance
20:
A(T) Σ
CC = C + - Csex + Tage Cage + Tsex.age Csexage
(9)
AC (C(eaxage)
for matrices CA 10. jaxd s.t. each entry CA:a.A is one iff groups g and h agree on the attributes included in A. particular, we have that Cg 1dxd the all-ones matrix. the entries Clex.g, h of C.es E (0, d are one iff groups g and h share the same sex attribute, the matrix Cage is analogous, and C.exage Ia the d identity. This structure visualized in Figure 1.
5

---

3.1.2 Efficiency and expressivity
As detailed in S,.1, this prior can be naturally extended to any number of attributes & using covariance matrix (T)) E gdxd specified by vector E R of 2 hyperparameters. Since S [log, dj, the total number hyperparameters (including E RRd)) is d 2* O(d), which much smaller than the ( complexity of the general case. We can further reduce this by fixing the entries of 0. constraining them to be identical, or setting them using external (e.g., multi-task) data.
Despite this reduction in hyperparameters, we can show that for suitable choice of т, the estimator AMMP (e))(T) recovers many estimators of interest, including the naive estimator and the (possibly offset) pooled estimator (see $C.2). This means that MAP with our structured prior should be able to outperform all four baselines from the previous section, if appropriately tuned.
3.2 Tuning by minimizing expected risk
Having specified a parameterized estimator, there remains the question of setting its parameters and One might want to treat this as hyperparameter tuning problem and use data-splitting approach; however, the dimensionality of the problem makes standard techniques either expensive or noisy, and data splitting introduces additional randomness and design decisions into already data-poor environment. We instead make continued use of our Gaussian assumption and turn to SURE, which given differentiable estimator IRd Rd returns unbiased estimate of its weighted MSE I. using sample data y ~ (1, Σ):
0² d (I((y) - y1 d + 2 aly)). R (y)
(10)
where given any W Odxd denote ||x|iw (x, Wx) Vx Rd Using SURE we can tune the parameters p by minimizing Rn (y) in a manner similar to empirical risk minimization.
3.2.1 Single-task SureMap
In the single-task setting, we fix 0 O& and tune the variance parameters T R Letting
A(T) = (A ( A (T), we define the single-task SureMap estimator as
SM (y) (O) (y) (Ia A(F))y
(11)
for 7 = arg min -
(12)
TERSO
MAP The optimization problem in the second line comes from substituting (()) into SURE (Eq. 10). is nonconvex, but we find that can be quickly solved to sufficient accuracy with L-BFGS-B [Byrd
et al., 1995], standard method for bound-constrained optimization of differentiable functions.
3.2.2 Multi-task SureMap
To generalize SureMap to the multi-task setting we propose to specify Ô and + by minimizing SURE aggregated across tasks, i... Σ Rp; (yr). While setting both parameters via direct optimization of this objective is the most straightforware approach, we find that it performs worse than single-task SureMap when there only few tasks (T < 5) and rarely improves significantly above the multi-task global and offset estimators. This can be explained by observing the few-task limit-i.e.
T= 1-in which case optimizing the aggregated SURE objective results in setting i yı and thus makes the multi-task estimator equivalent to the naive estimator.
We find that a better approach is to treat the choice of Ô as its own simultaneous mean estimation problem and apply the SureMap approach to it. In particular, our model y, ~ ((,, Σ,) and our prior μ4 ~ (e, A) imply that the samples y; ~ (N A Σ,) have mean @ and known covariances (apart from tuning parameters). Therefore, the MAP estimator of 0 itself given a hyperprior e ~ N (Os, I) Σ,) 'y
with covariance r 0 will have the form θ (I-1 ΣT (A + (,)
Ei,
To reduce the number tuning parameters, we use prior the same form as before by specifying T= A(v) for v E IR22 q. i.e., the same structured covariance as described in $3.1.1 but with separately tuned parameters (see $3.1.1 and SC.1). Substituting the meta-level MAP estimator of into the MAP estimator of μe and tuning the parameters T and by optimizing the sum of SUREs (Eq. 10)

---

across tasks yields our multi-task SureMap estimator (see $C.3 for details):
((ve(() (o(40). (xt) =y- + (O( - y.)
MAP
(13)
T
for 0(T,1)) = Σ M)
Ax(T) = 10--A-1100 (A-(()) + + (7):
1=1
T
M)(T,)) A (v) (v)
Σ (A(T) (1,))=22))) (A(T)+ + =Z)
+
==1
-) (2T - -a)] (14)
0 aarg min T.EER t=1
The optimization problem on the last line can again be approximately solved using L-BFGS-B.
3.3 Limitations
Various modeling assumptions impact performance of SureMap. For instance, the Gaussian error assumption is less appropriate when errors are heavy-tailed. In $G, Figure example of a target metric with heavy-tailed observation errors. We find that SureMap still performs well, but is longer superior previous approaches. One avenue for improvement would be to a variance-stabilizin transformation (e.g., Hawkins and Wixley [1986]) prior to applying SureMap.
consider MSE an
Note that SureMap achieves its improved accuracy by shrinking naive estimates towards less granular estimator (e.g., a pooled mean). As a result the estimation is biased towards less disparity, which could lead to overly optimistic conclusions about faimess. For this reason it is extremely important to examine not just the point estimates, but also confidence intervals. These can be obtained, for example, by viewing SureMap regression approach ($E) and leveraging inference techniques for regression.
Datasets
We evaluate our approach in several representative settings for disaggregated evaluation, including two tabular settings appearing in previous works [Miller et al., 2,21, Herlihy et al., 2024, Liu et al., 2024], and three new settings: a multi-task tabular setting based on state-level U.S. census data and both single-task and a multi-task ASR evaluation setting.
4.1 Tabular datasets
We consider three tabular datasets, two for the single-task and one for the multi-task setting, covering two important domains where fairness concerns can arise: healthcare records and demographic data. While we focus on the classification task and 0-1 error, SG we also report results for regression (Figures 8 & 9) and for classification with AUC as the target metric (Figures 10 & 12).
Diabetes. This is tabular dataset of Strack et al. [2014], containing around 100K patient records with six race, two sex, and three age categories. We evaluate logistic regression classifier trained to predict patient disposition after hospital stay (discharged or otherwise). The target metric the 0-1 error.
Adult. We use the classic Adult census dataset [Kohavi, 1996] to evaluate performance of an 1lama-3-70b-
in-context LLM caanee--seccfficalll or less than $50K after being provided with eight examples via a modification of the prompt template of Liu al. [2024]. The target metric is the 0-1 error, disaggregation is by race, sex, and age.
predicting whether person makes more
State- ACS (SLACS). This is a tabular dataset for multi-task setting derived from the census data for all U.S. states and Puerto Rico assembled by Ding et al. [2021]. Each datapoint corresponds to a person in one of nine race and two sex categories; we consider three age categories: below 25, 25-64, and over 64. The underlying task is to classify each person as earning either more or less than $50K. We train a regularized logistic model on the data from California, and seek to evaluate its performance on the other 50 states/teritories, which comprise the tasks. The target metric is the 0-1 error.
4.2 ASR datasets
We also introduce both single-task and multi-task speech recognition datasets, based on applying the popular Whisper ASR model [Radford et al., 2023] speciffccll whisper-tiny- the En- glish part of the Common Voice (CV) dataset [Ardila et al., 2020], which contains utterances from individuals in one of nine age and three sex categories.
7

---

Diabetes Classification (0-1 error)
Classification error)
Diabetes Classification (0-1 emori
naive
0.64
MAX Jarggn greAzON)
pooled
Matcomal gruuzi
1,20
3.2
Bock
3.50
Struct. Reg.
0.00
SureMap
3.0
9.5
0.0
003
3.01
825
0,13
0.814
601
0:507
Adult Classification (0-1 error)
Adult Classification (0 1 error)
Adult Classification (0-1 error)
0.10
nalve
022
pooled
MAKE Medcarge argn gro.ppp--
Bock
0.00
Struct. Reg.
SureMap
6.0-
0.7
Common Voice ASR (WER)
Common Voice ASR (WER)
Common Voice ASR (WER)
0.14
nolve
3.22
Medcarge gre a0
poaled
nattsmall crousd)
3.00
Bock
Struct. Reg.
SureMap
0.30
9:30
4.9
0.0
000
12:
(K3)
11611
30
122)
012
15K0
331
subsammping Imedian group size]
subsampling rate (median group size)
subsampling rete (mediaa group size!
Figure 2: Single-task evaluations on Diahetes (top, disaggregating by race, sex, and age), Adult (mid- dle, disaggregating by race, sex, and age), and Common Voice (bottom, disaggregating by sex and age). The MAE is averaged across all groups (left), large groups (center), or small groups (right). Large and small groups are defined as above and below median group size.
Common Voice. This is a single-task dataset obtained by combining the validation and test partitions of the CV dataset. We calculate the word-error rate (WER) across all the utterances of each individual, which becomes the target metric to be predicted.
Common Voice Clusters (CVC). This is multi-task ASR dataset. To construct it, we first cluster the utterances in the train partition of the CV dataset into 20 clusters by applying k-means to the sums of GloVe word embeddings [Pennington et al., 2014] of their corresponding texr strings. To model task relatedness, we then randomly reassign each utterance to a random cluster with probability [0, 1]. The resulting clusters are the tasks. The target metric is the word-error rate (WER) across all the utterances of each individual
given cluster. In most experiments we use a = 1. we also investigate what happens when varies between zero, i.e., the original clusters, and one, corresponding to identically distributed tasks.
but
2.
Evaluation
Our main metric is MAE relative to ground truth vector, which we take to be the mean of all available data for each subpopulation g E [d]. except those with fewer than 40 samples. In our main results we subsample with replacement from the entire dataset at different rates and track performance as function of the sizes of the resulting datasets. To obtain 95% confidence intervals we conduct 200 and 40 random trials at each subsampling rate in the single-task and multi-task settings, respectively.
5.1 Single-task
We compare SureMap to the naive (Eq. 2) and pooled (Eq. 3) baselines, as well as to the Bock estimator with shrinkage towards the pooled estimator (Eq. 17) and the structured regression estimator of Herlihy et al. [2024]. On both Diabetes and Adult, SureMap significantly outperforms all competitors (Figure 2, top middle), the greatest improvement is on subpopulations with limited data. In SG, we consider regression variant of Diabetes and observe similar results (Figure 8). On the Common Voice task, SureMap performs roughly similarly to Bock, while outperforming structured regression some subsampling rates (Figure 2, bottom); here again the gains are driven
8

---

State-Level ACS Classification (0-1 error)
task-level results (median grp. size 26)
0.16
naive
MT Back
0.14
pooled
multipl pallicaaivee improvement in MAE over nalve
MT SureMap
task-averaged MAE
0.12
Bock
Peneee naive
Reg.
0.30
SureMaa
global
0.00
offset
0.08
MT Bock
SureMoo
0.01
0,032
0.316
(3)
88)
(26)
(83)
(252)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Common Voice Clusters ASR (WER)
task-level results (median grp. size
21)
0.05
3.5
naive
MT Bock
0.08
pooled
multiplicative improvement in MAE over naive
SureMap
task-averaged MAE
0.07
Bock
nalve
Reg.
Sureklap
0.06
global
0.04
MT offset
Bock
0.03
SureMap
0.00
1.5
0.01
0.032
01
0.316
100
1.0
(21)
(57)
(210)
0.2
0.6 0.8 1.0
subsampling rate (median group size)
distance from ground truth MT median
Figure 3: Multi-task evaluations on SLACS (top, disaggregating by race, sex, and age) and CVC (bottom, disaggregating by sex and age). Left: Performance across different subsampling rates. Right: Multiplicative improvement in MAE over naive estimator on individual tasks; subsampling rate=0.1.
by better performance on small groups Pooling performs best when data is extremely limited, but it is not competitive with even modestly more data, and also underperforms on small groups.
5.2 Multi-task
In the multi-task setting, we use all the single-task methods as baselines while adding multi-task (MT) ones, including MT global (Eq. 4), MT offset (Eq. 5), and an MT extension of Bock (Eq. 6) in which 0g is set using the average across the group data on all other tasks. We first consider the SLACS task, for which Figure 3 (top left) shows that MT SureMap significantly outperforms other methods in the low-data regime while matching the best one (MT Bock) in the high-data regime. At subsampling rate 0.1, Figure 3 (top right) also shows that using multi-tast data leads to improvement of the fifty tasks, that the reduction in MAE over the naive estimator on a typical task is 2x, and that this improvement only loosely correlates with the task's 's ground truth distance from the multi-task median. On the other hand, it also shows that while MT Bock's ' improvements are typically smaller, on SLACS it improves performance for every state (including the two where MT SureMap is worse).
all but two
On the CVC task, the MT offset baseline is the most competitive, except at the lowest subsampling rate where pooling is better and at higher subsampling rates where it stops improving with additional data (Figure 3, bottom left). SureMap outperforms and all other methods across subsampling rates and its advantage is greatest in low-data regimes, where it even outperforms pooling. In the tesk- evaluation in Figure (bottom right) we see that on every task, MT SureMap attains an improvement of 2-3.5x over the naive baseline and almost always outperforms MT Bock. Furthermore, the latter performs substantially worse on tasks whose ground truth vectors are far away from the multi-task center while MT SureMap is not affected.
5.3 Ablations
We next look at how the degree of included intersectional effects and multi-task structure affect performance. In Figure 4 (left), we evaluate utility of including higher-order interactions in the struc- tured prior. We implement SureMap variants with up to fth-order interactions by setting TA for |A| E (f+1,.... k- } to zero (but not for |A| = k). We observe that including zeroth-order effects (E 0) single-task Mur (i.e. shrinkage to pooling) improves upon the usual single-pane parammee Gaussian prior (L = -1) in low-data regimes. Adding first-order effects (L 1) leads to substantial further im-

---

State-Level ACS Classification (0-1 error)
Common Voice Clusters ASR (WER)
0.16
0.050
naive
task-averaged MAE
0.12
0.14
SureMap (1)
SuneMap (-1]
pooted
task task-aves -esk-vveraedd MAE
0.060
0.045
pooled
Bock
Reg.
0.30
SunwMap (2)
0.035
SaraMap
MIl global
MT global
0.00
MTT SureMap (:1)
0.030
MT offset
0.08
MT SureMap
MIT SureMap
0.025
MT SureMap
MT Bock
MIT SureMap (2)
0.020
0.02
0.015
0.01
0,032
0.35 0.316
1.0
(3)
(8)
(26)
(83)
(252)
0.0
0.2
0.4
0.6
0.8
1.0
subsampling rate (median group size)
task similarity
Figure 4: Left: Comparison of SureMap variants on SLACS, The SureMap (f) variant sets to zero the
entries of T corresponding to interactions of size > l (except for the highest-order interactions). Right:
Evaluation of different methods as the interpolation coefficient that defines the CVC tasks is varied.
State-Level ACS Classification (0-1 error)
Common Voice Clusters ASR (WER)
nelve
0.12
pooled
0.033
nalve
Bock
task- task-aver lllavveraedd MAE
pooled
Reg.
Bock
task-averaged
Surehlap
Struct. Reg.
global
0.08
MT offset
0.025
MT global
SureMap
Bock
MT offset
0.08
SureMaa
0.020
Bock
MT SureMap
0.04
0.015
10
20
50
30
15
20
number of tasks
number of tasks
Figure 5: Performance as the number of tasks varies, evaluated on SLACS (left) and CVC (right).
provement, but second-orded effects (f - 2) make performance slightly worse. A similar effect be observed among the multi-task variants, except there is no loss (but also no gain) to using the highest- order variant (L 2). Overall, this study suggests that using the full-order SureMap is reasonable default but that most of the method's effectiveness comes from zeroth- and first-order effects.
Figure 4 (right) tracks performance as the task-similarity parameter defining the CVC task is varied. MT SureMap outperforms all methods at all settings and is also not as strongly affected by the task similarity, at least as it is defined for the CVC data. This suggests that the structured prior use may be useful even if the dataset means are quite different but the underlying evaluation problem (in this case estimating WER of ASR models) is the same.
Lastly, in Figure we study how the number of tasks affects multi-task performance. On both SLACS and CVC, MT SureMap outperforms all single-tas task baselines (the best one being the single-task SureMap) at = 2 tasks, i.e.. can take advantage even very little external information. In contrast, on CVC, the competitor multi-task methods (e.g.. MT offset and MT Bock) do not even outperform single-task methods until 2 5 tasks. These results demonstrate that, unlike these comparators, multi- task SureMap can be confidently used even when only one additional client's worth of data is available.
6 Conclusion
We have introduced SureMap, a disaggregated evaluation approach, which combines MAP estima- tion under a structured Gaussian prior with hyperparameter tuning via SURE. SureMap achieves substantial empirical improvements over strong baselines in both single-task and multi-task settings. Valuable future directions include improving robustness to heavy-tailed data and developing multi- task methods that can handle client privacy concerns. More broadly, we hope our work will have positive impact by allowing model users to more accurately identify fairness-related harms, the first step towards mitigating them. However, using SureMap and any other disaggregated evaluation approach must be done with care, so to not risk overconfidence in a model's 's faimesss.
10

---

Acknowledgments
We thank Nicolas Le Roux for discussions and feedback at the beginning of this effort.
References
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common Voice: A massively- multilingual speech corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, 2020.
Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wort- man Vaughan, W. Duncan Wadsworth, and Hanna Wallach. Designing disaggregated evaluations of Al systems: Choices, considerations, and tradeoffs. In Proceedings of the 2021 AAAI/ACM Conference AI, Ethics, and Society, 2021.
Mary Ellen Bock. Minimax estimators of the mean multivariate normal distribution. The Annals of Statistics, 3(1):209-22))) -28. 1975.
James R. Bunch, Christopher Nielsen, and Danny C. Sorensen. Rank-one modification of symmetric eigenproblem. Numerische Mathematik, 31:31-48, 1978.
Richard H. Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5): 190-1108, 1995.
Corinna Cortes and Mehryar Mohri. AUC optimization vs. error rate minimization. Advances in Neural Information Processing Systems, 2003.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring Adult: New datasets for fair machine learning. In Advances in Neural Information Processing Systems, 2021.
David L. Donoho and Iain M. Johnstone. Adapting to unknown smoothness via wavelet shrinkage. Journal of the American Statistical Association, 90(432):1200-122 1995.
Bradley Efron and Carl Morris. Stein's estimation rule and its competitors-an empirical Bayes approach. Journal of the American Statistical Association, 187341):117-130, 1973.
Sergey Feldman, Maya R. Gupta, and Bela A. Frigyik. Revisiting Stein's paradox: Multi-task averaging. Journal of Machine Learning Research, 15:3621-3666 2014.
Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. Bayesian Data Analysis. Electronic Edition, 2014.
Douglas M. Hawkins and R. J. J. Wixley. A note on the transformation of chi-squared variables to normality. The American Statistician, 40(4):296-298, 1986.
Christine Herlihy, Kimberly Truong, Alexandra Chouldechova, and Miroslav Dudík. A structured regression approach for evaluating model performance across intersectional subgroups. In Pro- ceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024.
Roger A. Horn and Charles R Johnson. Matrix Analysis. Cambridge University Press, second edition, 2013.
Willard James and Charles M. Stein. Estimation with quadratic loss. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, 1961.
Ron Kohavi. Scaling up the accuracy of Naive-Bayer Bayes classifiers: A decision-tred tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, 1996.
Sören Laue, Matthias Mitterreiter, and Joachim Giesen. Computing higher order derivatives of matrix and tensor expressions. In Advances in Neural Information Processing Systems, 2018.

---

Erich Leo Lehmann. Consistency and unbiasedness of certain nonparametric tests. Annals of Mathematical Statisties, 22(1):165-179, 1951.
Ker-Chau Li. From Stein's unbiased risk estimates to the method generalized cross validation. The Annals of Stafistics, 13(4): 1352-1377, 1985.
Jun S. Liu. Siegel's formula via Stein's identities. Statistics & Probability Letters, 21:247-251, 1994.
Yanchen Liu, Srishti Gautam, Jiaqi Ma, and Himabindu Lakkaraju. Confronting L.MS with traditional MI: Rethinking the fairness of large language models tabular classificattions. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024.
Andrew C. Miller, Leon A. Gatys, Joseph Futoma, and Emily Fox. Model-based metrics: Sample- efficient estimates of predictive model subpopulation performance. In Proceedings of the 6th Machine Learning for Healthcare Conference, 2021.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 2014.
Vihari Piratla, Soumen Chakrabarti, and Sunita Sarawagi. Active assessment of prediction services as accuracy surface over attribute combinations. In Advances in Neural Information Processing Systems, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Ro- bust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Sidney Siegel. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, 1956.
Charles M. Stein. Estimation of the mean of a multivariate normal distribution. The Annals of Statistics, 96):1133-11511 1981.
Beata Strack, Jonathan P. DeShazo, Chris Gennings, Juan L. Olmo, Sebastian Ventura, Krzysztof J. Cios, and John N. Clore. Impact of HbAlc measurement on hospital readmission rates: Analysis of 70,000 clinical database patient records. BioMed Research International, 2014, 2014.
Qing Wang and Alexandria Guo. An efficient variance estimator of AUC and its applications to binary classification. Statistics in Medicine, 39:4107-43499 2020.
Hilde Weerts, Miroslav Dudk, Richard Edgar, Adrin Jalali, Roman Lutz, and Michael Madaio. Fairlearn: Assessing and improving fairness of Al systems. Journal of Machine Learning Research, 24:1-8, 2023.
Xianchao Xie, C. Kou, and Lawrence D. Brown. SURE estimates for a heteroscedastic hierarchical model. Journal of the American Statistical Association, 165550)):146-111 1469-1479, 2012.
12

---

A
Notation
For p E [1, oo] we use I1-Ilp to denote the p-norm on Rd.
We use
-|Il to denote the spectral norm Rmx*
For any positive semi-definite matrix A ž Oaxd we use the notation ||-I|A IR R20 to denote
the vector norm
| Axx Ax) |
E Rd
For any positive integer k we use [k] to denote the set kl.
For any S we use 2 denote its powerset.
For any vector a E R2 we use a, to denote its ith entry. If is only defined as an expression then we will abuse notation and use (a); to refer to its ith entry.
For any vector E IR and any subset
[k] with elements 81 aem) ) to be the vector of the entries of a whose indices correspond to the elements of S sorted in ascending order.
8m define
as = (as,
For any subset S C [k] we assume X iterates over the elements in ascending order.
For any matrix A Rmxxn we use Ai, denote its (i, j)th entry and A,, its ith row. If A is only
defined as an expression then we will abuse notation and use (A)
j.s to refer to its (i. (ith)th entry.
For any k-tensor Z Rx with dimensions di,..., dg Zyo and any vector X [da] of indices
Ze to refer the the (cI,...;Ck entry of Z.
We use Om, mm E R and 0 0mxn: 1xx E mm to refer to all-zero and all-one vectors and matrices, and refer the n x identity matrix.
B Heteroskedastic James--Sein-tyy shrinkage estimation
For reference we state weighted version of Stein's 's unbiased risk estimate (SURE) [Stein, 1981]:
Lemma B.1. Suppose y Nu,, , Σ) for mean μ Rd and diagonal p.s.d. covariance Σ E Rdxd and consider a function Ed + Fd s.t. for every [d] the function Pa is almost differentiable in Yi and we have E. V ((y))l co. Then for any diagonal matrix W Rdddd we have Bl( =ll (y) -yllw Tr(WE) 2E (WEEpyy))
Proof. Expanding the squared norm and applying Stein's Lemma [Stein, 1981, Lemma 2] to the last term yields
E Ply) wlll = EEl - yll + Elly w + 2B(p(y) y, W(y - -4))
(15)
= =EllĀ(y) - yllw + Tr(WE) + 2E 2 (WEA(y) - -2Tr(WE))
now turn to James-Stein-type estimators of the form fa(y) = y bI P(y b) for
E RR, 0 E Rdd, and P E Rdxd, roughly extending the one considered by Bock [1975] (in a more
general form) to + La and b * Og- Setting x P(y 0), we apply Lemma B. show that the expected Σ -weighted MSE is
2
2c Tr())
4c Tx T Σ 'P)
E| p(y) 1113. d+E
|x|3
|x||
|
2c Tc
4c Tr xxx
E 3-
d
(16)
X X-1
|x| x-1
c² Σ 2c Tr
EP)
4c
1 1
< d
x
where to compute the derivative in SURE we made use of the matrix calculus tool of Laue et al. [2018] and the last line follows by von Neumann's 's trace inequality. This upper bound is minimized
Tr(P) 2 - Note that for I, this setting of exactly recovers the Bock estimator f Bock in Equation 6, apart from the positive-por correction.
13

---

On the other hand, for shrinking to the pooled mean p (y) known fact about the eigenvalues of symmetric rank-one updates (e.g. Bunch al. [1978, Theorem IJ)
1ax4 y
TH(2 2) we can apply well-
11 T---1
diagonal matrices to obtain || P
I
which yields the setting
in Feldman et al. [2014] of c = Tr(P) - 2 Σ PE = =d - 3:
p Bock (y) = fiPooled (y) + y papooled (y)
d-3
(y f poooled (P)ooledl))
(17)
Lastly, we note that it is straightforward to extend SureMap to the case of non-diagonal covariance matrices Σ, i.e., when we want to simultaneously release performance estimates for groups with different numbers of intersections (e.g., just race, just age, and both race and age). In this case there will be nonzero covariance between elements of the vector y, c.g., those corresponding to a specific age bracket and an intersection of that bracket with specific race. To handle this, one only needs to derive the SURE objective estimating the (non-diagonal) Σ Σ-1 -weighted risk of the MAP estimator with (the same, non-diagonal) covariance Σ; this be done with the following generalization of Lemma B.I:
Lemma B.2. Suppose y ~ (H, Σ) for mean μ Rdd and p.s.d. covariance E E Raxd, and consider a function ÿa Rd - Rd s.I. for every i [d] the function ùa a.e. difserentiable in yi and V y fuly)lli
Then for any p.s.d. W Rdxd have
d
A(y) - Mll = Ellp(y)) yllw + T((WE) 2E
ΣV O (WVyA(y) W))ss (18)
i=1j=1
where fly) is the Jacobian of f with entries VyngA(y) 0y, (,(y).
Proof. Note that since Ey = we have by a multivariate version of Stein's 's lemma (c.g.. Liu [1994, Lemma 11) that the following identity holds for any a.e. continuous f Rd IR s.t. [(y)] < oc V E [d]:
B[(y fy)l(y)] = Cov((y my).(())) E. Vyf(y))
(19)
Using this equality with f(y) (W y))s yields
(yl - 2 Bly) - yll + Elly - -lll + 2B(p(( - y. W(y - 14)
d
ElIA(y) vllw + Tr(WE) + 2E Σ EE (IWW(()))-(WW>) -
Ell ylw + Tr(WE) + 2I
E En (Wi., W..))
(20)
i=1==
d
d
A(y) yllw + Tr(WE) + 2E
((W.. v,yū(y)) Wes)
i=1 ==
d d
= EEI yl + +T<WE) + 2E
O (WV,A(y) W))as
14

---

C SureMap estimator
In this section we describe the prior in full for any number of attributes, prove expressivity results reference in $3.1.2, and describe how to compute the estimator using coordinate descent.
C.1 linear prior
Suppose each group g [d] corresponds to intersection g [==](da] of attributes, with
attribute a having da possible classes. 5 For example, the first attribute could be one of dj different age brackets and the second could be one of d2 different racial categories. For each subset A C [K] of attributes define random tensor ZA E IR Au with i.id. entries ZAic ~ (O,,, 1), where c E X A [da] is a specific class combination of the attribute A. Then the additive intersectional effects prior on the mean Hg of group g is the weighted sum
Mg 0g + Σ T.ZA
(21)
AE2I
with coefficients тA R corresponding to each A E 2 These hyperparameters thus determine the strength of the effect of attribute intersection A on the group means, with TạZ being added to the means of all groups g s.t. SA c, i.e. whose attribute intersection results in the specific class combination
Letting т = (To...., 7n)) E RR2 be the vector of hyperparameters тл and assuming Tкk > 0, this prior equivalent Gaussian prior μ NO, A(T)) with covariance
A(T) = Σ - Σ CA
(22)
4
AE
for matrices U E [0,1]xX[l.ea da with orthogonal column vectors, each one corresponding to different attribute intersection in Xaea! [da] and its gth entry indicating whether group is a subset of that intersection. Their outer product matrices CA UAU have entries CA:g.h that are h and zero otherwise, i.c. they indicate if groups and h have the same type (c.g. (senior, female)) of attribute intersection A (e.g. (age, sex)). In the simplest case, if we are disaggregating across just one attribute, i.c. h = and d1 = d, then A(T) = Fdla + 2 Tij Since k S [log2 d]. using this prior to define the covariance matrix reduces the number of hyperparameters to d 2* < 2d for MAP compared to (dd)) ( for general covariance A. )
C.2 Expressivity of the linear prior
Theorem C.1. Consider a disaggregated setting with k attributes with di,... da possible categories, respectively, and total number of groups d = - da- If we use diagonal covariance Σ E RRdxd satisfying En.k = o² /nn v h then the following holds V E R.d:
1. TA V A + [k]] and 0 E Rd then lim MAP
THK (O)(T)(() ) f f.uaive
2. TA 0 A A E (0, [k]]
MAP then lim
(O)AA(T) (y) (xpooled(y)
+00
THe
3. TA
A A + K and 0 E Rdd
then
lim
MAP (.A(T) (y)
THe
and 0 E IRd then lim 4. TA 4 A E ( }
(.A(T) (y) fapooled (y -0)
MAP
700
-A)
as desired.
$ Note that this does reduce the generality of our basic setup, which can be recovered with 1 and di d.
15

---

Proof. The first and third results can be easily shown using the fact that Σ is diagonal:
1
TMe 700 (
MAP lim
lim +00
L
+
+
E
by
y fi:uiiee (y)
(23)
[R
[A]
-1
0
TNN + (
MAP (v) lim
lim TM
2 [
+
2
+
+ y
0
(24)
[(k
The second result follows from the last by substituting 0 one.
02 we just need prove the latter
First note that
P (OT( A = (A-)) (1
(25)
- (A + Σ (y - 0) (y- just need to show that the second term approaches faooked (y -0). formula to compute
(26)
0). We use the Sherman-Morriso
Ia
A ( +1
(27)
+ 3
7h
[k.
and apply it again to compute
-1
L
Te 1
(A-* -
+ +
+ TBd
[
[M]
1
1e1
4 +
(28)
1
La
[[4]
[
[h
+
2 [o]
70
-8 Tr
Σ-1
[6]
Defining y 0, we have by L'Hôpital's rule that g [d]
lim TV 7000
( + -ーートーー
Ti T0
5n/ En/En.h
dg 2 9.
[e]
1=1 1 Eh,h
Tel
= lim
19.9
Ta 700
TIK
TM
1
Eg.9
+ Tod d
Σ
33
h 1 Eh,h
lim
dn A
T/6/ /Eg.
bn La
(29)
33.9
[()
En.a
T/
1+ S4,
1+ Vig.o
[Nl
1=1
TM
lim
78 7200
lim
T
Σ
Da.n (1t) +m,n)
710 +00
h=1
d
lim
nn (yn - 0%) = ppooled (y 0)
-
+000
h=1
as desired.
16

---

State-Level ACS Classification (0-1 error)
Common Voice Clusters ASR (WER)
nalve
naive
0.12
pooled
pooled
+ SureMeo
task task-avee aakkaagveeeee MAE
SureMap
plottal
global
task-averaged
MT offeet
NT. offfee
Bock
Bock
0.08
SureMap
0.025
SureMap
MT SureMap (SureSalve)
MT SureMop (Sure5olve)
0.08
(MetaMap)
0.020
SureMap (MetaMap)
0.04
10
20
30
40
50
30
15
20
number of tasks
number of tasks
Figure 6: Performance as the number of tasks varies, evaluated on SLACS (left) and CVC (right). This plot similar to Figure but demonstrates the superiority of the multi-task version of SureMap that we choose (MetaMap) relative to alternatives.
C.3 How to set the multi-task center
In the single-task case we set the prior mean = Od; this can also be done in the multi-task setting, Alternatively, we can use multi-task data to construct estimator of some true underlying multi-task mean and substitute the former for 0; for simplicity we will restrict to the ourselves to linear estimators θ- Σi M,y, where the matrices M....., My are independent of .I.....y7 To determine these matrices, we will assign them some parametric form and set those parameters by minimizing the sum of the E, -weighted risks across tasks, as estimated by SURE:
T
MAP (x.) (A Ex A
Mells
Σ
EE
(IA (x.)) - -vll 2V MMA (x))
1=1
(30)
T
dT
+
E, A,
yo) (-y +2TT(A.M A.)
t=1
for A, = (A-1+ E, 1
A ie.
a
MAP (y:) y A.
+ A, y1) = yt A.ly; -I M.y.).
C.3.1 SureSolve
This approach sets the prior mean by finding the 0 that minimizes the above SURE objective, which equivalent to solving an overconstrained linear system:
i = dl' + arg min BERd
|A.(0 y 2TH(A.)
AAT A.
Σ
A A.y.
(31)
Note that this estimator can be expressed in the desired linear form Ô = Σ M,Y with the
matrices M, ATE, A, Σ A, depending on the parameters determining A.
C.3.2 MetaMap
Alternatively, if we assume the prior is correct for some O and A, i.e. N00, + Σ,), then natural estimator to use for 0 given a fixed A is MAP estimator of Or with priorr/(O... A(O,,, I') for some covariance T. Define ΣA- = (EL (A + E,) - and note that YA Σn ΣT (A + Σ,) (o)))ly:.. y, distributed as ( ). Then the MAP estimator of 0
T
θ = (T-1 MイA)
E Y
(A + -
(32)
has the necessary form 0 Er M.y. for M, (T- matrices depend on both A and r. i.c. the covariance matrices of the prior and the meta-prior.
アン Σ,) In this case the
17

---

Algorithm 2: Multi task SureMap.
Input: target f: Z R. samples S 2 for each task 1...., T, partition (2g]g=1 of Z, each group an intersection of E Zyo attributes
// compute naive group means
for task & E [T] do
for group g [d] do
Re: 1 g
Vt:g
Lessnz f(z)
// estimate group variances
Σt= |
0² t n-d7 2i=1 2g=1 Eesiinz (f(z) V1:a)
for task LE [T] do
Σ,¹ +- -iiag(.)/oo
// compute auxiliary matrices (avoids inverting A) Method (r)):
// compute prior covariance (matrices CA are defined in Appendix C.
A - Eeeul CACA
Output: (Ia + AE,
// compute auxiliary matrices (avoids inverting Σ, and r)
Method (T v):
// compute prior and meta-prior covariances
AE2( r
+ Aczle U CA
Output: (I+rET E, (AZ," (.)-1)+ TE + +1,)
// estimates prior mean using MAP
Method (T, w):
L Output: Σ M,(T,
// optimize the sum of SUREs across tasks using L-BPGS-B
A, 0 arg min E (T) vill + 2'1 (A.(7) La))
T.eER. 20
// return an estimate of the group of each task f using MAP task E [T] do Output: y: (O()) y.)
D
Computation
Here we note additional computational details.
D.1 Nonnegativity of SureMap
In both the single and multi-task cases there is no guarantee that 0 is in the convex hull of the values y or (yr)? and in fact can negative. Since the quantities we are estimating are usually nonnegative, in practice we do a post-hoc correction forcing to be nonnegative.
D.2 Optimization of SureMap
Both the single-task and multi-task variants of SureMap require solving optimization problems, the former over ? (72) AC(M) (Eq. 12) and the latter also over 2² ( (Eq. 14), both of which are vectors in IR We find that both problems can be quickly and efficiently optimized using L-BFGS-B over the entire domain R and R22 respectively, using the default settings provided in
18

---

its SciPy implementation. 6 To initialize the algorithm we set all entries of 7 and v² to zero except those corresponding to the entire set [k], which we set one.' In the remainder of this section, we describe how to compute gradients (passed to L-BFGG-BB) BB) of the multi-task SureMap objective
dT
Er
A:
Yt
M.Ys
+2'TYA,M A.)
(33)
Σ
where the matrices M, differ based on whether we are using the SureSolve or MetaMap variant. The gradients are taken w.r.t. the tuning parameters, which are the coefficients -?...., : used define
the prior covariance A- Σ 1im1 of U.U.U. and, in the second case, the coefficients u ...... 2iz used Ei of U.T. Noting that A, (A+E, 1 'A- (Ia+
define the meta- meta-prior covariance
AE, ) and (A + _,) = A [I + E = Σ; (La+ AE, 1)-1) Σ, A.. we have the derivative O,A, = A,U,U, "; A. w.r.t T.. which yields 0 Tr(A;) = Tr(-A,; M At) and
0.1 (1)),,) 2(0 y.) AA yY)
(34)
=-2 - y.) A, yi)
D.2.1 SureSolve
First note that
0, T Ad] = - A7 = A + -AA))
(35)
=-Σ, + A.)UU ET - (A} + A.)
where the first step follows by Σ, A II AI 1 (Ia+ E 'E, - A,
-1
Then for M, =
ET= A E As)
A E.E. A. we have
T
T ΣATEFIA A,
-1 T
a.M. =
Σ A
A! A.
Σ
a. E.(ATE.'A.IM.
Σ E.'AA A.
'A, -A.,UA+A)UUU +A)J +Ai)
T
22 Σ A!E A,
E A, A.)M,
T -22 + 2
M
(36)
where B EET, AJ A!
Σx (A) A.)UU E. ' + +At) We can then compute
0 Tr(AM -2TH A, B BB.....) 1) - -TAAAU,UUE, A.Mi)
s=1
(37)
T Σ
-r
A
Bi, +
U.U. A
Bi,
M
"nttps://docs. www.sscccccccccooooeeooooooooooe wwgg/bbc/scipy///eeeeeeco http://eeeeeeeeeee/eeiieeeeee -iiiinsze-llfsb html "This initialization corresponds to setting the prior covariance to be the x identity.
We use indices i instead of subsets A here for simplicity.
19

---

and
T
T T θ,
a
At
Yt
Σ M,ys
Σ
Σ (yr
M.y.) A, Ex E A.(y, M,y,)
s=1
E
#=1 r=
T
2 Σ (O.MY.)) AT 0)
+ 2(y, A O.A 0)
T
4
y. B.B.
A C,,,,AA06 6)
(38)
40
B T A E. 0)
2(y, 0)
A M, A.U A, 0)
T
4 > (y. 0) B E, 0)
s=1
2(y: 0) A, E, A.U.U? E A.ly. 0)
D.3 MetaMap
Note that (A + Ex)-1= = (AE, L4) E,'A 'A so we can write the matrices representing
the estimator as M, = =TT-1 1 + Σ,- = (T-1 + Σ- Ar Thus
- 0A - -T--+++)
Σ A,A.M.
$=1
(39)
-MM.UU T.U A A, Σ M.
We can then compute
0 THA,,, - THA + A.M.J.U,E;;))
Σ TAAMMUUU E.AAA A.M.)
(40)
20

---

and
T T
0,
At
Yt
Mays
= ΣV Σ
yy M,y.) A, A M,y,)
8=1
s=1r=1
= = Σ (OM.y)) AT 6r)
+ + - A ÔT)
T
2
y. M.U.U, E
#=1
2 M, A,M,) A,E,'A Qr)
-
2(yr T E, A.U.U A., 6r)
T
2 D (y,
E θr)
s=1
2(y - ④r) T A Σ A.U,U E A.,y. θr)
(41)
Lastly, note that M, (T - ~ A. =L 1 TE, "A, and redefine the
derivative 0 be w.r.t. Ug. so that
O,M, - -,, M. + カー Σ; A.
(42)
Then we have 0 Tr(A,M,) = Tr(A, (E (2) A, Σズ M))) and
0, A Y1 0r) || E
= - V
(a.My.)) T TA A.,, A Qr)
(43)
1=1
=2 1- Σ E A.y,
+
- - 0r)
D.4 Handling groups with no data
is frequently the case that a specific group may not have any examples, i.e., ng = 0. and so we cannot define Eggg a² /ng. At the same time, we may need to handle the dimension associated with this group, either because we still need to report a value for it or because we are in the multi-task setting and other tasks do have examples of that group that may allow us get estimate. To handle this issue, in all computations we only use the precision matrix Σ 2-1 which can be casily defined in the case of ng using 2 ng/ Note in particular that (A+ E) - Σ -AE +1) 9.9
D.5 Handling near-singular covariances
Since would like to optimize over the entire domain 72 R22 but A Oaxa singular, we avoid inverting prior covariance matrices (i.e. A, I') in all computations. Note in particular that
A, -A-1+ -- (I. AΣ-1 AΣ-1)-1,
21

---

D.6 Handling group variances for AUC
While obtaining unbiased variance estimates for averages is straightforward, it is more involved for multi-point statistics such as AUC. For simplicity we just use (ng + 1)/ fl ) where ng the number of members group 9 with label i: this estimate derived from the variance estimate of the related Mann- SoinnWWitney U-statistic [Siegel, 1956]. However, future work may consider improvements based on more complicated approaches [Cortes and Mohri, 2003, Wang and Guo, 2020], or using bootstrapping techniques is done by Herlihy et al. [2024] for structured regression.
(0) (1)),
E Derivation of SureMap estimators via ridge regression
In this appendix we show that f and f can be derived as ridge regression estimators.
Similar to the notation in Appendix C, we consider l sensitive attributes (like age, etc.) indexed by E [k]. The ath sensitive attribute is denoted ga and takes values [da]. The joint sensitive
attribute g takes values in G [d]] [d2] x... x [ds] with the cardinality d | d1d2 ... dk- For A C [k]. write gA for the tuple (ga)
E.1
Single-task regression model
Each joint attribute g G identifies intersectional group (of order k). We seek to jointly fit means of all these groups, represented vector la E R based on the vector of empirical means E R GI,
We posit regression model
where E R| is feature matrix and E RIJT a vector of regression coefficients. The columns of + are referred to as features and indexed by J, where J is some index set. We assume a Gaussian prior on the parameter 3 ~ (O, K) and Gaussian distribution over observation errors, so y N($3, E), where K and E are, respectively, the prior covariance matrix and error covariance matrix.
The еπrог covariance matrix Σ is assumed fixed and positive definite, the prior covariance matrix K is viewed we assume that K is positive defimite (but that assumption is not necessary). Any generic forms of T, K and Σ can be considered. Here we exhibit specific form of and under which the MAP regression estimator is the same as fiSM, when provided with the same error covariance matrix Σ.
hyperparameter (with a specific structure reduce its dimension); for simplicity,
We consider structured form of matrix + with features being indicators of tuples of sensitive attribute values. The features are indexed by j E J, where
J ((A,c) A C [k], c [lae
and defined
(A,cA.c) 1{g c}.
(44)
We will also consider subsets of features that focus on specific subset of sensitive attributes A C [k]]
=== = (AAcc) [IaeA[da))
and write 4A [I for the submatrix composed of features indexed by j E JA.
The prior matrix K is diagonal, specified using a set hyperparameters T = (TΛ)ACI)) with diagonal entries
K(A.c).(A.)) = =
(45)
The MAP solution under the model described above obtained by solving
i arg min
(y
.33 M . + 3 K 1g] 1-100
which is a weighted ridge regression problem. Setting the gradient to zero, the solution must satisfy
24 Σ (y
+ 3 0.
22

---

Hence,
B ( Σ + K 1 Σ-
This yields a regression- estimator
~
2
D Σー + +K - y
(46)
In contrast, by Eqs. (11) and (7), the pam estimator is obtained as
a SM
(A -
where the matrix A (following Eq. 22 in Appendix C.1) depends on the hyperparameter vector as
A = Σ AUA
AC
where UA is precisely the submatrix PA defined above. Thus, writing AA,c 1219 for the column of & indexed by (A, c), we obtain
A
2
Σ Σ
B = $KΦ
AC
AC
cE Tal
Hence, the fiSM estimator can be rewritten as
f
(47)
Theorem E.1. With and K defined as above (Eqs. 44 and 45), we have ===== P. =
In the proof variant of Moeerrann-Mrrsso--oooooe
formula [Horn and Johnson, 2013, Eq. 0.7.4.1J, specialized to symmetric positive definite matrices:
Proposition E.1 (Symmetric SMW Formula), Let A E R and R E mmxm be symmetric positive definite matrices and let X E " Then
1A+XRX A + X A
A
Proof of Theorem E.I. suffices to show that
Φ Σーーー + ((*K ) +
We do this by direct calculation, using the symmetric SMW formula:
2 Σ +
lok K
[ 2K* ) $K o
($K4
(
( (BKKT)) (
= ((KK 1) 1)-
The first equality follows by the SMW formula with A K-1 R= and X TT. the second
by multiplying out the terms, and the third by the SMW formula with A- $K R Σ and X equal the d x d identity matrix.
E.2 Multi-task regression model
Consider multi-task setting with tasks indexed by / 1...., T. We write T [T' for the set of tasks. Multi-task setting can be reduced to single-task setting by viewing the task as an additional sensitive attribute, and performing the same analysis as for the single-task setting, but with G' =T x G, with the dimension = | = Td.
Formally, we seek to fit μı' E I9' based on the vector of empirical means y' 19'1, The entries of µı' and y' are denoted as pigg and vl.g for the task and the intersectional group g. We denote task-specific slices of these vectors p x and y¹ v'
23

---

|J] defined as Features are indexed by elements of J' = S x J, where S = T U {glob), so there are task- specific and global features. The feature matrix & E pl uses the single-task feature matrix + E ||||]]]| (defined in Eq. 44) as a building block. The matrix o has block structure, with [T| x [ blocks of size 9 x | defined as
O 0 gi
if - or glob,
Φ
otherwise.
We posit the regression model
# D'p',
where 3' E RJ With definition of ' as above, this boils down to for E ET As before, we assume a Gaussian prior and Gaussian enrors, 3 ~ ~'(, ~ ~N'''',
14g = (3, (lab)
The error covariance Σ' E IR 9'Ix|9"I has a block-diagonal form with single-task error covariance matrices Σ, E || for E [T], along the diagonal.
We consider structured form of the prior covariance matrix K' specified by two vectors of hyperpa- rameters: = (TA)AG()) and v (UA)AC(K)- The matrix K' diagonal and positive definite, with entries
T s E T. UA if s = glob.
K ((..Ae)) =
can also be viewed as block-diagonal matrix with |S] diagonal blocks of size |J]
if s T. K
K
V
if glob.
where K is the single-task prior matrix defined in Eq. (45), and V R|JX*|3 is analogous matrix based on the vector hyperparameters v rather than T, with diagonal entries
V(A.O).A..)) = UA.
(48)
The multi-task regression-based estimator is obtained by solving the resulting MAP regression problem. Similarly to the single-task case, the estimator takes form
Tey $b' 4p'
4 + k'
(49)
with the individual task estimates denoted p!!T
The multi-task SureMap estimator, introduced in $3.2.2, takes form
SM
y'+ 1,+++1,1 A (
(50)
where
T
i
r- + Σ AA -
Σ A + E,)-y'
(51)
t=1
and
A = $
and
T= $VΦ
Theorem E.2. With D', K' and E' defined above, we have p 10g
p.o SM for all t E [T]]
In the proof we use the following identities:
Proposition E.2. Let A, E Ru be symmetric positive definite matrices and let the n n identity matrix. Then
(A + 11-1111 -
=I -I-AAA- B(A+B)*
-I-
24

---

Proof. We have
(A- A B) A -
=I- -I-A(A+B]*
(A + () + -
= -BA +
where the first equality follows by the SMW formula (Proposition E.1), and the remaining equalities follow by simple algebraic manipulations.
Proof of Theorem E.2. Starting with Eq. (49), we have T
f क
E
4 + K'
Φ TT Σ'
(''K''') ) + ) E'y y
= =(1-E<("''0 T + Σン "")"'''''
(52)
where the second equality follows by the same reasoning as in the proof of Theorem E.1, and the third equality follows by Proposition E.2.
We next evaluate 4 'K' D'K'sTT Note that the matrices &', K' and have the following block structure:
K
Σ1
Φ'
K'
Σ'
K
Σn
Thus,
K
Φ'K' T
+
V
K
ΦKΦ
ΦV (I ..... I)
ΦKΦ
A
+
r I) = A' X
where in the last line we introduced the notation A' for the block- diakoddaaonnl matrix with T copics of matrix A along diagonal, and notation X for the matrix obtained by stacking copies of the [g] x [ identity matrix I on top of each other. Plugging the last expression back into Eq. (52), obtain
nog
I E' A +XXXX
XTX +
-y yy' E (N' 1 +
(A' + (M'+E'
r +x (A++ X (A' (
Σ¹) -'--->1-- -11 (53)
where the second equality follows by the SMW formula (Proposition E. with A = A' + A'+ERR
and X = X. We next focus simplifying the last term in the bracket in Eq. (53).
Since A' and E' are block- diagonal, the matrix (A + E')-1 is also block -iikonaa with blocks along the diagonal equal to (A + Σ,) for = 1...., T. Thus,
(A (A+E,)- -1
T AA
X (A' - (I I)
:
Σ tm1
-.)-1,
(A + -1
25

---

and similarly,
T y' Σ
X' +
A -yy
[=1
Therefore,
T +x (A' + Σ -
X (A' + +EE
r - - Σ
(A E+) (A+ -
Plugging this back in Eq. (53), obtain
n - -'-'-'((''+""'>>'" -A''+ =y -E'A'' + E') y
i
Using, again, the fact that matrices A' and E' are block-diagonal, the task-specific blocks of j must be equal
F nog
=y, E,(A + (y' -
= - (A د. - "y' 0) - ) A (Ō y') - -pr
where the second equality follows by Proposition E.2.
F Resources
F.1 Data and model resources
We make use of data / models with the following sources / licenses:
Strack et al. [2014]: CC Attribution License.
2. Weerts et al. [2023]: MIT License.
Ardila et al. [2020]: CCO License.
4. Radford et al. [2023]: Apache-2.0 License.
https://archive.. ics. wui. edu/ dtta///attaast//adutt:: CC 4.0 License
6. https://huggingface co/JaaackXD/Llanaaa
GGUF: Meta Llama 3 License
use the third and fourth resources to create a dataset Whisper model evaluations on Common Voice utterances, which result in the Common Voice and CVC tasks described in $4.2: we also use the last two resources to create a dataset of in-context evaluations of Llama 3 on the Adult dataset, which results in the Adult task described in $4.1. Both resources released under a CC BY 4.0 License and are available at https: //gttttttt.t//////////// com/ http///oook//ueeeaaaaa
Computational
By far the most computation was required to generate the Common Voice, CVC, and Adult tasks, which was done on a machine with two RTX-8000 GPUs and took about a week. As described above, the corresponding datasets are made publicly available and easy to re-use without any GPU access. Given these dataset, the main experiments were run on a 40-core machine and take a couple hours, with the vast majority of this time spent rumning the structured regression approach of Herlihy et al. [2024]; see Figure for a summary of the costs associated with each method evaluated this paper. Code for both generating the task data and reproducing the method evaluations is available at bstps://go thub. com/ mkhodak/ Aakhodak/SSrreMaa
26

---

State-Level ACS Classification
Common Voice Clusters ASR
10a
paoled
10
pooled
10
Bock
Bock
cost (seconds)
Struct. Reg.
cost (seconds)
Struct. Reg.
10a
SureMap
SureMap
global
10-1
g global
offset
10a1
Bock
MT Bock
MT SureMap
10"
MT SureMap
10.
10
5
10
2
11
20
number tasks
number of tasks
Figure 7: Cost of running the methods evaluated in this paper as a function of the number of tasks, with both axes sealed logarithmically. While they are 1-2 orders of magnitude more expensive than the seeeiines-wiee all have closed form Sxpressions-Sureeee orders of magnitude than structured regression [Herlihy al., 2024]. Note that for the most part the runtime of all methods will usually be dwarfed by the cost of inference.
and multi-task SureMap are also
Diabetes negression (MAE)
Regression
Regression
1. naive
MAETINg grauge)
pooled
Bock
Struct. Reg.
MAE DaEEtmall
SureMap
01
0.23
8,330
8-
8.65
0003
35
0.315
134011
0.31
6.32
0.8
E354
(201
subsampling rato (median group size)
Subsampling rate (median group Si20]
subsampling rate (median group size)
Figure Evaluations on the regression variant of Diabetes, using MAE as the target metric, disaggregating by race, and age. On the left the MAE is taken across all groups, while in the center it is only over large groups and on the left over small groups. Large and small are defined as the top and bottom half of all groups, respectively.
Regression (MSE)
Regression (MSE)
Regression
naive
pooled
MAE
Struct. Reg.
SureMap
0.2
6.50
8-3
Eutro
111
2.507
100
subsampling rate (meeiin group size)
subssmpling rate Imedian group size)
subsampling rate (median group sizei
Figure 9: Evaluations on the regression variant of Diabetes, using MSE as the target metric. disaggregating by race, sex, and age. On the left the MAE is taken across all groups, while in the center it is only over large groups and on the left over small groups. Large and small are defined as the top and bottom half of all groups, respectively.
G Additional evaluations
In Figures 8 & 9, we compare evaluation methods on the regression variant of the Diabetes task, where the goal is to predict patient's length of stay using ridge regression. In Figures 10 12, we compare evaluation methods compare evaluation methods on Common Voice and CVC with the character error rate (CER) as the target metric. And finally, in Figures 14 & 15, compare evaluation methods on Diabetes, Adult, Common Voice, SLACS, and CVC according to RMSE instead of MAE.
Diabetes and SLACS with AUC the target metric. In Figures 11 & 13, we
27

---

Diabetes Classification (AUC)
Diabetes Classification (AUC)
Diabetes Classification (AUC)
3.44
0.12
naive
3.2
pooled
0.10
Bock
Struct. Reg.
NAElsmoll
022
6.00
SureMap
2.3
9.6
0.6
004
0.30
0.02
0.32
13:40
Imedian group sizc]
Subsampling (median
subsampliig (madian group
Single-task evaluations on the Diabetes classification setting (disaggregating by race, sex, and age) when using AUC as the target metric. In the left column the RMSE is taken across all groups, while in the center it is only over large groups and on the right over small groups. Large and small are defined as the top and bottom half of all groups, respectively.
Figure 10:
Common Voice ASA (CER)
Common Voice ASR (CER)
Common Voice ASA ICER)
0.59
0.99
SMAG graups!
pooled
NAEtomall grouzs)
09r-
Back
Struct. Reg.
0.36
SureMap
0.3
0.0
082
0:22
0.01
subzampling Imedian group sizel
subsampling rate
subsampling (median group
Single-task evaluation on the Common Voice ASR setting (bottom, disaggregating by sex and age) when using CER as the target metric. In the left column the MAE is taken across all groups, while in the center it is only over large groups and the right over small groups. Large and small are defined as the top and bottom half of all groups, respectively.
Figure 11:
State-Level ACS Classification (AUC)
task-level results (median grp. size 33)
naiie
MT Bock
0.12
pooled
multplicative improvement in MAE over naive
MT SureMap
task: averaged MAE
nalve
0.10
Struct, Reg.
SureMap
0.08
NT global
offset
0.06
MT Bock
SureMop
(105)
0.50
0.75 1.00
1.25
1.50
1.75 2.00 2
subsampling rate (median group size)
distance from ground truth to MT median
Multi-task evaluations on state-level ACS data (disaggregating by race, sex, and age) when using AUC as the target metric. On the left is the performance across different subsampling rates while on the right we show (multiplicative) performance improvement over the naive estimator on different tasks at subsampling rate 0.1.
Figure 12:
Common Voice Clusters ASR (CER)
task-level results (median grp. size 22)
0.05
pooled
neive
mulppiccttive improvement in MAE over nalve
3.5
MT SureMap
MT Bock
task. averaged MAE
Back
nalve
Struct, Reg.
0.03
SureMap
MT global
offset
0.02
NT Bock
2.0
SureMep
1.5
(236)
0.4 0.5
subsampling rate (median group size)
distance from ground truth to MT median
Figure 13: Multi-task evaluations Common Voice clusters (disaggregating by sex and age) when using CER as the target metric. On the left is the performance across different subsampling rates while on the right we show (multiplicative) performance improvement over the naive estimator on different tasks at subsampling rate 0.
28

---

Diabetes Classification (0-1 emorr)
Diabetes Classification er-re
error)
Diabetes Classification (0-1 emort
0.200
0.12
pooled
naive
RMSE groups)
0.150
Bock
Struct. Reg.
0.000
SureMap
0.60
0.00
3130
1251
Adult Classification (0-1 emror)
Adult Classification (0-1 emrrr)
Adult Classification orror)
3.54
naive
3.2
soeattaage groups)
pooled
3.00
Bock
Struct. Reg.
SureMap
0.00
100
0.8
0.00
0.2
1951
0.0
190
B601
123001
0.23
151
341
1001
Common Voice (WER)
Common Voice (WER)
Common Voice ASR
0111%
0.16
naive
0.550
pooted
3:56
0.135
Struct. Reg.
Back
3:22
6.00
SureMap
0:30
002
3.3
0.3
2.00
0.314
12001
8.65
0.033
63
0.355
0.01
6,532
3,31
Subsampling rate (median group size)
subsampling rate (median Group size)
subsampling rate (madian group sizel
Figure 14: Single-task evaluations on the Diabetes classification setting (top, disaggregating by race, sex, and age), the Adult in-context classification setting (middle, disaggregating by race, sex, and age), and the Common Voice ASR setting (bottom, disaggregating by sex and age); these are the same evalu- ations as Figure 2 except with RMSE instead of MAE as the performance measure. In the left column the RMSE is taken across all groups, while in the center it is only over large groups and on the right over small groups. Large and small are defined as the top and bottom half of all groups, respectively.
State-Level ACS Classification (0-1 error)
task-level results (median grp. size 26)
0.225
nalive
MT Bock
task aveaagerrdddd RMSE
0.200
pooled
multip multiplicative improvement in RMSE over naive
MT SureMap
0.175
Bock
nalve
Reg:
0.150
Sureklap
global
0.100
MT offset
Bock
0.075
SureMop
0.050
0.025
0.01
0,032
0.1
0.316
1.0
131
10)
126)
(83)
(282)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Common Voice Clusters ASR (WER)
task-level results (median grp. size
21)
task averaged RMSE
0.10
0.12
pooled
naive
Bock
multiplicative improvement in RMSE over naive
3.5
MT SureMap
MT Bock
naive
Struct. Reg.
0.05
Sureklap
global
2.5
MT offset
Bock
2.0
SureMap
0.02
0U1
0.032
0.1
0.31.
1.0
1.0
(23)
(67)
(F10)
0.25
0.10 0.15
0.20
0.25
0.30
subsampling rate (median group size)
distance from ground truth to MT median
Figure 15: Multi-task evaluations on state-level ACS data (top, disaggregating by race, sex, and age) and Common Voice clusters (bottom, disaggregating by sex and age); these plots visualize the same evaluations as Figure 3 except they use RMSE instead of MAE as the performance measure. On the left the performance across different subsampling rates while on the right we show (multiplicative) performance improvement over the naive estimator on different tasks at subsampling rate 0.1.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: our abstract and introduction both describe the introduction of a new method for disaggregated evaluation, which we do.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: c.f. Sections 3.3 and 5.3.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification: c.f. Appendix C.2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: c.f. Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification:
https://githul: hwwp:aawww....////ooooo/aaaaaaaa
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/ /wwpddee/eeeeubbmmiiiimmma
iilyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.cooc wwtppppplcggcpcieececeeeooocopce gwte//gggee///eee CCodeSubmiieeeeece
/tte:////ggiideeeeeeuuuubiiiiiii iariommsssioooooooiiioo (eionnolccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: c.f. Section 5, Appendix D, and the linked code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: all figures except scatter plots have 95% confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
32

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: e.f. Appendix F.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: / http///eerpp//eeeeee/eee/ cc httppp/cccc/ecicoudeeleleeee.eee
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: have reviewed the Code of Ethics and work conforms.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: c.f. Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
33

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The one asset released is just evaluations of an open-source model on open- source data and so unlikely to be misused.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: e.f. Appendix F.I.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
34

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: we release new datasets in conjunction with three of the tasks described in Section 4; see that section and Appendix F.1 for further details. This data is provided at the code link.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codc/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "KzBp0kjERM", "reviewer": "Reviewer_gnhf", "summary": "This paper presents SureMap, a novel method for disaggregated evaluation, aimed at improving the estimation accuracy of performance metrics for machine learning models across different subpopulations. The proposed method is designed to address both single-task and multi-task settings, where multiple clients independently evaluate the same AI model on their respective data. SureMap leverages maximum a posteriori (MAP) estimation with a well-chosen Gaussian prior, fine-tuned using Stein’s unbiased risk estimate (SURE), to achieve high estimation accuracy. The authors evaluate SureMap across various domains, demonstrating significant improvements over existing baselines.", "questions": "1.\tImplementation Details: Can the authors provide more detailed pseudocode or a step-by-step guide for implementing the coordinate descent algorithm used in SureMap? This would help in better understanding and replicating the method.\n\t2.\tNon-Gaussian Data: How robust is SureMap to deviations from the Gaussian assumptions? Have the authors considered alternative distributions, and how would the method need to be adapted for such cases?\n\t3.\tScalability: What are the computational requirements of SureMap for very large datasets or a high number of tasks? Can the authors provide any benchmarks or comparisons in terms of runtime and memory usage?\n\t4.\tReal-World Applications: Can the authors provide examples of real-world applications where SureMap has been or could be successfully applied? This would help in contextualizing the method’s practical utility.", "limitations": "•\tGaussian Assumption: The reliance on Gaussian assumptions is acknowledged, and the authors suggest potential future work on other distributions like Student’s t.\n\t•\tData Integration Costs: The potential burden or cost of integrating data from multiple clients is mentioned, and the authors propose the use of model provider data as a mitigation strategy.\n\t•\tScalability: Although not deeply explored, the authors note the computational challenges and suggest future work to improve scalability and efficiency.\n\t•\tFairness and Over-Confidence: The authors caution against over-confidence in a model’s fairness based solely on disaggregated evaluations and emphasize the need for careful application of SureMap and similar methods.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: The introduction of SureMap for simultaneous mean estimation in disaggregated evaluation is novel, particularly in addressing both single-task and multi-task scenarios using a structured Gaussian prior and SURE for parameter tuning.\nQuality: The paper is well-structured, with a thorough theoretical foundation, clear methodological development, and comprehensive experiments. The approach of combining MAP estimation with SURE tuning is well-justified and effectively demonstrated.\nClarity: The paper is clearly written, with detailed explanations of the methods and assumptions. The inclusion of theoretical proofs and detailed descriptions of the datasets and experiments adds to the clarity.\nSignificance: The ability to improve disaggregated evaluation accuracy has significant implications for fairness in AI, as it allows for better assessment of model performance across different demographic groups. This work has the potential to impact how AI systems are evaluated and deployed, ensuring more equitable outcomes.", "weaknesses": "Complexity of Implementation: While the method is theoretically sound, the practical implementation of SureMap may be complex, particularly the coordinate descent algorithm used for tuning parameters. This could pose challenges for practitioners.\nDependence on Gaussian Assumptions: The method relies on Gaussian assumptions for the prior and noise distributions. In real-world scenarios, data distributions may deviate significantly from Gaussian, potentially affecting the performance of SureMap.\nScalability: The scalability of the method in very large-scale settings is not fully explored. The computational cost associated with MAP estimation and SURE tuning might be prohibitive for very large datasets or a very high number of tasks.\nEmpirical Evaluation: While the empirical results are promising, they are primarily based on synthetic and semi-synthetic datasets. More extensive evaluation on real-world datasets across diverse domains would strengthen the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! We address your concerns and questions below:\n\n1. [*While the method is theoretically sound, the practical implementation of SureMap may be complex, particularly the coordinate descent algorithm used for tuning parameters. This could pose challenges for practitioners. [...] Can the authors provide more detailed pseudocode or a step-by-step guide for implementing the coordinate descent algorithm used in SureMap?*]  \na. As discussed in the general response (Issue 2: Computation), we have updated the parameter optimization scheme post-submission to be L-BFGS. While SureMap is more involved than baselines such as using the naive estimate or the Bock estimator, it is as simple as or simpler than other recent approaches such as Structured Regression (Herlihy et al., 2024) and AAA (Piratla et al., 2021). The pseudo-code for our current method—which we will add in revision—can be summarized in two steps: (1) apply L-BFGS with SciPy’s default settings to the objective in Equation 10 (or Equation 11 for the multi-task case) and (2) run MAP estimation with the resulting mean and covariance.\n2. [*The method relies on Gaussian assumptions for the prior and noise distributions. In real-world scenarios, data distributions may deviate significantly from Gaussian, potentially affecting the performance of SureMap. [...] How robust is SureMap to deviations from the Gaussian assumptions? Have the authors considered alternative distributions, and how would the method need to be adapted for such cases?*]  \na. Please see our discussion of these points in the general response (Issue 1: Assumption). In particular, please note that we do not assume that the prior and the noise are Gaussian; we only assume that the summary statistics are. \n3. [*The scalability of the method in very large-scale settings is not fully explored. The computational cost associated with MAP estimation and SURE tuning might be prohibitive for very large datasets or a very high number of tasks. [...] What are the computational requirements of SureMap for very large datasets or a high number of tasks? Can the authors provide any benchmarks or comparisons in terms of runtime and memory usage?*]  \na. Please see our discussion of these points in the general response (Issue 2: Computation). In particular, SureMap is *not* computationally expensive for large datasets (it relies on cheap-to-compute summary statistics and so scales weakly with dataset size) nor for many tasks (as shown in Rebuttal Figure 2); in practice, we expect the cost of SureMap (less than a few seconds) is dominated by model inference costs. We will include this cost analysis in the revision.\n4. [*While the empirical results are promising, they are primarily based on synthetic and semi-synthetic datasets. More extensive evaluation on real-world datasets across diverse domains would strengthen the findings. [...] Can the authors provide examples of real-world applications where SureMap has been or could be successfully applied? This would help in contextualizing the method’s practical utility.*]  \na. Please note that none of our benchmarks are fully synthetic, in the sense of being entirely generated by artificial distributions, and many do not involve any synthetic aspects at all. For example, our single-task ASR setting involves a widely used model (Whisper) evaluated on a real-world dataset (Common Voice). In terms of specific applications, the Diabetes task is motivated by a real-world use-case (Obermeyer et al., 2019) and has been used in previous disaggregated evaluation research (Miller et al., 2021; Herlihy et al., 2024). At a higher level, disaggregated evaluation is the central step in any fairness assessment (Barocas et al., 2021; Herlihy et al., 2024), so we expect SureMap to be of use in a broad range of real-world settings by AI companies, regulators, journalists, and researchers.\n\n## References\nBarocas, Guo, Kamar, Krones, Morris, Wortman Vaughan, Wadsworth, Wallach. *Designing disaggregated evaluations of AI systems: Choices, considerations, and tradeoffs*. AIES 2021.  \nHerlihy, Truong, Chouldechova, Dudík. *A structured regression approach for evaluating model performance across intersectional subgroups*. FAccT 2024.  \nMiller, Gatys, Futoma, Fox. Model-based metrics: Sample-efficient estimates of predictive model subpopulation performance. ML4H 2021.  \nObermeyer, Powers, Vogeli, Mullainathan. *Dissecting racial bias in an algorithm used to manage the health of populations*. Science, 2019.  \nPiratla, Chakrabarti, Sarawagi. *Active assessment of prediction services as accuracy surface over attribute combinations*. NeurIPS 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jblKVh2daH", "reviewer": "Reviewer_3rXK", "summary": "This paper introduces SureMap, a new method for disaggregated evaluation of AI systems especially for multi-task setting. The authors model the problem as Gaussian mean estimation and use a structured covariance prior that captures intersectional effects. SureMap is evaluated on several datasets, including a new multi-task ASR dataset they introduce. They show SureMap generally outperforms existing methods, especially for small subgroups and in the multi-task setting.", "questions": "1. Have you considered non-Gaussian models? For example, a t-distribution might better handle heavy-tailed performance data that can occur in practice.\n2. The multi-task setting assumes clients are willing to share summary statistics. How realistic is this assumption? Could you explore privacy-preserving ways to share this information?\n3. How does SureMap compare to GP-based approaches like in [1]? The method aims to handle low-data regimes well, which seems relevant here.\n\n[1] Active assessment of prediction services as accuracy surface over attribute combinations. NeurIPS 2021", "limitations": "N.A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper tackles an important problem in fairness and evaluation of AI systems and formulation as a Gaussian estimation problem with a structured prior.\n2. Theoretical analysis showing SureMap can recover existing baselines.\n3. This paper introduces the multi-task ASR datasets for evaluation.\n4. The empirical results are strong, especially in the multi-task setting.", "weaknesses": "1. The Gaussian assumption may be too strong for some real-world settings\n2. No comparison to some recent methods like GP-based approaches\n3. The multi-task formulation assumes clients are willing to share data statistics", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! We address your concerns and questions below:\n\n1. [*The Gaussian assumption may be too strong for some real-world settings [...] Have you considered non-Gaussian models? For example, a t-distribution might better handle heavy-tailed performance data that can occur in practice.*]  \na. Please see our discussion of these points in the general response (Issue 1: Assumption). In particular, as noted in the paper we view a derivation of a robust version of the SureMap objective using Student’s t-distribution as a good direction for future work.\n2. [*The multi-task setting assumes clients are willing to share summary statistics. How realistic is this assumption? Could you explore privacy-preserving ways to share this information?*]  \na. There are many settings when this is not a concern, e.g. in situations when the companies are already sharing the performance statistics publicly as a form of disclosure to their customers or government regulators. In settings where this *is* a concern, it should be possible to apply techniques from the differential privacy (DP) literature to overcome these limitations (many DP techniques exist for releasing summary statistics, e.g. Biswas et al. (2020)). We leave a full investigation to future work.\n3. [*How does SureMap compare to GP-based approaches like in [(Piratla et al., 2021)]? The method aims to handle low-data regimes well, which seems relevant here.*]  \na. Piratla et al. (2021) study a different low-data setting where they allow the user to *actively* sample points on which to evaluate performance. This difference makes it somewhat difficult to compare our methods.\n\n## References\nBiswas, Dong, Kamath, Ullman. *CoinPress: Practical private mean and covariance estimation*. NeurIPS 2020.  \nPiratla, Chakrabarti, Sarawagi. *Active assessment of prediction services as accuracy surface over attribute combinations*. NeurIPS 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1nwZkXD42f", "reviewer": "Reviewer_3ZUA", "summary": "The author developed a disggregated evaluation method called SureMap, which has high estimation accuracy for both multi-task and single-task disggregated evaluations. SureMap transforms the problem into structured simultaneous Gaussian mean estimatio, incorporating external data. This method further combines Maximum A Posteriori (MAP) estimation and cross-validation-free tuning via Stein's risk estimate (SURE). Significant improvements in accuracy were observed in disggregated evaluation tasks.", "questions": "Disaggregated evaluation is a core task in the fairness assessment of AI systems. This paper provides a possible solution and gives the corresponding analysis.  \n\nI would like to point out that my expertise does not directly align with the specific field of this article. Nevertheless, I have carefully read the paper several times, attempting to provide constructive feedback. I look forward to reading the insights of other reviewers, whose expertise is more closely related to the subject, to further inform my final score.", "limitations": "see the weaknesses and questions above", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author introduces a new method, SureMap, which tunes the parameters of the selected Gaussian prior using SURE before applying MAP estimation. Only linear parameters are needed to recover several natural baselines for disggregated evaluation.\n2.  The author introduces disggregated evaluation datasets for both single-task and multi-task settings.\n3. SureMap shows good results in both single-task and multi-task settings.", "weaknesses": "1. I would like to know if there are any rules that need to be followed in the selection of disaggregated evaluation datasets to ensure fairness, as well as the reason why the disaggregated evaluation goal is set as the mean 0-1 error.\n2. The fairness assessment is an application biased towards real-world scenarios. Does using Gaussian distribution as a prior align with real-world evaluations？", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! We address your concerns and questions below:\n\n1. [*I would like to know if there are any rules that need to be followed in the selection of disaggregated evaluation datasets to ensure fairness, as well as the reason why the disaggregated evaluation goal is set as the mean 0-1 error.*]  \na. The goal of disaggregated evaluation is to evaluate fairness and thus help surface any fairness issues that need to be addressed. The question of selecting the right dataset and metric for disaggregated evaluation is an active area of research but beyond the scope of this paper. Some considerations include how well the data represents the intended uses of the AI system and how well the metric captures potential harms / benefits; see Barocas et al. (2021) for an in-depth discussion. In our case, for dataset selection we simply use all the available data, while for the performance metric note that 0-1 error is just one possible measure among many that our method can be applied to, including MAE, MSE, WER, AUC, and so on. In addition to the 0-1 error our experiments include results for MAE and WER.\n2. [*The fairness assessment is an application biased towards real-world scenarios. Does using Gaussian distribution as a prior align with real-world evaluations?*]  \na. Please see our discussion of these points in the general response (Issue 1: Assumption). In particular, please note that we do not assume that the prior and individual-level distributions are Gaussian, only that the summary statistics are. As discussed in the response, this approximation is quite reasonable for numerous performance metrics of practical interest, including the 0-1 error, MSE, WER, AUC, and so on.\n\n## References\nBarocas, Guo, Kamar, Krones, Morris, Wortman Vaughan, Wadsworth, Wallach. *Designing disaggregated evaluations of AI systems: Choices, considerations, and tradeoffs*. AIES 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PR0EGadbXV", "reviewer": "Reviewer_LQxL", "summary": "This paper studies disaggregated evaluation, which aims to estimate the performance of models on various subpopulations. This problem is challenging due to small sample sizes in subpopulations, thus leading to inaccurate performance estimates. This issue is magnified when multiple clients use the same AI model and require individualized evaluations, which is referred to as multi-task disaggregated evaluation. \n\nThis paper designs a method that transforms the problem into a structured simultaneous Gaussian mean estimation problem. The method is comprised of two components: first, conduct maximum a posteriori (MAP) estimation for  Gaussian mean estimation and then apply cross-validation-free tuning using Stein’s unbiased risk estimate (SURE). Furthermore, the method employs an additive intersectional effect prior for capturing relationships between subpopulations with a limited number of hyperparameters.\n\nThe method, namely SureMap, is evaluated on various disaggregated evaluation tasks across different domains, including automated speech recognition (ASR) and tabular census data. The method shows high estimation accuracy for both single-task and multi-task settings over naive estimation, pooled estimation, and Bock estimation. Moreover, the method improves performance estimation even in data-poor regimes.", "questions": "- For subpopulations with few samples, how does this paper compute their ground truth and evaluate the estimation? \n- It would be better to clarify the pooled estimator. For example, what does the notation $h$​ in Equation 3? \n- It would be better to explain the working of the baselines, such as the Bock method.", "limitations": "This work has discussed its limitations in the setup of Gaussian distribution.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper presents a novel method, SureMap, designed to improve the accuracy of performance estimation for models on various subpopulations. This is crucial for assessing the fairness and robustness of machine learning models, especially when subpopulations are small or data is scarce.\n- Across tabular census data and automated speech recognition datasets in both single task and multitask settings, the proposed method outperforms previous estimation approaches by up to 50%. The results are consistent for various sampling rates and task numbers up to 50.", "weaknesses": "- The efficiency discussion of the method needs to be expanded. It would be better to list the complexity of the proposed method with existing methods and report the actual runtime. \n- There are certain cases that the proposed method performs worse than the pooled estimation, such as in Figure 2. It would be better to analyze the reasons behind such results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! We address your concerns and questions below:\n\n1. [*The efficiency discussion of the method needs to be expanded. It would be better to list the complexity of the proposed method with existing methods and report the actual runtime.*]  \na. Please see our discussion of efficiency in the general response (Issue 2: Computation). We will include this analysis in the revision.\n2. [*There are certain cases that the proposed method performs worse than the pooled estimation, such as in Figure 2. It would be better to analyze the reasons behind such results.*]  \na. In Figure 2, pooling outperforms our approach at extremely low data regimes where the typical number of samples per group is 1. In this extreme regime, there may not be enough information to estimate group-level performance well and so the best thing to report is the overall mean (which is what the pooled estimator does).\n3. [*For subpopulations with few samples, how does this paper compute their ground truth and evaluate the estimation?*]  \na. As noted in the first paragraph of Section 5, we exclude subpopulations with fewer than twenty samples, as we cannot obtain a reasonable ground truth for them.\n4. [*It would be better to clarify the pooled estimator. For example, what does the notation $h$ in Equation 3?*]  \na. The pooled estimator just takes the average across all data samples, or equivalently a weighted average across the average performance on each subpopulation, with weights corresponding to the number of samples on each subpopulation. Equation 3 makes the latter explicit, and uses $h$ to index each subpopulation.\n5. [*It would be better to explain the working of the baselines, such as the Bock method.*]  \na. The Bock estimator is described in Section 3.1, and specified explicitly in Equation 6. All other baselines are also specified in Equations 2, 3, and 5. We will make this clearer in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZKEMEmQlOx", "reviewer": "Reviewer_UdGz", "summary": "The paper proposed SureMap, a promising method for solving multi-task disaggregated evaluation problem. The key innovation of SureMap lies on transforming the problem into structured simultaneous Gaussian mean estimation and incorporating external data, e.g. from the AI system creator or from their other clients. Experiments on disaggregated evaluation tasks in multiple domains show the promising performance.", "questions": "1) Can the authors provide some analysis or insights when the data contradicts with Gaussian assumption?\n2) Can the authors visualize/show the cost when incorporating data from clients?", "limitations": "I don't think there are any negative societal impacts of this work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) Well motivated.\n2) Introduce several datasets for disaggregated evaluation and propose method that uses SURE to tune the parameters of a well-chosen\nGaussian prior before applying MAP estimation.\n3) Experiments are promising. Competitive in both single task and multi-task evaluation.", "weaknesses": "As analyzed in Section 2.4:\n1) Gaussian assumption may result in underperformance on heavy-tailed data.\n2) Incorporating data from multiple clients may be costly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! We address your concerns and questions below:\n\n1. [*Can the authors provide some analysis or insights when the data contradicts with Gaussian assumption?*]  \na. Please see our discussion of this issue in the general response (Issue 1: Assumption).\n2. [*Can the authors visualize/show the cost when incorporating data from clients?*]  \na. Please see our discussion of efficiency in the general response (Issue 2: Computation) and in particular Rebuttal Figure 2. We will include this analysis and visualization in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First we would like to thank all the reviewers for their careful and thorough reviews. We are happy to see that reviewers found the problem we tackle important (Revs. gnhf, 3rXK), the paper well-structured and theoretically well-founded (Rev. gnhf), and the experiments convincing (Revs. LQxL, 3ZUA, 3rXK, UdGz). In this general response we would like to address two common points raised by reviewers about the assumptions and costs of our approach.

## Issue 1: Assumption
Several reviewers raised concerns about the assumptions used to derive our method (SureMap). The form of our estimator is motivated by a hierarchical model with a Gaussian prior and a Gaussian observation distribution. The Gaussian prior is used *only* to motivate the form of the estimator; the estimator is valid even when this assumption does not hold. What our approach does require is that the observations, which in our case correspond to summary statistics (e.g. average accuracy in each group), can be reasonably approximated by a Gaussian. Note that we do *not* assume that the individual-level performance metric (e.g. the accuracy of each example) is Gaussian.  We can usually expect the summary statistics (e.g., within-group average accuracy) to be approximately Gaussian due to the central limit theorem (CLT), which states that averages of samples from a bounded-variance distribution converge in distribution to a Gaussian with that distribution’s mean. In particular, this is guaranteed to be true for bounded performance measures, which includes many of the most important measures in ML including accuracy and (for the most part) word-error rate. CLT-like results also hold (and thus our method would also work) for other important performance measures that are not sample averages, including AUC. Thus **our Gaussianity assumption is well-justified in most target applications**.

A few reviewers correctly point out, as we also do in Section 2.4, that this approximation may work worse for heavier-tailed data. As a quick check of the severity of this problem, we re-evaluate SureMap and the baselines on Diabetes Regression (Figure 7 in the original submission) but using MSE rather than MAE as the performance measure to be estimated. MSE is likely to have heavier tails than MAE if the error distribution looks roughly Gaussian, since in that case the MSE follows a chi-squared distribution (sub-exponential) while MAE follows a one-sided Gaussian (sub-Gaussian). By comparing Rebuttal Figure 1 (left) with Submission Figure 7 (left), we see that on heavier tailed data SureMap no longer dominates, but it is also not significantly worse than the best baseline. This both suggests that SureMap is reasonably robust to data with heavier tails and reinforces our argument in Section 6 that disaggregated evaluation needs to be done with care. In this case for example, a statistician might first consider applying a variance-stabilizing transformation to the data before running their analysis.  For instance, in the case of MSE, it is known that the fourth root of a chi-squared random variable is approximately normally distributed (Hawkins et al., 1986), so one could apply SureMap to fourth root transformed MSE’s, and then transform back to produce estimates on the original scale. We view other approaches to making SureMap even more robust, e.g. by using Student’s t-distribution in the derivation of the optimization objective, as an important direction for future work.

## Issue 2: Computation
Some reviewers also raised concerns about the complexity of SureMap, including the computational complexity as the number of tasks increases and the implementation complexity of the coordinate descent scheme. While we also noted scalability as a limitation in Section 2.4 of the submission, after the deadline **we believe it is no longer a limitation**, largely for two reasons:
1. After the submission, we updated the optimization algorithm to use L-BFGS, which has a standard implementation in SciPy whose defaults work well in our settings. It is also quite efficient, with single and multi-task SureMap taking roughly one second to produce estimates for fifty tasks (c.f. Rebuttal Figure 2, noting the log scales), a very reasonable amount of time for running statistical data analysis. Notably, the same figure shows that multi-task SureMap scales slightly *better* than single-task SureMap with the number of tasks.
2. While the baselines are faster due to not having an optimization routine, we note that both they and SureMap are methods that evaluate model performance and therefore take as input the outputs of model inference. These are often quite expensive to generate in practice; for example, it took more than a GPU-day to generate the ASR dataset used in our evaluations. In comparison to this, a second or less of CPU time is a miniscule cost and thus not a concern for the practical usefulness of our method.

## References
Hawkins, Wixley. *A note on the transformation of chi-squared variables to normality*. The American Statistician, 1986.

Meta-review: This paper studies estimating model performance in multiple distribution, a problem that has good relevance to fairness. The reviewers are in agreement that the theoretical analysis is adequate, the newly introduced datasets for evaluation are useful, and the experimental results are convincing. The rebuttal further addressed the reviewers' concerns about Gaussianity assumption and computational efficiency. Yet in the discussion phase there seems to be a lack of enthusiasm in championing this paper. Due to these considerations this paper is on the borderline.
Decision: Accept (poster)