Title: SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation
Venue: NeurIPS 2024
Accepted: yes
Keywords: fairness

evaluation

multi-task

transfer
Abstract: Disaggregated evaluation&mdash;estimation of performance of a machine learning model on different subpopulations&mdash;is a core task when assessing performance and group-fairness of AI systems.
A key challenge is that evaluation data is scarce, and subpopulations arising from intersections of attributes (e.g., race, sex, age) are often tiny.
Today, it is common for multiple clients to procure the same AI model from a model developer, and the task of disaggregated evaluation is faced by each customer individually.  This gives rise to what we call the *multi-task disaggregated evaluation problem*, wherein multiple clients seek to conduct a disaggregated evaluation of a given model in their own data setting (task).  In this work we develop a disaggregated evaluation method called **SureMap** that has high estimation accuracy for both multi-task *and* single-task disaggregated evaluations of blackbox models.  SureMap's efficiency gains come from
(1) transforming the problem into structured simultaneous Gaussian mean estimation and (2) incorporating external data, e.g., from the AI system creator or from their other clients.  Our method combines *maximum a posteriori* (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE).
We evaluate SureMap on disaggregated evaluation tasks in multiple domains, observing significant accuracy improvements over several strong competitors.

SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation

Abstract

Disaggregated evaluation-estimation of performance of a machine learning model on different subpopulations-is a core task when assessing performance and group-fairness of AI systems. A key challenge is that evaluation data is scarce, and subpopulations arising from intersections of attributes (e.g., race, sex, age) are often tiny. Today, it is common for multiple clients to procure the same AI model from a model developer, and the task of disaggregated evaluation is faced by each customer individually. This gives rise to what we call the multi-task disaggregated evaluation problem , wherein multiple clients seek to conduct a disaggregated evaluation of a given model in their own data setting (task). In this work we develop a disaggregated evaluation method called SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations of blackbox models. SureMap's efficiency gains come from (1) transforming the problem into structured simultaneous Gaussian mean estimation and (2) incorporating external data, e.g., from the AI system creator or from their other clients. Our method combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE). We evaluate SureMap on disaggregated evaluation tasks in multiple domains, observing significant accuracy improvements over several strong competitors.

1 Introduction

Evaluation is a key challenge in modern AI, with much effort spent deciding what metrics to measure, with which methods, and on what data. This challenge is especially acute in fairness assessment, which requires not only high-quality data to run a model and score its outputs but also demographic information for defining groups. Due to the high cost of obtaining high-quality evaluation data, the issue of sample complexity-sample size needed to get a good performance estimate-remains salient, especially when we want to release not just one overall measure but instead to output a disaggregated evaluation that captures variation among demographic subpopulations of the data [Barocas et al., 2021]. For instance, we might want to assess group fairness by examining the variation in performance across groups of users defined by intersections of the demographic attributes age, race, and sex. The naive approach of independently evaluating each group's performance on its own data can fail because the sample sizes of intersectional groups rapidly decrease as we consider more attributes [Herlihy et al., 2024]. Recent work has shown how to improve upon naive methods by combining data from multiple subpopulations to inform their individual performance estimates [Miller et al., 2021, Herlihy et al., 2024].

In today's technology landscape it is common for multiple clients to procure the same model (e.g., an automated speech recognition or language model) from an AI developer, with each client performing a disaggregated evaluation of the same model on their own data . We refer to this problem as the multi-task disaggregated evaluation . We formalize and study this problem, showing that one can improve the disaggregated evaluations of individual clients by using multi-task data in the form of

summary statistics from other clients or from the model provider. Our approach uses the (out-ofdistribution) multi-task data to set the parameters of a multivariate normal prior and then performs maximum a posteriori (MAP) inference on the (in-distribution) client data. Formally, we model the problem as Gaussian mean estimation and design a simple-yet-expressive additive prior that can capture many different relationships between subpopulations. Drawing upon classical statistics, we fit prior parameters by minimizing Stein's unbiased risk estimator [SURE, Stein, 1981]. While motivated by multi-task considerations, we show that our method also performs well in the single-task setting.

1.1 Contributions

SureMap : We introduce a method that uses SURE to tune the parameters of a well-chosen Gaussian prior before applying MAP estimation. The prior is motivated by its attainment of a good efficiency-expressivity tradeoff, requiring only a linear (in the number of subpopulations) number of parameters to recover several natural baselines for disaggregated evaluation.

Datasets : Disaggregated evaluation has few benchmarks [Herlihy et al., 2024], so we introduce new ones for both the single-task and multi-task settings, covering automated speech recognition (ASR) and also tabular domains (with linear models and also in-context LLMs).

Single-task: We find that SureMap is always competitive with strong baselines from prior work, while improving significantly in some settings with intersectional sensitive attributes.

Multi-task: Incorporating data from multiple clients into SureMap yields significant improvements across all evaluated settings. This multi-task approach is more accurate even with just one additional task and is the only method to consistently outperform the naive and pooling baselines.

1.2 Related work

Disaggregated evaluation is a core task in the fairness assessment of AI systems [Barocas et al., 2021]. Past work has sought to improve estimation accuracy by combining information across different groups, e.g., via Bayesian modeling [Miller et al., 2021], Gaussian process approximation of loss surfaces [Piratla et al., 2021], and structured regression [Herlihy et al., 2024]. The last work found that classical James-Stein-type mean estimation [James and Stein, 1961, Bock, 1975] is often competitive, and so we adopt it as our first non-naive baseline. We also compare to structured regression itself, which turns out to have a tight mathematical connection to SureMap; indeed, apart from our use of Gaussian (ridge) rather than Laplace (lasso) priors (regularization)-as well as our use of a more flexible tuning based on SURE rather than cross-validation-the method of Herlihy et al. [2024] can be viewed as the discriminative counterpart to our generative approach (see §E for details).

Within the disaggregated evaluation literature we are the first to formulate and study multi-task disaggregated evaluation. This is an important direction because (a) model providers often have their own data or data from multiple clients that can inform the evaluation and (b) transferring information across distributions is a key way to handle very low-sample regimes. We also contribute several datasets that we hope will spur further development in disaggregated evaluation.

SureMap relies on applying classical mean estimation tools to quantities modeled as Gaussian means. Notably, Miller et al. [2021] model scores via well-studied distributions-e.g., Gaussians-but since scores are related non-linearly to metrics it is unclear if this can lead to similarly simple estimators. To tune parameters, we use SURE, a popular statistical approach [Li, 1985, Donoho and Johnstone, 1995]. Specifically, in the empirical Bayes tradition, we use it to set the MAP estimator of a hierarchical model. Using SURE to tune the scale of an isotropic Gaussian prior was shown to be asymptotically (in the dimension) optimal in the case of heteroskedastic data distributions [Xie et al., 2012]. Since disaggregated evaluation data is highly heteroskedastic due to variation in group size, this is positive evidence for our approach, although our prior is non-isotropic and has many more variance parameters.

2 Setup

We first describe the disaggregated evaluation problem (§2.1), recast it as a Gaussian mean estimation (§2.2), and motivate a multi-task variant (§2.3), all while introducing several baselines estimators.

2.1 Setting and baselines

We want to assess a predictive model p : X → Y under some distribution D over input space X and output space Y using error measure ℓ : Y × Y → R . For example, in image classification, D is a distribution over (image, label) pairs and ℓ is the 0-1 error. To simplify notation, we mainly deal with the composite function f ( z ) = ℓ ( y, p ( x )) acting on points z = ( x, y ) in the product space Z = X × Y .

In disaggregated evaluation, Z is assumed to be a union of d ≥ 1 disjoint subsets Z g , each associated to some subpopulation or group g ∈ [ d ] , where we use [ k ] to denote the set { 1 , . . . , k } . As a running example, suppose each point z ∈ Z is an individual whose sex s and age a are categorical variables with d 1 and d 2 possible values, respectively. Then d = d 1 d 2 and each point has an associated index g = ( s -1) d 2 + a denoting its intersection of sex s ∈ [ d 1 ] and age a ∈ [ d 2 ] . The task is then to use a set S ∼ D n of n ≥ 1 i.i.d. samples from the distribution D to estimate the vector of true subpopulation errors µ ∈ R d , with components µ g = E z ∼D|Z g [ f ( z )] . 1 We write ˆ µ ( S ) ∈ R d for an estimator of µ with components denoted as ˆ µ g ( S ) .

Empirically, we measure estimation accuracy, i.e., the performance of the performance estimate, via mean absolute error (MAE) L MAE µ ( ˆ µ ( S )) = 1 d ∥ ˆ µ ( S ) -µ ∥ 1 , which is easy to interpret and less sensitive to outliers than mean squared error (MSE). Our method development, however, is based on a count-weighted version of MSE, where we denote group counts as n g = | S ∩ Z g | :

We conclude the setup with two baselines. The first is the naive estimator returning group means: 2

While unbiased, ˆ µ naive can perform poorly on groups with few samples. The second baseline, the pooled estimator , returns an identical quantity-the overall sample mean-for all groups:

This estimator is generally biased (unless all group means are equal), but it can perform well in low-sample regimes thanks to a much lower variance.

2.2 A Gaussian model for disaggregated evaluation

We use a simple but natural model to aid in the design of disaggregated evaluation methods. Specifically, denoting the naive estimator by y = ˆ µ naive ( S ) ∈ R d , we model group g 's entry y g as being drawn from a Gaussian with (unknown) mean µ g and (known) variance σ 2 /n g , where σ 2 is shared across groups. This reduces the problem of disaggregated evaluation, as defined in §2.1, to that of estimating the mean of a multivariate Gaussian with known diagonal covariance Σ g,g = σ 2 /n g given a single sample y ∼ N ( µ , Σ ) . Our model has many advantages in the disaggregated evaluation setting:

By the central limit theorem, y is asymptotically normal with mean µ and diagonal covariance Σ for many distributions D of interest, even when the underlying data is non-Gaussian. Furthermore, because the methods derived from our model only take y and Σ as input, they can be applied even when the evaluated statistic is not the pointwise average assumed by the setup in §2.1, so long as y ∼ N ( µ , Σ ) holds asymptotically. An example of this is when y g = ˆ µ naive g ( S ) corresponds to the area under the ROC curve (AUC) computed over group g 's data S ∩ Z g [Lehmann, 1951]; we demonstrate SureMap's applicability to AUC empirically in §G (Figures 10 & 12).

While a shared variance is a strong assumption, it is perhaps the simplest way of incorporating the inductive bias that Σ g,g will be highly correlated with the inverse of n g , the number of samples from group g . In practice, we set σ 2 to be the pooled estimate 1 n -d ∑ d g =1 ∑ z ∈ S ∩Z g ( f ( z ) -y g ) 2 .

Gaussian mean estimation is one of the best-studied problem in statistics, with numerous well-tested baselines and approaches for developing new methods. In particular, we make significant use of the classic James-Stein approach [James and Stein, 1961, Bock, 1975], SURE [Stein, 1981], and empirical Bayesian estimation methods [ ? ].

In the multi-task setting, clients are likely to be unwilling to share their actual data but possibly more willing to share group summary statistics. Thus methods developed for our Gaussian model-which only require the group means y , group counts n , and an estimate of σ 2 -will be more broadly applicable than methods that act directly on the dataset S ⊂ Z .

This model can also be naturally extended-using a non-diagonal covariance Σ -to disaggregated evaluation with non-disjoint groups, e.g., to simultaneously estimate performance both for all women and for only women in their forties. In the interest of brevity we focus on the disjoint group setting.

2.3 The multi-task setting

We can easily extend this model to study multi-task disaggregated evaluation, in which for each task t = 1 , . . . , T (e.g., a client of the model provider) we observe a set S t ⊂ Z of n t samples from the task distribution D t . The goal is then to output T vectors ˆ µ t that are close on-average to the tasks' subpopulation errors µ t ; g = E z ∼D t [ f ( z ) | z ∈ Z g ] . Converting to our Gaussian model, we observe T vectors y t ∼ N ( µ t , Σ t ) -where we set Σ t ; g,g = σ 2 /n t ; g for some globally shared σ 2 and task-specific group count vectors n t ∈ Z d ≥ 0 -and must output T mean estimates ˆ µ t ( { y t } T t =1 ) ∈ R d .

We consider two natural multi-task baseline estimators. The first is the global naive estimator (or global estimator for short), which combines the data from all tasks, computes a single global vector of group averages, and uses it as the estimate for each task:

While low variance, the global estimator ignores variation across tasks. Our second baseline-the multi-task offset estimator -shifts the global estimate on each task to ensure that the task's pooled mean is preserved (thus accounting for the variation in pooled means across individual tasks):

3 Methods

In the last section we reduced the problem of disaggregated evaluation to that of estimating a mean µ ∈ R d given a sample y ∼ N ( µ , Σ ) , where Σ is known and diagonal. We now design a method, SureMap , for the latter problem. Our technical approach involves the following two steps:

Choosing a parameterized mean estimator. We use the MAP estimator under a multivariate normal prior that we design specifically for intersectional subpopulations.

Tuning the estimator's hyperparameters. We use SURE to estimate the quality of our estimator, which we then optimize over the choice of hyperparameters using the L-BFGS-B algorithm.

3.1 Designing a parameterized estimator

As mean estimation is a vast area, we use three criteria for designing an estimator: it should (1) dominate baselines such as ˆ µ naive and ˆ µ pooled ; (2) have relatively few hyperparameters; and (3) handle heteroskedasticity stemming from variation in group sizes. One natural source of candidates are James-Stein-type shrinkage estimators: the original James-Stein estimator famously dominates ˆ µ naive in MSE and has no hyperparameters to tune [James and Stein, 1961], satisfying our first two desiderata. Furthermore, while James and Stein [1961] assumed an isotropic Σ , subsequent estimators such as the following variant of an estimator due to Bock [1975] do handle heteroskedastic Σ : 3

where ( · ) + = max {· , 0 } , and θ ∈ R d is a default estimate towards which y is shrunk. 4

However, empirically we find that this often underperforms the pooled estimator in low-sample regimes; further, the form of ˆ µ Bock θ shows that the amount of shrinkage towards θ is the same for each coordinate g ∈ [ d ] , despite intuition suggesting that we should shrink less for groups g with more samples. Corrections to this tend to be involved and difficult to generalize [Efron and Morris, 1973].

We thus turn to a different family of well-known Gaussian mean estimators: those that return the mode of the posterior distribution assuming µ is sampled from the conjugate prior N ( θ , Λ ) with mean θ ∈ R d and positive-definite covariance Λ ∈ R d × d (e.g., Gelman et al. [2014, Equation 3.12]):

F,<25

F,25-64

M,25-64

F,>64

M,>64

F,<25

F,25-64

M,25-64

F,>64

M,>64

F,<25

F,25-64

M,25-64

F,>64

M,>64

F,<25

F,25-64

M,25-64

F,>64

M,>64

F,<25

F,25-64

M,25-64

F,>64

M,>64

F,<25

F,25-64

M,25-64

F,>64

M,>64

F,<25

F,25-64

M,25-64

F,>64

M,>64

F,<25

F,25-64

M,25-64

F,>64

M,>64

+

τ

2

sex

+

τ

2

age

+

τ

2

sex,age

Λ

(

τ

) =

τ

2

∅

MAP naturally handles heteroskedasticity by weighting low-variance coordinates g more heavily, satisfying our third criterion; however, its most general form has O ( d 2 ) hyperparameters, violating the second. We next use problem structure to restrict the number of hyperparameters needed to define Λ while still allowing ˆ µ MAP θ , Λ to express every baseline introduced in §2.1, satisfying our first criterion.

3.1.1 An additive intersectional effects prior

For brevity, we build up our estimator somewhat informally; a full description is in §C.1. We return to our simple example where each group g ∈ [ d ] corresponds to an intersection ( s, a ) ∈ [ d 1 ] × [ d 2 ] of two attributes: a sex s ∈ [ d 1 ] and an age a ∈ [ d 2 ] . A simple prior that additively incorporates individual attribute effects into intersectional group means is the following:

where θ ∈ R d and τ ∅ , τ sex , τ age , τ sex,age ∈ R ≥ 0 are hyperparameters, ζ ∼ N (0 , 1) is a scalar effect shared across all groups, the vector ζ sex ∈ R d 1 has its s th entry ζ sex s ∼ N (0 , 1) shared by all groups g whose sex is s , the vector ζ age ∈ R d 2 has its a th entry ζ age a ∼ N (0 , 1) shared by all groups g whose age is a , and the vector ζ ∼ N ( 0 d , I d ) contains an independent noise term ζ g for each group g .

The hyperparameter τ ∅ quantifies how much we expect all of the means to be shifted (by a shared positive or negative value) from the default θ . Hyperparameters τ sex and τ age express how large we expect contributions of sex and age alone to be towards the means. And finally, non-zero τ sex,age gives the prior flexibility to model heterogeneity across all intersectional groups.

Given the vector of hyperparameters τ = ( τ ∅ , . . . , τ sex,age ) ∈ R 4 ≥ 0 , the prior can be written more compactly as µ ∼ N ( θ , Λ ( τ )) , where the covariance is

for matrices C A ∈ { 0 , 1 } d × d s.t. each entry C A ; g,h is one iff groups g and h agree on the attributes included in A . In particular, we have that C ∅ = 1 d × d is the all-ones matrix, the entries C sex ; g,h of C sex ∈ { 0 , 1 } d × d are one iff groups g and h share the same sex attribute, the matrix C age is analogous, and C sex,age = I d is the d × d identity. This structure is visualized in Figure 1.

<latexi sh

1_b

64="32

S

UVwzZON

8C

r

W

5d

>A

TH

c

DL

R

EJy

7/

YB

0

u

p

m

j

o

v

P

X

M

Q

F

9

k

n

fg

G

+

<latexi sh

1_b

64="XAR0SkCcDqf

In

N9

z

Td

>

P

V

B

F

o

EU

g

G

L

Z

YjWm2

u8+

H

J

7

p

y

r

O5

K

M

Q

v

3

/

w

<latexi sh

1_b

64="rH83

m

nX

9C

V+dF0L

qT7g

>A

P

c

BNS

R

EO2

u

yU

j

M

KkzGo

J

/

v

f

Wp

5

I

w

Q

Z

D

Y

<latexi sh

1_b

64="0fzKL

JF

dP

5TXBEj

u

Y

>A

CQ

c

VD

S

R

n2

Ivg

U

+

wq

p

m

y

9H

M

/G

3

O

r

87

o

Z

N

k

W

3.1.2 Efficiency and expressivity

As detailed in §C.1, this prior can be naturally extended to any number of attributes k using a covariance matrix Λ ( τ ) ∈ R d × d specified by a vector τ ∈ R 2 k ≥ 0 of 2 k hyperparameters. Since k ≤ ⌊ log 2 d ⌋ , the total number of hyperparameters (including θ ∈ R d ) is d +2 k = O ( d ) , which is much smaller than the O ( d 2 ) complexity of the general case. We can further reduce this by fixing the entries of θ , constraining them to be identical, or setting them using external (e.g., multi-task) data.

Despite this reduction in hyperparameters, we can show that for a suitable choice of τ , the estimator ˆ µ MAP θ , Λ ( τ ) recovers many estimators of interest, including the naive estimator and the (possibly offset) pooled estimator (see §C.2). This means that MAP with our structured prior should be able to outperform all four baselines from the previous section, if appropriately tuned.

3.2 Tuning by minimizing expected risk

Having specified a parameterized estimator, there remains the question of setting its parameters θ and τ . One might want to treat this as a hyperparameter tuning problem and use a data-splitting approach; however, the dimensionality of the problem makes standard techniques either expensive or noisy, and data splitting introduces additional randomness and design decisions into an already data-poor environment. We instead make continued use of our Gaussian assumption and turn to SURE, which given a differentiable estimator ˆ µ : R d → R d returns an unbiased estimate of its weighted MSE L n µ using sample data y ∼ N ( µ , Σ ) :

where given any W ≻ 0 d × d we denote ∥ x ∥ 2 W = ⟨ x , Wx ⟩ ∀ x ∈ R d . Using SURE we can now tune the parameters of ˆ µ by minimizing ˆ R n µ ( y ) in a manner similar to empirical risk minimization.

3.2.1 Single-task SureMap

In the single-task setting, we fix θ = 0 d and tune the variance parameters τ ∈ R 2 k ≥ 0 . Letting A ( τ ) = ( Λ -1 ( τ ) + Σ -1 ) -1 Λ -1 ( τ ) , we define the single-task SureMap estimator as

The optimization problem in the second line comes from substituting ˆ µ MAP 0 d , Λ ( ˆ τ ) into SURE (Eq. 10). It is nonconvex, but we find that it can be quickly solved to sufficient accuracy with L-BFGS-B [Byrd et al., 1995], a standard method for bound-constrained optimization of differentiable functions.

3.2.2 Multi-task SureMap

To generalize SureMap to the multi-task setting we propose to specify ˆ θ and ˆ τ by minimizing SURE aggregated across tasks, i.e., ∑ T t =1 ˆ R n t µ t ( y t ) . While setting both parameters via direct optimization of this objective is the most straightforward approach, we find that it performs worse than single-task SureMap when there are only a few tasks ( T ≤ 5 ) and rarely improves significantly above the multi-task global and offset estimators. This can be explained by observing the few-task limit-i.e. T = 1 -in which case optimizing the aggregated SURE objective results in setting ˆ θ = y 1 and thus makes the multi-task estimator equivalent to the naive estimator.

We find that a better approach is to treat the choice of ˆ θ as its own simultaneous mean estimation problem and apply the SureMap approach to it. In particular, our model y t ∼ N ( µ t , Σ t ) and our prior µ t ∼ N ( θ , Λ ) imply that the samples y t ∼ N ( θ , Λ + Σ t ) have mean θ and known covariances (apart from tuning parameters). Therefore, the MAP estimator of θ itself given a hyperprior θ ∼ N ( 0 d , Γ ) with covariance Γ ≻ 0 will have the form ˆ θ = ( Γ -1 + ∑ T t =1 ( Λ + Σ t ) -1 ) -1 ∑ T t =1 ( Λ + Σ t ) -1 y t .

To reduce the number of tuning parameters, we use a prior of the same form as before by specifying Γ = Λ ( υ ) for υ ∈ R 2 k ≥ 0 , i.e., the same structured covariance as described in §3.1.1 but with separately tuned parameters (see §3.1.1 and §C.1). Substituting the meta-level MAP estimator of θ into the MAP estimator of µ t and tuning the parameters τ and υ by optimizing the sum of SUREs (Eq. 10)

across tasks yields our multi-task SureMap estimator (see §C.3 for details):

The optimization problem on the last line can again be approximately solved using L-BFGS-B.

3.3 Limitations

Various modeling assumptions impact performance of SureMap. For instance, the Gaussian error assumption is less appropriate when errors are heavy-tailed. In §G, Figure 9, we consider MSE as an example of a target metric with heavy-tailed observation errors. We find that SureMap still performs well, but is no longer superior to previous approaches. One avenue for improvement would be to use a variance-stabilizing transformation (e.g., Hawkins and Wixley [1986]) prior to applying SureMap.

Note that SureMap achieves its improved accuracy by shrinking naive estimates towards a less granular estimator (e.g., a pooled mean). As a result the estimation is biased towards less disparity , which could lead to overly optimistic conclusions about fairness. For this reason it is extremely important to examine not just the point estimates, but also confidence intervals. These can be obtained, for example, by viewing SureMap as a regression approach (§E) and leveraging inference techniques for regression.

4 Datasets

We evaluate our approach in several representative settings for disaggregated evaluation, including two tabular settings appearing in previous works [Miller et al., 2021, Herlihy et al., 2024, Liu et al., 2024], and three new settings: a multi-task tabular setting based on state-level U.S. census data and both a single-task and a multi-task ASR evaluation setting.

4.1 Tabular datasets

We consider three tabular datasets, two for the single-task and one for the multi-task setting, covering two important domains where fairness concerns can arise: healthcare records and demographic data. While we focus on the classification task and 0-1 error, in §G we also report results for regression (Figures 8 & 9) and for classification with AUC as the target metric (Figures 10 & 12).

Diabetes. This is a tabular dataset of Strack et al. [2014], containing around 100K patient records with six race, two sex, and three age categories. We evaluate a logistic regression classifier trained to predict patient disposition after a hospital stay (discharged or otherwise). The target metric is the 0-1 error.

Adult. We use the classic Adult census dataset [Kohavi, 1996] to evaluate performance of an in-context LLM learner-specifically llama-3-70b -in predicting whether a person makes more or less than $50K after being provided with eight examples via a modification of the prompt template of Liu et al. [2024]. The target metric is the 0-1 error, disaggregation is by race, sex, and age.

State-Level ACS (SLACS). This is a tabular dataset for multi-task setting derived from the census data for all U.S. states and Puerto Rico assembled by Ding et al. [2021]. Each datapoint corresponds to a person in one of nine race and two sex categories; we consider three age categories: below 25, 25-64, and over 64. The underlying task is to classify each person as earning either more or less than $50K. We train a regularized logistic model on the data from California, and seek to evaluate its performance on the other 50 states/territories, which comprise the tasks. The target metric is the 0-1 error.

4.2 ASR datasets

We also introduce both single-task and multi-task speech recognition datasets, based on applying the popular Whisper ASR model [Radford et al., 2023]-specifically whisper-tiny -on the English part of the Common Voice (CV) dataset [Ardila et al., 2020], which contains utterances from individuals in one of nine age and three sex categories.

Diabetes Classification (0-1 error)

Diabetes Classification (0-1 error)

Diabetes Classification (0-1 error)

0.16

naive

I.

0.14

0.08

pooled

(sdnou61

0.20

0.12

0.10

-f-

f-

Bock

Struct.Reg.

0.15

0.06

0.04

K

K

0.04

0.02

+-

SureMap

0.10

0.05

K

0.02

0.01

0.032

0.1

0.316

1.0

100

0.032

0.1

0.316

1.0

0.01

0.032

0.1

0.316

1.0

(5)

(15)

(48)

(150)

(476)

(35)

(110)

(348)

(1101)

(3481)

(3)

(8)

(25)

(6)

(251)

Adult Classification(0-1 error)

0.10

0.12

naive

I.

0.08

0.08

(sdno6

pooled

(sdnoub

0.10

0.06

-+-

-I-

Bock

Struct.Reg.

MAE(smallg

0.08

1

K

-f-

SureMap

0.06

0.04

0.02

0.04

0.02

0.02

0.01

0.032

0.316

1.0

0.032

(0.1

0.316

1.0

0.01

0.032

0.316

1.0

(4)

(12)

(119)

(375)

(19)

(59)

(187)

(591)

(1869)

(1)

(3)

(34)

(108)

CommonVoiceASR(WER)

CommonVoiceASR(WER)

CommonVoiceASR(WER)

0.14

0.14

naive

0.14

0.12

0.12

..

pooled

0.10

0.10

-+-

Bock

0.10

0.08

Struct. Reg.

MAE(small g

+

SureMap

0.08

0.06

I.

0.06

0.04

0.04

0.04

0.02

0.02

0.02

10'0

0.032

(6)

0.316

1.0

10'0

0.032

318

0.032

0.316

(2)

(6)

(60)

(681)

(4)

(12)

(1)

(2)

(6)

subsampling rate (median group size)

subsampling rate (median group size)

subsampling rate (median group size)

Common Voice. This is a single-task dataset obtained by combining the validation and test partitions of the CV dataset. We calculate the word-error rate (WER) across all the utterances of each individual, which becomes the target metric to be predicted.

Common Voice Clusters (CVC). This is a multi-task ASR dataset. To construct it, we first cluster the utterances in the train partition of the CV dataset into 20 clusters by applying k -means to the sums of GloVe word embeddings [Pennington et al., 2014] of their corresponding text strings. To model task relatedness, we then randomly reassign each utterance to a random cluster with probability α ∈ [0 , 1] . The resulting clusters are the tasks. The target metric is the word-error rate (WER) across all the utterances of each individual in a given cluster. In most experiments we use α = 1 2 , but we also investigate what happens when α varies between zero, i.e., the original clusters, and one, corresponding to identically distributed tasks.

5 Evaluation

Our main metric is MAE relative to a ground truth vector, which we take to be the mean of all available data for each subpopulation g ∈ [ d ] , except those with fewer than 40 samples. In our main results we subsample with replacement from the entire dataset at different rates and track performance as a function of the sizes of the resulting datasets. To obtain 95% confidence intervals we conduct 200 and 40 random trials at each subsampling rate in the single-task and multi-task settings, respectively.

5.1 Single-task

We compare SureMap to the naive (Eq. 2) and pooled (Eq. 3) baselines, as well as to the Bock estimator with shrinkage towards the pooled estimator (Eq. 17) and the structured regression estimator of Herlihy et al. [2024]. On both Diabetes and Adult, SureMap significantly outperforms all competitors (Figure 2, top & middle), the greatest improvement is on subpopulations with limited data. In §G, we consider a regression variant of Diabetes and observe similar results (Figure 8). On the Common Voice task, SureMap performs roughly similarly to Bock, while outperforming structured regression at some subsampling rates (Figure 2, bottom); here again the gains are driven

+

State-Level ACS Classification (0-1 error)

0.16

naive

0.14 -

pooled

MAE

-I--

Bock

0.12

---

Struct.Reg.

0.10

---

SureMap

task-average

MT global

0.08

X

MT offset

0.06

MT Bock

+

MTSureMap

0.04

0.02

0.01

0.032

0.1

0.316

1.0

(3)

(8)

(26)

(83)

(262)

Common Voice Clusters ASR (WER)

0.09

I.

naive

0.08

....

pooled

MAE

0.07-

Bock

Struct.Reg.

0.06

十--

SureMap

/erag

0.05

MT global

task-ave

0.04

MT offset

1

MT Bock

0.03

MTSureMap

0.02

0.01

0.01

0.032

0.1

0.316

1.0

(2)

(7)

(21)

(67)

(210)

subsampling rate (median group size)

task-level results (median grp.size =26)

5

multiplicativeimprovement

MT Bock

MTSureMap

inMAEovernaive

4

naive

3

X

2

1

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

task-level results (median grp. size = 21)

3.5

multiplicativeimprovement

MT Bock

MTSureMap

inMAEovernaive

3.0

naive

2.5

2.0

X

X

1.5

X

1.0

0.2

0.4

0.6

0.8

1.0

1.2

distancefrom ground truthtoMT median

by better performance on small groups. Pooling performs best when data is extremely limited, but it is not competitive with even modestly more data, and also underperforms on small groups.

5.2 Multi-task

In the multi-task setting, we use all the single-task methods as baselines while adding multi-task (MT) ones, including MT global (Eq. 4), MT offset (Eq. 5), and an MT extension of Bock (Eq. 6) in which θ g is set using the average across the group g data on all other tasks. We first consider the SLACS task, for which Figure 3 (top left) shows that MT SureMap significantly outperforms other methods in the low-data regime while matching the best one (MT Bock) in the high-data regime. At subsampling rate 0.1, Figure 3 (top right) also shows that using multi-task data leads to improvement on all but two of the fifty tasks, that the reduction in MAE over the naive estimator on a typical task is 2x, and that this improvement only loosely correlates with the task's ground truth distance from the multi-task median. On the other hand, it also shows that while MT Bock's improvements are typically smaller, on SLACS it improves performance for every state (including the two where MT SureMap is worse).

On the CVC task, the MT offset baseline is the most competitive, except at the lowest subsampling rate where pooling is better and at higher subsampling rates where it stops improving with additional data (Figure 3, bottom left). SureMap outperforms it and all other methods across all subsampling rates and its advantage is greatest in low-data regimes, where it even outperforms pooling. In the task-level evaluation in Figure 3 (bottom right) we see that on every task, MT SureMap attains an improvement of 2-3.5x over the naive baseline and almost always outperforms MT Bock. Furthermore, the latter performs substantially worse on tasks whose ground truth vectors are far away from the multi-task center while MT SureMap is not affected.

5.3 Ablations

We next look at how the degree of included intersectional effects and multi-task structure affect performance. In Figure 4 (left), we evaluate utility of including higher-order interactions in the structured prior. We implement SureMap variants with up to ℓ th-order interactions by setting τ A for | A | ∈ { ℓ +1 , . . . , k -1 } to zero (but not for | A | = k ). Weobserve that including zeroth-order effects ( ℓ = 0 ) in single-task SureMap (i.e., shrinkage to pooling) improves upon the usual single-parameter Gaussian prior ( ℓ = -1 ) in low-data regimes. Adding first-order effects ( ℓ = 1 ) leads to substantial further im-

State-Level ACS Classification (0-1 error)

0.16

.-I.

naive

0.14

..I..

pooled

MAE

F

SureMap-(-1).-

0.12

ask-average

e

0.10

MT global

0.08

MT SureMap (-1)

MTSureMap(O)

0.06

MT SureMap (1)

0.04

MTSureMap(2)

0.02

0.01

0.032

0.1

0.316

1.0

(3)

(8)

(26)

(83)

(262)

subsamplingrate (median group size)

CommonVoice Clusters ASR(WER)

0.050

naive

pooled

0.045

E

MAE

Bock

0.040

Struct.Reg.

SureMap

task-average

0.035

MT global

0.030

MT offset

MTBock

0.025

MT SureMap

0.020

0.015

0.0

0.2

0.4

0.6

0.8

1.0

tasksimilarity

State-Level ACS Classification (O-1 error)

I..............·....

.....naive

0.12

pooled

MAE

Bock

Struct.Reg.

task-averaged

0.10

SureMap

MT global

0.08

MT offset

MT Bock

0.06

MTSureMap

0.04

10

20

30

40

50

numberoftasks

CommonVoice ClustersASR (WER)

0.035

naive

MAE

pooled

task-averaged I

0.030

Bock

Struct.Reg.

SureMap

0.025

MT global-I

0.020

MT Bock

MTSureMap

0.015

5

10

15

20

numberoftasks

provement, but second-order effects ( ℓ = 2 ) make performance slightly worse. A similar effect can be observed among the multi-task variants, except there is no loss (but also no gain) to using the highestorder variant ( ℓ = 2 ). Overall, this study suggests that using the full-order SureMap is a reasonable default but that most of the method's effectiveness comes from zeroth- and first-order effects.

Figure 4 (right) tracks performance as the task-similarity parameter defining the CVC task is varied. MT SureMap outperforms all methods at all settings and is also not as strongly affected by the task similarity, at least as it is defined for the CVC data. This suggests that the structured prior we use may be useful even if the dataset means are quite different but the underlying evaluation problem (in this case estimating WER of ASR models) is the same.

Lastly, in Figure 5 we study how the number of tasks affects multi-task performance. On both SLACS and CVC, MT SureMap outperforms all single-task baselines (the best one being the single-task SureMap) at T = 2 tasks, i.e., it can take advantage of even very little external information. In contrast, on CVC, the competitor multi-task methods (e.g., MT offset and MT Bock) do not even outperform single-task methods until T ≥ 5 tasks. These results demonstrate that, unlike these comparators, multitask SureMap can be confidently used even when only one additional client's worth of data is available.

6 Conclusion

We have introduced SureMap, a disaggregated evaluation approach, which combines MAP estimation under a structured Gaussian prior with hyperparameter tuning via SURE. SureMap achieves substantial empirical improvements over strong baselines in both single-task and multi-task settings. Valuable future directions include improving robustness to heavy-tailed data and developing multitask methods that can handle client privacy concerns. More broadly, we hope our work will have a positive impact by allowing model users to more accurately identify fairness-related harms, the first step towards mitigating them. However, using SureMap and any other disaggregated evaluation approach must be done with care, so as to not risk overconfidence in a model's fairness.

Acknowledgments

We thank Nicolas Le Roux for discussions and feedback at the beginning of this effort.

References

Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common Voice: A massivelymultilingual speech corpus. In Proceedings of the 12th Language Resources and Evaluation Conference , 2020.

Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, W. Duncan Wadsworth, and Hanna Wallach. Designing disaggregated evaluations of AI systems: Choices, considerations, and tradeoffs. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , 2021.

Mary Ellen Bock. Minimax estimators of the mean of a multivariate normal distribution. The Annals of Statistics , 3(1):209-218, 1975.

James R. Bunch, Christopher P. Nielsen, and Danny C. Sorensen. Rank-one modification of the symmetric eigenproblem. Numerische Mathematik , 31:31-48, 1978.

Richard H. Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing , 16(5):1190-1208, 1995.

Corinna Cortes and Mehryar Mohri. AUC optimization vs. error rate minimization. In Advances in Neural Information Processing Systems , 2003.

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring Adult: New datasets for fair machine learning. In Advances in Neural Information Processing Systems , 2021.

David L. Donoho and Iain M. Johnstone. Adapting to unknown smoothness via wavelet shrinkage. Journal of the American Statistical Association , 90(432):1200-1224, 1995.

Bradley Efron and Carl Morris. Stein's estimation rule and its competitors-an empirical Bayes approach. Journal of the American Statistical Association , 68(341):117-130, 1973.

Sergey Feldman, Maya R. Gupta, and Bela A. Frigyik. Revisiting Stein's paradox: Multi-task averaging. Journal of Machine Learning Research , 15:3621-3662, 2014.

Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. Bayesian Data Analysis . Electronic Edition, 2014.

Douglas M. Hawkins and R. A. J. Wixley. A note on the transformation of chi-squared variables to normality. The American Statistician , 40(4):296-298, 1986.

Christine Herlihy, Kimberly Truong, Alexandra Chouldechova, and Miroslav Dudík. A structured regression approach for evaluating model performance across intersectional subgroups. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , 2024.

Roger A. Horn and Charles R Johnson. Matrix Analysis . Cambridge University Press, second edition, 2013.

Willard James and Charles M. Stein. Estimation with quadratic loss. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability , 1961.

Ron Kohavi. Scaling up the accuracy of Naive-Bayes classifiers: A decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining , 1996.

Sören Laue, Matthias Mitterreiter, and Joachim Giesen. Computing higher order derivatives of matrix and tensor expressions. In Advances in Neural Information Processing Systems , 2018.

Erich Leo Lehmann. Consistency and unbiasedness of certain nonparametric tests. Annals of Mathematical Statistics , 22(1):165-179, 1951.

Ker-Chau Li. From Stein's unbiased risk estimates to the method of generalized cross validation. The Annals of Statistics , 13(4):1352-1377, 1985.

Jun S. Liu. Siegel's formula via Stein's identities. Statistics & Probability Letters , 21:247-251, 1994.

Yanchen Liu, Srishti Gautam, Jiaqi Ma, and Himabindu Lakkaraju. Confronting LLMs with traditional ML: Rethinking the fairness of large language models in tabular classifications. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2024.

Andrew C. Miller, Leon A. Gatys, Joseph Futoma, and Emily Fox. Model-based metrics: Sampleefficient estimates of predictive model subpopulation performance. In Proceedings of the 6th Machine Learning for Healthcare Conference , 2021.

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing , 2014.

Vihari Piratla, Soumen Chakrabarti, and Sunita Sarawagi. Active assessment of prediction services as accuracy surface over attribute combinations. In Advances in Neural Information Processing Systems , 2021.

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning , 2023.

Sidney Siegel. Nonparametric Statistics for the Behavioral Sciences . McGraw-Hill, 1956.

Charles M. Stein. Estimation of the mean of a multivariate normal distribution. The Annals of Statistics , 9(6):1135-1151, 1981.

Beata Strack, Jonathan P. DeShazo, Chris Gennings, Juan L. Olmo, Sebastian Ventura, Krzysztof J. Cios, , and John N. Clore. Impact of HbA1c measurement on hospital readmission rates: Analysis of 70,000 clinical database patient records. BioMed Research International , 2014, 2014.

Qing Wang and Alexandria Guo. An efficient variance estimator of AUC and its applications to binary classification. Statistics in Medicine , 39:4107-4349, 2020.

Hilde Weerts, Miroslav Dud ´ k, Richard Edgar, Adrin Jalali, Roman Lutz, and Michael Madaio. Fairlearn: Assessing and improving fairness of AI systems. Journal of Machine Learning Research , 24:1-8, 2023.

Xianchao Xie, S. C. Kou, and Lawrence D. Brown. SURE estimates for a heteroscedastic hierarchical model. Journal of the American Statistical Association , 107(500):1465-1479, 2012.

A Notation

For p ∈ [1 , ∞ ] we use ∥·∥ p to denote the p -norm on R d .

We use ∥·∥ to denote the spectral norm on R m × n .

For any positive semi-definite matrix A ⪰ 0 d × d we use the notation ∥·∥ A : R d → R ≥ 0 to denote the vector norm ∥ x ∥ A = √ ⟨ x , Ax ⟩ = ∥ √ Ax ∥ 2 on x ∈ R d .

For any positive integer k we use [ k ] to denote the set { 1 , . . . , k } .

For any set S we use 2 S to denote its powerset.

For any vector a ∈ R d we use a i to denote its i th entry. If a is only defined as an expression then we will abuse notation and use ( a ) i to refer to its i th entry.

For any vector a ∈ R k and any subset S ⊂ [ k ] with elements s 1 < · · · < s m we define a S = ( a s 1 · · · a s m ) to be the vector of the entries of a whose indices correspond to the elements of S sorted in ascending order.

∈ · For any matrix A ∈ R m × n we use A i,j to denote its ( i, j ) th entry and A i, : its i th row. If A is only defined as an expression then we will abuse notation and use ( A ) i,j to refer to its ( i, j ) th entry.

For any subset S ⊂ [ k ] we assume × s S iterates over the elements in ascending order.

For any k -tensor Z ∈ R × k a =1 d a with dimensions d 1 , . . . , d k ∈ Z > 0 and any vector c ∈ × k a =1 [ d a ] of indices we use Z c to refer the the ( c 1 , . . . , c k ) th entry of Z .

We use 0 m , 1 m ∈ R m and 0 m × n , 1 m × n ∈ R m × n to refer to all-zero and all-one vectors and matrices, and I n to refer the n × n identity matrix.

B Heteroskedastic James-Stein-type shrinkage estimation

For reference we state a weighted version of Stein's unbiased risk estimate (SURE) [Stein, 1981]:

Lemma B.1. Suppose y ∼ N ( µ , Σ ) for mean µ ∈ R d and diagonal p.s.d. covariance Σ ∈ R d × d , and consider a function ˆ µ : R d → R d s.t. for every i ∈ [ d ] the function ˆ µ i is almost differentiable in y i and we have E ∥ ∇ y · ˆ µ ( y ) ∥ 1 < ∞ . Then for any diagonal matrix W ∈ R d × d we have E ∥ ˆ µ ( y ) -µ ∥ 2 W = E ∥ ˆ µ ( y ) -y ∥ 2 W -Tr( WΣ ) + 2 E [ ∇ y · ( WΣˆ µ ( y ))] .

Proof. Expanding the squared norm and applying Stein's Lemma [Stein, 1981, Lemma 2] to the last term yields

̸

We now turn to James-Stein-type estimators of the form ˆ µ ( y ) = y -c ∥ P ( y -b ) ∥ 2 Σ -1 P ( y -b ) for c ∈ R , θ ∈ R d , and P ∈ R d × d , roughly extending the one considered by Bock [1975] (in a more general form) to P = I d and b = 0 d . Setting x = P ( y -θ ) , we apply Lemma B.1 to show that the expected Σ -1 -weighted MSE is

̸

where to compute the derivative in SURE we made use of the matrix calculus tool of Laue et al. [2018] and the last line follows by von Neumann's trace inequality. This upper bound is minimized at c = Tr( P ) -2 ∥ Σ -1 2 PΣ 1 2 ∥ . Note that for P = I d this setting of c exactly recovers the Bock estimator ˆ µ Bock θ in Equation 6, apart from the positive-part correction.

On the other hand, for shrinking to the pooled mean ˆ µ pooled ( y ) = 1 d × d Σ -1 y Tr( Σ -1 ) we can apply a wellknown fact about the eigenvalues of symmetric rank-one updates (e.g. Bunch et al. [1978, Theorem 1]) to diagonal matrices to obtain ∥ Σ -1 2 PΣ 1 2 ∥ = ∥ ∥ ∥ ∥ I d -Σ -1 2 1 d × d Σ -1 2 Tr( Σ -1 ) ∥ ∥ ∥ ∥ = 1 , which yields the setting in Feldman et al. [2014] of c = Tr( P ) -2 ∥ Σ -1 2 PΣ 1 2 ∥ = d -3 :

Lastly, we note that it is straightforward to extend SureMap to the case of non-diagonal covariance matrices Σ , i.e., when we want to simultaneously release performance estimates for groups with different numbers of intersections (e.g., just race, just age, and both race and age). In this case there will be nonzero covariance between elements of the vector y , e.g., those corresponding to a specific age bracket and an intersection of that bracket with a specific race. To handle this, one only needs to derive the SURE objective estimating the (non-diagonal) Σ -1 -weighted risk of the MAP estimator with (the same, non-diagonal) covariance Σ ; this can be done with the following generalization of Lemma B.1:

Lemma B.2. Suppose y ∼ N ( µ , Σ ) for mean µ ∈ R d and p.s.d. covariance Σ ∈ R d × d , and consider a function ˆ µ : R d → R d s.t. for every i ∈ [ d ] the function ˆ µ i is a.e. differentiable in y i and E ∥ ∇ y · ˆ µ ( y ) ∥ 1 < ∞ . Then for any p.s.d. W ∈ R d × d we have

where ∇ y ˆ µ ( y ) is the Jacobian of ˆ µ with entries ∇ y ; i,j ˆ µ ( y ) = ∂ y j ˆ µ i ( y ) .

Proof. Note that since E y = µ we have by a multivariate version of Stein's lemma (e.g., Liu [1994, Lemma 1]) that the following identity holds for any a.e. continuous f : R d → R s.t. E | ∂ y i f ( y ) | < ∞∀ i ∈ [ d ] :

Using this equality with f ( y ) = ( W ( ˆ µ -y )) i yields

C SureMap estimator

In this section we describe the prior in full for any number of attributes, prove expressivity results reference in §3.1.2, and describe how to compute the estimator using coordinate descent.

C.1 A linear prior

Suppose each group g ∈ [ d ] corresponds to an intersection g ∈ × k a =1 [ d a ] of k attributes, with attribute a having d a possible classes. 5 For example, the first attribute could be one of d 1 different age brackets and the second could be one of d 2 different racial categories. For each subset A ⊂ [ k ] of attributes define a random tensor Z A ∈ R × a ∈ A d a with i.i.d. entries Z A ; c ∼ N (0 , 1) , where c ∈ × a ∈ A [ d a ] is a specific class combination of the attribute A . Then the additive intersectional effects prior on the mean µ g of group g is the weighted sum

with coefficients τ A ∈ R corresponding to each A ∈ 2 [ k ] . These hyperparameters thus determine the strength of the effect of attribute intersection A on the group means, with τ A Z A ; c being added to the means of all groups g s.t. g A = c , i.e. whose attribute intersection results in the specific class combination c .

Letting τ = ( τ ∅ , . . . , τ [ k ] ) ∈ R 2 k ≥ 0 be the vector of hyperparameters τ A and assuming τ [ k ] > 0 , this prior is equivalent to a Gaussian prior µ ∼ N ( θ , Λ ( τ )) with covariance

for matrices U A ∈ { 0 , 1 } d × ∏ a ∈ A d a with orthogonal column vectors, each one corresponding to a different attribute intersection in × a ∈ A [ d a ] and its g th entry indicating whether group g is a subset of that intersection. Their outer product matrices C A = U A U ⊤ A have entries C A ; g,h that are one if g A = h A and zero otherwise, i.e. they indicate if groups g and h have the same type (e.g. (senior, female)) of attribute intersection A (e.g. (age, sex)). In the simplest case, if we are disaggregating across just one attribute, i.e. k = 1 and d 1 = d , then Λ ( τ ) = τ 2 ∅ 1 d × d + τ 2 [ k ] I d . Since k ≤ ⌊ log 2 d ⌋ , using this prior to define the covariance matrix reduces the number of hyperparameters to d +2 k ≤ 2 d for ˆ µ MAP θ , Λ ( τ ) , compared to O ( d 2 ) for a general covariance Λ .

C.2 Expressivity of the linear prior

Theorem C.1. Consider a disaggregated setting with k attributes with d 1 , . . . , d k possible categories, respectively, and total number of groups d = ∏ k a =1 d a . If we use a diagonal covariance Σ ∈ R d × d satisfying Σ h,h = σ 2 /n h ∀ h then the following holds ∀ y ∈ R d :

̸

̸

as desired.

Proof. The first and third results can be easily shown using the fact that Σ is diagonal:

The second result follows from the last by substituting θ = 0 d , so we just need to prove the latter one. First note that

so we just need to show that the second term approaches ˆ µ pooled ( y -θ ) . Weuse the Sherman-Morrison formula to compute

and apply it again to compute

Defining δ = y -θ , we have by L'Hôpital's rule that ∀ g ∈ [ d ]

as desired.

State-Level ACS Classification (0-1 error)

I................

naive....

0.12

pooled

MAE

E

SureMap

MT global

task-averaged

0.10

MT offset

MT Bock

0.08

MT SureMap (θ=0)

MTSureMap(SureSolve)

0.06

MTSureMap(MetaMap)

0.04

10

20

30

40

50

numberoftasks

Common Voice Clusters ASR (WER)

naive

0.035

pooled

MAE

E

SureMap

MT global

task-averaged

0.030

MToffset

MT Bock

0.025

MT SureMap (θ=0)

MTSureMap_(SureSolve)

0.020

MTSureMap(MetaMap)

0.015

5

10

15

20

numberoftasks

C.3 How to set the multi-task center

In the single-task case we set the prior mean θ = 0 d ; this can also be done in the multi-task setting. Alternatively, we can use multi-task data to construct estimator ˆ θ of some true underlying multi-task mean θ and substitute the former for θ ; for simplicity we will restrict to the ourselves to linear estimators ˆ θ = ∑ T t =1 M t y t , where the matrices M 1 , . . . , M T are independent of y 1 , . . . , y T . To determine these matrices, we will assign them some parametric form and set those parameters by minimizing the sum of the Σ -1 t -weighted risks across tasks, as estimated by SURE:

C.3.1 SureSolve

This approach sets the prior mean by finding the θ that minimizes the above SURE objective, which is equivalent to solving an overconstrained linear system:

Note that this estimator can be expressed in the desired linear form ˆ θ = ∑ T t =1 M t y t , with the matrices M t = ( ∑ T s =1 A ⊤ s Σ -1 s A s ) -1 A ⊤ t Σ -1 t A t depending on the parameters determining Λ .

C.3.2 MetaMap

Alternatively, if we assume the prior is correct for some θ and Λ , i.e. y t ∼ N ( θ , Λ + Σ t ) , then a natural estimator to use for θ given a fixed Λ is a MAP estimator of ˆ θ Γ with prior N ( 0 d , Γ ) for some covariance Γ . Define Σ Λ = ( ∑ T t =1 ( Λ + Σ t ) -1 ) -1 and note that y Λ = Σ Λ ∑ T t =1 ( Λ + Σ t ) -1 y t is distributed as N ( θ , Σ Λ ) . Then the MAP estimator of θ

has the necessary form ˆ θ = ∑ T t =1 M t y t for M t = ( Γ -1 + Σ -1 Λ ) -1 ( Λ + Σ t ) -1 . In this case the matrices depend on both Λ and Γ , i.e. the covariance matrices of the prior and the meta-prior.

Algorithm 2: Multi-task SureMap.

D Computation

Here we note additional computational details.

D.1 Nonnegativity of SureMap

In both the single and multi-task cases there is no guarantee that θ is in the convex hull of the values in y or { y t } T t =1 , and in fact it can be negative. Since the quantities we are estimating are usually nonnegative, in practice we do a post-hoc correction forcing θ to be nonnegative.

D.2 Optimization of SureMap

Both the single-task and multi-task variants of SureMap require solving optimization problems, the former over τ 2 = ( τ 2 A ) A ⊆ [ k ] (Eq. 12) and the latter also over υ 2 = ( υ 2 A ) A ⊆ [ k ] (Eq. 14), both of which are vectors in R 2 k ≥ 0 . We find that both problems can be quickly and efficiently optimized using L-BFGS-B over the entire domain R 2 k ≥ 0 and R 2 · 2 k ≥ 0 , respectively, using the default settings provided in

its SciPy implementation. 6 To initialize the algorithm we set all entries of τ 2 and υ 2 to zero except those corresponding to the entire set [ k ] , which we set to one. 7 In the remainder of this section, we describe how to compute gradients (passed to L-BFGS-B) of the multi-task SureMap objective

where the matrices M t differ based on whether we are using the SureSolve or MetaMap variant. The gradients are taken w.r.t. the tuning parameters, which are the coefficients τ 2 1 , . . . , τ 2 2 k used to define the prior covariance Λ = ∑ 2 k i =1 τ 2 i U i U ⊤ i and, in the second case, the coefficients υ 2 1 , . . . , υ 2 2 k used to define the meta-prior covariance Γ = ∑ 2 k i =1 υ 2 i U i U ⊤ i . 8 Noting that A t = ( Λ + Σ -1 t ) -1 Λ -1 = ( I d + ΛΣ -1 t ) -1 and ( Λ + Σ t ) -1 = Λ -1 ( I d + Σ t Λ -1 ) -1 = Σ -1 t ( I d + ΛΣ -1 t ) -1 = Σ -1 t A t , we have the derivative ∂ i A t = -A t U i U ⊤ i Σ -1 t A t w.r.t. τ 2 i , which yields ∂ i Tr( A t ) = Tr( -A t U i U ⊤ i Σ -1 t A t ) and

D.2.1 SureSolve

First note that

where the first step follows by Σ -1 t A t = Σ -1 t ( I d + ΛΣ -1 t ) -1 = ( I d + Σ -1 t Λ ) -1 Σ -1 t = A ⊤ t Σ -1 t . Then for M t = ( ∑ T s =1 A ⊤ s Σ -1 s A s ) -1 A ⊤ t Σ -1 t A t we have

where B i,t = ( ∑ T s =1 A ⊤ s Σ -1 s A s ) -1 Σ -1 t ( A 2 t + A t ) U i U ⊤ i Σ -1 t ( A 2 t + A t ) . We can then compute

and

D.3 MetaMap

Note that ( Λ + Σ t ) -1 = Σ -1 t ( ΛΣ -1 t + I d ) = Σ -1 t A t so we can write the matrices representing the estimator as M t = ( Γ -1 + Σ -1 Λ ) -1 ( Λ + Σ t ) -1 = ( Γ -1 + ∑ T s =1 Σ -1 s A s ) -1 Σ -1 t A t . Thus

We can then compute

and

Lastly, note that M t = ( Γ -1 + Σ -1 Λ ) -1 Σ -1 t A t = ( I d + ΓΣ -1 Λ ) -1 ΓΣ -1 t A t and redefine the derivative ∂ i to be w.r.t. υ 2 i , so that

Then we have ∂ i Tr( A t M t ) = Tr( A t ( I d + ΓΣ -1 Λ ) -1 U i U ⊤ i ( Σ -1 t A t -Σ -1 Λ M t )) and

D.4 Handling groups with no data

It is frequently the case that a specific group g may not have any examples, i.e., n g = 0 , and so we cannot define Σ g,g = σ 2 /n g . At the same time, we may need to handle the dimension associated with this group, either because we still need to report a value for it or because we are in the multi-task setting and other tasks do have examples of that group that may allow us to get an estimate. To handle this issue, in all computations we only use the precision matrix Σ -1 , which can be easily defined in the case of n g = 0 using Σ -1 g,g = n g /σ 2 = 0 . Note in particular that ( Λ + Σ ) -1 = Σ -1 ( ΛΣ -1 + I d ) -1 .

D.5 Handling near-singular covariances

Since we would like to optimize over the entire domain τ 2 ∈ R 2 k ≥ 0 but Λ ( 0 2 k ) = 0 d × d is singular, we avoid inverting prior covariance matrices (i.e. Λ , Γ ) in all computations. Note in particular that A t = ( Λ -1 + Σ -1 ) -1 Λ -1 = ( I d + ΛΣ -1 ) -1 .

D.6 Handling group variances for AUC

While obtaining unbiased variance estimates for averages is straightforward, it is more involved for multi-point statistics such as AUC. For simplicity we just use ( n g +1) / (12 n g n (0) g n (1) g ) , where n ( i ) g is the number of members of group g with label i ; this estimate is derived from the variance estimate of the related Mann-Whitney U -statistic [Siegel, 1956]. However, future work may consider improvements based on more complicated approaches [Cortes and Mohri, 2003, Wang and Guo, 2020], or using bootstrapping techniques as is done by Herlihy et al. [2024] for structured regression.

E Derivation of SureMap estimators via ridge regression

In this appendix we show that ˆ µ SM and ˆ µ SM t can be derived as ridge regression estimators.

Similar to the notation in Appendix C, we consider k sensitive attributes (like sex, age, etc.) indexed by a ∈ [ k ] . The a th sensitive attribute is denoted g a and takes values in [ d a ] . The joint sensitive attribute g takes values in G = [ d 1 ] × [ d 2 ] ×··· × [ d k ] with the cardinality d = |G| = d 1 d 2 · · · d k . For A ⊆ [ k ] , write g A for the tuple ( g a ) a ∈ A .

E.1 Single-task regression model

Each joint attribute g ∈ G identifies an intersectional group (of order k ). We seek to jointly fit means of all these groups, represented as a vector µ ∈ R |G| , based on the vector of empirical means y ∈ R |G| . We posit a regression model

where Φ ∈ R |G|×|J| is a feature matrix and β ∈ R |J | is a vector of regression coefficients. The columns of Φ are referred to as features and indexed by j ∈ J , where J is some index set. We assume a Gaussian prior on the parameter β ∼ N ( 0 , K ) and a Gaussian distribution over observation errors, so y ∼ N ( Φ β , Σ ) , where K and Σ are, respectively, the prior covariance matrix and error covariance matrix.

The error covariance matrix Σ is assumed fixed and positive definite, the prior covariance matrix K is viewed as a hyperparameter (with a specific structure to reduce its dimension); for simplicity, we assume that K is positive definite (but that assumption is not necessary). Any generic forms of Φ , K and Σ can be considered. Here we exhibit a specific form of Φ and K under which the MAP regression estimator is the same as ˆ µ SM , when provided with the same error covariance matrix Σ .

We consider a structured form of matrix Φ , with features being indicators of tuples of sensitive attribute values. The features are indexed by j ∈ J , where

We will also consider subsets of features that focus on a specific subset of sensitive attributes A ⊆ [ k ] ,

and write Φ A ∈ R |G|×|J A | for the submatrix composed of features indexed by j ∈ J A .

The prior matrix K is diagonal, specified using a set of hyperparameters τ = ( τ A ) A ⊆ [ k ] , with diagonal entries

The MAP solution under the model described above is obtained by solving

which is a weighted ridge regression problem. Setting the gradient to zero, the solution must satisfy

and defined as

Hence,

This yields a regression-based estimator

In contrast, by Eqs. (11) and (7), the ˆ µ SM estimator is obtained as

where the matrix Λ (following Eq. 22 in Appendix C.1) depends on the hyperparameter vector τ as

where U A is precisely the submatrix Φ A defined above. Thus, writing ϕ A, c ∈ R |G| for the column of Φ indexed by ( A, c ) , we obtain

Hence, the ˆ µ SM estimator can be rewritten as

Theorem E.1. With Φ and K defined as above (Eqs. 44 and 45), we have ˆ µ reg = ˆ µ SM .

In the proof we use a variant of Sherman-Morrison-Woodbury formula [Horn and Johnson, 2013, Eq. 0.7.4.1], specialized to symmetric positive definite matrices:

Proposition E.1 (Symmetric SMW Formula) . Let A ∈ R n × n and R ∈ R m × m be symmetric positive definite matrices and let X ∈ R n × m . Then

Proof of Theorem E.1. It suffices to show that

We do this by direct calculation, using the symmetric SMW formula:

The first equality follows by the SMW formula with A = K -1 , R = Σ -1 and X = Φ ⊤ , the second by multiplying out the terms, and the third by the SMW formula with A -1 = ΦKΦ ⊤ , R -1 = Σ and X equal to the d × d identity matrix.

E.2 Multi-task regression model

Consider multi-task setting with tasks indexed by t = 1 , . . . , T . We write T = [ T ] for the set of tasks. Multi-task setting can be reduced to single-task setting by viewing the task id as an additional sensitive attribute, and performing the same analysis as for the single-task setting, but with G ′ = T × G , with the dimension d ′ = |G ′ | = Td .

Formally, we seek to fit µ ′ ∈ R |G ′ | based on the vector of empirical means y ′ ∈ R |G ′ | . The entries of µ ′ and y ′ are denoted as µ ′ t, g and y ′ t, g for the task t and the intersectional group g . We denote task-specific slices of these vectors as µ ′ t = µ ′ { t }×G and y ′ t = y ′ { t }×G .

Features are indexed by elements of J ′ = S × J , where S = T ∪ { glob } , so there are taskspecific and global features. The feature matrix Φ ′ ∈ R |G ′ |×|J ′ | uses the single-task feature matrix Φ ∈ R |G|×|J| (defined in Eq. 44) as a building block. The matrix Φ ′ has a block structure, with |T | × |S| blocks of size |G| × |J | defined as

We posit the regression model where

and

where β ′ ∈ R J ′ . With the definition of Φ ′ as above, this boils down to

As before, we assume a Gaussian prior and Gaussian errors, β ′ ∼ N ( 0 , K ′ ) , y ′ ∼ N ( Φ ′ β ′ , Σ ′ ) .

The error covariance Σ ′ ∈ R |G ′ |×|G ′ | has a block-diagonal form with single-task error covariance matrices Σ t ∈ R |G|×|G| , for t ∈ [ T ] , along the diagonal.

We consider a structured form of the prior covariance matrix K ′ specified by two vectors of hyperparameters: τ = ( τ 2 A ) A ⊆ [ k ] and υ = ( υ A ) A ⊆ [ k ] . The matrix K ′ is diagonal and positive definite, with entries

It can also be viewed as a block-diagonal matrix with |S| diagonal blocks of size |J | × |J | defined as

where K is the single-task prior matrix defined in Eq. (45), and V ∈ R |J |×|J | is an analogous matrix based on the vector of hyperparameters υ rather than τ , with diagonal entries

The multi-task regression-based estimator is obtained by solving the resulting MAP regression problem. Similarly to the single-task case, the estimator takes form

with the individual task estimates denoted ˆ µ ′ t reg .

The multi-task SureMap estimator, introduced in §3.2.2, takes form

.

Theorem E.2. With Φ ′ , K ′ and Σ ′ defined as above, we have ˆ µ ′ t reg = ˆ µ SM t for all t ∈ [ T ] .

In the proof we use the following identities:

Proposition E.2. Let A , B ∈ R n × n be symmetric positive definite matrices and let I be the n × n identity matrix. Then

Proof. We have

where the first equality follows by the SMW formula (Proposition E.1), and the remaining equalities follow by simple algebraic manipulations.

Proof of Theorem E.2. Starting with Eq. (49), we have

where the second equality follows by the same reasoning as in the proof of Theorem E.1, and the third equality follows by Proposition E.2.

We next evaluate Φ ′ K ′ Φ ′ ⊤ . Note that the matrices Φ ′ , K ′ and Σ ′ have the following block structure:

Thus,

where in the last line we introduced the notation Λ ′ for the block-diagonal matrix with T copies of matrix Λ along diagonal, and notation X for the matrix obtained by stacking T copies of the |G| ×|G| identity matrix I on top of each other. Plugging the last expression back into Eq. (52), we obtain

where the second equality follows by the SMW formula (Proposition E.1) with A = Λ ′ + Σ ′ , R = Γ , and X = X . We next focus on simplifying the last term in the bracket in Eq. (53).

Since Λ ′ and Σ ′ are block-diagonal, the matrix ( Λ ′ + Σ ′ ) -1 is also block-diagonal with blocks along the diagonal equal to ( Λ + Σ t ) -1 for t = 1 , . . . , T . Thus,

and similarly,

Therefore,

Plugging this back in Eq. (53), we obtain

Using, again, the fact that matrices Λ ′ and Σ ′ are block-diagonal, the task-specific blocks of ˆ µ ′ reg must be equal to

where the second equality follows by Proposition E.2.

F Resources

F.1 Data and model resources

We make use of data / models with the following sources / licenses:

Strack et al. [2014]: CC Attribution License.

Weerts et al. [2023]: MIT License.

Ardila et al. [2020]: CC0 License.

Radford et al. [2023]: Apache-2.0 License.

https://archive.ics.uci.edu/dataset/2/adult : CC BY 4.0 License

https://huggingface.co/JaaackXD/Llama-3-70B-GGUF : Meta Llama 3 License

We use the third and fourth resources to create a dataset of Whisper model evaluations on Common Voice utterances, which result in the Common Voice and CVC tasks described in §4.2; we also use the last two resources to create a dataset of in-context evaluations of Llama 3 on the Adult dataset, which results in the Adult task described in §4.1. Both resources are released under a CC BY 4.0 License and are available at https://github.com/mkhodak/SureMap .

F.2 Computational

By far the most computation was required to generate the Common Voice, CVC, and Adult tasks, which was done on a machine with two RTX-8000 GPUs and took about a week. As described above, the corresponding datasets are made publicly available and easy to re-use without any GPU access. Given these dataset, the main experiments were run on a 40-core machine and take a couple hours, with the vast majority of this time spent running the structured regression approach of Herlihy et al. [2024]; see Figure 7 for a summary of the costs associated with each method evaluated in this paper. Code for both generating the task data and reproducing the method evaluations is available at https://github.com/mkhodak/SureMap .

State-Level ACS Classification

CommonVoiceClustersASR

102

....

naive

.I..

naive

....

pooled

101

.I..

pooled

101

Bock

Bock

(seconds)

Struct.Reg.

cost(seconds

100

-+-

Struct.Reg.

SureMap

十-

SureMap

100

MT global

MT global

10-1

MT offset

MToffset

cost

10-1

MTBock

MT Bock

MTSureMap

10-2

MTSureMap

10-2

10-3

10-3

2

5

10

23

50

2

3

6

11

20

numberoftasks

numberoftasks

DiabetesRegression(MAE)

DiabetesRegression(MAE)

DiabetesRegression(MAE)

0.5

0.7

0.30

naive

0.6

0.4

(sdnob

0.25

pooled

Bock

(sdno6

0.5

W 0.3

MA

0.20

Struct.Reg.

0.4

0.2

0.15

SureMap

0.3

A

M

0.10

0.2

0.1

0.05

0.1

0.01

0.032

0.1

0.316

1.0

0.01

0.032

0.1

0.316

1.0

0.01

0.032

0.1

0.316

1.0

(5)

(15)

(48)

(150)

(476)

(35)

(110)

(348)

(1101)

(3481)

(3)

(8)

(25)

(79)

(251)

subsampling rate(median groupsize)

subsamplingrate(mediangroupsize)

subsamplingrate(mediangroupsize)

DiabetesRegression(MSE)

DiabetesRegression(MSE)

Diabetes Regression(MSE)

3.0

naive

4.0

2.5

(sdnob

2.0

pooled

(sdnob

3.5

Bock

3.0

MAE

2.0

1.5

Struct.Reg.

2.5

1.5

1.0

SureMap

2.0

1.5

1.0

0.5

1.0

0.5

0.5

0.01

0.032

0.1

0.316

1.0

0.01

0.032

0.1

0.316

1.0

0.01

0.032

0.1

0.316

1.0

(5)

(15)

(48)

(150)

(476)

(35)

(110)

(348)

(1101)

(3481)

(3)

(8)

(25)

(79)

(251)

subsamplingrate(median groupsize)

subsamplingrate(mediangroupsize)

subsamplingrate(median groupsize)

G Additional evaluations

In Figures 8 & 9, we compare evaluation methods on the regression variant of the Diabetes task, where the goal is to predict a patient's length of stay using ridge regression. In Figures 10 & 12, we compare evaluation methods on Diabetes and SLACS with AUC as the target metric. In Figures 11 & 13, we compare evaluation methods on Common Voice and CVC with the character error rate (CER) as the target metric. And finally, in Figures 14 & 15, we compare evaluation methods on Diabetes, Adult, Common Voice, SLACS, and CVC according to RMSE instead of MAE.

DiabetesClassification(AUC)

DiabetesClassification(AUC)

DiabetesClassification(AUC)

0.14

0.12

naive

0.18

0.12

(sdno6

0.10

pooled

(sdnob

0.16

E

MAI

0.10

0.08

MAE(large

0.08

0.06

SureMap

Bock

Struct.Reg.

MAE(smallg

0.14

0.12

0.10

0.06

0.08

0.04

0.06

0.04

0.02

0.02

0.04

0.02

0.01

0.032

0.1

0.316

1.0

0.01

0.032

0.1

0.316

1.0

0.01

0.032

0.1

0.316

1.0

(5)

(15)

(48)

(153)

(483)

(36)

(114)

(362)

(1143)

(3616)

(3)

(9)

(30)

(94)

(297)

subsamplingrate(median group size)

subsamplingrate(mediangroupsize)

subsamplingrate(median groupsize)

CommonVoiceASR(CER)

CommonVoiceASR(CER)

CommonVoiceASR(CER)

0.10

0.09

naive

0.08

(sdno6

0.08

pooled

(sdnob

0.08

0.07

Bock

MA

E

0.04

0.06

0.05

MAE(large

0.06

0.04

Struct.Reg.

SureMap

MAE(small g

0.06

0.04

0.03

0.02

0.02

0.02

0.01

0.01

0.032

0.1

0.316

1.0

0.01

0.032

0.1

0.316

1.0

0.01

0.032

0.1

0.316

1.0

(2)

(6)

(19)

(60)

(189)

(4)

(12)

(37)

(117)

(370)

(1)

(2)

(6)

(19)

(61)

subsamplingrate(median groupsize)

subsamplingrate(mediangroupsize)

subsamplingrate(mediangroupsize)

State-LevelACSClassification(AuC)

...

naive

0.12

pooled

MAE

Bock

aged

0.10

-

Struct.Reg.

SureMap

sk-averag

0.08

MT global

MT offset

0.06

MT.Bock

task

MTSureMap

0.04

0.01

0.032

0.1

0.316

1.0

(3)

(11)

(33)

(105)

(333)

subsampling rate(median group size)

task-levelresults(median grp.size =33)

multiplicativeimprovement

5

MT Bock

MT SureMap

overnaive

naive

4

MAE

B

=

2

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

2.25

distancefromgroundtruthtoMTmedian

Common Voice Clusters ASR (CER)

0.05

I.

naive

....

pooled

MA

E

0.04

Bock

task-averaged

Struct.Reg.

0.03

SureMap

MT global

MToffset

0.02

MTBock

MT-SureMap

0.01

0.01

0.032

0.1

0.316

1.0

(2)

(7)

(22)

(68)

(216)

subsampling rate (median group size)

task-level results (median grp. size = 22)

multiplicativeimprovement

3.5

X

MT Bock

MTSureMap

inMAEovernaive

3.0

naive

2.5

2.0

1.5

1.0

0.1

0.2

0.3

0.4

0.5

distancefromground truthtoMTmedian

Diabetes Classification(0-1error)

Diabetes Classification (0-1 error)

Diabetes Classification (0-1 error)

0.225

1,

0.14

I.

.. naive

0.175

0.200

(sdno6

0.12

pooled

0.25

0.150

0.10

-f-

Bock

0.20

-+-

Struct.Reg.

'1.

0.06

二

1.

-+-

SureMap

0.15

0.075

0.050

0.025

K

0.04

0.02

0.10

0.05

T

0.01

0.032

0.1

0.316

1.0

0.01

0.032

(0.1

0.316

1.0

0.01

0.032

(23

0.316

1.0

(5)

(15)

(48)

(150)

(476)

(35)

(110)

(348)

(1101)

(3481)

(3)

(8)

(6)

(251)

Adult Classification(0-1 error)

Adult Classification(0-1 error)

0.14

0.10

naive

0.16

0.12

(sdno.6

pooled

0.10

1

0.08

--日

Bock

0.12

-I-

Struct.Reg.

0.06

K

0.04

f-

SureMap

0.06

0.04 -

0.02

0.04

0.02

0.02

0.01

0.032

0.316

1.0

0.01

0.032

(0.1

0.316

1.0

0.01

0.032

0.316

1.0

(4)

(12)

(119)

(375)

(19)

(59)

(187)

(591)

(1869)

(1)

(3)

(34)

(108)

CommonVoiceASR(WER)

CommonVoiceASR(WER)

0.175

0.16

naive

0.18

0.150

..

pooled

0.125

-+-

Bock

0.14

RMSE

Struct.Reg.

0.100

+

+

SureMap

0.10

0.075

0.06

0.050

0.04

0.06

0.025

0.02

0.04

0.02

0.032

0.316

1.0

10'0

(0)

0.032

0.316

(6)

(60)

(681)

(4)

(1)

(2)

(19)

subsampling rate (median group size)

subsampling rate (median group size)

subsampling rate (median group size)

State-Level ACS Classification (0-1 error)

0.225

naive

RMSE

0.200

pooled

0.175

-I--

Bock

--

Struct.Reg.

ged

0.150

---

SureMap

task-averag

0.125

十

MT global

MT offset

0.100

MT Bock

0.075

MTSureMap

0.050

0.025

0.01

0.032

0.1

0.316

1.0

(3)

(8)

(26)

(83)

(262)

CommonVoice ClustersASR(WER)

0.12

I.

naive

RMSE

0.10

....

---

pooled

Bock

-I-

Struct.Reg.

task-averaged

0.08

+-

SureMap

+

MT global

0.06

十

MToffset

十

MT Bock

0.04

MTSureMap

0.02

0.01

0.032

0.1

0.316

1.0

(2)

(7)

(21)

(67)

(210)

subsampling rate (median group size)

task-level results (median grp. size = 26)

multiplicativeimprovement

6

MT Bock

MTSureMap

inRMSEovernaive

5

naive

4

3

X

2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

task-levelresults(median grp.size = 21)

multiplicativeimprovement

3.5

MTSureMap

MT Bock

inRMSEovernaive

naive

3.0

2.5

2.0

1.5

1.0

0.05

0.10

0.15

0.20

0.25

0.30

distance from ground truth to MT median

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: our abstract and introduction both describe the introduction of a new method for disaggregated evaluation, which we do.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: c.f. Sections 3.3 and 5.3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: c.f. Appendix D

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: https://github.com/mkhodak/SureMap

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: c.f. Section 5, Appendix D, and the linked code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: all figures except scatter plots have 95% confidence intervals.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: we have reviewed the Code of Ethics and our work conforms.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: c.f. Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The one asset released is just evaluations of an open-source model on opensource data and so unlikely to be misused.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: we release new datasets in conjunction with three of the tasks described in Section 4; see that section and Appendix F.1 for further details. This data is provided at the code link.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposed SureMap, a promising method for solving multi-task disaggregated evaluation problem. The key innovation of SureMap lies on transforming the problem into structured simultaneous Gaussian mean estimation and incorporating external data, e.g. from the AI system creator or from their other clients. Experiments on disaggregated evaluation tasks in multiple domains show the promising performance.

STRENGTHS:
1) Well motivated.
2) Introduce several datasets for disaggregated evaluation and propose method that uses SURE to tune the parameters of a well-chosen
Gaussian prior before applying MAP estimation.
3) Experiments are promising. Competitive in both single task and multi-task evaluation.

WEAKNESSES:
As analyzed in Section 2.4:
1) Gaussian assumption may result in underperformance on heavy-tailed data.
2) Incorporating data from multiple clients may be costly.

QUESTIONS:
1) Can the authors provide some analysis or insights when the data contradicts with Gaussian assumption?
2) Can the authors visualize/show the cost when incorporating data from clients?

LIMITATIONS:
I don't think there are any negative societal impacts of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies disaggregated evaluation, which aims to estimate the performance of models on various subpopulations. This problem is challenging due to small sample sizes in subpopulations, thus leading to inaccurate performance estimates. This issue is magnified when multiple clients use the same AI model and require individualized evaluations, which is referred to as multi-task disaggregated evaluation. \n\nThis paper designs a method that transforms the problem into a structured simultaneous Gaussian mean estimation problem. The method is comprised of two components: first, conduct maximum a posteriori (MAP) estimation for  Gaussian mean estimation and then apply cross-validation-free tuning using Stein’s unbiased risk estimate (SURE). Furthermore, the method employs an additive intersectional effect prior for capturing relationships between subpopulations with a limited number of hyperparameters.\n\nThe method, namely SureMap, is evaluated on various disaggregated evaluation tasks across different domains, including automated speech recognition (ASR) and tabular census data. The method shows high estimation accuracy for both single-task and multi-task settings over naive estimation, pooled estimation, and Bock estimation. Moreover, the method improves performance estimation even in data-poor regimes.\n\nSTRENGTHS:\n- This paper presents a novel method, SureMap, designed to improve the accuracy of performance estimation for models on various subpopulations. This is crucial for assessing the fairness and robustness of machine learning models, especially when subpopulations are small or data is scarce.\n- Across tabular census data and automated speech recognition datasets in both single task and multitask settings, the proposed method outperforms previous estimation approaches by up to 50%. The results are consistent for various sampling rates and task numbers up to 50.\n\nWEAKNESSES:\n- The efficiency discussion of the method needs to be expanded. It would be better to list the complexity of the proposed method with existing methods and report the actual runtime. \n- There are certain cases that the proposed method performs worse than the pooled estimation, such as in Figure 2. It would be better to analyze the reasons behind such results.\n\nQUESTIONS:\n- For subpopulations with few samples, how does this paper compute their ground truth and evaluate the estimation? \n- It would be better to clarify the pooled estimator. For example, what does the notation $h$​ in Equation 3? \n- It would be better to explain the working of the baselines, such as the Bock method.\n\nLIMITATIONS:\nThis work has discussed its limitations in the setup of Gaussian distribution.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author developed a disggregated evaluation method called SureMap, which has high estimation accuracy for both multi-task and single-task disggregated evaluations. SureMap transforms the problem into structured simultaneous Gaussian mean estimatio, incorporating external data. This method further combines Maximum A Posteriori (MAP) estimation and cross-validation-free tuning via Stein's risk estimate (SURE). Significant improvements in accuracy were observed in disggregated evaluation tasks.\n\nSTRENGTHS:\n1. The author introduces a new method, SureMap, which tunes the parameters of the selected Gaussian prior using SURE before applying MAP estimation. Only linear parameters are needed to recover several natural baselines for disggregated evaluation.\n2.  The author introduces disggregated evaluation datasets for both single-task and multi-task settings.\n3. SureMap shows good results in both single-task and multi-task settings.\n\nWEAKNESSES:\n1. I would like to know if there are any rules that need to be followed in the selection of disaggregated evaluation datasets to ensure fairness, as well as the reason why the disaggregated evaluation goal is set as the mean 0-1 error.\n2. The fairness assessment is an application biased towards real-world scenarios. Does using Gaussian distribution as a prior align with real-world evaluations？\n\nQUESTIONS:\nDisaggregated evaluation is a core task in the fairness assessment of AI systems. This paper provides a possible solution and gives the corresponding analysis.  \n\nI would like to point out that my expertise does not directly align with the specific field of this article. Nevertheless, I have carefully read the paper several times, attempting to provide constructive feedback. I look forward to reading the insights of other reviewers, whose expertise is more closely related to the subject, to further inform my final score.\n\nLIMITATIONS:\nsee the weaknesses and questions above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper introduces SureMap, a new method for disaggregated evaluation of AI systems especially for multi-task setting. The authors model the problem as Gaussian mean estimation and use a structured covariance prior that captures intersectional effects. SureMap is evaluated on several datasets, including a new multi-task ASR dataset they introduce. They show SureMap generally outperforms existing methods, especially for small subgroups and in the multi-task setting.

STRENGTHS:
1. This paper tackles an important problem in fairness and evaluation of AI systems and formulation as a Gaussian estimation problem with a structured prior.
2. Theoretical analysis showing SureMap can recover existing baselines.
3. This paper introduces the multi-task ASR datasets for evaluation.
4. The empirical results are strong, especially in the multi-task setting.

WEAKNESSES:
1. The Gaussian assumption may be too strong for some real-world settings
2. No comparison to some recent methods like GP-based approaches
3. The multi-task formulation assumes clients are willing to share data statistics

QUESTIONS:
1. Have you considered non-Gaussian models? For example, a t-distribution might better handle heavy-tailed performance data that can occur in practice.
2. The multi-task setting assumes clients are willing to share summary statistics. How realistic is this assumption? Could you explore privacy-preserving ways to share this information?
3. How does SureMap compare to GP-based approaches like in [1]? The method aims to handle low-data regimes well, which seems relevant here.

[1] Active assessment of prediction services as accuracy surface over attribute combinations. NeurIPS 2021

LIMITATIONS:
N.A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SureMap, a novel method for disaggregated evaluation, aimed at improving the estimation accuracy of performance metrics for machine learning models across different subpopulations. The proposed method is designed to address both single-task and multi-task settings, where multiple clients independently evaluate the same AI model on their respective data. SureMap leverages maximum a posteriori (MAP) estimation with a well-chosen Gaussian prior, fine-tuned using Stein’s unbiased risk estimate (SURE), to achieve high estimation accuracy. The authors evaluate SureMap across various domains, demonstrating significant improvements over existing baselines.\n\nSTRENGTHS:\nOriginality: The introduction of SureMap for simultaneous mean estimation in disaggregated evaluation is novel, particularly in addressing both single-task and multi-task scenarios using a structured Gaussian prior and SURE for parameter tuning.\nQuality: The paper is well-structured, with a thorough theoretical foundation, clear methodological development, and comprehensive experiments. The approach of combining MAP estimation with SURE tuning is well-justified and effectively demonstrated.\nClarity: The paper is clearly written, with detailed explanations of the methods and assumptions. The inclusion of theoretical proofs and detailed descriptions of the datasets and experiments adds to the clarity.\nSignificance: The ability to improve disaggregated evaluation accuracy has significant implications for fairness in AI, as it allows for better assessment of model performance across different demographic groups. This work has the potential to impact how AI systems are evaluated and deployed, ensuring more equitable outcomes.\n\nWEAKNESSES:\nComplexity of Implementation: While the method is theoretically sound, the practical implementation of SureMap may be complex, particularly the coordinate descent algorithm used for tuning parameters. This could pose challenges for practitioners.\nDependence on Gaussian Assumptions: The method relies on Gaussian assumptions for the prior and noise distributions. In real-world scenarios, data distributions may deviate significantly from Gaussian, potentially affecting the performance of SureMap.\nScalability: The scalability of the method in very large-scale settings is not fully explored. The computational cost associated with MAP estimation and SURE tuning might be prohibitive for very large datasets or a very high number of tasks.\nEmpirical Evaluation: While the empirical results are promising, they are primarily based on synthetic and semi-synthetic datasets. More extensive evaluation on real-world datasets across diverse domains would strengthen the findings.\n\nQUESTIONS:\n1.\tImplementation Details: Can the authors provide more detailed pseudocode or a step-by-step guide for implementing the coordinate descent algorithm used in SureMap? This would help in better understanding and replicating the method.\n\t2.\tNon-Gaussian Data: How robust is SureMap to deviations from the Gaussian assumptions? Have the authors considered alternative distributions, and how would the method need to be adapted for such cases?\n\t3.\tScalability: What are the computational requirements of SureMap for very large datasets or a high number of tasks? Can the authors provide any benchmarks or comparisons in terms of runtime and memory usage?\n\t4.\tReal-World Applications: Can the authors provide examples of real-world applications where SureMap has been or could be successfully applied? This would help in contextualizing the method’s practical utility.\n\nLIMITATIONS:\n•\tGaussian Assumption: The reliance on Gaussian assumptions is acknowledged, and the authors suggest potential future work on other distributions like Student’s t.\n\t•\tData Integration Costs: The potential burden or cost of integrating data from multiple clients is mentioned, and the authors propose the use of model provider data as a mitigation strategy.\n\t•\tScalability: Although not deeply explored, the authors note the computational challenges and suggest future work to improve scalability and efficiency.\n\t•\tFairness and Over-Confidence: The authors caution against over-confidence in a model’s fairness based solely on disaggregated evaluations and emphasize the need for careful application of SureMap and similar methods.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}