SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation

Authors: mikhail  khodak
lester  mackey
alexandra  chouldechova
miroslav  dudík
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 14:00:32

Abstract: Disaggregated evaluation&mdash;estimation of performance of a machine learning model on different subpopulations&mdash;is a core task when assessing performance and group-fairness of AI systems.
A key challenge is that evaluation data is scarce, and subpopulations arising from intersections of attributes (e.g., race, sex, age) are often tiny.
Today, it is common for multiple clients to procure the same AI model from a model developer, and the task of disaggregated evaluation is faced by each customer individually.  This gives rise to what we call the *multi-task disaggregated evaluation problem*, wherein multiple clients seek to conduct a disaggregated evaluation of a given model in their own data setting (task).  In this work we develop a disaggregated evaluation method called **SureMap** that has high estimation accuracy for both multi-task *and* single-task disaggregated evaluations of blackbox models.  SureMap's efficiency gains come from
(1) transforming the problem into structured simultaneous Gaussian mean estimation and (2) incorporating external data, e.g., from the AI system creator or from their other clients.  Our method combines *maximum a posteriori* (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE).
We evaluate SureMap on disaggregated evaluation tasks in multiple domains, observing significant accuracy improvements over several strong competitors.

Main paper: 
 introduction: 
 Evaluation is a key challenge in modern AI, with much effort spent deciding what metrics to measure, with which methods, and on what data. This challenge is especially acute in fairness assessment, which requires not only high-quality data to run a model and score its outputs but also demographic information for defining groups. Due to the high cost of obtaining high-quality evaluation data, the issue of sample complexity-sample size needed to get a good performance estimate-remains salient, especially when we want to release not just one overall measure but instead to output a disaggregated evaluation that captures variation among demographic subpopulations of the data [Barocas et al., 2021] . For instance, we might want to assess group fairness by examining the variation in performance across groups of users defined by intersections of the demographic attributes age, race, and sex. The naive approach of independently evaluating each group's performance on its own data can fail because the sample sizes of intersectional groups rapidly decrease as we consider more attributes [Herlihy et al., 2024] . Recent work has shown how to improve upon naive methods by combining data from multiple subpopulations to inform their individual performance estimates [Miller et al., 2021 , Herlihy et al., 2024] .In today's technology landscape it is common for multiple clients to procure the same model (e.g., an automated speech recognition or language model) from an AI developer, with each client performing a disaggregated evaluation of the same model on their own data. We refer to this problem as the multi-task disaggregated evaluation. We formalize and study this problem, showing that one can improve the disaggregated evaluations of individual clients by using multi-task data in the form of summary statistics from other clients or from the model provider. Our approach uses the (out-ofdistribution) multi-task data to set the parameters of a multivariate normal prior and then performs maximum a posteriori (MAP) inference on the (in-distribution) client data. Formally, we model the problem as Gaussian mean estimation and design a simple-yet-expressive additive prior that can capture many different relationships between subpopulations. Drawing upon classical statistics, we fit prior parameters by minimizing Stein's unbiased risk estimator [SURE, Stein, 1981] . While motivated by multi-task considerations, we show that our method also performs well in the single-task setting.1. SureMap: We introduce a method that uses SURE to tune the parameters of a well-chosen Gaussian prior before applying MAP estimation. The prior is motivated by its attainment of a good efficiency-expressivity tradeoff, requiring only a linear (in the number of subpopulations) number of parameters to recover several natural baselines for disaggregated evaluation. 2. Datasets: Disaggregated evaluation has few benchmarks [Herlihy et al., 2024] , so we introduce new ones for both the single-task and multi-task settings, covering automated speech recognition (ASR) and also tabular domains (with linear models and also in-context LLMs). 3. Single-task: We find that SureMap is always competitive with strong baselines from prior work, while improving significantly in some settings with intersectional sensitive attributes. 4. Multi-task: Incorporating data from multiple clients into SureMap yields significant improvements across all evaluated settings. This multi-task approach is more accurate even with just one additional task and is the only method to consistently outperform the naive and pooling baselines.Disaggregated evaluation is a core task in the fairness assessment of AI systems [Barocas et al., 2021] . Past work has sought to improve estimation accuracy by combining information across different groups, e.g., via Bayesian modeling [Miller et al., 2021] , Gaussian process approximation of loss surfaces [Piratla et al., 2021] , and structured regression [Herlihy et al., 2024] . The last work found that classical James-Stein-type mean estimation [James and Stein, 1961, Bock, 1975] is often competitive, and so we adopt it as our first non-naive baseline. We also compare to structured regression itself, which turns out to have a tight mathematical connection to SureMap; indeed, apart from our use of Gaussian (ridge) rather than Laplace (lasso) priors (regularization)-as well as our use of a more flexible tuning based on SURE rather than cross-validation-the method of Herlihy et al.[2024] can be viewed as the discriminative counterpart to our generative approach (see §E for details).Within the disaggregated evaluation literature we are the first to formulate and study multi-task disaggregated evaluation. This is an important direction because (a) model providers often have their own data or data from multiple clients that can inform the evaluation and (b) transferring information across distributions is a key way to handle very low-sample regimes. We also contribute several datasets that we hope will spur further development in disaggregated evaluation.SureMap relies on applying classical mean estimation tools to quantities modeled as Gaussian means. Notably, Miller et al. [2021] model scores via well-studied distributions-e.g., Gaussians-but since scores are related non-linearly to metrics it is unclear if this can lead to similarly simple estimators. To tune parameters, we use SURE, a popular statistical approach [Li, 1985, Donoho and Johnstone, 1995] . Specifically, in the empirical Bayes tradition, we use it to set the MAP estimator of a hierarchical model. Using SURE to tune the scale of an isotropic Gaussian prior was shown to be asymptotically (in the dimension) optimal in the case of heteroskedastic data distributions [Xie et al., 2012] . Since disaggregated evaluation data is highly heteroskedastic due to variation in group size, this is positive evidence for our approach, although our prior is non-isotropic and has many more variance parameters. 
setup: 
 We first describe the disaggregated evaluation problem ( §2.1), recast it as a Gaussian mean estimation ( §2.2), and motivate a multi-task variant ( §2.3), all while introducing several baselines estimators.We want to assess a predictive model p : X → Y under some distribution D over input space X and output space Y using error measure ℓ : Y × Y → R. For example, in image classification, D is a distribution over (image, label) pairs and ℓ is the 0-1 error. To simplify notation, we mainly deal with the composite function f (z) = ℓ(y, p(x)) acting on points z = (x, y) in the product space Z = X × Y.In disaggregated evaluation, Z is assumed to be a union of d ≥ 1 disjoint subsets Z g , each associated to some subpopulation or group g ∈ [d] , where we use [k] to denote the set {1, . . . , k}. As a running example, suppose each point z ∈ Z is an individual whose sex s and age a are categorical variables with d 1 and d 2 possible values, respectively. Then d = d 1 d 2 and each point has an associated index g = (s -1)d 2 + a denoting its intersection of sex s ∈ [d 1 ] and age a ∈ [d 2 ]. The task is then to use a set S ∼ D n of n ≥ 1 i.i.d. samples from the distribution D to estimate the vector of true subpopulation errors µ ∈ R d , with components µ g = E z∼D|Zg [f (z)]. 1 We write μ(S) ∈ R d for an estimator of µ with components denoted as μg (S).Empirically, we measure estimation accuracy, i.e., the performance of the performance estimate, via mean absolute error (MAE) L MAE µ (μ(S)) = 1 d ∥μ(S) -µ∥ 1 , which is easy to interpret and less sensitive to outliers than mean squared error (MSE). Our method development, however, is based on a count-weighted version of MSE, where we denote group counts as n g = |S ∩ Z g |:L n µ (μ(S)) = 1 d d g=1 n g (μ g (S) -µ g ) 2 .(1)We conclude the setup with two baselines. The first is the naive estimator returning group means:EQUATIONWhile unbiased, μnaive can perform poorly on groups with few samples. The second baseline, the pooled estimator, returns an identical quantity-the overall sample mean-for all groups:EQUATIONThis estimator is generally biased (unless all group means are equal), but it can perform well in low-sample regimes thanks to a much lower variance.We use a simple but natural model to aid in the design of disaggregated evaluation methods. Specifically, denoting the naive estimator by y = μnaive (S) ∈ R d , we model group g's entry y g as being drawn from a Gaussian with (unknown) mean µ g and (known) variance σ 2 /n g , where σ 2 is shared across groups. This reduces the problem of disaggregated evaluation, as defined in §2.1, to that of estimating the mean of a multivariate Gaussian with known diagonal covariance Σ g,g = σ 2 /n g given a single sample y ∼ N (µ, Σ). Our model has many advantages in the disaggregated evaluation setting: 1. By the central limit theorem, y is asymptotically normal with mean µ and diagonal covariance Σ for many distributions D of interest, even when the underlying data is non-Gaussian. Furthermore, because the methods derived from our model only take y and Σ as input, they can be applied even when the evaluated statistic is not the pointwise average assumed by the setup in §2.1, so long as y ∼ N (µ, Σ) holds asymptotically. An example of this is when y g = μnaive g (S) corresponds to the area under the ROC curve (AUC) computed over group g's data S ∩ Z g [Lehmann, 1951] ; we demonstrate SureMap's applicability to AUC empirically in §G (Figures 10 & 12) . 2. While a shared variance is a strong assumption, it is perhaps the simplest way of incorporating the inductive bias that Σ g,g will be highly correlated with the inverse of n g , the number of samples from group g. In practice, we set σ 2 to be the pooled estimate1 n-d d g=1z∈S∩Zg (f (z) -y g ) 2 . 3. Gaussian mean estimation is one of the best-studied problem in statistics, with numerous well-tested baselines and approaches for developing new methods. In particular, we make significant use of the classic James-Stein approach [James and Stein, 1961, Bock, 1975] , SURE [Stein, 1981] , and empirical Bayesian estimation methods [Robbins, 1964] . 4. In the multi-task setting, clients are likely to be unwilling to share their actual data but possibly more willing to share group summary statistics. Thus methods developed for our Gaussian model-which only require the group means y, group counts n, and an estimate of σ 2 -will be more broadly applicable than methods that act directly on the dataset S ⊂ Z. ; in the next section, assume ng > 0 ∀ g.This model can also be naturally extended-using a non-diagonal covariance Σ-to disaggregated evaluation with non-disjoint groups, e.g., to simultaneously estimate performance both for all women and for only women in their forties. In the interest of brevity we focus on the disjoint group setting.We can easily extend this model to study multi-task disaggregated evaluation, in which for each task t = 1, . . . , T (e.g., a client of the model provider) we observe a set S t ⊂ Z of n t samples from the task distribution D t . The goal is then to output T vectors μt that are close on-average to the tasks' subpopulation errorsµ t;g = E z∼Dt [f (z)|z ∈ Z g ].Converting to our Gaussian model, we observe T vectors y t ∼ N (µ t , Σ t )-where we set Σ t;g,g = σ 2 /n t;g for some globally shared σ 2 and task-specific group count vectors n t ∈ Z d ≥0 -and must output T mean estimates μt ({y t } T t=1 ) ∈ R d . We consider two natural multi-task baseline estimators. The first is the global naive estimator (or global estimator for short), which combines the data from all tasks, computes a single global vector of group averages, and uses it as the estimate for each task:EQUATIONWhile low variance, the global estimator ignores variation across tasks. Our second baseline-the multi-task offset estimator-shifts the global estimate on each task to ensure that the task's pooled mean is preserved (thus accounting for the variation in pooled means across individual tasks):EQUATION)3 MethodsIn the last section we reduced the problem of disaggregated evaluation to that of estimating a mean µ ∈ R d given a sample y ∼ N (µ, Σ), where Σ is known and diagonal. We now design a method, SureMap, for the latter problem. Our technical approach involves the following two steps: 1. Choosing a parameterized mean estimator. We use the MAP estimator under a multivariate normal prior that we design specifically for intersectional subpopulations. 2. Tuning the estimator's hyperparameters. We use SURE to estimate the quality of our estimator, which we then optimize over the choice of hyperparameters using the L-BFGS-B algorithm. 
designing a parameterized estimator: 
 As mean estimation is a vast area, we use three criteria for designing an estimator: it should (1) dominate baselines such as μnaive and μpooled ; (2) have relatively few hyperparameters; and(3) handle heteroskedasticity stemming from variation in group sizes. One natural source of candidates are James-Stein-type shrinkage estimators: the original James-Stein estimator famously dominates μnaive in MSE and has no hyperparameters to tune [James and Stein, 1961] , satisfying our first two desiderata. Furthermore, while James and Stein [1961] assumed an isotropic Σ, subsequent estimators such as the following variant of an estimator due to Bock [1975] do handle heteroskedastic Σ:EQUATIONwhere (•) + = max{•, 0}, and θ ∈ R d is a default estimate towards which y is shrunk.4 However, empirically we find that this often underperforms the pooled estimator in low-sample regimes; further, the form of μBock θ shows that the amount of shrinkage towards θ is the same for each coordinate g ∈ [d], despite intuition suggesting that we should shrink less for groups g with more samples. Corrections to this tend to be involved and difficult to generalize [Efron and Morris, 1973] .We thus turn to a different family of well-known Gaussian mean estimators: those that return the mode of the posterior distribution assuming µ is sampled from the conjugate prior N (θ, Λ) with mean θ ∈ R d and positive-definite covariance Λ ∈ R d×d (e.g., Gelman et al. [2014, Equation 3 .12]): EQUATION)Λ ← A⊆[k] τ 2 A CA Output: (I d + ΛΣ -1 ) -1 // optimize SURE using L-BFGS-B τ ← arg min τ ∈R 2 k ≥0 ∥A(τ )y∥ 2 Σ -1 -2 Tr(A(τ ))// estimate group means using MAP Output: y -A(τ )y U K G G Q a 2 A q l t C P b 7 9 X f v 8 O t B F Z e o H T H E a K X a c i E Z y h k 8 a t 3 c O H C F k x j h D u U S t r 4 L 6 8 O h q 3 / K A d z E D f k 7 A m P q n R H W 9 7 r W i S 8 U J B i l w y Y 4 Z h k O P I M o 2 C S y i b U W E g Z / y W X c P Q 0 Z Q p M C M 7 u 7 + k + 0 6 Z 0 C T T 7 q V I Z + q / G 5 Y p Y 6 Y q d p O K 4 Y 1 Z 9 C r x f 9 6 w w O R 0 Z E W a F w g p f w t K C k k x o 1 U Z d C I 0 c J R T R x j X w t 1 K + Q 3 T j K O r b C 4 l V u 4 P E 0 i i A z + 0 U Z U U J z a K F f X D s n Y 6 z q k k W 0 l l 0 3 U Y LV 3 + d m 6 z Q H L o 8 k 5 m + j J k B K V L o o k A J l 7 k G p m I J F / H t c a V f / A F t R J a e 4 S i H v m L X q U g E Z + i o Q X N t 5 z 5 C V g w i h D v U y r J r K K 9 2 B 0 0 / a A X j o l 9 B W A O f 1 N U Z r H r N a J j x Q k G K X D J j e m G Q Y 9 8 y j Y J L K B t R Y S B n / N b t 7 z m Y M g W m b 8 f 3 l 3 T T M U O a Z N q 9 F O m Y f T 9 h m T J m p G L X q R j e m M 9 a R X 6 n 9 Q p M D v t W p H m B k P K J U V J I i h m t w q B D o Y G j H D n A u B b u V s p v m G Y c X W Q f X G L l / j C E J N r 2 Q x t V T n F i o 1 h R P y x r p e 2 U i r I V V T Z c h u H n x L 6 C 8 9 1 W u N / a P 9 3 z 2 0 d 1 m n N k n W y Q L R K S A 9 I m J 6 R D u o Q T Sx 7 I I 3 n y n r 2 / 3 j / v / 6 R 1 y q t n f p M P 5 b 2 + A W f p r G Q = < / l a t e x i t >+ ω 2 age < l a t e x i t s h a 1 _ b a s e 6 4 = " 0 f z K L K J F i d P i 5 T X B E j t J u a P 0 Y t E = " > A A A C Q X i c b V D L S s R A E J z 4 d n 2 t e h I v g 0 E Q l S U R U Y + L X j w q u C p s 1 m U y 2 9 H B m S T M d M Q l R L / G q 3 6 E X + E n e B O v X p y s O f h q G C i q u r u m K 0 y l M O h 5 L 8 7 Q 8 M j o 2 P j E Z G 1 q e m Z 2 r j 6 / c G q S T H N o 8 U Q m + j x k B q S I o Y U C J Z y n G p g K J Z y F 1 w e l f n Y D 2 o g k P s F + C h 3 F L m M R C c 7 Q U t 3 6 0 s Z d g C z r B g i 3 q F V u 4 H a T X U J x s d W t u 1 7 D G x T 9 C / w K u K S q o + 6 8 U w 9 6 C c 8 U x M g l M 6 b t e y l 2 c q Z R c A l F L c g M p I x f 2 / 1 t C 2 O m w H T y w Q 0 F X b V M j 0 a J t i 9 G O m C / T + R M G d N X o e 1 U D K / M b 6 0 k / 9 P a G U Z 7 n V z E a Y Y Q 8 y + j K J M U E 1 o G Q n t C A 0 f Z t 4 B x L e x f K b 9 i m n G 0 s f 1 w C Z W 9 o Q d R s O 7 6 e V A 6 h V E e h I q 6 f l E p T a u U V F 5 S RK 3 h B B H G y 6 f h G U T m F c B K G i r j + o l J Z V S q o o q U H d Z u j / T u w v O N 9 q + j v N n d N t t 3 V Q p T l J l s k K W S c + 2 S U t c k R O S J t w 8 k A e y R N 5 d l 6 c N + f d + f h q r T n V z B L 5 U b X x T x B C s Q k = < / l a t e x i t > !(ω ) = ω 2 → Figure 1: Matrices C A ∈ {0, 1}d×d , whose linear combination defines the covariance Λ(τ ) of our additive intersectional effects prior (Eq. 9). In this example, there are two categories for sex (F,M) and three for age (<25,25-64,>64), yielding d = 6 groups. Shaded squares are 1s, unshaded are 0s.MAP naturally handles heteroskedasticity by weighting low-variance coordinates g more heavily, satisfying our third criterion; however, its most general form has O(d 2 ) hyperparameters, violating the second. We next use problem structure to restrict the number of hyperparameters needed to define Λ while still allowing μMAP θ,Λ to express every baseline introduced in §2.1, satisfying our first criterion.For brevity, we build up our estimator somewhat informally; a full description is in §C. where θ ∈ R d and τ ∅ , τ sex , τ age , τ sex,age ∈ R ≥0 are hyperparameters, ζ ∼ N (0, 1) is a scalar effect shared across all groups, the vector ζ sex ∈ R d1 has its sth entry ζ sex s ∼ N (0, 1) shared by all groups g whose sex is s, the vector ζ age ∈ R d2 has its ath entry ζ age a ∼ N (0, 1) shared by all groups g whose age is a, and the vector ζ ∼ N (0 d , I d ) contains an independent noise term ζ g for each group g.The hyperparameter τ ∅ quantifies how much we expect all of the means to be shifted (by a shared positive or negative value) from the default θ. Hyperparameters τ sex and τ age express how large we expect contributions of sex and age alone to be towards the means. And finally, non-zero τ sex,age gives the prior flexibility to model heterogeneity across all intersectional groups.Given the vector of hyperparameters τ = (τ ∅ , . . . , τ sex,age ) ∈ R 4 ≥0 , the prior can be written more compactly as µ ∼ N (θ, Λ(τ )), where the covariance isEQUATIONfor matrices C A ∈ {0, 1} d×d s.t. each entry C A;g,h is one iff groups g and h agree on the attributes included in A. In particular, we have that C ∅ = 1 d×d is the all-ones matrix, the entries C sex;g,h of C sex ∈ {0, 1} d×d are one iff groups g and h share the same sex attribute, the matrix C age is analogous, and C sex,age = I d is the d × d identity. This structure is visualized in Figure 1 .As detailed in §C.1, this prior can be naturally extended to any number of attributes k using a covariance matrix Λ(τ) ∈ R d×d specified by a vector τ ∈ R 2 k ≥0 of 2 k hyperparameters. Since k ≤ ⌊log 2 d⌋, the total number of hyperparameters (including θ ∈ R d ) is d + 2 k = O(d),which is much smaller than the O(d 2 ) complexity of the general case. We can further reduce this by fixing the entries of θ, constraining them to be identical, or setting them using external (e.g., multi-task) data.Despite this reduction in hyperparameters, we can show that for a suitable choice of τ , the estimator μMAP θ,Λ(τ ) recovers many estimators of interest, including the naive estimator and the (possibly offset) pooled estimator (see §C.2). This means that MAP with our structured prior should be able to outperform all four baselines from the previous section, if appropriately tuned.Having specified a parameterized estimator, there remains the question of setting its parameters θ and τ . One might want to treat this as a hyperparameter tuning problem and use a data-splitting approach; however, the dimensionality of the problem makes standard techniques either expensive or noisy, and data splitting introduces additional randomness and design decisions into an already data-poor environment. We instead make continued use of our Gaussian assumption and turn to SURE, which given a differentiable estimator μ : R d → R d returns an unbiased estimate of its weighted MSE L n µ using sample data y ∼ N (µ, Σ):EQUATIONwhere given any W ≻ 0 d×d we denote ∥x∥ 2 W = ⟨x, Wx⟩ ∀ x ∈ R d . Using SURE we can now tune the parameters of μ by minimizing Rn µ (y) in a manner similar to empirical risk minimization.In the single-task setting, we fix θ = 0 d and tune the variance parameters τ ∈ R 2 k ≥0 . Letting A(τ ) = (Λ -1 (τ ) + Σ -1 ) -1 Λ -1 (τ ), we define the single-task SureMap estimator asEQUATIONfor τ = arg minEQUATION)The optimization problem in the second line comes from substituting μMAP 0 d ,Λ(τ ) into SURE (Eq. 10). It is nonconvex, but we find that it can be quickly solved to sufficient accuracy with L-BFGS-B [Byrd et al., 1995] , a standard method for bound-constrained optimization of differentiable functions.To generalize SureMap to the multi-task setting we propose to specify θ and τ by minimizing SURE aggregated across tasks, i.e.,Rnt µt (y t ). While setting both parameters via direct optimization of this objective is the most straightforward approach, we find that it performs worse than single-task SureMap when there are only a few tasks (T ≤ 5) and rarely improves significantly above the multi-task global and offset estimators. This can be explained by observing the few-task limit-i.e. T = 1-in which case optimizing the aggregated SURE objective results in setting θ = y 1 and thus makes the multi-task estimator equivalent to the naive estimator.We find that a better approach is to treat the choice of θ as its own simultaneous mean estimation problem and apply the SureMap approach to it. In particular, our model y t ∼ N (µ t , Σ t ) and our prior µ t ∼ N (θ, Λ) imply that the samples y t ∼ N (θ, Λ+Σ t ) have mean θ and known covariances (apart from tuning parameters). Therefore, the MAP estimator of θ itself given a hyperprior θ ∼ N (0 d , Γ) with covariance Γ ≻ 0 will have the formθ = Γ -1 + T t=1 (Λ + Σ t ) -1 -1 T t=1 (Λ + Σ t ) -1 y t .To reduce the number of tuning parameters, we use a prior of the same form as before by specifying Γ = Λ(υ) for υ ∈ R 2 k ≥0 , i.e., the same structured covariance as described in §3.1.1 but with separately tuned parameters (see §3.1.1 and §C.1). Substituting the meta-level MAP estimator of θ into the MAP estimator of µ t and tuning the parameters τ and υ by optimizing the sum of SUREs (Eq. 10) across tasks yields our multi-task SureMap estimator (see §C.3 for details):EQUATION)The optimization problem on the last line can again be approximately solved using L-BFGS-B.Various modeling assumptions impact performance of SureMap. For instance, the Gaussian error assumption is less appropriate when errors are heavy-tailed. In §G, Figure 9 , we consider MSE as an example of a target metric with heavy-tailed observation errors. We find that SureMap still performs well, but is no longer superior to previous approaches. One avenue for improvement would be to use a variance-stabilizing transformation (e.g., Hawkins and Wixley [1986] ) prior to applying SureMap.Note that SureMap achieves its improved accuracy by shrinking naive estimates towards a less granular estimator (e.g., a pooled mean). As a result the estimation is biased towards less disparity, which could lead to overly optimistic conclusions about fairness. For this reason it is extremely important to examine not just the point estimates, but also confidence intervals. These can be obtained, for example, by viewing SureMap as a regression approach ( §E) and leveraging inference techniques for regression. 
datasets: 
 We evaluate our approach in several representative settings for disaggregated evaluation, including two tabular settings appearing in previous works [Miller et al., 2021 , Herlihy et al., 2024 , Liu et al., 2024] , and three new settings: a multi-task tabular setting based on state-level U.S. census data and both a single-task and a multi-task ASR evaluation setting.We consider three tabular datasets, two for the single-task and one for the multi-task setting, covering two important domains where fairness concerns can arise: healthcare records and demographic data. While we focus on the classification task and 0-1 error, in §G we also report results for regression (Figures 8 & 9 ) and for classification with AUC as the target metric (Figures 10 & 12) .Diabetes. This is a tabular dataset of Strack et al. [2014] , containing around 100K patient records with six race, two sex, and three age categories. We evaluate a logistic regression classifier trained to predict patient disposition after a hospital stay (discharged or otherwise). The target metric is the 0-1 error.Adult. We use the classic Adult census dataset [Kohavi, 1996] to evaluate performance of an in-context LLM learner-specifically llama-3-70b-in predicting whether a person makes more or less than $50K after being provided with eight examples via a modification of the prompt template of Liu et al. [2024] . The target metric is the 0-1 error, disaggregation is by race, sex, and age.. This is a tabular dataset for multi-task setting derived from the census data for all U.S. states and Puerto Rico assembled by Ding et al. [2021] . Each datapoint corresponds to a person in one of nine race and two sex categories; we consider three age categories: below 25, 25-64, and over 64. The underlying task is to classify each person as earning either more or less than $50K. We train a regularized logistic model on the data from California, and seek to evaluate its performance on the other 50 states/territories, which comprise the tasks. The target metric is the 0-1 error.We also introduce both single-task and multi-task speech recognition datasets, based on applying the popular Whisper ASR model [Radford et al., 2023] -specifically whisper-tiny-on the English part of the Common Voice (CV) dataset [Ardila et al., 2020] , which contains utterances from individuals in one of nine age and three sex categories. Large and small groups are defined as above and below median group size.Common Voice. This is a single-task dataset obtained by combining the validation and test partitions of the CV dataset. We calculate the word-error rate (WER) across all the utterances of each individual, which becomes the target metric to be predicted.Common Voice Clusters (CVC). This is a multi-task ASR dataset. To construct it, we first cluster the utterances in the train partition of the CV dataset into 20 clusters by applying k-means to the sums of GloVe word embeddings [Pennington et al., 2014] of their corresponding text strings. To model task relatedness, we then randomly reassign each utterance to a random cluster with probability α ∈ [0, 1]. The resulting clusters are the tasks. The target metric is the word-error rate (WER) across all the utterances of each individual in a given cluster. In most experiments we use α = 1 2 , but we also investigate what happens when α varies between zero, i.e., the original clusters, and one, corresponding to identically distributed tasks. 
evaluation: 
 Our main metric is MAE relative to a ground truth vector, which we take to be the mean of all available data for each subpopulation g ∈ [d], except those with fewer than 40 samples. In our main results we subsample with replacement from the entire dataset at different rates and track performance as a function of the sizes of the resulting datasets. To obtain 95% confidence intervals we conduct 200 and 40 random trials at each subsampling rate in the single-task and multi-task settings, respectively.We compare SureMap to the naive (Eq. 2) and pooled (Eq. 3) baselines, as well as to the Bock estimator with shrinkage towards the pooled estimator (Eq. 17) and the structured regression estimator of Herlihy et al. [2024] . On both Diabetes and Adult, SureMap significantly outperforms all competitors (Figure 2 , top & middle), the greatest improvement is on subpopulations with limited data. In §G, we consider a regression variant of Diabetes and observe similar results (Figure 8 ). On the Common Voice task, SureMap performs roughly similarly to Bock, while outperforming structured regression at some subsampling rates (Figure 2 , bottom); here again the gains are driven by better performance on small groups. Pooling performs best when data is extremely limited, but it is not competitive with even modestly more data, and also underperforms on small groups.In the multi-task setting, we use all the single-task methods as baselines while adding multi-task (MT) ones, including MT global (Eq. 4), MT offset (Eq. 5), and an MT extension of Bock (Eq. 6) in which θ g is set using the average across the group g data on all other tasks. We first consider the SLACS task, for which Figure 3 (top left) shows that MT SureMap significantly outperforms other methods in the low-data regime while matching the best one (MT Bock) in the high-data regime. At subsampling rate 0.1, Figure 3 (top right) also shows that using multi-task data leads to improvement on all but two of the fifty tasks, that the reduction in MAE over the naive estimator on a typical task is 2x, and that this improvement only loosely correlates with the task's ground truth distance from the multi-task median. On the other hand, it also shows that while MT Bock's improvements are typically smaller, on SLACS it improves performance for every state (including the two where MT SureMap is worse).On the CVC task, the MT offset baseline is the most competitive, except at the lowest subsampling rate where pooling is better and at higher subsampling rates where it stops improving with additional data (Figure 3 , bottom left). SureMap outperforms it and all other methods across all subsampling rates and its advantage is greatest in low-data regimes, where it even outperforms pooling. In the task-level evaluation in Figure 3 (bottom right) we see that on every task, MT SureMap attains an improvement of 2-3.5x over the naive baseline and almost always outperforms MT Bock. Furthermore, the latter performs substantially worse on tasks whose ground truth vectors are far away from the multi-task center while MT SureMap is not affected.We next look at how the degree of included intersectional effects and multi-task structure affect performance. In Figure 4 (left), we evaluate utility of including higher-order interactions in the structured prior. We implement SureMap variants with up to ℓth-order interactions by setting τ A for |A| ∈ {ℓ+1, . . . , k -1} to zero (but not for |A| = k). We observe that including zeroth-order effects (ℓ = 0) in single-task SureMap (i.e., shrinkage to pooling) improves upon the usual single-parameter Gaussian prior (ℓ = -1) in low-data regimes. Adding first-order effects (ℓ = 1) leads to substantial further im- provement, but second-order effects (ℓ = 2) make performance slightly worse. A similar effect can be observed among the multi-task variants, except there is no loss (but also no gain) to using the highestorder variant (ℓ = 2). Overall, this study suggests that using the full-order SureMap is a reasonable default but that most of the method's effectiveness comes from zeroth-and first-order effects.Figure 4 (right) tracks performance as the task-similarity parameter defining the CVC task is varied. MT SureMap outperforms all methods at all settings and is also not as strongly affected by the task similarity, at least as it is defined for the CVC data. This suggests that the structured prior we use may be useful even if the dataset means are quite different but the underlying evaluation problem (in this case estimating WER of ASR models) is the same.Lastly, in Figure 5 we study how the number of tasks affects multi-task performance. On both SLACS and CVC, MT SureMap outperforms all single-task baselines (the best one being the single-task SureMap) at T = 2 tasks, i.e., it can take advantage of even very little external information. In contrast, on CVC, the competitor multi-task methods (e.g., MT offset and MT Bock) do not even outperform single-task methods until T ≥ 5 tasks. These results demonstrate that, unlike these comparators, multitask SureMap can be confidently used even when only one additional client's worth of data is available. 
conclusion: 
 We have introduced SureMap, a disaggregated evaluation approach, which combines MAP estimation under a structured Gaussian prior with hyperparameter tuning via SURE. SureMap achieves substantial empirical improvements over strong baselines in both single-task and multi-task settings. Valuable future directions include improving robustness to heavy-tailed data and developing multitask methods that can handle client privacy concerns. More broadly, we hope our work will have a positive impact by allowing model users to more accurately identify fairness-related harms, the first step towards mitigating them. However, using SureMap and any other disaggregated evaluation approach must be done with care, so as to not risk overconfidence in a model's fairness. • For any set S we use 2 S to denote its powerset.• For any vector a ∈ R d we use a i to denote its ith entry. If a is only defined as an expression then we will abuse notation and use (a) i to refer to its ith entry. • For any vector a ∈ R k and any subset S ⊂ [k] with elements s 1 < • • • < s m we define a S = (a s1 • • • a sm ) to be the vector of the entries of a whose indices correspond to the elements of S sorted in ascending order.• For any subset S ⊂ [k] we assume × s∈S iterates over the elements in ascending order.• For any matrix A ∈ R m×n we use A i,j to denote its (i, j)th entry and A i,: its ith row. If A is only defined as an expression then we will abuse notation and use (A) i,j to refer to its (i, j)th entry. Proof. Expanding the squared norm and applying Stein's Lemma [Stein, 1981, Lemma 2] to the last term yieldsEQUATIONWe now turn to James-Stein-type estimators of the form μ(y) = y -c ∥P(y-b)∥ 2 Σ -1 P(y -b) for c ∈ R, θ ∈ R d, and P ∈ R d×d , roughly extending the one considered by Bock [1975] (in a more general form) to P ̸ = I d and b ̸ = 0 d . Setting x = P(yθ), we apply Lemma B.1 to show that the expected Σ -1 -weighted MSE isE∥μ(y) -µ∥ 2 Σ -1 = d + E c 2 -2c Tr(P) ∥x∥ 2 Σ -1 + 4c Tr(xx ⊤ Σ -1 P) ∥x∥ 4 Σ -1 = d + E c 2 -2c Tr(P) ∥x∥ 2 Σ -1 + 4c Tr(Σ -1 2 xx ⊤ Σ -1 2 Σ -1 2 PΣ 1 2 ) ∥x∥ 4 Σ -1 ≤ d + E c 2 ∥Σ∥ -2c Tr(ΣP) + 4c∥Σ -1 2 PΣ 1 2 ∥ ∥x∥ 2 Σ -1 (16)where to compute the derivative in SURE we made use of the matrix calculus tool of Laue et al. [2018] On the other hand, for shrinking to the pooled mean μpooled (y) = 1 d×d Σ -1 y Tr(Σ -1 ) we can apply a wellknown fact about the eigenvalues of symmetric rank-one updates (e.g. Bunch et al. [1978, Theorem 1] )to diagonal matrices to obtain ∥Σ -1 2 PΣ 1 2 ∥ = I d -Σ -1 2 1 d×d Σ -1 2 Tr(Σ -1 )= 1, which yields the setting in Feldman et al. [2014] EQUATIONLastly, we note that it is straightforward to extend SureMap to the case of non-diagonal covariance matrices Σ, i.e., when we want to simultaneously release performance estimates for groups with different numbers of intersections (e.g., just race, just age, and both race and age). In this case there will be nonzero covariance between elements of the vector y, e.g., those corresponding to a specific age bracket and an intersection of that bracket with a specific race. EQUATIONwhere ∇ y μ(y) is the Jacobian of μ with entries ∇ y;i,j μ(y) = ∂ yj μi (y).Proof. Note that since Ey = µ we have by a multivariate version of Stein's lemma (e.g., Liu [1994, Lemma 1 ]) that the following identity holds for any a.e. continuous f :EQUATIONUsing this equality with f (y) = (W(μy)) i yieldsEQUATION 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The one asset released is just evaluations of an open-source model on opensource data and so unlikely to be misused.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: c.f. Appendix F.1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: we release new datasets in conjunction with three of the tasks described in Section 4; see that section and Appendix F.1 for further details. This data is provided at the code link. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section we describe the prior in full for any number of attributes, prove expressivity results reference in §3.1.2, and describe how to compute the estimator using coordinate descent.Suppose each group g ∈ [d] corresponds to an intersection g ∈ × k a=1 [d a ] of k attributes, with attribute a having d a possible classes. 5 For example, the first attribute could be one of d 1 different age brackets and the second could be one of d 2 different racial categories. For each subset A ⊂ [k] of attributes define a random tensor Z A ∈ R × a∈A da with i.i.d. entries Z A;c ∼ N (0, 1), where c ∈ × a∈A [d a ] is a specific class combination of the attribute A. Then the additive intersectional effects prior on the mean µ g of group g is the weighted sumEQUATIONwith coefficients τ A ∈ R corresponding to each A ∈ 2 [k] . These hyperparameters thus determine the strength of the effect of attribute intersection A on the group means, with τ A Z A;c being added to the means of all groups g s.t. g A = c, i.e. whose attribute intersection results in the specific class combination c.Letting τ = (τ ∅ , . . . , τ [k] ) ∈ R 2 k ≥0be the vector of hyperparameters τ A and assuming τ [k] > 0, this prior is equivalent to a Gaussian prior µ ∼ N (θ, Λ(τ )) with covarianceEQUATIONfor matrices U A ∈ {0, 1} d× a∈A da with orthogonal column vectors, each one corresponding to a different attribute intersection in × a∈A [d a ] and its gth entry indicating whether group g is a subset of that intersection. Their outer product matrices C A = U A U ⊤ A have entries C A;g,h that are one if g A = h A and zero otherwise, i.e. they indicate if groups g and h have the same type (e.g. (senior, female)) of attribute intersection A (e.g. (age, sex)). In the simplest case, if we are disaggregating across just one attribute, i.e. k = 1 and d 1 = d, then Λ(τ ) = τ 2 ∅ 1 d×d + τ 2 [k] I d .∈ R d×d satisfying Σ h,h = σ 2 /n h ∀ h then the following holds ∀ y ∈ R d : 1. If τ A = 0 ∀ A ̸ = [k] and θ ∈ R d then lim τ [k] →∞ μMAP θ,Λ(τ ) (y) = μnaive (y) 2. If τ A = 0 ∀ A ̸ ∈ {∅, [k]} then lim τ ∅ →∞ τ [k] →0 μMAP 0 d ,Λ(τ ) (y) = μpooled (y) 3. If τ A = 0 ∀ A ̸ = [k] and θ ∈ R d then lim τ [k] →0 μMAP θ,Λ(τ ) (y) = θ 4. If τ A = 0 ∀ A ̸ ∈ {∅, [k]} and θ ∈ R d then lim τ ∅ →∞ τ [k] →0 μMAP θ,Λ(τ ) (y) = μpooled (y -θ) + θ as desired.Proof. The first and third results can be easily shown using the fact that Σ is diagonal:EQUATIONThe second result follows from the last by substituting θ = 0 d , so we just need to prove the latter one. First note thatEQUATIONso we just need to show that the second term approaches μpooled (y-θ). We use the Sherman-Morrison formula to computeEQUATIONand apply it again to computeEQUATIONDefining δ = yθ, we have by L'Hôpital's rule thatEQUATIONas desired. In the single-task case we set the prior mean θ = 0 d ; this can also be done in the multi-task setting. Alternatively, we can use multi-task data to construct estimator θ of some true underlying multi-task mean θ and substitute the former for θ; for simplicity we will restrict to the ourselves to linear estimators θ = T t=1 M t y t , where the matrices M 1 , . . . , M T are independent of y 1 , . . . , y T . To determine these matrices, we will assign them some parametric form and set those parameters by minimizing the sum of the Σ -1 t -weighted risks across tasks, as estimated by SURE:EQUATIONforA t = (Λ -1 + Σ -1 t ) -1 Λ -1 , i.e. μMAP θ,Λ (y t ) = y t + A t ( θ -y t ) = y t -A t (y t - T s=1 M s y s ).This approach sets the prior mean by finding the θ that minimizes the above SURE objective, which is equivalent to solving an overconstrained linear system:θ = dT + arg min θ∈R d T t=1 ∥A t (θ -y t )∥ 2 Σ -1 t -2 Tr(A t ) = T t=1 A ⊤ t Σ -1 t A t -1 T t=1 A ⊤ t Σ -1 t A t y t(31) Note that this estimator can be expressed in the desired linear form θ = T t=1 M t y t , with the matricesM t = T s=1 A ⊤ s Σ -1 s A s -1A ⊤ t Σ -1 t A t depending on the parameters determining Λ.Alternatively, if we assume the prior is correct for some θ and Λ, i.e. y t ∼ N (θ, Λ + Σ t ), then a natural estimator to use for θ given a fixed Λ is a MAP estimator of θΓ with prior N (0 d , Γ) for some covariance Γ. Define Σ Λ = (T t=1 (Λ + Σ t ) -1 ) -1 and note that y Λ = Σ Λ T t=1 (Λ + Σ t ) -1 y t is distributed as N (θ, Σ Λ ). Then the MAP estimator of θ θΓ = Γ -1 + Σ -1 Λ -1 Σ -1 Λ y Λ = T t=1 Γ -1 + Σ -1 Λ -1 (Λ + Σ t ) -1 y t (32) has the necessary form θ = T t=1 M t y t for M t = (Γ -1 + Σ -1 Λ ) -1 (Λ + Σ t ) -1 .In this case the matrices depend on both Λ and Γ, i.e. the covariance matrices of the prior and the meta-prior. = T t=1 |St| σ 2 ← 1 n-dT T t=1 d g=1 z∈S t ∩Zg (f (z) -yt;g) 2 for task t ∈ [T ] do Σ -1 t ← diag(nt)/σ 2// compute auxiliary matrices (avoids inverting Λ) Method At(τ ): // compute prior covariance (matrices CA are defined in Appendix C.1)Λ ← A∈2 [k] τ 2 A CA Output: (I d + ΛΣ -1 t ) -1// compute auxiliary matrices (avoids inverting Σ -1 t and Γ) Method Mt(τ , υ):// compute prior and meta-prior covariancesΛ ← A∈2 [k] τ 2 A CA Γ ← A∈2 [k] υ 2 A CA Output: I d + Γ T s=1 Σ -1 s (ΛΣ -1 s + I d ) -1 -1 ΓΣ -1 t (ΛΣ -1 t + I d ) -1// estimates prior mean using MAP Method θ(τ , υ): Output: T t=1 Mt(τ , υ)yt // optimize the sum of SUREs across tasks using L-BFGS-B τ , υ = arg minτ ,υ∈R 2 k ≥0 T t=1 ∥At(τ )( θ(τ , υ) -yt)∥ 2 Σ -1 t + 2 Tr (At(τ )(Mt(τ , υ) -I d ))// return an estimate of the group means of each task t using MAP for task t ∈ [T ] do Output: yt + At(τ )( θ(τ , υ) -yt)Here we note additional computational details.In both the single and multi-task cases there is no guarantee that θ is in the convex hull of the values in y or {y t } T t=1 , and in fact it can be negative. Since the quantities we are estimating are usually nonnegative, in practice we do a post-hoc correction forcing θ to be nonnegative.Both the single-task and multi-task variants of SureMap require solving optimization problems, the former over τ 2 = (τ 2 A ) A⊆[k] (Eq. 12) and the latter also over υ 2 = (υ 2 A ) A⊆[k] (Eq. 14), both of which are vectors in R 2 k ≥0 . We find that both problems can be quickly and efficiently optimized using L-BFGS-B over the entire domain R 2 k ≥0 and R 2•2 k ≥0 , respectively, using the default settings provided in its SciPy implementation. 6 To initialize the algorithm we set all entries of τ 2 and υ 2 to zero except those corresponding to the entire set [k], which we set to one. 7 In the remainder of this section, we describe how to compute gradients (passed to L-BFGS-B) of the multi-task SureMap objectivedT + T t=1 E t A t y t - T s=1 M s y s 2 Σ -1 t + 2 Tr(A t M t -A t ) (33)where the matrices M t differ based on whether we are using the SureSolve or MetaMap variant. The gradients are taken w.r.t. the tuning parameters, which are the coefficients τ 2 1 , . . . , τ 2 2 k used to define the prior covariance Λ =2 k i=1 τ 2 i U i U ⊤i and, in the second case, the coefficients υ 2 1 , . . . , υ 2 2 k used to define the meta-prior covariance Γ =EQUATIONEQUATIONFirst note thatEQUATIONwhere the first step follows byEQUATION)whereB i,t = T s=1 A ⊤ s Σ -1 s A s -1 Σ -1 t (A 2 t + A t )U i U ⊤ i Σ -1 t (A 2 t + A t ). We can then compute∂ i Tr(A t M t ) = -2 Tr A t B i,t - T s=1 B i,s M t -Tr(A t U i U ⊤ i Σ -1 t A t M t ) = -2 Tr A t B i,t + U i U ⊤ i Σ -1 t A t - T s=1 B i,s M t (37)and∂ i A t y t - T s=1 M s y s 2 Σ -1 t = ∂ i T s=1 T r=1 (y t -M s y s ) ⊤ A ⊤ t Σ -1 t A t (y t -M r y r ) = -2 T s=1 (∂ i M s y s ) ⊤ A ⊤ t Σ -1 t A t (y t -θ) + 2(y t -θ) ⊤ A ⊤ t Σ -1 t ∂ i A t (y t -θ) = 4 T s=1 y ⊤ s B ⊤ i,s A ⊤ t Σ -1 t A t (y t -θ) -4 θ⊤ T s=1 B ⊤ i,s A ⊤ t Σ -1 t A t (y t -θ) -2(y t -θ) ⊤ A ⊤ t Σ -1 t A t U i U ⊤ i Σ -1 t A t (y t -θ) = 4 T s=1 (y s -θ) ⊤ B ⊤ i,s A ⊤ t Σ -1 t A t (y t -θ) -2(y t -θ) ⊤ A ⊤ t Σ -1 t A t U i U ⊤ i Σ -1 t A t (y t -θ) (38) D.3 MetaMap Note that (Λ + Σ t ) -1 = Σ -1 t (ΛΣ -1 t + I d ) = Σ -1t A t so we can write the matrices representing the estimator asM t = (Γ -1 + Σ -1 Λ ) -1 (Λ + Σ t ) -1 = (Γ -1 + T s=1 Σ -1 s A s ) -1 Σ -1 t A t .Thus∂ i M t = (Γ -1 + Σ -1 Λ ) -1 Σ -1 t ∂ i A t -(Γ -1 + Σ -1 Λ ) -1 T s=1 Σ -1 s ∂ i A s M t = -M t U i U ⊤ i Σ -1 t A t + T s=1 M s U i U ⊤ i Σ -1 s A s M t (39)We can then computeEQUATIONandEQUATIONThen we have∂ i Tr(A t M t ) = Tr(A t (I d + ΓΣ -1 Λ ) -1 U i U ⊤ i (Σ -1 t A t -Σ -1 Λ M t ))andEQUATIONIt is frequently the case that a specific group g may not have any examples, i.e., n g = 0, and so we cannot define Σ g,g = σ 2 /n g . At the same time, we may need to handle the dimension associated with this group, either because we still need to report a value for it or because we are in the multi-task setting and other tasks do have examples of that group that may allow us to get an estimate. To handle this issue, in all computations we only use the precision matrix Σ -1 , which can be easily defined in the case of n g = 0 using Σ -1 g,g = n g /σ 2 = 0. Note in particular that(Λ + Σ) -1 = Σ -1 (ΛΣ -1 + I d ) -1 .Since we would like to optimize over the entire domain τ 2 ∈ R 2 k ≥0 but Λ(0 2 k ) = 0 d×d is singular, we avoid inverting prior covariance matrices (i.e. Λ, Γ) in all computations. Note in particular thatA t = (Λ -1 + Σ -1 ) -1 Λ -1 = (I d + ΛΣ -1 ) -1 .While obtaining unbiased variance estimates for averages is straightforward, it is more involved for multi-point statistics such as AUC. For simplicity we just use (n g + 1)/(12n g n (0)g n (1) g ), where n (i)g is the number of members of group g with label i; this estimate is derived from the variance estimate of the related Mann-Whitney U -statistic [Siegel, 1956] . However, future work may consider improvements based on more complicated approaches [Cortes and Mohri, 2003, Wang and Guo, 2020] , or using bootstrapping techniques as is done by Herlihy et al. [2024] for structured regression.In this appendix we show that μSM and μSM t can be derived as ridge regression estimators. Similar to the notation in Appendix C, we consider k sensitive attributes (like sex, age, etc.) indexed by a ∈ [k]. The ath sensitive attribute is denoted g a and takes values in[d a ]. The joint sensitive attribute g takes values in G = [d 1 ] × [d 2 ] × • • • × [d k ] with the cardinality d = |G| = d 1 d 2 • • • d k . For A ⊆ [k],write g A for the tuple (g a ) a∈A .Each joint attribute g ∈ G identifies an intersectional group (of order k). We seek to jointly fit means of all these groups, represented as a vector µ ∈ R |G| , based on the vector of empirical means y ∈ R |G| . We posit a regression model µ = Φβ where Φ ∈ R |G|×|J | is a feature matrix and β ∈ R |J | is a vector of regression coefficients. The columns of Φ are referred to as features and indexed by j ∈ J , where J is some index set. We assume a Gaussian prior on the parameter β ∼ N (0, K) and a Gaussian distribution over observation errors, so y ∼ N (Φβ, Σ), where K and Σ are, respectively, the prior covariance matrix and error covariance matrix.The error covariance matrix Σ is assumed fixed and positive definite, the prior covariance matrix K is viewed as a hyperparameter (with a specific structure to reduce its dimension); for simplicity, we assume that K is positive definite (but that assumption is not necessary). Any generic forms of Φ, K and Σ can be considered. Here we exhibit a specific form of Φ and K under which the MAP regression estimator is the same as μSM , when provided with the same error covariance matrix Σ.We consider a structured form of matrix Φ, with features being indicators of tuples of sensitive attribute values. The features are indexed by j ∈ J , whereJ = (A, c) : A ⊆ [k], c ∈ a∈A [d a ] ,and defined asEQUATIONWe will also consider subsets of features that focus on a specific subset of sensitive attributes A ⊆ [k],J A = (A, c) : c ∈ a∈A [d a ] ,and write Φ A ∈ R |G|×|J A | for the submatrix composed of features indexed by j ∈ J A .The prior matrix K is diagonal, specified using a set of hyperparameters τ = (τ A ) A⊆[k] , with diagonal entriesEQUATIONThe MAP solution under the model described above is obtained by solvingβ = arg min β (y -Φβ) ⊤ Σ -1 (y -Φβ) + β ⊤ K -1 β ,which is a weighted ridge regression problem. Setting the gradient to zero, the solution must satisfy-2Φ ⊤ Σ -1 (y -Φβ) + 2K -1 β = 0.Hence,β = (Φ ⊤ Σ -1 Φ + K -1 ) -1 Φ ⊤ Σ -1 y.This yields a regression-based estimatorEQUATIONIn contrast, by Eqs. ( 11) and ( 7), the μSM estimator is obtained asμSM = (Λ -1 + Σ -1 ) -1 Σ -1 y,where the matrix Λ (following Eq. 22 in Appendix C.1) depends on the hyperparameter vector τ asΛ = A⊆[k] τ 2 A U A U ⊤ A ,where U A is precisely the submatrix Φ A defined above. Thus, writing ϕ A,c ∈ R |G| for the column of Φ indexed by (A, c), we obtainΛ = A⊆[k] τ 2 A Φ A Φ ⊤ A = A⊆[k] c∈ a∈A [da] τ 2 A ϕ A,c ϕ ⊤ A,c = ΦKΦ ⊤ .Hence, the μSM estimator can be rewritten asEQUATION)Theorem E.1. With Φ and K defined as above (Eqs. 44 and 45), we have μreg = μSM .In the proof we use a variant of Sherman-Morrison-Woodbury formula [Horn and Johnson, 2013, Eq. 0.7.4 .1], specialized to symmetric positive definite matrices: Proposition E.1 (Symmetric SMW Formula). Let A ∈ R n×n and R ∈ R m×m be symmetric positive definite matrices and let X ∈ R n×m . Then(A + XRX ⊤ ) -1 = A -1 -A -1 X(R -1 + X ⊤ A -1 X) -1 X ⊤ A -1 .Proof of Theorem E.1. It suffices to show thatΦ(Φ ⊤ Σ -1 Φ + K -1 ) -1 Φ ⊤ = (ΦKΦ ⊤ ) -1 + Σ -1 -1 .We do this by direct calculation, using the symmetric SMW formula:Φ(Φ ⊤ Σ -1 Φ + K -1 ) -1 Φ ⊤ = Φ K -KΦ ⊤ (Σ + ΦKΦ ⊤ ) -1 ΦK Φ ⊤ = (ΦKΦ ⊤ ) -(ΦKΦ ⊤ ) (Σ + (ΦKΦ ⊤ ) -1 (ΦKΦ ⊤ ) = (ΦKΦ ⊤ ) -1 + Σ -1 -1 .The first equality follows by the SMW formula with A = K -1 , R = Σ -1 and X = Φ ⊤ , the second by multiplying out the terms, and the third by the SMW formula with A -1 = ΦKΦ ⊤ , R -1 = Σ and X equal to the d × d identity matrix.Consider multi-task setting with tasks indexed by t = 1, . . . , T . We write T = [T ] for the set of tasks. Multi-task setting can be reduced to single-task setting by viewing the task id as an additional sensitive attribute, and performing the same analysis as for the single-task setting, but withG ′ = T × G, with the dimension d ′ = |G ′ | = T d.Formally, we seek to fit µ ′ ∈ R |G ′ | based on the vector of empirical means y ′ ∈ R |G ′ | . The entries of µ ′ and y ′ are denoted as µ ′ t,g and y ′ t,g for the task t and the intersectional group g. We denote task-specific slices of these vectors as µ ′ t = µ ′ {t}×G and y ′ t = y ′ {t}×G .Features are indexed by elements of J ′ = S × J , where S = T ∪ {glob}, so there are taskspecific and global features. The feature matrix Φ ′ ∈ R |G ′ |×|J ′ | uses the single-task feature matrix Φ ∈ R |G|×|J | (Φ ′ t,s = Φ if s = t or s = glob, 0 |G|×|J | otherwise.We posit the regression modelµ ′ = Φ ′ β ′ ,whereβ ′ ∈ R J ′ .With the definition of Φ ′ as above, this boils down toµ ′ t = Φ(β ′ t + β ′ glob ) for all t ∈ T .As before, we assume a Gaussian prior and Gaussian errors,β ′ ∼ N (0, K ′ ), y ′ ∼ N (Φ ′ β ′ , Σ ′ ).The error covarianceΣ ′ ∈ R |G ′ |×|G ′ | has a block-diagonal form with single-task error covariance matrices Σ t ∈ R |G|×|G| , for t ∈ [T ], along the diagonal.We consider a structured form of the prior covariance matrix K ′ specified by two vectors of hyperparameters:τ = (τ 2 A ) A⊆[k] and υ = (υ A ) A⊆[k] .The matrix K ′ is diagonal and positive definite, with entriesK ′ (s,A,c),(s,A,c) = τ 2 A if s ∈ T , υ A if s = glob.It can also be viewed as a block-diagonal matrix with |S| diagonal blocks of size |J | × |J | defined asK ′ s,s = K if s ∈ T , V if s = glob,where K is the single-task prior matrix defined in Eq. ( 45), and V ∈ R |J |×|J | is an analogous matrix based on the vector of hyperparameters υ rather than τ , with diagonal entriesEQUATION)The multi-task regression-based estimator is obtained by solving the resulting MAP regression problem. Similarly to the single-task case, the estimator takes formEQUATIONwith the individual task estimates denoted μ′ t reg .The multi-task SureMap estimator, introduced in §3.2.2, takes formEQUATIONwhereEQUATIONand Λ = ΦKΦ ⊤ and Γ = ΦVΦ ⊤ .Theorem E.2. With Φ ′ , K ′ and Σ ′ defined as above, we have μ′t reg = μSM t for all t ∈ [T ].In the proof we use the following identities: Proposition E.2. Let A, B ∈ R n×n be symmetric positive definite matrices and let I be the n × n identity matrix. Then(A -1 + B -1 ) -1 A -1 = I -A(A + B) -1 = B(A + B) -1 .Proof. We have(A -1 + B -1 ) -1 A -1 = A -A(A + B) -1 A A -1 = I -A(A + B) -1 = (A + B)(A + B) -1 -A(A + B) -1 = B(A + B) -1 ,where the first equality follows by the SMW formula (Proposition E.1), and the remaining equalities follow by simple algebraic manipulations.Proof of Theorem E.2. Starting with Eq. ( 49), we haveEQUATIONwhere the second equality follows by the same reasoning as in the proof of Theorem E.1, and the third equality follows by Proposition E.2.We next evaluate Φ ′ K ′ Φ ′ ⊤ . Note that the matrices Φ ′ , K ′ and Σ ′ have the following block structure:Φ ′ =   Φ Φ . . . . . . Φ Φ   , K ′ =    K . . . K V    , Σ ′ =   Σ 1 . . . Σ T   .Thus,Φ ′ K ′ Φ ′ ⊤ =   Φ . . . Φ     K . . . K     Φ ⊤ . . . Φ ⊤   +   Φ . . . Φ   V Φ ⊤ • • • Φ ⊤ =   ΦKΦ ⊤ . . . ΦKΦ ⊤   +   I . . . I   ΦVΦ ⊤ (I • • • I) =   Λ . . . Λ   +   I . . . I   Γ (I • • • I) = Λ ′ + XΓX ⊤ ,where in the last line we introduced the notation Λ ′ for the block-diagonal matrix with T copies of matrix Λ along diagonal, and notation X for the matrix obtained by stacking T copies of the |G| × |G| identity matrix I on top of each other. Plugging the last expression back into Eq. ( 52), we obtainEQUATION)where the second equality follows by the SMW formula (Proposition E.1) with A = Λ ′ + Σ ′ , R = Γ, and X = X. We next focus on simplifying the last term in the bracket in Eq. ( 53).Since Λ ′ and Σ ′ are block-diagonal, the matrix (Λ ′ + Σ ′ ) -1 is also block-diagonal with blocks along the diagonal equal to (Λ + Σ t ) -1 for t = 1, . . . , T . Thus,X ⊤ (Λ ′ + Σ ′ ) -1 X = (I • • • I)   (Λ + Σ 1 ) -1 . . . (Λ + Σ T ) -1     I . . . I   = T t=1 (Λ + Σ t ) -1 ,and similarly,X ⊤ (Λ ′ + Σ ′ ) -1 y ′ = T t=1 (Λ + Σ t ) -1 y ′ t .Therefore,Γ -1 + X ⊤ (Λ ′ + Σ ′ ) -1 X -1 X ⊤ (Λ ′ + Σ ′ ) -1 y ′ = Γ -1 + T t=1 (Λ + Σ t ) -1 -1 T t=1 (Λ + Σ t ) -1 y ′ t = θ.Plugging this back in Eq. ( 53), we obtainμ′ reg = y ′ -Σ ′ (Λ ′ + Σ ′ ) -1 y ′ -(Λ ′ + Σ ′ ) -1 X θ = y ′ -Σ ′ (Λ ′ + Σ ′ ) -1   y ′ -   I . . . I   θ  .Using, again, the fact that matrices Λ ′ and Σ ′ are block-diagonal, the task-specific blocks of μ′ reg must be equal toμ′ t reg = y ′ t -Σ t (Λ + Σ t ) -1 (y ′ t -θ) = y ′ t -Λ -1 + Σ -1 t -1 Λ -1 (y ′ t -θ) = y ′ t + Λ -1 + Σ -1 t -1 Λ -1 ( θ -y ′ t ) = μSM t ,where the second equality follows by Proposition E.2.We make use of data / models with the following sources / licenses: 1. Strack et al. [2014] : CC Attribution License. 2. Weerts et al. [2023] : MIT License. 3. Ardila et al. [2020] : CC0 License. 4. Radford et al. [2023] : Apache-2.0 License. 5. https://archive.ics.uci.edu/dataset/2/adult: CC BY 4.0 License 6. https://huggingface.co/JaaackXD/Llama-3-70B-GGUF: Meta Llama 3 License We use the third and fourth resources to create a dataset of Whisper model evaluations on Common Voice utterances, which result in the Common Voice and CVC tasks described in §4.2; we also use the last two resources to create a dataset of in-context evaluations of Llama 3 on the Adult dataset, which results in the Adult task described in §4.1. Both resources are released under a CC BY 4.0 License and are available at https://github.com/mkhodak/SureMap.By far the most computation was required to generate the Common Voice, CVC, and Adult tasks, which was done on a machine with two RTX-8000 GPUs and took about a week. As described above, the corresponding datasets are made publicly available and easy to re-use without any GPU access. Given these dataset, the main experiments were run on a 40-core machine and take a couple hours, with the vast majority of this time spent running the structured regression approach of Herlihy et al. [2024] ; see Figure 7 for a summary of the costs associated with each method evaluated in this paper. Code for both generating the task data and reproducing the method evaluations is available at https://github.com/mkhodak/SureMap.Figure 7 : Cost of running the methods evaluated in this paper as a function of the number of tasks, with both axes scaled logarithmically. While they are 1-2 orders of magnitude more expensive than the baselines-which all have closed form expressions-SureMap and multi-task SureMap are also 1-2 orders of magnitude than structured regression [Herlihy et al., 2024] . Note that for the most part the runtime of all methods will usually be dwarfed by the cost of inference.Figure 8 : Evaluations on the regression variant of Diabetes, using MAE as the target metric, disaggregating by race, sex, and age. On the left the MAE is taken across all groups, while in the center it is only over large groups and on the left over small groups. Large and small are defined as the top and bottom half of all groups, respectively.Figure 9 : Evaluations on the regression variant of Diabetes, using MSE as the target metric, disaggregating by race, sex, and age. On the left the MAE is taken across all groups, while in the center it is only over large groups and on the left over small groups. Large and small are defined as the top and bottom half of all groups, respectively.In Figures 8 & 9 , we compare evaluation methods on the regression variant of the Diabetes task, where the goal is to predict a patient's length of stay using ridge regression. In Figure 10 : Single-task evaluations on the Diabetes classification setting (disaggregating by race, sex, and age) when using AUC as the target metric. In the left column the RMSE is taken across all groups, while in the center it is only over large groups and on the right over small groups. Large and small are defined as the top and bottom half of all groups, respectively.Figure 11 : Single-task evaluation on the Common Voice ASR setting (bottom, disaggregating by sex and age) when using CER as the target metric. In the left column the MAE is taken across all groups, while in the center it is only over large groups and on the right over small groups. Large and small are defined as the top and bottom half of all groups, respectively.Figure 12 : Multi-task evaluations on state-level ACS data (disaggregating by race, sex, and age) when using AUC as the target metric. On the left is the performance across different subsampling rates while on the right we show (multiplicative) performance improvement over the naive estimator on different tasks at subsampling rate 0.1.Figure 13 : Multi-task evaluations on Common Voice clusters (disaggregating by sex and age) when using CER as the target metric. On the left is the performance across different subsampling rates while on the right we show (multiplicative) performance improvement over the naive estimator on different tasks at subsampling rate 0.1. Figure 14 : Single-task evaluations on the Diabetes classification setting (top, disaggregating by race, sex, and age), the Adult in-context classification setting (middle, disaggregating by race, sex, and age), and the Common Voice ASR setting (bottom, disaggregating by sex and age); these are the same evaluations as Figure 2 except with RMSE instead of MAE as the performance measure. In the left column the RMSE is taken across all groups, while in the center it is only over large groups and on the right over small groups. Large and small are defined as the top and bottom half of all groups, respectively. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: c.f. Section 6. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Feldman et al. [2014] show that using d in the numerator instead of Bock's Tr Σ ∥Σ∥ performs better; in §B we show that their modified form can be derived by minimizing an upper bound on the Σ -1 -weighted MSE.We typically use θ = 0, but in §B we derive a variant shrinking towards θ = μpooled (y).Note that this does not reduce the generality of our basic setup, which can be recovered with k = 1 and d1 = d.https://docs.scipy.org/doc/scipy/reference/optimize.minimize-lbfgsb.htmlThis initialization corresponds to setting the prior covariance to be the d × d identity.We use indices i instead of subsets A here for simplicity. 


Decision: This paper studies estimating model performance in multiple distribution, a problem that has good relevance to fairness. The reviewers are in agreement that the theoretical analysis is adequate, the newly introduced datasets for evaluation are useful, and the experimental results are convincing. The rebuttal further addressed the reviewers' concerns about Gaussianity assumption and computational efficiency. Yet in the discussion phase there seems to be a lack of enthusiasm in championing this paper. Due to these considerations this paper is on the borderline.

Reviews: review: paper_summary: paper_summary: This paper presents SureMap, a novel method for disaggregated evaluation, aimed at improving the estimation accuracy of performance metrics for machine learning models across different subpopulations. The proposed method is designed to address both single-task and multi-task settings, where multiple clients independently evaluate the same AI model on their respective data. SureMap leverages maximum a posteriori (MAP) estimation with a well-chosen Gaussian prior, fine-tuned using Stein’s unbiased risk estimate (SURE), to achieve high estimation accuracy. The authors evaluate SureMap across various domains, demonstrating significant improvements over existing baselines.
strength_weakness: strength_weakness: Complexity of Implementation: While the method is theoretically sound, the practical implementation of SureMap may be complex, particularly the coordinate descent algorithm used for tuning parameters. This could pose challenges for practitioners.
Dependence on Gaussian Assumptions: The method relies on Gaussian assumptions for the prior and noise distributions. In real-world scenarios, data distributions may deviate significantly from Gaussian, potentially affecting the performance of SureMap.
Scalability: The scalability of the method in very large-scale settings is not fully explored. The computational cost associated with MAP estimation and SURE tuning might be prohibitive for very large datasets or a very high number of tasks.
Empirical Evaluation: While the empirical results are promising, they are primarily based on synthetic and semi-synthetic datasets. More extensive evaluation on real-world datasets across diverse domains would strengthen the findings.
questions: questions: 1.	Implementation Details: Can the authors provide more detailed pseudocode or a step-by-step guide for implementing the coordinate descent algorithm used in SureMap? This would help in better understanding and replicating the method.
	2.	Non-Gaussian Data: How robust is SureMap to deviations from the Gaussian assumptions? Have the authors considered alternative distributions, and how would the method need to be adapted for such cases?
	3.	Scalability: What are the computational requirements of SureMap for very large datasets or a high number of tasks? Can the authors provide any benchmarks or comparisons in terms of runtime and memory usage?
	4.	Real-World Applications: Can the authors provide examples of real-world applications where SureMap has been or could be successfully applied? This would help in contextualizing the method’s practical utility.
limitations: limitations: •	Gaussian Assumption: The reliance on Gaussian assumptions is acknowledged, and the authors suggest potential future work on other distributions like Student’s t.
	•	Data Integration Costs: The potential burden or cost of integrating data from multiple clients is mentioned, and the authors propose the use of model provider data as a mitigation strategy.
	•	Scalability: Although not deeply explored, the authors note the computational challenges and suggest future work to improve scalability and efficiency.
	•	Fairness and Over-Confidence: The authors caution against over-confidence in a model’s fairness based solely on disaggregated evaluations and emphasize the need for careful application of SureMap and similar methods.
paper_summary: paper_summary: This paper introduces SureMap, a new method for disaggregated evaluation of AI systems especially for multi-task setting. The authors model the problem as Gaussian mean estimation and use a structured covariance prior that captures intersectional effects. SureMap is evaluated on several datasets, including a new multi-task ASR dataset they introduce. They show SureMap generally outperforms existing methods, especially for small subgroups and in the multi-task setting.
strength_weakness: strength_weakness: 1. The Gaussian assumption may be too strong for some real-world settings
2. No comparison to some recent methods like GP-based approaches
3. The multi-task formulation assumes clients are willing to share data statistics
questions: questions: 1. Have you considered non-Gaussian models? For example, a t-distribution might better handle heavy-tailed performance data that can occur in practice.
2. The multi-task setting assumes clients are willing to share summary statistics. How realistic is this assumption? Could you explore privacy-preserving ways to share this information?
3. How does SureMap compare to GP-based approaches like in [1]? The method aims to handle low-data regimes well, which seems relevant here.

[1] Active assessment of prediction services as accuracy surface over attribute combinations. NeurIPS 2021
limitations: limitations: N.A
paper_summary: paper_summary: The author developed a disggregated evaluation method called SureMap, which has high estimation accuracy for both multi-task and single-task disggregated evaluations. SureMap transforms the problem into structured simultaneous Gaussian mean estimatio, incorporating external data. This method further combines Maximum A Posteriori (MAP) estimation and cross-validation-free tuning via Stein's risk estimate (SURE). Significant improvements in accuracy were observed in disggregated evaluation tasks.
strength_weakness: strength_weakness: 1. I would like to know if there are any rules that need to be followed in the selection of disaggregated evaluation datasets to ensure fairness, as well as the reason why the disaggregated evaluation goal is set as the mean 0-1 error.
2. The fairness assessment is an application biased towards real-world scenarios. Does using Gaussian distribution as a prior align with real-world evaluations？
questions: questions: Disaggregated evaluation is a core task in the fairness assessment of AI systems. This paper provides a possible solution and gives the corresponding analysis.  

I would like to point out that my expertise does not directly align with the specific field of this article. Nevertheless, I have carefully read the paper several times, attempting to provide constructive feedback. I look forward to reading the insights of other reviewers, whose expertise is more closely related to the subject, to further inform my final score.
limitations: limitations: see the weaknesses and questions above
paper_summary: paper_summary: This paper studies disaggregated evaluation, which aims to estimate the performance of models on various subpopulations. This problem is challenging due to small sample sizes in subpopulations, thus leading to inaccurate performance estimates. This issue is magnified when multiple clients use the same AI model and require individualized evaluations, which is referred to as multi-task disaggregated evaluation. 

This paper designs a method that transforms the problem into a structured simultaneous Gaussian mean estimation problem. The method is comprised of two components: first, conduct maximum a posteriori (MAP) estimation for  Gaussian mean estimation and then apply cross-validation-free tuning using Stein’s unbiased risk estimate (SURE). Furthermore, the method employs an additive intersectional effect prior for capturing relationships between subpopulations with a limited number of hyperparameters.

The method, namely SureMap, is evaluated on various disaggregated evaluation tasks across different domains, including automated speech recognition (ASR) and tabular census data. The method shows high estimation accuracy for both single-task and multi-task settings over naive estimation, pooled estimation, and Bock estimation. Moreover, the method improves performance estimation even in data-poor regimes.
strength_weakness: strength_weakness: - The efficiency discussion of the method needs to be expanded. It would be better to list the complexity of the proposed method with existing methods and report the actual runtime. 
- There are certain cases that the proposed method performs worse than the pooled estimation, such as in Figure 2. It would be better to analyze the reasons behind such results.
questions: questions: - For subpopulations with few samples, how does this paper compute their ground truth and evaluate the estimation? 
- It would be better to clarify the pooled estimator. For example, what does the notation $h$​ in Equation 3? 
- It would be better to explain the working of the baselines, such as the Bock method.
limitations: limitations: This work has discussed its limitations in the setup of Gaussian distribution.
paper_summary: paper_summary: The paper proposed SureMap, a promising method for solving multi-task disaggregated evaluation problem. The key innovation of SureMap lies on transforming the problem into structured simultaneous Gaussian mean estimation and incorporating external data, e.g. from the AI system creator or from their other clients. Experiments on disaggregated evaluation tasks in multiple domains show the promising performance.
strength_weakness: strength_weakness: As analyzed in Section 2.4:
1) Gaussian assumption may result in underperformance on heavy-tailed data.
2) Incorporating data from multiple clients may be costly.
questions: questions: 1) Can the authors provide some analysis or insights when the data contradicts with Gaussian assumption?
2) Can the authors visualize/show the cost when incorporating data from clients?
limitations: limitations: I don't think there are any negative societal impacts of this work.

Comments: title: Thanks for the rebuttal
comment: Thanks for the rebuttal. I don't have further questions.