Title: Distribution-Free Statistical Dispersion Control for Societal Applications
Venue: NeurIPS 2023
Accepted: yes
Keywords: societal dispersion

distribution-free uncertainty quantification
Abstract: Explicit finite-sample statistical guarantees on model performance are an important ingredient in responsible machine learning.  Previous work has focused mainly on bounding either the expected loss of a predictor or the probability that an individual prediction will incur a loss value in a specified range.  However, for many high-stakes applications it is crucial to understand and control the \textit{dispersion} of a loss distribution, or the extent to which different members of a population experience unequal effects of algorithmic decisions. We initiate the study of distribution-free control of statistical dispersion measures with societal implications and propose a simple yet flexible framework that allows us to handle a much richer class of statistical functionals beyond previous work. Our methods are verified through experiments in toxic comment detection, medical imaging, and film recommendation.

Distribution-Free Statistical Dispersion Control for Societal Applications

Abstract

Explicit finite-sample statistical guarantees on model performance are an important ingredient in responsible machine learning. Previous work has focused mainly on bounding either the expected loss of a predictor or the probability that an individual prediction will incur a loss value in a specified range. However, for many high-stakes applications it is crucial to understand and control the dispersion of a loss distribution, or the extent to which different members of a population experience unequal effects of algorithmic decisions. We initiate the study of distribution-free control of statistical dispersion measures with societal implications and propose a simple yet flexible framework that allows us to handle a much richer class of statistical functionals beyond previous work. Our methods are verified through experiments in toxic comment detection, medical imaging, and film recommendation.

1 Introduction

Learning-based predictive algorithms are widely used in real-world systems and have significantly impacted our daily lives. However, many algorithms are deployed without sufficient testing or a thorough understanding of likely failure modes. This is especially worrisome in high-stakes application areas such as healthcare, finance, and autonomous transportation. In order to address this critical challenge and provide tools for rigorous system evaluation prior to deployment, there has been a rise in techniques offering explicit and finite-sample statistical guarantees that hold for any unknown data distribution and black-box algorithm, a paradigm known as distribution-free uncertainty quantification (DFUQ). In [1], a framework is proposed for selecting a model based on bounds on expected loss produced using validation data. Subsequent work [30] goes beyond expected loss to provide distribution-free control for a class of risk measures known as quantile-based risk measures (QBRMs) [8]. This includes (in addition to expected loss): median, value-at-risk (VaR), and conditional value-at-risk (CVaR) [23]. For example, such a framework can be used to get bounds on the 80th percentile loss or the average loss of the 10% worst cases.

While this is important progress towards the sort of robust system verification necessary to ensure the responsible use of machine learning algorithms, in some scenarios measuring the expected loss or value-at-risk is not enough. As models are increasingly deployed in areas with long-lasting societal consequences, we should also be concerned with the dispersion of error across the population, or the extent to which different members of a population experience unequal effects of decisions made based on a model's prediction. For example, a system for promoting content on a social platform may offer less appropriate recommendations for the long tail of niche users in service of a small set

of users with high and typical engagement, as shown in [19]. This may be undesirable from both a business and societal point of view, and thus it is crucial to rigorously validate such properties in an algorithm prior to deployment and understand how the outcomes disperse . To this end, we offer a novel study providing rigorous distribution-free guarantees for a broad class of functionals including key measures of statistical dispersion in society. We consider both differences in performance that arise between different demographic groups as well as disparities that can be identified even if one does not have reliable demographic data or chooses not to collect them due to privacy or security concerns. Well-studied risk measures that fit into our framework include the Gini coefficient [33] and other functions of the Lorenz curve as well as differences in group measures such as the median [5]. See Figure 4 for a further illustration of loss dispersion.

LorenzCurves

GroupDifferences

1.0

1.0

0.8

h1

hi

Group1

0.8

h2

nare

0.8

h2

SS07

S

0.6

Group2

0.6

Sha

0.6

LineofEquality

β

Median

0.4

0.4

S

S

0.4

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

β

In order to provide rigorous guarantees for socially important measures that go beyond expected loss or other QBRMs, we provide two-sided bounds for quantiles and nonlinear functionals of quantiles. Our framework is simple yet flexible and widely applicable to a rich class of nonlinear functionals of quantiles, including Gini coefficient, Atkinson index, and group-based measures of inequality, among many others. Beyond our method for controlling this richer class of functionals, we propose a novel numerical optimization method that significantly tightens the bounds when data is scarce, extending earlier techniques [21, 30]. We conduct experiments on toxic comment moderation, detecting genetic mutations in cell images, and online content recommendation, to study the impact of our approach to model selection and tailored bounds.

To summarize our contributions, we: (1) initiate the study of distribution-free control of societal dispersion measures; (2) generalize the framework of [30] to provide bounds for nonlinear functionals of quantiles; (3) develop a novel optimization method that substantially tightens the bounds when data is scarce; (4) apply our framework to high-impact NLP, medical, and recommendation applications.

2 Problem setup

We consider a black-box model that produces an output Z on every example. Our algorithm selects a predictor h , which maps an input Z ∈ Z to a prediction h ( Z ) ∈ ˆ Y . A loss function ℓ : ˆ Y × Y → R quantifies the quality of a prediction ˆ Y ∈ ˆ Y with respect to the target output y ∈ Y . Let ( Z, Y ) be drawn from an unknown joint distribution P over Z × Y . We define the random variable X h := ℓ ( h ( Z ) , Y ) as the loss induced by h on P . The cumulative distribution function (CDF) of the random variable X h is F h ( x ) := P ( X h ≤ x ) . For brevity, we sometimes use X and F when we do not need to explicitly consider h . We define the inverse of a CDF (also called inverse CDF) F as F -( p ) = inf { x : F ( x ) ≥ p } for any p ∈ R . Finally, we assume access to a set of validation samples ( Z, Y ) 1: n = { ( Z 1 , Y 1 ) , . . . , ( Z n , Y n ) } for the purpose of achieving distribution-free CDF control with mild assumptions on the loss samples X 1: n . We emphasize that the 'distribution-free'

requirement is on ( Z, Y ) 1: n instead of X 1: n , because the loss studied on the validation dataset is known to us and we can take advantage of properties of the loss such as boundedness.

3 Statistical dispersion measures for societal applications

In this section, we motivate our method by studying some widely-used measures of societal statistical dispersion. There are key gaps between the existing techniques for bounding QBRMs and those needed to bound many important measures of statistical dispersion. We first define a QBRM:

Definition 1 (Quantile-based Risk Measure) . Let ψ ( p ) be a weighting function such that ψ ( p ) ≥ 0 and ∫ 1 0 ψ ( p ) dp = 1 . The quantile-based risk measure defined by ψ is

A QBRM is a linear functional of F -, but quantifying many common group-based risk dispersion measures (e.g. Atkinson index) also involves forms like nonlinear functions of the (inverse) CDF or nonlinear functionals of the (inverse) CDF, and some (like maximum group differences) further involve nonlinear functions of functionals of the loss CDF. Thus a much richer framework for achieving bounds is needed here.

For clarity, we use J as a generic term to denote either the CDF F or its inverse F -depending on the context, and summarize the building blocks as below: (i) nonlinear functions of J , i.e. ξ ( J ) ; (ii) functionals in the form of integral of nonlinear functions of J , i.e. ∫ ψ ( p ) ξ ( J ( p )) dp for a weight function ψ ; (iii) composed functionals as nonlinear functions of functionals for the functional T ( J ) with forms in (ii) , i.e. ζ ( T ( J )) for a non-linear function ζ .

3.1 Standard measures of dispersion

We start by introducing some classic non-group-based measures of dispersion. Those measures usually quantify wealth or consumption inequality within a social group (or a population) instead of quantifying differences among groups. Note that for all of these measures we only consider non-negative losses X , and assume that ∫ 1 0 F -( p ) dp > 0 1 .

Gini family of measures. Gini coefficient [33, 34] is a canonical measure of statistical dispersion, used for quantifying the uneven distribution of resources or losses. It summarizes the Lorenz curve introduced in Figure 4. From the definition of Lorenz curve, the greater its curvature is, the greater inequality there exists; the Gini coefficient is measuring the ratio of the area that lies between the line of equality (the 45 ◦ line) and the Lorenz curve to the total area under the line of equality.

Definition 2 (Gini coefficient) . For a non-negative random variable X , the Gini coefficient is

where X ′ is an independent copy of X . G ( X ) ∈ [0 , 1] , with 0 indicating perfect equality.

Because of the existence of the denominator in the Gini coefficient calculation, unlike in QBRM we need both an upper and a lower bound for F -(see Section 4.1.1). In the appendix, we also introduce the extended Gini family.

Atkinson index. The Atkinson index [2, 19] is another renowned dispersion measure defined on the non-negative random variable X (e.g., income, loss), and improves over the Gini coefficient in that it is useful in determining which end of the distribution contributes most to the observed inequality by choosing an appropriate inequality-aversion parameter ε ≥ 0 . For instance, the Atkinson index becomes more sensitive to changes at the lower end of the income distribution as ε increases.

̸

Definition 3 (Atkinson index) . For a non-negative random variable X , for any ε ≥ 0 , the Atkinson index is defined as the following if ε = 1 :

And for ε = 1 , A (1 , X ) := lim ε → 1 A ( ε, X ) , which will converge to a form involving the geometric mean of X . A ( ε, X ) ∈ [0 , 1] , and 0 indicates perfect equality (see appendix for details).

The form of Atkinson index includes a nonlinear function of F -, i.e. ( F -) 1 -ε , but this type of nonlinearity is easy to tackle since the function is monotonic w.r.t. the range of F -(see Section 4.2.1).

Remark 1. The reason we study the CDF of X and not X 1 -ε is that it allows us to simultaneously control the Atkinson index for all ε 's.

In addition, there are many other important measures of dispersion involving more complicated types of nonlinearity such as the quantile of extreme observations and mean of range. Those measures are widely used in forecasting weather events or food supply. We discuss and formulate these dispersion measures in the appendix.

3.2 Group-based measures of dispersion

Another family of dispersion measures refer to minimizing differences in performance across possibly overlapping groups in the data defined by (protected) attributes like race and gender. Under equal opportunity [11], false positive rates are made commensurate, while equalized odds [11] aims to equalize false positive rates and false negative rates among groups. More general attempts to induce fairly-dispersed outcomes include CVaR fairness [32] and multi-calibration [14, 15]. Our framework offers the flexibility to control a wide range of measures of a group CDF F g , i.e. T ( F g ) , as well as the variation of T ( F g ) between groups. As an illustration of the importance of such bounds, [5] finds that the median white family in the United States has eight times as much wealth as the median black family; this motivates a dispersion measure based on the difference in group medians.

Absolute/quadratic difference of risks and beyond. The simplest way to measure the dispersion of a risk measure (like median) between two groups are quantities such as | T ( F g ) -T ( F g ′ ) | or [ T ( F g ) -T ( F g ′ )] 2 . Moreover, one can study ξ ( T ( F g ) -T ( F g ′ )) for some general nonlinear functions. These types of dispersion measures are widely used in algorithmic fairness [11, 20].

CVaR-fairness risk measure and its extensions. In [32], the authors further consider a distribution for each group, P g , and a distribution over group indices, P Idx . Letting CVaR α, P Z ( Z ) := E Z ∼P Z [ Z | Z > α ] for any distribution P Z , they define the following dispersion for the expected loss of group g (i.e. µ g := E X ∼P g [ X ] ) for α ∈ (0 , 1) :

A natural extension would be D CV,α ( T ( F g )) for general functional T ( F g ) , which we can write in a more explicit way [23]:

The function [ T ( F g ) -ρ ] + is a nonlinear function of T ( F g ) , but it is a monotonic function when ρ is fixed and its further composition with the expectation operation is still monotonic, which can be easily dealt with.

Uncertainty quantification of risk measures. In [4], the authors study the problem of uncertainty of risk assessment, which has important consequences for societal measures.They formulate a deviation-based approach to quantify uncertainty for risks, which includes forms like: ρ ξ ( P Idx ) := E g ∼P Idx [ ξ ( T ( F g ))] for different types of nonlinear functions ξ . Examples include variance uncertainty quantification, where E g ∼P Idx ( T ( F g ) -E g ∼P Idx T ( F g ) ) 2 ; and E ψ [ ξ ( F -( α ))] := ∫ 1 0 ξ ( F -( α )) ψ ( α ) dα to quantify how sensitive the α -VaR value is w.r.t its parameter α for some non-negative weight function ψ .

4 Distribution-free control of societal dispersion measures

In this section, we introduce a simple yet general framework to obtain rigorous upper bounds on the statistical dispersion measures discussed in the previous section. We will provide a high-level summary of our framework in this section, and leave detailed derivations and most examples to the

appendix. Our discussion will focus on quantities related to the inverse of CDFs, but similar results could be obtained for CDFs.

In short, our framework involves two steps: produce upper and lower bounds on the CDF (and thus inverse CDF) of the loss distribution, and use these to calculate bounds on a chosen target risk measure. First, we will describe our extension of the one-sided bounds in [30] to the two-sided bounds necessary to control many societal dispersion measures of interest. Then we will describe how these CDF bounds can be post-processed to provide control on risk measures defined by nonlinear functions and functionals of the CDF. Finally, we will offer a novel optimization method for tightening the bounds for a chosen, possibly complex, risk measure.

4.1 Methods to obtain confidence two-sided bounds for CDFs

For loss values { X i } n i =1 , let X (1) ≤ . . . ≤ X ( n ) denote the corresponding order statistics. For the uniform distribution over [0,1], i.e. U (0 , 1) , let U 1 , . . . , U n ∼ iid U (0 , 1) denote the corresponding order statistics U (1) ≤ . . . ≤ U ( n ) . We will also make use of the following:

Proposition 1. For the CDF F of X , if there exists two CDFs F U , F L such that F U ⪰ F ⪰ F L 2 , then we have F -L ⪰ F -⪰ F -U .

We use ( ˆ F δ n,L , ˆ F δ n,U ) to denote a (1 -δ ) -confidence bound pair ( (1 -δ ) -CBP), which satisfies P ( ˆ F δ n,U ⪰ F ⪰ ˆ F δ n,L ) ≥ 1 -δ .

We extend the techniques developed in [30], wherein one-sided (lower) confidence bounds on the uniform order statistics are used to bound F . This is done by considering a one-sided minimum goodness-of-fit (GoF) statistic of the following form: S := min 1 ≤ i ≤ n s i ( U ( i ) ) , where s 1 , . . . , s n : [0 , 1] → R are right continuous monotone nondecreasing functions. Thus, P ( ∀ i : F ( X ( i ) ) ≥ s -i ( s δ )) ≥ 1 -δ, for s δ = inf r { r : P ( S ≥ r ) ≥ 1 -δ } . Given this step function defined by s 1 , . . . , s n , it is easy to construct ˆ F δ n,L via conservative completion of the CDF. [30] found that a Berk-Jones bound could be used to choose appropriate s i 's, and is typically much tighter than using the Dvoretzky-Kiefer-Wolfowitz (DKW) inequality to construct a bound.

4.1.1 A reduction approach to constructing upper bounds of CDFs

Now we show how we can leverage this approach to produce two-sided bounds. In the following lemma we show how a CDF upper bound can be reduced to constructing lower bounds.

Furthermore, let R ( x ) be defined as 1 -L n if x < X (1) ; 1 -L n -i +1 if X ( i ) ≤ x < X ( i +1) for i ∈ { 1 , 2 , · · · , n -1 } ; 1 if X ( n ) ≤ x . Then, F ⪯ R .

Thus, we can simultaneously obtain ( ˆ F δ n,L , ˆ F δ n,U ) by setting L i = s -i ( s δ ) and applying (different) CDFconservative completions. In practice, the CDF upper bound can be produced via post-processing of the lower bound. One clear advantage of this approach is that it avoids the need to independently produce a pair of bounds where each bound must hold with probability 1 -δ/ 2 .

4.2 Controlling statistical dispersion measures

Having described how to obtain the CDF upper and lower bounds ( ˆ F δ n,L , ˆ F δ n,U ) , we next turn to using these to control various important risk measures such as the Gini coefficient and group differences. We will only provide high-level descriptions here and leave details to the appendix.

4.2.1 Control of nonlinear functions of CDFs

First we consider bounding ξ ( F -) , which maps F -to another function of R .

Control for a monotonic function. We start with the simplest case, where ξ is a monotonic function in the range of X . For example, if ξ is an increasing function, and with probability at least 1 -δ , ˆ F δ n,U ⪰ F ⪰ ˆ F δ n,L ; then further by Proposition 1, we have that ξ ( ˆ F δ, -n,L ) ⪰ ξ ( ˆ F -) ⪰ ξ ( ˆ F δ, -n,U ) holds with probability at least 1 -δ . This property could be utilized to provide bounds for the Gini coefficient or Atkinson index by controlling the numerator and denominator separately as integrals of monotonic functions of F -.

Example 1 (Gini coefficient) . If given a (1 -δ ) -CBP ( ˆ F δ n,L , ˆ F δ n,U ) and ˆ F δ n,L ⪰ 0 3 , we can provide the following bound for the Gini coefficient. Notice that

Given F -( p ) ≥ 0 (since we only consider non-negative losses, i.e. X is always non-negative), we know

with probability at least 1 -δ .

Control for absolute and polynomial functions. Many societal dispersion measures involve absolute-value functions, e.g., the Hoover index or maximum group differences. We must also control polynomial functions of inverse CDFs, such as in the CDFs of extreme observations. For any polynomial function ϕ ( s ) = ∑ k =0 α k s k , if k is odd, s k is monotonic w.r.t. s ; if k is even, s k = | s | k . Thus, we can group α k s k according to the sign of α k and whether k is even or odd, and flexibly use the upper and lower bounds already established for the absolute value function and monotonic functions to obtain an overall upper bound.

Example 2. If we have ( T δ L ( F g ) , T δ U ( F g )) such that T δ L ( F g ) ≤ T ( F g ) ≤ T δ U ( F g ) holds for all g we consider, then we can provide high probability upper bounds for

for any polynomial functions or the absolute function ξ . For example, with probability at least 1 -δ

Control for a general function. To handle general nonlinearities, we introduce the class of functions of bounded total variation. Roughly speaking, if a function is of bounded total variation on an interval, it means that its range is bounded on that interval. This is a very rich class including all continuously differentiable or Lipchitz continuous functions. The following theorem shows that such functions can always be decomposed into two monotonic functions.

Theorem 1. For ( ˆ F δ n,L , ˆ F δ n,U ) , if ξ is a function with bounded total variation on the range of X , there exists increasing functions f 1 , f 2 with explicit and calculable forms, such that with probability at least 1 -δ , ξ ( F -) ⪯ f 1 ( ˆ F δ, -n,L ) -f 2 ( ˆ F δ, -n,U ) .

As an example, recall that [4] studies forms like

to quantify how sensitive the α -VaR value is w.r.t its parameter α . For nonlinear functions beyond polynomials, consider the example where ξ = e x + e -x . This can be readily bounded since it is a mixture of monotone functions.

4.2.2 Control of nonlinear functionals of CDFs and beyond

Finally, we show how our techniques from the previous section can be applied to handle the general form ∫ 1 0 ξ ( T ( F -( α ))) ψ ( α ) dα , where ψ can be a general function (not necessarily non-negative) and ξ can be a functional of the inverse CDF. To control ξ ( T ( F -)) , we can first obtain two-sided bounds for T ( F -) if T ( F -) is in the class of QBRMs or in the form of ∫ ψ ( p ) ξ 2 ( F -( p )) dp for some nonlinear function ξ 2 (as in [4]). We can also generalize the weight functions in QBRMs from non-negative to general weight functions once we notice that ψ can be decomposed into two non-negative functions, i.e. ψ = max { ψ, 0 } -max {-ψ, 0 } . Then, we can provide upper bounds for terms like ∫ max { ψ, 0 } ξ ( F -( p )) dp by adopting an upper bound for ξ ( F -) .

4.3 Numerical optimization towards tighter bounds for statistical functionals

Having described our framework for obtaining CDF bounds and controlling rich families of risk measures, we return to the question of how to produce the CDF bounds. One drawback of directly using the bound returned by Berk-Jones is that it is not weight function aware, i.e., it does not leverage knowledge of the target risk measures. This motivates the following numerical optimization method, which shows significant improvement over previous bounds including DKW and Berk-Jones bounds (as well as the truncated version proposed in [30]).

Our key observation is that for any 0 ≤ L 1 ≤ · · · ≤ L n ≤ 1 , we have P ( ∀ i, F ( X ( i ) ) ≥ L i ) ≥ n ! ∫ 1 L n dx n ∫ x n L n -1 dx n -1 · · · ∫ x 2 L 1 dx 1 , where the right-hand side integral is a function of { L i } n i =1 and its partial derivatives can be calculated exactly by the package in [21]. Consider controlling ∫ 1 0 ψ ( p ) F -( p ) dp as an example. For any { L i } n i =1 satisfying P ( ∀ i, F ( X ( i ) ) ≥ L i ) ≥ 1 -δ, one can use conservative CDF completion to obtain ˆ F δ n,L , i.e.

where L n +1 is 1 , L 0 = 0 , and X ( n +1) = ∞ or a known upper bound for X . Then, we can formulate tightening the upper bound as an optimization problem:

such that P ( ∀ i, F ( X ( i ) ) ≥ L i ) ≥ 1 -δ, and 0 ≤ L 1 ≤ · · · ≤ L n ≤ 1 . We optimize the above problem with gradient descent and a simple post-processing procedure to make sure the obtained { ˆ L i } n i =1 strictly satisfy the above constraints. In practice, we re-parameterize { L i } n i =1 with a network ϕ θ that maps n random seeds to a function of the L i 's, and transform the optimization objective from { L i } n i =1 to θ . We find that a simple parameterized neural network model with 3 fully-connected hidden layers of dimension 64 is enough for good performance and robust to hyper-parameter settings.

Notice that there is always a feasible solution. We can use binary search to efficiently find (a good approximate of) γ ∗ .

5 Experiments

With our experiments, we aim to examine the contributions of our methodology in two areas: bound formation and responsible model selection.

5.1 Learn then calibrate for detecting toxic comments

Using the CivilComments dataset [6], we study the application of our approach to toxic comment detection under group-based fairness measures. CivilComments is a large dataset of online comments

labeled for toxicity as well as the mention of protected sensitive attributes such as gender, race, and religion. Our loss function is the Brier Score, a proper scoring rule that measures the accuracy of probabilistic predictions, and we work in the common setting where a trained model is calibrated post-hoc to produce confidence estimates that are more faithful to ground-truth label probabilities. We use a pre-trained toxicity model and apply a Platt scaling model controlled by a single parameter to optimize confidence calibration. Our approach is then used to select from a set of hypotheses, determined by varying the scaling parameter in the range [0 . 25 , 2] (where scaling parameter 1 recovers the original model). See the Appendix for more details on the experimental settings and our bound optimization technique.

5.1.1 Bounding complex expressions of group dispersion

First, we investigate the full power of our framework by applying it to a complex statistical dispersion objective. Our overall loss objective considers both expected mean across groups as well as the maximum difference between group medians, and can be expressed as: L = ❊ g [ T 1 ( F g )] + λ sup g,g ′ | T 2 ( F g ) -T 2 ( F g ′ ) | , where T 1 is expected loss and T 2 is a smoothed version of a median (centered around β = 0 . 5 with spread parameter a = 0 . 1 ). Groups are defined by intersectional attributes: g ∈ G = { black female, white female, black male, white male } . We use 100 and 200 samples from each group, and select among 50 predictors. For each group, we use our numerical optimization framework to optimize a bound on O = T 1 ( F g ) + T 2 ( F g ) using the predictor (and accompanying loss distribution) chosen under the Berk-Jones method. Results are shown in Table 1. We compare our numerically-optimized bound (NN-Opt.) to the bound given by Berk-Jones as well as an application of the DKW inequality to lower-bounding a CDF.

Our framework enables us to choose a predictor that fits our specified fairness criterion, and produces reasonably tight bounds given the small sample size and the convergence rate of 1 √ n . Moreover, there is a large gain in tightness from numerical optimization in the case where n = 100 , especially with respect to the bound on the maximum difference in median losses (0.076 vs. 0.016). These results show that a single bound can be flexibly optimized to improve on multiple objectives at once via our numerical method, a key innovation point for optimizing bounds reflecting complex societal concerns like differences in group medians [5].

5.1.2 Optimizing bounds on measures of group dispersion

Having studied the effects of applying the full framework, we further investigate whether our method for numerical optimization can be used to get tight and flexible bounds on functionals of interest. First, β -CVaR is a canonical tail measure, and we bound the loss for the worst-off 1 -β proportion of predictions (with β = 0 . 75 ). Next, we bound a specified interval of the VaR ( [0 . 5 , 0 . 9] ), which is useful when a range of quantiles of interest is known but flexibility to answer different queries within the range is important. Finally, we consider a worst-quantile weighting function ψ ( p ) = p , which penalizes higher loss values on higher quantiles, and study a smooth delta function around β = 0 . 5 , a more robust version of a median measure. We focus on producing bounds using only 100 samples from a particular intersectionally-defined protected group, in this case black females, and all measures are optimized with the same hyperparameters. The bounds produced via numerical optimization (NN-Opt.) are compared to the bounds in [30] (as DKW has been previously shown to produce weak CDF bounds), including the typical Berk-Jones bound as well as a truncated version tailored to particular quantile ranges. See Table 2 and the Appendix for results.

The numerical optimization method induces much tighter bounds than Berk-Jones on all measures, and also improves over the truncated Berk-Jones where it is applicable. Further, whereas the truncated Berk-Jones bound will give trivial control outside of [ β min , β max ] , the numerically-optimized bound not only retains a reasonable bound on the entire CDF, but even improves on Berk-Jones with respect to the bound on expected loss in all cases. For example, after adapting to CVaR, the numericallyoptimized bound gives a bound on the expected loss of 0.23, versus 0.25 for Berk-Jones and 0.50 for Truncated Berk-Jones. Thus numerical optimization produces both the best bound in the range of interest as well as across the rest of the distribution, showing the value of adapting the bound to the particular functional and loss distribution while still retaining the distribution-free guarantee.

5.2 Investigating bounds on standard measures of dispersion

Next, we aim to explore the application of our approach to responsible model selection under non-group-based fairness measures, and show how using our framework leads to a more balanced distribution of loss across the population. Further details for both experiments can be found in the Appendix.

5.2.1 Controlling balanced accuracy in detection of genetic mutation

RxRx1 [31] is a task where the input is a 3-channel image of cells obtained by fluorescent microscopy, the label indicates which of 1,139 genetic treatments the cells received, and there is a batch effect that creates a challenging distribution shift across domains. Using a model trained on the train split of the RxRx1 dataset, we evaluate our method with an out-of-distribution validation set to highlight the distribution-free nature of the bounds. We apply a threshold to model output in order to produce prediction sets, or sets of candidate labels for a particular task instance. Prediction sets are scored with a balanced accuracy metric that equally weights sensitivity and specificity, and our overall objective is: L = T 1 ( F ) + λT 2 ( F ) , where T 1 is expected loss, T 2 is Gini coefficient, and λ = 0 . 2 . We choose among 50 predictors (i.e. model plus threshold) and use 2500 population samples to produce our bounds. Results are shown in Figure 2.

e

nare

1.00

Expected

0.4

Sha

0.75

Expected+Gini

LoS

S

Expected

Expected+Gini

LO

S

S

0.50

Line of Equality

0.2

Gini

um.

0.25

0.0

0.00

10-7

10-5

10-3

10-1

0.0

0.2

0.4

0.6

0.8

1.0

Threshold

β

The plot on the left shows how the bounds on the expected loss T 1 , scaled Gini coefficient λT 2 , and total objective L vary across the different hypotheses (i.e. model and threshold combination for producing prediction sets). The bold points indicate the optimal threshold choice for each quantity. On the right is shown the Lorenz curves (a typical graphical expression of Gini) of the loss distributions induced by choosing a hypothesis based on the expected loss bound versus the bound on the total objective. Incorporating the bound on Gini coefficient in hypothesis selection leads to a more equal loss distribution. Taken together, these figures illustrate how the ability to bound a non-group based

dispersion measure like the Gini coefficient can lead to less skewed outcomes across a population, a key goal in societal applications.

5.2.2 Producing recommendation sets for the whole population

Using the MovieLens dataset [12], we test whether better control on another important non-group based dispersion measure, the Atkinson index (with ϵ = 0 . 5 ), leads to a more even distribution of loss across the population. We train a user/item embedding model, and compute a loss that balances precision and recall for each set of user recommendations. Results are shown in Figure 3. Tighter control of the Atkinson index leads to a more dispersed distribution of loss across the population, even for subgroups defined by protected attributes like age and gender that are unidentified for privacy or security reasons.

AtkinsonIndex

0.10

ho

Bound

Share

1.0

0.8

Other

MaleOver55

Share

1.0

0.8

Other

MaleOver55

0.08

0.06

h1

SS

LO

0.6

0.4

Line of equality

SSO

0.6

0.4

Line of equality

cum.

um.

0.04

0.2

C

0.2

0.02

0.0

0.0

6

4

2

Threshold

0

2

6

0.0

0.2

0.4

β

0.6

0.8

1.0

0.0

0.2

0.4

β

0.6

0.8

1.0

6 Related work

The field of distribution-free uncertainty quantification has its roots in conformal prediction [27]. The coverage guarantees of conformal prediction have recently been extended and generalized to controlling the expected loss of loss functions beyond coverage [1, 3]. The framework proposed by [30] offers the ability to select predictors beyond expected loss, to include a rich class of quantilebased risk measures (QBRMs) like CVaR and intervals of the VaR; they also introduce a method for achieving tighter bounds on certain QBRMs by focusing the statistical power of the Berk-Jones bound on a certain quantile range. Note that these measures cannot cover the range of dispersion measures studied in this work.

There is a rich literature studying both standard and group-based statistical dispersion measures, and their use in producing fairer outcomes in machine learning systems. Some work in fairness has aimed at achieving coverage guarantees across groups [24, 25], but to our knowledge there has not been prior work exploring controlling loss functions beyond coverage, such as the plethora of loss functions aimed at characterizing fairness, which can be expressed as group-based measures (cf. Section 3.2). Other recent fairness work has adapted some of the inequality measures found in economics. [32] aims to enforce that outcomes are not too different across groups defined by protected attributes, and introduces a convex notion of group CVaR, and [24] propose a DFUQ method of equalizing coverage between groups. [19] studies distributional inequality measures like Gini and Atkinson index since demographic group information is often unavailable, while [7] use the notion of Lorenz efficiency to generate rankings that increase the utility of both the worst-off users and producers.

7 Conclusion

In this work, we focus on a rich class of statistical dispersion measures, both standard group-based, and show how these measures can be controlled. In addition, we offer a novel numerical optimization method for achieving tighter bounds on these quantities. We investigate the effects of applying our framework via several experiments and show that our methods lead to more fair model selection and tighter bounds. We believe our study offers a significant step towards the sort of thorough and transparent validation that is critical for applying machine learning algorithms to applications with societal implications.

References

Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. arXiv:2110.01052 , November 2021.

Anthony B Atkinson et al. On the Measurement of Inequality. Journal of Economic Theory , 2(3):244-263, 1970.

Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution-Free, Risk-Controlling Prediction Sets. arXiv:2101.02703 , August 2021.

Mohammed Berkhouch, Ghizlane Lakhnati, and Marcelo Brutti Righi. Spectral Risk Measures and Uncertainty. arXiv preprint arXiv:1905.07716 , 2019.

Neil Bhutta, Andrew Chang, Lisa Dettling, and Joanne Hsu. Disparities in Wealth by Race and Ethnicity in the 2019 Survey of Consumer Finances. FEDS Notes , 2020, 09 2020.

Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference , pages 491-500, 2019.

Virginie Do, Sam Corbett-Davies, Jamal Atif, and Nicolas Usunier. Two-Sided Fairness in Rankings via Lorenz Dominance. Advances in Neural Information Processing Systems , 34:8596-8608, 2021.

Kevin Dowd and David Blake. After VaR: The Theory, Estimation, and Insurance Applications of Quantile-Based Risk Measures. Journal of Risk & Insurance , 73(2):193-229, June 2006.

Chengyue Gong and Xingchao Liu. Bi-Objective Trade-off with Dynamic Barrier Gradient Descent. Advances in Neural Information Processing Systems , 2021.

Laura Hanu and Unitary team. Detoxify. Github. https://github.com/unitaryai/detoxify, 2020.

Moritz Hardt, Eric Price, and Nati Srebro. Equality of Opportunity in Supervised Learning. Advances in Neural Information Processing Systems , 29, 2016.

Maxwell F Harper and Joseph A Konstan. The Movielens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems , 5(4):1-19, 2015.

HO Hartley and HA David. Universal Bounds for Mean Range and Extreme Observation. The Annals of Mathematical Statistics , pages 85-99, 1954.

Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the (Computationally-Identifiable) Masses. In International Conference on Machine Learning , pages 1939-1948. PMLR, 2018.

Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 2564-2572. PMLR, 2018.

Bruce P Kennedy, Ichiro Kawachi, and Deborah Prothrow-Stith. Income Distribution and Mortality: Cross-Sectional Ecological Study of the Robin Hood Index in the United States. British Medical Journal , 312(7037):1004-1007, 1996.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A Benchmark of in-the-Wild Distribution Shifts, 2021.

Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified Uncertainty Calibration. Advances in Neural Information Processing Systems , 32, 2019.

Tomo Lazovich, Luca Belli, Aaron Gonzales, Amanda Bower, Uthaipon Tantipongpipat, Kristian Lum, Ferenc Huszar, and Rumman Chowdhury. Measuring Disparate Outcomes of Content Recommendation Algorithms with Distributional Inequality Metrics. Patterns , 3(8):100568, 2022.

Jean-Yves Le Boudec. Rate Adaptation, Congestion Control and Fairness: A Tutorial. Web Page, November , 4, 2005.

Amit Moscovich, Boaz Nadler, and Clifford Spiegelman. On the Exact Berk-Jones Statistics and Their p -Value Calculation. Electronic Journal of Statistics , 10(2):2329-2354, 2016.

John Platt et al. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances in Large Margin Classifiers , 10(3):61-74, 1999.

R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of Conditional Value-at-Risk. Journal of Risk , 2:21-42, 2000.

Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel Candès. With Malice Toward None: Assessing Uncertainty via Equalized Coverag. Harvard Data Science Review , 2(2):4, 2020.

Yaniv Romano, Stephen Bates, and Emmanuel J. Candès. Achieving Equalized Odds by Resampling Sensitive Attributes. arXiv:2006.04292 , June 2020.

Halsey Lawrence Royden and Patrick Fitzpatrick. Real Analysis , volume 32. Macmillan New York, 1988.

Glenn Shafer and Vladimir Vovk. A Tutorial on Conformal Prediction. Journal of Machine Learning Research , 9(3), 2008.

Galen R. Shorack and Jon A. Wellner. Empirical Processes with Applications to Statistics . Society for Industrial and Applied Mathematics, January 2009.

Anthony F Shorrocks. The Class of Additively Decomposable Inequality Measures. Econometrica: Journal of the Econometric Society , pages 613-625, 1980.

Jake C Snell, Thomas P Zollo, Zhun Deng, Toniann Pitassi, and Richard Zemel. Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictions. International Conference on Learning Representations (ICLR) , 2023.

James Taylor, Berton Earnshaw, Ben Mabey, Mason Victors, and Jason Yosinski. Rxrx1: An Image Set for Cellular Morphological Variation Across Many Experimental Batches. In International Conference on Learning Representations (ICLR) , volume 22, page 23, 2019.

Robert Williamson and Aditya Menon. Fairness Risk Measures. In International conference on machine learning , pages 6786-6797. PMLR, 2019.

Shlomo Yitzhaki. Relative Deprivation and the Gini Coefficient. The Quarterly Journal of Economics , 93(2):321-324, 1979.

Shlomo Yitzhaki and Edna Schechtman. The Gini Methodology: a Primer on a Statistical Methodology . Springer, 2013.

Appendix

Broader impact. The broader impact of the proposed framework is significant, as it extends the ability to gain trust in machine learning systems. However there are important concerns and limitations.

Focus on performance metrics In this paper we propose a range of performance metrics, which extend well beyond standard metrics concerning expected loss. However, in many situations these metrics are not sufficient to capture the effects of the machine learning system. Often a number of different metrics are required to provide a clearer picture of model performance, while some effects are difficult to capture in any metric. Also, while the measures studied offer the ability to more evenly distribute a quantity across a population, they do not offer guarantees to individuals. Finally, achieving a more equal distribution of the relevant quantity (e.g., loss or income) may have negative impacts on some segments of the population.

Limitations These are summarized in the Conclusion but are expanded upon here. An important assumption in this work, and in distribution-free uncertainty quantification more generally, is that the examples seen in deployment are drawn from the same distribution as those in the validation set that are used to construct the bounds. Although this is an active area of research, here we make this assumption, and the quality of the bounds produced may degrade if the assumption is violated. A second limitation is that the scope of hypotheses and predictors we can select from is limited, due to theoretical constraints: a correction must be performed based on the size of the hypothesis set. Finally, the generated bounds may not be tight, depending on the amount of available validation data and unavoidable limits of the techniques used to produce the bounds. We did some comparisons to Empirical values of the measures we obtained bounds for in the experiments; more extensive studies would be useful to elucidate the value of the bounds in practice.

Organization of the Appendix. (1) In Appendix A, we provide detailed statements and derivations of our methodology presented Section 4.2.1, including how to obtain bounds for those measures mentioned in Section 3; (2) in Appendix B, we introduce further societal dispersion measures, beyond those presented in Section 3 and corresponding bounds; (3) in Appendix C, we investigate the extension of our results to multi-dimensional settings; (4) lastly, in Appendix D and E, we provide more complete details and results from our experiments (Section 5).

A Derivations and proofs for bounding methods

Section A.1, we first consider how to control, or provide upper bounds on, various quantities when we are given ( ˆ F δ n,L , ˆ F δ n,U ) , which are constructed by { X i } n i =1 , such that

where the randomness is taken over { X i } n i =1 .

Then, in Section A.2, we will show how we obtain ( ˆ F δ, -n,L , ˆ F δ, -n,U ) by extending the arguments in [30]. In addition, we show details in Section A.2.2 on how we go beyond the methods in [30] and provide a numerical optimization method for tighter bounds.

Proof of Proposition 1. We briefly describe the the proof for Proposition 1. The proof is mainly based on [30], but we include it here for completeness. Notice for any non-decreasing function G : R → R (not just a CDF), there exists the (general) inverse of G as G -( p ) = inf { x : G ( x ) ≥ p } for any p ∈ R .

Proposition 2 (Restatement of Proposition 1) . For the CDF F of X , if there exists two increasing functions F U , F L such that F U ⪰ F ⪰ F L , then we have F -L ⪰ F -⪰ F -U .

Proof. For any two non-decreasing function G ( p ) and C ( p ) , by the definition of the general inverse function, G ( G -( p )) ≥ p . If C ⪰ G , we therefore have C ( G -( p )) ≥ G ( G -( p )) ≥ p . Applying C -to both sides yields C -( C ( G -( p ))) ≥ C -( p ) . But x ≥ C -◦ C ( x ) (see e.g. Proposition 3 on p. 6

of [28]) and thus G -( p ) ≥ C -( p ) . Plugging in F and F U as G and C , this can yield F -⪰ F -U . The other direcion is similar.

A.1 Control of nonlinear functions of CDFs (Section 4.2.1)

A.1.1 Control for monotonic functions

Recall that we start with the simplest case where ξ is a monotonic function in the range of X . It is straightforward to have the following claim.

Claim 1. If we have ˆ F δ, -n,L ⪯ F ⪯ ˆ F δ, -n,U with probability at least 1 -δ for some δ ∈ (0 , 1) , if ξ is an increasing function, then

with probability at least 1 -δ . Similarly, if ξ is a decreasing function, then ξ ( ˆ F δ, -n,L ) ⪯ ξ ( ˆ F -) ⪯ ξ ( ˆ F δ, -n,U ) with probability at least 1 -δ .

We show how this could be applied to provide bounds for Gini coefficient and Atkinson index by controlling the numerator and denominator separately as integrals of monotonic functions of F -.

Example 3 (Gini coefficient) . If given a (1 -δ ) -CBP ( ˆ F δ n,L , ˆ F δ n,U ) and ˆ F δ n,L ⪰ 0 4 , we can provide the following bound for the Gini coefficient. Notice that

Given F -( p ) ≥ 0 (since we only consider non-negative losses, i.e. X is always non-negative), we know

with probability at least 1 -δ .

Example 4 (Atkinson index) . First, we present the complete version of Atkinson index. Namely,

̸

Notice that for ε ≥ 0 , ( · ) 1 -ε and ln( · ) are increasing functions, thus, for Atkinson index and a (1 -δ ) -

̸

1 ; 1 -exp( ∫ 1 0 ln( ˆ F δ, -n,U ( p )) dp ) ∫ 1 0 ˆ F δ, -n,L ( p ) dp , if ε = 1 . Then, with probability at least 1 -δ , A δ U ( ε, X ) is an upper bound for A ( ε, X ) for all ε ∈ [0 , 1) .

As mentioned in Remark 1, instead of calculating bounds separately for each ε , simple post-processing enables us to efficiently issue a family of bounds.

Example 5 (CVaR fairness-risk measures and beyond) . Recall that for α ∈ (0 , 1) ,

The function [ T ( F g ) -ρ ] + is an increasing function when ρ is fixed and its further composition with the expectation operation is still increasing. If we have ( T δ L ( F g ) , T δ U ( F g )) such that T δ L ( F g ) ≤

T ( F g ) ≤ T δ U ( F g ) 5 for all g with probability at least 1 -δ , then we have

and the first term of RHS can be minimized easily since it is a convex function of ρ .

A.1.2 Control for absolute and polynomial functions

Recall that if s L ≤ s ≤ s U , then

More generally, for any polynomial function ϕ ( s ) = ∑ k =0 α k s k . Notice if k is odd, s k is monotonic w.r.t. s and we can bound

So, for ϕ ( F -) , we can plug in ˆ F δ, -n,L and ˆ F δ, -n,U to replace s U and s L to obtain an upper bound with probability at least (1 -δ ) . The derivation for the lower bound is similar. We summarize our results as the following proposition.

Proposition 3. If given a (1 -δ ) -CBP, then with probability at least 1 -δ , ( ˆ F δ n,L , ˆ F δ n,U ) ,

Moreover, for any polynomial function ϕ ( s ) = ∑ k =0 α k s k , we have

Example 6. If we have ( T δ L ( F g ) , T δ U ( F g )) such that T δ L ( F g ) ≤ T ( F g ) ≤ T δ U ( F g ) holds for all g we consider, then we can provide high probability upper bounds for

for any polynomial functions or the absolute function ξ . For example, with probability at least 1 -δ ,

We will further show in Appendix B how our results are applied to specific examples.

A.1.3 Control for a general function

To handle general non-linearity, we need to introduce the class of functions of bounded variation on a certain interval, which is a very rich class that includes all the functions that are continuously differentiable or Lipchitz continuous on that interval.

Definition 4 (Functions of bounded total variation [26]) . Define the set of paritions on [ a, b ] as

Then, the total variation of a continuous real-valued function ξ , defined on [ a, b ] ⊂ R is defined as

where Π is the set of all partitions, and we say a function ξ is of bounded variation, i.e. ξ ∈ BV ([ a, b ]) iff V b a ( ξ ) < ∞ .

Recall that X ≥ 0 in our cases, then, for ξ ( F -) , we can have the following bound.

Theorem 2 (A restatement & formal version of Theorem 1) . For a (1 -δ ) -CBP ( ˆ F δ n,L , ˆ F δ n,U ) , for any p ∈ [0 , 1] such that the total variation of ξ is finite on [0 , ˆ F δ, -n,L ( p )] , then

Moreover, if ξ is continuously differentiable on [0 , ˆ F δ, -n,L ( p )] , we can express V s 0 ( ξ ) as ∫ x 0 | dξ ds ( s ) | ds for any x ∈ [0 , ˆ F δ, -n,L ( p )] .

Proof. By the property of functions of bounded total variation [26], if ξ is of bounded total variation on [0 , ˆ F δ, -n,L ( p )] , then, we have that: for any x ∈ [0 , ˆ F δ, -n,L ( p )]

where both f 1 ( x ) := V x 0 ( ξ ) and f 2 ( x ) := V x 0 ( ξ ) -ξ ( x ) are increasing functions. Moreover,

if ξ is continuously differentiable.

Thus, by taking advantage of the monotonicity, we have

So, if ξ is of bounded variation on the range of X , then

A.2 Methods to obtain confidence two-sided bounds for CDFs (Section 4.1)

We provide details for two-sided bounds and our numerical methods in the following.

A.2.1 The reduction approach to constructing upper bounds of CDFs (Section 4.1.1)

We here provide the proof of Lemma 1.

Lemma 2 (A restatement & formal version of Lemma 1) . For 0 ≤ L 1 ≤ L 2 · · · ≤ L n ≤ 1 , since P ( ∀ i : F ( X ( i ) ) ≥ L i ) ≥ P ( ∀ i : U ( i ) ≥ L i ) by [30], if we further have P ( ∀ i : U ( i ) ≥ L i ) ≥ 1 -δ, then we have

Furthermore, let R ( x ) be defined as

Then, F ⪯ R .

Proof. Notice that for given order statistics { X ( i ) } n i =1 , let P { X ( i ) } n i =1 denote the probability taken over the randomness of { X ( i ) } n i =1 , and P X denote the probability taken over the randomness of X , which is an independent random variable drawn from F . Let us denote B = -X , and B ( i ) as the i -th order statistic for samples {-X i } n i =1 . It is easy to see that B ( n -i +1) = -X ( i ) . We also denote P B as the probability taken over the randomness of B , and F B as the CDF of B .

where we use the fact that F -B ( U ( n -i +1) ) is of the same distribution as B ( n -i +1) and the last inequality follows from Proposition 1, eq. 24 on p.5 of [28].

Notice that P ( ∀ i : U ( n -i +1) > L n -i +1 ) = P ( ∀ i : U ( n -i +1) ≥ L n -i +1 ) , and according to [30] and our assumption, P ( ∀ i : F ( X ( i ) ) ≥ L i ) ≥ P ( ∀ i : U ( i ) ≥ L i ) ≥ 1 -δ .

The conservative construction of R satisfies R ⪰ F straightforwardly if ∀ i : lim ϵ → 0 + F ( X ( i ) -ϵ ) ≤ 1 -L n -i +1 ) holds. Thus, we know R ⪰ F with probability at least 1 -δ . Our proof is complete.

UniformDistribution

Empirical Distribution

1.0

0.8

0.6

β

0.4

0.2

0.0

(ZT)x(TT)x(0t)x(6)x(8)x（L)x(9)x(S)x(b)x(E)x(z)x(T)x

0

2

3

LoSS

A.2.2 Details of numerical optimization method (Section 4.3)

Now, we introduce the details of our numerical optimization method. Recall that one drawback of the QBRMbounding approach is that it is not weight function aware: when controlling ∫ 1 0 ψ ( p ) F -( p ) dp for a non-negative weight function ψ , the procedure ignores the structure of ψ , as it first obtains ˆ F δ n,L , then provides an upper bound ∫ 1 0 ψ ( p ) ˆ F δ, -n,L ( p ) dp .

Our numerical approach can overcome that drawback and can also easily be applied to handle mixtures of multiple functionals. The bounds obtained by our method are significantly tighter than those provided by methods in [30] in the regime of small data size. Notice that the small data size regime is the one people care about because when the data size is large, all the bounds we discussed will converge to the same value, and the gap between different bounds will shrink to 0 as the data size grows.

First, by [ ? ] and Proposition 1, eq. 24 on p.5 of [28], we have for any 0 ≤ L 1 ≤ · · · ≤ L n ≤ 1 ,

where the right-hand side integral is a function of { L i } n i =1 and its partial derivatives can be exactly calculated by the package in [21]. Specifically, the package in [21] enables us to calculate

for any positive integer n . Notice that the partial derivative of υ ( L 1 , L 2 , · · · , L n , 1) with respect to L i is:

which we can also use the package in [21] to calculate the partial derivatives.

Consider providing upper or lower bounds for ∫ 1 0 ψ ( p ) F -( p ) dp for non-negative weight function ψ as an example. For any { L i } n i =1 satisfying P ( ∀ i, F ( X ( i ) ) ≥ L i ) ≥ 1 -δ , one can use conservative CDF completion in [30] to obtain ˆ F δ n,L , i.e. ∫ 1 0 ψ ( p ) ξ ( ˆ F δ, -n,L ( p )) dp = ∑ n +1 i =1 ξ ( X ( i ) ) ∫ L i L i -1 ψ ( p ) dp , where L n +1 is 1 , L 0 = 0 , and X ( n +1) = ∞ or a known upper bound for X . Then, we can formulate tightening the upper bound as an optimization problem:

such that

Similarly, for the lower bound, we can use the CDF completion mentioned in Theorem 1, and construct ˆ F δ n,U , then, we can study the following lower bound for ∫ 1 0 ψ ( p ) F -( p ) dp ,

where X (0) = 0 .

Parameterized model approach. Notice the above optimization problem formulation has a drawback: if more samples are drawn, i.e. n increases, then the number of parameters we need to optimize also increases. In practice, we re-parameterize { L i } n i =1 as the following:

where g i are random Gaussian seeds. This is of the same spirit as using random seeds in generative models. We find that a simple parameterized neural network model with 3 fully-connected hidden layers of dimension 64 is enough for good performance and robust to hyper-parameter settings. Take the upper bound optimization problem as an example; using the new parameterized model, we have

such that

where L 0 = 0 , L n +1 = 1 , X ( n +1) = ∞ or a known upper bound for X . We can solve the above optimization problem using heuristic methods such as [9].

Post-processing for a rigorous guarantee for constraints. Notice that we may not ensure the constraint n ! ∫ 1 L n ( θ ) dx n ∫ x n L n -1 ( θ ) dx n -1 · · · ∫ x 2 L 1 ( θ ) dx 1 ≥ 1 -δ is satisfied in the above optimization because we may use surrogates like Langrange forms in our optimization processes. To make sure the constraint is strictly satisfied, we can do the following post-processing: let us denote the obtained L i 's by optimizing (1) as L i ( ˆ θ ) . Then, we look for γ ∗ ∈ [0 , L n ( ˆ θ )] such that

Notice there is always a feasible solution as when γ = L n ( ˆ θ ) ,

and υ ( L 1 ( ˆ θ ) -γ, · · · , L n ( ˆ θ ) -γ, 1) is a decreasing function of γ . We can use binary search to efficiently find (a good approximate of) γ ∗ .

B Other dispersion measures and calculation

B.1 Lorenz curve & the extended Gini family

Lorenz curve. In the main context, Lorenz curve has been mentioned in reference to Gini coefficient and Atkinson index. To be more complete, we further demonstrate the definition of Lorenz curve in its mathematical form.

Definition 5 (Lorenz curve) . The definition of Lorenz curve is a function: for t ∈ [0 , 1] ,

We can obtain a lower bound and an upper bound function for the Lorenz curve. Given a (1 -δ ) -CBP ( ˆ F δ n,L , ˆ F δ n,U ) and ˆ F δ n,L ⪰ 0 , we can construct a lower bound function L δ L ( t ) :

and an upper bound can be obtained by

With probability at least 1 -δ , the true Lorenz curve sits between the upper bound function and the lower bound function for all t ∈ [0 , 1] .

The extended Gini family. The Gini coefficient can further give rise to the extended Gini family, which is a family of variability and inequality measures that depends on one parameter - the extended Gini parameter. The definition is as follows.

Definition 6 (The extended Gini family[34]) . The extended Gini coefficient is given by

where ν > 0 is the extended Gini parameter and Cov ( · , · ) is the covariance.

For the extended Gini coefficient, choosing different ν 's corresponds to different weighting schemes applied to the vertical distance between the egalitarian line and the Lorenz curve; and if ν = 2 , it is the standard Gini coefficient.

Given a (1 -δ ) -CBP ( ˆ F δ n,L , ˆ F δ n,U ) and ˆ F δ n,L ⪰ 0 , we can construct upper bound for G . Let

then G δ U ( ν, X ) ⪰ G ( ν, X ) with probability at least 1 -δ .

B.2 Generalized entropy index

The generalized entropy index [29] is another measure of inequality in a population. Specifically, the definition is: for real number α

̸

It is not hard to further expand the expressions and write the generalized entropy index as:

̸

̸

Notice that ( · ) α is a monotonic function for the case α = 0 , 1 , and ln( · ) is also a monotonic function, so the bound can be obtained similarly as in the case of Atkinson index. For instance, for α > 1 , given a (1 -δ ) -CBP ( ˆ F δ n,L , ˆ F δ n,U ) ,

Other cases can be tackled in a similar way, which we will not reiterate here.

B.3 Hoover index

The Hoover index [16] is equal to the percentage of the total population's income that would have to be redistributed to make all the incomes equal.

Definition 7 (Hoover index) . For a non-negative random variable X , the Hoover index is defined as

Hoover index involves forms like | F --µ | for µ = ∫ 1 0 F -( p ) dp . This type of nonlinear structure can be dealt with the absolute function results mentioned in Appendix A.1.2.

For Hoover index and a (1 -δ ) -CBP ( ˆ F δ n,L , ˆ F δ n,U ) , let us define

Then, with probability at least 1 -δ , H U ( , X ) is an upper bound for H ( X ) .

B.4 Extreme observations & mean range

For example, a city may need to estimate the cost of damage to public amenities due to rain in a certain month. The loss for each day of a month is X 1 , · · · , X k i.i.d drawn from F , and the administration hopes to estimate and control the dispersion of the losses in a month so that they can accurately allocate resources. This involves quantities such as range ( max i ∈ [ k ] X i -min j ∈ [ k ] X j ) or quantiles of extreme observations ( max i ∈ [ k ] X i ). The CDF of extreme observations such as max i ∈ [ k ] X i involves a nonlinear function of F , i.e. ( F ( x )) k .

Example 7 (Quantiles of extreme observations) . The CDF of max i ∈ [ k ] X i is F k . Thus, by the result of Appendix A.1.2, if given a (1 -δ ) -CBP ( ˆ F δ n,L , ˆ F δ n,U ) and 1 ⪰ ˆ F δ, -n,U ⪰ ˆ F δ, -n,L ⪰ 0 , with probability at least 1 -δ ,

We also have

Similarly, for min i ∈ [ k ] X i , the CDF is 1 -(1 -F ) k , thus, we have

We also want to emphasize, even if, X is not necessarily non-negative, we can apply the polynomial method in Appendix A.1.2 for ˆ F δ, -n,U and ˆ F δ, -n,L .

Example 8 (Mean range) . By [13], if we further have prior knowledge that X is of continuous distribution, the mean of max i ∈ [ k ] X i -min j ∈ [ k ] X j can be expressed as:

Notice that both F and F -are increasing. Thus, if given a (1 -δ ) -CBP ( ˆ F δ n,L , ˆ F δ n,U ) , ˆ F δ n,L ⪰ 0 , then with probability at least 1 -δ ,

is an upper bound of the mean range.

There are many other interesting societal dispersion measures that could be handled by our framework, such as those in [19]. For example, they study tail share that captures 'the top 1% of people own X share of wealth", which could be easily handled with the tools provided here. We will leave those those examples to readers.

C Extension to multi-dimensional cases and applications

We briefly discuss extending our approach to multi-dimensional losses. Unfortunately, there is not a gold-standard definition of quantiles in the multi-dimensional case, and thus we only discuss functionals of CDFs and provide an example. For multi-dimensional samples { X i } n i =1 , each of k dimensions, i.e. X i = ( X i 1 , · · · , X i k ) , for any k -dimensional vector x = ( x 1 , · · · , x k ) , define empirical CDF

where we abuse the notation ⪯ to mean all of X i 's coordinates are smaller than x 's.

By classic DKW inequality, we have with probability at least 1 -δ ,

Meanwhile, we can further adopt Frechet-Hoeffeding bound, which gives,

where F i is the CDF of the i-th coordinate. Then, we can construct ( ˆ F δ/k,i n,L , ˆ F δ/k,i n,U ) such that ( ˆ F δ/k,i n,L ⪯ F i ⪯ ˆ F δ/k,i n,U ) , with probability at last 1 -δ/k . Thus, by union bound,

for all x with probability at last 1 -δ .

We have

with probability at last 1 -2 δ .

Example 9 (Gini correlation coefficient [34]) . The Gini correlation coefficient for two non-negative random variable X and Y are defined as

where F X , F Y are marginal CDFs of X,Y and F X,Y is the joint CDF. One can use the multidimensional CDF bounds and our previous methods to provide bounds for the Gini correlation coeffiecient.

D Experiment details

This section contains additional details for the experiments in Section 5. We set δ = 0 . 05 (before statistical corrections for multiple tests) in all experiments unless otherwise explicitly stated. Whenever we are bounding measures on multiple hypotheses, we perform a correction for the size of the hypothesis set. Additionally, when we bound measures on multiple distributions (e.g. demographic groups), we also perform a correction. Our code will be released publicly upon the publication of this article.

D.1 CivilComments (Section 5.1)

Our set of hypotheses are a toxicity model combined with a Platt scaler [22], where the model is fixed and we vary the scaling parameter in the range [0 . 25 , 2] while fixing the bias term to 0 . We use a pre-trained toxicity model from the popular python library Detoxify 6 [10] and perform Platt Scaling using code from the python library released by [18] 7 . A Platt calibrator produces output according to:

where w,b are learnable parameters and v is the log odds of the prediction. Thus we form our hypothesis set by varying the parameter w while fixing b to 0 . Examples are drawn from the train split of CivilComments, which totals 269,038 data points.

The loss metric for our CivilComments experiments is the Brier Score. For n data points, Brier score is calculated as:

where f i is prediction confidence and o i is the outcome (0 or 1).

D.1.1 Bounding complex objectives (Section 5.1.1)

We randomly sample 100,000 test points for calculating the empirical values in Table 1, and draw our validation points from the remaining data. We perform a Bonferroni correction on δ = 0 . 05 for the size of the set of hypotheses as well as the number of distributions on which we bound our measures (in this case the number of groups, 4). We set λ = 1 . 0 .

Numerical optimization details (including training strategy and hyperparameters) are the same as Section 5.1.2, explained below in Appendix D.1.2. For each group g we optimize the objective

where F g is the CDF bound for group, T 1 is expected loss, and T 2 is a smoothed version of a median with a = 0 . 01 (see Appendix D.1.2 and Figure 5).

For comparison, the DKW inequality is applied to get a CDF lower bound, which is then transformed to an upper bound via the reduction approach in Section 4.1.1. To get the lower bound b l 1: n , we set:

D.1.2 Numerical optimization examples (Section 5.1.2)

We parameterize the bounds with a fully connected network with 3 hidden layers of dimension 64. The n gaussian seeds are of size 32, which is also the input dimension for the network. Training is performed in two stages, where the network is first trained to approximate a Berk-Jones bound, and then optimized for some specified objective O . In both stages of training we aim to push the training error to zero or as close as possible (i.e. 'overfit'), since we are optimizing a bound and do not seek generalization. The model is first trained for 100,000 epochs to output the Berk-Jones bound using a mean-squared error loss. Then optimization on O is performed for a maximum of 10,000 epochs, and validation is performed every 25 epochs, where we choose the best model according to the bound on O . Both stages of optimization use the Adam optimizer with a learning rate 0 . 00005 , and for the second stage the constraint weight is set to λ = 0 . 00005 . We perform post-processing to ensure the constraint holds (see Section A.2.2). For some denominator m (in our case m = 10 6 ) we set γ = 1 m , 2 m , 3 m , ... and check the constraint until it is satisfied.

This approach is applied to both the experiments in Section 5.1.1 and Section 5.1.2. Details on the objective for Section 5.1.1 are above in Appendix D.1.1. In Section 5.1.2, we set δ = 0 . 01 and our metrics for optimization are described below:

CVaR CVaR is a measure of the expected loss for the items at or above some quantile level β . We set β = 0 . 75 , and thus we bound the expected loss for the worst-off 25% of the population.

VaR-Interval In the event that different stakeholders are interested in the VaR for different quantile levels β , we may want to select a bound based on some interval of the VaR [ β min , β max ] . We perform our experiment with β min = 0 . 5 , β max = 0 . 9 , which includes the median ( β = 0 . 5 ) through the worst-case loss exluding a small batch of outliers ( β = 0 . 9 ).

Quantile-Weighted We apply a weighting function to the quantile loss ψ ( p ) = p , such that the loss incurred by the worst-off members of a population are weighted more heavily.

Smoothed Median We study a more robust version of a median:

with β = 0 . 5 and a = 0 . 01 , similar to a normal distribution extremely concentrated around its mean. See Figure 5 for an illustration of such a weighting.

D.2 Bounds on standard measures (Section 5.2)

This section contains additional details for the experiments in Section 5.2.

Smoothed Median

(d)

0.498

0.499

0.500

0.501

0.502

p

D.2.1 RxRx1 (Section 5.2.1)

We use the code released by [17] 8 to pre-train a model on the train split of RxRx1 [31] and we evaluate our algorithm on the OOD val split with 9854 total samples. We randomly sample 2500 items for use in validation (bounding and model selection), and use the remainder of the data points for illustrating the empirical distribution induced by the different hypotheses. The thresholds which are combined with the pre-trained model to form our hypothesis set are evenly spaced in [ -8 , 0] under the log transformation with base 10, thus leaving the thresholds in the range [10 -8 , 1] .

Balanced accuracy is calculated as:

where Y is the set of ground truth labels (which in this experiment will always be one label), ˆ Y is a set of predictions, and k is the number of classes.

D.2.2 MovieLens-1M (Section 5.2.2)

MovieLens-1M [12] is a publicly available dataset. We filter all ratings below 5 stars, a typical pre-processing step, and filter any users with less than 15 5-star ratings, leaving us with 4050 users. For each user, the 5 most recently watched items are added to the test set, while the remaining (earlier) items are added to the train set. We train a user/item embedding model using the popular python recommender library LightFM 9 with a WARP ranking loss for 30 epochs and an embedding dimension of 16.

For recommendation set ˆ I we compute a loss combining recall and precision against a user test set I of size k :

where α = 0 . 5 . We randomly sample 1500 users for validation, and use the remaining users to plot the empirical distributions. The 100 hypotheses tested are evenly spaced between the minimum and maximum scores of any user/item pair in the score matrix.

E Additional results for numerical optimization (Section 5.1.2)

Figure 6 compares the learned bounds G opt to the Berk-Jones ( G BJ ) and Truncated Berk-Jones ( G BJ -t ) bounds, as well as the empirical CDF of the real loss distribution.

CVaR

VaRInterval

1.0

Femp

1.0

Femp

Gopt

Gopt

0.8

0.8

GBJ

GBJ

0.6

GBJ-t

0.6

GBJ-t

S

S

5

S

5

S

0.4

0.4

0.2

0.2

0.0

0.0

0.5

0.6

0.7

0.8

0.9

1.0

0.5

0.6

0.7

0.8

0.9

1.0

β

β

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study extends the literature on distribution-free uncertainty quantification by providing bounds for various statistical dispersion measures, particularly those commonly employed in fairness evaluations of algorithms. The authors enhance the theoretical foundation by introducing additional results that allow bounding any functions with bounded total variation on the cumulative distribution function (CDF). Additionally, they propose a numerical optimization technique that achieve tighter bounds than previous methods. Empirical experiments validate the effectiveness of these tighter bounds and their relevance to ensuring fairness in model selection.\n\nSTRENGTHS:\nThe paper addresses an existing gap in the literature on bounding other functions of the observed loss of an algorithm beyond means and QRBMs [1]. This is particularly relevant in the case of group-based measures that often appear in fairness analyses. The technical results that enable this are insightful and the proposed numerical optimization algorithm looks promising. The comprehensive empirical evaluation of the proposed methodology is commendable and provides valuable insights.\n\n[1] Jake C Snell, Thomas P Zollo, Zhun Deng, Toniann Pitassi, and Richard Zemel. Quantile risk control: A flexible framework for bounding the probability of high-loss predictions. arXiv preprint arXiv:2212.13629, 2022.\n\nWEAKNESSES:\n1. Significance\n\n\tI am a bit torn on the significance of this contribution. The contribution is two-fold: provide the theoretical underpinnings for bounds of other types of functions on the CDF and provide a numerical optimization method that gives tighter bounds than the truncated Berk-Jones introduced in [1]. However, the technical results are not particularly insightful or novel and the bounds provided by the numerical optimization are comparable with the truncated Berk-Jones ones. On the other hand, the immediate relevance to fairness applications is noteworthy and this practical relevance alone makes the work valuable and suitable for presentation at the conference.\n\n2. Paper structure\n\n\tThe writing in its current form lacks sufficient focus and intuition. The authors include many results in the main paper, while relegating most of the technical and experimental insights to the appendix. Consequently, the paper becomes excessively dense in certain sections. To address this, I would suggest narrowing the focus to a select few measures and demonstrating how the novel results apply, while providing intuitive explanations. The remaining measures could be included in the appendix. Similarly, a more extensive discussion on the numerical optimization method would greatly enhance the paper's clarity. By adopting these suggestions, the paper can strike a better balance between comprehensive coverage and presenting key insights effectively.\n\nQUESTIONS:\nSee weaknesses +:\n\n* I understand that the numerical optimization gives tighter bounds, but how do you ensure that these bounds are valid?\n\nLIMITATIONS:\nLimitations were adequately addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper demonstrates how to give distribution-free probabilistic control of certain statistical dispersion measures that are relevant in societal applications. An example is the GINI coefficient.\n\nThe paper technically builds on and slightly extends the work of Snell et al., which shows how to control quantile-based risk measures.\n\nSTRENGTHS:\nCongratulations to the authors on the great paper. It is the best in my batch of 6.\nI am familiar with the line of work on quantile-based risk measures, and am happy to see it used in such an interesting application domain.\nThe GINI coefficient and friends are a unique set of application domains, with large practical importance. Though I do not necessarily see large methodological novelty from the statistical point of view in this paper, I do believe that it has the potential for large impact in the social sciences (politics/economics, etc.) if the techniques are adopted.\n\nWEAKNESSES:\nI will be an advocate for the paper's acceptance, but I would really encourage the authors to do a major rewrite during the revision stage. The writing and presentation needs work, in my opinion.\nI have taken careful notes below. They are a bit stream-of-consciousness, so please forgive me for the tone.\n\nWRITING FEEDBACK:\n\nSection 4 reads really strangely to me right now.\nIt contains essentially no technical content, but is written in a technical way.\nIt’s more of an advertisement for the appendix. But it is vague, so I do not understand what has been proven, and what technical tools were used to do that until after I read the appendix (which 99% of your readers will NOT do).\nE.g. Under “Control for absolute and polynomial functions” you say “we can control these” but don’t give a formal statement saying what that means. Also, is it a corollary of some main result, or is it a bespoke proof? We don’t know until we read the appendix.\nFor that reason, I’m not really learning anything from section 4 as a reader, which is not good.\nIf you need to make space, cut 4.3 which is not important for main text.\n There are many places where the math just doesn’t make sense, and false statements are made, which makes the paper confusing. (To be clear, I believe all the statements are formally true, but not always for the reason given.) Examples:\n(1) “building blocks mentioned above as linear functionals of J: (i) nonlinear functions of J, i.e. \\xi(J)”. What is this supposed to mean? J represents either F or F^- so \\xi is a functional, not a function right? And you’re saying it’s nonlinear, so how can it be linear? As a side note, this “building blocks” section really didn’t work for me in terms of clarifying the later examples. It is so mathematical! Who are you writing for, a theorist, an applied statistician, or a social scientist? You have to decide, because the latter does not know what a function of a functional is, and the middle one doesn’t care and will probably be turned off by that language in my honest opinion.\n\nAs an alternative, just say that you are trying to control “functions of a whole CDF, like the GINI coefficient.” You can delineate the exact distinctions later when you’ve already made your main piont. “nonlinear function of a functional” and similar language is definitely distracting, not helping.\n\n(2) Under “Control for a monotonic function”, this does not hold because of Proposition 1. Proposition 1 is a deterministic statement, and you need Proposition 3 to make the probabilistic statement. This is confusing because it is obviously false to say that “by proposition 1 you have X with probability 1-\\delta”.\n\n(3) The statement of Proposition 3 is false. The second line holds only with high probability.\n\n(4) In Appendix C, it’s not “By classic DKW inequality” — it should say “By the classic DKW inequality and a Union bound/Bonferroni correction” because the classic DKW inequality does not have a k in it. Also, I find this subsection uninteresting and would consider greatly shortening it by just stating “you can use Bonferroni correction” and then giving the example.\n\nThe paper strikes a strange balance of being needlessly complex while presenting almost no theoretical detail in the main manuscript. It is fine to defer the most general form to the appendix, but in my honest opinion, the hybrid that is happening now is not very effective. Here are a few suggestions and places where I think the writing is ineffective.\n\n(1) Simplify superscripts and subscripts. The letter “F” gets way too many superscripts/subscripts/hats. It is hard for an unfamiliar reader to keep track of what’s happening. But it can be simplified with essentially no loss: for example, because you never write F_{n/2} or F^{delta/2} at all in the main paper, it suffices to remove those scripts entirely.\n\n(2) Remove technical minutiae. Remark 1 can be removed and the same point can be made as a half-sentence somewhere. Don’t use J as a generic notation for F or F^-, it’s very confusing. \n\n(3) I would add at least one actual technical result in the main body of the paper. Everything is in the appendix right now. I definitely respect the reasoning behind that choice, but right now it really isn’t working. The reality is that the paper is very mathematical, but without some clear theoretical statement (Theorem 2 from appendix would suffice), it’s not clear what the main result is and why it works. Furthermore, things like Proposition 1 do not help. Proposition 1 is obvious (does not require proof) and it isn’t directly connected to any of the claims you make in the paper. It should definitely be in the appendix. Whereas Theorem 2 is your main hammer, so you should highlight it. And you should explicitly state somewhere that the \n\nThe sentence starting with “Roughly speaking, a function of bounded total variation…” is totally out of place given how technical the previous parts of the section have been.\n\nQUESTIONS:\nN/A, just let me know if I have misunderstood anything above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGiven a trained predictor, this paper proposes a framework for bounding various measures of dispersion of the loss distribution.\n\nThe approach taken here is similar to that of conformal prediction, where one uses a calibration set to estimate the distribution of the loss. In this paper, the authors use the empirical CDF of the loss function on the calibration set to build confidence bands for the true (unknown) CDF. These confidence bands can then be used to obtain bounds on the dispersion measure of interest.  This approach is demonstrated on several measures of dispersion: Gini coefficient, Atkinson index, difference of group medians, and others.\n\nSTRENGTHS:\n* This paper addresses an important class of problems.\n* The proposed methodology has a broad scope as it is applicable to any ML system and any data distribution (so long as it is i.i.d. and no distribution shift has occurred).\n* The method should be easy to incorporate into standard ML packages and workflows. Thus, it has a high potential impact on ML practice.\n* In terms of originality, this is a moderate contribution and seems mostly like a follow-up paper to [1]. However its applicative focus is sufficiently different. Also, it contains some technical novelty. Chiefly, the gradient-based optimization approach for the construction of tight confidence bands for a given dispersion measure.\n* On the experimental front, the proposed approach is demonstrated on a diverse set of dispersion measures and data sets, showing its real-world applicability.\n\n[1] Jake Snell, Thomas P Zollo, Zhun Deng, Toniann Pitassi, Richard Zemel. \"Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictions\" ICLR (2023).\n\nWEAKNESSES:\nThe results in this paper are based on the construction of confidence bands for the CDF of the loss and in particular on the use of goodness-of-fit statistics for the construction of these bands. This idea is well-known in the field of statistics and the authors should do a better job of connecting with that literature. In particular, the paper [2] from 1995 proposed constructing confidence bands for the CDF using the Berk-Jones statistic, just like the baseline used in the current paper, but is not cited.\n\nMinor points for improvement:\n1. The fonts used in the figures are a bit small.\n2. The notation in Section 2 is confusing. I believe it would be better to use the standard notation: X for inputs and Y for outputs. Z can be used for loss(h(X),Y).\n3. The last paragraph of Section 3 is a bit much. Consider breaking it down and providing specific simple examples for clarity.\n4. Page 4, line 139: a space is missing in \"measures.They\"\n\n[2] Art Owen. \"Nonparametric likelihood confidence bands for a distribution function\". Journal of the American Statistical Association (1995)\n\nQUESTIONS:\n* In many applications, one is given a finite dataset to work with. In which case, under the current paper's framework there is a need to decide how to split the data into a primary training set and an additional calibration set (which the authors refer to as validation set, however this term is different from that used in relation to cross-validation). Are there any guidelines that you can provide on this matter? e.g. minimal size of the calibration needed set to achieve a certain error.\n\n* Using a fully-connected NN to parameterize the confidence boundaries in the optimization seems like overkill. I suggest that the authors try a much simpler parameterization, such as low-degree polynomials or using the first few elements of a Fourier series. This should be sufficient to express any smooth boundary accurately and is likely to converge faster due to the smaller number of parameters.\n\nLIMITATIONS:\n* This paper only considers one-sided confidence bands for the CDF and the bounds on dispersion that result from them. In some cases it may be sensible to consider two-sided confidence bands but this point is not discussed in the paper. It seems that the repeated integration method for one-sided bands on which this paper is based can only be used for computing one-sided bands. I think this limitation should be at least mentioned.  One could approximate a two-sided 1-alpha level boundary by combining an upper and lower band at the level of 1-alpha/2. However, this is not optimal (and is also not mentioned in the paper). Note that there are methods in the literature that compute two-sided confidence bands for the CDF that are based on similar ideas and they could be incorporated in some manner into the framework presented in the paper (but this is too much to ask for in a revision!).\n\n* Distribution shift is not mentioned anywhere, though I don't see this as a clear limitation of the proposed framework. Depending on the type of distributional change, one might be able to construct wider confidence bands that take it into account.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a methodology for distribution-free confidence intervals for a wide class of dispersion measures. This is motivated by validating the performance of machine learning algorithms with respect to more complex notions of performance, such as group-based measures of loss balance or the Gini coefficient of the allocation of the losses. This paper studies the problem in more generality than previous work, which focused on losses that are empirical means of quantities or quantile-based risk measures. The paper demonstrates the proposed methodology on several solid benchmark tasks.\n\nSTRENGTHS:\nThe problem is well-motivated, and the need for such guarantees about (dispersion) risk measures is convincing. Such techniques would help audit and verify aspects of the performance of machine learning algorithms. I find the introduction and abstract to be clear and set the scene well.\n\nThe work involves some non-trivial mathematics, and it is a formidable technical extension of previous work. A reader of previous work would not be able to apply it to the risk measures studied herein without this manuscript.\n\nThe examples are serviceable and demonstrate the breadth of the proposed method.\n\nWEAKNESSES:\nMy main concern is with the technical presentation in Section 4. I am fairly knowledgeable about this literature, but I still found it challenging to read at first. The authors have a lot of technical content to cover with limited space, so right now much of it is deferred to the appendix. I understand that this is challenging, and I appreciate all the work that has gone into this draft. Nonetheless, I hope this can be improved in the next iteration. In general, my opinion is probably too much content for the amount of space that is currently allocated, so perhaps more space can be made for this section.\n\nMy first suggestion is that the authors state the desired property of the output of their algorithm explicitly at the beginning of Chapter 4 or perhaps earlier -- if I understand correctly the goal is to give a 1-delta confidence interval for some risk measure. The output of the algorithm is this confidence interval. This is in contrast to some other works in risk control and conformal prediction, where the output of the algorithm is a choice of a parameter that controls the coverage rate or other risk level. When the authors of this work use the word \"control\" in the abstract and at various other points, I was initially confused as to what they meant. \n\nSecondly, it's not immediately clear what the proposed algorithm is when reading section 4. The section seems quite compressed, and as a result the reader is left with a lot of work to put together the pieces from the relatively lean presentation. I would find it helpful to see an algorithm environment or pseudocode explaining exactly the steps to compute the upper bound for at least one of the cases considered.\n\nIt goes without saying that when it comes to presentation choices, it is somewhat a matter of taste. I understand the authors may wish to go a different direction than my suggestions above. Still, iterating on this initial draft would be very helpful. In good faith, I'm giving the paper a score of \"7\" in anticipation that the final version incorporates some improvements to the presentation.\n\nQUESTIONS:\nTheorem 1 seems like the technical core. Is this impression correct? If so, more commentary about the functions f_1 and f_2 would be helpful. These exist in principal, but for which cases are they actually tractable?\n\nThere is currently some notational pain with the overloading of \\xi. Sometimes it is a function R to R, sometimes it is a functional. 4.1 is titled \"Control of nonlinear functions ...\" but then starts off with \"xi(F^-) which maps F^- to another function of R\", which is interpretting it as a function. However, later on it is said that xi is monotonic, which is invoking the interpretation of xi as a function R to R rather than a function. Similarly, 4.2 is about \"control of nonlinear functionals\" but then in the integral expression in line 183, F^- is evaluated at alpha, which means this expression is uses \\xi as a function, not functional.  I don't doubt that it's all correct, but it was hard for me to read at first. It would help the object \\xi is defined explicitly somewhere, so that exactly what type of object (function vs functional) it is is stated. \n\nMinor:\n-- Berk-Jones and DKW should be cited.\n-- I know of two other references in the conformal prediction literature that use uniform bounds on the CDF like Berk-Jones; https://arxiv.org/abs/2104.08279 and https://arxiv.org/abs/2304.06158 . These works are somewhat technically related, although I understand the use of the bounds on the loss distribution is conceptually different than using the bounds on the conformal score function.\n\nLIMITATIONS:\nThe authors do a good job explaining the context and limitations of their work. I would say the main limitation is that these techniques (like most techniques in this field) do not handle distribution shifts, and are assuming an iid data sample. This is explicitly addressed in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}