SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {1}{\alpha (\alpha - 1)} \int_ {0} ^ {1} \left[ \left(\frac {F ^ {-} (p)}{\int_ {0} ^ {1} F ^ {-} (p) d p}\right) ^ {\alpha} - 1 \right] d p \leq \frac {1}{\alpha (\alpha - 1)} \int_ {0} ^ {1} \left[ \left(\frac {\hat {F} _ {n , L} ^ {\delta , -} (p)}{\int_ {0} ^ {1} \hat {F} _ {n , U} ^ {\delta , -} (p) d p}\right) ^ {\alpha} - 1 \right] d p.
$$
Other cases can be tackled in a similar way, which we will not reiterate here.
B.3 Hoover index
The Hoover index [16] is equal to the percentage of the total population's income that would have to be redistributed to make all the incomes equal.
Definition 7 (Hoover index). For a non-negative random variable  $X$ , the Hoover index is defined as
$$
H (X) = \frac {\int_ {0} ^ {1} | F ^ {-} (p) - \int_ {0} ^ {1} F ^ {-} (q) d q | d p}{2 \int_ {0} ^ {1} F ^ {-} (p) d p}
$$
Hoover index involves forms like  $|F^{-} - \mu|$  for  $\mu = \int_0^1 F^{-}(p) dp$ . This type of nonlinear structure can be dealt with the absolute function results mentioned in Appendix A.1.2.
For Hoover index and a  $(1 - \delta)$ -CBP  $(\hat{F}_{n,L}^{\delta},\hat{F}_{n,U}^{\delta})$ , let us define
$$
H _ {U} (X) = \frac {\int_ {0} ^ {1} \max \{| \hat {F} _ {n , L} ^ {\delta , -} (p) - \int_ {0} ^ {1} \hat {F} _ {n , U} ^ {\delta , -} (q) d q | , | \hat {F} _ {n , U} ^ {\delta , -} (p) - \int_ {0} ^ {1} \hat {F} _ {n , L} ^ {\delta , -} (q) d q | \} d p}{2 \int_ {0} ^ {1} \hat {F} _ {n , U} ^ {\delta , -} (p) d p}.
$$
Then, with probability at least  $1 - \delta$ ,  $H_{U}(,X)$  is an upper bound for  $H(X)$ .
B.4 Extreme observations & mean range
For example, a city may need to estimate the cost of damage to public amenities due to rain in a certain month. The loss for each day of a month is  $X_{1}, \dots, X_{k}$  i.i.d drawn from  $F$ , and the administration hopes to estimate and control the dispersion of the losses in a month so that they can accurately allocate resources. This involves quantities such as range  $(\max_{i \in [k]} X_{i} - \min_{j \in [k]} X_{j})$  or quantiles of extreme observations  $(\max_{i \in [k]} X_{i})$ . The CDF of extreme observations such as  $\max_{i \in [k]} X_{i}$  involves a nonlinear function of  $F$ , i.e.  $(F(x))^k$ .
Example 7 (Quantiles of extreme observations). The CDF of  $\max_{i\in [k]}X_i$  is  $F^k$ . Thus, by the result of Appendix A.1.2, if given a  $(1 - \delta)$ -CBP  $(\hat{F}_{n,L}^{\delta},\hat{F}_{n,U}^{\delta})$  and  $1\succeq \hat{F}_{n,U}^{\delta , - } \succeq \hat{F}_{n,L}^{\delta , - } \succeq 0$ , with probability at least  $1 - \delta$ ,
$$
(\hat {F} _ {n, L} ^ {\delta , -}) ^ {k} \preceq F ^ {k} \preceq (\hat {F} _ {n, U} ^ {\delta , -}) ^ {k}.
$$
$$
(\hat {F} _ {n, U} ^ {\delta , -}) ^ {k} \preceq F ^ {k} \preceq (\hat {F} _ {n, L} ^ {\delta , -}) ^ {k}.
$$
Similarly, for  $\min_{i\in [k]}X_i$ , the CDF is  $1 - (1 - F)^{k}$ , thus, we have
$$
1 - \left(1 - \hat {F} _ {n, U} ^ {\delta , -}\right) ^ {k} \preceq F ^ {k} \preceq 1 - \left(1 - \hat {F} _ {n, L} ^ {\delta , -}\right) ^ {k}.
$$
We also want to emphasize, even if,  $X$  is not necessarily non-negative, we can apply the polynomial method in Appendix A.1.2 for  $\hat{F}_{n,U}^{\delta, -}$  and  $\hat{F}_{n,L}^{\delta, -}$ .
Example 8 (Mean range). By [13], if we further have prior knowledge that  $X$  is of continuous distribution, the mean of  $\max_{i\in [k]}X_i - \min_{j\in [k]}X_j$  can be expressed as:
$$
k \int F ^ {-} (x) \left[ F ^ {k - 1} (x) - F ^ {k} (x) \right] d F (x) = k \int_ {0} ^ {1} F ^ {-} \left(F ^ {-} (p)\right) \left[ F ^ {k - 1} \left(F ^ {-} (p)\right) - F ^ {k} \left(F ^ {-} (p)\right) \right] d p
$$
Notice that both  $F$  and  $F^{-}$  are increasing. Thus, if given a  $(1 - \delta)$ -CBP  $(\hat{F}_{n,L}^{\delta},\hat{F}_{n,U}^{\delta})$ ,  $\hat{F}_{n,L}^{\delta}\succeq 0$ , then with probability at least  $1 - \delta$ ,
$$
\int_ {0} ^ {1} \hat {F} _ {n, L} ^ {\delta , -} (\hat {F} _ {n, L} ^ {\delta , -} (p)) [ (\hat {F} _ {n, U} ^ {\delta}) ^ {k} (\hat {F} _ {n, L} ^ {\delta , -} (p)) - (\hat {F} _ {n, L} ^ {\delta}) ^ {k} (\hat {F} _ {n, U} ^ {\delta , -} (p)) ] d p
$$
is an upper bound of the mean range.
There are many other interesting societal dispersion measures that could be handled by our framework, such as those in [19]. For example, they study tail share that captures "the top  $1\%$  of people own  $X$  share of wealth", which could be easily handled with the tools provided here. We will leave those those examples to readers.
C Extension to multi-dimensional cases and applications
We briefly discuss extending our approach to multi-dimensional losses. Unfortunately, there is not a gold-standard definition of quantiles in the multi-dimensional case, and thus we only discuss functionals of CDFs and provide an example. For multi-dimensional samples  $\{X_i\}_{i=1}^n$ , each of  $k$  dimensions, i.e.  $\mathbf{X}_i = (X_1^i, \dots, X_k^i)$ , for any  $k$ -dimensional vector  $\mathbf{x} = (x_1, \dots, x_k)$ , define empirical CDF
$$
\hat {F} _ {n} (\pmb {x}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {1} \{\pmb {X} _ {i} \preceq \pmb {x} \}.
$$
where we abuse the notation  $\preceq$  to mean all of  $X_{i}$ 's coordinates are smaller than  $x$ 's.
By classic DKW inequality, we have with probability at least  $1 - \delta$
$$
| \hat {F} _ {n} (\boldsymbol {x}) - F (\boldsymbol {x}) | \leq \sqrt {\frac {\ln (k (n + 1) / \delta)}{2 n}}.
$$
Meanwhile, we can further adopt Frechet-Hoeffding bound, which gives,
$$
\max  \{1 - k + \sum_ {i = 1} ^ {k} F _ {i} \left(x _ {i}\right), 0 \} \leq F (\boldsymbol {x}) \leq \min  \left\{F _ {1} \left(x _ {1}\right), \dots , F _ {k} \left(x _ {k}\right) \right\}
$$
where  $F_{i}$  is the CDF of the i-th coordinate. Then, we can construct  $(\hat{F}_{n,L}^{\delta /k,i},\hat{F}_{n,U}^{\delta /k,i})$  such that  $(\hat{F}_{n,L}^{\delta /k,i}\preceq F_i\preceq \hat{F}_{n,U}^{\delta /k,i})$ , with probability at last  $1 - \delta /k$ . Thus, by union bound,
$$
\max  \{1 - k + \sum_ {i = 1} ^ {k} \hat {F} _ {n, L} ^ {\delta / k, i} (x _ {i}), 0 \} \leq F (\boldsymbol {x}) \leq \min  \{\hat {F} _ {n, U} ^ {\delta / k, 1} (x _ {1}), \dots , \hat {F} _ {n, U} ^ {\delta / k, k} (x _ {k}) \}
$$
for all  $\pmb{x}$  with probability at last  $1 - \delta$ .
$$
F (\boldsymbol {x}) \geq \max  \{1 - k + \sum_ {i = 1} ^ {k} \hat {F} _ {n, L} ^ {\delta / k, i} (x _ {i}), 0, \hat {F} _ {n} (\boldsymbol {x}) - \sqrt {\frac {\ln (k (n + 1) / \delta)}{2 n}} \}
$$
$$
F (\boldsymbol {x}) \leq \min  \{\hat {F} _ {n, U} ^ {\delta / k, 1} (x _ {1}), \dots , \hat {F} _ {n, U} ^ {\delta / k, k} (x _ {k}), \hat {F} _ {n} (\boldsymbol {x}) + \sqrt {\frac {\ln (k (n + 1) / \delta)}{2 n}} \}
$$
with probability at last  $1 - 2\delta$
Example 9 (Gini correlation coefficient [34]). The Gini correlation coefficient for two non-negative random variable  $X$  and  $Y$  are defined as
$$
\Gamma_ {X, Y} := \frac {\operatorname {C o v} (X , F _ {Y} (Y))}{\operatorname {C o v} (X , F _ {X} (X))} = \frac {\int \int \left(F _ {X , Y} (x , y) - F _ {X} (x) F _ {Y} (Y)\right) d x d F _ {Y} (y)}{\operatorname {C o v} (X , F _ {X} (X))},
$$
where  $F_{X}, F_{Y}$  are marginal CDFs of  $X, Y$  and  $F_{X,Y}$  is the joint CDF. One can use the multidimensional CDF bounds and our previous methods to provide bounds for the Gini correlation coefficient.
This section contains additional details for the experiments in Section 5. We set  $\delta = 0.05$  (before statistical corrections for multiple tests) in all experiments unless otherwise explicitly stated. Whenever we are bounding measures on multiple hypotheses, we perform a correction for the size of the hypothesis set. Additionally, when we bound measures on multiple distributions (e.g. demographic groups), we also perform a correction. Our code will be released publicly upon the publication of this article.
D.1 CivilComments (Section 5.1)
Our set of hypotheses are a toxicity model combined with a Platt scalar [22], where the model is fixed and we vary the scaling parameter in the range  $[0.25, 2]$  while fixing the bias term to 0. We use a pre-trained toxicity model from the popular python library Detoxify [10] and perform Platt Scaling using code from the python library released by [18]. A Platt calibrator produces output according to:
$$
h (v) = \frac {1}{1 + \exp (w v + b)}
$$
where  $w, b$  are learnable parameters and  $v$  is the log odds of the prediction. Thus we form our hypothesis set by varying the parameter  $w$  while fixing  $b$  to 0. Examples are drawn from the train split of CivilComments, which totals 269,038 data points.
The loss metric for our CivilComments experiments is the Brier Score. For  $n$  data points, Brier score is calculated as:
$$
L = \frac {1}{n} \sum_ {i = 1} ^ {n} (f _ {i} - o _ {i}) ^ {2}
$$
where  $f_{i}$  is prediction confidence and  $o_{i}$  is the outcome (0 or 1).
$^{6}$ https://github.com/unitaryai/detoxify
<sup>7</sup>https://github.com/p-lambda/verified_calibration
D.1.1 Bounding complex objectives (Section 5.1.1)
We randomly sample 100,000 test points for calculating the empirical values in Table 1, and draw our validation points from the remaining data. We perform a Bonferroni correction on  $\delta = 0.05$  for the size of the set of hypotheses as well as the number of distributions on which we bound our measures (in this case the number of groups, 4). We set  $\lambda = 1.0$ .
Numerical optimization details (including training strategy and hyperparameters) are the same as Section 5.1.2, explained below in Appendix D.1.2. For each group  $g$  we optimize the objective
$$
\mathcal {O} = T _ {1} \left(F _ {g}\right) + T _ {2} \left(F _ {g}\right)
$$
where  $F_{g}$  is the CDF bound for group,  $T_{1}$  is expected loss, and  $T_{2}$  is a smoothed version of a median with  $a = 0.01$  (see Appendix D.1.2 and Figure 5).
For comparison, the DKW inequality is applied to get a CDF lower bound, which is then transformed to an upper bound via the reduction approach in Section 4.1.1. To get the lower bound  $b_{1:n}^{l}$ , we set:
$$
b _ {i} ^ {l} = \max (0, \frac {\# \mathrm {p o i n t s} \leqslant \frac {i}{n}}{n} - \sqrt {\frac {\log (\frac {2}{\delta})}{2 n}})
$$
D.1.2 Numerical optimization examples (Section 5.1.2)
We parameterize the bounds with a fully connected network with 3 hidden layers of dimension 64. The  $n$  gaussian seeds are of size 32, which is also the input dimension for the network. Training is performed in two stages, where the network is first trained to approximate a Berk-Jones bound, and then optimized for some specified objective  $O$ . In both stages of training we aim to push the training error to zero or as close as possible (i.e. "overfit"), since we are optimizing a bound and do not seek generalization. The model is first trained for 100,000 epochs to output the Berk-Jones bound using a mean-squared error loss. Then optimization on  $O$  is performed for a maximum of 10,000 epochs, and validation is performed every 25 epochs, where we choose the best model according to the bound on  $O$ . Both stages of optimization use the Adam optimizer with a learning rate 0.00005, and for the second stage the constraint weight is set to  $\lambda = 0.00005$ . We perform post-processing to ensure the constraint holds (see Section A.2.2). For some denominator  $m$  (in our case  $m = 10^6$ ) we set  $\gamma = \frac{1}{m}$ ,  $\frac{2}{m}$ ,  $\frac{3}{m}$ , ... and check the constraint until it is satisfied.
This approach is applied to both the experiments in Section 5.1.1 and Section 5.1.2. Details on the objective for Section 5.1.1 are above in Appendix D.1.1. In Section 5.1.2, we set  $\delta = 0.01$  and our metrics for optimization are described below:
CVaR CVaR is a measure of the expected loss for the items at or above some quantile level  $\beta$ . We set  $\beta = 0.75$ , and thus we bound the expected loss for the worst-off  $25\%$  of the population.
VaR-Interval In the event that different stakeholders are interested in the VaR for different quantile levels  $\beta$ , we may want to select a bound based on some interval of the VaR  $[\beta_{min}, \beta_{max}]$ . We perform our experiment with  $\beta_{min} = 0.5, \beta_{max} = 0.9$ , which includes the median  $(\beta = 0.5)$  through the worst-case loss excluding a small batch of outliers  $(\beta = 0.9)$ .
Quantile-Weighted We apply a weighting function to the quantile loss  $\psi(p) = p$ , such that the loss incurred by the worst-off members of a population are weighted more heavily.
Smoothe d Median We study a more robust version of a median:
$$
\psi (p; \beta) = \frac {1}{a \sqrt {\pi}} \exp (- \frac {(p - \beta) ^ {2}}{a ^ {2}})
$$
with  $\beta = 0.5$  and  $a = 0.01$ , similar to a normal distribution extremely concentrated around its mean. See Figure 5 for an illustration of such a weighting.
D.2 Bounds on standard measures (Section 5.2)
This section contains additional details for the experiments in Section 5.2.
Figure 5: Plot of smoothed median function with  $\beta = 0.5$  and  $a = 0.01$
D.2.1 RxRx1 (Section 5.2.1)
We use the code released by  $[17]^8$  to pre-train a model on the train split of RxRx1 [31] and we evaluate our algorithm on the OOD val split with 9854 total samples. We randomly sample 2500 items for use in validation (bounding and model selection), and use the remainder of the data points for illustrating the empirical distribution induced by the different hypotheses. The thresholds which are combined with the pre-trained model to form our hypothesis set are evenly spaced in  $[-8,0]$  under the log transformation with base 10, thus leaving the thresholds in the range  $[10^{-8},1]$ .
Balanced accuracy is calculated as:
$$
L (\hat {Y}, Y) = 1 - \frac {1}{2} \left(\operatorname {S e n s} (\hat {Y}, Y) + \operatorname {S p e c} (\hat {Y}, Y)\right), \text {w h e r e}
$$
$$
\operatorname {S e n s} (\hat {Y}, Y) = \frac {| \hat {Y} \cap Y |}{| Y |} \text {a n d} \operatorname {S p e c} (\hat {Y}, Y) = \frac {k - | Y | - | \hat {Y} \setminus Y |}{k - | Y |}.
$$
where  $Y$  is the set of ground truth labels (which in this experiment will always be one label),  $\hat{Y}$  is a set of predictions, and  $k$  is the number of classes.
D.2.2 MovieLens-1M (Section 5.2.2)
MovieLens-1M [12] is a publicly available dataset. We filter all ratings below 5 stars, a typical pre-processing step, and filter any users with less than 15 5-star ratings, leaving us with 4050 users. For each user, the 5 most recently watched items are added to the test set, while the remaining (earlier) items are added to the train set. We train a user/item embedding model using the popular python recommender library LightFM with a WARP ranking loss for 30 epochs and an embedding dimension of 16.
For recommendation set  $\hat{I}$  we compute a loss combining recall and precision against a user test set  $I$  of size  $k$ :
$$
L = \alpha l _ {r} (\hat {I}, I) ^ {2} + (1 - \alpha) l _ {p} (\hat {I}, I) ^ {2}, \text {w h e r e}
$$
$$
l _ {r} (\hat {I}, I) = 1 - \frac {1}{k} \sum_ {i \in I} \mathbb {1} \{i \in \hat {I} \} \text {a n d} l _ {p} (\hat {I}, I) = 1 - \frac {1}{| \hat {I} |} \sum_ {i \in \hat {I}} \mathbb {1} \{i \in I \}
$$
where  $\alpha = 0.5$ . We randomly sample 1500 users for validation, and use the remaining users to plot the empirical distributions. The 100 hypotheses tested are evenly spaced between the minimum and maximum scores of any user/item pair in the score matrix.
<sup>8</sup>https://github.com/p-lambda/wilds
<sup>9</sup>https://github.com/lyst/lightfm
E Additional results for numerical optimization (Section 5.1.2)
Figure 6 compares the learned bounds  $G_{opt}$  to the Berk-Jones ( $G_{BJ}$ ) and Truncated Berk-Jones ( $G_{BJ - t}$ ) bounds, as well as the empirical CDF of the real loss distribution.
Figure 6: Learning tighter bounds on functionals of interest for protected groups. On the left, a bound is optimized for CVaR with  $\beta = 0.75$ , and on the right a bound is optimized for the VaR Interval [0.5, 0.9]. In both cases the optimized bounds are tightest on both the target metric as well as the mean, illustrating the power of adaptation both to particular quantile ranges as well as real loss distributions.