SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 1: For  $i, j \in \{0, 1\}$ , the classifiers  $\widehat{Y} = f(X)$  and  $\widehat{A} = h(X)$  satisfy
$$
\frac {U _ {i}}{\hat {r} _ {i , j}} \leq \widehat {\alpha} _ {i, j} \leq 1 - \frac {U _ {i}}{\hat {r} _ {i , j}}.
$$
We now expand on the implications of this assumptions. Recall that  $U_{i} = \mathbb{P}(\widehat{A} = i, A \neq i)$ ,  $\hat{r}_{i,j} = \mathbb{P}(\widehat{A} = i, Y = j)$ , and  $\widehat{\alpha}_{i,j} = \mathbb{P}(\widehat{Y} = 1 \mid \widehat{A} = i, Y = j)$ . Thus Assumption 1 states,
$$
\mathbb {P} (\widehat {A} = i, A \neq i) \leq \mathbb {P} (\widehat {Y} = 1, \widehat {A} = i, Y = j) \leq \mathbb {P} (\widehat {A} = i, Y = j) - \mathbb {P} (\widehat {A} = i, A \neq i). \tag {10}
$$
Immediately, it is clear that Eq. (10) implies
$$
\begin{array}{l} \mathbb {P} (\widehat {A} = i, A \neq i) \leq \mathbb {P} (\widehat {A} = i, Y = j) - \mathbb {P} (\widehat {A} = i, A \neq i) (11) \\ \Longrightarrow \mathbb {P} (\widehat {A} = i, A \neq i) \leq \frac {1}{2} \mathbb {P} (\widehat {A} = i, Y = j) (12) \\ \end{array}
$$
Now, the left inequality of Eq. (10) states
$$
\mathbb {P} (\widehat {A} = i, A \neq i) \leq \mathbb {P} (\widehat {Y} = 1, \widehat {A} = i, Y = j) \tag {13}
$$
and the right inequality of Eq. (10) states
$$
\mathbb {P} (\widehat {Y} = 1, \widehat {A} = i, Y = j) \leq \mathbb {P} (\widehat {A} = i, Y = j) - \mathbb {P} (\widehat {A} = i, A \neq i)
$$
$$
\begin{array}{l} \mathbb {P} (\widehat {A} = i, A \neq i) \leq \mathbb {P} (\widehat {A} = i, Y = j) - \mathbb {P} (\widehat {Y} = 1, \widehat {A} = i, Y = j) (14) \\ = \mathbb {P} (\widehat {Y} = 0, \widehat {A} = i, Y = j) (15) \\ \end{array}
$$
If  $j = 1$ , then this implies
$$
\mathbb {P} (\widehat {A} = i, A \neq i) \leq \mathbb {P} (\widehat {Y} = 1, \widehat {A} = i, Y = 1) \tag {16}
$$
$$
\mathbb {P} (\widehat {A} = i, A \neq i) \leq \mathbb {P} (\widehat {Y} = 0, \widehat {A} = i, Y = 1) \tag {17}
$$
and if  $j = 0$
$$
\mathbb {P} (\widehat {A} = i, A \neq i) \leq \mathbb {P} (\widehat {Y} = 1, \widehat {A} = i, Y = 0) \tag {18}
$$
$$
\mathbb {P} (\widehat {A} = i, A \neq i) \leq \mathbb {P} (\widehat {Y} = 0, \widehat {A} = i, Y = 0) \tag {19}
$$
Any reasonable classifier  $\widehat{Y}$  would have the properties
$$
\mathbb {P} (\widehat {Y} = 0, \widehat {A} = i, Y = 1) \leq \mathbb {P} (\widehat {Y} = 1, \widehat {A} = i, Y = 1) \tag {20}
$$
$$
\mathbb {P} (\widehat {Y} = 1, \widehat {A} = i, Y = 0) \leq \mathbb {P} (\widehat {Y} = 0, \widehat {A} = i, Y = 0) \tag {21}
$$
Thus, Assumption 1 is met when
$$
\mathbb {P} (\widehat {A} = i, A \neq i) \leq \mathbb {P} (\widehat {Y} = j, \widehat {A} = i, Y \neq j) \tag {22}
$$
We only prove the result for  $|\Delta_{\mathrm{TPR}}(f)|$  as the proof for  $|\Delta_{\mathrm{FPR}}(f)|$  is completely analogous.
Proof. The rules of conditional probability and the law of total probability allow us to decompose  $\alpha_{1,1}$  and  $\alpha_{0,1}$  in the following manner,
$$
\begin{array}{l} \alpha_ {1, 1} = \mathbb {P} (\hat {Y} = 1 \mid A = 1, Y = 1) (23) \\ = \frac {\mathbb {P} (\hat {Y} = 1 , A = 1 , Y = 1)}{\mathbb {P} (A = 1 , Y = 1)} (24) \\ = \frac {\sum_ {i \in \{0 , 1 \}} \mathbb {P} (\hat {Y} = 1 , A = 1 , Y = 1 , \hat {A} = i)}{\sum_ {j \in \{0 , 1 \}} \sum_ {i \in \{0 , 1 \}} \mathbb {P} (\hat {Y} = j , A = 1 , Y = 1 , \hat {A} = i)} (25) \\ = \frac {\sum_ {i \in \{0 , 1 \}} \mathbb {P} (\hat {Y} = 1 , A = 1 , Y = 1 \mid \hat {A} = i) \cdot \mathbb {P} (\hat {A} = i)}{\sum_ {j \in \{0 , 1 \}} \sum_ {i \in \{0 , 1 \}} \mathbb {P} (\hat {Y} = j , A = 1 , Y = 1 \mid \hat {A} = i) \cdot \mathbb {P} (\hat {A} = i)} (26) \\ \end{array}
$$
$$
\begin{array}{l} \alpha_ {0, 1} = \mathbb {P} (\hat {Y} = 1 \mid A = 0, Y = 1) (27) \\ = \frac {\mathbb {P} (\hat {Y} = 1 , A = 0 , Y = 1)}{\mathbb {P} (A = 0 , Y = 1)} (28) \\ = \frac {\mathbb {P} (\hat {Y} = 1 , Y = 1) - \mathbb {P} (\hat {Y} = 1 , A = 1 , Y = 1)}{\mathbb {P} (Y = 1) - \mathbb {P} (A = 1 , Y = 1)} (29) \\ = \frac {\mathbb {P} (\hat {Y} = 1 , Y = 1) - \left[ \sum_ {i \in \{0 , 1 \}} \mathbb {P} (\hat {Y} = 1 , A = 1 , Y = 1 \mid \hat {A} = i) \cdot \mathbb {P} (\hat {A} = i) \right]}{\mathbb {P} (Y = 1) - \left[ \sum_ {j \in \{0 , 1 \}} \sum_ {i \in \{0 , 1 \}} \mathbb {P} (\hat {Y} = j , A = 1 , Y = 1 \mid \hat {A} = i) \cdot \mathbb {P} (\hat {A} = i) \right]} (30) \\ \end{array}
$$
Therefore,  $\Delta_{\mathrm{TPR}}(f) = \alpha_{1,1} - \alpha_{0,1}$  is a function of the four probabilities given by
$$
\mathbb {P} (\hat {Y} = j, A = 1, Y = 1 \mid \hat {A} = i) \tag {31}
$$
which are unidentifiable in a demographically scarce regime and therefore not computable.
The Fréchet inequalities tell us that for  $i,j\in \{0,1\}$
$$
\mathbb {P} (\hat {Y} = j, A = 1, Y = 1 \mid \hat {A} = i) \geq \max  \left\{\mathbb {P} (\hat {Y} = j, Y = 1 \mid \hat {A} = i) - \mathbb {P} (A = 0 \mid \hat {A} = i), 0 \right\} \tag {32}
$$
$$
\mathbb {P} (\hat {Y} = j, A = 1, Y = 1 \mid \hat {A} = i) \leq \min  \left\{\mathbb {P} (\hat {Y} = j, Y = 1 \mid \hat {A} = i), \mathbb {P} (A = 1 \mid \hat {A} = i) \right\}. \tag {33}
$$
Observe that,  $\Delta_{\mathrm{TPR}}(f)$  is an increasing function with respect to the two probabilities
$$
\mathbb {P} (\hat {Y} = 1, A = 1, Y = 1 \mid \hat {A} = i) \tag {34}
$$
and a decreasing one with respect to the two probabilities,
$$
\mathbb {P} (\hat {Y} = 0, A = 1, Y = 1 \mid \hat {A} = i). \tag {35}
$$
As a result,  $\Delta_{\mathrm{TPR}}(f)$  is maximal when  $\mathbb{P}(\hat{Y} = 1, A = 1, Y = 1 \mid \hat{A} = i)$  achieve their maximum values and  $\mathbb{P}(\hat{Y} = 0, A = 1, Y = 1 \mid \hat{A} = i)$  achieve their minimum values. On the other hand,  $\Delta_{\mathrm{TPR}}(f)$  is minimal when  $\mathbb{P}(\hat{Y} = 1, A = 1, Y = 1 \mid \hat{A} = i)$  achieve their minimum values and  $\mathbb{P}(\hat{Y} = 0, A = 1, Y = 1, \mid \hat{A} = i)$  achieve their maximum values. With these facts, we now provide the upper bound. Recall from Appendix A.1 that Assumption 1 implies
$$
\mathbb {P} (\widehat {A} = i, A \neq i) \leq \frac {1}{2} \mathbb {P} (\widehat {A} = i, Y = j) \tag {36}
$$
$$
\mathbb {P} (\widehat {A} = i, A \neq i) \leq \mathbb {P} (\widehat {Y} = j, \widehat {A} = i, Y \neq j) \leq \mathbb {P} (\widehat {Y} = j, \widehat {A} = i, Y = j) \tag {37}
$$
With Assumption 1 we first provide the values of min  $[\mathbb{P}(\hat{Y} = 0, A = 1, Y = 1, | \hat{A} = i)]$ . First,
$$
\begin{array}{l} \mathbb {P} (\hat {Y} = 0, Y = 1, | \hat {A} = 1) - \mathbb {P} (A = 0 | \hat {A} = 1) = \frac {\mathbb {P} (\hat {Y} = 0 , \hat {A} = 1 , Y = 1)}{P (\hat {A} = 1)} - \frac {U _ {1}}{\mathbb {P} (\hat {A} = 1)} (39) \\ \geq 0 (40) \\ \end{array}
$$
because  $\mathbb{P}(\hat{Y} = 0, \hat{A} = 1, Y = 1) - U_1 \geq 0$ . Second,
$$
\begin{array}{l} \mathbb {P} (\hat {Y} = 0, Y = 1, | \hat {A} = 0) - \mathbb {P} (A = 0 | \hat {A} = 0) = \frac {\mathbb {P} (\hat {Y} = 0 , \hat {A} = 0 , Y = 1)}{P (\hat {A} = 0)} - \frac {\mathbb {P} (A = 0 , \hat {A} = 0)}{\mathbb {P} (\hat {A} = 0)} (41) \\ = \frac {\mathbb {P} (\hat {Y} = 0 , \hat {A} = 0 , Y = 1)}{P (\hat {A} = 0)} - \frac {\mathbb {P} (\hat {A} = 0) - U _ {0}}{\mathbb {P} (\hat {A} = 0)} (42) \\ = \frac {\mathbb {P} (\hat {Y} = 0 , \hat {A} = 0 , Y = 1)}{P (\hat {A} = 0)} - \frac {\mathbb {P} (\hat {A} = 0) - U _ {0}}{\mathbb {P} (\hat {A} = 0)} (43) \\ = \frac {\mathbb {P} (\hat {Y} = 0 , \hat {A} = 0 , Y = 1)}{P (\hat {A} = 0)} + \frac {U _ {0}}{\mathbb {P} (\hat {A} = 0)} - 1 (44) \\ \end{array}
$$
Now note that,
$$
\begin{array}{l} \mathbb {P} (\hat {Y} = 0, \hat {A} = 0, Y = 1) = \mathbb {P} (\hat {A} = 0, Y = 1) - \mathbb {P} (\hat {Y} = 1, \hat {A} = 0, Y = 1) (45) \\ \leq \mathbb {P} (\hat {A} = 0, Y = 1) - U _ {0} (46) \\ \end{array}
$$
where the second equality is due to Assumption 1. As a result
$$
\begin{array}{l} \frac {\mathbb {P} (\hat {Y} = 0 , \hat {A} = 0 , Y = 1)}{P (\hat {A} = 0)} + \frac {U _ {0}}{\mathbb {P} (\hat {A} = 0)} - 1 \leq \frac {\mathbb {P} (\hat {A} = 0 , Y = 1) - U _ {0}}{P (\hat {A} = 0)} + \frac {U _ {0}}{\mathbb {P} (\hat {A} = 0)} - 1 (47) \\ = \frac {\mathbb {P} (\hat {A} = 0 , Y = 1)}{P (\hat {A} = 0)} - 1 \leq 0 (48) \\ \end{array}
$$
$$
\min  \left[ \mathbb {P} (\hat {Y} = 0, A = 1, Y = 1 \mid \hat {A} = 1) \right] = \mathbb {P} (\hat {Y} = 0, Y = 1, \mid \hat {A} = 1) - \mathbb {P} (A = 0 \mid \hat {A} = 1) \tag {49}
$$
$$
\min  \left[ \mathbb {P} (\hat {Y} = 0, A = 1, Y = 1 \mid \hat {A} = 0) \right] = 0 \tag {50}
$$
Now we provide the values of max  $[\mathbb{P}(\hat{Y} = 1, A = 1, Y = 1, | \hat{A} = i)]$ . First,
$$
\begin{array}{l} \mathbb {P} (A = 1 \mid \hat {A} = 1) = \frac {\mathbb {P} (A = 1 , \hat {A} = 1)}{\mathbb {P} (\hat {A} = 1)} (51) \\ = \frac {\mathbb {P} (\hat {A} = 1) - U _ {1}}{\mathbb {P} (\hat {A} = 1)} (52) \\ \geq \frac {\mathbb {P} (\hat {A} = 1 , Y = 1) - U _ {1}}{\mathbb {P} (\hat {A} = 1)} (53) \\ \geq \frac {\mathbb {P} (\hat {A} = 1 , Y = 1) - \mathbb {P} (\hat {Y} = 0 , \hat {A} = 1 , Y = 1)}{\mathbb {P} (\hat {A} = 1)} (54) \\ = \frac {\mathbb {P} (\hat {Y} = 1 , \hat {A} = 1 , Y = 1)}{\mathbb {P} (\hat {A} = 1)} = \mathbb {P} (\hat {Y} = 1, Y = 1 \mid \hat {A} = 1) (55) \\ \end{array}
$$
Second,
$$
\begin{array}{l} \mathbb {P} (A = 1 \mid \hat {A} = 0) = \frac {\mathbb {P} (A = 1 , \hat {A} = 0)}{\mathbb {P} (\hat {A} = 0)} (56) \\ \leq \frac {\mathbb {P} (\hat {Y} = 1 , \hat {A} = 0 , Y = 1)}{\mathbb {P} (\hat {A} = 0)} = \mathbb {P} (\hat {Y} = 1, Y = 1 \mid \hat {A} = 0) (57) \\ \end{array}
$$
$$
\max  \left[ \mathbb {P} \left(\hat {Y} = 1, A = 1, Y = 1 \mid \hat {A} = 1\right) \right] = \mathbb {P} \left(\hat {Y} = 1, Y = 1 \mid \hat {A} = 1\right) \tag {58}
$$
$$
\max  \left[ \mathbb {P} (\hat {Y} = 1, A = 1, Y = 1 \mid \hat {A} = 0) \right] = \mathbb {P} (A = 1 \mid \hat {A} = 0) \tag {59}
$$
Plugging these 4 values into  $\Delta_{\mathrm{TPR}}$  will yield the upper bound,
$$
B _ {1} + C _ {0, 1} = \frac {\hat {r} _ {1 , 1}}{\hat {r} _ {1 , 1} + \Delta U} \widehat {\alpha} _ {1, 1} - \frac {\hat {r} _ {0 , 1}}{\hat {r} _ {0 , 1} - \Delta U} \widehat {\alpha} _ {0, 1} + U _ {0} \left(\frac {1}{\hat {r} _ {1 , 1} + \Delta U} + \frac {1}{\hat {r} _ {0 , 1} - \Delta U}\right) \tag {60}
$$
One can similarly use the assumptions to derive the lower bound,
$$
B _ {1} - C _ {1, 1} = \frac {\hat {r} _ {1 , 1}}{\hat {r} _ {1 , 1} + \Delta U} \hat {\alpha} _ {1, 1} - \frac {\hat {r} _ {0 , 1}}{\hat {r} _ {0 , 1} - \Delta U} \hat {\alpha} _ {0, 1} - U _ {1} \left(\frac {1}{\hat {r} _ {1 , 1} + \Delta U} + \frac {1}{\hat {r} _ {0 , 1} - \Delta U}\right) \tag {61}
$$
and thus  $|\Delta_{\mathrm{TPR}}| \leq \max \{|B_1 + C_{0,1}|, |B_1 - C_{1,1}|\}$ . One can use same arguments to derive the upper bound for  $|\Delta_{\mathrm{FPR}}|$ .
We prove the result for  $|\Delta_{\mathrm{TPR}}(f)|$ . We first start by proving the existence portion of the theorem.
Let  $\widehat{A} = h(X)$  be a sensitive attribute classifier with errors  $U_{0}$  and  $U_{1}$  that produces rates  $\hat{r}_{i,j} = \mathbb{P}(\widehat{A} = i,Y = j)$ . Let  $\mathcal{F}$  be the set of classifiers for  $Y$  such that  $\forall f\in \mathcal{F}$ ,  $f$  and  $h$  satisfy Assumption 1. Consider any  $f\in \mathcal{F}$  with group conditional probabilities,  $\widehat{\alpha}_{i,j} = \mathbb{P}(\widehat{Y} = 1\mid \widehat{A} = i,Y = j)$ . Since we are only proving the result for  $|\Delta_{\mathrm{TPR}}(f)|$ , set  $j = 1$ . Consider the  $xy$  plane, with the  $x$ -axis being  $\widehat{\alpha}_{0,1}$  and the  $y$ -axis being  $\widehat{\alpha}_{1,1}$ . We know,
$$
\frac {U _ {i}}{\hat {r} _ {i , 1}} \leq \widehat {\alpha} _ {i, 1} \leq 1 - \frac {U _ {i}}{\hat {r} _ {i , 1}} \quad \Longrightarrow \quad \frac {U _ {i}}{\hat {r} _ {i , 1}} \leq \frac {1}{2}. \tag {62}
$$
The two equations above define a rectangular region in the  $xy$  plane with a center  $(\frac{1}{2},\frac{1}{2})$ , meaning any classifier  $f\in \mathcal{F}$ , has  $\widehat{\alpha}_{i,j}$  that are in this region.
Now, denote  $\bar{\mathcal{F}}$  to be a set of classifiers for  $Y$ , with group conditional probabilities  $\underline{\hat{\alpha}}_{i,1}$ , that satisfy the condition,
$$
\frac {\hat {r} _ {0 , 1}}{\hat {r} _ {0 , 1} - \Delta U} \underline {{\alpha}} _ {0, 1} - \frac {\hat {r} _ {1 , 1}}{\hat {r} _ {1 , 1} + \Delta U} \underline {{\alpha}} _ {1, 1} = \frac {\Delta U}{2} \left(\frac {1}{\hat {r} _ {1 , 1} + \Delta U} + \frac {1}{\hat {r} _ {0 , 1} - \Delta U}\right). \tag {63}
$$
This condition defines a line in the  $xy$  plane meaning any classifier in  $\bar{\mathcal{F}}$  has  $\underline{\hat{\alpha}}_{i,1}$  that are on this line. Now observe that the classifier  $\bar{f} \in \bar{\mathcal{F}}$  with  $\underline{\hat{\alpha}}_{i,1} = \frac{1}{2}$ , satisfy the above condition because,
$$
\begin{array}{l} \frac {\hat {r} _ {0 , 1}}{\hat {r} _ {0 , 1} - \Delta U} \left(\frac {1}{2}\right) - \frac {\hat {r} _ {1 , 1}}{\hat {r} _ {1 , 1} + \Delta U} \left(\frac {1}{2}\right) = \frac {1}{2} \left(\frac {\hat {r} _ {0 , 1}}{\hat {r} _ {0 , 1} - \Delta U} - \frac {\hat {r} _ {1 , 1}}{\hat {r} _ {1 , 1} + \Delta U}\right) (64) \\ = \frac {1}{2} \left(\frac {\hat {r} _ {0 , 1} - \Delta U + \Delta U}{\hat {r} _ {0 , 1} - \Delta U} - \frac {\hat {r} _ {1 , 1} + \Delta U - \Delta U}{\hat {r} _ {1 , 1} + \Delta U}\right) (65) \\ = \frac {1}{2} \left(1 + \frac {\Delta U}{\hat {r} _ {0 , 1} - \Delta U} - 1 + \frac {\Delta U}{\hat {r} _ {1 , 1} + \Delta U}\right) (66) \\ = \frac {\Delta U}{2} \left(\frac {1}{\hat {r} _ {1 , 1} + \Delta U} + \frac {1}{\hat {r} _ {0 , 1} - \Delta U}\right) (67) \\ \end{array}
$$
This implies that the line defined by Eq. (63) intersects the rectangular region that Assumption 1 defines. As a result,  $\mathcal{F} \cap \bar{\mathcal{F}}$  is not empty, meaning there exists a classifier  $\bar{f} \in \mathcal{F}$  with group conditional probabilities  $\widehat{\underline{\alpha}}_{i,1}$  that also satisfies the condition,
$$
\frac {\hat {r} _ {0 , 1}}{\hat {r} _ {0 , 1} - \Delta U} \underline {{\hat {\alpha}}} _ {0, 1} - \frac {\hat {r} _ {1 , 1}}{\hat {r} _ {1 , 1} + \Delta U} \underline {{\hat {\alpha}}} _ {1, 1} = \frac {\Delta U}{2} \left(\frac {1}{\hat {r} _ {1 , 1} + \Delta U} + \frac {1}{\hat {r} _ {0 , 1} - \Delta U}\right). \tag {68}
$$
Now we prove that such a classifier has minimal bounds. Theorem 1 tells us that for  $f \in \mathcal{F}$
$$
\left| \Delta_ {\mathrm {T P R}} (f) \right| \leq B _ {\mathrm {T P R}} (f) \triangleq \max  \left\{\left| B _ {1} + C _ {0, 1} \right|, \left| B _ {1} - C _ {1, 1} \right| \right\}
$$
Note that  $B_{1}$  is linear in  $\widehat{\alpha}_{1,1}$  and  $\widehat{\alpha}_{0,1}$  and that  $C_{0,1}$  and  $C_{1,1}$  are constants such that  $B_{1} + C_{0,1} \geq B_{1} - C_{1,1}$  simply because  $B_{1} + C_{0,1}$  is the upper bound for  $\Delta_{TPR}$  and  $B_{1} - C_{0,1}$  is the lower bound. Since these bounds are affine functions shifted by a constant, then min-max  $\{|B_1 + C_{0,1}|, |B_1 - C_{1,1}|\}$  necessarily occurs when
$$
B _ {1} + C _ {0, 1} = - B _ {1} - C _ {1, 1} \tag {69}
$$
$$
2 B _ {1} = - \left(C _ {1, 1} + C _ {0, 1}\right) \tag {70}
$$
have minimal upper bounds on  $|\Delta_{\mathrm{TPR}}|$ . After rearranging terms, this condition is precisely
$$
\frac {\hat {r} _ {0 , 1}}{\hat {r} _ {0 , 1} - \Delta U} \widehat {\alpha} _ {0, 1} - \frac {\hat {r} _ {1 , 1}}{\hat {r} _ {1 , 1} + \Delta U} \widehat {\alpha} _ {1, 1} = \frac {\Delta U}{2} \left(\frac {1}{\hat {r} _ {1 , 1} + \Delta U} + \frac {1}{\hat {r} _ {0 , 1} - \Delta U}\right). \tag {71}
$$
FIFA 2020 (Awasthi et al. [5]): The task is to learn a classifier  $f$ , using FIFA 2020 player data [29], that determines if a soccer players wage is above  $(Y = 1)$  or below  $(Y = 0)$  the median wage based on the player's age and their overall attribute. The sensitive attribute  $A$  is player nationality and the player's name is used to learn the sensitive attribute predictor  $h$ . We consider two scenarios, when  $A \in \{\text{English}, \text{Argentine}\}$  (English  $= 0$ ) and  $A \in \{\text{French}, \text{Spanish}\}$  (French  $= 0$ ). To learn the sensitive attribute predictor  $h$ , we train a Bidirectional Encoder Representations from Transformers (BERT) model [26] using an Adam optimizer [27] for 5 epochs. To learn the label classifier  $f$ , we train a Random Forest classifier. On a test dataset, we generate our predictions  $\hat{Y} = f(X)$  and  $\hat{A} = h(X)$  to yield a dataset over  $(\hat{A}, Y, \hat{Y})$ . We utilize the bootstrap method to generate 1,000 samples of this dataset and, for each sample, perform the same correction algorithms as before to yield  $f_{\widehat{\text{fair}}}$  and  $f_{\text{opt}}$  and calculate the same metrics as done in the synthetic data experiments. For  $A \in \{\text{English}, \text{English}\}$  the nationality predictor,  $h$ , achieves an error of  $U = 0.033$  with  $U_0 \approx 0.008$  and  $U_1 \approx 0.015$ . For  $A \in \{\text{French}, \text{Spanish}\}$  the nationality predictor,  $h$ , achieves an error of  $U = 0.053$  with  $U_0 \approx 0.025$  and  $U_1 \approx 0.028$ . The results for both experiments are displayed in Fig. C.1 and Fig. C.2. In addition, we include Fig. C.1d, Fig. C.1e, Fig. C.2d, and Fig. C.2e, which display the difference between  $B_{\text{TPR}}(f_{\widehat{\text{fair}}})$  and  $B_{\text{TPR}}(f_{\text{opt}})$  and between  $B_{\text{FPR}}(f_{\widehat{\text{fair}}})$  and  $B_{\text{FPR}}(f_{\text{opt}})$ . These figures depict that the bounds of  $f_{\text{opt}}$  are smaller than the bounds of  $f_{\widehat{\text{fair}}}$  because the histograms of  $B_{\text{TPR}}(f_{\widehat{\text{fair}}}) - B_{\text{TPR}}(f_{\text{opt}})$  and  $B_{\text{FPR}}(f_{\widehat{\text{fair}}}) - B_{\text{FPR}}(f_{\text{opt}})$  are strictly to the right of the dashed green vertical line at 0, indicating that both differences are greater than or equal to 0.
ACSPublicCoverage (Ding et al. [15]): The task is to learn a classifier  $f$ , using the 2018 state census data, that determines if a low-income individual, not eligible for Medicare, has coverage from public health insurance ( $Y = 1$ ) or does not ( $Y = 0$ ). The sensitive attribute  $A$  is sex (Female = 0) and with a separate dataset (containing the same features used to learn  $f$  (disregarding sex)), we learn the sensitive attribute predictor  $h$ . We work with the 2018 California census data. To learn the sensitive attribute predictor  $h$  and label classifier  $f$ , we train two Random Forest classifiers. On a test dataset, we generate our predictions  $\hat{Y} = f(X)$  and  $\hat{A} = h(X)$  to yield a dataset over  $(\hat{A}, Y, \hat{Y})$ . We utilize the bootstrap method to generate 1,000 samples of this dataset and, for each sample, perform the same correction algorithms as before to yield  $f_{\widehat{\text{fair}}}$  and  $f_{\text{opt}}$  and calculate the same metrics as done in the previous experiments. The sex predictor,  $h$ , achieves an error of  $U = 0.07$  with  $U_0 \approx 0.05$  and  $U_1 \approx 0.02$ . The results for this experiment are in Fig. C.3. In addition, we include Fig. C.3b, which displays the difference between  $B_{\text{TPR}}(f_{\widehat{\text{fair}}})$  and  $B_{\text{TPR}}(f_{\text{opt}})$ . This figure depicts that the TPR bounds of  $f_{\text{opt}}$  are smaller than the TPR bounds of  $f_{\widehat{\text{fair}}}$  because the histogram of  $B_{\text{TPR}}(f_{\widehat{\text{fair}}}) - B_{\text{TPR}}(f_{\text{opt}})$  is strictly to the right of the dashed green vertical line at 0, indicating that this difference is greater than 0.
CheXpert (Irvin et al. [24]): CheXpert is a large public dataset for chest radiograph interpretation, consisting of 224,316 chest radio graphs of 65,240 patients, with labeled annotations for 14 observations (positive, negative, or unlabeled) including cardiomegaly, atelectasis, consolidation,
and several others. The task is to learn a classifier  $f$  to determine if an X-ray contains an annotation for any abnormal condition  $(Y = 1)$  or does not  $(Y = 0)$ . The sensitive attribute  $A$  is sex (Female  $= 0$ ) and with a separate set of X-rays (different from those used to learn  $f$ ) we learn the sensitive attribute predictor  $h$ . To learn both  $f$  and  $h$  we use a DenseNet121 convolutional neural network architecture. Images are fed into the network with size  $320 \times 320$  pixels. We use the Adam optimizer [27] with default  $\beta$ -parameters of  $\beta_{1} = 0.9, \beta_{2} = 0.999$  and a fixed learning rate of  $1 \times 10^{-4}$ . Batches are sampled using a fixed batch size of 16 images and we train for 5 epochs. On a test dataset, we generate our predictions  $\hat{Y} = f$  and  $\hat{A} = h$  to yield a dataset over  $(\hat{A}, Y, \hat{Y})$ . We utilize the bootstrap method to generate 1,000 samples of this dataset, and for each sample, perform the same correction algorithms as before to yield  $f_{\widehat{\text{fair}}}$  and  $f_{\text{opt}}$  and calculate the same metrics as done in the previous experiments. The sex predictor,  $h$ , achieves an error of  $U = 0.023$  with  $U_{0} \approx 0.008$  and  $U_{1} \approx 0.015$ . The results for this experiment are in Fig. 3.
C Additional Figures & Results
Table 1: Verification of Assumption 1
Dataset	\( \widehat{\alpha}_{i,j} \)	\( \frac{U_i}{\widehat{r}_{i,j}} \)	Actual Value	\( 1 - \frac{U_i}{\widehat{r}_{i,j}} \)
CheXpert	\( \widehat{\alpha}_{1,1} \)	0.03	0.80	0.97
\( \widehat{\alpha}_{0,1} \)	0.02	0.78	0.98
\( \widehat{\alpha}_{1,0} \)	0.17	0.22	0.83
\( \widehat{\alpha}_{0,0} \)	0.11	0.19	0.89
FIFA 2020 (English & Argentina)	\( \widehat{\alpha}_{1,1} \)	0.07	0.93	0.94
\( \widehat{\alpha}_{0,1} \)	0.05	0.67	0.95
\( \widehat{\alpha}_{1,0} \)	0.13	0.26	0.87
\( \widehat{\alpha}_{0,0} \)	0.03	0.05	0.97
FIFA 2020 (French & Spanish)	\( \widehat{\alpha}_{1,1} \)	0.08	0.86	0.92
\( \widehat{\alpha}_{0,1} \)	0.13	0.83	0.87
\( \widehat{\alpha}_{1,0} \)	0.11	0.13	0.89
\( \widehat{\alpha}_{0,0} \)	0.10	0.10	0.95
ACSPublicCoverage (California)	\( \widehat{\alpha}_{1,1} \)	0.13	0.39	0.87
\( \widehat{\alpha}_{0,1} \)	0.24	0.33	0.76
(a)  $\Delta_{\mathrm{TPR}}$  and  $B_{\mathrm{TPR}}$
(b)  $\Delta_{\mathrm{FPR}}$  and  $B_{\mathrm{FPR}}$
(d) Difference between  $B_{\mathrm{TPR}}(f_{\mathrm{fair}})$  and  $B_{\mathrm{TPR}}(f_{\mathrm{opt}})$
Figure C.1: Results for FIFA 2020 dataset with  $A \in \{\text{English, Argentine}\}$ .
(c) Expected loss
(e) Difference between  $B_{\mathrm{FPR}}(f_{\mathrm{fair}})$  and  $B_{\mathrm{FPR}}(f_{\mathrm{opt}})$
(a)  $\Delta_{\mathrm{TPR}}$  and  $B_{\mathrm{TPR}}$
(d) Difference between  $B_{\mathrm{TPR}}(f_{\widehat{\mathrm{fair}}})$  and  $B_{\mathrm{TPR}}(f_{\mathrm{opt}})$
Figure C.2: Results for FIFA 2020 dataset with  $A \in \{\text{French}, \text{Spanish}\}$ .
(b)  $\Delta_{\mathrm{FPR}}$  and  $B_{\mathrm{FPR}}$
(c) Expected loss
(e) Difference between  $B_{\mathrm{FPR}}(f_{\widehat{\mathrm{fair}}})$  and  $B_{\mathrm{FPR}}(f_{\mathrm{opt}})$
(a)  $\Delta_{\mathrm{TPR}}$  and  $B_{\mathrm{TPR}}$
Figure C.3: Results for ACSPublicCoverage 2018 California census data
(b) Difference between  $B_{\mathrm{TPR}}(f_{\widehat{\mathrm{fair}}})$  and  $B_{\mathrm{TPR}}(f_{\mathrm{opt}})$