SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A more recent line of works using Optimal Transport (OT) [GDBFL19, BYF20, SRT+20, SMBN21, BDB22] has shown that it is possible to improve fairness without such tradeoffs by uniformizing distributions in different groups. Our method improves fairness and performance simultaneously by using OT. However, OT-based methods can suffer from additional errors induced by the optimal transport mapping. In our method, OT-induced errors that occur at the label sources can be covered by the label model, since those labeling functions that are made worse are down-weighted in label model fitting. A limitation of our method is that its resulting fairness scores are unpredictable in advance, while methods having fairness-accuracy tradeoffs typically come with tunable parameters to a priori control this tradeoff. However, as we argued in Section 5, our method is compatible with such methods, so that it is possible to control the fairness of the end model by adopting other methods as well.
Another related line of research is fairness under noisy labels [WLL21, KL22, WZN+23, ZZL+23]. [WLL21, WGH+22] suggests a new loss function that considers the noise level in labels to improve fairness. [KL22] studies PAC learning under a noisy label setting with fairness considerations. [WZN+23] introduces a regularizer to boost fairness under label noise, focusing on the performance difference in subpopulations. Finally,  $[ZZL^{+}23]$  also proposes a regularizer based on mutual information to improve fairness of representations under label noise. While our method shares the assumption of label noise, the main difference is that we have access to multiple noisy estimates—whose noise rates we can learn—and where we can apply fairness notions to each separate estimate. In contrast, other methods consider only one noisy label source—which corresponds to pseudolabels generated by label models in weak supervision. Thus, their methods can be used additionally with SBM, like other previous fair ML methods such as optimal threshold. Also, the typical weak supervision pipeline does not include training for individual label sources — so that techniques involving optimizing loss functions and regularization do not fit this setting.
Data slice discovery Data slice denotes a group of data examples that share a common attribute or characteristic  $\left[\mathrm{EVS}^{+}22\right]$ . Slice discovery aims to discover semantically meaningful subgroups from unstructured input data. Typical usage of slice discovery is to identify data slices where the model underperforms. For example, [DVMWVdM19] analyzed off-the-shelf object recognition models' performance with additional context annotations, revealing that model performance can vary depending on the geographical location where the image is taken.  $\left[\mathrm{WFT}^{+}19\right]$  shows that the melanoma detection models do not work well for dermascopic images with skin markings since they use those markings as a spurious feature. [ORDCR20] found pneumothorax classification models in chest X-rays underperform for chest x-ray images with chest drains. To discover such data slices without additional annotations, a variety of strategies have been employed. [ORDCR20, SDA  $^{+}20$ , EVS  $^{+}22$  ] used clustering or mixture modeling with dimensionality reduction such as U-MAP  $\left[\mathrm{SDA}^{+}20\right]$ , SVD  $\left[\mathrm{EVS}^{+}22\right]$ . [ddWLB22] solves an optimization problem to find high loss data points given a subgroup size.
While typical slice discovery methods require access to true labels to find underperforming data slices, weak supervision setting does not allow it. To work this around, we replace true labels with pseudolabels generated by label model. And, slice discovery methods are applied for outputs of each LF. Thus, the discovered data slices represent data points where each LF disagrees with the aggregated pseudolabels.
C Algorithm details
Figure 5: Overall WS pipeline involving SBM
Algorithm 2: ESTIMATE ACCURACY (TRIPLET) [FCS+20]
Parameters: Weak label values  $\lambda^1,\dots,\lambda^m$  for  $i,j,k\in \{1,2,\dots,m\}$  ( $i\neq j\neq k$ ) do  $|\hat{a}^i| \gets \sqrt{\hat{\mathbb{E}}[\lambda^i\lambda^j]\hat{\mathbb{E}}[\lambda^i\lambda^k] / \hat{\mathbb{E}}[\lambda^j\lambda^k]}$ $|\hat{a}^j| \gets \sqrt{\hat{\mathbb{E}}[\lambda^i\lambda^j]\hat{\mathbb{E}}[\lambda^j\lambda^k] / \hat{\mathbb{E}}[\lambda^i\lambda^k]}$ $|\hat{a}^k| \gets \sqrt{\hat{\mathbb{E}}[\lambda^i\lambda^k]\hat{\mathbb{E}}[\lambda^j\lambda^k] / \hat{\mathbb{E}}[\lambda^i\lambda^j]}$  end for return ResolveSign ( $\hat{a}^i$ )  $\forall i\in \{1,2,\dots,m\}$
Algorithm 3: TRANSPORT
Parameters: Source input  $X_{src}$ , destination input  $X_{dst}$ , source weak label values  $\lambda_{src}$ , destination weak label values  $\lambda_{dst}$ , Optimal Transport type  $O$ , the number of nearest neighbors  $k = 1$   
if OT type  $O$  is linear then  
 $\tilde{X}_{src} \gets \text{LINEAROT}(X_{src}, X_{dst})$  [KS84]  
else if OT type  $O$  is sinkhorn then  
 $\tilde{X}_{src} \gets \text{SINKHORNOT}(X_{src}, X_{dst})$  [Cut13]  
else  
 $\tilde{X}_{src} \gets X_{src}$   
end if  
 $\tilde{\lambda}_{src} \gets kNN(\tilde{X}_{src}, X_{dst}, \lambda_{dst}, k)$   
return  $\tilde{\lambda}_{src}$
In this appendix section, we discuss the details of the algorithms we used. The overall WS pipeline including SBM is illustrated in Figure 5. Our method is placed in the first step as a refinement of
Algorithm 4: LINEAROT (OT-L)[KS84]
Parameters: Source input  $X_{src}$ , destination input  $X_{dst}$ $\mu_s, \mu_t \gets \text{MEAN}(X_{src}), \text{MEAN}(X_{dst})$ $\Sigma_s, \Sigma_t \gets \text{COV}(X_{src}), \text{COV}(X_{dst})$ $A \gets \Sigma_s^{-1/2}(\Sigma_s^{1/2}\Sigma_t\Sigma_s^{1/2})^{1/2}\Sigma_s^{-1/2}$ $b \gets \mu_t - A\mu_s$ $n_{src} \gets \text{size}(X_{src})$   
for  $i \in \{1,2,\dots,n_{src}\}$  do  
 $\tilde{X}_{src}[i] \gets AX_{src}[i] + b$   
end for  
return  $\tilde{X}_{src}$
Algorithm 5: SINKHORNOT (OT-S)[Cut13]
Parameters: Source input  $X_{src}$ , destination input  $X_{dst}$ , Entropic regularization parameter  $\eta = 1$ $M \leftarrow$  PAIRWISE DISTANCE  $(X_{src}, X_{dst})$ $n_{src}, n_{dst} \leftarrow$  size  $(X_{src})$ , size  $(X_{dst})$ $a \leftarrow (1 / n_{src}, \dots, 1 / n_{src})^T \in \mathbb{R}^{n_{src}}$ $b \leftarrow (1 / n_{dst}, \dots, 1 / n_{dst})^T \in \mathbb{R}^{n_{dst}}$   
Set  $K \in \mathbb{R}^{n_{src} \times n_{dst}}$  such that  $K_{ij} = \exp(-M_{ij} / \eta)$   
Initialize  $v = (1, \dots, 1)^T \in \mathbb{R}^{n_{dst}}$   
while not converged do  
\[
u \leftarrow \frac{a}{Kv}
\]
\[
v \leftarrow \frac{b}{KTu}
\]
end while  
 $\pi \leftarrow \text{diag}(u) K \text{diag}(v)$ $\tilde{\pi} = n_{src} \pi$   
for  $i \in \{1, 2, \dots, n_{src}\}$  do  
 $\tilde{X}_{src}[i] \leftarrow \sum_{j=1}^{n_{dst}} \tilde{\pi}_{ij} X_{dst}[j]$   
end for  
return  $\tilde{X}_{src}$
noisy labels. In this step, we improve the noisy labels by transporting the low accuracy group to the high accuracy group. To identify low accuracy and high accuracy groups, we estimate accuracy using Algorithm 2. After estimating accuracies, we transport the low accuracy group to the high accuracy group and get refined labels for the low accuracy group by Algorithm 3. Linear OT (Optimal Transport) is used to estimate a mapping. Sinkhorn OT approximates optimal coupling using Sinkhorn iteration. After obtaining the optimal coupling, data points from the low accuracy group are mapped to the coupled data points in the high accuracy group. If OT type is not given, data points from the low accuracy group are mapped to the 1-nearest neighbor in the high accuracy group. After obtaining mapped points, weak labels associated with mapped points in the target group are used for the data points from the low accuracy group.
The next step is a standard weak supervision pipeline step – training a label model and running inference to obtain pseudolabels. Finally, we train the end model. We used the off-the-shelf label model Snorkel [BRL+19] for the label model and logistic regression as the end model in all experiments unless otherwise stated. For the optimal transport algorithm implementation, we used the Python POT package [FCG+21].
D Theory details
Accuracy is defined as  $P(\lambda(x) = y) = P(\lambda(x) = 1, y = 1) + P(\lambda(x) = -1, y = -1)$ . For ease of notation, below we set  $\phi(x, x^{\mathrm{center}_0}) = (1 + \|x - x^{\mathrm{center}_0}\|)^{-1}$ .
Theorem 4.1. Let  $g_1^{(k)}$  be an arbitrary sequence of functions such that  $\lim_{k\to \infty}\mathbb{E}_{x'\in g_1^{(k)}(\mathcal{X})}[\| x' - x^{center_0}\| ]\to \infty$ . Suppose our assumptions above are met; in particular, that the label  $y$  is independent of the observed features  $x = I(z)$  or  $x^{\prime} = g_{1}^{(k)}(z),\forall k$ , conditioned on the latent features  $z$ . Then,
$$
\lim _ {k \to \infty} \mathbb {E} _ {x ^ {\prime} \in g _ {1} ^ {(k)} (\mathcal {Z})} [ P (\lambda (x ^ {\prime}) = y) ] = \frac {1}{2},
$$
Proof of Theorem 4.1. Because  $\lim_{k\to \infty}\mathbb{E}_{x'\in g_1^{(k)}(\mathcal{Z})}[\| x' - x^{\mathrm{center}_0}\| ]\to \infty$  , we have
$$
\lim  _ {k \rightarrow \infty} \mathbb {E} _ {x ^ {\prime} \in g _ {1} ^ {(k)} (\mathcal {Z})} [ \phi (x ^ {\prime}, x ^ {\text {c e n t e r} _ {0}}) ] = 0
$$
and because  $Z = \sum_{i\in \{0,1\}}\sum_{j\in \{0,1\}}P(\lambda (x^{\prime}) = i,y = j)$
$$
\begin{array}{l} \lim  _ {k \rightarrow \infty} \mathbb {E} _ {x ^ {\prime} \in g _ {1} ^ {(k)} (\mathcal {Z})} [ P (\lambda (x ^ {\prime}) = y) ] = \lim  _ {k \rightarrow \infty} \mathbb {E} _ {x ^ {\prime} \in g _ {1} ^ {(k)} (\mathcal {Z})} [ P (\lambda (x ^ {\prime}) = 1, y = 1) + P (\lambda (x ^ {\prime}) = - 1, y = - 1) ] \\ = \lim  _ {k \rightarrow \infty} \mathbb {E} _ {x ^ {\prime} \in g _ {1} ^ {(k)} (\mathcal {Z})} \left[ \frac {1}{Z} \exp \left(\theta_ {0} (1) (1) \phi \left(x ^ {\prime}, x ^ {\text {c e n t e r} _ {0}}\right)\right) + \frac {1}{Z} \exp \left(\theta_ {0} (- 1) (- 1) \phi \left(x ^ {\prime}, x ^ {\text {c e n t e r} _ {0}}\right)\right)\right] \\ = \lim  _ {k \rightarrow \infty} \mathbb {E} _ {x ^ {\prime} \in g _ {1} ^ {(k)} (\mathcal {Z})} \left[ \frac {2}{Z} \exp \left(\theta_ {0} \phi \left(x ^ {\prime}, x ^ {\text {c e n t e r} _ {0}}\right)\right)\right] \\ = \lim  _ {k \rightarrow \infty} \mathbb {E} _ {x ^ {\prime} \in g _ {1} ^ {(k)} (\mathcal {Z})} \left[ \frac {2 \exp \left(\theta_ {0} \phi \left(x ^ {\prime} , x ^ {\text {c e n t e r} _ {0}}\right)\right)}{2 \left(\exp \left(\theta_ {0} \phi \left(x ^ {\prime} , x ^ {\text {c e n t e r} _ {0}}\right)\right) + \exp \left(- \theta_ {0} \phi \left(x ^ {\prime} , x ^ {\text {c e n t e r} _ {0}}\right)\right)\right)} \right] \\ = \lim  _ {k \rightarrow \infty} \mathbb {E} _ {x ^ {\prime} \in g _ {1} ^ {(k)} (\mathcal {Z})} \left[ \frac {\exp \left(\theta_ {0} \phi \left(x ^ {\prime} , x ^ {\text {c e n t e r} _ {0}}\right)\right)}{\exp \left(\theta_ {0} \phi \left(x ^ {\prime} , x ^ {\text {c e n t e r} _ {0}}\right)\right) + \exp \left(- \theta_ {0} \phi \left(x ^ {\prime} , x ^ {\text {c e n t e r} _ {0}}\right)\right)} \right] \\ = \frac {1}{2}. \\ \end{array}
$$
since we get
$$
\lim  _ {k \rightarrow \infty} \mathbb {E} _ {x ^ {\prime} \in g _ {1} ^ {(k)} (\mathcal {Z})} \exp \left(\theta_ {0} \phi \left(x ^ {\prime}, x ^ {\text {c e n t e r} _ {0}}\right)\right) = 1
$$
$$
\lim  _ {k \rightarrow \infty} \mathbb {E} _ {x ^ {\prime} \in g _ {1} ^ {(k)} (\mathcal {Z})} \exp (- \theta_ {0} \phi (x ^ {\prime}, x ^ {\text {c e n t e r} _ {0}})) = 1
$$
from  $\lim_{k\to \infty}\mathbb{E}_{x^{\prime}\in g_1^{(k)}(\mathcal{Z})}[\phi (x^{\prime},x^{\mathrm{center}_0})] = 0$
Theorem 4.2. Let  $\tau = \max \left(\frac{\mathbf{r}(\Sigma_0)}{n_0},\frac{\mathbf{r}(\Sigma_1)}{n_1},\frac{t}{\min(n_0,n_1)},\frac{t^2}{\max(n_0,n_1)^2}\right)$  and  $C$  be a constant. Using Algorithm 1, for any  $t > 0$ , we bound the difference
$$
\begin{array}{l} \left. \left| \mathbb {E} _ {x \in \mathcal {Z}} [ P (\lambda (x) = y) ] - \mathbb {E} _ {x ^ {\prime} \in \mathcal {X}} [ P (\lambda (\hat {h} (x ^ {\prime})) = y) ] \right| \right. \\ \leq 4 \theta_ {0} C \sqrt {\tau \mathbf {r} (\Sigma_ {1})}. \\ \end{array}
$$
with probability  $1 - e^{-t} - \frac{1}{n_1}$ .
We use the result from [FLF19], which bounds the difference of the true  $h$  and empirical  $\hat{h}$  Monge estimators between two distributions  $P_0(x)$  and  $P_1(x')$ . Let  $\mathbf{r}(\Sigma), \lambda_{\min}(\Sigma)$  and  $\lambda_{\max}(\Sigma)$  denote the effective rank, minimum and maximum eigenvalues of matrix  $\Sigma$  respectively. Then,
Lemma D.1 ([FLF19]). Let  $P_0(x)$  and  $P_{1}(x^{\prime})$  be sub-Gaussian distributions on  $\mathcal{X} = \mathbb{R}^{d}$  with expectations  $\mu_0,\mu_1$  and positive-definite covariance operators  $\Sigma_0,\Sigma_1$  respectively. We observe  $n_0$  points from the distribution  $P_{0}(x)$  and  $n_1$  points from the distribution  $P_{1}(x^{\prime})$ . We assume that
$$
c _ {1} <   \min  _ {j \in \{0, 1 \}} \lambda_ {\min } (\Sigma_ {j}) \leq \max  _ {j \in \{0, 1 \}} \lambda_ {\max } (\Sigma_ {j}) \leq c _ {2},
$$
for fixed constants  $0 < c_{1} \leq c_{2} < \infty$ . Further, we assume  $n_0 \geq c\mathbf{r}(\Sigma_0)$  and  $n_1 \geq d$  for a sufficiently large constant  $c > 0$ .
Then, for any  $t > 0$ , we have with probability at least  $1 - e^{-t} - \frac{1}{n_1}$  that
$$
\mathbb {E} _ {x ^ {\prime} \in \mathcal {X}} [ \left\| h (x ^ {\prime}) - \hat {h} (x ^ {\prime}) \right\| ] \leq C \sqrt {\tau \mathbf {r} (\Sigma_ {1})},
$$
where  $C$  is a constant independent of  $n_0, n_1, d, \mathbf{r}(\Sigma_0), \mathbf{r}(\Sigma_1)$  and  $\tau = \max\left(\frac{\mathbf{r}(\Sigma_0)}{n_0}, \frac{\mathbf{r}(\Sigma_1)}{n_1}, \frac{t}{\min(n_0, n_1)}, \frac{t^2}{\max(n_0, n_1)^2}\right)$ .
We will also use the following:
Lemma D.2. The probability  $P(\lambda(x) = y)$  is  $L$ -Lipschitz with respect to  $x \in \mathcal{X}$ . Specifically,  $\forall x_1, x_2 \in \mathcal{X}$ ,
$$
\left| P \left(\lambda \left(x _ {1}\right) = y\right) - P \left(\lambda \left(x _ {2}\right) = y\right) \right| \leq 4 \theta_ {0} \| x _ {1} - x _ {2} \|.
$$
Proof. We will demonstrate that, because  $\| \nabla_x P(\lambda(x) = y) \|$  is bounded above by  $4\theta_0$ ,  $P(\lambda(x) = y)$  must be  $4\theta_0$ -Lipschitz with respect to  $x$ . First,
$$
\begin{array}{l} P (\lambda (x) = y) = \frac {\exp \left(\theta_ {0} \phi \left(x , x ^ {\text {c e n t e r} _ {0}}\right)\right)}{\exp \left(\theta_ {0} \phi \left(x , x ^ {\text {c e n t e r} _ {0}}\right)\right) + \exp \left(- \theta_ {0} \phi \left(x , x ^ {\text {c e n t e r} _ {0}}\right)\right)} \\ = \frac {1}{1 + \exp \left(- 2 \theta_ {0} \phi \left(x , x ^ {\text {c e n t e r} _ {0}}\right)\right)} \\ = \sigma (2 \theta_ {0} \phi (x, x ^ {\mathrm {c e n t e r} _ {0}})), \\ \end{array}
$$
where  $\sigma$  denotes the sigmoid function. Note that  $\frac{d}{du}\sigma(u) = \frac{\exp(-u)}{(1 + \exp(-u))^2}$  for  $u \in \mathbb{R}$  and that  $\nabla_x[2\theta_0\phi(x, x^{\mathrm{center}_0})] = 2\theta_0\nabla_x\left[\frac{1}{1 + \|x - x^{\mathrm{center}_0}\|}\right] = 2\theta_0\frac{\nabla_x\left[\|x - x^{\mathrm{center}_0}\|\right]}{(1 + \|x - x^{\mathrm{center}_0}\|)^2}$ . Further, note that  $\nabla_x[\|x - x^{\mathrm{center}_0}\|] = \frac{x - x^{\mathrm{center}_0}}{\|x - x^{\mathrm{center}_0}\|}$  because we use Euclidean norm. Thus,
$$
\begin{array}{l} \left\| \nabla_ {x} P (\lambda (x) = y) \right\| = \left\| \nabla_ {x} \sigma \left(2 \theta_ {0} \phi \left(x, x ^ {\text {c e n t e r} _ {0}}\right)\right) \right\| \\ = \left| \left| 2 \theta_ {0} \frac {\exp \left(- 2 \theta_ {0} \phi \left(x , x ^ {\text {c e n t e r} _ {0}}\right)\right)}{\left(1 + \exp \left(- 2 \theta_ {0} \phi \left(x , x ^ {\text {c e n t e r} _ {0}}\right)\right)\right) ^ {2} \left(1 + \| x - x ^ {\text {c e n t e r} _ {0}} \|\right) ^ {2}} \cdot \nabla_ {x} [ \| x - x ^ {\text {c e n t e r} _ {0}} \| ] \right| \right| \\ = \left\| 2 \theta_ {0} \frac {\exp \left(- 2 \theta_ {0} \phi \left(x , x ^ {\text {c e n t e r} _ {0}}\right)\right)}{\left(1 + \exp \left(- 2 \theta_ {0} \phi \left(x , x ^ {\text {c e n t e r} _ {0}}\right)\right)\right) ^ {2}} \phi \left(x, x ^ {\text {c e n t e r} _ {0}}\right) ^ {2} \cdot \frac {x - x ^ {\text {c e n t e r} _ {0}}}{\| x - x ^ {\text {c e n t e r} _ {0}} \|} \right\| \\ = \left\| 2 \theta_ {0} \sigma \left(2 \theta_ {0} \phi \left(x, x ^ {\text {c e n t e r} _ {0}}\right)\right) \left(1 - \sigma \left(2 \theta_ {0} \phi \left(x, x ^ {\text {c e n t e r} _ {0}}\right)\right)\right) \phi \left(x, x ^ {\text {c e n t e r} _ {0}}\right) ^ {2} \cdot \frac {x - x ^ {\text {c e n t e r} _ {0}}}{\| x - x ^ {\text {c e n t e r} _ {0}} \|} \right\| \\ <   2 \theta_ {0} \left\| 1 \cdot (1 - (- 1)) \phi (x, x ^ {\text {c e n t e r} _ {0}}) ^ {2} \cdot \frac {x - x ^ {\text {c e n t e r} _ {0}}}{\| x - x ^ {\text {c e n t e r} _ {0}} \|} \right\| \\ = 4 \theta_ {0} | | \phi (x, x ^ {\text {c e n t e r} _ {0}}) ^ {2} \cdot 1 | | \\ \leq 4 \theta_ {0}. \\ \end{array}
$$
Thus,  $\| \nabla_x P(\lambda(x) = y) \|$  is bounded above by  $4\theta_0$ . We now use this fact to demonstrate that  $P(\lambda(x) = y)$  is  $4\theta_0$ -Lipschitz with respect to  $x$ .
For  $0 \leq v \leq 1$ , let  $s(v) = x_1 + (x_2 - x_1)v$ . For  $x$  between  $x_1$  and  $x_2$  inclusive and for  $v$  such that  $s(v) = x$ , since  $\|\nabla_x P(\lambda(x) = y)\| = \left|\frac{d}{dv} P(\lambda(s(v)) = y)\right| \leq 4\theta_0$ , we have
$$
\begin{array}{l} | P (\lambda (x _ {1}) = y) - P (\lambda (x _ {2}) = y) | = \left| \int_ {v = 0} ^ {1} \left[ \frac {d}{d v} P (\lambda (s (v)) = y) \right] \| s (v) \| d v \right| \\ \leq \int_ {v = 0} ^ {1} \left| \frac {d}{d v} P (\lambda (s (v)) = y) \right| \| s (v) \| d v \\ \leq \int_ {v = 0} ^ {1} 4 \theta_ {0} \| s (v) \| d v \\ = 4 \theta_ {0} \| x _ {1} - x _ {2} \|. \\ \end{array}
$$
Proof of Theorem 4.2. Now we are ready to complete our proof. We have,
$$
\begin{array}{l} \left. \left| \mathbb {E} _ {x \in \mathcal {Z}} [ P (\lambda (x) = y) ] - \mathbb {E} _ {x ^ {\prime} \in \mathcal {X}} [ P (\lambda (\hat {h} (x ^ {\prime})) = y) ] \right| \right. \\ \leq \mathbb {E} _ {z \in \mathcal {Z}} [ | P (\lambda (I (z)) = y) - P (\lambda (\hat {h} (g _ {1} (z))) = y | ] \\ = \mathbb {E} _ {z \in \mathcal {Z}} [ | P (\lambda (h (g _ {1} (z))) = y) - P (\lambda (\hat {h} (g _ {1} (z))) = y) ] \\ \leq \mathbb {E} _ {z \in \mathcal {Z}} \left[ 4 \theta_ {0} | | h (g _ {1} (z)) - \hat {h} (g _ {1} (z)) | | \right] (\text {b y}) \\ = \mathbb {E} _ {x ^ {\prime} \in \mathcal {X}} \left[ 4 \theta_ {0} | | h (x ^ {\prime}) - \hat {h} (x ^ {\prime}) | | \right] \\ \leq 4 \theta_ {0} C \sqrt {\tau \mathbf {r} (\Sigma_ {1})}, \\ \end{array}
$$
where the last line holds with probability at least  $1 - e^{-t} - \frac{1}{n_1}$  by Lemma D.1.
E.1 Dataset details
Table 7: Summary of real datasets
Domain	Datasets	Y	A	P(Y=1|A=1)	P(Y=1|A=0)	ΔDP
Tabular	Adult	Income (≥50000)	Sex	0.3038	0.1093	0.1945
Bank	Subscription	Age (≥25)	0.1093	0.2399	0.1306
NLP	CivilComments	Toxicity	Black	0.3154	0.1057	0.2097
HateXplain	Toxicity	Race	0.8040	0.4672	0.3368
Vision	CelebA	Gender	Young	0.3244	0.6590	0.3346
UTKFace	Gender	Asian	0.4856	0.4610	0.0246
Adult The adult dataset \(\left[\mathrm{K}^{+}96\right]\) has information about the annual income of people and their demographics. The classification task is to predict whether the annual income of a person exceeds \\(50,000 based on demographic features. Demographic features include age, work class, education, marital status, occupation, relationship, race, capital gain and loss, work hours per week, and native country. The group variable is whether age is greater than 25 or not. The training data has 32,561 examples, and the test data has 16,281 examples.
BankMarketing The bank marketing dataset [MCR14] was collected during the direct marketing campaigns of a Portuguese banking institution from 2008 to 2013. Features consist of demographic attributes, financial attributes, and the attributes related to the interaction with the campaign. The task is to classify whether a client will make a deposit subscription or not. The group variable is, whether age is greater than 25 or not. The total number of instances is 41,188, where the training dataset has 28,831 rows and the test dataset has 12,357 rows.
CivilComments The CivilComments dataset  $\left[\mathrm{BDS}^{+}19\right]$  is constructed from data collected in Jigsaw online conversation platform. We used the dataset downloaded from  $\left[\mathrm{KSM}^{+}21\right]$ . Features are extracted using BERT [DCLT18] embeddings from comments, and the task is to classify whether a given comment is toxic or not. This dataset has annotations about the content type of comments, e.g. race, age, religion, etc. We chose "black" as the group variable. The total number of instances is 402,820, where the training dataset has 269,038 rows and the test dataset has 133,782 rows.
HateXplain The HateXplain dataset  $\left[\mathrm{MSY}^{+}21\right]$  is comments collected from Twitter and Gab and labeled through Amazon Mechanical Turk. Features are extracted using BERT embeddings from comments. The task is to classify whether a given comment is toxic or not. This dataset has three label types - hate, offensive, or normal. We used hate and offensive label types as "toxic" label, that is, if
one of two labels types is 1 for a row, the row has label 1. The dataset has annotations about the target communities - race, religion, gender, LGBTQ. We used the race group as the group variable. The total number of instances is 17,281, where the training set has 15,360 rows and the test set has 1,921 rows.
CelebA The CelebA dataset [LLWT15] is the image datasets that has images of faces with various attributes. Features are extracted using CLIP[RKH $^{+}$ 21] embeddings from images. We tried gender classification task from this dataset—various other tasks can be constructed. We used “Young” as the group variable. The total number of instances is 202,599, where the training set has 162,770 rows and the test set has 39,829 rows.
UTKFace UTKFace dataset [ZSQ17] consists of faces images in the wild with annotations of age, gender, and ethnicity. Features are extracted using CLIP embeddings from images. This dataset has attributes of age, gender, race and we chose gender as label (Male:0, Female: 1), solving gender classification. We used the race as the group variable, encoding "Asian" group as 1, other race groups as 0. The total number of instances is 23,705, where the training set has 18,964 rows and the test set has 4,741 rows.
E.2 Application of LIFT
To use smooth embeddings in the transport step of tabular datasets (adult and bank marketing), we convert tabular data rows into text sentences and then get BERT embeddings.
Adult First we decided r.pronoun for each row r from  $<\text{r.sex}>$  and then generated sentences with the following template. "<r.pronoun> is in her  $<\text{r.age_range}>$ s. <r.pronoun> is from  $<\text{r.nationality}>$ . <r.pronoun> works in  $<\text{r.workclass}>$ . Specifically, <r.pronoun> has a  $<\text{r.job}>$  job. <r.pronoun> works  $<\text{r.working_hour}>$  hours per week. <r.pronoun> total education year is  $<\text{r.education Years}>$ . <r.pronoun> is  $<\text{r.marital_status}>$ . <r.pronoun> is  $<\text{r.race}>$ ".
An example sentence is "She is in her 20s. She is from United-States. She works in private sector. Specifically, she has a sales job. She works 30 hours per week. Her total education year is 6 years. She is married. She is White."