SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For analysis purposes, we split the sample into several subsets and pass each subset to the corresponding estimator (the correspondence is explained later). Figure 1 shows an overview of the sample splitting and the correspondence between the subsets and estimators. We construct the histogram of the sensitive attribute  $S_{i}$  from the sample  $D_{n}$ , denoted as  $n_{\cdot} = (n_{1},\dots,n_{M})$ , where  $n_s = |\{i\in [n]:S_i = s\} |$  (upper left in Figure 1). We also construct group-wise samples  $D_{s} = \{(X_{i},Y_{i}):i\in [n],S_{i} = s\}$ . For each  $s\in [M]$ , we divide  $D_{s}$  into  $D_{1,s},D_{2,s}$ , and  $D_{3,s}$  such that  $|D_{b,s}| = n_{b,s}\geq \lfloor n_s / 3\rfloor$  for  $b\in [3]$  (lower left in Figure 1). We use  $n_{\cdot},D_{1,s},D_{2,s}$ , and  $D_{3,s}$  to estimate  $\hat{p}_s,\widehat{\| \beta_s\|},\tilde{\beta}_s$ , and  $\hat{\mu}_{s}$ , respectively, where  $\| \beta_s\|$  is the estimator for  $\| \beta^{*}s\|$ . We obtain  $\| \widehat{\beta}_s\|$  from the combination of  $\hat{p}_s$  and  $\| \widehat{\beta_s}\|$  (middle in Figure 1). Furthermore, for each  $s\in [M]$ , we create a copy of  $D_{s}$ , denoted as  $D^{\prime}s$ , and divide it into  $D^{\prime}1,s$  and  $D^{\prime}2,s$  such that  $|D^{\prime}b,s|\coloneqq n^{\prime}b,s\geq \lfloor n_s / 2\rfloor$  for  $b\in [2]$ . We use  $D^{\prime}1,s$  and  $D_{2,s}^{\prime}$  to estimate  $\hat{\beta}_s^\prime$  and  $\hat{\mu}_s^\prime$ , respectively (right in Figure 1).
We describe the construction of each estimator below. We first define some notations. Let the  $i$ th element of  $D_{b,s}$  and  $D_{b,s}^{\prime}$  be denoted as  $(X_{b,s,i},Y_{b,s,i})$  and  $(X_{b,s,i}^{\prime},Y_{b,s,i}^{\prime})$ , respectively. We use the matrix notations  $X_{b,s} = (X_{b,s,1}\dots X_{b,s,n_{b,s}})^{\top}$ ,  $X_{b,s}^{\prime} = (X_{b,s,1}^{\prime}\dots X_{b,s,n_{b,s}^{\prime}}^{\prime})^{\top}$ ,  $Y_{b,s} = (Y_{b,s,1}\dots Y_{b,s,n_{b,s}})^{\top}$ , and  $Y_{b,s}^{\prime} = (Y_{b,s,1}^{\prime}\dots Y_{b,s,n_{b,s}^{\prime}}^{\prime})^{\top}$ . We define the ordinary least square estimators for the subset of the sample  $D_{b,s}$  and  $D_{b,s}^{\prime}$  as follows:
$$
\hat {\beta} _ {b, s} = \left(\frac {1}{n _ {b , s}} X _ {b, s} ^ {\top} X _ {b, s}\right) ^ {- 1} \left(\frac {1}{n _ {b , s}} X _ {b, s} ^ {\top} Y _ {b, s}\right),
$$
$$
\hat {\beta} _ {b, s} ^ {\prime} = \left(\frac {1}{n _ {b , s} ^ {\prime}} (X _ {b, s} ^ {\prime}) ^ {\top} X _ {b, s} ^ {\prime}\right) ^ {- 1} \left(\frac {1}{n _ {b , s} ^ {\prime}} (X _ {b, s} ^ {\prime}) ^ {\top} Y _ {b, s} ^ {\prime}\right).
$$
We construct each estimator as follows:
$(\hat{p} s)$  We use the empirical mean defined as  $\hat{p}_s = n_s / n$
$(\widehat{\|\beta_s\|})$  We use the norm of the OLS estimator. We define  $\| \widehat{\beta_s}\| = \| \widehat{\beta}_{1,s}\|$  if  $n_{s} > 18d$ , and  $\| \widehat{\beta_s}\| = 0$  otherwise.
$(\widehat{\|\beta_{\cdot}\|})$  Since  $\overline{\|\beta_{\cdot}^{*}\|} = \sum_{s \in [M]} p_s \|\beta_s^*\|$ , we construct its estimator as  $\widehat{\|\beta_{\cdot}\|} = \sum_{s \in [M]} \hat{p}_s \widehat{\|\beta_s\}}$ .
$(\tilde{\beta}_s)$  We use the normalized ordinary least square estimator;  $\tilde{\beta}_s = \hat{\beta}_{2,s} / \| \hat{\beta}_{2,s}\|$  if  $n_s > 18d$ , and  $\tilde{\beta}_s = 0$  otherwise.
$(\hat{\mu}_s)$  We use the empirical mean;  $\hat{\mu}_s = \frac{1}{n_{3,s}}\sum_{i = 1}^{n_{3,s}}X_{3,s,i}$ .
$(\hat{\beta}_s^{\prime})$  We employ the ordinary least square estimator;  $\hat{\beta}_s^\prime = \hat{\beta}_{1,s}^\prime$  if  $n_s > 12d$ , and  $\hat{\beta}_s^\prime = 0$  otherwise.
$(\hat{\mu}_s^{\prime})$  We use the empirical mean;  $\hat{\mu}_s^\prime = \frac{1}{n_{2,s}^\prime}\sum_{i = 1}^{n_{2,s}^\prime}X_{2,s,i}^\prime$  if  $n_s > 12d$ , and  $\hat{\mu}_s^\prime = 0$  otherwise.
Some estimators change their behavior based on the condition  $n_s > 18d$  or  $n_s > 12d$ , which is done for technical purposes in later analyses.
Recall that the final regressor is constructed as follows:
$$
\widehat {f} _ {n} (x, s) = \widehat {\| \beta . \|} \Bigl \langle \widetilde {\beta} _ {s}, x - \widehat {\mu} _ {s} \Bigr \rangle + \sum_ {s ^ {\prime} \in [ M ]} \widehat {p} _ {s ^ {\prime}} \Bigl \langle \widehat {\beta} _ {s} ^ {\prime}, \widehat {\mu} _ {s} ^ {\prime} \Bigr \rangle .
$$
Algorithm 1 shows the algorithm for our estimator.
C Bayes Optimal Regressor under Our Modell
This section presents the proof of Lemma 1, demonstrating the Bayes optimal regressor under the model Eq (2). To establish this, we make use of a key result from the work of Chzhen et al. [7]:
Theorem 9 (Chzhen et al. [7]). Assume, for each  $s \in [M]$ ,  $\nu_{f^{*}|s}$  has a density. Then,
$$
\inf _ {f: \mathrm {D P}} \mathbf {E} \Big [ (f (X, S) - f ^ {*} (X, S)) ^ {2} \Big ] = \inf _ {\nu} \sum_ {s \in [ M ]} p _ {s} W _ {2} ^ {2} (\nu_ {f ^ {*} | s}, \nu).
$$
where the infimum is taken over all the regressors that satisfy the demographic parity. Moreover, letting  $f_{\mathrm{DP}}^*$  and  $\nu^*$  be the minimizer of the lhs and rhs, respectively, we have  $\nu_{f_{\mathrm{DP}}^*} = \nu^*$  and
$$
f _ {\mathrm {D P}} ^ {*} (x, s) = \left(\sum_ {s ^ {\prime} \in [ M ]} p _ {s ^ {\prime}} F _ {f ^ {*} | s ^ {\prime}} ^ {- 1}\right) \circ F _ {f ^ {*} | s} \left(f ^ {*} (x, s)\right). \tag {9}
$$
Here, we denote  $\nu_{f^*}$  as the distribution of  $f^{*}(X,S)$ ,  $F_{f|s}$  as the cumulative distribution function of  $f(X,S)$  conditioned on  $S = s$ , and  $F_{f|s}^{-1}$  as the inverse cumulative distribution function, given by  $F_{f|s}^{-1}(t) = \inf \{y\in \mathbb{R}|F_{f|s}(y)\geq t\}$ .
Building upon the results of Theorem 9, we establish the proof of Lemma 1.
Proof of Lemma 1. Building upon Theorem 9, we can derive the Bayes optimal regressor under the model Eq (2) by obtaining closed expressions of the cumulative and inverse cumulative distribution functions  $F_{f^{*}|s}$  and  $F_{f^{*}|s}^{-1}$ . To obtain these closed forms, we apply certain transformations to  $f^{*}(X,S)$  that render it a random variable following a standard normal distribution. Let  $\Phi$  and  $\Phi^{-1}$  be the CDF and inverse CDF of the standard normal distribution, respectively. Through elementary calculations, we have:
$$
\begin{array}{l} F _ {f ^ {*} \mid s} (t) = \mathbb {P} \{f ^ {*} (X, S) \leq t \mid S = s \} \\ = \mathbb {P} \left\{\left\langle \beta_ {s} ^ {*}, X \right\rangle \leq t \mid S = s \right\} \\ = \mathbb {P} \bigg \{\frac {1}{\sigma_ {X} \| \beta_ {s} ^ {*} \|} \langle \beta_ {s} ^ {*}, X - \mu_ {s} \rangle \leq \frac {1}{\sigma_ {X} \| \beta_ {s} ^ {*} \|} (t - \langle \beta_ {s} ^ {*}, \mu_ {s} \rangle) | S = s \bigg \}. \\ \end{array}
$$
Here, we can readily observe that  $\frac{1}{\sigma_X\|\beta_s^*\|}\langle \beta_s^*,X - \mu_s\rangle$  follows the standard normal distribution under conditioned on  $S = s$ , as  $X\sim N(\mu_s,\sigma_XI)$  conditioned on  $S = s$ . Consequently, we have
$$
F _ {f ^ {*} \mid s} (t) = \Phi \left(\frac {1}{\sigma_ {X} \| \beta_ {s} ^ {*} \|} \left(t - \langle \beta_ {s} ^ {*}, \mu_ {s} \rangle\right)\right). \tag {10}
$$
The inverse function of  $F_{f^{*}|s}^{-1}(t)$  can be obtained by equating the right-hand side to  $p$  and solving the resulting equation for  $t$ , which leads to
$$
F _ {f ^ {*} \mid s} ^ {- 1} (p) = \sigma_ {X} \| \beta_ {s} ^ {*} \| \Phi^ {- 1} (p) + \langle \beta_ {s} ^ {*}, \mu_ {s} \rangle . \tag {11}
$$
By substituting Eqs (10) and (11) into Eq (9) in Theorem 9, we obtain the desired claim.
D Details of Fairness Analysis
In this section, we provide evidence of the guarantee of our estimator's fairness consistency. Specifically, we present the following theorem.
Theorem 10. For any  $\delta \in (0,1]$ , the regressor in Eq (6) is  $(1/2, \delta)$ -consistently fair.
We prove the above claim by utilizing Theorem 2, which is shown in the main body, as follows:
Proof of Theorem 10. We can confirm the claim by comparing the bound obtained in Theorem 2 with the definition of  $(\frac{1}{2},\delta)$ -consistent fairness in Definition 2. In particular, we can set  $C = 4B\sigma_X\sqrt{\frac{48\ln(M / \delta)}{\min_{s\in[M]}p_s}}$  and  $n_0 = n\geq 48\ln (M / \delta) / \min_s p_s$  to satisfy the definition of  $(\frac{1}{2},\delta)$ -consistent fairness.
Next, we provide the proof of Theorem 2. To this end, we prove the following two theorems:
Theorem 11. Let  $\hat{f}_n$  be the estimator of  $f_{\mathrm{DP}}^*$  defined in Eq (6). Then, almost surely, we have
$$
W _ {2} \left(\nu_ {\hat {f} _ {n} | s}, \nu_ {\hat {f} _ {f n} | s ^ {\prime}}\right) \leq 2 B \left(\left| \left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle \right| \vee \left| \left\langle \tilde {\beta} _ {s ^ {\prime}}, \mu_ {s ^ {\prime}} - \hat {\mu} _ {s ^ {\prime}} \right\rangle \right|\right).
$$
Theorem 12. If  $n \geq (48\ln(M / \delta) - 36d) / \min_s p_s$ , we have for  $\delta \in (0,1)$
$$
\mathbb {P} \left\{\exists s \in [ M ], \left| \left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle \right| > \sigma_ {X} \sqrt {\frac {4 8 \ln (M / \delta)}{\min _ {s} n p _ {s} + 3 6 d}} \right\} \leq \delta .
$$
Combining Theorems 11 and 12 immediately yields Theorem 2.
Proof of Theorem 11. This proof investigates the distribution of  $\nu_{\hat{f}_n|s}$ . It is straightforward to verify that, conditioned on  $S = s$ ,  $\hat{f}_n(X,S)$  follows the Gaussian distribution with mean
$$
\widehat {\lVert \boldsymbol {\beta} . \rVert} \Big \langle \tilde {\boldsymbol {\beta}} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \Big \rangle + \sum_ {s ^ {\prime} \in [ M ]} \hat {p} _ {s ^ {\prime}} \Big \langle \hat {\beta} _ {s ^ {\prime}} ^ {\prime}, \hat {\mu} _ {s ^ {\prime}} ^ {\prime} \Big \rangle ,
$$
and variance
$$
\sigma_ {X} ^ {2} \widehat {\left\| \beta . \right\|} ^ {2}.
$$
We can thus evaluate the Wasserstein distance between the distributions  $\nu_{\hat{f} n|s}$  and  $\nu \hat{f}n|s'$  using the Wasserstein distance between Gaussian distributions. Given two Gaussian distributions  $N(\mu, \sigma^2)$  and  $N(\mu', \sigma'^2)$ , the 2-Wasserstein distance between them are obtained [20] as
$$
W _ {2} ^ {2} (N (\mu , \sigma^ {2}), N (\mu^ {\prime}, \sigma^ {2})) = (\mu - \mu^ {\prime}) ^ {2} + (\sigma - \sigma^ {\prime}) ^ {2}.
$$
$$
\begin{array}{l} W _ {2} ^ {2} (\nu_ {\hat {f} _ {n} | s}, \nu_ {\hat {f} _ {n} | s ^ {\prime}}) = \widehat {| | \beta . | |} ^ {2} \Big (\left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle - \left\langle \tilde {\beta} _ {s ^ {\prime}}, \mu_ {s ^ {\prime}} - \hat {\mu} _ {s ^ {\prime}} \right\rangle \Big) ^ {2} \\ \leq 4 B ^ {2} \left(\left| \left\langle \tilde {\beta} _ {s}, \mu - \hat {\mu} \right\rangle \right| \vee \left| \left\langle \tilde {\beta} _ {s ^ {\prime}}, \mu - \hat {\mu} \right\rangle \right|\right) ^ {2}, \\ \end{array}
$$
which concludes the claim.
Proof of Theorem 12. We start by deriving the concentration inequality for  $\langle \tilde{\beta}_s, \mu_s - \hat{\mu}_s \rangle$  conditioned on  $\tilde{\beta}_s$  and  $n$ . Note that  $\tilde{\beta}_s = 0$  if  $n_s \leq 18d$ . Conditioning on  $\tilde{\beta}_s$  and  $n$ , we observe that  $\langle \tilde{\beta}_s, \mu_s - \hat{\mu}_s \rangle$  follows a Gaussian distribution with mean zero and variance  $\sigma_X^2 / n_{3,s}$ . Therefore, for any  $s \in [M]$  and  $t > 0$ ,
$$
\mathbb {P} \Big \{\left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle > t \Big | \tilde {\beta} _ {s}, n. \Big \} \leq \mathbb {1} \{n _ {s} > 1 8 d \} \exp \biggl (- \frac {n _ {3 , s} t ^ {2}}{2 \sigma_ {X} ^ {2}} \biggr).
$$
Taking the expectation with respect to  $\tilde{\beta}_s$  and using the fact that  $n_{3,s} \geq \lfloor n_s / 3 \rfloor \geq n_s / 6$  for  $n_s \geq 6$ , we obtain the following inequality for  $s \in [M]$  and  $t > 0$ :
$$
\mathbb {P} \Bigl \{\Bigl \langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \Bigr \rangle > t \Big | n. \Bigr \} \leq \mathbb {1} \{n _ {s} > 1 8 d \} \exp \biggl (- \frac {n _ {s} t ^ {2}}{1 2 \sigma_ {X} ^ {2}} \biggr).
$$
Using the union bound for  $t > 0$ , we have
$$
\mathbb {P} \left\{\exists s \in [ M ], \left\langle \tilde {\beta} _ {s}, \mu_ {s} - \hat {\mu} _ {s} \right\rangle > t \mid n. \right\} \leq \sum_ {s \in [ M ]} \mathbb {1} \left\{n _ {s} > 1 8 d \right\} \exp \left(- \frac {n _ {s} t ^ {2}}{1 2 \sigma_ {X} ^ {2}}\right). \tag {12}
$$
We now derive a sufficient condition on  $t$  such that the expectation of the right-hand side in Eq (12) is less than  $\delta$ . First, we note that
$$
\begin{array}{l} \mathbb {1} \{n _ {s} > 1 8 d \} \exp \left(- \frac {n _ {s} t ^ {2}}{1 2 \sigma_ {X} ^ {2}}\right) \\ \leq \mathbb {1} \left\{n _ {s} > 1 8 d \right\} \exp \left(- \frac {\left(n _ {s} + 1 8 d\right) t ^ {2}}{1 2 \sigma_ {X} ^ {2}} \frac {n _ {s}}{n _ {s} + 1 8 d}\right) \\ \leq \exp \left(- \frac {\left(n _ {s} + 1 8 d\right) t ^ {2}}{2 4 \sigma_ {X} ^ {2}}\right) \\ \end{array}
$$
Taking the expectation and substituting the moment-generating function of the binomial distribution, we obtain
$$
\begin{array}{l} \mathbf {E} \Bigg [ \exp \bigg (- \frac {(n _ {s} + 1 8 d) t ^ {2}}{2 4 \sigma_ {X} ^ {2}} \bigg) \Bigg ] \\ \leq \exp \left(- \frac {1 8 d t ^ {2}}{2 4 \sigma_ {X} ^ {2}}\right) \left(1 - p _ {s} + p _ {s} e ^ {- \frac {t ^ {2}}{2 4 \sigma_ {X} ^ {2}}}\right) ^ {n} \\ \leq \exp \biggl (- \frac {1 8 d t ^ {2}}{2 4 \sigma_ {X} ^ {2}} - n p _ {s} \biggl (1 - e ^ {- \frac {t ^ {2}}{2 4 \sigma_ {X} ^ {2}}} \biggr) \biggr). \\ \end{array}
$$
Since  $1 - e^{-x} \geq (1 - e^{-1})x$  for  $x \in [0,1]$ , if  $t^2 / 24\sigma_X^2 \leq 1$ , we have
$$
\begin{array}{l} \mathbf {E} \left[ \exp \left(- \frac {\left(n _ {s} + 1 8 d\right) t ^ {2}}{2 4 \sigma_ {X} ^ {2}}\right) \right] \\ \leq \exp \left(- \frac {1 8 d t ^ {2}}{2 4 \sigma_ {X} ^ {2}} - \frac {(1 - e ^ {- 1}) n p _ {s} t ^ {2}}{2 4 \sigma_ {X} ^ {2}}\right). \\ \end{array}
$$
Hence,  $\mathbf{E}[\exp (-\frac{(n_s + 18d)t^2}{24\sigma_X^2})]\leq \delta /M$  if  $t\geq \sigma_X\sqrt{\frac{24\ln(M / \delta)}{(1 - e^{-1})np_s + 18d}}\leq \sigma_X\sqrt{\frac{48\ln(M / \delta)}{\min_snp_s + 36d}}$  because  $(1 - e^{-1})\geq 1 / 2.$  To ensure  $t^2 /24\sigma_X^2\leq 1$  , we require  $n\geq (48\ln (M / \delta) - 36d) / \min_{s}p_{s}$
E Proofs for Norm and Direction Estimators
This section presents the proofs for Theorem 4 and Theorem 5. Our strategy for proving these theorems is to use the hyperellipsoid to interpret the distribution of the OLS estimator. Specifically, we begin by defining  $\Sigma_{n} = \frac{1}{n} X^{\top}X$  and expressing the OLS estimator  $\hat{\beta}$  as
$$
\hat {\beta} = \beta^ {*} + \Sigma_ {n} ^ {- 1} \left(\frac {1}{n} X ^ {\top} \xi\right), \tag {13}
$$
where  $\xi$  follows a zero-mean Gaussian distribution. Eq (13) shows that, conditioned on  $X$ ,  $\hat{\beta}$  follows a multivariate Gaussian distribution with mean  $\beta^{*}$  and covariance matrix  $\frac{\sigma_{\xi}^{2}}{n}\Sigma_{n}^{-1}$ . We establish that, under the condition  $\| \hat{\beta}\| = r$ ,  $\hat{\beta}$  is supported on a hyperellipsoid  $E(r,\beta ,\frac{n}{\sigma_{\xi}^{2}}\Sigma_{n})$ , where  $E(r,c,A) = \{x\in \mathbb{R}^d:(x - c)^\top A(x - c)\leq r\}$  denotes the hyperellipsoid with  $r > 0$ ,  $c\in \mathbb{R}^d$ , and a symmetric and positive-definite matrix  $A\in \mathbb{R}^{d\times d}$ .
To prove Theorem 4 and Theorem 5, we adopt the following strategy. First, we provide an approximation of the hyperellipsoid  $E(r,c,A)$  using the maximum eigenvalue of  $A^{-1}$ , i.e.,  $\lambda_{\max}(A^{-1})$ . In our context,  $A = \frac{n}{\sigma_{\xi}^{2}} \Sigma_{n}$ , and we then focus on the concentration inequalities regarding  $\lambda_{\max}(\Sigma_n^{-1})$ . Finally, we combine these tools to prove both theorems.
Lemmas regarding hyperellipsoid. We present two lemmas that relate to the approximation of the hyperellipsoid  $E(r, c, A)$ . Specifically, we demonstrate the following two lemmas:
Lemma 2. For  $r > 0$ ,  $c \in \mathbb{R}^d$ , and a symmetric and positive-definite matrix  $A \in \mathbb{R}^{d \times d}$ , we have  $E(r, c, A) \subseteq E(r\lambda_{\max}(A^{-1}), c, I)$ .
Lemma 3. For  $r > 0$ ,  $c \in \mathbb{R}^d$ , and a symmetric and positive-definite matrix  $A \in \mathbb{R}^{d \times d}$ , if  $r\lambda_{\max}(A^{-1}) \leq \|c\|^2$ , we have
$$
\inf  _ {x \in E (r, c, A)} \left\langle \frac {c}{\| c \|}, \frac {x}{\| x \|} \right\rangle \geq \sqrt {1 - \frac {r}{\| c \| ^ {2}} \lambda_ {\max } (A ^ {- 1})}.
$$
These lemmas provide insight into the approximation of the hyperellipsoid  $E(r, c, A)$  for a given positive value of  $r$ , vector  $c$  in  $\mathbb{R}^d$ , and positive-definite symmetric matrix  $A$  in  $\mathbb{R}^{d \times d}$ . Lemma 2 states that the hyperellipsoid  $E(r, c, A)$  is contained within a hyperellipsoid  $E(r\lambda_{\max}(A^{-1}), c, I)$ . Lemma 3 shows that, under certain conditions, the minimum angle between a point in  $E(r, c, A)$  and the vector  $c$  is bounded below by a quantity that depends on  $r, c$ , and  $A$ .
Proof of Lemma 2. It is trivial that  $A - \lambda_{\min}(A)I$  is positive semi-definite. Equivalently, we have for any  $x \in \mathbb{R}^d$ ,
$$
\begin{array}{l} x ^ {\top} (A - \lambda_ {\min } (A) I) x \geq 0 \\ \Longleftrightarrow x ^ {\top} A x \geq x ^ {\top} \lambda_ {\min } (A) I x. \tag {14} \\ \end{array}
$$
From Eq (14), for any  $x \in E(r, c, A)$ , we have
$$
x ^ {\top} \lambda_ {\min } (A) I x \leq x ^ {\top} A x \leq r.
$$
Hence, for any  $x \in E(r, c, A)$ , we have
$$
x ^ {\top} I x \leq \frac {r}{\lambda_ {\operatorname* {m i n}} (A)} = \lambda_ {\max } (A ^ {- 1}) r,
$$
which indicates  $x \in E(r\lambda_{\max}(A^{-1}), c, I)$ .
Proof of Lemma 3. Let  $\bar{c} = c / \|c\|$ , and define a set  $\bar{E}(r, c, A) = \{x \in \mathbb{S}d - 1 : \exists \gamma > 0, \gamma x \in E(r, c, A)\}$ . Then,  $x \in \bar{E}(r, c, A)$  if and only if
$$
\inf  _ {\gamma > 0} (\gamma x - c) ^ {\top} A (\gamma x - c) \leq r. \tag {15}
$$
We can rewrite the left-hand side of Eq (15) as
$$
\begin{array}{l} \gamma^ {2} \langle x, A x \rangle - 2 \gamma \langle c, A x \rangle + \langle c, A c \rangle \\ = \langle x, A x \rangle \left(\gamma - \frac {\langle c , A x \rangle}{\langle x , A x \rangle}\right) ^ {2} + \langle c, A c \rangle - \frac {\langle x , A c \rangle^ {2}}{\langle x , A x \rangle}. \\ \end{array}
$$
$$
\inf  _ {\gamma > 0} (\gamma x - c) ^ {\top} A (\gamma x - c) = \langle c, A c \rangle - \frac {(\langle x , A c \rangle \lor 0) ^ {2}}{\langle x , A x \rangle}.
$$
Consequently,  $x\in \bar{E} (r,c,A)$  if and only if
$$
\langle x, A x \rangle \left(\langle \bar {c}, A \bar {c} \rangle - \frac {r}{\| c \| ^ {2}}\right) \leq \left(\langle x, A \bar {c} \rangle \vee 0\right) ^ {2}. \tag {16}
$$
From Lemma 2, we have
$$
\begin{array}{l} \inf  _ {x \in E (r, c, A)} \left\langle \bar {c}, \frac {x}{\| x \|} \right\rangle \geq \inf  _ {x \in E (\lambda_ {\max } (A ^ {- 1}) r, c, I)} \left\langle \bar {c}, \frac {x}{\| x \|} \right\rangle \\ = \inf  _ {x \in \bar {E} \left(\lambda_ {\max } \left(A ^ {- 1}\right) r, c, I\right)} \langle \bar {c}, x \rangle . \tag {17} \\ \end{array}
$$
By Eq (16),  $x \in \bar{E}(\lambda_{\max}(A^{-1})r, c, I)$  if and only if
$$
1 - \frac {r}{\| c \| ^ {2}} \lambda_ {\max } \left(A ^ {- 1}\right) \leq \left(\langle x, \bar {c} \rangle \vee 0\right) ^ {2}. \tag {18}
$$
Combining Eqs (17) and (18) and the assumption yields the claim.
Least eigenvalue of the empirical covariance matrix. The previous lemmas, Lemmas 2 and 3, provide valuable insight into analyzing the randomness regarding  $\xi$ . However, to account for the randomness of  $X$ , we must also control the lower bound on the least eigenvalue of  $A$  in Lemmas 2 and 3, which corresponds to the least eigenvalue of  $\frac{1}{n} X^{\top} X$  in our context. To this end, we leverage the high probability bound presented by Mourtada [15] based on the small-ball condition. We state the following probabilistic bound and expectation bound.
Lemma 4. For  $\mu \in \mathbb{R}^d$  and  $\sigma_X^2 >0$ , let  $X_{1},\ldots ,X_{n}\stackrel {\mathrm{iid}}{\sim}N(\mu ,\sigma_X^2 I)$ , and let  $X = (X_{1}\dots X_{n})^{\top}$ . Then, for  $n > 6d$ , we have
$$
\mathbb {P} \left\{\lambda_ {\min } \left(\frac {1}{n} X ^ {\top} X\right) <   t \right\} \leq \left(\frac {2 1 e ^ {1 0}}{\sigma_ {X} ^ {2}} t\right) ^ {n / 6}.
$$
Lemma 5. For  $\mu \in \mathbb{R}^d$  and  $\sigma_X^2 >0$ , let  $X_{1},\ldots ,X_{n}\stackrel {\mathrm{iid}}{\sim}N(\mu ,\sigma_X^2 I)$ , and let  $X = (X_{1}\dots X_{n})^{\top}$ . Then, for  $n > 6d$ , we have
$$
\mathbf {E} \left[ \lambda_ {\max } \left(\left(\frac {1}{n} X ^ {\top} X\right) ^ {- 1}\right) \right] \leq \frac {2 1 e ^ {1 0}}{\sigma_ {X} ^ {2}} \left(1 + \frac {6}{n - 6}\right).
$$
To prove Lemma 4, we utilize Corollary 3 in Mourtada [15]. Specifically, we use the following theorem.
Theorem 13 (Corollary 3 in Mourtada [15]). Let  $X$  be a random vector in  $\mathbb{R}^d$  such that  $\mathbf{E}[\|X\|^2] < +\infty$ , and let  $\Sigma = \mathbf{E}[XX^\top]$ . Let  $\hat{\Sigma}_n = \frac{1}{n}\sum_{i=1}^{n}X_iX_i^\top$ , where  $X_i$  are i.i.d. copies of  $X$ . Given  $C > 0$  and  $\alpha \in (0,1]$ , assume that for every  $\theta \in \mathbb{R}^d \setminus \{0\}$  and  $t > 0$ ,
$$
\mathbb {P} \left\{\langle \theta , X \rangle^ {2} \leq t ^ {2} \left\| \Sigma^ {1 / 2} \theta \right\| ^ {2} \right\} \leq (C t) ^ {\alpha}. \tag {19}
$$
Then, if  $d / n\leq \alpha /6$  ,for every  $t > 0$
$$
\hat {\Sigma} _ {n} \succeq t \Sigma
$$
with probability at least  $1 - (C^{\prime}t)^{\alpha n / 6}$ , where  $C^\prime = 3C^4 e^{1 + 9 / \alpha}$ .
Eq (19) is known as the small-ball condition.
Proof of Lemma 4. To take an advantage of Theorem 13, we need to ensure that  $X_{i}$  satisfies the small-ball condition in Eq (19). Let  $\Sigma_{n} = \frac{1}{n} X^{\top}X$ . Then, the expected value of  $\Sigma_{n}$  is equal to  $\sigma_X^2 I + \mu \mu^\top \coloneqq \Sigma$ , i.e.,  $\mathbf{E}[\Sigma_n] = \Sigma$ . Given  $\theta \in \mathbb{R}^{d} \setminus \{0\}$ ,  $\langle \theta, X_i \rangle^2 / \sigma_X^2 \| \theta \|^2$  follows the non-central  $\chi^2$  distribution with degree of freedom 1 and non-centrality parameter  $\langle \theta, \mu \rangle^2 / \sigma_X^2 \| \theta \|^2$ . Consequently, we verify the satisfaction of the small-ball condition of  $X_{i}$  by confirming that for a random variable  $Z$  following the non-central  $\chi^2$  distribution with degree of freedom 1 and non-centrality parameter  $\lambda^2$ , there exists  $C$  and  $\alpha \in (0,1]$  such that
$$
\mathbb {P} \{Z \leq t ^ {2} \} \leq (C t) ^ {\alpha}.
$$
The cumulative distribution function of the non-central  $\chi^2$  distribution with degree of freedom 1 has a closed-form using the error function (See [12] and references therein). Specifically, letting  $\operatorname{erf}(z)$  be the error function, defined as
$$
\operatorname {e r f} (z) = \frac {2}{\sqrt {\pi}} \int_ {0} ^ {z} e ^ {- x ^ {2}} d x,
$$
the cumulative distribution function of  $Z$  is obtained as
$$
\mathbb {P} \{Z \leq t ^ {2} \} = \frac {1}{2} \left(\operatorname {e r f} \left(\frac {t - \lambda}{\sqrt {2}}\right) + \operatorname {e r f} \left(\frac {t + \lambda}{\sqrt {2}}\right)\right).
$$
Since  $e^{-x^2}$  is an even function, we have
$$
\mathbb {P} \{Z \leq t ^ {2} \} = \frac {1}{\sqrt {2 \pi}} \left(\int_ {0} ^ {\lambda + t} e ^ {- x ^ {2} / 2} d x + \int_ {0} ^ {t - \lambda} e ^ {- x ^ {2} / 2} d x\right)
$$
$$
\begin{array}{l} = \frac {1}{\sqrt {2 \pi}} \left(\int_ {0} ^ {\lambda + t} e ^ {- x ^ {2} / 2} d x + \int_ {\lambda - t} ^ {0} e ^ {- x ^ {2} / 2} d x\right) \\ = \frac {1}{\sqrt {2 \pi}} \left(\int_ {0} ^ {\lambda + t} e ^ {- x ^ {2} / 2} d x - \int_ {0} ^ {\lambda - t} e ^ {- x ^ {2} / 2} d x\right) \\ = \frac {1}{\sqrt {2 \pi}} \int_ {\lambda - t} ^ {\lambda + t} e ^ {- x ^ {2} / 2} d x. \\ \end{array}
$$
Noting that  $e^{-x^2 / 2} \leq e^{-\frac{(0 \vee (\lambda - t))^2}{2}}$  for  $x \in (\lambda - t, \lambda + t)$ , we have
$$
\mathbb {P} \{Z \leq t ^ {2} \} \leq \frac {1}{\sqrt {2 \pi}} \int_ {\lambda - t} ^ {\lambda + t} d x = \sqrt {\frac {2}{\pi}} e ^ {- \frac {(0 \vee (\lambda - t)) ^ {2}}{2}} t. \tag {20}
$$