SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Therefore,  $2\exp \left(-t^{2} / 2 + 2\ell^{2}\right)\leq 2\exp \left(-t^{2} / (10\ell^{2})\right)$  for  $t^2\geq 5\ell^2$  . On the other hand,
$$
2 \exp \left(- t ^ {2} / \left(1 0 \ell^ {2}\right)\right) > 2 e ^ {- 1 / 2} > 1, \quad \text {w h e n} t ^ {2} <   5 \ell^ {2}.
$$
Thus, in this case the following holds for all  $t > 0$ :
$$
\Pr \left[ | \tilde {X} | > t \right] \leq 2 \exp \left(- t ^ {2} / \left(1 0 \ell^ {2}\right)\right) \tag {27}
$$
The above results also apply to "complements" of the thresholded Gaussians. In particular, let  $\overline{\mathcal{D}}_{\ell}$  be the distribution of  $\{X \sim N(0,1) \mid X \leq \ell \Leftrightarrow -X \geq -\ell\}$  which is equivalently  $\{-X \sim N(0,1) \mid X \geq -\ell\}$  to which the above analysis can be directly applied. This yields, that if  $\tilde{X} \sim \overline{\mathcal{D}}_{\ell}$  then for any  $t > 0$
$$
\Pr \left[ | \tilde {X} | > t \right] \leq 2 \cdot \exp (- t ^ {2} / 2), \quad \text {i f} \ell \geq 0, \tag {28}
$$
$$
\Pr \left[ | \tilde {X} | > t \right] \leq 2 \cdot \exp \left(- t ^ {2} / K _ {1} ^ {2}\right), \quad \text {i f} \ell <   0, \tag {29}
$$
where  $K_{1} = \max \{\sqrt{20},|\ell |\sqrt{10}\}$
E.1 Bag mean and covariance estimation error bounds
In this section we shall be concerned with random variables  $\tilde{X}$  which are sampled from  $\mathcal{D}_{\ell}$  with probability  $p$  and from  $\overline{\mathcal{D}}_{\ell}$  with probability  $(1 - p)$ . Let us denote this distribution by  $\hat{\mathcal{D}}(p, \ell)$ . Using Lemmas E.3, E.4 and (28), (29), we obtain the following lemma.
Lemma E.5. Let  $\tilde{X} \sim \hat{\mathcal{D}}(p, \ell)$  for some  $p \in (0,1)$ . Then for any  $t > 0$ ,
$$
\Pr \left[ | \tilde {X} | > t \right] \leq 2 \cdot \exp \left(- t ^ {2} / K _ {1} ^ {2}\right) \tag {30}
$$
where  $K_{1} = \max \{\sqrt{20},|\ell |\sqrt{10}\}$  . In particular,  $\| \tilde{X}\|_{\psi_2} = O(|\ell |)$
We however, shall also require similar bounds for the mean-zero version of such distributions. To begin with we state an easy lemma bounding the mean of  $\tilde{X}$ .
Lemma E.6. Let  $\tilde{X} \sim \hat{\mathcal{D}}(p, \ell)$  for some  $p \in (0,1)$ . Then,  $\left|\mathbb{E}\left[\tilde{X}\right]\right| \leq \gamma_{\ell} = \max \{\gamma_0, 2\ell\}$ , where  $\gamma_0 > 0$  is some constant.
Proof. Let us consider the case of  $\ell > 0$  (and  $\ell \leq 0$  follows analogously). When  $\ell < 2$ , it is easy to see that desired expectation is  $O(1)$ . Further, the expectation over  $\overline{\mathcal{D}}_{\ell}$  is  $O(\ell)$  for any  $\ell > 0$ , since it is a convex combination of the expectation of a half gaussian which has  $O(1)$  expectation, and a gaussian truncated from below at 0 and above at  $\ell$ , which has  $O(\ell)$  expectation. To complete the argument we need to bound the expectation over  $\mathcal{D}_{\ell}$ . Using (21) and  $(1/\sqrt{2\pi}) \int_{\ell}^{\infty} x \exp(-x^{2}/2) dx = (1/\sqrt{2\pi}) \exp(-\ell^{2}/2)$ , we obtain  $\mathbb{E}_{X \sim \mathcal{D}_{\ell}}[X] = (1/\sqrt{2\pi}) \exp(-\ell^{2}/2) \overline{\Phi}(\ell)^{-1}$  and the lower bound from Prop. E.2 along with  $\ell \geq 2$  yields an upper bound of  $2\ell$  on the expectation.
With the setup as in Lemma E.5 define  $\widehat{X} \coloneqq \tilde{X} - \mathbb{E}\left[\tilde{X}\right]$ . Clearly,  $\mathbb{E}\left[\widehat{X}\right] = 0$ . Further,
$$
\left| \hat {X} \right| > t \Rightarrow \left| \tilde {X} \right| + \left| \mathbb {E} [ \tilde {X} ] \right| > t \Rightarrow \left| \tilde {X} \right| > t - \gamma_ {\ell}.
$$
$$
\Pr \left[ | \widehat {X} | > t \right] \leq \Pr \left[ | \tilde {X} | > t - \gamma_ {\ell} \right] \leq 2 \cdot \exp \left(- (t - \gamma_ {\ell}) ^ {2} / K _ {1} ^ {2}\right) \tag {31}
$$
Let  $K_{2} = \max \{2K_{1},\sqrt{2}\gamma_{\ell}\}$ . Now,
$$
\begin{array}{l} t \geq 2 \gamma_ {\ell} \Rightarrow | t | \leq 2 | t - \gamma_ {\ell} | \Rightarrow \frac {t ^ {2}}{4 K _ {1} ^ {2}} \leq \frac {(t - \gamma_ {\ell}) ^ {2}}{K _ {1} ^ {2}} \\ \Rightarrow 2 \cdot \exp \left(- \left(t - \gamma_ {\ell}\right) ^ {2} / K _ {1} ^ {2}\right) \leq 2 \cdot \exp \left(- t ^ {2} / K _ {2} ^ {2}\right) \tag {32} \\ \end{array}
$$
On the other hand, when  $0 \leq t < 2\gamma_{\ell}$ ,  $t^{2} / K_{2}^{2} \geq 1 / 2$ , and thus
$$
2 \cdot \exp \left(- t ^ {2} / K _ {2} ^ {2}\right) \geq 2 e ^ {- 1 / 2} > 1.
$$
$$
\Pr \left[ | \widehat {X} | > t \right] \leq 2 \cdot \exp \left(- t ^ {2} / K _ {2} ^ {2}\right) \tag {33}
$$
for all  $t > 0$ , where  $K_{2} = O(|\ell |)$ .
E.1.1 Concentration of mean estimate using Hoeffding's Bound
Let us first state the Hoeffding's concentration bound for subgaussian random variables.
Theorem E.7 (Theorem 2.6.2 of [32]). Let  $X_{1},\ldots ,X_{N}$  be independent, mean-zero, sub-gaussian random variables. Then, for every  $\varepsilon \geq 0$
$$
\Pr \left[ \left| \frac {\sum_ {i = 1} ^ {N} X _ {i}}{N} \right| \geq \varepsilon \right] \leq 2 \cdot \exp \left(\frac {- c \varepsilon^ {2} N ^ {2}}{\sum_ {i = 1} ^ {N} \| X _ {i} \| _ {\psi_ {2}} ^ {2}}\right), \tag {34}
$$
where  $c > 0$  is some absolute constant.
For the rest of this section we shall fix  $\ell \in \mathbb{R}$  and  $p\in (0,1)$ . Consider vector valued random variable  $\mathbf{X} = (X^{(1)},\ldots ,X^{(d)})$  with independent coordinates where
-  $X^{(1)} = \tilde{X} -\mathbb{E}[\tilde{X} ]$  where  $\tilde{X}\sim \hat{\mathcal{D}} (p,\ell)$  . From the previous subsection, we have  $\| X^{(1)}\|_{\psi_2} = O(|\ell |)$
- For  $i = 2, \dots, d$ ,  $X^{(i)} \sim N(0,1)$  and therefore  $\| X^{(i)}\|_{\psi_2} = O(1)$ .
Using the above bounds on the subgaussian norms, and applying Theorem E.7 to bound the error in each coordinate by  $\varepsilon/\sqrt{d}$  and taking a union bound we obtain the following lemma.
Lemma E.8. Let  $\mathbf{X}_1, \ldots, \mathbf{X}_N$  be  $N$  iid samples of  $\mathbf{X}$ . Then for every  $\varepsilon \geq 0$ ,
$$
\Pr \left[ \left\| \frac {\sum_ {i = 1} ^ {N} \mathbf {X} _ {i}}{N} \right\| _ {2} \geq \varepsilon \right] \leq 2 \cdot \exp \left(\frac {- c _ {0} \varepsilon^ {2} N}{d O (\ell^ {2})}\right) + 2 (d - 1) \cdot \exp \left(- c _ {0} \varepsilon^ {2} N / d\right) \tag {35}
$$
for some absolute constant  $c_0 > 0$ . In particular, if  $N > O\left((d / \varepsilon^2)O(\ell^2)\log (d / \delta)\right)$ ,
$$
\operatorname * {P r} \left[ \left\| \frac {\sum_ {i = 1} ^ {N} \mathbf {X} _ {i}}{N} \right\| _ {2} \geq \varepsilon \right] \leq \delta ,
$$
for any  $\delta > 0$ .
E.1.2 Concentration of covariance estimate
Consider the vector random variable  $\mathbf{X}$  defined in the previous subsection. It is mean-zero and so by Defn. 3.4.1 and Lemma 3.4.2 of [32],
$$
\sup  _ {\mathbf {x} \in S ^ {d - 1}} \| \langle \mathbf {x}, \mathbf {X} \rangle \| _ {\psi_ {2}} = O (\ell), \tag {36}
$$
using the bounds on the subgaussian norms of the coordinates of  $\mathbf{X}$  given in the previous subsection. Using this we can directly apply Proposition 2.1 of [31] to obtain the following lemma.
Lemma E.9. Let  $\mathbf{X}_1, \ldots, \mathbf{X}_N$  be  $N$  iid samples of  $\mathbf{X}$ , then if  $N > O\left((d / \varepsilon^2)O(\ell^2)\log (1 / \delta)\right)$ ,
$$
\Pr \left[ \left\| \frac {\sum_ {i = 1} ^ {N} \mathbf {X} _ {i} \mathbf {X} _ {i} ^ {\mathsf {T}}}{N} - \mathbb {E} \left[ \mathbf {X X} ^ {\mathsf {T}} \right] \right\| _ {2} \geq \varepsilon \right] \leq \delta , \tag {37}
$$
for any  $\varepsilon, \delta > 0$ .
E.1.3 Mean and covariance estimate bounds for non-centered vector r.v.s
Distribution  $\mathcal{D}_{\mathrm{asymvec}}(p,\ell)$ . We revisit the definition of  $\mathbf{X}$  in Sec. E.1.1 and instead define distribution  $\mathcal{D}_{\mathrm{asymvec}}(p,\ell)$  over  $\mathbf{Z} = (Z^{(1)},\ldots,Z^{(d)})$  with independent coordinates by taking  $Z^{(1)} = \tilde{X}$  where  $\tilde{X} \sim \hat{\mathcal{D}}(p,\ell)$  and for  $i = 2,\dots,d$ ,  $Z^{(i)} \sim N(0,1)$ .
Clearly,  $\mathbf{X} = \mathbf{Z} - \mathbb{E}[\mathbf{Z}]$ . For convenience, we shall use the following notation:
$$
\boldsymbol {\mu} _ {Z} := \mathbb {E} [ \mathbf {Z} ] \quad \text {a n d ,} \quad \hat {\boldsymbol {\mu}} _ {Z} := \frac {\sum_ {i = 1} ^ {N} \mathbf {Z} _ {i}}{N}, \tag {38}
$$
$$
\boldsymbol {\Sigma} _ {Z} := \mathbb {E} \left[ \left(\mathbf {Z} - \boldsymbol {\mu} _ {Z}\right) \left(\mathbf {Z} - \boldsymbol {\mu} _ {Z}\right) ^ {\top} \right] \quad \text {a n d ,} \quad \hat {\boldsymbol {\Sigma}} _ {Z} := \frac {\sum_ {i = 1} ^ {N} \left(\mathbf {Z} _ {i} - \hat {\boldsymbol {\mu}} _ {Z}\right) \left(\mathbf {Z} _ {i} - \hat {\boldsymbol {\mu}} _ {Z}\right) ^ {\top}}{N}, \tag {39}
$$
where  $\mathbf{Z}_i$  is an iid sample of  $\mathbf{Z}$  and  $\mathbf{X}_i = \mathbf{Z}_i - \boldsymbol{\mu}_Z$ , for  $i = 1, \dots, N$ . We have the following lemma.
Lemma E.10. For any  $\varepsilon, \delta \in (0,1)$ , if  $N > O\left((d / \varepsilon^2)O(\ell^2)\log (d / \delta)\right)$ , then w.p.  $1 - \delta$  the following hold simultaneously,
$$
\left\| \hat {\boldsymbol {\mu}} _ {Z} - \boldsymbol {\mu} _ {Z} \right\| _ {2} \leq \varepsilon / 2, \tag {40}
$$
$$
\left\| \dot {\boldsymbol {\Sigma}} _ {Z} - \boldsymbol {\Sigma} _ {Z} \right\| _ {2} \leq \varepsilon \tag {41}
$$
Proof. We begin by applying Lemmas E.8 and E.9 to  $\mathbf{X}$  and the iid samples  $\mathbf{X}_1, \ldots, \mathbf{X}_N$  so that their conditions hold with  $\varepsilon / 2$  and  $\delta / 2$ . Taking a union bound we get that the following simultaneously hold with probability at least  $1 - \delta$ :
$$
\left\| \frac {\sum_ {i = 1} ^ {N} \mathbf {X} _ {i}}{N} \right\| _ {2} \leq \varepsilon / 2, \tag {42}
$$
$$
\left\| \frac {\sum_ {i = 1} ^ {N} \mathbf {X} _ {i} \mathbf {X} _ {i} ^ {\mathsf {T}}}{N} - \mathbb {E} \left[ \mathbf {X X} ^ {\mathsf {T}} \right] \right\| _ {2} \leq \varepsilon / 2. \tag {43}
$$
By the definitions above, (42) directly implies (40).
Now, observe that  $\boldsymbol{\Sigma}_{Z} = \mathbb{E}[\mathbf{X}\mathbf{X}^{\top}]$ . On the other hand, letting  $\zeta \coloneqq \hat{\pmb{\mu}}_Z - \pmb{\mu}_Z = (\sum_{i=1}^{N} \mathbf{X}_i) / N$  we simplify  $\hat{\Sigma}_{Z}$  as,
$$
\begin{array}{l} \frac {\sum_ {i = 1} ^ {N} \left(\mathbf {Z} _ {i} - \hat {\boldsymbol {\mu}} _ {Z}\right) \left(\mathbf {Z} _ {i} - \hat {\boldsymbol {\mu}} _ {Z}\right) ^ {\mathsf {T}}}{N} = \frac {\sum_ {i = 1} ^ {N} \left(\mathbf {X} _ {i} - \boldsymbol {\zeta}\right) \left(\mathbf {X} _ {i} - \boldsymbol {\zeta}\right) ^ {\mathsf {T}}}{N} \\ = \frac {\sum_ {i = 1} ^ {N} \mathbf {X} _ {i} \mathbf {X} _ {i} ^ {\top} - \mathbf {X} _ {i} \boldsymbol {\zeta} ^ {\top} - \boldsymbol {\zeta} \mathbf {X} _ {i} ^ {\top} + \boldsymbol {\zeta} \boldsymbol {\zeta} ^ {\top}}{N} \\ = \frac {\sum_ {i = 1} ^ {N} \mathbf {X} _ {i} \mathbf {X} _ {i} ^ {\mathsf {T}}}{N} - 2 \zeta \zeta^ {\mathsf {T}} + \zeta \zeta^ {\mathsf {T}} \\ = \frac {\sum_ {i = 1} ^ {N} \mathbf {X} _ {i} \mathbf {X} _ {i} ^ {\intercal}}{N} - \zeta \zeta^ {\intercal} \tag {44} \\ \end{array}
$$
Thus, the LHS of (41) is at most,
$$
\begin{array}{l} \left\| \frac {\sum_ {i = 1} ^ {N} \mathbf {X} _ {i} \mathbf {X} _ {i} ^ {\mathsf {T}}}{N} - \mathbb {E} [ \mathbf {X X} ^ {\mathsf {T}} ] - \boldsymbol {\zeta} \boldsymbol {\zeta} ^ {\mathsf {T}} \right\| _ {2} \leq \left\| \frac {\sum_ {i = 1} ^ {N} \mathbf {X} _ {i} \mathbf {X} _ {i} ^ {\mathsf {T}}}{N} - \mathbb {E} [ \mathbf {X X} ^ {\mathsf {T}} ] \right\| _ {2} + \| \boldsymbol {\zeta} \boldsymbol {\zeta} ^ {\mathsf {T}} \| _ {2} \\ \leq \varepsilon / 2 + \varepsilon^ {2} / 4 \leq \varepsilon , \tag {45} \\ \end{array}
$$
since we have shown that  $\| \boldsymbol {\zeta}\| _2 = \| \hat{\pmb{\mu}}_Z - \pmb {\mu}_Z\| _2\leq \varepsilon /2$
Finally, we prove a version of the above lemma under a symmetric psd transformation. Let  $\mathbf{A}$  be a psd matrix s.t.  $\lambda_{\mathrm{max}}$  is the maximum eigenvalue of  $\mathbf{A}^2 = \mathbf{A}\mathbf{A}$  i.e.,  $\sqrt{\lambda_{\mathrm{max}}}$  is the maximum eigenvalue of  $\mathbf{A}$ . Then, if we define  $\tilde{\mathbf{Z}} := \mathbf{A}\mathbf{Z}$  and  $\tilde{\mathbf{Z}}_i$  as iid samples of  $\tilde{\mathbf{Z}}, i = 1,\dots,N$  and analogous to (38) and (39), define  $\mu_{\tilde{Z}},\hat{\mu}_{\tilde{Z}},\Sigma_{\tilde{Z}}$  and  $\hat{\Sigma}_{\tilde{Z}}$ , we have the following lemma which follows directly from Lemma E.10 the  $\sqrt{\lambda_{\mathrm{max}}}$  upper bound on the operator norm of  $\mathbf{A}$ .
Lemma E.11. For any  $\varepsilon, \delta \in (0,1)$ , if  $N > O\left((d / \varepsilon^2)O(\ell^2)\log (d / \delta)\right)$ , then w.p.  $1 - \delta$  the following hold simultaneously,
$$
\left\| \hat {\boldsymbol {\mu}} _ {\tilde {Z}} - \boldsymbol {\mu} _ {\tilde {Z}} \right\| _ {2} \leq \varepsilon \sqrt {\lambda_ {\max }} / 2, \tag {46}
$$
$$
\left\| \tilde {\boldsymbol {\Sigma}} _ {\tilde {Z}} - \boldsymbol {\Sigma} _ {\tilde {Z}} \right\| _ {2} \leq \varepsilon \lambda_ {\max } \tag {47}
$$
E.2 Estimating Covariance of differences
First we begin this subsection with a simple observation. If  $X$  and  $Y$  are two random variables such that,
$$
\Pr \left[ | X | > t \right], \Pr \left[ | Y | > t \right] \leq 2 \cdot \exp \left(- t ^ {2} / K _ {1} ^ {- 2}\right), \quad \forall t > 0, \tag {48}
$$
then  $X - Y$  is a random variable such that,
$$
\Pr \left[ | X - Y | > t \right] \leq \Pr \left[ | X | > t / 2 \right] + \Pr \left[ | Y | > t / 2 \right] \leq 4 \cdot \exp \left(- t ^ {2} / (2 K _ {1}) ^ {2}\right).
$$
It is easy to see that,
$$
4 \cdot \exp \left(- t ^ {2} / (2 K _ {1}) ^ {2}\right) \leq 2 \cdot \exp \left(- t ^ {2} / (4 K _ {1}) ^ {2}\right),
$$
when  $t^2 \geq 8K_1^2$ . On the other hand, when  $t^2 < 8K_1^2$ ,  $2 \cdot \exp \left(-t^{2} / (4K_{1})^{2}\right) > 1$ . Thus,
$$
\Pr \left[ | X - Y | > t \right] \leq 2 \cdot \exp \left(- t ^ {2} / \left(4 K _ {1}\right) ^ {2}\right) \tag {49}
$$
In this subsection shall consider  $\tilde{X} \sim \mathcal{D}(p, q, \ell)$  to be defined as follows, for some  $\ell \in \mathbb{R}, p, q \in [0,1)$  s.t.  $p + q < 1$ .  $\tilde{X} = U - V$  where:
with probability  $p$ ,  $U \sim \mathcal{D}_{\ell}$  and  $V \sim \mathcal{D}_{\ell}$  independently,
with probability  $q$ ,  $U \sim \overline{\mathcal{D}}_{\ell}$  and  $V \sim \overline{\mathcal{D}}_{\ell}$  independently,
with probability  $(1 - p - q) / 2$ ,  $U \sim \mathcal{D}_{\ell}$  and  $V \sim \overline{\mathcal{D}}_{\ell}$  independently,
with probability  $(1 - p - q) / 2$ ,  $U \sim \overline{\mathcal{D}}_{\ell}$  and  $V \sim \mathcal{D}_{\ell}$  independently.
From the above it is clear that  $\mathbb{E}[\tilde{X}] = 0$ . Further, from (49) and Lemmas E.3, E.4 and (28), (29),
$$
\Pr \left[ | \tilde {X} | > t \right] \leq 2 \cdot \exp \left(- t ^ {2} / K _ {1} ^ {2}\right) \tag {50}
$$
for  $t > 0$ , where  $K_{1} = \max \{4\sqrt{20}, 4|\ell| \sqrt{10}\}$ . In particular,  $\| \tilde{X} \|_{\psi_2} = O(\ell)$ .
Let us now define a distribution  $\mathcal{D}_{\mathrm{diffvec}}(p,q,\ell)$  vector valued random variable with independent coordinates  $\mathbf{X} = (X^{(1)},\ldots ,X^{(d)})$ , where
-  $X^{(1)} \sim \mathcal{D}(p, q, \ell)$ , for some  $\ell \in \mathbb{R}, p, q \in [0, 1)$  s.t.  $p + q < 1$ .
-  $\mathbf{X}^{(j)}$  is the difference of two iid  $N(0,1)$  random variables, for  $j = 2,\dots ,d$ . In particular, the subgaussian norm of these coordinates is  $O(1)$ .
From, the above it is clear that  $\mathbb{E}[\mathbf{X}] = \mathbf{0}$ , and by Defn. 3.4.1 and Lemma 3.4.2 of [32], (36) is applicable to  $\mathbf{X}$  as defined above. Thus, letting  $\hat{\mathbf{X}} := \mathbf{A}\mathbf{X}$ , where  $\mathbf{A}$  is as used in the previous subsection we have,
Lemma E.12. For  $\mathbf{X}$  defined above, the statement of Lemma E.9 is applicable, and (37) implies that the following holds:
$$
\left\| \frac {\sum_ {i = 1} ^ {N} \hat {\mathbf {X}} _ {i} \hat {\mathbf {X}} _ {i} ^ {\mathsf {T}}}{N} - \mathbb {E} \left[ \hat {\mathbf {X}} \hat {\mathbf {X}} ^ {\mathsf {T}} \right] \right\| _ {2} \leq \varepsilon \lambda_ {\max } \tag {51}
$$
E.3 Proof of Lemma 3.1
Our proof shall utilize the following normalization of LTFs in a Gaussian space.
Lemma E.13. Suppose  $f(\mathbf{X}) = \mathrm{pos}\left(\mathbf{r}^{\top}\mathbf{X} + c\right)$ , where  $\| \mathbf{r}\| _2 > 0$  and  $\mathbf{X}\sim N(\boldsymbol {\mu},\boldsymbol {\Sigma})$  s.t.  $\boldsymbol{\Sigma}$  is positive definite. Let  $\Gamma \coloneqq \Sigma^{1 / 2}$  be symmetric p.d., and  $\mathbf{U}$  be any orthonormal transformation satisfying  $\mathbf{U}\Gamma \mathbf{r} / \| \Gamma \mathbf{r}\| _2 = \mathbf{e}_1$  where  $\mathbf{e}_1$  is the vector with 1 in the first coordinate and 0 in the rest. Then, letting  $\mathbf{Z}\sim N(\mathbf{0},\mathbf{I})$  so that  $\mathbf{X} = \Gamma \mathbf{U}^{\top}\mathbf{Z} + \boldsymbol{\mu}$ ,
$$
f (\mathbf {X}) = \operatorname {p o s} \left(\mathbf {r} ^ {\mathsf {T}} \mathbf {X} + c\right) = \operatorname {p o s} \left(\mathbf {e} _ {1} ^ {\mathsf {T}} \mathbf {Z} + \ell\right) \tag {52}
$$
where  $\ell = \left(\mathbf{r}^{\mathrm{T}}\pmb {\mu} + c\right) / \| \mathbf{Tr}\|_{2}$ .
Proof. We have  $\mathbf{X} = \hat{\mathbf{X}} + \boldsymbol{\mu}$  where  $\hat{\mathbf{X}} \sim N(\mathbf{0}, \mathbf{\Sigma})$ . Thus,  $\hat{\mathbf{X}} = \Gamma \mathbf{U}^{\top} \mathbf{Z} \Rightarrow \mathbf{Z} = \mathbf{U} \Gamma^{-1} \hat{\mathbf{X}}$ , using  $\mathbf{U}^{\top} = \mathbf{U}^{-1}$ . Now,  $f(\mathbf{X})$  can be written as
$$
\begin{array}{l} \left. \operatorname {p o s} \left(\frac {\mathbf {r} ^ {\top} (\hat {\mathbf {X}} + \boldsymbol {\mu}) + c}{\| \mathbf {T r} \| _ {2}}\right) = \operatorname {p o s} \left(\frac {\mathbf {r} ^ {\top} \boldsymbol {\Gamma} \mathbf {U} ^ {\top} \mathbf {U} \boldsymbol {\Gamma} ^ {- 1} \hat {\mathbf {X}} + \mathbf {r} ^ {\top} \boldsymbol {\mu} + c}{\| \boldsymbol {\Gamma} \mathbf {r} \| _ {2}}\right) \right. \\ = \operatorname {p o s} \left(\frac {\left(\mathbf {U} \boldsymbol {\Gamma} \mathbf {r}\right) ^ {\top} \mathbf {Z}}{\| \boldsymbol {\Gamma} \mathbf {r} \| _ {2}} + \frac {\mathbf {r} ^ {\top} \boldsymbol {\mu} + c}{\| \boldsymbol {\Gamma} \mathbf {r} \| _ {2}}\right) \\ = \operatorname {p o s} \left(\mathbf {e} _ {1} ^ {\top} \mathbf {Z} + \frac {\mathbf {r} ^ {\top} \boldsymbol {\mu} + c}{\| \boldsymbol {\Gamma} \mathbf {r} \| _ {2}}\right) \tag {53} \\ \end{array}
$$
Proof. (of Lemma 3.1) Using the normalization in Lemma E.13, we can write  $\mathbf{X} = \mathbf{A}\mathbf{Z} + \boldsymbol{\mu}$  where  $\mathbf{A} = \boldsymbol{\Sigma}^{1/2}\mathbf{U}^{\top}$  such that  $\mathbf{X} \sim N(\boldsymbol{\mu}, \boldsymbol{\Sigma}) \equiv \mathbf{Z} \sim N(\mathbf{0}, \mathbf{I})$ . From the condition in (52) we can write the samples in Step 2 of Alg. 1 as  $\mathbf{x}_i = \mathbf{A}\mathbf{z}_i + \boldsymbol{\mu}$  where  $\mathbf{z}_i$  are sampled from  $\mathcal{D}_{\mathrm{asymvec}}(k/q,-\ell)$  (see Sec.
E.1.3) for  $\ell$  as given in Lemma E.13. Note that the maximum eigenvalue of  $\mathbf{A}^2$  is  $\lambda_{\mathrm{max}}$  which is the maximum eigenvalue of  $\pmb{\Sigma}$ . Thus, one can apply Lemma E.11 to  $\hat{\pmb{\mu}}_B$  and  $\hat{\pmb{\Sigma}}_B$ .
Further, the difference vectors sampled in Step 6 can be written as  $\overline{\mathbf{x}}_i = \mathbf{A}\overline{\mathbf{z}}_i$  where  $\overline{\mathbf{z}}_i$  are sampled from  $\mathcal{D}_{\mathrm{diffvec}}(p,p', - \ell)$  (see Sec. E.2) where  $p = \binom{k}{2} / \binom{q}{2}$  is the probability of sampling a pair of 1-labeled feature-vectors from a bag, and  $p' = \binom{q-k}{2} / \binom{q}{2}$  is that of sampling a pair of 0-labeled feature-vectors. Thus, one can apply Lemma E.12 to  $\hat{\mathbf{\Sigma}}_D$ .
Using both the above applications with the error probability  $\delta /2$  and using union bound we complete the proof.
F Experimental Details and Results
The implementations of the algorithms in this paper (Algs. 2, 3, 4) and of the random LTF algorithm are in python using numpy libraries. The code for the SDP algorithms of [25, 26] for bag sizes 2 and 3 is adapted from the publicly available codebase<sup>1</sup>. The experimental code was executed on a 16-core CPU and 128 GB RAM machine running linux in a standard python environment.
F.2 Experimental Results
In the following  $d$  denotes the dimension of the feature-vectors,  $q$  the size of the bags, with  $k / q \in (0,1)$  the bag label proportion, and  $m$  be the number of sampled bags in the training dataset. The instance-level test set is of size 1000 in all the experiments, and the reported metric is the accuracy over the test set.
Standard Gaussian without LTF offset. Here we primarily wish to evaluate the Algorithm 3 using unbalanced bag oracles such that  $k \neq q / 2$ . For  $d \in \{10,50\}$ ,  $(q,k) \in \{(3,1),(10,8),(50,35)\}$  and  $m \in \{100,500,2000\}$  we create 25 datasets. In each LLP dataset, we (i) sample a random unit vector  $\mathbf{r}^*$  and let  $f(\mathbf{x}) \coloneqq \mathrm{pos}\left(\mathbf{r}^{*\top}\mathbf{x}\right)$ , (ii) sample  $m$  training bags from  $\mathsf{Ex}(f,N(\mathbf{0},\mathbf{I}),q,k)$ , (iii) sample 1000 test instances  $(\mathbf{x},f(\mathbf{x}))$ ,  $\mathbf{x} \gets N(\mathbf{0},\mathbf{I})$ . We also evaluate the Algorithm 2 on these datasets. For comparison we have the random LTF algorithm (R) in which we sample 100 random LTFs and return the one that performs best on the training set. The results are reported in Table 5. We also evaluate the SDP algorithm (S) in [26] for  $(q,k) = (3,1)$  using  $m \in \{50,100,200\}$  (since the SDP algorithms do not scale to larger number of bags) whose results are reported in Table 6.
Table 2 reports a concise set of comparative scores of Algorithms 2, 3 and random LTF (R) using 2000 bags and of the SDP algorithms (S) with 200 bags.
Centered and general Gaussian. Here we evaluate Algorithms 2 and 4, and we have both balanced as well as unbalanced bag oracles. In particular, for  $d \in \{10,50\}$ ,  $(q,k) \in \{(2,1),(3,1),(10,5),(10,8),(50,25),(50,35)\}$  and  $m \in \{100,500,2000\}$  we create 25 datasets similar to the previous case, except that for each dataset we first sample  $\mu$  and  $\Sigma$  and use  $N(\mathbf{0},\Sigma)$  for sampling feature-vectors in the centered Gaussian case and use  $N(\boldsymbol{\mu},\boldsymbol{\Sigma})$  for sampling feature-vectors in the general Gaussian case. We perform the following set of experiments in each case. For the cases when bags are balanced, i.e.  $(q,k) \in \{(2,1),(10,5),(50,25)\}$ , for each our Algorithms 2 and 4 we evaluate their two possible solutions on the test data and report the better number.
- With LTF offset. We sample  $(\mathbf{r}_*, c_*)$  and create a dataset using  $\mathrm{pos}(\mathbf{r}_*^\top \mathbf{x} + c_*)$  as the labeling function. Table 11 reports the test accuracy scores for Algorithm 4 and random LTF (R) with  $m = \{100, 500, 2000\}$  for centered and general Gaussians. Table 12 reports the corresponding scores for the SDP algorithm (S) [25, 26] with  $m = \{50, 100, 200\}$  and  $(q, k) \in \{(2, 1), (3, 1)\}$ . Table 4 provides concise comparative scores with  $m = 2000$  for Algorithm 4 and random LTF and  $m = 200$  for the SDP algorithm (S).
- Without LTF offset. We sample an  $\mathbf{r}_{*}$  and create a dataset using  $\mathrm{pos}(\mathbf{r}_{*}^{\top}\mathbf{x})$  as the labeling function. Table 7 reports the test accuracy scores for Algorithm 2 and random LTF (R)
<sup>1</sup>https://github.com/google-research/google-research/tree/master/Algorithms_and_Hardness_for_Learning_Lineal Thresholds_from_Label_Proportions (license included in the repository)
for centered and general Gaussians. Table 12 reports the scores for the SDP algorithm (S) [25, 26] on  $(q,k)\in \{(2,1),(3,1)\}$  with  $m = \{50,100,200\}$ . Table 3 provides concise comparative scores with  $m = 2000$  for Algorithm 4 and random LTF and  $m = 200$  for the SDP algorithm (S).