SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ | \hat {Q} _ {\tau} (X _ {\text {t e s t}}) - Q _ {\tau} (U | X = X _ {\text {t e s t}}) | ^ {2} \right] \\ \leq 4 L ^ {2} h ^ {2} + \mathbb {E} _ {X _ {\text {t e s t}}} \left[ \mathbb {E} _ {X _ {1: n}} \left[ \frac {C _ {4} ^ {\prime} (\underline {{p}} ^ {- 1} , \underline {{M}} , \log (n M))}{n (X _ {\text {t e s t}})} \mathbb {1} \{n (X _ {\text {t e s t}}) \geq C _ {2} \} + 4 M ^ {2} \mathbb {1} \{n (X _ {\text {t e s t}}) <   C _ {2} \} \right] \right] \\ \leq 4 L ^ {2} h ^ {2} + C _ {4} ^ {\prime} C h ^ {- d} \cdot \frac {1}{n} + C _ {7} C h ^ {- d} \cdot \frac {1}{n} \\ = 4 L ^ {2} h ^ {2} + C _ {2} ^ {\prime} \left(\underline {{p}} ^ {- 1}, \underline {{r}} ^ {- 1}, M, \log (n M)\right) h ^ {- d} \cdot \frac {1}{n}. \\ \end{array}
$$
For the special case of  $L = 0$ , we can select  $h = \Theta(M)$ , hence reaching a fast rate of convergence of order  $O\left(\frac{C_2'}{n}\right)$ .
If  $L > 0$ , by selecting  $h = n^{-\frac{1}{d + 2}}L^{\frac{2}{d + 2}}(dC_2')^{-\frac{1}{d + 2}}$ , we get the convergence result that
$$
\mathbb {E} \left[ | \hat {Q} _ {\tau} (F _ {X}) - Q _ {\tau} (F _ {X}) | ^ {2} \right] \leq L ^ {\frac {2 d}{d + 2}} \cdot n ^ {- \frac {2}{d + 2}} \left(d C _ {2} ^ {\prime} \big (L, \underline {{p}} ^ {- 1}, \underline {{r}} ^ {- 1}, M, \log (n) \big)\right) ^ {\frac {2}{d + 2}},
$$
where the dependence of  $C_2'$  on  $\log(n)$  is polynomial. We conclude that
$$
\mathbb {E} \left[ | \hat {Q} _ {\tau} (F _ {X}) - Q _ {\tau} (F _ {X}) | ^ {2} \right] = \tilde {O} \left(L ^ {\frac {2 d}{d + 2}} n ^ {- \frac {2}{d + 2}}\right).
$$
Remark 2. A for large  $d$  cases, the left behind term  $\left(dC_2'\right)\frac{2}{d + 2}$  is almost 1, compared to other terms. The major contribution to the error upper bound is made by the  $L^{\frac{2d}{d + 2}}n^{-\frac{2}{d + 2}}$  term. If  $L$  is replaced by  $kL$ , then  $n$  must be at least  $k^d n$  to keep the upper bound to remain the same, which implies that the contribution of  $L$  is much more significant than  $n$ .
As for the lower bound, Theorem 2 follows immediately from Theorem 3.2 in Györfi et al. [2002] with Hölderness parameter  $(1,L)$ . Note that the class  $\mathcal{P}^L$  we construct has a conditional distribution identical to the Gaussian of the unit variance, which automatically fulfills our Assumption 1 (b) and (c). The only additional requirement in  $\mathcal{P}^L$  compared to their  $\mathcal{D}^{(1,L)}$  is that we require the existence of  $\mu(x) = 0$  for some  $x \in [0,1]^d$ . In their proof, they construct a sub-class based on the division of  $[0,1]^d$  into smaller cubes, where the mean function  $\mu(x)$  is  $(1,L)$ -Hölder (i.e.  $L$ -Lipschitz) on each cube while being zero on the boundary of those cubes. Hence our additional requirement of  $\exists x \in [0,1]^d$  such that  $\mu(x) = 0$  does not affect the result.
Lemma 14 (Theorem 3.2 in Györfi et al. [2002]). Consider a conditional mean estimation problem (i.e. regression problem) with the estimator sequence  $\{\hat{\mu}_n\}$  and true conditional mean  $\mu$ . For the class  $\mathcal{P}^L$ , the sequence
$$
a _ {n} = L ^ {\frac {2 d}{d + 2}} n ^ {- \frac {2}{d + 2}}
$$
is a lower minimax rate of convergence. In particular,
$$
\liminf_{n\to \infty}\inf_{\hat{\mu}_{n}}\sup_{(X,Y)\sim \mathcal{P},\mathcal{P}\in \mathcal{P}^{L}}\frac{\mathbb{E}\left[||\hat{\mu}_{n} - \mu||^{2}\right]}{L^{\frac{2d}{d + 2}}n^{-\frac{2}{d + 2}}}\geq C_{8} > 0,
$$
for some constant  $C_8$  independent of  $L$ .
Proof of Theorem 2. Since for any Gaussian distribution with known variance, estimating its mean is equivalent to estimating its any  $\tau^{-\mathrm{th}}$  quantile, Theorem 2 is a direct corollary from Lemma 14, once one verifies that  $\mathcal{P}^L$  satisfies Assumption 1.
Verifying Assumption 1 (a): Since the  $\tau^{\mathrm{th}}$  conditional quantile is of the type  $\mu(x) + \Phi(\tau)$  and  $\mu(X)$  is  $L$ -Lipschitz, the conditional quantile is also of course  $L$ -Lipschitz.
Verifying Assumption 1 (b): Because  $\mu(x)$  is  $L$ -Lipschitz and  $\mathcal{X}$  is bounded, its image  $\mu(\mathcal{X})$  is bounded. Furthermore, as we know  $0 \in \mu(\mathcal{X})$ , we can bound  $\mu(\mathcal{X})$  by  $[-L\sqrt{d}, L\sqrt{d}]$ , leading to the boundedness of the conditional quantile set  $\mu(\mathcal{X}) + \Phi(r)$ .
Verifying Assumption 1 (c): The pdf around  $\mu(x) + \Phi(\tau)$  is of the same shape as that of standard Gaussian at  $\Phi(\tau)$ , which is definitely lower bounded from zero in a neighborhood.
I Proofs in Section 3.2
Proof of Theorem 3. By the definition of mutual independence, we have  $\forall A\in \mathcal{F}(A),B\in \mathcal{F}(U)$
$$
\mathbb {P} (U \in B | Z \in z (A)) \mathbb {P} (X \in A | Z \in z (A)) = \mathbb {P} (U \in B, X \in A | Z \in z (A)).
$$
$$
\begin{array}{l} \mathbb {P} (U \in B | Z \in z (A)) = \frac {\mathbb {P} (U \in B , X \in A | Z \in z (A))}{\mathbb {P} (X \in A | Z \in z (A))} \\ = \frac {\mathbb {P} (U \in B , X \in A , Z \in z (A)) / \mathbb {P} (Z \in z (A))}{\mathbb {P} (X \in A , Z \in z (A)) / \mathbb {P} (Z \in z (A))} \\ = \frac {\mathbb {P} (U \in B , X \in A , Z \in z (A))}{\mathbb {P} (X \in A , Z \in z (A))} \\ = \frac {\mathbb {P} (U \in B , X \in A)}{\mathbb {P} (X \in A)} \\ = \mathbb {P} (U \in B | X \in A), \\ \end{array}
$$
leading to the conclusion that
$$
F _ {U | X = x} = F _ {U | Z = z (x)}.
$$
Thus, if we have any consistency guarantee or mean squared error guarantee on
$$
| \hat {Q} _ {\tau} (U | Z = z (x)) - Q _ {\tau} (U | Z = z (x)) |,
$$
then we also get the same result for
$$
| \hat {Q} _ {\tau} (U | Z = z (x)) - Q _ {\tau} (U | X = x) |.
$$
Notice that the dimension of  $\mathcal{Z}$  is  $d_0\leq d$  , we can conclude the proof.
Proof of Theorem 4. By definition of quantiles, we know
$$
\mathbb {P} \left(U \leq Q _ {\tau} (U | Z) \mid Z\right) = F _ {U | Z} \left(Q _ {\tau} (U | Z)\right) \geq \tau .
$$
Since we assume that  $F_{U|Z}$  is absolutely continuous with respect to the Lebesgue measure, the Radon-Nikodym derivative exists, which is the conditional probability density function. Thus  $F_{U|Z}$  must be locally Lipschitz. Then for any fixed bounded neighborhood  $I$  containing  $Q_{\tau}(U|Z)$ ,  $F_{U|Z}$  is Lipschitz on  $\bar{I}$  with some Lipschitz constant  $L(\bar{I})$ . Therefore  $\forall \epsilon > 0$  and small enough such that  $B(Q_{\tau}(U|Z), \epsilon) \subset I$ ,
$$
F _ {U | Z} \left(Q _ {\tau} (U | Z) - \epsilon\right) \geq F _ {U | Z} \left(Q _ {\tau} (U | Z)\right) - L (I) \epsilon .
$$
By the definition of quantiles, we have
$$
F _ {U | Z} \left(Q _ {\tau} (U | Z) - \epsilon\right) <   \tau .
$$
$$
F _ {U | Z} \left(Q _ {\tau} (U | Z)\right) <   L (I) \epsilon + \tau .
$$
Taking  $\epsilon \to 0$ , we have
$$
F _ {U | Z} \left(Q _ {\tau} (U | Z)\right) \leq \tau .
$$
Combining it with the fact that  $F_{U|Z}(Q_{\tau}(U|Z)) \geq \tau$ , we prove that
$$
\mathbb {P} \left(U \leq Q _ {\tau} (U | Z) \mid Z\right) = F _ {U | Z} \left(Q _ {\tau} (U | Z)\right) = \tau .
$$
Proof of Theorem 5. We only prove the middle inequality, since the remaining two hold from similar arguments.
By a similar argument as Proposition 2, if one knows  $Z^{(d_2)}$ , then the best single point decision they can make with respect to conditional expectation of pinball loss is
$$
Q_{\tau}(U|Z^{(d_{2})})\in \operatorname *{arg  min}_{u\in \mathbb{R}}\mathbb{E}[l_{\tau}(u,U)|Z^{(d_{2})}].
$$
Since  $Z^{(d_1)} \in \mathcal{F}(Z^{(d_2)})$ , if another learner makes a decision based on the first  $d_1$  components of  $Z^{(d_2)}$ , say,  $Q_{\tau}(U|Z^{(d_1)})$ , then they must suffer a pinball loss no smaller than that  $Z^{(d_2)}$  learner, which is
$$
\mathbb {E} \left[ l _ {\tau} \left(Q _ {\tau} (U | Z ^ {(d _ {2})}), U\right) | Z ^ {(d _ {2})} \right] \leq \mathbb {E} \left[ l _ {\tau} \left(Q _ {\tau} (U | Z ^ {(d _ {1})}), U\right) | Z ^ {(d _ {2})} \right].
$$
By taking the expectation with respect to  $Z^{(d_2)}$ , the tower property of the conditional expectation tells us that
$$
\begin{array}{l} \mathbb {E} \left[ l _ {\tau} \left(Q _ {\tau} (U | Z ^ {(d _ {2})}), U\right) \right] = \mathbb {E} \left[ \mathbb {E} \left[ l _ {\tau} \left(Q _ {\tau} (U | Z ^ {(d _ {2})}), U\right) | Z ^ {(d _ {2})} \right] \right] \\ \leq \mathbb {E} \left[ \mathbb {E} \left[ l _ {\tau} \left(Q _ {\tau} (U | Z ^ {(d _ {1}}), U) | Z ^ {(d _ {2})} \right] \right] \right] \\ = \mathbb {E} \left[ l _ {\tau} \left(Q _ {\tau} (U | Z ^ {(d _ {1})}), U\right) \right]. \\ \end{array}
$$
Noticing that  $Y = U + \hat{f}(X)$  and  $l_{\tau}(a + c, b + c) = l_{\tau}(a, b)$ , we have
$$
l _ {\tau} (u + \hat {f} (X), Y) = l _ {\tau} (u, U),
$$
which verifies the proof finally.
Remark 3. One may wonder if one can conduct the proof of Theorem 5 in a simpler way via Jensen's inequality since  $l_{\tau}(\cdot, \cdot)$  is convex with respect to its first component. But unfortunately, such an argument does not work for the quantile case, since in general,
$$
\mathbb {E} \left[ Q _ {\tau} (U | X) \right] \neq Q _ {\tau} (U).
$$
J Proofs of Auxiliary Lemmas in Appendix H
Proof of Lemma 6. Denote the probability measurement by  $P$ . From the definition of pinball loss and expectation, we have
$$
\begin{array}{l} \mathbb {E} _ {U} \left[ l _ {\tau} (u, U) \right] = \int_ {- \infty} ^ {u ^ {-}} (\tau - 1) (U - u) d P + \int_ {u ^ {-}} ^ {\infty} \tau (U - u) d P \\ = \int_ {- \infty} ^ {u} (\tau - 1) (U - u) d P + \int_ {u} ^ {\infty} \tau (U - u) d P \\ \end{array}
$$
For a small but positive  $\Delta > 0$ , we have
$$
\begin{array}{l} \mathbb {E} _ {U} \left[ l _ {\tau} (u - \Delta , U) \right] - \mathbb {E} _ {U} \left[ l _ {\tau} (u, U) \right] \\ = \int_ {- \infty} ^ {u ^ {-} - \Delta} (\tau - 1) (U - u + \Delta) d P + \int_ {u ^ {-} - \Delta} ^ {\infty} \tau (U - u + \Delta) d P \\ - \int_ {- \infty} ^ {u ^ {-}} (\tau - 1) (U - u) d P - \int_ {u ^ {-}} ^ {\infty} \tau (U - u) d P \\ = \int_ {- \infty} ^ {u ^ {-}} (\tau - 1) (U - u + \Delta) d P + \int_ {u ^ {-}} ^ {\infty} \tau (U - u + \Delta) d P \\ - \int_ {- \infty} ^ {u ^ {-}} (\tau - 1) (U - u) d P - \int_ {u ^ {-}} ^ {\infty} \tau (U - u) d P \\ - \int_ {u ^ {-} - \Delta} ^ {u ^ {-}} (\tau - 1) (U - u + \Delta) d P + \int_ {u ^ {-} - \Delta} ^ {u ^ {-}} \tau (U - u + \Delta) d P \\ = \Delta \left[ \int_ {- \infty} ^ {u ^ {-}} (\tau - 1) \mathrm {d} P + \int_ {u ^ {-}} ^ {\infty} \tau \mathrm {d} P \right] \\ + \int_ {u ^ {-} - \Delta} ^ {u ^ {-}} (U - u + \Delta) \mathrm {d} P \\ = \Delta (\tau - F (u ^ {-})) + \int_ {u ^ {-} - \Delta} ^ {u ^ {-}} (U - u + \Delta) d P. \\ \end{array}
$$
Note that any cdf is càdlàg (right continuous with left limits). Thus,  $\forall \epsilon > 0$ ,  $\exists \delta > 0$ , s.t.  $\forall \Delta < \delta$
$$
\left| F \left(u ^ {-} - \Delta\right) - F \left(u ^ {-}\right) \right| \leq \epsilon .
$$
$$
\begin{array}{l} 0 \leq \int_ {u ^ {-} - \Delta} ^ {u ^ {-}} (U - u + \Delta) \mathrm {d} P \\ \leq \int_ {u ^ {-} - \Delta} ^ {u ^ {-}} \Delta \mathrm {d} P \\ <   \Delta \epsilon . \\ \end{array}
$$
$$
\Delta (\tau - F (u ^ {-})) \leq \mathbb {E} _ {U} \left[ l _ {\tau} (u - \Delta , U) \right] - \mathbb {E} _ {U} \left[ l _ {\tau} (u, U) \right] \leq \Delta (\tau - F (u ^ {-}) + \epsilon).
$$
$$
\begin{array}{l} \tau - F (u ^ {-}) \leq \lim  _ {\Delta \rightarrow 0 ^ {+}} \inf  _ {\Delta} \frac {\mathbb {E} _ {U} [ l _ {\tau} (u - \Delta , U) ] - \mathbb {E} _ {U} [ l _ {\tau} (u , U) ]}{\Delta} \\ \leq \limsup_{\Delta \to 0^{+}}\frac{\mathbb{E}_{U}[l_{\tau}(u - \Delta,U)] - \mathbb{E}_{U}[l_{\tau}(u,U)]}{\Delta} \\ \leq \tau - F (u ^ {-}) + \epsilon . \\ \end{array}
$$
Since  $\epsilon$  can be arbitrarily small, the limit exists, and
$$
\partial_ {-} \mathbb {E} _ {U \sim F} [ l _ {\tau} (u, U) ] = - \lim  _ {\Delta \to 0 ^ {+}} \frac {\mathbb {E} _ {U} [ l _ {\tau} (u - \Delta , U) ] - \mathbb {E} _ {U} [ l _ {\tau} (u , U) ]}{\Delta} = F (u ^ {-}) - \tau .
$$
The other side of semi-derivative can be derived in a similar way, which we omit for simplicity.
Lemma 15. For any  $a_1, a_2, b_1, b_2 \in \mathbb{R}$ , we have
$$
\left| \max  \left\{a _ {1}, b _ {1} \right\} - \max  \left\{a _ {2}, b _ {2} \right\} \right| \leq \max  \left\{\left| a _ {1} - a _ {2} \right|, \left| b _ {1} - b _ {2} \right| \right\}.
$$
Proof of Lemma 15. If  $a_i \geq b_i$  (or  $a_i \leq b_i$ ) simultaneously for  $i = 1, 2$ , the claim is straightforward. Without loss of generality, we assume  $a_1 < b_1 < b_2 < a_2$ . Then
$$
\mathrm {L H S} = a _ {2} - b _ {1} \leq a _ {2} - a _ {1} = \mathrm {R H S}.
$$
Proof of Lemma 13. As for part (a), we notice that
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{1 + \zeta} \right] = \sum_ {j = 0} ^ {n} \frac {1}{j + 1} \mathrm {C} _ {n} ^ {j} p ^ {j} (1 - p) ^ {n - j} \\ = \frac {1}{(n + 1) p} \sum_ {j = 0} ^ {n} \mathrm {C} _ {n + 1} ^ {j + 1} p ^ {j + 1} (1 - p) ^ {n - j} \\ \leq \frac {1}{(n + 1) p} \sum_ {j = 0} ^ {n + 1} C _ {n + 1} ^ {j} p ^ {j} (1 - p) ^ {n - j + 1} \\ = \frac {1}{(n + 1) p} (p + 1 - p) ^ {n + 1} = \frac {1}{(n + 1) p}, \\ \end{array}
$$
where  $C_n^j$  is the number of combinations of  $(n,l)$ .
Since  $\frac{1}{k} \leq \frac{2}{k + 1}$  for any  $k \geq 1$ , we have
$$
\mathbb {E} \left[ \frac {1}{\zeta} \mathbb {1} \{\zeta > 0 \} \right] \leq \mathbb {E} \left[ \frac {2}{1 + \zeta} \right] \leq \frac {2}{(n + 1) p}.
$$
For part (b), we prove its symmetric argument that  $\forall r > np$
$$
\mathbb {P} (\zeta \geq r) \leq \frac {r q}{(r - n p) ^ {2}},
$$
where  $q = 1 - p$
The probability ratio of two adjacent value is
$$
\begin{array}{l} \frac {\mathbb {P} (\zeta = r + 1)}{\mathbb {P} (\zeta = r)} = \frac {(n - r) p}{(r + 1) q} \\ \leq \frac {(n - r) p}{r q} \\ = 1 - \frac {r - n p}{r q} \\ <   1. \\ \end{array}
$$
Hence for any  $k\geq r$  , we have
$$
\mathbb {P} (\zeta = k) \leq \mathbb {P} (\zeta = r) \cdot \left(1 - \frac {r - n p}{r q}\right) ^ {k - r}.
$$
Summing all these  $k$ 's from  $r$  to  $\infty$ , we have
$$
\begin{array}{l} \mathbb {P} (\zeta \geq r) = \sum_ {k = r} ^ {\infty} \mathbb {P} (\zeta = k) \\ \leq \mathbb {P} (\zeta = r) \sum_ {k = r} ^ {\infty} \left(1 - \frac {r - n p}{r q}\right) ^ {k - r} \\ = \mathbb {P} (\zeta = r) \frac {1}{1 - \left(1 - \frac {r - n p}{r q}\right)} \\ = \mathbb {P} (\zeta = r) \frac {r q}{r - n p}. \\ \end{array}
$$
From a similar argument of inspecting the probability ratio between  $\lceil np\rceil \leq k\leq r$  , we know that
$$
\mathbb {P} (\zeta = k) \geq \mathbb {P} (\zeta = r).
$$
There are at least  $r - \lceil np \rceil + 1 \geq r - np$  such  $k$ 's (including  $r$  itself), so we have
$$
1 \geq (r - n p) \mathbb {P} (\zeta = r),
$$
which leads to the conclusion that
$$
\mathbb {P} (\zeta \geq r) \leq \frac {r q}{(r - n p) ^ {2}}, \quad \forall r > n p.
$$
If we have  $r < np$ , then we repeat the above arguments by replacing  $r$  with  $n - r$  and reversing  $p$  and  $q$ , we can prove
$$
\mathbb {P} (\zeta \leq r) \leq \frac {(n - r) p}{(n p - r) ^ {2}}, \quad \forall r <   n p.
$$