SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma B.1 (Bounding of Privatised Error). Let  $f_{\pi}^{\mathrm{DP}}$  be the LPDT estimator in Algorithm 1. Suppose  $f_{\pi}$  is the decision tree regressor defined in (2). Suppose  $\pi$  is generated by Algorithm 2. Let Assumption 3.2 and 3.3 hold. Then, if we take  $n_l \asymp n_q / 2^s$ , there holds
$$
\mathcal {R} _ {L, \mathrm {P}} (f _ {\pi} ^ {\mathrm {D P}}) - \mathcal {R} _ {L, \mathrm {P}} (f _ {\pi}) \lesssim \frac {2 ^ {2 s} \cdot \log n}{n \varepsilon^ {2}}
$$
with probability  $\mathrm{P}^n\otimes \mathrm{Q}^{n_q}\otimes \mathrm{R}^n$  at least  $1 - 1 / n_{q}^{2} - 4 / n^{2}$
Lemma B.2 (Bounding of Sample Error). Suppose  $f_{\pi}$  and  $\overline{f}_{\pi}$  are the decision tree regressor and the population decision tree regressor defined in (2) and (1), respectively. Let Assumption 3.2, and 3.3 hold. Then, if we take  $n_l \asymp n_q / 2^s$ , there holds
$$
\mathcal {R} _ {L, \mathrm {P}} (f _ {\pi}) - \mathcal {R} _ {L, \mathrm {P}} (\overline {{f}} _ {\pi}) \lesssim \frac {2 ^ {s}}{n}
$$
with probability  $\mathrm{P}^n\otimes \mathrm{Q}^{n_q}$  at least  $1 - 1 / n_{q}^{2} - 1 / n^{2}$
Lemma B.3 (Bounding of Approximation Error). Suppose  $\overline{f}_{\pi}$  is the population decision tree regressor defined in (1). Let Assumption 3.2 holds. Then there holds
$$
\mathcal {R} _ {L, \mathrm {P}} (\overline {{f}} _ {\pi}) - \mathcal {R} _ {L, \mathrm {P}} (f ^ {*}) \leq 2 c _ {L} ^ {2} 2 ^ {- (2 \alpha s / d) \wedge s}.
$$
Proof of Theorem 3.1. For each conditional distribution  $\mathrm{R}\left(\tilde{U}_i,\tilde{Y}_i|X_i = x,Y_i = y\right)$ , we can compute the density ratio as
$$
\begin{array}{l} \sup_{x,x^{\prime}\in \mathbb{R}^{d},  y,y^{\prime}\in [-M,M],S\in \sigma (S)}\frac{\operatorname{R}\left((\tilde{U},\tilde{Y})\in S|X_{i} = x,Y_{i} = y\right)}{\operatorname{R}\left((\tilde{U},\tilde{Y})\in S|X_{i} = x^{\prime},Y_{i} = y^{\prime}\right)} \\ \leq \sup  _ {x, x ^ {\prime} \in \mathbb {R} ^ {d}} \frac {d \mathrm {R} (\tilde {U} | X _ {i} = x)}{d \mathrm {R} (\tilde {U} | X _ {i} = x ^ {\prime})} \cdot \sup  _ {y, y ^ {\prime} \in [ - M, M ]} \frac {d \mathrm {R} (\tilde {Y} | Y _ {i} = y)}{d \mathrm {R} (\tilde {Y} | Y _ {i} = y ^ {\prime})}. \tag {8} \\ \end{array}
$$
(i) The first part is characterized by the random perturbation mechanism. Since the conditional density is identical if  $x$  and  $x'$  belongs to a same  $A_j$ , we have
$$
\sup_{x,x^{\prime}\in \mathbb{R}^{d}}\frac{d\mathrm{R}(\tilde{U} |X_{i} = x)}{d\mathrm{R}(\tilde{U} |X_{i} = x^{\prime})} = \sup_{u,u^{\prime}\in \{0,1\}^{\mid \mathcal{T}\mid}}\frac{d\mathrm{R}(\tilde{U} |U_{i} = u)}{d\mathrm{R}(\tilde{U} |U_{i} = u^{\prime})}.
$$
The right-hand side can be computed as
$$
\frac {d \mathrm {R} (\tilde {U} | U _ {i} = u)}{d \mathrm {R} (\tilde {U} | U _ {i} = u ^ {\prime})} = \frac {\prod_ {j \in \mathcal {I}} \mathbb {P} [ \tilde {U} _ {i} ^ {j} | U _ {i} ^ {j} = u ^ {j} ]}{\prod_ {j \in \mathcal {I}} \mathbb {P} [ \tilde {U} _ {i} ^ {j} | U _ {i} ^ {j} = u ^ {\prime j} ]}. \tag {9}
$$
By definition,  $u$  and  $u'$  are one-hot vectors and differ at most on two entries. Without loss of generality, assume they differ on the first two elements. Also, the procedure in (3) yields that  $e^{-\varepsilon/4} \leq \mathbb{P}[\tilde{U}_i^j | U_i^j = 0] / \mathbb{P}[\tilde{U}_i^j | U_i^j = 0] \leq e^{\varepsilon/4}$ . As a result, (9) becomes
$$
\sup  _ {u, u ^ {\prime} \in \{0, 1 \} ^ {| \mathcal {I} |}} \frac {d \mathrm {R} (\tilde {U} | U _ {i} = u)}{d \mathrm {R} (\tilde {U} | U _ {i} = u ^ {\prime})} = \sup  _ {u, u ^ {\prime} \in \{0, 1 \} ^ {| \mathcal {I} |}} \frac {\prod_ {j = 1 , 2} \mathbb {P} \left[ \tilde {U} _ {i} ^ {j} \mid U _ {i} ^ {j} = u ^ {j} \right]}{\prod_ {j = 1 , 2} \mathbb {P} \left[ \tilde {U} _ {i} ^ {j} \mid U _ {i} ^ {j} = u ^ {\prime} j \right]} \leq e ^ {\varepsilon / 4 + \varepsilon / 4} = e ^ {\varepsilon / 2}. \tag {10}
$$
(ii) For the second part, there holds
$$
\begin{array}{l} \sup  _ {y, y ^ {\prime} \in [ - M, M ]} \frac {d \mathrm {R} (\tilde {Y} | Y _ {i} = y)}{d \mathrm {R} (\tilde {Y} | Y _ {i} = y ^ {\prime})} = \sup  _ {y, y ^ {\prime} \in [ - M, M ]} \frac {\exp \left(- \frac {\varepsilon}{4 M} | \tilde {Y} - y |\right)}{\exp \left(- \frac {\varepsilon}{4 M} | \tilde {Y} - y ^ {\prime} |\right)} \\ \leq \sup  _ {y, y ^ {\prime} \in [ - M, M ]} \exp \left(\frac {\varepsilon}{4 M} | y - y ^ {\prime} |\right) \leq e ^ {\varepsilon / 2}. \tag {11} \\ \end{array}
$$
Bringing (10) and (11) into (8) yields the desired conclusion.
C.2 Proof of Theorem 3.4
Proof of Theorem 3.4. Bringing Proposition B.1, B.2, and B.3 into the decomposition (7), we have
$$
\mathcal {R} _ {L, \mathrm {P}} (f _ {\pi} ^ {\mathrm {D P}}) - \mathcal {R} _ {L, \mathrm {P}} (f _ {\pi}) \lesssim \frac {2 ^ {2 s} \cdot \log n}{n \varepsilon^ {2}} + \frac {2 ^ {s}}{n} + 2 ^ {- (2 \alpha s / d) \wedge s}
$$
holds with probability  $\mathrm{P}^n\otimes \mathrm{Q}^{n_q}\otimes \mathrm{R}^n$  at least  $1 - 2 / n_{q}^{2} - 5 / n^{2}$  . When  $(2\alpha s / d)\wedge p = 2\alpha s / d,$  by choosing  $2^{s}\asymp (n\varepsilon^{2} / \log n)^{d / (2\alpha +2d)}$  , i.e.  $s\asymp \log n\varepsilon^2$  , we obtain  $\mathcal{R}_{L,\mathrm{P}}(f_{\pi}^{\mathrm{DP}}) - \mathcal{R}_{L,\mathrm{P}}(f_{\pi})\lesssim$ $(n\varepsilon^{2} / \log n)^{-\frac{\alpha}{\alpha + d}}$  . When  $(2\alpha s / d)\wedge s = s$  , by choosing  $2^{s}\asymp (n\varepsilon^{2} / \log n)^{1 / 3}$  , i.e.  $s\asymp \log n\varepsilon^2$  , we obtain  $\mathcal{R}_{L,\mathrm{P}}(f_{\pi}^{\mathrm{DP}}) - \mathcal{R}_{L,\mathrm{P}}(f_{\pi})\lesssim \left(n\varepsilon^{2} / \log n\right)^{-1 / 3}$  . Note that  $2\alpha s / d\geq s$  if and only if  $\frac{\alpha}{\alpha + d}\geq 1 / 3.$  As a result, we get
$$
\mathcal {R} _ {L, \mathrm {P}} (f _ {\pi} ^ {\mathrm {D P}}) - \mathcal {R} _ {L, \mathrm {P}} (f _ {\pi}) \lesssim \left(\frac {\log n}{n \varepsilon^ {2}}\right) ^ {\frac {\alpha}{\alpha + d} \wedge \frac {1}{3}}
$$
with probability  $\mathrm{P}^n\otimes \mathrm{Q}^{n_q}\otimes \mathrm{R}^n$  at least  $1 - 2 / n_{q}^{2} - 5 / n^{2}$
C.3 Proof of Theorem 3.5
Proof of Theorem 3.5. We consider such  $\mathrm{P}_c\in \mathcal{P}$  such that for each  $d\mathrm{P}_{c,X}(x) = c$  for some constant  $c$  and  $x\in [0,1 / 2]$ , and  $d\mathrm{P}_{c,Y|X}(0) = 1$ . Then for such  $\mathrm{P}_c$ , the Bayes function is  $f(x) = 0$  and  $\mathcal{R}_{L,\mathrm{P}_c}^* = 0$ . For the excess risk of  $f_{\pi}^{\mathrm{DP}}$ , we have
$$
\begin{array}{l} \mathcal {R} _ {L, \mathrm {P}} (f _ {\pi} ^ {\mathrm {D P}}) = \sum_ {j \in \mathcal {I}} \int_ {A _ {j}} \left(\frac {\sum_ {i = 1} ^ {n} \tilde {Y} _ {i} \cdot \tilde {U} _ {i} ^ {j}}{\sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}}\right) ^ {2} d \mathrm {P} _ {c, X} (x) \\ = \sum_ {j \in \mathcal {I}} \frac {1 6 M ^ {2}}{\varepsilon^ {2}} \left(\frac {\sum_ {i = 1} ^ {n} \xi_ {i} \cdot \tilde {U} _ {i} ^ {j}}{\sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}}\right) ^ {2} \int_ {A _ {j}} d P _ {c, X} (x). \tag {12} \\ \end{array}
$$
We first take expectation with respect to  $\xi$ . There holds
$$
\mathbb {E} _ {\xi} \left[ \left(\sum_ {i = 1} ^ {n} \xi_ {i} \cdot \tilde {U} _ {i} ^ {j}\right) ^ {2} \right] = \sum_ {i = 1} ^ {n} \mathrm {E} _ {\xi} \xi_ {i} ^ {2} \left(\tilde {U} _ {i} ^ {j}\right) ^ {2}. \tag {13}
$$
For standard Laplace random variable  $\xi$ , there holds  $\mathrm{E}_{\xi}\xi^{2} = 2$ . Each  $\tilde{U}_i^j$  takes either  $1 - \frac{1}{e^{\varepsilon / 4} + 1}$  or  $-\frac{1}{e^{\varepsilon / 4} + 1}$ , which implies  $(\tilde{U}_i^j)^2 \geq \frac{1}{(e^{\varepsilon / 4} + 1)^2}$ . Together, (13) becomes
$$
\mathbb {E} _ {\xi} \left[ \left(\sum_ {i = 1} ^ {n} \xi_ {i} \cdot \tilde {U} _ {i} ^ {j}\right) ^ {2} \right] \geq \frac {2 n}{(e ^ {\varepsilon / 4} + 1) ^ {2}}.
$$
Bringing this into (12) leads to
$$
\mathbb {E} _ {\xi} \left[ \mathcal {R} _ {L, \mathrm {P}} \left(f _ {\pi} ^ {\mathrm {D P}}\right) \right] \geq \frac {3 2 M ^ {2} n}{\varepsilon^ {2} \left(e ^ {\varepsilon / 4} + 1\right) ^ {2}} \sum_ {j \in \mathcal {I}} \left(\frac {1}{\sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}}\right) ^ {2} \int_ {A _ {j}} d \mathrm {P} _ {c, X} (x). \tag {14}
$$
Next, we take the expectation of  $\tilde{U}_i^j$  with respect to both  $\mathrm{P}$  and  $\mathrm{P}_R$ . Specifically, we have
$$
\begin{array}{l} \mathbb {E} _ {\tilde {U}} \left[ \frac {1}{(\sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}) ^ {2}} \right] = \int_ {0} ^ {\infty} \mathbb {P} \left(\frac {1}{(\sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}) ^ {2}} > t\right) d t \\ = \int_ {0} ^ {\infty} \mathbb {P} \left(- \frac {1}{\sqrt {n t}} \leq \frac {1}{\sqrt {n}} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j} \leq \frac {1}{\sqrt {n t}}\right) d t \\ \geq \int_ {0} ^ {1 / n} \mathbb {P} \left(- \frac {1}{\sqrt {n t}} \leq \frac {1}{\sqrt {n}} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j} \leq \frac {1}{\sqrt {n t}}\right) d t \tag {15} \\ \end{array}
$$
For any  $n$ , there exists some  $\mathrm{P}_c$  with  $c < 1 / (2\sqrt{n})$ . For such  $\mathrm{P}_c$ , we have
$$
\sqrt {n} \int_ {A _ {j}} d \mathrm {P} _ {c, X} (x) = \sqrt {n} \cdot 2 ^ {- s} \cdot c \leq 2 ^ {- s - 1}
$$
since when  $n_l = 0$ , each grid has volume  $2^{-s}$ . When  $t < 1 / n$ , we have  $1 / \sqrt{nt} \geq 1 \geq 2 \cdot 2^{-s - 1}$  for any  $s$ , which implies  $1 / \sqrt{nt} \geq 2 \cdot \sqrt{n} \int_{A_j} d\mathrm{P}_{c,X}(x)$ . Bring this into (15), we have
$$
\mathbb {E} _ {\tilde {U}} \left[ \frac {1}{\left(\sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}\right) ^ {2}} \right] \geq \int_ {0} ^ {1 / n} \mathbb {P} \left(- \frac {1}{2 \sqrt {n t}} \leq \frac {1}{\sqrt {n}} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j} - \sqrt {n} \int_ {A _ {j}} d \mathrm {P} _ {c, X} (x) \leq \frac {1}{2 \sqrt {n t}}\right) d t. \tag {16}
$$
Using the central limit theorem, we know
$$
\mathbb {P} \left(- \frac {1}{2 \sqrt {n t}} \leq \frac {1}{\sqrt {n}} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j} - \sqrt {n} \int_ {A _ {j}} d \mathrm {P} _ {c, X} (x) \leq \frac {1}{2 \sqrt {n t}}\right) \gtrsim 1
$$
for  $t\leq 1 / n$  . Then, (16) becomes
$$
\mathbb {E} _ {\tilde {U}} \left[ \frac {1}{\left(\sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}\right) ^ {2}} \right] \gtrsim \int_ {0} ^ {1 / n} d t = \frac {1}{n}.
$$
This together with (14) yields
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {R} _ {L, \mathrm {P}} \left(f _ {\pi} ^ {\mathrm {D P}}\right) \right] \geq \frac {3 2 M ^ {2} n}{\varepsilon^ {2} \left(e ^ {\varepsilon / 4} + 1\right) ^ {2}} \sum_ {j \in \mathcal {I}} \mathbb {E} _ {\tilde {U}} \left[ \left(\frac {1}{\sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}}\right) ^ {2} \right] \int_ {A _ {j}} d \mathrm {P} _ {c, X} (x) \\ \gtrsim n \sum_ {j \in \mathcal {I}} \frac {1}{n} \int_ {A _ {j}} d \mathrm {P} _ {c, X} (x) = \sum_ {j \in \mathcal {I}} \int_ {A _ {j}} d \mathrm {P} _ {c, X} (x) = 1. \\ \end{array}
$$
C.4 Proofs of results in Section B
To prove lemmas in Section B, we first present several technical results. Their proof will be found in Section C.4
Lemma C.1. Suppose  $\zeta_{i}, i = 1, \dots, n$  are independent random variables such that  $a_{i} \leq \zeta_{i} \leq b_{i}$ . Then there holds
$$
\mathbb {P} \left[ \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \zeta_ {i} - \mathbb {E} \frac {1}{n} \sum_ {i = 1} ^ {n} \zeta_ {i} \right| \geq t \right] \leq 2 e ^ {- \frac {2 n ^ {2} t ^ {2}}{\sum_ {i = 1} ^ {n} (b _ {i} - a _ {i}) ^ {2}}}
$$
for any  $t > 0$
Lemma C.2. Suppose  $\xi_{i}, i = 1, \dots, n$  are independent standard sub-exponential random variables with parameters  $(\nu, \beta)$  [55, Definition 2.9]. Then there holds
$$
\mathbb {P} \left[ \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \xi_ {i} - \mathbb {E} \frac {1}{n} \sum_ {i = 1} ^ {n} \xi_ {i} \right| \geq t \right] \leq 2 e ^ {- \frac {n t ^ {2}}{2 \nu^ {2}}}.
$$
for any  $t > 0$ . Moreover, a standard Laplace random variable is sub-exponential with parameters  $(\sqrt{2},1)$ .
Lemma C.3. Let  $\pi$  be a partition generated from Algorithm 2. Suppose Assumption 3.2 holds. Then for any  $D = \{(X_1, Y_1), \dots, (X_n, Y_n)\}$  drawn i.i.d. from  $\mathbf{P}$  and any  $x \in \mathcal{X}$ , there holds
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {1} \{X _ {i} \in A (x) \} - \int_ {A (x)} d \mathrm {P} _ {X} \left(x ^ {\prime}\right) \right| \leq \sqrt {\frac {\overline {{c}} \cdot 2 ^ {1 - s} (4 d + 5) \log n}{n}} + \frac {2 (4 d + 5) \log n}{3 n} + \frac {4}{n}
$$
with probability  $\mathrm{P}^n$  at least  $1 - 1 / n^{2}$
Lemma C.4. Let  $\pi$  be a partition generated from Algorithm 2. Suppose Assumption 3.2 holds. Then for any  $D = \{(X_1, Y_1), \dots, (X_n, Y_n)\}$  drawn i.i.d. from  $\mathrm{P}$  and any  $x \in \mathcal{X}$ , there holds
$$
\begin{array}{l} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} Y _ {i} \mathbf {1} \{X _ {i} \in A (x) \} - \int_ {A (x)} f ^ {*} \left(x ^ {\prime}\right) d \mathrm {P} _ {X} \left(x ^ {\prime}\right) \right| \\ \leq M \sqrt {\frac {\bar {c} \cdot 2 ^ {1 - s} (4 d + 5) \log n}{n}} + \frac {2 M (4 d + 5) \log n}{3 n} + \frac {4 M}{n} \\ \end{array}
$$
Lemma C.5. Let  $\pi$  be a partition generated from Algorithm 2 using public data set  $D^{pub} = \{(X_{1}^{pub},Y_{1}^{pub}),\dots ,(X_{n_{q}}^{pub},Y_{n_{q}}^{pub})\}$ . Suppose Assumption 3.2 and 3.3 hold. Suppose  $n\geq n_q\geq 2^s\log^2 n$ . If we take  $n_l = n_q / 2^s$ , there holds
$$
\int_ {A (x)} d \mathrm {P} _ {X} \left(x ^ {\prime}\right) \asymp 2 ^ {- s}
$$
with probability  $1 - 1 / n_q^2$ . Moreover, the analogous conclusion holds for the empirical probability measure, i.e.
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {1} \{X _ {i} \in A (x) \} \asymp 2 ^ {- s}.
$$
with probability  $1 - 1 / n_q^2 - 1 / n^2$ .
Lemma C.6. Let  $\pi := \left\{A_i^j, i \in \{1, \dots, p\}, j \in \{1, \dots, 2^s\}\right\} = \{A_j, j \in \mathcal{I}\}$  be the variance reduction decision tree partition generated by Algorithm 2. Then for any  $A_i^j := \times_{i=1}^d [a_i, b_i] \in \pi$ , denote the depth of  $A_i^j$  as  $\mathrm{depth}(A_i^j) = i$ . Then for any  $A_i^j \in \pi$ , there holds
$$
2 ^ {- 1} \sqrt {d} \cdot 2 ^ {- i / d} \leq \operatorname {d i a m} \left(A _ {i} ^ {j}\right) \leq 2 \sqrt {d} \cdot 2 ^ {- i / d}
$$
This implies that for any  $A_{j}\in \pi$ , we have
$$
2 ^ {- 1} \sqrt {d} \cdot 2 ^ {- \operatorname {d e p t h} (A _ {j}) / d} \leq \operatorname {d i a m} (A _ {j}) \leq 2 \sqrt {d} \cdot 2 ^ {- \operatorname {d e p t h} (A _ {j}) / d}.
$$
Lemma C.7. Let  $\pi := \left\{A_i^j, i \in \{1, \dots, p\}, j \in \{1, \dots, 2^s\}\right\} = \{A_j, j \in \mathcal{I}\}$  be the variance reduction decision tree partition generated by Algorithm 2. Then for any  $A_i^j \in \pi$ , denote the depth of  $A_i^j$  as  $\mathrm{depth}(A_i^j) = i$ . Then there holds
$$
\sum_ {j \in \mathcal {I}} 2 ^ {p - \operatorname {d e p t h} (A _ {j})} = 2 ^ {s}.
$$
Proof of Lemma B.1. We intend to bound
$$
\begin{array}{l} \mathcal {R} _ {L, \mathrm {P}} (f _ {\pi} ^ {\mathrm {D P}}) - \mathcal {R} _ {L, \mathrm {P}} (f _ {\pi}) = \int_ {\mathcal {X}} \left| f _ {\pi} ^ {\mathrm {D P}} (x) - f _ {\pi} (x) \right| ^ {2} d \mathrm {P} _ {X} (x) \\ = \int_ {\mathcal {X}} \left| \frac {\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} \tilde {Y} _ {i} \cdot \tilde {U} _ {i} ^ {j}}{\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} \tilde {U} _ {i} ^ {j}} - \frac {\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} Y _ {i} \cdot U _ {i} ^ {j}}{\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} U _ {i} ^ {j}} \right| ^ {2} d \mathrm {P} _ {X} (x). \\ \end{array}
$$
For each  $j \in \mathcal{I}$  and any  $x \in A_j$ , the point-wise error can be decomposed as
$$
\begin{array}{l} \left| \frac {\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} \tilde {Y _ {i}} \cdot \tilde {U _ {i}} ^ {j}}{\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} \tilde {U _ {i}} ^ {j}} - \frac {\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} Y _ {i} \cdot U _ {i} ^ {j}}{\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} U _ {i} ^ {j}} \right| ^ {2} \\ \leq 3 \cdot \left(\underbrace {\left| \frac {\sum_ {i = 1} ^ {n} (\tilde {Y} _ {i} - Y _ {i}) \tilde {U} _ {i} ^ {j}}{\sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}} \right| ^ {2}} _ {(I)} + \underbrace {\left| \frac {\frac {e ^ {\varepsilon / 4} + 1}{e ^ {\varepsilon / 4} - 1} \sum_ {i = 1} ^ {n} Y _ {i} \tilde {U} _ {i} ^ {j} \sum_ {i = 1} ^ {n} U _ {i} ^ {j} - \sum_ {i = 1} ^ {n} Y _ {i} U _ {i} ^ {j} \sum_ {i = 1} ^ {n} U _ {i} ^ {j}} _ {\frac {e ^ {\varepsilon / 4} + 1}{e ^ {\varepsilon / 4} - 1} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j} \sum_ {i = 1} ^ {n} U _ {i} ^ {j}} \right| ^ {2}} _ {(I I)}\right) \\ + \underbrace {\left|\begin{array}{c}\sum_ {i = 1} ^ {n} Y _ {i} U _ {i} ^ {j} \sum_ {i = 1} ^ {n} U _ {i} ^ {j} - \frac {e ^ {\varepsilon / 4} + 1}{e ^ {\varepsilon / 4} - 1} \sum_ {i = 1} ^ {n} Y _ {i} U _ {i} ^ {j} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}\\\hline \frac {e ^ {\varepsilon / 4} + 1}{e ^ {\varepsilon / 4} - 1} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j} \sum_ {i = 1} ^ {n} U _ {i} ^ {j}\end{array}\right| ^ {2}\left. \right)} _ {(I I I)} \\ \end{array}
$$
using triangular inequality. We bound the three parts separately.
(i) For the numerator of  $(I)$ , note that  $\tilde{Y}_i - Y_i = \frac{4M}{\varepsilon}\xi_i$  and  $\tilde{U}_i^j \in [-1,1]$ , Lemma C.2 yields that,  $(\tilde{Y}_i - Y_i)\tilde{U}_i^j$  are sub-exponential random variables with parameter  $(\frac{\varepsilon}{2\sqrt{2}M}, 1)$  and consequently
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\tilde {Y} _ {i} - Y _ {i}\right) \tilde {U} _ {i} ^ {j} \right| \leq \sqrt {\frac {1 2 8 M ^ {2} \cdot \log n}{n \varepsilon^ {2}}} \tag {17}
$$
with probability at least  $1 - 1/n^2$ . For the denominator, applying Lemma C.1 yields
$$
\left| \frac {1}{n} \frac {e ^ {\varepsilon / 4} + 1}{e ^ {\varepsilon / 4} - 1} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j} - \frac {1}{n} \sum_ {i = 1} ^ {n} U _ {i} ^ {j} \right| \leq \sqrt {\frac {\log n}{n}} \tag {18}
$$
with probability at least  $1 - 2 / n^2$ . Moreover, by Lemma C.5, we have
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} U _ {i} ^ {j} = \sum_ {i = 1} ^ {n} \mathbf {1} \{X _ {i} \in A _ {j} \} \gtrsim \frac {1}{2 ^ {s}}
$$
holds with probability at least  $1 - 1 / n_q^2 - 1 / n^2$ . Then, we can guarantee that
$$
\begin{array}{l} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j} \right| \geq \left| \frac {e ^ {\varepsilon / 4} - 1}{e ^ {\varepsilon / 4} + 1} \frac {1}{n} \sum_ {i = 1} ^ {n} U _ {i} ^ {j} \right| - \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j} - \frac {e ^ {\varepsilon / 4} - 1}{e ^ {\varepsilon / 4} + 1} \frac {1}{n} \sum_ {i = 1} ^ {n} U _ {i} ^ {j} \right| (19) \\ \gtrsim \frac {1}{2 ^ {s}} - \frac {e ^ {\varepsilon / 4} - 1}{e ^ {\varepsilon / 4} + 1} \sqrt {\frac {\log n}{n}} \gtrsim \frac {1}{2 ^ {s}} (20) \\ \end{array}
$$
with probability  $1 - 1 / n_q^2 -3 / n^2$ . This together with (17) yields
$$
(I) \lesssim \frac {2 ^ {2 s} \cdot \log n}{n \varepsilon^ {2}}. \tag {21}
$$
(ii) For (II), we first cancel the  $\sum_{i=1}^{n} U_i^j$  on both the numerator and the denominator. Then, applying Lemma C.1, we get
$$
\left| \frac {1}{n} \frac {e ^ {\varepsilon / 4} + 1}{e ^ {\varepsilon / 4} - 1} \sum_ {i = 1} ^ {n} Y _ {i} \tilde {U} _ {i} ^ {j} - \frac {1}{n} \sum_ {i = 1} ^ {n} Y _ {i} U _ {i} ^ {j} \right| \leq \sqrt {\frac {M \cdot \log n}{n}}
$$
with probability  $1 - 2 / n^2$  since  $|Y_{i}|\leq M$ . Combining this with (19), we get
$$
(I I) \lesssim \frac {2 ^ {2 s} \cdot \log n}{n}. \tag {22}
$$
(iii) For (III), note that  $U_{i}^{j} \in \{0,1\}$ , we have  $\sum_{i=1}^{n} Y_{i} U_{i}^{j} \leq M \sum_{i=1}^{n} U_{i}^{j}$ . Thus,
$$
\begin{array}{l} \left| \frac {\sum_ {i = 1} ^ {n} Y _ {i} U _ {i} ^ {j} \sum_ {i = 1} ^ {n} U _ {i} ^ {j} - \frac {e ^ {\varepsilon / 4} + 1}{e ^ {\varepsilon / 4} - 1} \sum_ {i = 1} ^ {n} Y _ {i} U _ {i} ^ {j} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}}{\frac {e ^ {\varepsilon / 4} + 1}{e ^ {\varepsilon / 4} - 1} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j} \sum_ {i = 1} ^ {n} U _ {i} ^ {j}} \right| ^ {2} \leq M ^ {2} \left| \frac {\sum_ {i = 1} ^ {n} U _ {i} ^ {j} - \frac {e ^ {\varepsilon / 4} + 1}{e ^ {\varepsilon / 4} - 1} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}}{\frac {e ^ {\varepsilon / 4} + 1}{e ^ {\varepsilon / 4} - 1} \sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}} \right| ^ {2} \\ \lesssim M ^ {2} \frac {2 ^ {2 s} \cdot \log n}{n} \\ \end{array}
$$
where the last inequality follows from (18) and (19). Combining this with (21) and (22), we have
$$
\left| \frac {\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} \tilde {Y} _ {i} \cdot \tilde {U} _ {i} ^ {j}}{\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} \tilde {U} _ {i} ^ {j}} - \frac {\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} Y _ {i} \cdot U _ {i} ^ {j}}{\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} U _ {i} ^ {j}} \right| ^ {2} \lesssim \frac {2 ^ {2 s} \cdot \log n}{n \varepsilon^ {2}}
$$
with probability  $\mathrm{P}^n\times \mathrm{Q}^{n_q}\times \mathrm{R}$  at least  $1 - 4 / n^{2} - 1 / n_{q}^{2}$  for any  $x\in \mathcal{X}$ . This directly implies the desired bound of  $\mathcal{R}_{L,\mathrm{P}}(f_{\pi}^{\mathrm{DP}}) - \mathcal{R}_{L,\mathrm{P}}(f_{\pi})$
Proof of Lemma B.2. We intend to bound
$$
\begin{array}{l} \mathcal {R} _ {L, \mathrm {P}} (f _ {\pi}) - \mathcal {R} _ {L, \mathrm {P}} (\overline {{f}} _ {\pi}) = \int_ {\mathcal {X}} \left| f _ {\pi} ^ {\mathrm {D P}} (x) - f _ {\pi} (x) \right| ^ {2} d \mathrm {P} _ {X} (x) \\ = \int_ {\mathcal {X}} \left| \frac {\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} Y _ {i} U _ {i} ^ {j}}{\sum_ {i , j} \mathbf {1} \{x \in A _ {j} \} U _ {i} ^ {j}} - \frac {\sum_ {j \in \mathcal {I} _ {p}} \mathbf {1} \{x \in A _ {j} \} \int_ {A _ {j}} f ^ {*} (x ^ {\prime}) d \mathrm {P} _ {X} (x)}{\sum_ {j \in \mathcal {I} _ {p}} \mathbf {1} \{x \in A _ {j} \} \int_ {A _ {j}} d \mathrm {P} _ {X} (x)} \right| ^ {2} d \mathrm {P} _ {X} (x). \\ \end{array}
$$
For any fixed  $j$ , we have
$$
\begin{array}{l} \left| f _ {\pi} ^ {\mathrm {D P}} (x) - f _ {\pi} (x) \right| ^ {2} \leq \left| \frac {\frac {1}{n} \sum_ {i = 1} ^ {n} Y _ {i} U _ {i} ^ {j}}{\frac {1}{n} \sum_ {i = 1} ^ {n} U _ {i} ^ {j}} - \frac {\int_ {A _ {j}} f ^ {*} (x ^ {\prime}) d \mathrm {P} _ {X} (x ^ {\prime})}{\int_ {A _ {j}} d \mathrm {P} _ {X} (x)} \right| ^ {2} \\ \leq \underbrace {\left| \begin{array}{c} \frac {1}{n} \sum_ {i = 1} ^ {n} Y _ {i} U _ {i} ^ {j} \int_ {A _ {j}} d \mathrm {P} _ {X} \left(x ^ {\prime}\right) - \int_ {A _ {j}} f ^ {*} \left(x ^ {\prime}\right) d \mathrm {P} \left(x ^ {\prime}\right) \int_ {A _ {j}} d \mathrm {P} _ {X} \left(x ^ {\prime}\right) \\ \hline \frac {1}{n} \sum_ {i = 1} ^ {n} U _ {i} ^ {j} \int_ {A _ {j}} d \mathrm {P} _ {X} (x) \end{array} \right| ^ {2}} _ {(I)} \\ + \underbrace {\left| \frac {\int_ {A _ {j}} f ^ {*} \left(x ^ {\prime}\right) d P \left(x ^ {\prime}\right) \int_ {A _ {j}} d P _ {X} \left(x ^ {\prime}\right) - \frac {1}{n} \sum_ {i = 1} ^ {n} U _ {i} ^ {j} \int_ {A _ {j}} f ^ {*} \left(x ^ {\prime}\right) d P _ {X} \left(x ^ {\prime}\right)}{\frac {1}{n} \sum_ {i = 1} ^ {n} U _ {i} ^ {j} \int_ {A _ {j}} d P _ {X} (x)} \right| ^ {2}} _ {(I I)} \\ \end{array}
$$
We bound two terms separately. For  $(I)$ , Lemma C.5 yields
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} U _ {i} ^ {j} \int_ {A _ {j}} d \mathrm {P} _ {X} (x) \gtrsim 2 ^ {- 2 s} \tag {23}
$$
with probability  $1 - 1 / n_q^2 - 1 / n^2$ . For the numerator, Lemma C.4 yields
$$
\left| \sum_ {i = 1} ^ {n} Y _ {i} U _ {i} ^ {j} - \int_ {A _ {j}} f ^ {*} \left(x ^ {\prime}\right) d \mathrm {P} _ {X} \left(x ^ {\prime}\right) \right| ^ {2} \left| \int_ {A _ {j}} d \mathrm {P} _ {X} \left(x ^ {\prime}\right) \right| ^ {2} \lesssim \frac {1}{2 ^ {s} \cdot n} \cdot 2 ^ {- 2 s}.
$$
Together, we get  $(I) \lesssim 2^{s} / n$ . Analogously, by Lemma C.3, we have
$$
\left| \int_ {A _ {j}} d \mathrm {P} _ {X} \left(x ^ {\prime}\right) - \sum_ {i = 1} ^ {n} U _ {i} ^ {j} \right| ^ {2} \left| \int_ {A _ {j}} f ^ {*} \left(x ^ {\prime}\right) d \mathrm {P} _ {X} \left(x ^ {\prime}\right) \right| ^ {2} \leq \frac {1}{2 ^ {p} \cdot n} \cdot M ^ {2} \cdot 2 ^ {- 2 s}.
$$
This together with (23) yields  $(II)\lesssim 2^{s} / n$ . The bound of  $(I)$  and  $(II)$  together yields the desired conclusion.
Proof of Lemma B.3. We intend to bound