SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| P _ {\geq 2 m} ^ {(d)} \operatorname {R e L U} (x) \right\| _ {L ^ {2} (\mu_ {d})} ^ {2} = \sum_ {k \geq m} \frac {(2 k - 3) ! ! ^ {2} B (d , 2 k)}{\beta (\frac {1}{2} , \frac {d - 1}{2}) ^ {2} \prod_ {j = 0} ^ {k} (d + 2 j - 1) ^ {2}}
$$
The proof of this lemma is deferred to Appendix G.1.
We also require a key result from [19], which lower bounds the approximation error of an inner product function.
Definition 9.  $f:\mathcal{S}^{d - 1}(1)\times \mathcal{S}^{d - 1}(1)\to \mathbb{R}$  is an inner product function if  $f(x,x^{\prime}) = \phi (\langle x,x^{\prime}\rangle)$  for some  $\phi :[-1,1]\rightarrow \mathbb{R}$
Definition 10.  $g:\mathcal{S}^{d - 1}(1)\times \mathcal{S}^{d - 1}(1)\to \mathbb{R}$  is a separable function if  $g(x,x^{\prime}) = \psi (\langle v,x\rangle ,\langle v^{\prime},x^{\prime}\rangle)$  for some  $v,v^{\prime}\in \mathcal{S}^{d - 1}(1)$  and  $\psi :[-1,1]^2\to \mathbb{R}$
Let  $\tilde{\nu}_d$  be the uniform distribution over  $S^{d-1}(1) \times S^{d-1}(1)$ . We note that if  $(x, x') \sim \tilde{\nu}_d$ , then  $\langle x, x' \rangle \sim \mu_d$ . For an inner product function  $f$ , we thus have  $\|f\|_{L^2(\tilde{\nu}_d)} = \|f\|_{L^2(\mu_d)}$ . We overload notation and let  $\left\|P_k^{(d)}f\right\|_{L^2(\mu_d)} = \left\|P_k^{(d)}\phi\right\|_{L^2(\mu_d)}$ .
Lemma 39. [19, Theorem 3] Let  $f$  be an inner product function and  $g_1, \ldots, g_r$  be separable functions. Then, for any  $k \geq 1$ ,
$$
\left\| f - \sum_ {i = 1} ^ {r} g _ {i} \right\| _ {L ^ {2} (\tilde {\nu} _ {d})} ^ {2} \geq \| P _ {k} f \| _ {L ^ {2} (\mu_ {d})} \cdot \left(\| P _ {k} f \| _ {L ^ {2} (\mu_ {d})} - \frac {2 \sum_ {i = 1} ^ {r} \| g _ {r} \| _ {L ^ {2} (\tilde {\nu} _ {d})}}{B (d , k) ^ {1 / 2}}\right).
$$
We now can prove Theorem 4
Proof of Theorem 4. We begin with the lower bound. Let  $x = U\left( \begin{array}{c}x_{1}\\ x_{2} \end{array} \right)$ , where  $x_{1},x_{2}\in \mathbb{R}^{d / 2}$ . Assume that there exists some  $\theta$  such that  $\| f^{*} - N_{\theta}\|_{L^{2}(\mathcal{X})}\leq \epsilon$ . Then
$$
\begin{array}{l} \epsilon^ {2} \geq \mathbb {E} _ {x} \left[ \left(f ^ {*} (x) - N _ {\theta} (x)\right) ^ {2} \right] \\ = \mathbb {E} _ {r \sim \mu} \left[ \mathbb {E} _ {x _ {1} \sim \mathcal {S} ^ {d - 1} (\sqrt {r}), x _ {2} \sim \mathcal {S} ^ {d - 1} (\sqrt {d - r})} \left(f ^ {*} (x) - N _ {\theta} (x)\right) ^ {2} \right], \\ \end{array}
$$
where  $r$  is the random variable defined as  $r = \| x_1\|^2$  and  $\mu$  is the associated measure. The equality comes from the fact that conditioned on  $r$ ,  $x_{1}$  and  $x_{2}$  are independent and distributed uniformly on the spheres of radii  $\sqrt{r}$  and  $\sqrt{d - r}$ , respectively. We see that  $\mathbb{E}r = d / 2$ , and thus
$$
\mathbb {P} (| r - d / 2 | > d / 4) \leq \exp (- \Omega (d))
$$
Let  $\delta = \inf_{r\in [d / 4,3d / 4]}\mathbb{E}_{x_1\sim \mathcal{S}^{d - 1}(\sqrt{r}),x_2\sim \mathcal{S}^{d - 1}(\sqrt{d - r})}(f^* (x) - N_\theta (x))^2$  . We get the bound
$$
\epsilon^ {2} \geq \delta \cdot \mathbb {P} (r \in [ d / 4, 3 d / 4 ]),
$$
and thus  $\delta \leq \epsilon^2 (1 - \exp (-\Omega (d)))$ . Therefore there exists an  $r\in [d / 4,3d / 4]$  such that
$$
\mathbb {E} _ {x _ {1} \sim \mathcal {S} ^ {d - 1} (\sqrt {r}), x _ {2} \sim \mathcal {S} ^ {d - 1} (\sqrt {d - r})} (f ^ {*} (x) - N _ {\theta} (x)) ^ {2} \leq \epsilon^ {2} / 2.
$$
Next, see that when  $\| x_1\|^2 = r,\| x_2\|^2 = d - r$  , we have that
$$
x ^ {T} A x = \frac {2}{\sqrt {d}} \langle x _ {1}, x _ {2} \rangle = 2 \sqrt {\frac {r (d - r)}{d}} \langle \overline {{x}} _ {1}, \overline {{x}} _ {2} \rangle ,
$$
where now  $\overline{x}_1, \overline{x}_2 \sim \mathcal{S}^{d/2-1}(1)$  i.i.d. Defining  $q(z) = \mathrm{ReLU}\left(2\sqrt{\frac{r(d-r)}{d}}z\right) - c_0$ , we thus have
$$
\bar {q} \left(\left\langle \bar {x} _ {1}, \bar {x} _ {2} \right\rangle\right) = \operatorname {R e L U} \left(x ^ {T} A x\right) - c _ {0} = f ^ {*} (x).
$$
Furthermore, defining  $\overline{x} = \left(\frac{\overline{x}_1}{\overline{x}_2}\right)$ , choosing the parameter vector  $\overline{\theta} = (a,\overline{W},b_1,b_2)$ , where  $\overline{W} = WU\left(\sqrt{r}\cdot I\quad 0\quad \sqrt{d - r}\cdot I\right)$  yields a network so that  $N_{\overline{\theta}}(\overline{x}) = N_{\theta}(x)$ . Therefore we get that the new network  $N_{\overline{\theta}}$  satisfies
$$
\mathbb {E} _ {\overline {{x}} _ {1}, \overline {{x}} _ {2}} \left[ \left(\bar {q} \left(\langle \overline {{x}} _ {1}, \overline {{x}} _ {2} \rangle\right) - N _ {\bar {\theta}} (\overline {{x}})\right) ^ {2} \right] \leq \epsilon^ {2} / 2,
$$
where  $\overline{x}_1, \overline{x}_2$  are drawn i.i.d over  $\mathrm{Unif}(S^{d/2-1}(1))$ .
We aim to invoke Lemma 39. We note that  $(\overline{x}_1,\overline{x}_2)\sim \tilde{\nu}_{d / 2}$ , and that  $\overline{q}$  is an inner product function. Define  $g_{i}(\overline{x}) = a_{i}\sigma (\overline{w}_{i}^{T}\overline{x} +b_{1,i})$ . We see that  $g_{i}$  is a separable function, and also that
$$
N _ {\overline {{\theta}}} (\overline {{x}}) = \sum_ {i = 1} ^ {m} g _ {i} (\overline {{x}}) + b _ {2}.
$$
Hence  $N_{\overline{\theta}}$  ) is the sum of  $m + 1$  separable functions. We can bound the a single function as
$$
\begin{array}{l} \left| g _ {i} (\bar {x}) \right| \leq \left| a _ {i} \right| C _ {\sigma} \left(1 + \left| \bar {w} _ {i} ^ {T} \bar {x} + b _ {1, i} \right|\right) ^ {\alpha_ {\sigma}} \\ \leq C _ {\sigma} B \left(1 + \sqrt {d} \| \bar {w} _ {i} \| _ {\infty} + B\right) ^ {\alpha_ {\sigma}} \\ \leq C _ {\sigma} B \left(1 + B d ^ {3 / 2} / 2 + B\right) ^ {\alpha_ {\sigma}} \\ \leq \left(B d ^ {3 / 2}\right) ^ {\alpha_ {\sigma} + 1}, \\ \end{array}
$$
since  $\left\| \overline{W}\right\|_{\infty} \leq \max (\sqrt{r}, \sqrt{d - r})\| W U\|_{\infty} \leq B d / 2$ . Therefore by Lemma 39
$$
\mathbb {E} _ {\overline {{x}} _ {1}, \overline {{x}} _ {2}} \Big [ \big (\overline {{q}} (\langle \overline {{x}} _ {1}, \overline {{x}} _ {2} \rangle) - N _ {\overline {{\theta}}} (\overline {{x}}) \big) ^ {2} \Big ] \geq \| P _ {\geq k} \overline {{q}} \| _ {L ^ {2}} \cdot \Bigg (\| P _ {\geq k} \overline {{q}} \| _ {L ^ {2}} - \frac {2 (m + 1) (B d ^ {3 / 2}) ^ {\alpha_ {\sigma} + 1}}{\sqrt {B _ {d / 2 , k}}} \Bigg).
$$
By Lemma 38, we have that
$$
\| P _ {\geq 2 m} \mathrm {R e L U} (x) \| _ {L ^ {2} (\mu_ {d / 2})} ^ {2} = \sum_ {k \geq m} \frac {(2 k - 3) ! ! ^ {2} B (d / 2 , 2 k)}{\beta (\frac {1}{2} , \frac {d / 2 - 1}{2}) ^ {2} \prod_ {j = 0} ^ {k} (d / 2 + 2 j - 1) ^ {2}}
$$
Simplifying, we have that
$$
\begin{array}{l} \frac {(2 k - 3) ! ! ^ {2} B (d / 2 , 2 k)}{\prod_ {j = 0} ^ {k} (d / 2 + 2 j - 1) ^ {2}} = \frac {(2 k - 3) ! ! ^ {2}}{(2 k) !} \cdot \frac {(d / 2 + 2 k - 3) ! (d / 2 + 4 k - 2)}{(d / 2 - 2) ! \prod_ {j = 0} ^ {k} (d / 2 + 2 j - 1) ^ {2}} \\ = \frac {(2 k - 3) ! ! ^ {2}}{(2 k) !} \cdot \frac {(d / 2 + 4 k - 2)}{(d / 2 + 2 k - 1) ^ {2} (d / 2 + 2 k - 3)} \prod_ {j = 0} ^ {k - 2} \frac {d / 2 + 2 j}{d / 2 + 2 j - 1} \\ \geq \frac {(2 k - 3) ! ! ^ {2}}{(2 k) !} \cdot \frac {(d / 2 + 4 k - 2)}{(d / 2 + 2 k - 1) ^ {2} (d / 2 + 2 k - 3)} \\ \geq \frac {1}{2 k (2 k - 1) (2 k - 2)} \cdot \frac {(d / 2 + 4 k - 2)}{(d / 2 + 2 k - 1) ^ {2} (d / 2 + 2 k - 3)} \\ \end{array}
$$
By Gautschi's inequality, we can bound
$$
\beta \bigg (\frac {1}{2}, \frac {d - 2}{4} \bigg) = \frac {\sqrt {\pi} \Gamma (\frac {d - 2}{4})}{\Gamma (\frac {d}{4})} \leq \sqrt {\pi} \bigg (\frac {d}{4} - 1 \bigg) ^ {- 1 / 2} \leq 4 d ^ {- 1 / 2}.
$$
$$
\begin{array}{l} \| P _ {2 k} \operatorname {R e L U} (x) \| _ {L ^ {2} \left(\mu_ {d / 2}\right)} ^ {2} \geq \frac {1}{2 k (2 k - 1) (2 k - 2) 1 6} \cdot \frac {(d / 2 + 4 k - 2) d}{(d / 2 + 2 k - 1) ^ {2} (d / 2 + 2 k - 3)} \\ \geq \frac {1}{1 2 8 k ^ {3}} \cdot \frac {(d / 2 + 4 k - 2) d}{(d / 2 + 2 k - 1) ^ {2} (d / 2 + 2 k - 3)} \\ \geq \frac {1}{1 2 8 k ^ {3} d} \\ \end{array}
$$
for  $k\leq d / 4$  .Altogether,
$$
\| P _ {\geq 2 m} \mathrm {R e L U} (x) \| _ {L ^ {2} (\mu_ {d / 2})} ^ {2} \geq \frac {1}{1 2 8 d} \sum_ {k = m} ^ {d / 4} \frac {1}{k ^ {3}} \geq \frac {1}{5 1 2 m ^ {2} d}
$$
for  $m \leq d/8$ . Since  $\overline{q}(z) = \mathrm{ReLU}\left(2\sqrt{\frac{r(d - r)}{d}z}\right) - c_0 = 2\sqrt{\frac{r(d - r)}{d}}\mathrm{ReLU}(z) - c_0$ , we have that
$$
\begin{array}{l} \| P _ {\geq 2 m} \bar {q} \| _ {L ^ {2} (\mu)} \geq 4 \frac {r (d - r)}{d} \| P _ {\geq 2 m} \operatorname {R e L U} (x) \| \\ \geq \sqrt {3 d} \cdot \sqrt {\frac {1}{5 1 2 m ^ {2} d}} \\ \geq \frac {1}{1 6 m}. \\ \end{array}
$$
We thus have, for any integer  $k < d / 8$
$$
\begin{array}{l} \epsilon^ {2} / 2 \geq \mathbb {E} _ {\overline {{x}} _ {1}, \overline {{x}} _ {2}} \left[ \left(\bar {q} \left(\langle \overline {{x}} _ {1}, \overline {{x}} _ {2} \rangle\right) - N _ {\bar {\theta}} (\overline {{x}})\right) ^ {2} \right] \\ \geq \| P _ {\geq 2 k} \bar {q} \| _ {L ^ {2}} \cdot \left(\| P _ {\geq 2 k} \bar {q} \| _ {L ^ {2}} - \frac {2 (m + 1) \left(B d ^ {3 / 2}\right) ^ {\alpha_ {\sigma} + 1}}{\sqrt {B _ {d / 2 , 2 k}}}\right) \\ \end{array}
$$
Choose  $\epsilon \leq \frac{1}{512k^2}$ ; we then must have
$$
\frac {2 (m + 1) \left(B d ^ {3 / 2}\right) ^ {\alpha_ {\sigma} + 1}}{\sqrt {B _ {d / 2 , 2 k}}} \geq \frac {1}{3 2 k},
$$
$$
\begin{array}{l} (m + 1) (B d ^ {3 / 2}) ^ {\alpha_ {\sigma} + 1} \geq \frac {1}{6 4 k} B (d / 2, 2 k) ^ {1 / 2} \geq d ^ {k} 2 ^ {- k} \cdot \frac {1}{6 4 k \sqrt {(2 k) !}} \\ = C _ {1} \exp \left(k \log d - \log k - \frac {1}{2} \log (2 k)! - k \log 2\right) \\ \geq C _ {1} \exp (k \log d - \log k - k \log (2 k) - k \log 2) \\ \geq C _ {1} \exp \left(k \log \frac {d}{k} - \log k - 2 k \log 2\right) \\ \geq C _ {1} \exp \left(C _ {2} k \log \frac {d}{k}\right) \\ \end{array}
$$
for any  $k \leq C_3d$ . Selecting  $k = \left\lfloor \sqrt{\frac{1}{512\epsilon}} \right\rfloor$  yields
$$
\max  (m, B) \geq C _ {1} \exp \left(C _ {2} \epsilon^ {- 1 / 2} \log (d \epsilon)\right) \cdot d ^ {- 3 / 2} \geq C _ {1} \exp \left(C _ {2} \epsilon^ {- 1 / 2} \log (d \epsilon)\right)
$$
for  $\epsilon$  less than a universal constant  $c_{3}$ .
We next show the upper bound,
It is easy to see that Assumptions 1 and 3 are satisfied. Next, since the verification of Assumptions 2 and 4 only required Lipschitzness, those assumptions are satisfied as well with  $\ell, \chi = 1$ . Finally, we have
$$
\mathbb {E} _ {x} \left[ f ^ {*} (x) ^ {2} \right] \leq \mathbb {E} _ {x} \left[ \operatorname {R e L U} ^ {2} \left(x ^ {T} A x\right) \right] = \frac {1}{2} \mathbb {E} _ {x} \left[ \left(x ^ {T} A x\right) ^ {2} \right] = \frac {d}{d + 2} <   1.
$$
Next, observe that  $\| \mathbb{K}f^{*}\|_{L^2}\lesssim d^{-2}$ . Define  $\overline{A} = \sqrt{\frac{d + 2}{2d}} A$ . This scaling ensures  $\left\| x^T\overline{A} x\right\|_{L^2} = 1$ . Then, we can write  $f^{*}(x) = g^{*}(x^{T}\overline{A} x)$  for  $g^{*}(z) = \sqrt{\frac{2d}{d + 2}}\mathrm{ReLU}(z) - c_{0}$ . For  $\epsilon >0$ , define the smoothed ReLU  $\mathrm{ReLU}_{\epsilon}(z)$  as
$$
\mathrm {R e L U} _ {\epsilon} (z) = \left\{ \begin{array}{l l} 0 & z \leq - \epsilon \\ \frac {1}{4 \epsilon} (x + \epsilon) ^ {2} & - \epsilon \leq 0 \leq \epsilon \\ x & x \geq \epsilon \end{array} \right..
$$
One sees that  $\mathrm{ReLU}_{\epsilon}$  is twice differentiable with  $\| \mathrm{ReLU}_{\epsilon} \|_{1,\infty} \leq 1$  and  $\| \mathrm{ReLU}_{\epsilon} \|_{2,\infty} = \frac{1}{2\epsilon}$ .
We select the test function  $q$  to be  $q(z) = \sqrt{\frac{2d}{d + 2}}\mathrm{ReLU}_{\epsilon}(\overline{\eta}^{-1}\| \mathbb{K}f^{*}\|_{L^2}^{-1}\cdot z) - c_0$ . We see that
$$
q (\bar {\eta} (\mathbb {K} f ^ {*}) (x)) = \operatorname {R e L U} _ {\epsilon} \Big (\| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1} (\mathbb {K} f ^ {*}) (x) \Big),
$$
$$
\begin{array}{l} \left\| f ^ {*} - q (\bar {\eta} (\mathbb {K} f ^ {*}) (x)) \right\| _ {L ^ {2}} \\ = \sqrt {\frac {2 d}{d + 2}} \left\| \operatorname {R e L U} \left(x ^ {T} \bar {A} x\right) - \operatorname {R e L U} _ {\epsilon} \left(\left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} ^ {- 1} \left(\mathbb {K} f ^ {*}\right) (x)\right) \right\| _ {L ^ {2}} \\ \leq \left\| \operatorname {R e L U} \left(x ^ {T} \bar {A} x\right) - \operatorname {R e L U} _ {\epsilon} \left(x ^ {T} \bar {A} x\right) \right\| _ {L ^ {2}} + \left\| \operatorname {R e L U} _ {\epsilon} \left(x ^ {T} \bar {A} x\right) - \operatorname {R e L U} _ {\epsilon} \left(\left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} ^ {- 1} (\mathbb {K} f ^ {*}) (x)\right) \right\| _ {L ^ {2}} \\ \lesssim \left\| \operatorname {R e L U} \left(x ^ {T} \bar {A} x\right) - \operatorname {R e L U} _ {\epsilon} \left(x ^ {T} \bar {A} x\right) \right\| _ {L ^ {2}} + \left\| x ^ {T} \bar {A} x - \| K f ^ {*} \| _ {L ^ {2}} ^ {- 1} \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} \\ \lesssim \left\| \mathrm {R e L U} (x ^ {T} \overline {{A}} x) - \mathrm {R e L U} _ {\epsilon} (x ^ {T} \overline {{A}} x) \right\| _ {L ^ {2}} + L d ^ {- 1 / 1 2} \log d, \\ \end{array}
$$
where the first inequality follows from Lipschitzness and the second inequality is Corollary 3, using  $\kappa = 1$ .
There exists a constant upper bound for the density of  $x^T \overline{A} x$ , and thus we can upper bound
$$
\left\| \operatorname {R e L U} \left(x ^ {T} \bar {A} x\right) - \operatorname {R e L U} _ {\epsilon} \left(x ^ {T} \bar {A} x\right) \right\| _ {L ^ {2}} ^ {2} \lesssim \int_ {0} ^ {\epsilon} \frac {1}{\epsilon^ {2}} z ^ {4} d z \lesssim \epsilon^ {3}.
$$
Furthermore since  $\overline{\eta} = \Theta (\| \mathbb{K}f^{*}\|_{L^2}^{-1}\iota^{-\chi})$  , we get that  $\overline{\eta}^{-1}\| \mathbb{K}f^{*}\|_{L^2}^{-1} = \Theta (\iota^{\chi})$  , and thus
$$
\begin{array}{l} \sup  _ {\in [ - 1, 1 ]} | q (z) | = \sup  _ {z \in [ - \Theta (\iota^ {x}), \Theta (\iota^ {x}) ]} | \operatorname {R e L U} _ {\epsilon} (z) | = \operatorname {p o l y} (\iota) \\ \sup  _ {z \in [ - 1, 1 ]} | q ^ {\prime} (z) | = \overline {{\eta}} ^ {- 1} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1} \sup  _ {z \in [ - \Theta (\iota x), \Theta (\iota x) ]} | (\operatorname {R e L U} _ {\epsilon}) ^ {\prime} (z) | = \operatorname {p o l y} (\iota) \\ \sup  _ {z \in [ - 1, 1 ]} | q ^ {\prime \prime} (z) | = \left(\overline {{\eta}} ^ {- 1} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1}\right) ^ {2} \sup  _ {z \in [ - \Theta (\iota^ {x}), \Theta (\iota^ {x}) ]} | (\operatorname {R e L U} _ {\epsilon}) ^ {\prime \prime} (z) | = \operatorname {p o l y} (\iota) \epsilon^ {- 1} \\ \end{array}
$$
$$
\mathbb {E} _ {x} \left[ \left(f (x; \hat {\theta}) - f ^ {*} (x)\right) ^ {2} \right] \lesssim \tilde {O} \left(\frac {d ^ {4}}{\operatorname* {m i n} (n , m _ {1} , m _ {2})} + \frac {\epsilon^ {- 2}}{m _ {1}} + \sqrt {\frac {\epsilon^ {- 4}}{n}} + \epsilon^ {3} + d ^ {- 1 / 6}\right).
$$
Choosing  $\epsilon = d^{-1/4}$  yields the desired result. As for the sample complexity, we have  $\|q\|_{2,\infty} = \tilde{O}(\epsilon^{-1}) = \tilde{O}(d^{1/4})$ , and so the runtime is  $\mathrm{poly}(d, m_1, m_2, n)$ .
G.1 Proof of Lemma 38
Proof of Lemma 38. For any integer  $k$ , we define the quantities  $A_{2k}^{(d)}$ ,  $B_{2k+1}^{(d)}$  as
$$
A _ {2 k} ^ {(d)} := \int_ {0} ^ {1} x G _ {2 k} ^ {(d)} (x) d \mu_ {d} (x)
$$
$$
B _ {2 k + 1} ^ {(d)} = \int_ {0} ^ {1} G _ {2 k + 1} ^ {(d)} (x) d \mu_ {d} (x).
$$
We also let  $Z_{d} = \frac{\Gamma(d / 2)}{\sqrt{\pi}\Gamma(\frac{d - 1}{2})}$  to be the normalization constant.
Integration by parts yields
$$
\begin{array}{l} A _ {2 k} ^ {(d)} = Z _ {d} \int_ {0} ^ {1} G _ {2 k} ^ {(d)} (x) x \left(1 - x ^ {2}\right) ^ {\frac {d - 3}{2}} d x \\ = - Z _ {d} \cdot G _ {2 k} ^ {(d)} (x) \cdot \frac {1}{d - 1} \left(1 - x ^ {2}\right) ^ {\frac {d - 1}{2}} \big | _ {0} ^ {1} + \frac {Z _ {d}}{d - 1} \cdot \frac {2 k (2 k + d - 2)}{d - 1} \int_ {0} ^ {1} G _ {2 k - 1} ^ {(d + 2)} (x) \left(1 - x ^ {2}\right) ^ {\frac {d - 1}{2}} d x \\ = \frac {Z _ {d}}{d - 1} G _ {2 k} ^ {(d)} (0) + \frac {Z _ {d}}{Z _ {d + 2}} \cdot \frac {2 k (2 k + d - 2)}{(d - 1) ^ {2}} \cdot B _ {2 k - 1} ^ {(d + 2)} \\ \end{array}
$$
From Corollary 5 we have
$$
G _ {2 k} ^ {(d)} (0) = \frac {(2 k - 1) ! !}{\Pi_ {j = 0} ^ {k - 1} (d + 2 j - 1)} (- 1) ^ {k}.
$$
$$
A _ {2 k} ^ {(d)} = Z _ {d} \cdot \frac {(2 k - 1) ! !}{(d - 1) \Pi_ {j = 0} ^ {k - 1} (d + 2 j - 1)} (- 1) ^ {k} + \frac {2 k (2 k + d - 2)}{(d - 1) ^ {2}} \cdot \frac {Z _ {d}}{Z _ {d + 2}} B _ {2 k - 1} ^ {(d + 2)}. \tag {25}
$$
The recurrence formula yields
$$
\begin{array}{l} B _ {2 k + 1} ^ {(d)} = \int_ {0} ^ {1} G _ {2 k + 1} ^ {(d)} (x) d \mu_ {d} (x) (26) \\ = \int_ {0} ^ {1} \left[ \frac {4 k + d - 2}{2 k + d - 2} x G _ {2 k} ^ {(d)} (x) - \frac {2 k}{2 k + d - 2} G _ {2 k - 1} ^ {(d)} (x) \right] d \mu_ {d} (x) (27) \\ = \frac {4 k + d - 2}{2 k + d - 2} A _ {2 k} ^ {(d)} - \frac {2 k}{2 k + d - 2} B _ {2 k - 1} ^ {(d)}. (28) \\ \end{array}
$$
I claim that
$$
A _ {2 k} ^ {(d)} = \left\{ \begin{array}{l l} \frac {Z _ {d}}{d - 1} & k = 0 \\ (- 1) ^ {k + 1} Z _ {d} \frac {(2 k - 3) ! !}{\prod_ {j = 0} ^ {k} (d + 2 j - 1)} & k \geq 1 \end{array} \right. \text {a n d} B _ {2 k + 1} ^ {(d)} = (- 1) ^ {k} Z _ {d} \frac {(2 k - 1) ! !}{\prod_ {j = 0} ^ {k} (d + 2 j - 1)}.
$$
We proceed by induction on  $k$ . For the base cases, we first have
$$
\begin{array}{l} A _ {0} ^ {(d)} = \int_ {0} ^ {1} x d \mu_ {d} (x) = \int_ {0} ^ {1} Z _ {d} x (1 - x ^ {2}) ^ {\frac {d - 3}{2}} \\ = \int_ {0} ^ {1} \frac {Z _ {d}}{d - 1} d u \\ = \frac {Z _ {d}}{d - 1}, \\ \end{array}
$$
where we use the substitution  $u = \left(1 - x^{2}\right)^{\frac{d - 3}{2}}$ . Next,
$$
B _ {1} ^ {(d)} = \int_ {0} ^ {1} x d \mu_ {d} (x) = A _ {0} ^ {(d)} = \frac {Z _ {d}}{d - 1}.
$$
Next, eq. (25) gives
$$
\begin{array}{l} A _ {2} ^ {(d)} = Z _ {d} \cdot \frac {- 1}{(d - 1) ^ {2}} + \frac {2 d}{(d - 1) ^ {2}} \cdot \frac {Z _ {d}}{d + 1} \\ = \frac {Z _ {d}}{(d - 1) (d + 1)}. \\ \end{array}
$$
Finally, eq. (26) gives
$$
\begin{array}{l} B _ {3} ^ {(d)} = \frac {d + 2}{d} A _ {2} ^ {(d)} - \frac {2}{d} B _ {1} ^ {(d)} \\ = \frac {Z _ {d}}{d - 1} \left[ \frac {d + 2}{d (d + 1)} - \frac {2}{d} \right] \\ = - \frac {Z _ {d}}{(d - 1) (d + 1)}. \\ \end{array}
$$
Therefore the base case is proven for  $k = 0,1$ .
Now, assume that the claim is true for some  $k \geq 1$  for all  $d$ . We first have
$$
\begin{array}{l} A _ {2 k + 2} ^ {(d)} = Z _ {d} \cdot \frac {(2 k + 1) ! !}{(d - 1) \Pi_ {j = 0} ^ {k} (d + 2 j - 1)} (- 1) ^ {k + 1} + \frac {(2 k + 2) (2 k + d)}{(d - 1) ^ {2}} \cdot \frac {Z _ {d}}{Z _ {d + 2}} B _ {2 k + 1} ^ {(d + 2)} \\ = Z _ {d} \cdot \frac {(2 k + 1) ! !}{(d - 1) \Pi_ {j = 0} ^ {k} (d + 2 j - 1)} (- 1) ^ {k + 1} + Z _ {d} \cdot \frac {(2 k + 2) (2 k + d)}{(d - 1) ^ {2}} \cdot \frac {(2 k - 1) ! !}{\Pi_ {j = 0} ^ {k} (d + 2 j + 1)} (- 1) ^ {k} \\ = (- 1) ^ {k + 1} Z _ {d} \cdot \frac {(2 k - 1) ! !}{\Pi_ {j = 0} ^ {k + 1} (d + 2 j - 1)} \left[ \frac {(d + 2 k + 1) (2 k + 1)}{d - 1} - \frac {(2 k + 2) (2 k + d)}{d - 1} \right] \\ = (- 1) ^ {k + 1} Z _ {d} \cdot \frac {(2 k - 1) ! !}{\prod_ {j = 0} ^ {k + 1} (d + 2 j - 1)} \left[ \frac {- d + 1}{d - 1} \right] \\ = (- 1) ^ {k + 2} Z _ {d} \cdot \frac {(2 k - 1) ! !}{\Pi_ {j = 0} ^ {k + 1} (d + 2 j - 1)}. \\ \end{array}
$$
$$
\begin{array}{l} B _ {2 k + 3} ^ {(d)} = \frac {4 k + d + 2}{2 k + d} A _ {2 k + 2} ^ {(d)} - \frac {2 k + 2}{2 k + d} B _ {2 k + 1} ^ {(d)} \\ = (- 1) ^ {k} Z _ {d} \left[ \frac {4 k + d + 2}{2 k + d} \frac {(2 k - 1) ! !}{\prod_ {j = 0} ^ {k + 1} (d + 2 j - 1)} - \frac {2 k + 2}{2 k + d} \frac {(2 k - 1) ! !}{\prod_ {j = 0} ^ {k} (d + 2 j - 1)} \right] \\ = (- 1) ^ {k} Z _ {d} \frac {(2 k - 1) ! !}{\prod_ {j = 0} ^ {k + 1} (d + 2 j - 1)} \left[ \frac {(4 k + d + 2) - (2 k + 2) (d + 2 k + 1)}{2 k + d} \right] \\ = (- 1) ^ {k} Z _ {d} \frac {(2 k - 1) ! !}{\prod_ {j = 0} ^ {k + 1} (d + 2 j - 1)} \left[ \frac {- (2 k + 1) (2 k + d)}{2 k + d} \right] \\ = (- 1) ^ {k + 1} Z _ {d} \frac {(2 k + 1) ! !}{\prod_ {j = 0} ^ {k + 1} (d + 2 j - 1)}. \\ \end{array}
$$
Therefore by induction the claim holds for all  $k, d$ .
The Gegenbauer expansion of ReLU is given by
$$
\operatorname {R e L U} (x) = \sum_ {i = 0} ^ {\infty} \left\langle \operatorname {R e L U}, G _ {i} ^ {(d)} \right\rangle_ {L ^ {2} (\mu_ {d})} B (d, i) G _ {i} ^ {(d)} (x).
$$
Note that  $\mathrm{ReLU}(x) = \frac{1}{2} (x + |x|)$ . Since  $|x|$  is even, the only nonzero odd Gegenbauer coefficient is for  $G_{1}^{(d)}$ . In this case,
$$
\langle \mathrm {R e L U}, G _ {1} ^ {(d)} \rangle_ {L ^ {2} (\mu_ {d})} = \frac {1}{2} \mathbb {E} _ {x \sim \mu_ {d}} [ x ^ {2} ] = \frac {1}{2 d ^ {2}}.
$$
Also,  $B(d,1) = d$ . Next, we see that
$$
\left\langle \operatorname {R e L U}, G _ {2 k} ^ {(d)} \right\rangle_ {L ^ {2} \left(\mu_ {d}\right)} = \int_ {- 1} ^ {1} \operatorname {R e L U} (x) G _ {2 k} ^ {(d)} (x) d \mu_ {d} (x) = \int_ {0} ^ {1} x G _ {2 k} ^ {(d)} (x) d \mu_ {d} (x) = A _ {2 k} ^ {(d)}.
$$
Plugging in our derivation for  $A_{2k}^{(d)}$  gives the desired result.