SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We prove here Corollary 4.5 through a result by Allen-Zhu et al. [3, Theorem 1 and Example 3.1]. We first give a simplified version of this result and then we show how to use it to prove Corollary 4.5.
Consider learning some unknown distribution  $\mathcal{D}$  of data points  $z = (x,y)\in \mathbb{R}^d\times \mathcal{V}$ , where  $x$  is the input point and  $y$  is the label. Without loss of generality, assume  $\| x\| _2 = 1$  and  $x_{d} = 1 / 2$ . Consider a loss function  $L:\mathbb{R}^k\times \mathcal{V}\to \mathbb{R}$  such that for every  $y\in \mathcal{V}$ , the function  $L(\cdot ,y)$  is non-negative, convex, 1-Lipschitz continuous and  $L(0,y)\in [0,1]$ . This includes both the cross-entropy loss and the  $L_{2}$ -regression loss (for bounded  $\mathcal{V}$ ).
Let  $g: \mathbb{R} \to \mathbb{R}$  be a smooth activation function such that  $g(z) = e^{z}$ ,  $\sin(z)$ ,  $\operatorname{sigmoid}(z)$ ,  $\tanh(z)$  or is a low degree polynomial.
Define  $F^{\star}:\mathbb{R}^{d}\to \mathbb{R}^{k}$  such that  $OPT = \mathbb{E}_{\mathcal{D}}[L(F^{\star}(x),y)]$  is the smallest population error made by a neural network of the form  $F^{\star} = A^{\star}g(W^{\star}x)$ , where  $A^{\star}\in \mathbb{R}^{k\times p}$  and  $W^{\star}\in \mathbb{R}^{p\times d}$ . Assume for simplicity that the rows of  $W^{*}$  have  $\ell_2$ -norm 1 and each element of  $A^{*}$  is less or equal than 1.
Define a ReLU neural network  $F(x, W_0) = A_0 \sigma(W_0 x + b_0)$ , where  $A_0 \in \mathbb{R}^{k \times m}$ ,  $W_0 \in \mathbb{R}^{m \times d}$ , the entries of  $W_0$  and  $b_0$  are i.i.d. random Gaussians from  $\mathcal{N}(0, 1/m)$  and the entries of  $A$  are i.i.d. random Gaussians from  $\mathcal{N}(0, \varepsilon_A)$ , for  $\varepsilon_A \in (0, 1]$ . We train the weights  $W$  of this neural network through stochastic gradient descent over a dataset with  $N$  i.i.d. samples from  $\mathcal{D}$ , i.e., we update  $W_{t+1} = W_t - \eta g_t$ , where  $\mathbb{E}[g_t] = \nabla \mathbb{E}_{\mathcal{D}}[L(F(x, W_0 + W_t), y)]$ .
Theorem J.1 (Theorem 1 of Allen-Zhu et al. [3]). Let  $\varepsilon \in (0, O(1 / pk))$ , choose  $\varepsilon_A = \varepsilon / \widetilde{\Theta}(1)$  for the initialization and learning rate  $\eta = \widetilde{\Theta}\left(\frac{1}{\varepsilon km}\right)$ . SGD finds a set of parameters such that
$$
\frac {1}{J} \sum_ {n = 0} ^ {J - 1} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ L \left(F (x; W ^ {(0)} + W _ {t}), y\right) \right] \leq O P T + \varepsilon
$$
with probability  $1 - e^{-c\log^2 m}$  over the random initialization, for a sufficiently large constant  $c$ , with
$$
s i z e m = \frac {\mathrm {p o l y} (k , p)}{\mathrm {p o l y} (\varepsilon)} a n d s a m p l e c o m p l e x i t y \min  \{N, J \} = \frac {\mathrm {p o l y} (k , p , \log m)}{\varepsilon^ {2}}.
$$
Theorem J.1 shows that it is possible to achieve the population error OPT by training a two-layer ReLU network with SGD, and quantifies the number of samples needed to do so.
We make the following assumption to address the population error in our setting.
Assumption J.2. Let  $g: \mathbb{R} \to \mathbb{R}$  be a smooth activation function such that  $g(z) = e^{z}$ ,  $\sin(z)$ , sigmoid(z),  $\tanh(z)$  or is a low degree polynomial. For all time-steps  $t$ , we assume that there exists a target network  $F^{\star, t}: \mathbb{R}^d \to \mathbb{R}^k$ , with
$$
F ^ {\star , t} = \left(f _ {1} ^ {\star , t}, \dots , f _ {k} ^ {\star , t}\right) \quad \text {a n d} \quad f _ {r} ^ {\star , t} (x) = \sum_ {i = 1} ^ {p} a _ {r, i} ^ {\star , t} g (\langle w _ {1, i} ^ {\star , t}, x \rangle) \langle w _ {2, i} ^ {\star , t}, x \rangle
$$
where  $w_{1,i}^{\star ,t}\in \mathbb{R}^d,w_{2,i}^{\star ,t}\in \mathbb{R}^d$  , and  $a_{r,i}^{\star ,t}\in \mathbb{R}$  , such that
$$
\mathbb {E} \big [ \left\| F ^ {\star , t} - Q ^ {t} - \eta_ {t} ^ {- 1} \nabla h (\pi^ {t}) \right\| _ {L _ {2} (v ^ {t})} ^ {2} \big ] \leq O P T.
$$
We assume for simplicity  $\| w_{1,i}^{\star ,t}\| _2 = \| w_{2,i}^{\star ,t}\| _2 = 1$  and  $|a_{r,i}^{\star ,t}|\leq 1$
Assumptions similar to Assumption J.2 have already been made in the literature, such as the bias assumption in the compatible function approximation framework studied by [1]. The term  $OPT$  represents the minimum error incurred by a target network parameterized as  $F^{\star ,t}$  when solving the regression problem in Line 1 of Algorithm 1.
We are now ready to prove Corollary 4.5, which uses Algorithm 4 to obtain an unbiased estimate of the current Q-function. We assume to be in the same setting as Theorem J.1
Proof of Corollary 4.5. We aim to find a policy  $\pi^T$  such that
$$
V ^ {\star} (\mu) - \mathbb {E} \left[ V ^ {T} (\mu) \right] \leq \varepsilon . \tag {66}
$$
Suppose the total number of iterations, that is policy optimization steps, in AMPO is  $T$ . We need the bound in Assumption (A1) to hold for all  $T$  with probability  $1 - e^{-c\log^2 m}$ , which means that at each iteration the bound should hold with probability  $1 - T^{-1}e^{-c\log^2 m}$ . Through Algorithm 4, the expected number of samples needed to obtain an unbiased estimate of the current Q-function is  $(1 - \gamma)^{-1}$ . Therefore, using Theorem J.1, at each iteration of AMPO we need at most
$$
\frac {\operatorname {p o l y} (k , p , \log m , \log T)}{\varepsilon_ {\text {a p p r o x}} ^ {2} (1 - \gamma)}
$$
samples for SGD to find parameters that satisfy Assumption (A1) with probability  $1 - T^{-1}e^{-c\log^2 m}$ . To obtain (66), we need
$$
\frac {1}{1 - \gamma} \left(1 - \frac {1}{\nu_ {\mu}}\right) ^ {T} \left(1 + \frac {\mathcal {D} _ {0} ^ {\star}}{\eta_ {0} (\nu_ {\mu} - 1)}\right) \leq \frac {\varepsilon}{2} \quad \text {a n d} \quad \frac {2 (1 + \nu_ {\mu}) \sqrt {C _ {v} \varepsilon_ {\text {a p p r o x}}}}{1 - \gamma} \leq \frac {\varepsilon}{2}. \tag {67}
$$
Solving for  $T$  and  $\varepsilon_{\mathrm{approx}}$  and multiplying them together, we obtain the sample complexity of AMPO, that is
$$
\widetilde {\mathcal {O}} \left(\frac {\mathrm {p o l y} (k , p , \log m) C _ {v} ^ {2} \nu_ {\mu} ^ {5}}{\varepsilon^ {4} (1 - \gamma) ^ {6}}\right).
$$
Due to the statement of Theorem J.1, we cannot guarantee the approximation error incurred by the learner network to be smaller than  $OPT$ . Consequently, we have that
$$
\varepsilon \geq \frac {4 (1 + \nu_ {\mu}) \sqrt {C _ {v} O P T}}{1 - \gamma}.
$$
A similar bound can be applied to any proof that contains the bias assumption introduced by [1].
We can obtain an improvement over Corollary 4.5 by using the relaxed assumptions in Appendix G, in particular using the condition in (65).
Corollary J.3. In the setting of Theorem 4.3, replace Assumption (A1) with the condition
$$
\mathbb {E} \left[ \mathbb {E} \left\| f ^ {t + 1} - Q ^ {t} - \eta_ {t} ^ {- 1} \nabla h \left(\pi^ {t}\right) \right\| _ {L _ {2} (v ^ {t})} ^ {2} \right] \leq \frac {\nu_ {\mu} ^ {2}}{T ^ {2}} \left(1 - \frac {1}{\nu_ {\mu}}\right) ^ {- 2 (T - t)} \varepsilon_ {\text {a p p r o x}}, \tag {68}
$$
for all  $t < T$ . Let the parameterization class  $\mathcal{F}^{\Theta}$  consist of sufficiently wide shallow ReLU neural networks. Using an exponentially increasing step-size and solving the minimization problem in Line 1 with SGD as in (19), the number of samples required by AMPO to find an  $\varepsilon$ -optimal policy with high probability is  $\widetilde{\Theta}(C_v^2\nu_\mu^4/\varepsilon^4(1-\gamma)^6)$ .
Proof. The proof follows that of Corollary 4.5. Using Theorem J.1, at each iteration  $t$  of AMPO, we need at most
$$
\frac {T ^ {2}}{\nu_ {\mu} ^ {2}} \left(1 - \frac {1}{\nu_ {\mu}}\right) ^ {2 (T - t)} \frac {\operatorname {p o l y} (k , p , \log m , \log T)}{\varepsilon_ {\mathrm {a p p r o x}} ^ {2} (1 - \gamma)}
$$
samples for SGD to find parameters that satisfy condition (68) with probability  $1 - T^{-1}e^{-c\log^2 m}$ . Summing over  $T$  total iterations of AMPO we obtain that the total number of samples needed is
$$
\begin{array}{l} \sum_ {t \leq T} \frac {T ^ {2}}{\nu_ {\mu} ^ {2}} \left(1 - \frac {1}{\nu_ {\mu}}\right) ^ {2 (T - t)} \frac {\mathsf {p o l y} (k , p , \log m , \log T)}{\varepsilon_ {\mathrm {a p p r o x}} ^ {2} (1 - \gamma)} \\ = \frac {T ^ {2}}{\nu_ {\mu} ^ {2}} \left(1 - \frac {1}{\nu_ {\mu}}\right) ^ {2 T} \frac {\mathsf {p o l y} (k , p , \log m , \log T)}{\varepsilon_ {\mathrm {a p p r o x}} ^ {2} (1 - \gamma)} \sum_ {t \leq T} \left(1 - \frac {1}{\nu_ {\mu}}\right) ^ {- 2 t} \\ = \frac {T ^ {2}}{\nu_ {\mu} ^ {2}} \left(1 - \frac {1}{\nu_ {\mu}}\right) ^ {2 T} \frac {\operatorname {p o l y} (k , p , \log m , \log T)}{\varepsilon_ {\text {a p p r o x}} ^ {2} (1 - \gamma)} \frac {\left(\left(1 - \frac {1}{\nu_ {\mu}}\right) ^ {- 2 (T + 1)} - 1\right)}{\left(\left(\frac {1}{1 - \nu_ {\mu}}\right) ^ {- 2} - 1\right)} \\ \leq \mathcal {O} \left(\frac {T ^ {2}}{\nu_ {\mu} ^ {2}} \frac {\mathsf {p o l y} (k , p , \log m , \log T)}{\varepsilon_ {\mathrm {a p p r o x}} ^ {2} (1 - \gamma)}\right) \\ \end{array}
$$
Replacing  $T$  and  $\varepsilon_{\mathrm{approx}}$  with the solutions of (67) gives the result.
At this stage, it is important to note that choosing a method different from the one proposed by Allen-Zhu et al. [4] to solve Line 1 in Algorithm 1 of our paper with neural networks can lead to alternative, and possibly better, sample complexity results for AMPO. For example, we can obtain a sample complexity result for AMPO that does not involve a target network using results from [39] and [16], although this requires introducing more notation and background results compared to Corollary 4.5 (since in [16] they employ a temporal-difference-based algorithm, that is Algorithm 3 in their work, to obtain a neural network estimate  $\widehat{Q}^t$  of  $Q^t$ , while in [39] they provide a method based on Fourier transforms to approximate a target function through shallow ReLU networks). We outline below the steps in order to do so (and additional details including the precise statements of the results we use and how we use them are provided thereafter for the sake of completeness).
Step 1) We first split the approximation error in Assumption (A1) into a critic error  $\mathbb{E}[\sqrt{\|\widehat{Q}^t - Q^t\|_{L_2(v^t)}^2}] \leq \varepsilon_{\mathrm{crit}}$  and an actor error  $\mathbb{E}[\sqrt{\|f^{t + 1} - \widehat{Q}^t - \eta_t^{-1}\nabla h(\pi^t)\|_{L_2(v^t)}^2}] \leq \varepsilon_{\mathrm{actor}}$ . In this case, the linear convergence rate in our Theorem 4.3 becomes
$$
V ^ {\star} (\mu) - \mathbb {E} \left[ V ^ {T} (\mu) \right] \leq \frac {1}{1 - \gamma} \bigg (1 - \frac {1}{\nu_ {\mu}} \bigg) ^ {T} \bigg (1 + \frac {\mathcal {D} _ {0} ^ {\star}}{\eta_ {0} (\nu_ {\mu} - 1)} \bigg) + \frac {2 (1 + \nu_ {\mu}) \sqrt {C _ {v}} (\varepsilon_ {\mathrm {c r i t i c}} + \varepsilon_ {\mathrm {a c t o r}})}{1 - \gamma}.
$$
[We can obtain this alternative statement by modifying the passages in Appendix D.2. In particular, writing  $f^{t + 1} - Q^t - \eta_t^{-1}\nabla h(\pi^t) = (f^{t + 1} - \widehat{Q}^t - \eta_t^{-1}\nabla h(\pi^t)) + (Q^t - \widehat{Q}^t)$  and bounding the two terms with the same procedure in Appendix D.2 leads to this alternative expression for the error.]
We will next deal with the critic error and actor error separately.
Step 2) Critic error. Under a realizability assumption that we provide below along with the statement of the theorem (Assumption 2 in [16]), Theorem 1 from [16] gives that the sample complexity required to obtain  $\mathbb{E}[\sqrt{\|Q^t - Q^t\|_{L_2(d_\mu^t\cdot\pi^t)}^2} ]\leq \varepsilon$  is  $\widetilde{O} (\varepsilon^{-4}(1 - \gamma)^{-2})$ , while the required network width is  $\widetilde{O} (\varepsilon^{-2})$ .
Step 3) Actor error. Using Theorem E.1 from [39], we obtain that  $\mathbb{E}[\sqrt{\|f^{t+1} - \widehat{Q}^t - \eta_t^{-1}\nabla h(\pi^t)\|_{L_2(v^t)}^2}]$  can be made arbitrarily small by tuning the width of  $f^{t+1}$ , without using further samples.
Step 4) Replacing Equation (67) with the sample complexity of the critic, we obtain the following corollary on the sample complexity of AMPO, which does not depend on the error made by a target network.
Corollary J.4. In the setting of Theorem 4.3, let the parameterization class  $\mathcal{F}^{\Theta}$  consist of sufficiently wide shallow ReLU neural networks. Using an exponentially increasing step-size and using the techniques above to update  $f^{\theta}$ , the number of samples required by AMPO to find an  $\varepsilon$ -optimal policy with high probability is  $\widetilde{\mathcal{O}}(C_v^2\nu_\mu^5/\varepsilon^4(1-\gamma)^7)$ .
To the best of our knowledge, this result improves upon the previous best result on the sample complexity of a PG method with neural network parameterization [16], i.e.,  $\widetilde{\mathcal{O}}(C_v^2/\varepsilon^6 (1 - \gamma)^9)$ .
We now provide the statements of the aforementioned results we used.
Recalling Theorem 1 in [16] and its assumptions. Consider the following space of mappings:
$$
\mathcal {H} _ {\bar {\nu}} = \{v: \mathbb {R} ^ {d} \to \mathbb {R} ^ {d}: \sup  _ {w \in \mathbb {R} ^ {d}} \| v (w) \| _ {2} \leq \bar {\nu} \},
$$
and the function class:
$$
\mathcal {F} _ {\bar {\nu}} = \left\{g (\cdot) = \mathbb {E} _ {w _ {0} \sim \mathcal {N} (0, I _ {d})} [ \langle v (w _ {0}), \cdot \rangle \mathbb {I} \{\langle w _ {0}, \cdot \rangle > 0 \} ]: v \in \mathcal {H} _ {\bar {\nu}} \right\}.
$$
Consider the following realizability assumption for the Q-function.
Assumption J.5 (Assumption 2 in [16]). For any  $t \geq 0$ , we assume that  $Q^t \in \mathcal{F}_{\bar{\nu}}$  for some  $\bar{\nu} > 0$ .
Theorem J.6 (Theorem 1 in [16]). Under Assumption 2 in [16], for any error probability  $\delta \in (0,1)$ , let
$$
\ell (m ^ {\prime}, \delta) = 4 \sqrt {\log (2 m ^ {\prime} + 1)} + 4 \sqrt {\log (T / \delta)},
$$
and  $R > \bar{\nu}$ . Then, for any target error  $\varepsilon > 0$ , number of iterations  $T' \in \mathbb{N}$ , network width
$$
m ^ {\prime} > \frac {1 6 \left(\bar {\nu} + \left(R + \ell (m ^ {\prime} , \delta)\right) \left(\bar {\nu} + R\right)\right) ^ {2}}{(1 - \gamma) ^ {2} \varepsilon^ {2}},
$$
and step-size
$$
\alpha_ {C} = \frac {\varepsilon^ {2} (1 - \gamma)}{(1 + 2 R) ^ {2}},
$$
Algorithm 3 in [16] yields the following bound:
$$
\mathbb {E} \Big [ \sqrt {\| \widehat {Q} ^ {t} - Q ^ {t} \| _ {L _ {2} (d _ {\mu} ^ {t} \cdot \pi^ {t})} ^ {2}} \mathbb {I} _ {A _ {2}} \Big ] \leq \frac {(1 + 2 R) \bar {\nu}}{\varepsilon (1 - \gamma) \sqrt {T ^ {\prime}}} + 3 \varepsilon ,
$$
where  $A_{2}$  holds with probability at least  $1 - \delta$  over the random initializations of the critic network  $\widehat{Q}^t$ .
As indicated in [16], a consequence of this result is that in order to achieve a target error less than  $\varepsilon > 0$ , a network width of  $m' = \widetilde{O}\left(\frac{\bar{\nu}^4}{\varepsilon^2}\right)$  and iteration complexity  $O\left(\frac{(1 + 2\bar{\nu})^2\bar{\nu}^2}{(1 - \gamma)^2\varepsilon^4}\right)$  suffice.
The statement of Theorem 1 in [16] can be readily applied to obtain the sample complexity of the critic.
Recalling Theorem E.1 in [39] and its assumptions Let  $g:\mathbb{R}^n\to \mathbb{R}$  be given and define the modulus of continuity  $\omega_{g}$  as
$$
\omega_ {g} (\delta) := \sup  _ {x, x ^ {\prime} \in \mathbb {R} ^ {n}} \left\{g (x) - g \left(x ^ {\prime}\right): \max  \left(\| x \| _ {2}, \| x ^ {\prime} \| _ {2}\right) \leq 1 + \delta , \| x - x ^ {\prime} \| _ {2} \leq \delta \right\}.
$$
If  $g$  is continuous, then  $\omega_g$  is not only finite for all inputs, but moreover  $\lim_{\delta \to 0} \omega_g(\delta) \to 0$ .
Denote  $\| p \|_{L_1} = \int |p(w)| dw$ . Define a sample from a signed density  $p: \mathbb{R}^{n+1} \to \mathbb{R}$  with  $\| p \|_{L_1} < \infty$  as  $(w, b, s)$ , where  $(w, b) \in \mathbb{R}$  is sampled from the probability density  $|p| / \| p \|_{L_1}$  and  $s = \text{sign}(p(w, b))$
Theorem J.7 (Theorem E.1 in [39]). Let  $g: \mathbb{R}^n \to \mathbb{R}$ ,  $\delta > 0$  and  $\omega_g(\delta)$  be as above and define for  $x \in \mathbb{R}^n$
$$
M := \sup  _ {\| x \| \leq 1 + \delta} | g (x) |, \qquad g _ {| \delta} (x) = f (x) \mathbb {I} [ \| x \| \leq 1 + \delta ], \qquad \alpha := \frac {\delta}{\sqrt {\delta} + \sqrt {2 \log (2 M / \omega_ {g} (\delta))}}.
$$
Let  $G_{\alpha}$  be a gaussian distribution on  $\mathbb{R}^n$  with mean 0 and variance  $\alpha^2\mathcal{I}$ . Define the Gaussian convolution  $l = g_{|\delta} * G_{\alpha}$  with Fourier transform  $\widehat{l}$  satisfying radial decomposition  $\widehat{l}(w) = |\widehat{l}(w)| \exp(2\pi i\theta_h(w))$ . Let  $P$  be a probability distribution supported on  $\| x \| \leq 1$ . Additionally define
$$
c := g (0) g (0) \int | \widehat {l} (w) | \big [ \cos (2 \pi (\theta_ {l} (w) - \| w \| _ {2})) - 2 \pi \| w \| _ {2} \sin (2 \pi (\theta_ {l} (w) - \| w \| _ {2})) \big ] d w
$$
$$
a = \int w | \widehat {l} (w) | d w
$$
$$
r = \sqrt {n} + 2 \sqrt {\log \frac {2 4 \pi^ {2} (\sqrt {d} + 7) ^ {2} \| g _ {| \delta} \| _ {L _ {1}}}{\omega_ {g} (\delta)}}
$$
$$
p := 4 \pi^ {2} | \widehat {l} (w) | \cos (2 \pi (\| w \| _ {2} - b)) \mathbb {I} [ | b | \leq \| w \| \leq r ],
$$
and for convenience create fake (weight, bias, sign) triples
$$
(w, b, s) _ {m + 1} := (0, c, m \text {s i g n} (c)), \quad (w, b, s) _ {m + 2} := (a, 0, m), \quad (w, b, s) _ {m + 3} := (- a, 0, - m).
$$
$$
| c | \leq M + 2 \sqrt {n} \| g _ {| \delta} \| _ {L _ {1}} (2 \pi \alpha^ {2}) ^ {- d / 2},
$$
$$
\left\| p \right\| _ {L _ {1}} \leq 2 \left\| g _ {| \delta} \right\| _ {L _ {1}} \sqrt {\frac {(2 \pi) ^ {3} n}{(2 \pi \alpha^ {2}) ^ {n + 1}}},
$$
and with probability at least  $1 - 3\lambda$  over a draw of  $((s_j, w_j, b_j))_{j=1}^m$  from  $p$
$$
\sqrt {\left\| g - \frac {1}{m} \sum_ {j = 1} ^ {m + 3} s _ {j} \sigma (\langle w _ {j} , x \rangle + b _ {j}) \right\| _ {L (P)}} \leq 3 \omega_ {g} (\delta) + \frac {r \| p \| _ {L _ {1}}}{\sqrt {m}} \left[ 1 + \sqrt {2 \log (1 / \lambda)} \right].
$$
We can then characterize the error of the actor by choosing  $x = (s,a)$ ,  $g = \widehat{Q}^t +\eta_t^{-1}\nabla h(\pi^t)$ , and  $f^{t + 1} = \frac{1}{m}\sum_{j = 1}^{m + 3}s_j\sigma (\langle w_j,x\rangle +b_j)$ . We can then make the actor error arbitrarily small by tuning the network width  $m$  and  $\delta$  (note that, since both  $\widehat{Q}^t$  and  $f^t$  are continuous neural networks,  $g$  is a continuous function).