SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
T _ {1} \geq \frac {1}{\lambda (\theta)} \log \left(\frac {2 M _ {2}}{K _ {1} \delta}\right), \tag {18}
$$
where  $M_2 = 1 + \max_{s \in S} c(s, \pi_\theta(s)) > 0$  is the maximum of the continuous function  $c(\cdot, \pi_\theta(\cdot))$  over  $S$ , then applying the Lipschitz condition of  $c(\cdot, \pi_\theta(\cdot))$  yields
$$
\begin{array}{l} \sum_ {t = 0} ^ {T _ {1}} \gamma^ {t} | c (s _ {t}, \pi_ {\theta} (s _ {t})) - c (s _ {t} ^ {\prime}, \pi_ {\theta} (s _ {t} ^ {\prime})) | \leq \sum_ {t = 0} ^ {T _ {1}} \gamma^ {t} K _ {2} \| s _ {t} - s _ {t} ^ {\prime} \| \\ \leq \sum_ {t = 0} ^ {T _ {1}} K _ {1} K _ {2} e ^ {(\lambda (\theta) + \log \gamma) t} \delta \\ \leq K _ {1} K _ {2} \delta \frac {e ^ {\frac {\lambda (\theta) + \log \gamma}{\lambda (\theta)} \log \left(\frac {2 M _ {2}}{K _ {1} \delta}\right)} + 2 (\lambda (\theta) + \log \gamma)}{e ^ {(\lambda (\theta) + \log \gamma)} - 1} \\ = \frac {e ^ {2 (\lambda (\theta) + \log \gamma)} K _ {2} K _ {1} ^ {\frac {- \log \gamma}{\lambda (\theta)}} (2 M _ {2}) ^ {1 + \frac {\log \gamma}{\lambda (\theta)}}}{e ^ {(\lambda (\theta) + \log \gamma)} - 1} \delta^ {\frac {- \log \gamma}{\lambda (\theta)}} \\ \end{array}
$$
where  $K_{2} > 0$  is the Lipschitz constant of  $c(\cdot ,\pi_{\theta}(\cdot))$  over compact set  $S$
On the other hand, the tail terms in  $J(\cdot)$  is bounded by
$$
\begin{array}{l} \sum_ {t = T _ {1} + 1} ^ {\infty} \gamma^ {t} | c (s _ {t}, \pi_ {\theta} (s _ {t})) - c (s _ {t} ^ {\prime}, \pi_ {\theta} (s _ {t} ^ {\prime})) | \leq \sum_ {t = T _ {1} + 1} ^ {\infty} 2 M _ {2} \gamma^ {t} \\ \leq \sum_ {t = T _ {1}} ^ {\infty} 2 M _ {2} \gamma^ {t} \\ = 2 M _ {2} \frac {\gamma^ {T _ {1}}}{1 - \gamma} \\ \leq \frac {2 M _ {2}}{1 - \gamma} \left(\frac {K _ {1}}{2 M _ {2}}\right) ^ {\frac {- \log \gamma}{\lambda (\theta)}} \delta^ {\frac {- \log \gamma}{\lambda (\theta)}} \\ \end{array}
$$
using  $|c(s_{t},\pi_{\theta}(s_{t})) - c(s_{t}^{\prime},\pi_{\theta}(s_{t}^{\prime}))| \leq 2M_{2}$ . Combining the above two inequalities yields
$$
\begin{array}{l} \left| V ^ {\pi_ {\theta}} \left(s _ {0} ^ {\prime}\right) - V ^ {\pi_ {\theta}} \left(s _ {0}\right) \right| \\ \leq \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left| c \left(s _ {t}, \pi_ {\theta} \left(s _ {t}\right)\right) - c \left(s _ {t} ^ {\prime}, \pi_ {\theta} \left(s _ {t} ^ {\prime}\right)\right) \right| \\ = \sum_ {t = 0} ^ {T _ {1}} \gamma^ {t} | c (s _ {t}, \pi_ {\theta} (s _ {t})) - c (s _ {t} ^ {\prime}, \pi_ {\theta} (s _ {t} ^ {\prime})) | + \sum_ {t = T _ {1} + 1} ^ {\infty} \gamma^ {t} | c (s _ {t}, \pi_ {\theta} (s _ {t})) - c (s _ {t} ^ {\prime}, \pi_ {\theta} (s _ {t} ^ {\prime})) | \\ \leq \left(\frac {e ^ {2 (\lambda (\theta) + \log \gamma)} K _ {2} K _ {1} ^ {\frac {- \log \gamma}{\lambda (\theta)}} (2 M _ {2}) ^ {1 + \frac {\log \gamma}{\lambda (\theta)}}}{e ^ {(\lambda (\theta) + \log \gamma)} - 1} + \frac {2 M _ {2}}{1 - \gamma} \left(\frac {K _ {1}}{2 M _ {2}}\right) ^ {\frac {- \log \gamma}{\lambda (\theta)}}\right) \delta^ {\frac {- \log \gamma}{\lambda (\theta)}} \\ \end{array}
$$
Proof. For the ease of notation, let  $s_t = s_t^\theta$ ,  $s_t' = s_t^{\theta'}$ ,  $u(s) = \pi_\theta(s)$  and  $u'(s) = \pi_{\theta'}(s)$ .
$$
\begin{array}{l} V ^ {\pi_ {\theta^ {\prime}}} (s _ {0}) - V ^ {\pi_ {\theta}} (s _ {0}) = \sum_ {t = 0} ^ {\infty} \gamma^ {t} c \left(s _ {t} ^ {\prime}; u ^ {\prime} \left(s _ {t} ^ {\prime}\right)\right) - V ^ {\pi_ {\theta}} (s _ {0}) \\ = \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(c \left(s _ {t} ^ {\prime}; u ^ {\prime} \left(s _ {t} ^ {\prime}\right)\right) + V ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}\right) - V ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}\right)\right) - V ^ {\pi_ {\theta}} \left(s _ {0}\right) \\ = \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(c \left(s _ {t} ^ {\prime}; u ^ {\prime} \left(s _ {t} ^ {\prime}\right)\right) + \gamma V ^ {\pi_ {\theta}} \left(s _ {t + 1} ^ {\prime}\right) - V ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}\right) + V ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}\right) - \gamma V ^ {\pi_ {\theta}} \left(s _ {t + 1} ^ {\prime}\right)\right) - V ^ {\pi_ {\theta}} \left(s _ {0}\right) \\ = \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(Q ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}, u ^ {\prime} \left(s _ {t} ^ {\prime}\right)\right) - V ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}\right)\right) + \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(V ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}\right) - \gamma V ^ {\pi_ {\theta}} \left(s _ {t + 1} ^ {\prime}\right)\right) - V ^ {\pi_ {\theta}} \left(s _ {0}\right). \\ \end{array}
$$
Using the fact that  $\gamma^t V^{\pi_\theta}(x_{t+1}') \to 0$  as  $t \to \infty$  from (A.3) yields
$$
\begin{array}{l} V ^ {\pi_ {\theta^ {\prime}}} \left(s _ {0}\right) - V ^ {\pi_ {\theta}} \left(s _ {0}\right) = \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(Q ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}, u ^ {\prime} \left(s _ {t} ^ {\prime}\right)\right) - V ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}\right)\right) + V ^ {\pi_ {\theta}} \left(s _ {0} ^ {\prime}\right) - V ^ {\pi_ {\theta}} \left(s _ {0}\right) \\ = \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(Q ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}, u ^ {\prime} \left(s _ {t} ^ {\prime}\right)\right) - V ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}\right)\right) \\ \end{array}
$$
and the proof is completed using  $s_0' = s_0$ .
B.3 Proof of Theorem 4.2
Proof. First, we will show that  $Q^{\pi_{\theta}}(s,a)$  is  $\frac{-\log\gamma}{\lambda(\theta)}$ -Hölder continuous with respect to  $a$ . Note that for any given  $a \in \mathcal{A}$  and any  $a' \in \mathcal{A}$  such that  $\| a - a' \| \ll 1$ ,
$$
\begin{array}{l} \left| Q ^ {\pi_ {\theta}} (s, a) - Q ^ {\pi_ {\theta}} \left(s, a ^ {\prime}\right) \right| \leq \left| c (s, a) - c \left(s, a ^ {\prime}\right) \right| + \gamma \left| V ^ {\pi_ {\theta}} \left(f (s, a)\right) - V ^ {\pi_ {\theta}} \left(f \left(s, a ^ {\prime}\right)\right) \right| \\ \leq K _ {1} \| a - a ^ {\prime} \| + \gamma \| f (s, a) - f (s, a ^ {\prime}) \| ^ {- \frac {\log \gamma}{\lambda (\theta)}} \\ \leq K _ {1} \| a - a ^ {\prime} \| + \gamma K _ {2} \| a - a ^ {\prime} \| ^ {- \frac {\log \gamma}{\lambda (\theta)}} \\ \leq K _ {3} \| a - a ^ {\prime} \| ^ {- \frac {\log \gamma}{\lambda (\theta)}} \\ \end{array}
$$
for some  $K_{3} > 0$  using the locally Lipschitz continuity of  $c$  and  $f$ .
Note that  $V^{\pi_{\theta}}(s) = Q^{\pi_{\theta}}(s, \pi_{\theta}(s))$ , combining it with Lemma 4.1 yields
$$
\begin{array}{l} | J \left(\theta^ {\prime}\right) - J (\theta) | \leq \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left| Q ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}, u ^ {\prime} \left(s _ {t} ^ {\prime}\right)\right) - V ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}\right) \right| \\ = \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left| Q ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}, \pi_ {\theta^ {\prime}} \left(s _ {t} ^ {\prime}\right)\right) - Q ^ {\pi_ {\theta}} \left(s _ {t} ^ {\prime}, \pi_ {\theta} \left(s _ {t} ^ {\prime}\right)\right) \right| \\ \leq \sum_ {t = 0} ^ {\infty} \gamma^ {t} K _ {3} \| \pi_ {\theta^ {\prime}} (s _ {t} ^ {\prime}) - \pi_ {\theta} (s _ {t} ^ {\prime}) \| ^ {- \frac {\log \gamma}{\lambda (\theta)}} \\ \leq \sum_ {t = 0} ^ {\infty} \gamma^ {t} K _ {3} K _ {4} \| \theta^ {\prime} - \theta \| ^ {\frac {- \log \gamma}{\lambda (\theta)}} \\ = \frac {K _ {3} K _ {4}}{1 - \gamma} \| \theta^ {\prime} - \theta \| ^ {\frac {- \log \gamma}{\lambda (\theta)}} \\ \end{array}
$$
using the fact that  $\pi_{\theta}(s)$  is Lipschitz continuous in a neighborhood of  $(\theta ,s)\in \mathbb{R}^{N}\times \mathcal{S}$  for some constant  $K_{4} > 0$  and we complete the proof.
C From the perspective of fractal theory
We will go through some basic concepts in fractal theory that are related to the study of non-smooth functions.
C.1 The Hausdorff dimension
We will show that the Hausdorff dimension is well-defined: First, it is clear that when  $\delta < 1$ ,  $\mathcal{H}_{\delta}^{s}(F)$  is non-increasing with respect to  $s$ . Thus,  $\mathcal{H}^s (F)$  is non-increasing as well. Let  $s\geq 0$  such that  $\mathcal{H}^s (F) < \infty$ , then for any  $t > s$  and any  $\delta$ -cover  $\{U_i\}$  of  $F$ , we have
$$
\sum_ {i = 1} ^ {\infty} \left| U _ {i} \right| ^ {t} \leq \delta^ {t - s} \sum_ {i = 1} ^ {\infty} \left| U _ {i} \right| ^ {s}
$$
which implies  $\mathcal{H}^t (F) = 0$  by taking infimum on both sides and letting  $\delta \rightarrow 0$ . Therefore, the set  $\{s\geq 0:0 < \mathcal{H}^s (F) < \infty \}$  contains at most one point, which further implies  $\inf \{s\geq 0:\mathcal{H}^s (F) = 0\} = \sup \{s\geq 0:\mathcal{H}^s (F) = \infty \}$ .
More details regarding the well-posedness of Hausdorff dimension can be found in [3, 8]. In particular, one can easily verify that the Hausdorff dimension coincides with the standard dimension (i.e.  $s \in \mathbb{N}$ ) when  $F$  is a regular manifold. Typically, the Hausdorff dimension of a fractal is not an integer, and we will be exploiting this fact through the section. A famous example is the Weierstrass function as shown in Figure 7. A comparison of Figure 2e and Figure 7c (they have the same scale) gives some sense about how non-smooth the objective function can be in practice.
(a) The double sector  $S(x,\phi ,\psi)$
Figure 7: (a) shows how the double sector looks like. In (b) and (c), the Weierstrass function is given by  $W(x) = \sum_{n=0}^{\infty} a^n \cos(b^n \pi x)$  where  $a = 0.6, b = 7$ . The Hausdorff dimension of its loss curve is calculated as  $\dim_H \mathcal{L}_W = 2 + \log_b a \simeq 1.73$ . Also, according to [12], such  $W(x)$  is nowhere differentiable when  $0 < a < 1$  and  $ab \geq 1$ .
(b) The  $W(x)$  over  $x\in [-2,2]$
(c) Magnified:  $x \in [0.5, 0.5001]$ .
C.2 Non-existence of tangent plane
Actually, when  $J(\cdot)$  is Lipschitz continuous on any compact subset of  $\mathbb{R}^N$ , by the Rademacher's Theorem, we know that it is differentiable almost everywhere which implies the existence of tangent plane. As it comes to fractal landscapes, however, the tangent plane itself does not exist for almost every  $\theta \in \mathbb{R}^N$ , which makes all policy gradient algorithms ill-posed. Although similar results were obtained for higher-dimensional cases as in [26], we focus on the two-dimensional case so that it provides a more direct geometric intuition. First, we introduce the notion of  $s$ -sets:
Definition C.1. Let  $F \subset \mathbb{R}^2$  be a Borel set and  $s \geq 0$ , then  $F$  is called an  $s$ -set if  $0 < \mathcal{H}^s(F) < \infty$ .
The intuition is that: when the dimension of fractal  $F$  is a fraction between 1 and 2, then there is no direction along which a significant part of  $F$  concentrates within a small double sector with vertex  $x$  as shown in Figure 7a. To be precise, let  $S(x, \phi, \psi)$  denote the double sector and  $r > 0$ , then we say that  $F$  has a tangent at  $x \in F$  if there exists a direction  $\phi$  such that for every angle  $\phi > 0$ , it has
1.  $\lim \sup_{r\to 0}\frac{\mathcal{H}^s(F\cap\mathcal{B}(x,r))}{(2r)^s} >0;$
2.  $\lim_{r\to 0}\frac{\mathcal{H}^s(F\cap(\mathcal{B}(x,r)\backslash S(x,\phi,\psi)))}{(2r)^s} = 0;$
where the first condition states that the set  $F$  behaves like a fractal around  $x$ , and the second condition implies that the part of  $F$  lies outside of any double sector  $S(x,\phi ,\psi)$  is negligible when  $r\to 0$ . Then, the main result is as follows:
Proposition C.1. (Non-existence of tangent planes, [8]) If  $F \subset \mathbb{R}^2$  is an  $s$ -set with  $1 < s < 2$ , then at almost all points of  $F$ , no tangent exists.
Therefore, "estimate the gradient" no longer makes sense since there does not exist a tangent line/plane at almost every point on the loss surface. This means that all policy gradient algorithms are ill-posed since there is no gradient for them to estimate at all.
C.3 Accumulated uncertainty
Another issue that may emerge during training process is the accumulation of uncertainty. To see how the uncertainty entered at each step accumulates and eventually blows up when generating a path along fractal boundaries, let us consider the following toy problem: Suppose that the distance between the initial point  $\theta_0\in \mathbb{R}^N$  and the target  $\theta^{*}$  is  $d > 0$ , and step size  $\delta_{k} > 0$  is adapted at the  $k$ -th step, as shown in Figure 8a. If there exists  $c > 0$  such that the projection  $\langle \theta^{*} - \theta_{0},\theta_{k + 1} - \theta_{k}\rangle \geq cd\delta_{k}$  for all  $k\in \mathbb{N}$  which implies that the angle between the direction from  $\theta_{k}$  to  $\theta_{k + 1}$  and the true direction  $\theta^{*} - \theta_{0}$  does not exceed  $\arccos (c)$ , in this case, a successful path  $\{\theta_k\}$  that converges to  $\theta^{*}$  should give
$$
\sum_ {k = 0} ^ {\infty} c d \delta_ {k} \leq \sum_ {k = 0} ^ {\infty} \langle \theta^ {*} - \theta_ {0}, \theta_ {k + 1} - \theta_ {k} \rangle = \langle \theta^ {*} - \theta_ {0}, \theta^ {*} - \theta_ {0} \rangle = d ^ {2}
$$
using  $\theta_{k}\to \theta^{*}$  as  $k\rightarrow \infty$  , which is equivalent to  $\sum_{k = 0}^{\infty}\delta_k\leq \frac{d}{c}$
(a) Generating path from  $\theta_0$  to  $\theta^{*}$ .
(b) Update  $\theta_{n + 1}$  from  $\theta_{n}$
Figure 8: Illustrations of the statistical challenges in implementing policy gradient algorithms on a fractal loss surface.
On the other hand, when walking on the loss surface, it is not guaranteed to follow the correct direction precisely all the time. For any small step size  $\delta > 0$ , the uncertainty fraction  $u(\delta)$  involved in every single step can be estimated by the following result [20]:
Proposition C.2. Let  $\delta > 0$  be the step size and  $\beta = N + 1 - \dim_H J$  where  $\dim_H J$  is the Hausdorff dimension of loss surface of  $J(\cdot)$ , then the uncertainty  $u(\delta) \sim \mathcal{O}(\delta^{\beta})$  when  $\delta \ll 1$ .
Therefore, we may assume that there exists another  $c' > 0$  such that the uncertainty  $U_{k}$  at the  $k$ -th step has  $U_{k} \leq c'\delta_{k}^{\beta}$  for all  $k = 0,1,\ldots$ . Then, the accumulated uncertainty
$$
U = \sum_ {k = 0} ^ {\infty} U _ {k} \leq c ^ {\prime} \sum_ {k = 0} ^ {\infty} \delta_ {k} ^ {\beta}
$$
is bounded when  $\beta = 1$  (i.e. boundary is smooth) using the earlier result  $\sum_{k=0}^{\infty} \delta_k \leq \frac{d}{c}$ . However, the convergence of  $\sum_{k=0}^{\infty} \delta_k$  no longer guarantees the convergence of  $\sum_{k=0}^{\infty} \delta_k^\beta$  when  $\beta < 1$ , and a counterexample is the following series:
$$
\delta_ {k} = \frac {1}{k (\log (k + 2)) ^ {2}}
$$
for all  $k = 0,1,\ldots$ , which implies the uncertainty accumulated over the course of iterations may increase dramatically and eventually cause the sequence  $\theta_{k}$  to become random when walking on fractal boundaries.