SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tao Wang
UC San Diego
taw003@ucsd.edu
Sylvia Herbert
sherbert@ucsd.edu
Sicun Gao
sicung@ucsd.edu
Policy gradient lies at the core of deep reinforcement learning (RL) in continuous domains. Despite much success, it is often observed in practice that RL training with policy gradient can fail for many reasons, even on standard control problems with known solutions. We propose a framework for understanding one inherent limitation of the policy gradient approach: the optimization landscape in the policy space can be extremely non-smooth or fractal for certain classes of MDPs, such that there does not exist gradient to be estimated in the first place. We draw on techniques from chaos theory and non-smooth analysis, and analyze the maximal Lyapunov exponents and Hölder exponents of the policy optimization objectives. Moreover, we develop a practical method that can estimate the local smoothness of objective function from samples to identify when the training process has encountered fractal landscapes. We show experiments to illustrate how some failure cases of policy optimization can be explained by such fractal landscapes.
Deep reinforcement learning has achieved much success in various applications [23, 30, 38], but they also often fail, especially in continuous spaces, on control problems that other methods can readily solve. The understanding of such failure cases is still limited. For instance, the training process of reinforcement learning is unstable and the learning curve can fluctuate during training in ways that are hard to predict. The probability of obtaining satisfactory policies can also be inherently low in reward-sparse or highly nonlinear control tasks. Existing analysis of the failures focuses on limitations of the sampling and optimization algorithms, such as function approximation errors [35, 39], difficulty in data collection [34], and aggressive updates in the policy space [28]. There has not been much study of potentially deeper causes of failures that may be inherent in the formulation of policy optimization problems.
Motivated by the common observation that small updates in the policy parameters can significantly change the performance, we analyze the smoothness of the optimization landscapes in policy optimization. Drawing on chaos theory, we introduce the concept of maximal Lyapunov exponent (MLE) [17] to the RL setting to measure the exponential rate of trajectory divergence in MDP. It seems contradictory that a trajectory in chaotic systems can be both exponentially divergent and uniformly bounded at the same time, and we will show that these two conflicting facts combine to yield the fractal structure in the optimization landscape. Intuitively, the objective function is non-differentiable when the rate of trajectory divergence exceeds the decay rate of discount factor. Furthermore, this finding indicates that the fluctuations observed in the loss curve are not just due to the numerical or sampling error but rather reflect the intrinsic properties of the corresponding MDP.
We should emphasize that the fractal landscapes that we will demonstrate are stronger than various existing results on the non-smoothness [2, 7]. Most non-smooth objectives that have been studied still assume is local Lipschitz continuity or piecewise smoothness that implies differentiability almost everywhere (such as  $f(x) = |x|$ ). Instead, by showing that the loss landscape of policy optimization can be fractal, we demonstrate the absence of descent directions, which causes the failure of first-order
methods in general. Since such behavior is an intrinsic property of the underlying dynamical systems, the results show fundamental limitations of policy gradient methods on certain classes of MDPs.
The paper is organized as follows. In Section 3 and 4, we will introduce the preliminaries and develop the theory for deterministic policies. In particular, we show that the optimization landscape is fractal, even when all elements within the MDP are deterministic. Next, we consider stochastic policies and provide an example to show how non-smoothness can still occur if without additional assumptions. In Section 5, we turn the theoretical analysis into a practical sampling-based method for estimating the Hölder exponent to determine whether the optimization objective is differentiable at a specific parameter vector. It can also indicate if the training process has encountered fractal regions by comparing the regression slope with some fixed threshold. In Section 6, we show experiments that demonstrate and compare the landscapes of different MDPs.
Policy gradient and Q-learning methods. Policy gradient methods [33, 41] formulate RL as an optimization problem in the parameter space, with many variations such as natural policy gradient [16], deterministic policy gradient [29], deep deterministic policy gradient [18], trust region policy optimization [27] and proximal policy optimization [28], were proposed. As all of these algorithms aim to estimate the gradient of the objective function over the policy parameters, they become ill-posed when the objective is non-differentiable, which is the focus of our analysis.
Another popular approach for model-free RL is Q-learning methods, which approximate the Q-function of the policy at each step [22, 40]. As neural networks become more and more popular, they are employed as function approximators in deep Q-learning algorithms [9, 13, 37]. Since the foundation of Q-learning methods is established upon the estimation of value functions, a poor approximation can completely ruin the entire training process. In this paper, we will show that the value functions in a certain class of MDPs exhibit significant non-smoothness, making them challenging to represent using existing methods.
Chaos in machine learning. Chaotic behaviors due to randomness in the learning dynamics have been reported in other learning problems [6, 21, 25]. For instance, when training recurrent neural networks for a long period, the outcome behaves like a random walk due to the problems of vanishing and the exploding gradients [4]. It served as motivation for the work [24], which points out that the chaotic behavior in finite-horizon model-based reinforcement learning problems may be caused by long chains of nonlinear computation. A similar observation was made in [31]. However, we show that in RL, the objective function is provably smooth if the time horizon is finite and the underlying dynamics is differentiable. Instead, we focus on the general context of infinite-horizon problems in MDPs, in which case the objective function can become non-differentiable.
Loss landscape of policy optimization. It has been shown that the objective functions in finite state-space MDPs are smooth [1, 42], which enables the use of gradient-based methods and direct policy search. It also explains why the classical RL algorithms in [32] are provably efficient in finite space settings. Also, such smoothness results can be extended to some continuous state-space MDPs with special structures. For instance, the objective function in Linear Quadratic Regulator (LQR) problems is almost smooth [10] as long as the cost is finite. Similar results are obtained for the  $\mathcal{H}_2 / \mathcal{H}_{\infty}$  problem [43]. For the robust control problem, although the objective function may not be smooth, it is locally Lipschitz continuous, which implies differentiability almost everywhere, and further leads to global convergence of direct policy search [11]. There is still limited theoretical study of loss landscapes of policy optimization for nonlinear and complex MDPs. We aim to partially address this gap by pointing out the possibility that the loss landscape can be highly non-smooth and even fractal, which is far more complex than the previous cases.
3.1 Dynamical Systems as Markov Decision Processes
We consider Markov Decision Processes (MDPs) that encode continuous control problems for dynamical systems defined by difference equations of the form:
$$
s _ {t + 1} = f \left(s _ {t}, a _ {t}\right), \tag {1}
$$
where  $s_t \in S \subset \mathbb{R}^n$  is the state at time  $t$ ,  $s_0$  is the initial state and  $a_t \sim \pi_\theta(\cdot | s_t) \in \mathcal{A} \subset \mathbb{R}^m$  is the action taken at time  $t$  based on a policy parameterized by  $\theta \in \mathbb{R}^p$ . We assume that both the state space  $S$  and the action space  $\mathcal{A}$  are compact. The objective function of the RL problem to minimize is defined by  $V^{\pi_\theta}$  of policy  $\pi_\theta$ :
$$
J (\theta) = V ^ {\pi_ {\theta}} \left(s _ {0}\right) = \mathbb {E} _ {a _ {t} \sim \pi_ {\theta} \left(\cdot \mid s _ {t}\right)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} c \left(s _ {t}, a _ {t}\right) \right], \tag {2}
$$
where  $\gamma \in (0,1)$  is the discount factor and  $c(s,a)$  is the cost function. The following assumptions are made throughout this paper:
- (A.1)  $f: \mathbb{R}^n \times \mathbb{R}^m \to \mathbb{R}^n$  is Lipschitz continuous over any compact domains (i.e., locally Lipschitz continuous);
- (A.2) The cost function  $c: \mathbb{R}^n \times \mathbb{R}^m \to \mathbb{R}$  is non-negative and locally Lipschitz continuous everywhere;
- (A.3) The state space is closed under transitions, i.e., for any  $(s,a)\in S\times \mathcal{A}$ , the next state  $s^{\prime} = f(s,a)\in \mathcal{S}$ .
3.2 Policy gradient methods
Policy gradient methods estimate the gradient of the objective  $J(\cdot)$  with respect to the parameters of the policies. A commonly used form is
$$
\nabla J (\theta) = \mathbb {E} _ {a _ {t} \sim \pi_ {\theta} (\cdot | s _ {t})} \left[ \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) A ^ {\pi_ {\theta}} \left(s _ {t}, a _ {t}\right) \right], \tag {3}
$$
where  $\pi_{\theta}(\cdot |\cdot)$  is a stochastic policy parameterized by  $\theta$ .  $A^{\pi_{\theta}}(s,a) = Q^{\pi_{\theta}}(s,a) - V^{\pi_{\theta}}(s)$  is the advantage function often used for variance reduction and  $Q^{\pi_{\theta}}(\cdot ,\cdot)$  is the  $Q$ -value function of  $\pi_{\theta}$ . The theoretical guarantee of the convergence of policy gradient methods is typically established by the argument that the tail term  $\gamma^{t}\nabla_{\theta}V^{\pi_{\theta}}(s)$  diminishes as  $t$  increases, for any  $s\in S$  [33]. For such claims to hold, two assumptions are needed:
-  $\nabla_{\theta}V^{\pi_{\theta}}(s)$  exists and is continuous for all  $s\in S$
-  $\|\nabla_{\theta} V^{\pi_{\theta}}(s)\|$  is uniformly bounded over  $S$ .
The second assumption is automatically satisfied if the first assumption holds in the case that  $S$  is either finite or compact. However, as we will see in Section 4 and 6, the existence of  $\nabla_{\theta}V^{\pi_{\theta}}(\cdot)$  may fail in many continuous MDPs even if  $S$  is compact, which challenges the fundamental well-posedness of policy gradient methods.
3.3 Maximal Lyapunov Exponents
Behaviors of chaotic systems have sensitive dependence on their initial conditions. To be precise, consider the system  $s_{t + 1} = F(s_t)$  with initial state  $s_0\in \mathbb{R}^n$ , and suppose that a small perturbation  $\Delta Z_0$  is made to  $s_0$ . The divergence from the original trajectory of the system under this perturbation at time  $t$ , say  $\Delta Z(t)$ , can be estimated by  $\| \Delta Z(t)\| \simeq e^{\lambda t}\| \Delta Z_0\|$  with some  $\lambda$  that is called the Lyapunov exponent. For chaotic systems, Lyapunov exponents are typically positive, which implies an exponential divergence rate of the separation of nearby trajectories [19]. Since the Lyapunov exponent at a given point may depend on the direction of the perturbation  $\Delta Z_0$ , and we are interested in identifying the largest divergence rate, the maximal Lyapunov exponent (MLE) is formally defined as follows:
Definition 3.1. (Maximal Lyapunov exponent) For the dynamical system  $s_{t + 1} = F(s_t), s_0 \in \mathbb{R}^n$ , the maximal Lyapunov exponent  $\lambda_{\mathrm{max}}$  at  $s_0$  is defined as the largest value such that
$$
\lambda_ {\max } = \lim  _ {t \rightarrow \infty} \sup  _ {\| \Delta Z _ {0} \| \rightarrow 0} \frac {1}{t} \log \frac {\| \Delta Z (t) \|}{\| \Delta Z _ {0} \|}. \tag {4}
$$
Note that systems with unstable equilibria, not necessarily chaotic, can have positive MLEs.
3.4 Fractal Landscapes
The Hausdorff dimension is the most fundamental concept in fractal theory. We first introduce the concept of  $\delta$ -cover and Hausdorff measure:
Definition 3.2. ( $\delta$ -cover) Let  $\{U_i\}$  be a countable collection of sets of diameter at most  $\delta$  (i.e.  $|U_i| = \sup \{\|x - y\| : x, y \in U_i\} \leq \delta$ ) and  $F \subset \mathbb{R}^N$ , then  $\{U_i\}$  is a  $\delta$ -cover of  $F$  if  $F \subset \cup_{i=1}^{\infty} U_i$ .
Definition 3.3. (Hausdorff measure) For any  $F \subset \mathbb{R}^N$  and  $s \geq 0$ , let
$$
\mathcal {H} _ {\delta} ^ {s} (F) = \inf  \{\sum_ {i = 1} ^ {\infty} | U _ {i} | ^ {s}: \{U _ {i} \} i s a \delta - c o v e r o f F \}.
$$
Then we call the limit  $\mathcal{H}^s (F) = \lim_{\delta \to 0}\mathcal{H}_\delta^s (F)$  the  $s$  -dimensional Hausdorff measure of  $F$
The definition of Hausdorff dimension follows immediately:
Definition 3.4. (Hausdorff dimension) Let  $F\subset \mathbb{R}^N$  be a subset, then its Hausdorff dimension
$$
\dim_ {H} F = \inf  \{s \geq 0: \mathcal {H} ^ {s} (F) = 0 \} = \sup  \{s \geq 0: \mathcal {H} ^ {s} (F) = \infty \}.
$$
And we introduce the notion of  $\alpha$ -Hölder continuity that extends the concept of Lipschitz continuity:
Definition 3.5. ( $\alpha$ -Hölder continuity) Let  $\alpha > 0$  be a scalar. A function  $g: \mathbb{R}^N \to \mathbb{R}$  is  $\alpha$ -Hölder continuous at  $x \in \mathbb{R}^N$  if there exist  $C > 0$  and  $\delta > 0$  such that
$$
| g (x) - g (y) | \leq C \| x - y \| ^ {\alpha}
$$
for all  $y \in \mathcal{B}(x, \delta)$ , where  $\mathcal{B}(x, \delta)$  denotes the open ball of radius  $\delta$  centered at  $x$ .
The definition reduces to Lipschitz continuity when  $\alpha = 1$ . A function is not differentiable, if the largest Hölder exponent at a given point is less than 1. Just as smoothness is commonly associated with Lipschitz continuity, fractal behavior is closely related to Hölder continuity. In particular, for an open set  $F \subset \mathbb{R}^k$  and a continuous mapping  $\eta: F \to \mathbb{R}^p$  with  $p > k$ , the image set  $\eta(F)$  is fractal when its Hausdorff dimension  $\dim_H \eta(F)$  is strictly greater than  $k$ , which occurs when  $\eta: F \to \mathbb{R}^p$  is  $\alpha$ -Hölder continuous with exponent  $\alpha < 1$ :
Proposition 3.1. ([8]) Let  $F \subset \mathbb{R}^k$  be a subset and suppose that  $\eta : F \to \mathbb{R}^p$  is  $\alpha$ -Hölder continuous where  $\alpha > 0$ , then  $\dim_H \eta(F) \leq \frac{1}{\alpha} \dim_H F$ .
It implies that if the objective function is  $\alpha$ -Hölder for some  $\alpha < 1$ , its loss landscape  $\mathcal{L}_J = \{(\theta, J(\theta)) \in \mathbb{R}^{N+1} : \theta \in \mathbb{R}^N\}$  can be fractal. Further discussion of the theory on fractals can be found in Appendix C.
4 Fractal Landscapes in the Policy Space
In this section, we will show that the objective  $J(\theta)$  in policy optimization can be non-differentiable when the system has positive MLEs. We will first consider Hölder continuity of  $V^{\pi_{\theta}}(\cdot)$  and  $J(\cdot)$  with deterministic policies in 4.1 and 4.2, and then discuss the case of stochastic policies in 4.3.
4.1 Hölder Exponent of  $V^{\pi_{\theta}}(\cdot)$
We first consider a deterministic policy  $\pi_{\theta}$  that maps states to actions  $a = \pi_{\theta}(s)$  instead of distributions. Consider a fixed policy parameter  $\theta \in \mathbb{R}^p$  such that the MLE of (1), namely  $\lambda (\theta)$ , is greater than  $-\log \gamma$ . Let  $s_0^\prime \in S$  be another initial state that is close to  $s_0$ , i.e.,  $\delta = \| s_0' - s_0\| > 0$  is small enough. According to the assumption (A.3) and the compactness of the state space, we can find a constant  $M > 0$  such that both  $\| s_t\| \leq M$  and  $\| s_t'\| \leq M$  for all  $t \in \mathbb{N}$ , where  $\{s_t\}_{t=1}^{\infty}$  and  $\{s_t'\}_{t=1}^{\infty}$  are the trajectories starting from  $s_0$  and  $s_0'$ , respectively. Motivated by (4), we further make the following assumptions:
- (A.4) There exists  $K_{1} > 0$  such that  $\| s_t' - s_t\| \leq K_1\delta e^{\lambda (\theta)t}$  for all  $t\in \mathbb{N}$  and  $\delta = \| s_0^{\prime} - s_0\| >0$
- (A.5) The policy  $\pi : \mathbb{R}^N \times \mathbb{R}^n \to \mathbb{R}^m$  is locally Lipschitz continuous everywhere.
We then have following theorem, and it provides a lower bound for the Hölder exponent of  $J$  whose detailed proof can be found in Appendix B.1.
Theorem 4.1. (Non-smoothness of  $V^{\pi_{\theta}}$ ) Assume (A.1)-(A.5) and the parameterized policy  $\pi_{\theta}(\cdot)$  is deterministic. Let  $\lambda(\theta)$  denote the MLE of (1) at  $\theta \in \mathbb{R}^{N}$ . Suppose that  $\lambda(\theta) > -\log \gamma$ , then  $V^{\pi_{\theta}}(\cdot)$  is  $\frac{-\log \gamma}{\lambda(\theta)}$ -Hölder continuous at  $s_0$ .
Proof sketch of Theorem 4.1: Suppose that  $p \in (0,1]$  is some constant for which we would like to prove that  $V^{\pi_{\theta}}(s)$  is  $p$ -Hölder continuous at  $s = s_0$ , and here we take  $p = -\frac{\log \gamma}{\lambda(\theta)}$ .
According to Definition 3.5, it suffices to find some  $C' > 0$  such that  $|V^{\pi_{\theta}}(s_0') - V^{\pi_{\theta}}(s_0)| \leq C' \delta^p$  when  $\delta = \| s_0 - s_0' \| \ll 1$ . Consider the relaxed form
$$
\left| V ^ {\pi_ {\theta}} \left(s _ {0} ^ {\prime}\right) - V ^ {\pi_ {\theta}} \left(s _ {0}\right) \right| \leq \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left| c \left(s _ {t}, \pi_ {\theta} \left(s _ {t}\right)\right) - c \left(s _ {t} ^ {\prime}, \pi_ {\theta} \left(s _ {t} ^ {\prime}\right)\right) \right| \leq C ^ {\prime} \delta^ {p}. \tag {5}
$$
Now we split the entire series into three parts as shown in Figure 1: the sum of first  $T_{2}$  terms, the sum from  $t = T_{2} + 1$  to  $T_{3} - 1$ , and the sum from  $t = T_{3}$  to  $\infty$ . First, applying (A.4) to the sum of the first  $T_{2}$  terms yields
$$
\sum_ {t = 0} ^ {T _ {2}} \gamma^ {t} \left| c \left(s _ {t}, \pi_ {\theta} \left(s _ {t}\right)\right) - c \left(s _ {t} ^ {\prime}, \pi_ {\theta} \left(s _ {t} ^ {\prime}\right)\right) \right| \leq \frac {e ^ {(\lambda (\theta) + \log \gamma) T _ {2}}}{1 - \gamma} K _ {1} K _ {2} \delta , \tag {6}
$$
where  $K_{2} > 0$  is the Lipschitz constant obtained by (A.2) and (A.5). If we wish to bound the right-hand side of (6) by some term of order  $\mathcal{O}(\delta^p)$  when  $\delta \ll 1$ , the length  $T_{2}(\delta) \in \mathbb{N}$  should satisfy
$$
T _ {2} (\delta) \simeq C _ {1} + \frac {p - 1}{\lambda (\theta) + \log \gamma} \log (\delta), \tag {7}
$$
where  $C_1 > 0$  is some constant independent of  $p$  and  $\delta$ .
Figure 1: An illustration of the two series (7) and (9) that need to cover the entire  $\mathbb{R}$  when  $\delta \to 0$ .
Next, for the sum of the tail terms in  $V^{\pi_{\theta}}(\cdot)$  starting from  $T_{3}\in \mathbb{N}$ , it is automatically bounded by
$$
\sum_ {t = T _ {3}} ^ {\infty} \gamma^ {t} \left| c \left(s _ {t}, \pi_ {\theta} \left(s _ {t}\right)\right) - c \left(s _ {t} ^ {\prime}, \pi_ {\theta} \left(s _ {t} ^ {\prime}\right)\right) \right| \leq \frac {2 M _ {2} e ^ {T _ {3} \log \gamma}}{1 - \gamma}, \tag {8}
$$
where  $M_2 = \max_{s \in S} c(s, \pi_\theta(s))$  is the maximum of continuous function  $c(\cdot, \pi_\theta(\cdot))$  over the compact domain  $S$  (and hence exists). If we bound the right-hand side of (8) by a term of order  $\mathcal{O}(\delta^p)$ , it yields
$$
T _ {3} (\delta) \simeq C _ {2} + \frac {p}{\log \gamma} \log (\delta), \tag {9}
$$
for some independent constant  $C_2 > 0$ . Since the sum of (6) and (8) provides a good estimate of  $V^{\pi_{\theta}}$  only if  $T_{3}(\delta) - T_{2}(\delta)\leq N_{0}$  for some  $N_0 > 0$  as  $\delta \rightarrow 0$ , otherwise there would be infinitely many terms in the middle as  $\delta \rightarrow 0$  that cannot be controlled by any  $\mathcal{O}(\delta^p)$  terms. In this case, we have
$$
\left(C _ {2} - C _ {3}\right) + \left(\frac {p}{\log \gamma} - \frac {p - 1}{\lambda (\theta) + \log \gamma}\right) \log (\delta) \leq N _ {0}, \tag {10}
$$
as  $\log (\delta)\to -\infty$  , which implies that the slopes satisfy the inequality
$$
\frac {p}{\log \gamma} - \frac {p - 1}{\lambda (\theta) + \log \gamma} \geq 0, \tag {11}
$$
where the equality holds when  $p = -\frac{\log \gamma}{\lambda(\theta)}$ . Thus,  $V^{\pi_{\theta}}(s)$  is  $-\frac{\log \gamma}{\lambda(\theta)}$ -Hölder continuous at  $s = s_0$ .
On the other hand, the following counterexample shows that Theorem 4.1 has provided the strongest Hölder-continuity result for  $V^{\pi_{\theta}}(s)$  at  $s = s_0$  under the assumptions (A.1)-(A.5):
Example 4.1. Consider a one-dimensional MDP  $s_{t + 1} = f(s_t, a_t)$  where
$$
f (s, a) = \left\{ \begin{array}{l l} - 1, & a \leq - 1, \\ a, & - 1 <   a <   1, \\ 1, & a \geq 1, \end{array} \right. \tag {12}
$$
with state space  $\mathcal{S} = [-1,1]$  and cost function  $c(s,a) = |s|$ . Let the policy be linear, namely  $\pi_{\theta}(s) = \theta \cdot s$  and  $\theta \in \mathbb{R}$ . It can be verified that all assumptions (A.1)-(A.5) are satisfied. Now let  $s_0 = 0$  and  $\theta > 1$ , then applying (4) directly yields
$$
\lambda (\theta) = \limsup_{t\to \infty}\limsup_{\| \Delta Z_{0}\| \to 0}\frac{1}{t}\log \frac{\| \Delta Z(t)\|}{\| \Delta Z_{0}\|} = \limsup_{t\to \infty}\limsup_{\| \Delta Z_{0}\| \to 0}\frac{1}{t}\log \frac{\| \Delta Z_{0}\|\theta^{t}}{\| \Delta Z_{0}\|} = \log \theta .
$$
Let  $\delta > 0$  be sufficiently small, then
$$
V ^ {\pi_ {\theta}} (\delta) = \sum_ {t = 0} ^ {\infty} \delta \gamma^ {t} \theta^ {t} = \sum_ {t = 0} ^ {T _ {0} (\delta)} \delta \gamma^ {t} \theta^ {t} + \sum_ {t = T _ {0} (\delta)} ^ {\infty} \gamma^ {t} \geq \frac {\gamma^ {T _ {0} (\delta)}}{1 - \gamma}
$$
where  $T_0(\delta) = 1 + \left\lfloor \frac{-\log\delta}{\log\theta}\right\rfloor \in \mathbb{N}$  and  $\lfloor \cdot \rfloor$  is the flooring function. Therefore, we have
$$
\left| V ^ {\pi_ {\theta}} (\delta) - V ^ {\pi_ {\theta}} (0) \right| = V ^ {\pi_ {\theta}} (\delta) \geq \frac {\gamma^ {\frac {- \log \delta}{\log \theta} + 1}}{1 - \gamma} = \frac {\gamma}{1 - \gamma} \delta^ {\frac {- \log \gamma}{\log \theta}}.
$$
Remark 4.1. Another way to see why it is theoretically impossible to prove  $p$ -Hölder continuity for  $V^{\pi_{\theta}}$  for any  $p > -\frac{\log\gamma}{\lambda(\theta)}$ : notice that the inequality (10) no longer holds as  $\log \delta \to -\infty$  since
$$
\frac {p}{\log \gamma} - \frac {p - 1}{\lambda (\theta) + \log \gamma} <   0.
$$
Thus,  $p = \frac{-\log \gamma}{\lambda(\theta)}$  is the largest Hölder exponent of  $V^{\pi_{\theta}}(\cdot)$  that can be proved in the worst case.
Remark 4.2. The value function  $V^{\pi_{\theta}}(s)$  is Lipschitz continuous at  $s = s_0$  when the maximal Lyapunov exponent  $\lambda(\theta) < -\log \gamma$ , since there exists a constant  $K'$  such that
$$
\begin{array}{l} | V ^ {\pi_ {\theta}} (s _ {0} ^ {\prime}) - V ^ {\pi_ {\theta}} (s _ {0}) | \leq \sum_ {t = 0} ^ {\infty} \gamma^ {t} | c (s _ {t}, \pi_ {\theta} (s _ {t})) - c (s _ {t} ^ {\prime}, \pi_ {\theta} (s _ {t} ^ {\prime})) | \\ \leq \sum_ {t = 0} ^ {\infty} \gamma^ {t} K ^ {\prime} \delta e ^ {\lambda (\theta) t} \\ \leq \delta K ^ {\prime} \sum_ {t = 0} ^ {\infty} e ^ {(\lambda (\theta) + \log \gamma) t} \\ \leq \frac {K ^ {\prime} \delta}{1 - e ^ {(\lambda (\theta) + \log \gamma)}} \\ \end{array}
$$
where  $\delta = \| s_0 - s_0' \| > 0$  is the difference in the initial state.
4.2 Hölder Exponent of  $J(\cdot)$
The following lemma establishes a direct connection between  $J(\theta)$  and  $J(\theta')$  through value functions:
Lemma 4.1. Suppose that  $\theta, \theta' \in \mathbb{R}^p$ , then