SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
V ^ {\pi_ {\theta^ {\prime}}} (s _ {0}) - V ^ {\pi_ {\theta}} (s _ {0}) = \sum_ {t = 0} ^ {\infty} \gamma^ {t} (Q ^ {\pi_ {\theta}} (s _ {t} ^ {\theta^ {\prime}}, \pi_ {\theta^ {\prime}} (s _ {t} ^ {\theta^ {\prime}})) - V ^ {\pi_ {\theta}} (s _ {t} ^ {\theta^ {\prime}}))
$$
where  $\{s_t^{\theta'}\}_{t=0}^{\infty}$  is the trajectory generated by the policy  $\pi_{\theta'}(\cdot)$ .
The proof can be found in the Appendix B.2. Notice that indeed we have  $J(\theta') = V^{\pi_{\theta'}}(s_0)$  and  $J(\theta) = V^{\pi_{\theta}}(s_0)$ , substituting with these two terms in the previous lemma and doing some calculations lead to the following main theorem whose proof can be found in the Appendix B.3:
Theorem 4.2. (Non-smoothness of  $J$ ) Assume (A.1)-(A.5) and the parameterized policy  $\pi_{\theta}(\cdot)$  is deterministic. Let  $\lambda(\theta)$  denote the MLE of (1) at  $\theta \in \mathbb{R}^p$ . Suppose that  $\lambda(\theta) > -\log \gamma$ , then  $J(\cdot)$  is  $\frac{-\log \gamma}{\lambda(\theta)}$ -Hölder continuous at  $\theta$ .
Remark 4.3. In fact, the set of assumptions (A.1)-(A.5) is quite general and does not exclude the case of constant cost functions  $c(s,a) \equiv \text{const}$ , which always results in a smooth landscape regardless of the underlying dynamics, even though they are rarely used in practice. However, recall that the  $\frac{-\log\gamma}{\lambda(\theta)}$ -Hölder continuity is a result of exponential divergence of nearby trajectories, when a cost function can continuously distinguish two separate trajectories (e.g., quadratic costs) with a discount factor close to 1, the landscape will be fractal as shown in Section 6. Another way to see it is to look into the relaxation in (5) where the Hölder continuity is obtained from the local Lipschitz continuity of  $c(s,a)$ , i.e.,  $|c(s,\pi_{\theta}(s)) - c(s',\pi_{\theta}(s'))| \leq K_2\| s - s' \|$ . Therefore, the Hölder continuity is tight if for any  $\delta > 0$ , there exists  $s_0' \in \mathcal{B}(s_0,\delta)$  such that  $|c(s_t,\pi_\theta (s_t)) - c(s_t',\pi_\theta (s_t'))| \geq K_3\| s_t - s_t'\|$  with some  $K_{3} > 0$  for all  $t \in \mathbb{N}$ . We will leave the further investigation for future studies.
The following example illustrates how the smoothness of loss landscape changes with  $\lambda (\theta)$  and  $\gamma$
Example 4.2. (Logistic model) Consider the following MDP:
$$
s _ {t + 1} = \left(1 - s _ {t}\right) a _ {t}, \quad s _ {0} = 0. 9, \tag {13}
$$
where the policy  $a_{t}$  is given by deterministic linear function  $a_{t} = \pi_{\theta}(s_{t}) = \theta s_{t}$ . The objective function is defined as  $J(\theta) = \sum_{t=0}^{\infty} \gamma^{t} (s_{t}^{2} + 0.1 a_{t}^{2})$  where  $\gamma \in (0,1)$  is the discount factor. It is well-known that (13) begins to exhibit chaotic behavior with positive MLEs (as shown in Figure 2a) when  $\theta \geq 3.3$  [15], so we plot the graphs of  $J(\theta)$  for different discount factors over the interval  $\theta \in [3.3, 3.9]$ . From Figure 2b to 2d, the non-smoothness becomes more and more significant as  $\gamma$  grows. In particular, Figure 2e shows that the value of  $J(\theta)$  fluctuates violently even within a very small interval of  $\theta$ , suggesting a high degree of non-differentiability in this region.
(a) MLE  $\lambda (\theta)$
(b)  $\gamma = 0.5$
(c)  $\gamma = 0.9$
(d)  $\gamma = 0.99$
(e) Magnified.
Figure 2: The value of MLE  $\lambda (\theta)$  for  $\theta \in [3.3,3.9]$  is shown in 2a. The graph of objective function  $J(\theta)$  for different values of  $\gamma$  are shown in 2b-2e where  $J(\theta)$  is estimated by the sum of first 1000 terms in the infinite series.
4.3 Stochastic Policies
The original MDP (1) becomes stochastic when a stochastic policy is employed. First, let us consider the slightly modified version of MLE for stochastic policies:
$$
\tilde {\lambda} _ {m a x} = \lim  _ {t \rightarrow \infty} \sup  _ {\| \Delta Z _ {0} \| \rightarrow 0} \frac {1}{t} \log \frac {\| \mathbb {E} _ {\pi} [ \Delta Z _ {\omega} (t) ] \|}{\| \Delta Z _ {0} \|}. \tag {14}
$$
where  $\Delta Z_0 = s_0' - s_0$  is a small perturbation made to the initial state and  $\Delta Z_{\omega}(t) = s_t^\prime (\omega) - s_t(\omega)$  is the difference in the sample path at time  $t\in \mathbb{N}$  and sample  $\omega \in \Omega$ . Since this definition is consistent with that in (4) when sending the variance to 0, we use the same notation  $\lambda (\theta)$  to denote the MLE at given  $\theta \in \mathbb{R}^N$  and again assume  $\lambda (\theta) > - \log \gamma$ . Since policies in most control and robotics environments are deterministic, this encourages the variance to converge to 0 during training.
However, unlike the deterministic case where the Hölder continuity result was proved under the assumption that the policy  $\pi_{\theta}(s)$  is locally Lipschitz continuous, stochastic policies instead provide a probability distribution from which the action is sampled. Thus, a stochastic policy cannot be locally Lipschitz continuous in  $\theta$  when approaching its deterministic limit. For instance, consider the one-dimensional Gaussian distribution  $\pi_{\theta}(a|s)$  where  $\theta = [\mu, \sigma]^T$  denotes the mean and variance. As the variance  $\sigma$  approaches 0,  $\pi_{\theta}(a|s)$  becomes more and more concentrated at  $a = \mu s$ , and eventually converges to the Dirac delta function  $\delta(a - \mu s)$ , which means that  $\pi_{\theta}(a|s)$  cannot be Lipschitz continuous within a neighborhood of any  $\theta = [\mu, 0]^T$  even though its deterministic limit  $\pi_{\theta}(s) = \mu s$  is indeed Lipschitz continuous. The following example illustrates that in this case, the Hölder exponent of the objective function  $J(\cdot)$  can still be less than 1:
Example 4.3. Suppose that the one-dimensional MDP  $s_{t+1} = f(s_t, a_t)$  where  $f(s, a)$  is defined as in (12) over the state space  $S = [-1,1]$  and action space  $\mathcal{A} = [0,\infty)$ . The cost function is  $c(s, a) = s + 1$ . Also, the parameter space is  $\theta = [\theta_1, \theta_2]^T \in \mathbb{R}^2$  and the policy  $\pi_\theta(\cdot|s) \sim \mathcal{U}(|\theta_1|s + |\theta_2|, |\theta_1|s + 2|\theta_2|)$  is a uniform distribution. It is easy to verify that all required assumptions are satisfied. Let the initial state  $s_0 = 0$  and  $\theta_1 > 1, \theta_2 = 0$ , then applying (14) directly yields  $\lambda(\theta) = \log \theta_1$  similarly as in Example 4.1. Now suppose that  $\theta_2' > 0$  is small and  $\theta' = [\theta_1, \theta_2']^T$ , then for any  $\omega \in \Omega$  in the sample space, the sampled trajectory  $\{s_t'\}$  generated by  $\pi_{\theta'}$  has
$$
s _ {t + 1} ^ {\prime} (\omega) \geq \theta_ {1} s _ {t} ^ {\prime} (\omega) + \theta_ {2} ^ {\prime} > \theta_ {1} s _ {t} ^ {\prime} (\omega) \geq \theta_ {1} ^ {t} s _ {1} ^ {\prime} (\omega) \geq \theta_ {1} ^ {t} \left(\theta_ {2} ^ {\prime}\right)
$$
when  $s_{t+1}'(\omega) < 1$ . Thus, we have  $s_{t+1}'(\omega) = 1$  for all  $\omega \in \Omega$  and  $t \geq T_0(\theta') = 1 + \left\lfloor \frac{-\log \theta_2'}{\log \theta_1} \right\rfloor$ , which further leads to
$$
J (\theta^ {\prime}) = \frac {1}{1 - \gamma} + \sum_ {t = 0} ^ {\infty} \gamma^ {t} \mathbb {E} _ {\pi_ {\theta^ {\prime}}} [ s _ {t} ^ {\prime} ] \geq J (\theta) + \sum_ {t = T _ {0} (\delta)} ^ {\infty} \gamma^ {t} \mathbb {E} _ {\pi_ {\theta^ {\prime}}} [ s _ {t} ^ {\prime} ] \geq \frac {\gamma}{1 - \gamma} (\theta_ {2} ^ {\prime}) ^ {\frac {- \log \gamma}{\log \theta_ {1}}}
$$
using the fact that  $J(\theta) = \frac{1}{1 - \gamma}$ . Plugging  $\| \theta - \theta' \| = \theta_2'$  into the above inequality yields
$$
J \left(\theta^ {\prime}\right) - J (\theta) \geq \frac {\gamma}{1 - \gamma} \| \theta^ {\prime} - \theta \| ^ {\frac {- \log \gamma}{\log \theta_ {1}}}. \tag {15}
$$
where the Hölder exponent is again  $\frac{-\log\gamma}{\lambda(\theta)}$  as in Example 4.1.
Remark 4.4. Consider the 1-Wasserstein distance as defined in [36] between the distribution  $\delta(a - \mu s)$  and  $\mathcal{U}(|\theta_1|s + |\theta_2|, |\theta_1|s + 2|\theta_2|)$ , which is given by  $W_1(\theta_1, \theta_2) = \frac{3|\theta_2|}{2}$ . It is Lipschitz continuous at  $\theta_2 = 0$ , even though the non-smooth result in (15) holds. Therefore, probability distribution metrics, such as the Wasserstein distance, are too "coarse" to capture the full fractal nature of the objective function. This also suggests that further assumptions regarding the pointwise smoothness of probability density functions are necessary to create a smooth landscape with stochastic policies, even though they may exclude the case of  $\sigma \to 0$  as discussed earlier.
5 Estimating Hölder Exponents from Samples
In the previous sections, we have seen that the objective function  $J(\theta)$  can be highly non-smooth and thus gradient-based methods may not work well in the policy parameter space. The question is: how can we determine whether the objective function  $J(\theta)$  is differentiable at some  $\theta = \theta_0$  or not in high-dimensional settings? Note that  $J(\theta)$  may have different levels of smoothness along different directions. To address it, we propose a statistical method to estimate the Hölder exponent. Consider the objective function  $J(\theta)$  and a probability distribution whose variance is finite. Consider the isotropic Gaussian distribution  $X \sim \mathcal{N}(\theta_0, \sigma^2\mathcal{I}_p)$  where  $\mathcal{I}_p$  is the  $p \times p$  identity matrix. For continuous objective function  $J(\cdot)$ , then its variance matrix can be expressed as
$$
\begin{array}{l} V a r (J (X)) = \mathbb {E} _ {X \sim \mathcal {N} (\theta_ {0}, \sigma^ {2} \mathcal {I})} [ J (X) - \mathbb {E} _ {X \sim \mathcal {N} (\theta_ {0}, \sigma^ {2} \mathcal {I} _ {p})} [ J (X) ]) ^ {2} ] \\ = \mathbb {E} _ {X \sim \mathcal {N} \left(\theta_ {0}, \sigma^ {2} \mathcal {I} _ {p}\right)} \left[ \left(J (X) - J \left(\xi^ {\prime}\right)\right) ^ {2} \right] \\ \end{array}
$$
where  $\xi' \in \mathbb{R}^p$  is obtained from applying the intermediate value theorem to  $\mathbb{E}_{X \sim \mathcal{N}(\theta_0, \sigma^2\mathcal{I}_p)}[J(X)]$  and hence not a random variable. If  $J(\cdot)$  is locally Lipschitz continuous at  $\theta_0$ , say  $|J(\theta) - J(\theta_0)| \leq K\|\theta - \theta_0\|$  for some  $K > 0$  when  $\|\theta - \theta_0\|$  is small, then it has the following approximation
$$
V a r (J (X)) \leq K ^ {2} \mathbb {E} _ {X \sim \mathcal {N} \left(\theta_ {0}, \sigma^ {2} \mathcal {I} _ {p}\right)} [ \| X - \xi^ {\prime} \| ^ {2} ] \simeq (V a r (X)) ^ {2} \sim \mathcal {O} (\sigma^ {2}) \tag {16}
$$
when  $\sigma \ll 1$ . Therefore, (16) provides a way to directly determine whether the Hölder exponent of  $J(\cdot)$  at any given  $\theta \in \mathbb{R}^p$  is less than 1, especially when the dimension  $p$  is large. In particular, taking the logarithm on both sides of (16) yields
$$
\log \operatorname {V a r} _ {\sigma} (J (X)) \leq C + 2 \log \sigma \tag {17}
$$
for some constant  $C$  where the subscript in  $Var_{\sigma}(J(X))$  indicates its dependence on the standard deviation  $\sigma$  of  $X$ . Thus, the log-log plot of  $Var_{\sigma}(J(X))$  versus  $\sigma$  is expected to be close to a straight line with slope  $k \geq 2$  when  $J(\theta)$  is locally Lipschitz continuous around  $\theta = \theta_0$ . Therefore, one can determine the smoothness by sampling around  $\theta_0$  with different variances and estimating the slope via linear regression. Usually,  $J(\theta)$  is Lipschitz continuous at  $\theta = \theta_0$  when the slope  $k$  is close to or greater than 2, and it is non-differentiable if the slope is less than 2.
In this section, we will validate the theory presented in this paper through common RL tasks. All environments are adopted from The OpenAI Gym Documentation [5] with continuous control input. The experiments are conducted in two steps: first, we randomly sample a parameter  $\theta_0$  from a Gaussian distribution and estimate the gradient  $\eta(\theta_0)$  from (3); second, we evaluate  $J(\theta)$  at  $\theta = \theta_0 + \delta \eta(\theta_0)$  for each small  $\delta > 0$ . According to our results, the loss curve is expected to become smoother as  $\gamma$  decreases, since smaller  $\gamma$  makes the Hölder exponent  $\frac{-\log \gamma}{\lambda(\theta)}$  larger. In the meantime, the policy gradient method (3) should give a better descent direction while the true objective function  $J(\cdot)$  becoming smoother.
Notice that a single sample path can always be non-smooth when the policy is stochastic and hence interferes the desired observation, we use stochastic policies to estimate the gradient in (3), and apply their deterministic version (by setting variance equal to 0) when evaluating  $J(\theta)$ . Regarding the infinite series, we use the sum of first 1000 terms to approximate  $J(\theta)$ . The stochastic policy is given by  $\pi_{\theta}(\cdot |s)\sim \mathcal{N}(u(s),\sigma^2\mathcal{I}_p)$  where the mean  $u(s)$  is represented by the 2-layer neural network  $u(s) = W_2\tanh (W_1s)$  where  $W_{1}\in \mathcal{M}_{r\times n}(\mathbb{R})$  and  $W_{2}\in \mathcal{M}_{m\times r}(\mathbb{R})$  are weight matrices. Let  $\theta = [W_1,W_2]^T$  denote the vectorized policy parameter. For the width of the hidden layer, we use  $r = 8$  for the inverted pendulum and acrobot, and  $r = 64$  for the hopper.
(a)  $\gamma = 0.9$  ,deter-(b)  $\gamma = 0.9$  ,stochasministic.tic.
(c)  $\gamma = 0.99$  ,deter-(d)  $\gamma =$  ministic. stochastic.
0.99,(e)  $y = 1.980x + 2.088.$
Figure 3: The experimental results of inverted pendulum. In 3e, the linear regression result is obtained for  $\gamma = 0.9$ . The loss curves  $J(\theta)$  are presented in 3a-3d where  $\theta = \theta_0 + \delta \eta (\theta_0)$  with step size  $10^{-7}$ .
Inverted Pendulum. The inverted pendulum task is a standard test case for RL algorithms, and here we use it as an example of non-chaotic system. The initial state is always taken as  $s_0 = [-1, 0]^T$  ( $[0, 0]^T$  is the upright position), and quadratic cost function  $c(s, a) = s_t^T Q s_t + 0.001 \| a_t \|^2$ , where  $Q = \mathrm{diag}(1, 0.1)$  is a  $2 \times 2$  diagonal matrix,  $s_t \in \mathbb{R}^2$  and  $a_t \in \mathbb{R}$ . The initial parameter is given by  $\theta_0 \sim \mathcal{N}(0, 0.05^2 \mathcal{I})$ . In Figure 4a and 4c, we see that the loss curve is close to a straight line within a very small interval, which indicates the local smoothness of  $\theta_0$ . It is validated by the estimate of the Hölder exponent of  $J(\theta)$  at  $\theta = \theta_0$  which is based on (16) by sampling many parameters around  $\theta_0$  with different variance. In Figure 3e, the slope  $k = 1.980$  is very close to 2 so Lipschitz continuity (and hence differentiability) is verified at  $\theta = \theta_0$ . As a comparison, the loss curve of single random sample path is totally non-smooth as shown in Figure 3b and 3d.
Acrobot. The acrobot system is well-known for its chaotic behavior and hence we use it as the main test case. Here we use the cost function  $c(s, a) = s_t^T Q s_t + 0.005 \| a_t \|^2$ , where  $Q = \mathrm{diag}(1, 1, 0.1, 0.1)$ ,  $s_t \in \mathbb{R}^4$  and  $a_t \in \mathbb{R}$ . The initial state is  $s_0 = [1, 0, 0, 0]^T$ . The initial parameter is again sampled from  $\theta_0 \sim \mathcal{N}(0, 0.05^2 \mathcal{I})$ . From Figure 4a-4c, the non-smoothness grows as  $\gamma$  increases and finally becomes completely non-differentiable when  $\gamma = 0.99$  which is the most common value used for discount factor. It partially explains why the acrobot task is difficult to policy gradient methods. In Figure 4e, the Hölder exponent of  $J(\theta)$  at  $\theta = \theta_0$  is estimated as  $\alpha \simeq 0.43155 < 1$ , which further indicates non-differentiability around  $\theta_0$ .
(a)  $\gamma = 0.8$  ,deter-(b)  $\gamma = 0.9$  ,deterministic. ministic.
Figure 4: The experimental results of acrobot. In Figure 4e, the linear regression result is obtained for  $\gamma = 0.9$ . The loss curves  $J(\theta)$  are presented in 4a-4d where  $\theta = \theta_0 + \delta \eta (\theta_0)$  with step size  $10^{-7}$ .
0.99,(e)  $y = 0.8631x +$  2.041
Hopper. Now we consider the Hopper task in which the cost function is defined  $c(s, a) = (1.25 - s[0]) + 0.001 \|a\|^2$ , where  $s[0]$  is the first coordinate in  $s \in \mathbb{R}^{11}$  which indicates the height of hopper. Because the number of parameters involved in the neural network is larger, the initial parameter is instead sampled from  $\theta_0 \sim \mathcal{N}(0, 10^2 \mathcal{I})$ . As we see that in Figure 5a, the loss curve is almost a straight line when  $\gamma = 0.8$ , and it starts to exhibit non-smoothness when  $\gamma = 0.9$  and becomes totally non-differentiable when  $\gamma = 0.99$ . A supporting evidence by the Hölder exponent estimation is provided in Figure 5e where the slope is far less than 2.
(a)  $\gamma = 0.8$  ministic.
(b)  $\gamma = 0.9$  ministic.
(c)  $\gamma = 0.99$  ,deter-(d) ministic. sto
(d)  $\gamma =$  stochastic.
0.99,(e)  $y = 0.5036x - 1.250.$
Figure 5: The experimental results of hopper. In Figure 5e, the linear regression result is obtained for  $\gamma = 0.9$ . The loss curves  $J(\theta)$  are presented in 5a-5d where  $\theta = \theta_0 + \delta \eta (\theta_0)$  with step size  $10^{-3}$ .
In this paper, we initiate the study of chaotic behavior in reinforcement learning, especially focusing on how it is reflected on the fractal landscape of objective functions. A method to statistically estimate the Hölder exponent at some given parameter is proposed, so that one can figure out if the training process has encountered fractal landscapes or not. We believe that the theory established in this paper can help to explain many existing results in reinforcement learning, such as the hardness of complex control tasks and the fluctuating behavior of training curves. It also poses a serious question to the well-posedness of policy gradient methods given the fact that no gradient exists in many continuous state-space RL problems. Being aware of the fact that the non-smoothness of loss landscapes is an intrinsic property of the model, rather than a consequence of any numerical or statistical errors, we conjecture that the framework developed in this paper might provide new insights into the limitations of a wider range of deep learning problems beyond the realm of reinforcement learning.
Our work is supported by NSF Career CCF 2047034, NSF AI Institute CCF 2112665, ONR YIP N00014-22-1-2292, NSF CCF DASS 2217723, and Amazon Research Award. The authors thank Zichen He, Bochao Kong and Xie Wu for insightful discussions.
[1] A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
[2] A. Bagirov, N. Karmitsa, and M. M. Makela. Introduction to Nonsmooth Optimization. Springer, 2014.
[3] M. Barnsley. Fractals Everywhere. Academic Press, Inc., 1988.
[4] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157-166, 1994.
[5] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540, 2016.
[6] A. Camuto, G. Deligiannidis, M. A. Erdogdu, M. Gürbüzbalaban, U. Şimşekli, and L. Zhu. Fractal structure and generalization properties of stochastic optimization algorithms. arXiv preprint arXiv:2106.04881, 2021.
[7] F. H. Clarke. Methods of Dynamic and Nonsmooth Optimization. SIAM, 1989.
[8] K. J. Falconer. Fractal Geometry: Mathematical Foundations and Applications. John Wiley, 1990.
[9] J. Fan, Z. Wang, Y. Xie, and Z. Yang. A theoretical analysis of deep Q-learning. volume 120, pages 486-489, 2020.
[10] M. Fazel, R. Ge, S. M. Kakade, and M. Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. Proceedings of the 35th International Conference on Machine Learning, pages 1467-1476, 2018.
[11] X. Guo and B. Hu. Convergence of direct policy search for state-feedback  $\mathcal{H}_{\infty}$  robust control: A revisit of nonsmooth synthesis with Goldstein subdifferential. arXiv preprint arXiv:2210.11577, 2022.
[12] G. H. Hardy. Weierstrass's non-differentiable function. Transactions of the American Mathematical Society, 17:301-325, 1916.
[13] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, G. Dulac-Arnold, J. Agapiou, J. Leibo, and A. Gruslys. Deep Q-learning from demonstrations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018.
[14] M. W. Hirsch, S. Smale, and R. L. Devaney. Differential Equations, Dynamical Systems, and an Introduction to Chaos. Academic Press, 2013.
[15] D. W. Jordan and P. Smith. Nonlinear Ordinary Differential Equations: An introduction for Scientists and Engineers. Oxford University Press, 2007.
[16] S. M. Kakade. A natural policy gradient. Advances in Neural Information Processing Systems 14, pages 1531-1538, 2001.
[17] H. Kantz. A robust method to estimate the maximal Lyapunov exponent of a time series. Physics Letter A, 185:77-87, 1994.
[37] H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double Q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), 2016.
[38] O. Vinyals, I. Babuschkin, W. Czarnecki, and et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575:350-354, 2019.
[39] R. Wang, D. P. Foster, and S. M. Kakade. What are the statistical limits of offline RL with linear function approximation? arXiv preprint arXiv:2010.11895, 2020.
[40] C. J. C. H. Watkins and P. Dayan. Q-learning. Machine Learning, 8:279-292, 1992.
[41] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 1992.
[42] L. Xiao. On the convergence rates of policy gradient methods. arXiv preprint arXiv:2201.07443, 2022.
[43] K. Zhang, B. Hu, and T. Basar. Policy optimization for  $\mathcal{H}_2$  linear control with  $\mathcal{H}_{\infty}$  robustness guarantee: Implicit regularization and global convergence. SIAM Journal on Control and Optimization, 59(6):4081-4109, 2021.
A A Brief introduction to chaos theory
As mentioned in the Introduction, chaos exists in many systems in the real world. Although no universal definition of chaos can be made, there are, indeed, three features that a chaotic system usually possesses [14]:
- Dense periodic points;
- Topological transitivity;
- Sensitive dependence on initial conditions;
In some cases, some of these properties imply the others. It is important to note that, despite the appearance of chaos is always accompanied by high unpredictability, the chaotic behavior is entirely deterministic and is not a consequence of randomness. Another interesting fact is that trajectories in a chaotic system are usually bounded, which drives us to think about the convergence of policy gradient methods beyond the boundedness of state spaces.
(a) Lorenz attractor.
(b) Rössler attractor.
Figure 6: The Lorenz system and Rössler system are standard examples of chaotic systems, in which a small perturbation in the initial state can result in a significant divergence in the entire trajectory.
Actually, it can be summarized from the results in this paper that for a given MDP, the following three features contribute most to its chaotic behavior:
- Infinite time horizon  $(t\to \infty)$
- Continuous state space  $(\|\Delta Z_0\| \to 0)$
- Exponential divergence ( $\lambda_{\text{max}} > 0$ );
Since these features are not necessarily bound to certain types of continuous state-space MDPs, it would be exciting for future studies to investigate other types of MDPs using the framework developed in the paper.
B Proofs omitted in Section 4
Proof. Suppose that  $s_0' \in S$  is another initial state close to  $s_0$  and  $\delta = \| s_0 - s_0' \|$ . Let  $T_1 \in \mathbb{N}$  be the smallest integer that satisfies