Title: Fractal Landscapes in Policy Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement learning

policy gradient

non-smooth landscape
Abstract: Policy gradient lies at the core of deep reinforcement learning (RL) in continuous domains. Despite much success, it is often observed in practice that RL training with policy gradient can fail for many reasons, even on standard control problems with known solutions. We propose a framework for understanding one inherent limitation of the policy gradient approach: the optimization landscape in the policy space can be extremely non-smooth or fractal for certain classes of MDPs, such that there does not exist gradient to be estimated in the first place. We draw on techniques from chaos theory and non-smooth analysis, and analyze the maximal Lyapunov exponents and H\"older exponents of the policy optimization objectives. Moreover, we develop a practical method that can estimate the local smoothness of objective function from samples to identify when the training process has encountered fractal landscapes. We show experiments to illustrate how some failure cases of policy optimization can be explained by such fractal landscapes.

Fractal Landscapes in Policy Optimization

Abstract

Policy gradient lies at the core of deep reinforcement learning (RL) in continuous domains. Despite much success, it is often observed in practice that RL training with policy gradient can fail for many reasons, even on standard control problems with known solutions. We propose a framework for understanding one inherent limitation of the policy gradient approach: the optimization landscape in the policy space can be extremely non-smooth or fractal for certain classes of MDPs, such that there does not exist gradient to be estimated in the first place. We draw on techniques from chaos theory and non-smooth analysis, and analyze the maximal Lyapunov exponents and Hölder exponents of the policy optimization objectives. Moreover, we develop a practical method that can estimate the local smoothness of objective function from samples to identify when the training process has encountered fractal landscapes. We show experiments to illustrate how some failure cases of policy optimization can be explained by such fractal landscapes.

1 Introduction

Deep reinforcement learning has achieved much success in various applications [23, 30, 38], but they also often fail, especially in continuous spaces, on control problems that other methods can readily solve. The understanding of such failure cases is still limited. For instance, the training process of reinforcement learning is unstable and the learning curve can fluctuate during training in ways that are hard to predict. The probability of obtaining satisfactory policies can also be inherently low in reward-sparse or highly nonlinear control tasks. Existing analysis of the failures focuses on limitations of the sampling and optimization algorithms, such as function approximation errors [35, 39], difficulty in data collection [34], and aggressive updates in the policy space [28]. There has not been much study of potentially deeper causes of failures that may be inherent in the formulation of policy optimization problems.

Motivated by the common observation that small updates in the policy parameters can significantly change the performance, we analyze the smoothness of the optimization landscapes in policy optimization. Drawing on chaos theory, we introduce the concept of maximal Lyapunov exponent (MLE) [17] to the RL setting to measure the exponential rate of trajectory divergence in MDP. It seems contradictory that a trajectory in chaotic systems can be both exponentially divergent and uniformly bounded at the same time, and we will show that these two conflicting facts combine to yield the fractal structure in the optimization landscape. Intuitively, the objective function is non-differentiable when the rate of trajectory divergence exceeds the decay rate of discount factor. Furthermore, this finding indicates that the fluctuations observed in the loss curve are not just due to the numerical or sampling error but rather reflect the intrinsic properties of the corresponding MDP.

We should emphasize that the fractal landscapes that we will demonstrate are stronger than various existing results on the non-smoothness [2, 7]. Most non-smooth objectives that have been studied still assume is local Lipschitz continuity or piecewise smoothness that implies differentiability almost everywhere (such as f ( x ) = | x | ). Instead, by showing that the loss landscape of policy optimization can be fractal, we demonstrate the absence of descent directions, which causes the failure of first-order

methods in general. Since such behavior is an intrinsic property of the underlying dynamical systems, the results show fundamental limitations of policy gradient methods on certain classes of MDPs.

The paper is organized as follows. In Section 3 and 4, we will introduce the preliminaries and develop the theory for deterministic policies. In particular, we show that the optimization landscape is fractal, even when all elements within the MDP are deterministic. Next, we consider stochastic policies and provide an example to show how non-smoothness can still occur if without additional assumptions. In Section 5, we turn the theoretical analysis into a practical sampling-based method for estimating the Hölder exponent to determine whether the optimization objective is differentiable at a specific parameter vector. It can also indicate if the training process has encountered fractal regions by comparing the regression slope with some fixed threshold. In Section 6, we show experiments that demonstrate and compare the landscapes of different MDPs.

2 Related work

Policy gradient and Q-learning methods. Policy gradient methods [33, 41] formulate RL as an optimization problem in the parameter space, with many variations such as natural policy gradient [16], deterministic policy gradient [29], deep deterministic policy gradient [18], trust region policy optimization [27] and proximal policy optimization [28], were proposed. As all of these algorithms aim to estimate the gradient of the objective function over the policy parameters, they become ill-posed when the objective is non-differentiable, which is the focus of our analysis.

Another popular approach for model-free RL is Q-learning methods, which approximate the Qfunction of the policy at each step [22, 40]. As neural networks become more and more popular, they are employed as function approximators in deep Q-learning algorithms [9, 13, 37]. Since the foundation of Q-learning methods is established upon the estimation of value functions, a poor approximation can completely ruin the entire training process. In this paper, we will show that the value functions in a certain class of MDPs exhibit significant non-smoothness, making them challenging to represent using existing methods.

Chaos in machine learning. Chaotic behaviors due to randomness in the learning dynamics have been reported in other learning problems [6, 21, 25]. For instance, when training recurrent neural networks for a long period, the outcome behaves like a random walk due to the problems of vanishing and the exploding gradients [4]. It served as motivation for the work [24], which points out that the chaotic behavior in finite-horizon model-based reinforcement learning problems may be caused by long chains of nonlinear computation. A similar observation was made in [31]. However, we show that in RL, the objective function is provably smooth if the time horizon is finite and the underlying dynamics is differentiable. Instead, we focus on the general context of infinite-horizon problems in MDPs, in which case the objective function can become non-differentiable.

Loss landscape of policy optimization. It has been shown that the objective functions in finite state-space MDPs are smooth [1, 42], which enables the use of gradient-based methods and direct policy search. It also explains why the classical RL algorithms in [32] are provably efficient in finite space settings. Also, such smoothness results can be extended to some continuous state-space MDPs with special structures. For instance, the objective function in Linear Quadratic Regulator (LQR) problems is almost smooth [10] as long as the cost is finite. Similar results are obtained for the H 2 / H ∞ problem [43]. For the robust control problem, although the objective function may not be smooth, it is locally Lipschitz continuous, which implies differentiability almost everywhere , and further leads to global convergence of direct policy search [11]. There is still limited theoretical study of loss landscapes of policy optimization for nonlinear and complex MDPs. We aim to partially address this gap by pointing out the possibility that the loss landscape can be highly non-smooth and even fractal, which is far more complex than the previous cases.

3 Preliminaries

3.1 Dynamical Systems as Markov Decision Processes

We consider Markov Decision Processes (MDPs) that encode continuous control problems for dynamical systems defined by difference equations of the form:

where s t ∈ S ⊂ R n is the state at time t , s 0 is the initial state and a t ∼ π θ ( ·| s t ) ∈ A ⊂ R m is the action taken at time t based on a policy parameterized by θ ∈ R p .We assume that both the state space S and the action space A are compact. The objective function of the RL problem to minimize is defined by V π θ of policy π θ :

where γ ∈ (0 , 1) is the discount factor and c ( s, a ) is the cost function. The following assumptions are made throughout this paper:

(A.1) f : R n × R m → R n is Lipschitz continuous over any compact domains (i.e., locally Lipschitz continuous);

(A.2) The cost function c : R n × R m → R is non-negative and locally Lipschitz continuous everywhere;

(A.3) The state space is closed under transitions, i.e., for any ( s, a ) ∈ S × A , the next state s ′ = f ( s, a ) ∈ S .

3.2 Policy gradient methods

Policy gradient methods estimate the gradient of the objective J ( · ) with respect to the parameters of the policies. A commonly used form is

where π θ ( ·|· ) is a stochastic policy parameterized by θ . A π θ ( s, a ) = Q π θ ( s, a ) -V π θ ( s ) is the advantage function often used for variance reduction and Q π θ ( · , · ) is the Q -value function of π θ . The theoretical guarantee of the convergence of policy gradient methods is typically established by the argument that the tail term γ t ∇ θ V π θ ( s ) diminishes as t increases, for any s ∈ S [33]. For such claims to hold, two assumptions are needed:

∇ θ V π θ ( s ) exists and is continuous for all s ∈ S ;

∥∇ θ V π θ ( s ) ∥ is uniformly bounded over S .

The second assumption is automatically satisfied if the first assumption holds in the case that S is either finite or compact. However, as we will see in Section 4 and 6, the existence of ∇ θ V π θ ( · ) may fail in many continuous MDPs even if S is compact, which challenges the fundamental well-posedness of policy gradient methods.

3.3 Maximal Lyapunov Exponents

Behaviors of chaotic systems have sensitive dependence on their initial conditions. To be precise, consider the system s t +1 = F ( s t ) with initial state s 0 ∈ R n , and suppose that a small perturbation ∆ Z 0 is made to s 0 . The divergence from the original trajectory of the system under this perturbation at time t , say ∆ Z ( t ) , can be estimated by ∥ ∆ Z ( t ) ∥ ≃ e λt ∥ ∆ Z 0 ∥ with some λ that is called the Lyapunov exponent. For chaotic systems, Lyapunov exponents are typically positive, which implies an exponential divergence rate of the separation of nearby trajectories [19]. Since the Lyapunov exponent at a given point may depend on the direction of the perturbation ∆ Z 0 , and we are interested in identifying the largest divergence rate, the maximal Lyapunov exponent (MLE) is formally defined as follows:

Definition 3.1. (Maximal Lyapunov exponent) For the dynamical system s t +1 = F ( s t ) , s 0 ∈ R n , the maximal Lyapunov exponent λ max at s 0 is defined as the largest value such that

Note that systems with unstable equilibria, not necessarily chaotic, can have positive MLEs.

3.4 Fractal Landscapes

The Hausdorff dimension is the most fundamental concept in fractal theory. We first introduce the concept of δ -cover and Hausdorff measure:

Definition 3.2. ( δ -cover) Let { U i } be a countable collection of sets of diameter at most δ (i.e. | U i | = sup {∥ x -y ∥ : x, y ∈ U i } ≤ δ ) and F ⊂ R N , then { U i } is a δ -cover of F if F ⊂ ∪ ∞ i =1 U i .

Definition 3.3. (Hausdorff measure) For any F ⊂ R N and s ≥ 0 , let

Then we call the limit H s ( F ) = lim δ → 0 H s δ ( F ) the s -dimensional Hausdorff measure of F .

The definition of Hausdorff dimension follows immediately:

Definition 3.4. (Hausdorff dimension) Let F ⊂ R N be a subset, then its Hausdorff dimension

And we introduce the notion of α -Hölder continuity that extends the concept of Lipschitz continuity:

Definition 3.5. ( α -Hölder continuity) Let α > 0 be a scalar. A function g : R N → R is α -Hölder continuous at x ∈ R N if there exist C > 0 and δ > 0 such that

for all y ∈ B ( x, δ ) , where B ( x, δ ) denotes the open ball of radius δ centered at x .

The definition reduces to Lipschitz continuity when α = 1 . A function is not differentiable, if the largest Hölder exponent at a given point is less than 1 . Just as smoothness is commonly associated with Lipschitz continuity, fractal behavior is closely related to Hölder continuity. In particular, for an open set F ⊂ R k and a continuous mapping η : F → R p with p > k , the image set η ( F ) is fractal when its Hausdorff dimension dim H η ( F ) is strictly greater than k , which occurs when η : F → R p is α -Hölder continuous with exponent α < 1 :

Proposition 3.1. ([8]) Let F ⊂ R k be a subset and suppose that η : F → R p is α -Hölder continuous where α > 0 , then dim H η ( F ) ≤ 1 α dim H F .

It implies that if the objective function is α -Hölder for some α < 1 , its loss landscape L J = { ( θ, J ( θ )) ∈ R N +1 : θ ∈ R N } can be fractal. Further discussion of the theory on fractals can be found in Appendix C.

4 Fractal Landscapes in the Policy Space

In this section, we will show that the objective J ( θ ) in policy optimization can be non-differentiable when the system has positive MLEs. We will first consider Hölder continuity of V π θ ( · ) and J ( · ) with deterministic policies in 4.1 and 4.2, and then discuss the case of stochastic policies in 4.3.

4.1 Hölder Exponent of V π θ ( · )

We first consider a deterministic policy π θ that maps states to actions a = π θ ( s ) instead of distributions. Consider a fixed policy parameter θ ∈ R p such that the MLE of (1), namely λ ( θ ) , is greater than -log γ . Let s ′ 0 ∈ S be another initial state that is close to s 0 , i.e., δ = ∥ s ′ 0 -s 0 ∥ > 0 is small enough. According to the assumption (A.3) and the compactness of the state space, we can find a constant M > 0 such that both ∥ s t ∥ ≤ M and ∥ s ′ t ∥ ≤ M for all t ∈ N , where { s t } ∞ t =1 and { s ′ t } ∞ t =1 are the trajectories starting from s 0 and s ′ 0 , respectively. Motivated by (4), we further make the following assumptions:

(A.4) There exists K 1 > 0 such that ∥ s ′ t -s t ∥ ≤ K 1 δe λ ( θ ) t for all t ∈ N and δ = ∥ s ′ 0 -s 0 ∥ > 0 .

(A.5) The policy π : R N × R n → R m is locally Lipschitz continuous everywhere.

We then have following theorem, and it provides a lower bound for the Hölder exponent of J whose detailed proof can be found in Appendix B.1.

Theorem 4.1. (Non-smoothness of V π θ ) Assume (A.1)-(A.5) and the parameterized policy π θ ( · ) is deterministic. Let λ ( θ ) denote the MLE of (1) at θ ∈ R N . Suppose that λ ( θ ) > -log γ , then V π θ ( · ) is -log γ λ ( θ ) -Hölder continuous at s 0 .

Proof sketch of Theorem 4.1: Suppose that p ∈ (0 , 1] is some constant for which we would like to prove that V π θ ( s ) is p -Hölder continuous at s = s 0 , and here we take p = -log γ λ ( θ ) .

According to Definition 3.5, it suffices to find some C ′ > 0 such that | V π θ ( s ′ 0 ) -V π θ ( s 0 ) | ≤ C ′ δ p when δ = ∥ s 0 -s ′ 0 ∥ ≪ 1 . Consider the relaxed form

Now we split the entire series into three parts as shown in Figure 1: the sum of first T 2 terms, the sum from t = T 2 +1 to T 3 -1 , and the sum from t = T 3 to ∞ . First, applying (A.4) to the sum of the first T 2 terms yields

where K 2 > 0 is the Lipschitz constant obtained by (A.2) and (A.5). If we wish to bound the right-hand side of (6) by some term of order O ( δ p ) when δ ≪ 1 , the length T 2 ( δ ) ∈ N should satisfy

where C 1 > 0 is some constant independent of p and δ .

Figure 1: An illustration of the two series (7) and (9) that need to cover the entire R when δ → 0 .

Next, for the sum of the tail terms in V π θ ( · ) starting from T 3 ∈ N , it is automatically bounded by

where M 2 = max s ∈S c ( s, π θ ( s )) is the maximum of continuous function c ( · , π θ ( · )) over the compact domain S (and hence exists). if we bound the right-hand side of (8) by a term of order O ( δ p ) , it yields

for some independent constant C 2 > 0 . Since the sum of (6) and (8) provides a good estimate of V π θ only if T 3 ( δ ) -T 2 ( δ ) ≤ N 0 for some N 0 > 0 as δ → 0 , otherwise there would be infinitely many terms in the middle as δ → 0 that cannot be controlled by any O ( δ p ) terms. In this case, we have

as log( δ ) →-∞ , which implies that the slopes satisfy the inequality

where the equality holds when p = -log γ λ ( θ ) . Thus, V π θ ( s ) is -log γ λ ( θ ) -Hölder continuous at s = s 0 .

On the other hand, the following counterexample shows that Theorem 4.1 has provided the strongest Hölder-continuity result for V π θ ( s ) at s = s 0 under the assumptions (A.1)-(A.5):

Example 4.1. Consider a one-dimensional MDP s t +1 = f ( s t , a t ) where

with state space S = [ -1 , 1] and cost function c ( s, a ) = | s | . Let the policy be linear, namely π θ ( s ) = θ · s and θ ∈ R . It can be verified that all assumptions (A.1)-(A.5) are satisfied. Now let s 0 = 0 and θ > 1 , then applying (4) directly yields

Let δ > 0 be sufficiently small, then

where T 0 ( δ ) = 1 + ⌊ -log δ log θ ⌋ ∈ N and ⌊·⌋ is the flooring function. Therefore, we have

Remark 4.1. Another way to see why it is theoretically impossible to prove p -Hölder continuity for V π θ for any p > -log γ λ ( θ ) : notice that the inequality (10) no longer holds as log δ →-∞ since

Thus, p = -log γ λ ( θ ) is the largest Hölder exponent of V π θ ( · ) that can be proved in the worst case.

Remark 4.2. The value function V π θ ( s ) is Lipschitz continuous at s = s 0 when the maximal Lyapunov exponent λ ( θ ) < -log γ , since there exists a constant K ′ such that

where δ = ∥ s 0 -s ′ 0 ∥ > 0 is the difference in the initial state.

4.2 Hölder Exponent of J ( · )

The following lemma establishes a direct connection between J ( θ ) and J ( θ ′ ) through value functions:

Lemma 4.1. Suppose that θ, θ ′ ∈ R p , then

where { s θ ′ t } ∞ t =0 is the trajectory generated by the policy π θ ′ ( · ) .

The proof can be found in the Appendix B.2. Notice that indeed we have J ( θ ′ ) = V π θ ′ ( s 0 ) and J ( θ ) = V π θ ( s 0 ) , substituting with these two terms in the previous lemma and doing some calculations lead to the following main theorem whose proof can be found in the Appendix B.3:

Theorem 4.2. (Non-smoothness of J ) Assume (A.1)-(A.5) and the parameterized policy π θ ( · ) is deterministic. Let λ ( θ ) denote the MLE of (1) at θ ∈ R p . Suppose that λ ( θ ) > -log γ , then J ( · ) is -log γ λ ( θ ) -Hölder continuous at θ .

Remark 4.3. In fact, the set of assumptions (A.1)-(A.5) is quite general and does not exclude the case of constant cost functions c ( s, a ) ≡ const , which always results in a smooth landscape regardless of the underlying dynamics, even though they are rarely used in practice. However, recall that the -log γ λ ( θ ) -Hölder continuity is a result of exponential divergence of nearby trajectories, when a cost function can continuously distinguish two separate trajectories (e.g., quadratic costs) with a discount factor close to 1 , the landscape will be fractal as shown in Section 6. Another way to see it is to look into the relaxation in (5) where the Hölder continuity is obtained from the local Lipschitz continuity of c ( s, a ) , i.e., | c ( s, π θ ( s )) -c ( s ′ , π θ ( s ′ )) | ≤ K 2 ∥ s -s ′ ∥ . Therefore, the Hölder continuity is tight if for any δ > 0 , there exists s ′ 0 ∈ B ( s 0 , δ ) such that | c ( s t , π θ ( s t )) -c ( s ′ t , π θ ( s ′ t )) | ≥ K 3 ∥ s t -s ′ t ∥ with some K 3 > 0 for all t ∈ N . We will leave the further investigation for future studies.

The following example illustrates how the smoothness of loss landscape changes with λ ( θ ) and γ : Example 4.2. (Logistic model) Consider the following MDP:

where the policy a t is given by deterministic linear function a t = π θ ( s t ) = θs t . The objective function is defined as J ( θ ) = ∑ ∞ t =0 γ t ( s 2 t +0 . 1 a 2 t ) where γ ∈ (0 , 1) is the discount factor. It is well-known that (13) begins to exhibit chaotic behavior with positive MLEs (as shown in Figure 2a) when θ ≥ 3 . 3 [15], so we plot the graphs of J ( θ ) for different discount factors over the interval θ ∈ [3 . 3 , 3 . 9] . From Figure 2b to 2d, the non-smoothness becomes more and more significant as γ grows. In particular, Figure 2e shows that the value of J ( θ ) fluctuates violently even within a very small interval of θ , suggesting a high degree of non-differentiability in this region.

(a) MLE

λ

(

θ

)

.

(b)

γ

= 0

.

5

.

(c)

γ

= 0

.

9

.

(d)

γ

= 0

.

99

.

(e) Magnified.

0.50

2.9

y= 0.5

12.5

6°0 = A

0.25

0.00

2.8

2.7

12.0

(e)

0.25

2.6

11.5

05°0

11.0

SL'0

2.5

1.00

2.4

10.5

1.25

10.0

3.3

3.4

3.5

3.6

3.7

3.8

3.4

3.5

3.6

θ

3.7

EE

3.4

3.5

3.6

θ

'E

3.8

y= 0.99

112.5

66°0 =/

115

110.0

110

107.5

(e)r

105.0

105

102.5

0'00T

100

97.5

95.0

95

3.4

3.5

3.6

3.7

3.8

0.00000

0.00002

0.00004

0.00006

0.000080.00010

+3.9900000000

4.3 Stochastic Policies

The original MDP (1) becomes stochastic when a stochastic policy is employed. First, let us consider the slightly modified version of MLE for stochastic policies:

where ∆ Z 0 = s ′ 0 -s 0 is a small pertubation made to the initial state and ∆ Z ω ( t ) = s ′ t ( ω ) -s t ( ω ) is the difference in the sample path at time t ∈ N and sample ω ∈ Ω . Since this definition is consistent with that in (4) when sending the variance to 0 , we use the same notation λ ( θ ) to denote the MLE at given θ ∈ R N and again assume λ ( θ ) > -log γ . Since policies in most control and robotics environments are deterministic, this encourages the variance to converge to 0 during training.

However, unlike the deterministic case where the Hölder continuity result was proved under the assumption that the policy π θ ( s ) is locally Lipschitz continuous, stochastic policies instead provide a probability distribution from which the action is sampled. Thus, a stochastic policy cannot be locally Lipschitz continuous in θ when approaching its deterministic limit. For instance, consider the one-dimensional Gaussian distribution π θ ( a | s ) where θ = [ µ, σ ] T denotes the mean and variance. As the variance σ approaches 0 , π θ ( a | s ) becomes more and more concentrated at a = µs , and eventually converges to the Dirac delta function δ ( a -µs ) , which means that π θ ( a | s ) cannot be Lipschitz continuous within a neighborhood of any θ = [ µ, 0] T even though its deterministic limit π θ ( s ) = µs is indeed Lipschitz continuous. The following example illustrates that in this case, the Hölder exponent of the objective function J ( · ) can still be less than 1 :

Example 4.3. Suppose that the one-dimensional MDP s t +1 = f ( s t , a t ) where f ( s, a ) is defined as in (12) over the state space S = [ -1 , 1] and action space A = [0 , ∞ ) . The cost function is c ( s, a ) = s + 1 . Also, the parameter space is θ = [ θ 1 , θ 2 ] T ∈ R 2 and the policy π θ ( ·| s ) ∼ U ( | θ 1 | s + | θ 2 | , | θ 1 | s +2 | θ 2 | ) is a uniform distribution. It is easy to verify that all required assumptions are satisfied. Let the initial state s 0 = 0 and θ 1 > 1 , θ 2 = 0 , then applying (14) directly yields λ ( θ ) = log θ 1 similarly as in Example 4.1. Now suppose that θ ′ 2 > 0 is small and θ ′ = [ θ 1 , θ ′ 2 ] T , then for any ω ∈ Ω in the sample space, the sampled trajectory { s ′ t } generated by π θ ′ has

when s ′ t +1 ( ω ) < 1 . Thus, we have s ′ t +1 ( ω ) = 1 for all ω ∈ Ω and t ≥ T 0 ( θ ′ ) = 1 + ⌊ -log θ ′ 2 log θ 1 ⌋ , which further leads to

using the fact that J ( θ ) = 1 1 -γ . Plugging ∥ θ -θ ′ ∥ = θ ′ 2 into the above inequality yields

where the Hölder exponent is again -log γ λ ( θ ) as in Example 4.1.

Remark 4.4. Consider the 1 -Wasserstein distance as defined in [36] between the distribution δ ( a -µs ) and U ( | θ 1 | s + | θ 2 | , | θ 1 | s +2 | θ 2 | ) , which is given by W 1 ( θ 1 , θ 2 ) = 3 | θ 2 | 2 . It is Lipschitz continuous at θ 2 = 0 , even though the non-smooth result in (15) holds. Therefore, probability distribution metrics, such as the Wasserstein distance, are too "coarse" to capture the full fractal nature of the objective function. This also suggests that further assumptions regarding the pointwise smoothness of probability density functions are necessary to create a smooth landscape with stochastic policies, even though they may exclude the case of σ → 0 as discussed earlier.

5 Estimating Hölder Exponents from Samples

In the previous sections, we have seen that the objective function J ( θ ) can be highly non-smooth and thus gradient-based methods may not work well in the policy parameter space. The question is: how can we determine whether the objective function J ( θ ) is differentiable at some θ = θ 0 or not in high-dimensional settings? Note that J ( θ ) may have different levels of smoothness along different directions. To address it, we propose a statistical method to estimate the Hölder exponent. Consider the objective function J ( θ ) and a probability distribution whose variance is finite. Consider the isotropic Gaussian distribution X ∼ N ( θ 0 , σ 2 I p ) where I p is the p × p identity matrix. For continuous objective function J ( · ) , then its variance matrix can be expressed as

where ξ ′ ∈ R p is obtained from applying the intermediate value theorem to E X ∼N ( θ 0 ,σ 2 I p ) [ J ( X )] and hence not a random variable. If J ( · ) is locally Lipschitz continuous at θ 0 , say | J ( θ ) -J ( θ 0 ) | ≤ K ∥ θ -θ 0 ∥ for some K > 0 when ∥ θ -θ 0 ∥ is small, then it has the following approximation

when σ ≪ 1 . Therefore, (16) provides a way to directly determine whether the Hölder exponent of J ( · ) at any given θ ∈ R p is less than 1, especially when the dimension p is large. In particular, taking the logarithm on both sides of (16) yields

for some constant C where the subscript in V ar σ ( J ( X )) indicates its dependence on the standard deviation σ of X . Thus, the log-log plot of V ar σ ( J ( X )) versus σ is expected to be close to a straight line with slope k ≥ 2 when J ( θ ) is locally Lipschitz continuous around θ = θ 0 . Therefore, one can determine the smoothness by sampling around θ 0 with different variances and estimating the slope via linear regression. Usually, J ( θ ) is Lipschitz continuous at θ = θ 0 when the slope k is close to or greater than 2 , and it is non-differentiable if the slope is less than 2.

6 Experiments

In this section, we will validate the theory presented in this paper through common RL tasks. All environments are adopted from The OpenAI Gym Documentation [5] with continuous control input. The experiments are conducted in two steps: first, we randomly sample a parameter θ 0 from a Gaussian distribution and estimate the gradient η ( θ 0 ) from (3); second, we evaluate J ( θ ) at θ = θ 0 + δη ( θ 0 ) for each small δ > 0 . According to our results, the loss curve is expected to become smoother as γ decreases, since smaller γ makes the Hölder exponent -log γ λ ( θ ) larger. In the meantime, the policy gradient method (3) should give a better descent direction while the true objective function J ( · ) becoming smoother.

Notice that a single sample path can always be non-smooth when the policy is stochastic and hence interferes the desired observation, we use stochastic policies to estimate the gradient in (3), and apply their deterministic version (by setting variance equal to 0) when evaluating J ( θ ) . Regarding the infinite series, we use the sum of first 1000 terms to approximate J ( θ ) . The stochastic policy is given by π θ ( ·| s ) ∼ N ( u ( s ) , σ 2 I p ) where the mean u ( s ) is represented by the 2-layer neural network u ( s ) = W 2 tanh( W 1 s ) where W 1 ∈ M r × n ( R ) and W 2 ∈ M m × r ( R ) are weight matrices. Let θ = [ W 1 , W 2 ] T denote the vectorized policy parameter. For the width of the hidden layer, we use r = 8 for the inverted pendulum and acrobot, and r = 64 for the hopper.

(a)

γ

= 0

.

9

,

deter-

ministic.

(b)

γ

= 0

.

9

, stochas-

tic.

(c)

γ

= 0

.

99

, deter-

ministic.

(d)

γ

=

0

.

99

,

stochastic.

(e)

y

= 1

.

980

x

+

2

.

088

.

0.001000

+4.5078e1

Urear Regression

0.000350

0.000975

0.000325

0.000300

0.000275

0.000250

0.000225

0.000200

4G

4

0.000900

0.000825

0.000950

0.000925

0.000875

0.000850

0.000800

250

450

00

((x)ueA

2#+

18-5

38-I

000000

200000

0.00004

5

0.00006

800000

050000

0.00000

200000

0.00004

0.00006

0.00008

9.00010

000000

9.00002

0:00004

6

0.00006

0.00008

0.00010

9.00000

200000

0.00904

0.00006

0.00008

0.00030

Inverted Pendulum. The inverted pendulum task is a standard test case for RL algorithms, and here we use it as an example of non-chaotic system. The initial state is always taken as s 0 = [ -1 , 0] T ( [0 , 0] T is the upright position), and quadratic cost function c ( s, a ) = s T t Qs t +0 . 001 ∥ a t ∥ 2 , where Q = diag (1 , 0 . 1) is a 2 × 2 diagonal matrix, s t ∈ R 2 and a t ∈ R . The initial parameter is given by θ 0 ∼ N (0 , 0 . 05 2 I ) . In Figure 4a and 4c, we see that the loss curve is close to a straight line within a very small interval, which indicates the local smoothness of θ 0 . It is validated by the estimate of the Hölder exponent of J ( θ ) at θ = θ 0 which is based on (16) by sampling many parameters around θ 0 with different variance. In Figure 3e, the slope k = 1 . 980 is very closed to 2 so Lipschitz continuity (and hence differentiability) is verified at θ = θ 0 . As a comparison, the loss curve of single random sample path is totally non-smooth as shown in Figure 3b and 3d.

Acrobot. The acrobot system is well-known for its chaotic behavior and hence we use it as the main test case. Here we use the cost function c ( s, a ) = s T t Qs t + 0 . 005 ∥ a t ∥ 2 , where Q = diag (1 , 1 , 0 . 1 , 0 . 1) , s t ∈ R 4 and a t ∈ R . The initial state is s 0 = [1 , 0 , 0 , 0] T . The initial parameter is again sampled from θ 0 ∼ N (0 , 0 . 05 2 I ) . From Figure 4a-4c, the non-smoothness grows as γ increases and finally becomes completely non-differentiable when γ = 0 . 99 which is the most common value used for discount factor. It partially explains why the acrobot task is difficult to policy gradient methods. In Figure 4e, the Hölder exponent of J ( θ ) at θ = θ 0 is estimated as α ≃ 0 . 43155 < 1 , which further indicates non-differentiability around θ 0 .

(a)

γ

= 0

.

8

,

deter-

ministic.

(b)

γ

= 0

.

9

,

deter-

ministic.

(c)

γ

= 0

.

99

, deter-

ministic.

(d)

γ

=

0

.

99

,

stochastic.

(e)

y

= 0

.

8631

x

+

2

.

041

-512.54

78.778

78.776

1060

1400

1300

Linear Regression

felf

78.774

78.772

78.770

1020

1000

1200

1100

1000

((x)ueA

2#-

18-1

1P

1-

0.0000

0.0002

0.0004

0:0006

0.0008

0.0010

78.768

0.00000

0.00002

0.00004

6

0.00006

0.00008

0.00010

0.00000

0.00002

0:00004

0.00004

0.00008

0.00010

0.00000

0.00002

0.00004

6

0.00006

0.00008

0.00010

38~+

1o-1

32-4

30-1

10-?

Hopper. Now we consider the Hopper task in which the cost function is defined c ( s, a ) = (1 . 25 -s [0]) + 0 . 001 ∥ a ∥ 2 , where s [0] is the first coordinate in s ∈ R 11 which indicates the height of hopper. Because the number of parameters involved in the neural network is larger, the initial parameter is instead sampled from θ 0 ∼ N (0 , 10 2 I ) . As we see that in Figure 5a, the loss curve is almost a straight line when γ = 0 . 8 , and it starts to exhibit non-smoothness when γ = 0 . 9 and becomes totally non-differentiable when γ = 0 . 99 . A supporting evidence by the Hölder exponent estimation is provided in Figure 5e where the slope is far less than 2 .

(a)

γ

= 0

.

8

,

deter-

ministic.

(b)

γ

= 0

.

9

,

deter-

ministic.

(c)

γ

= 0

.

99

, deter-

ministic.

(d)

γ

=

0

.

99

,

stochastic.

(e)

y

= 0

.

5036

x

-

1

.

250

.

0.010

+9.664e1

0.05

1900

Linear Pegrestion

0.009

0.04

1900

0.008

0.006

0.007

0.005

0.004

0.01

0.03 -

0.01

0.02

0.00

1800

1500

1600

1500

1700

((x)eA

18-2

10-1

0.003

0.0

0.2

0.4

6

0.6

20/0

0.0

0.2

0.4

6

0.6

10

d.2

8.4

0.6

0.0

0.2

0.4

5

0.6

38~+

36-7

36-4

10-1

10

I0-1

19-2

10-1

7 Conclusion

In this paper, we initiate the study of chaotic behavior in reinforcement learning, especially focusing on how it is reflected on the fractal landscape of objective functions. A method to statistically estimate the Hölder exponent at some given parameter is proposed, so that one can figure out if the training process has encountered fractal landscapes or not. We believe that the theory established in this paper can help to explain many existing results in reinforcement learning, such as the hardness of complex control tasks and the fluctuating behavior of training curves. It also poses a serious question to the well-posedness of policy gradient methods given the fact that no gradient exists in many continuous state-space RL problems. Being aware of the fact that the non-smoothness of loss landscapes is an intrinsic property of the model, rather than a consequence of any numerical or statistical errors, we conjecture that the framework developed in this paper might provide new insights into the limitations of a wider range of deep learning problems beyond the realm of reinforcement learning.

References

A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research , 22(98):1-76, 2021.

A. Bagirov, N. Karmitsa, and M. M. Mäkelä. Introduction to Nonsmooth Optimization . Springer, 2014.

M. Barnsley. Fractals Everywhere . Academic Press, Inc., 1988.

Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks , 5(2):157-166, 1994.

G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540 , 2016.

A. Camuto, G. Deligiannidis, M. A. Erdogdu, M. Gürbüzbalaban, U. ¸ Sim¸ sekli, and L. Zhu. Fractal structure and generalization properties of stochastic optimization algorithms. arXiv preprint arXiv:2106.04881 , 2021.

F. H. Clarke. Methods of Dynamic and Nonsmooth Optimization . SIAM, 1989.

K. J. Falconer. Fractal Geometry: Mathematical Foundations and Applications . John Wiley, 1990.

J. Fan, Z. Wang, Y. Xie, and Z. Yang. A theoretical analysis of deep Q-learning. volume 120, pages 486-489, 2020.

M. Fazel, R. Ge, S. M. Kakade, and M. Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. Proceedings of the 35th International Conference on Machine Learning , pages 1467-1476, 2018.

X. Guo and B. Hu. Convergence of direct policy search for state-feedback H ∞ robust control: A revisit of nonsmooth synthesis with Goldstein subdifferential. arXiv preprint arXiv:2210.11577 , 2022.

G. H. Hardy. Weierstrass's non-differentiable function. Transactions of the American Mathematical Society , 17:301-325, 1916.

T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, G. Dulac-Arnold, J. Agapiou, J. Leibo, and A. Gruslys. Deep Qlearning from demonstrations. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), 2018.

M. W. Hirsch, S. Smale, and R. L. Devaney. Differential Equations, Dynamical Systems, and an Introduction to Chaos . Academic Press, 2013.

D. W. Jordan and P. Smith. Nonlinear Ordinary Differential Equations: An introduction for Scientists and Engineers . Oxford University Press, 2007.

S. M. Kakade. A natural policy gradient. Advances in Neural Information Processing Systems 14 , pages 1531-1538, 2001.

H. Kantz. A robust method to estimate the maximal Lyapunov exponent of a time series. Physics Letter A , 185:77-87, 1994.

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

E. N. Lorenz. The Essence of Chaos . University of Washington Press, 1995.

S. W. McDonald, C. Grebogia, E. Ott, and J. A. Yorke. Fractal basin boundaries. Physica , 17D:125-153, 1985.

L. Metz, C. D. Freeman, S. S. Schoenholz, and T. Kachman. Gradients are not all you need. 2021.

J. Millán, D. Posenato, and E. Dedieu. Continuous-action Q-learning. Machine Learning , 49:247-265, 2002.

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, and et al. Human-level control through deep reinforcement learning. Nature , 518:529-533, 2015.

P. Parmas, C. E. Rasmussen, J. Peters, and K. Doya. PIPPS: Flexible model-based policy search robust to the curse of chaos. Proceedings of the 35th International Conference on Machine Learning , pages 4065-4074, 2018.

R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on Machine Learning , pages 1310-1318, 2013.

D. Preiss. Geometry of measures in R n : Distribution, rectifiability, and densities. Annals of Mathematics , 125(3):537-643, 1987.

J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. Proceedings of the 32nd International Conference on Machine Learning , pages 1889-1897, 2015.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. Proceedings of the 31st International Conference on Machine Learning , pages 387-395, 2014.

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. Driessche, T. Graepel, and D. Hassabis. Mastering the game of Go without human knowledge. Nature , 550:354-359, 2017.

H. Suh, M. Simchowitz, K. Zhang, and R. Tedrake. Do differentiable simulators give better policy gradients? Proceedings of the 39th International Conference on Machine Learning , 162:20668-20696, 2022.

R. S. Sutton and A. Barto. Reinforcement Learning: an Introduction . MIT Press, 1998.

R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems 12 , pages 1057-1063, 1999.

P. Thomas and E. Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. Proceedings of The 33rd International Conference on Machine Learning , pages 2139-2148, 2016.

J. N. Tsitsiklis and B. V . Roy. Analysis of temporal-difference learning with function approximation. Advances in Neural Information Processing Systems 9 , pages 1075-1081, 1996.

S. S. Vallender. Calculation of the Wasserstein distance between probability distributions on the line. Theory of Probability and its Applications , 18(4):784-786, 1974.

H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double Q-learning. Proceedings of the AAAI Conference on Artificial Intelligence , 30(1), 2016.

O. Vinyals, I. Babuschkin, W. Czarnecki, and et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature , 575:350-354, 2019.

R. Wang, D. P. Foster, and S. M. Kakade. What are the statistical limits of offline RL with linear function approximation? arXiv preprint arXiv:2010.11895 , 2020.

C. J. C. H. Watkins and P. Dayan. Q-learning. Machine Learning , 8:279-292, 1992.

R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning , 8:229-256, 1992.

L. Xiao. On the convergence rates of policy gradient methods. arXiv preprint arXiv:2201.07443 , 2022.

K. Zhang, B. Hu, and T. Ba¸ sar. Policy optimization for H 2 linear control with H ∞ robustness guarantee: Implicit regularization and global convergence. SIAM Journal on Control and Optimization , 59(6):4081-4109, 2021.

A A Brief introduction to chaos theory

As mentioned in the Introduction, chaos exists in many systems in the real world. Although no universal definition of chaos can be made, there are, indeed, three features that a chaotic system usually possesses [14]:

Dense periodic points;

Topological transitivity;

Sensitive dependence on initial conditions;

In some cases, some of these properties imply the others. It is important to note that, despite the appearance of chaos is always accompanied by high unpredictability, the chaotic behavior is entirely deterministic and is not a consequence of randomness. Another interesting fact is that trajectories in a chaotic system are usually bounded, which drives us to think about the convergence of policy gradient methods beyond the boundedness of state spaces.

(a) Lorenz attractor.

(b) Rössler attractor.

50

17.5

40

15.0

12.5

30

Z

10.0

Z

20

7.5

5.0

10

2.5

0.0

30

20

10

-20-15-10-505101520

-10

0

y

-6-4-2

X

20

X

2

4

6

8

Actually, it can be summarized from the results in this paper that for a given MDP, the following three features contribute most to its chaotic behavior:

Infinite time horizon ( t →∞ ) ;

Continuous state space ( ∥ ∆ Z 0 ∥ → 0 ) ;

Exponential divergence ( λ max > 0 ) ;

Since these features are not necessarily bound to certain types of continuous state-space MDPs, it would be exciting for future studies to investigate other types of MDPs using the framework developed in the paper.

B Proofs omitted in Section 4

B.1 Proof of Theorem 4.1

Proof. Suppose that s ′ 0 ∈ S is another initial state close to s 0 and δ = ∥ s 0 -s ′ 0 ∥ . Let T 1 ∈ N be the smallest integer that satisfies

where M 2 = 1 + max s ∈S c ( s, π θ ( s )) > 0 is the maximum of the continuous function c ( · , π θ ( · )) over S , then applying the Lipschitz condition of c ( · , π θ ( · )) yields

where K 2 > 0 is the Lipschitz constant of c ( · , π θ ( · )) over compact set S . On the other hand, the tail terms in J ( · ) is bounded by

using | c ( s t , π θ ( s t )) -c ( s ′ t , π θ ( s ′ t )) | ≤ 2 M 2 . Combining the above two inequalities yields

and we complete the proof.

B.2 Proof of Lemma 4.1

Proof. For the ease of notation, let s t = s θ t , s ′ t = s θ ′ t , u ( s ) = π θ ( s ) and u ′ ( s ) = π θ ′ ( s ) .

Using the fact that γ t V π θ ( x ′ t +1 ) → 0 as t →∞ from (A.3) yields

and the proof is completed using s ′ 0 = s 0 .

B.3 Proof of Theorem 4.2

Proof. First, we will show that Q π θ ( s, a ) is -log γ λ ( θ ) -Hölder continuous with respect to a . Note that for any given a ∈ A and any a ′ ∈ A such that ∥ a -a ′ ∥ ≪ 1 ,

for some K 3 > 0 using the locally Lipschitz continuity of c and f .

Note that V π θ ( s ) = Q π θ ( s, π θ ( s )) , combining it with Lemma 4.1 yields

using the fact that π θ ( s ) is Lipschitz continuous in a neighborhood of ( θ, s ) ∈ R N ×S for some constant K 4 > 0 and we complete the proof.

C From the perspective of fractal theory

We will go through some basic concepts in fractal theory that are related to the study of non-smooth functions.

C.1 The Hausdorff dimension

We will show that the Hausdorff dimension is well-defined: First, it is clear that when δ < 1 , H s δ ( F ) is non-increasing with respect to s . Thus, H s ( F ) is non-increasing as well. Let s ≥ 0 such that H s ( F ) < ∞ , then for any t > s and any δ -cover { U i } of F , we have

which implies H t ( F ) = 0 by taking infimum on both sides and letting δ → 0 . Therefore, the set { s ≥ 0 : 0 < H s ( F ) < ∞} contains at most one point, which further implies inf { s ≥ 0 : H s ( F ) = 0 } = sup { s ≥ 0 : H s ( F ) = ∞} .

More details regarding the well-posedness of Hausdorff dimension can be found in [3, 8]. In particular, one can easily verify that the Hausdorff dimension coincides with the standard dimension (i.e. s ∈ N ) when F is a regular manifold. Typically, the Hausdorff dimension of a fractal is not an integer, and we will be exploiting this fact through the section. A famous example is the Weierstrass function as shown in Figure 7. A comparison of Figure 2e and Figure 7c (they have the same scale) gives some sense about how non-smooth the objective function can be in practice.

0.15

0.10

Φ+y

0.05

0.00

(x)M

W(x)

0.05

0.10

Φ-

0.15

0.20

B(x,r)

0.25

2.0

1.5

1.0

0.5

0.0

X

1.0

1.5

2.0

D

X

8

1e5+5e1

10

Figure 7: (a) shows how the double sector looks like. In (b) and (c), the Weierstrass function is given by W ( x ) = ∑ ∞ n =0 a n cos( b n πx ) where a = 0 . 6 , b = 7 . The Hausdorff dimension of its loss curve is calculated as dim H L W = 2 + log b a ≃ 1 . 73 . Also, according to [12], such W ( x ) is nowhere differentiable when 0 < a < 1 and ab ≥ 1 .

C.2 Non-existence of tangent plane

Actually, when J ( · ) is Lipschitz continuous on any compact subset of R N , by the Rademacher's Theorem, we know that it is differentiable almost everywhere which implies the existence of tangent plane. As it comes to fractal landscapes, however, the tangent plane itself does not exist for almost every θ ∈ R N , which makes all policy gradient algorithms ill-posed. Although similar results were obtained for higher-dimensional cases as in [26], we focus on the two-dimensional case so that it provides a more direct geometric intuition. First, we introduce the notion of s -sets:

The intuition is that: when the dimension of fractal F is a fraction between 1 and 2, then there is no direction along which a significant part of F concentrates within a small double sector with vertex x as shown in Figure 7a. To be precise, let S ( x, ϕ, ψ ) denote the double sector and r > 0 , then we say that F has a tangent at x ∈ F if there exists a direction ϕ such that for every angle ϕ > 0 , it has

where the first condition states that the set F behaves like a fractal around x , and the second condition implies that the part of F lies outside of any double sector S ( x, ϕ, ψ ) is negligible when r → 0 . Then, the main result is as follows:

Proposition C.1. (Non-existence of tangent planes, [8]) If F ⊂ R 2 is an s -set with 1 < s < 2 , then at almost all points of F , no tangent exists.

Therefore, "estimate the gradient" no longer makes sense since there does not exist a tangent line/plane at almost every point on the loss surface. This means that all policy gradient algorithms are ill-posed since there is no gradient for them to estimate at all.

C.3 Accumulated uncertainty

Another issue that may emerge during training process is the accumulation of uncertainty. To see how the uncertainty entered at each step accumulates and eventually blows up when generating a path along fractal boundaries, let us consider the following toy problem: Suppose that the distance between the initial point θ 0 ∈ R N and the target θ ∗ is d > 0 , and step size δ k > 0 is adapted at the k -th step, as shown in Figure 8a. If there exists c > 0 such that the projection ⟨ θ ∗ -θ 0 , θ k +1 -θ k ⟩ ≥ cdδ k for all k ∈ N which implies that the angle between the direction from θ k to θ k +1 and the true direction θ ∗ -θ 0 does not exceed arccos( c ) , in this case, a successful path { θ k } that converges to θ ∗ should give

using θ k → θ ∗ as k →∞ , which is equivalent to ∑ ∞ k =0 δ k ≤ d c .

9*

02

00

01

(a) Generating path from θ 0 to θ ∗ .

(b) Update

θ

n

+1

from

θ

8

J(0)

0n+1

n

.

On the other hand, when walking on the loss surface, it is not guaranteed to follow the correct direction precisely all the time. For any small step size δ > 0 , the uncertainty fraction u ( δ ) involved in every single step can be estimated by the following result [20]:

Proposition C.2. Let δ > 0 be the step size and β = N + 1 -dim H J where dim H J is the Hausdorff dimension of loss surface of J ( · ) , then the uncertainty u ( δ ) ∼ O ( δ β ) when δ ≪ 1 .

Therefore, we may assume that there exists another c ′ > 0 such that the uncertainty U k at the k -th step has U k ≤ c ′ δ β k for all k = 0 , 1 , ... . Then, the accumulated uncertainty

is bounded when β = 1 (i.e. boundary is smooth) using the earlier result ∑ ∞ k =0 δ k ≤ d c . However, the convergence of ∑ ∞ k =0 δ k no longer guarantees the convergence of ∑ ∞ k =0 δ β k when β < 1 , and a counterexample is the following series:

for all k = 0 , 1 , ... , which implies the uncertainty accumulated over the course of iterations may increase dramatically and eventually cause the sequence θ k to become random when walking on fractal boundaries.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the dynamics of policy gradient training in\nreinforcement learning, and argues that the policy gradient often does\nnot exist due to the lack of regularity of the training objective. The\nauthors derive a threshold on the \"maximum Lypanunov exponent\", a\nquantity they introduce which captures the sensitivity of trajectories\nto policy parameters, beyond which sufficient regularity (and existence\nof the policy gradient) cannot be guaranteed. Beyond this, the authors\ndemonstrate how the Holder coefficient of the training objective as a\nfunction of the policy parameters can be estimated from sample\ntrajectories, and validate via numerical simulations that the Holder\ncoefficient estimated from samples correlates with the stability of\npolicy gradient training.\n\nSTRENGTHS:\nThis work studies an interesting an important problem that deep RL is\nfaced with. The chaos-theory-inspired approach taken by the authors is\nrelevant, and in particular, it is nice to see how the Holder exponent\ncan be approximated from samples. The experimental results are fairly\ninteresting, demonstrating a clear correlation between the estimated\nHolder coefficient and the stability of training.\n\nWEAKNESSES:\nAs I understand it, the main theoretical result of the paper is that for\na small enough Lyapunov exponent, we can prove that the policy gradient\nobjective is smooth enough, but not the converse. That is, for a large\nenough Maximum Lyapunov Exponent, it is shown that sometimes the\nobjective is non-smooth, but clearly it may also still be smooth\n(consider for example any MDP with a constant reward function). It is\nclaimed on line 197 that when the Maximum Lyapunov Exponent is above the\nthreshold, the objective is \"very unlikely\" to be Lipschitz continuous,\nbut I do not find any convincing justification for this beyond\nintuition.\n\nMoreover, even if the objective is not Lipschitz continuous, it is not\nclear to me that this should necessarily raise alarms with regard to\npolicy optimization. A function that is non-diffentiable at even one\npoint (that is, a set of measure zero!) is not Lipschitz continuous, but\nI would not expect this to necessarily pose any severe optimization\nproblems for policy gradient algorithms. If the objective turns out to\nbe fractal (as suggested by the title), that would probably be much more\nsevere, but fractals are not really discussed at all in the paper. The\nauthors at one point claim that a lower Holder exponent *can* imply a\n\"more fractal\" landscape, but no evidence is given that loss landscapes\ntend to be fractal in the RL problems that we encounter.\n\nFurthermore, while I find the method of estimating the Holder exponent\nfrom samples interesting, there are many assumptions made (such as the\n\"tightness\" of the Holder continuity of the objective) that I would not\nnecessarily expect to hold in practice. Consequently, it is not clear to\nme that the constant extracted from this scheme would be an accurate\nrepresentation of the Holder exponent.\n\nThe Maximum Lyapunov Exponent introduced in this work is abbreviated to\nMLE, which should probably be avoided.\n\nQUESTIONS:\nLine 57 says \"Consequently, it is not surprising that [DDPG, TRPO,\nPPO, etc] often have a low probability of obtaining a satisfactory\nsolution\". Is this true? Some of these algorithms have proven to be\nquite successful (albeit very sensitive).\n\nIn section 3.3, the \"divergence\" terms $\\Delta Z(t)$ should be defined\nexplicitly. Moreover, the definition of the Maximum Lyapunov Exponent\ncan be made more clear. It defines $\\lambda_{max}$ as the \"largest\nvalue\" such that equation (6) holds, but it's not clear what the\nmaximum is being taken with respect to here. From the sentence\npreceding definition 3.1, I think there should be a supremum over all\nperturbation directions before the limits.\n\nOn line 155, the paragraph starts by mentioning Example 4.1 which\nthe reader hasn't encountered yet. This doesn't read well.\n\nI do not understand the argument about Lipschitz (dis)continuity for\nstochastic policies around line 223. It says in the zero-variance\nlimit that  $\\pi_\\theta(a\\mid s) =\n\\delta(a - \\mu s)$ is\na Dirac measure and cannot be Lipschitz continuous, despite the fact\nthat the deterministic limit\n$\\pi_\\theta(s) = \\mu s$ is Lipschitz continuous, but you are surely\nreferring to 2 distinct notions of Lipschitz continuity here. The\nformer (as far as I can tell) is a statement about the Lipschitz\ncontinuity of the map $a\\mapsto \\pi_\\theta(a\\mid s)$, whereas the\nlatter is a statement about the Lipschitz continuity of the map\n$s\\mapsto\\pi_\\theta(s)$. These are different things, and it's not\nclear to me why it is meaningful to compare them. The fair comparison\nas I see it would be the map $s\\mapsto \\pi_\\theta(\\cdot\\mid s)$ in the\nstochastic case, which can indeed be Lipschitz continuous if the image\nis metrized by the Wasserstein distance.\n\nOn line 89, it says \"The sample space $\\mathcal{S}$ is\ncompact\". Should it not say that the state space is compact?\n\nWhy is $R$ used to denote a cost function? This is confusing.\n\nLine 110, why does the second assumption only \"apparently\" hold?\n\nOn line 118, what does $\\simeq$ mean? Is $\\|\\Delta Z(t)\\|\\simeq\ne^{\\lambda t}\\|\\Delta Z_0\\|$ the same as\n$\\|\\Delta Z(t)\\| = e^{\\lambda t} + o(1)$?\n\nWhy is assumption A4 natural? Under which conditions on the MDP/policy\nis A4 satisfied?\n\nOther policy gradient works have proposed conditions for which the\npolicy gradient is defined (see e.g. \"Deterministic Policy Gradient\nAlgorithms\" by Silver et. al, 2014; Theorems 1, 2 and conditions A1,\nA2, B1).\nIs your Theorem 2 related to these conditions? Is it easier to verify\nthe conditions of Theorem 2?\n\nIn the experiments, how exactly was the policy gradient computed? In\nparticular, how are you estimating the advantage in equation (3)? How\nare you computing the policy gradient for deterministic policies?\nAlso, equation (3) for $\\eta$ looks like it is already computing a\ngradient w.r.t. parameters, so $\\nabla_\\theta\\eta(\\theta_0)$ does not\nappear to have the right shape, especially in the expression $\\theta =\n\\theta_0 + \\delta\\nabla_\\theta\\eta(\\theta_0)$.\nI'm assuming by $\\nabla_\\theta\\eta$ you just mean $\\eta$.\n\nAre there any promising directions that RL researchers can follow in\norder to solve the \"fractal landscape\" problem?\n\nLIMITATIONS:\nSome assumptions are made for the estimation of the Holder exponent\nthat seem very restrictive. Also, it is not clear to me if the\ncondition on the maximum Lyapunov exponent is easier to verify or more\nintuitive than those given in earlier works on policy gradient\nmethods, which also serve the purpose to ensure that the policy\ngradient exists.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper points out that the value function of some MDPs is not differentiable or even fractal in the infinite horizon setting.\n\nThis raises the question of the well-formedness of policy gradient methods.\n\nA method for statistically estimating the Hölder exponent is presented to find out whether the training process has encountered fractal landscapes.\n\nSTRENGTHS:\n* The paper describes a fundamental problem that has received little or no attention to date.\n* The presentation is well-founded and provides deep insights.\n* The Appendix is a very nice addition to the main text.\n\nRemark:\\\nI observed the presumably fractal structure of the Q-function of an MDP myself in 2017. However, I decided against elaborating and publishing this observation further at that time. I would like to congratulate the authors for their sound and profound elaboration of the description of this phenomenon.\n\nWEAKNESSES:\nI see no weakness.\n\nMinor notes:\n\nSome of the capitalization is inconsistent \"Hölder Exponent\", \"Hölder exponent\". \\\nI suggest:\\\n“Maximum Lyapunov Exponents” -> “maximum Lyapunov exponent”\\\n„Hölder Exponent“ -> „Hölder exponent“\n\nThere are a couple of typos:\\\n„updare” -> “update”\\\n“even small update” -> “even small updates”\\\n„transitons” -> „transitions”\\\n“pertubation” -> “perturbation”\\\n“issues[27]” -> “issues [27]”\n\nThe sentence “Similar result obtained for the H2/H∞  problem [33].” is incomplete. Perhaps it should read “Similar results are obtained for the H2/H∞  problem [33].”\n\nTo make the use of hyphens consistent, it should be \"non-linear\" instead of \" nonlinear\", analogous to \"non-smooth\".\n\n\nIn Subsection 4.1 and 4.2, the mathematical expressions should be in bold, e.g. with boldmath\n\n\nIn the References there are still some unintentional, wrong lowercase letters:\\\ngoldstein\\\nlyapunov\n\nIn “Since these features are not necessarily binded to certain types of continuous state-space MDPs”, instead of \"binded\" it should rather be \"bound\" or \"tied\".\n\nQUESTIONS:\nWould you agree with the view that for stochastic MDPs where the probability distribution of the subsequent states P(s'|s,a) is reasonably smooth and a continuum of subsequent states is possible (probably the condition is that the distribution function is differentiable) there can be no more fractal value function? The probability distribution acts like a smoothing kernel and makes everything differentiable.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to understand the optimization landscape of the policy gradient method in the search for a non-smooth or fractal landscape. The method is based on the chaos theory and non-smooth analysis of the optimization objective. In particular, the paper proposed a method to estimate the Hölder exponents, which is used to identify non-smooth landscapes. The theoretical analysis is discussed with empirical validation.\n\nSTRENGTHS:\n- The paper tackles an important and fundamental problem in policy gradient.\n- Overall a well-written paper with a comprehensive presentation.\n- The theoretical results are explained well with necessary assumptions and analysis.\n- Empirical results are presented in support of the theoretical analysis.\n\nWEAKNESSES:\n- Some experimental details (algorithm, return curve, random seed) are missing.\n- Unclear how the proposed method can be integrated into various effective policy gradient methods (e.g., Vanilla Policy Gradient, PPO).\n\nQUESTIONS:\nIt is unclear what the conclusion of the analysis on the non-smoothness of stochastic policies is in Section 4.3. Are the stochastic policies preferred over deterministic ones if the MDP (task) has a non-smooth landscape? \n\nSome experimental details are missing. The analysis was conducted on several Gym environments; however, the learning curve and RL algorithm details are absent. What algorithm was used for policy learning? Was it a vanilla policy gradient? Was batching employed for parameter updates, or were they updated after every sample? Providing a detailed algorithm would address these questions.\n\nHow the analysis depends on any particular type of policy gradient implementation, like with a batch of the sample or a single sample for parameter update?\n\nTo assess the algorithm's performance, it is crucial to examine its reward. Therefore, in addition to the loss curve, a return curve would provide a better understanding of the challenges involved in solving a complex task with a non-smooth landscape.\n\nAcrobot is mentioned as (lines 301-302) almost unsolvable by the policy gradient method. However, many policy gradient-based algorithms, such as PPO, can already solve this task. See CleanRL benchmark results for PPO (https://github.com/vwxyzjn/cleanrl). Is the analysis in this paper restricted to some particular policy gradient-based methods?\n\nIs the experiment run for multiple random seeds? The execution of the environments (MDP) changes in each episode; thus, multiple runs would give a better understanding. Adding multiple seed runs or explaining why that is unnecessary is crucial.\n\nThe Axis labels of Figures 2, 3, 4, and 5 are too small and hard to read.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tries to argue that in a certain class of continuous state-action MDPs, the policy gradient landscape can be non-smooth. It does so by showing that the Holder exponent of the policy objective $J(\\theta)$ is less than one, when the MDP's maximum Lyapunov exponent is greater than $-\\log \\gamma$. The paper complements this result by analytically analyzing multiple simple MDPs and also by empirically showing this non-smooth behavior for popular MDPs like Acrobot. \n\nUnfortunately, I propose rejecting this paper because I am not confident in its proofs and cannot follow the logic completely. Although, it is possible that I am missing something, in which case I hope that the author's rebuttal can help alleviate my concerns (please see the Weaknesses and Questions sections below.)\n\nSTRENGTHS:\n### Originality and significance:\nThe paper deals with the fundamental issue of characterizing the nature of the policy optimization landscape in continuous state-action MDPs. This setting is highly practical, and to the best of my knowledge, not widely studied. One of the primary advantage of policy gradient methods over value based methods is their ability to naturally deal with continuous action-spaces. And as the paper claims, the optimization landscape is non-smooth which might create significant difficulties for gradient based methods. This knowledge is very useful for both algorithm developers and practitioners. The approach itself seems elementary and novel to me (although I have very limited knowledge about related works). (Also, I mean \"elementary\" in a good way. This seems like those first fundamental results which other people build upon.)\n\n### Quality and clarity: \nThe paper motivates the problem well and provides a thorough literature survey, with an introductory background on chaos and fractal theory included in the appendix, which I think would be appreciated by the readers new to these fields. The main analytical results (Holder exponent of the policy gradient objective being less than one) are complemented by a number of worked out examples (simple MDPs) and corroboration by multiple simulation studies (visualizing the landscape of popular environments such as Acrobot and Humanoid showing the non-smooth nature of the landscape), providing a well-rounded treatment of the subject matter.\n\nWEAKNESSES:\n## Major issues (these affect my score significantly):\n\nThe major weaknesses of the paper are a lack of a coherent argument and flaws in the proofs.\n\n### 1. Disconnect between the stated result and the non-smoothness claim.\nThe main results (Theorem 4.2, line 192) says that the policy gradient objective $J$ is Holder with some exponent less than one. And the paper then concludes that $J$ is non-smooth in the policy parameters. I don't completely understand this connection. I think that if a function is $\\alpha$-Holder for some $0 < \\alpha < 1$ and is not Holder continuous of any other $\\alpha' > \\alpha$, i.e. in some sense $\\alpha$ is \"max Holder coefficient\" (not sure what the terminology is), then it would be non-smooth (can the authors provide the relevant result here as well). However, in the paper, the main result just shows that $J$ is $\\alpha$-Holder for some $\\alpha < 1$; it doesn't show that this is the maximum possible Holder coefficient. There is a discussion following Theorem 4.1, from line 152-177 that argues that the Holder coefficient for the value function should be less than one. However, if that is a result, it should be the main theorem statement (it does seem very important). The non-smoothness could then follow from the bound on the Holder coefficient, and the theorem (which the authors should state) that connects this max Holder coefficient to the non-smoothness of the function.\n\nFurther, from the experimental results (Section 6), all the figures plot $J$ vs $\\delta$. But how that is connected to the non-smoothness of the landscape is a little bit unclear. More explanation would be very helpful. \n\nThe conclusion of the paper says \"It also poses a serious question to the well-posedness of policy gradient methods given the fact that no gradient exists in many  continuous state-space RL problems.\" But I am unable to attribute this to a specific theorem or empirical result given in the paper. \n\n### 2. Potential flaws in the proofs.\nI should state that it is quite possible that these are not really flaws, and I am just unable to follow the proofs. In that case, I would appreciate if the authors could help me here. It is also possible that these are just local problems, and the veracity of the final claims remains valid.\n\n**(a) Proof of Theorem 4.1 (line 455, page 13):**\n\nIn the display following Eq. 18, I don't follow the inequality (ignoring the term $K_1K_2 \\delta$)\n\n$$ \\sum_{t=0}^{T_1} e^{(\\lambda(\\theta) + \\log \\gamma)t} \\leq \\frac{e^{\\frac{\\lambda(\\theta) + \\log \\gamma}{\\lambda(\\theta)} \\log \\left( \\frac{2M}{K_1 \\delta} \\right) + 1}}{e^{\\lambda(\\theta) + \\log \\gamma} - 1}. $$\n\nWhat I am able to compute is the following. Since $\\lambda(\\theta) + \\log \\gamma > 0$, the function $e^{(\\lambda(\\theta) + \\log \\gamma)x}$ is increasing. Therefore, using the fact that $T_1 \\geq \\frac{\\log(2M / K_1 \\delta)}{\\lambda(\\theta)}$, we get\n\n$$ \\sum_{t=0}^{T_1} e^{(\\lambda(\\theta) + \\log \\gamma)t} = \\frac{e^{(\\lambda(\\theta) + \\log \\gamma) (T_1 + 1)} - 1}{e^{(\\lambda(\\theta) + \\log \\gamma)} - 1} \\geq \\frac{e^{\\frac{(\\lambda(\\theta) + \\log \\gamma)}{\\lambda(\\theta)} \\left[ \\log \\left( \\frac{2M}{K_1 \\delta} \\right) + \\lambda(\\theta) \\right] - 1}}{e^{(\\lambda(\\theta) + \\log \\gamma)} - 1}.$$\n\n**The most important thing to note is the inequality sign is flipped.** Maybe this could be fixed using $T_1 \\leq \\frac{\\log(2M / K_1 \\delta)}{\\lambda(\\theta)}$, but then could we guarantee that $T_1$ would exist?\n\nSimilarly, I don't understand the inequality sign in the second big display on page 13: $\\frac{\\gamma^{T_1}}{1 - \\gamma} \\leq \\frac{1}{1 - \\gamma} (K_1 / 2M)^{-\\log \\gamma / \\lambda(\\theta)} \\delta^{-\\log \\gamma / \\lambda(\\theta)}$. \n\nFinally, I don't understand how the Holder continuity result follows from the last equation on page 13. Seems like one step is missing on line 464.\n\n**(b) Proof of Theorem 4.2 (line 466, page 14):**\nI don't understand the third line in the first display. In particular, how to show that\n$$ R(s_t'; u'(s_t')) + \\gamma V^{\\pi_\\theta}(s_{t+1}') = Q^\\theta (s_t', u'(s_t')).$$\n\nUsing Bellman equation (for deterministic dynamics), we know that $Q^\\theta (s_t', u'(s_t')) = R(s_t'; u'(s_t')) + \\gamma V^{\\pi_\\theta} ( f(s_t', \\pi_\\theta(s_t'))$. But is it necessary that $f(s_t', \\pi_\\theta(s_t') = s_{t+1}') = f(s_t', \\pi_{\\theta'}(s_t'))$?\n\n**(c) (MINOR) Proof of Theorem 4.3 (line 471, page 14):**\nI am unable to follow the second line and the fourth line in the first display. (I guess, I just need to think more carefully for these..)\n\n**(d) (MINOR) Example 4.1 (line 183, page 5):** \nI don't understand why $||\\Delta Z(t)|| = ||\\Delta Z_0|| \\theta^t$? I do understand that $||\\Delta Z(1)|| = | f(s_0', \\theta s_0') |$, but is $f(s_0', \\theta s_0') = \\theta s_0'$?\n\nAlso, I don't understand if we get the result $V(\\delta) - V(0) \\geq \\frac{\\gamma}{1 - \\gamma} \\delta^{(-\\log \\gamma / \\log \\theta)}$, why does it mean that $p = -\\log \\gamma / \\log \\theta$ is the largest Holder coefficient? For instance, we know that $V(\\delta) - V(0) = V(\\delta) = \\delta \\sum_{t=0}^\\infty (\\gamma \\theta)^t = \\frac{\\delta}{1 - \\gamma \\theta} \\geq \\frac{\\delta^2}{1 - \\gamma \\theta}$, for small $\\delta$ and $\\theta$. Does that mean $p = 2$ is the largest Holder coefficient? Some additional explanation would be very helpful.\n\n### 3. There is almost no exposition on how the policy optimization landscape has fractal properties.\nThere is an appendix that discusses fractal theory. However, since \"fractal\" appears in the title of the paper, these details (IMHO) shouldn't be relegated to the appendix. In particular, after skimming that appendix, it seems that the discussion there is on general fractal theory, and no explicit connection to the paper's main results (Holder exponent and non-smoothness of the objective function) is made. The main question that remains in my mind is how is the non-smoothness of the landscape connected to it being fractal? (BTW, I think the non-smoothness result in itself is of sufficient interest, and the fractal part could be safely dropped.)\n\n\n\n## Minor issues / suggestions (these do not affect my score as much; please ignore them if you don't agree):\n- line 15: References 17, 24 are not about policy gradient methods\n- line 19 (the probability of obtaining a satisfactory solution can be surprisingly low, or even close to zero in certain tasks such as acrobot): a reference would be nice. I remember the following paper talks about PG methods failing with non-zero probability (maybe that's a little bit relevant?): Mei et al. (2021) Understanding the Effect of Stochasticity in Policy Optimization. NeurIPS. \n- line 25 (Considering that even a small update in policy space can significantly change the performance metric): a reference here would be nice\n- lines 27-38: some symbols ($\\gamma$, $\\sigma$) or choices (such as the policy parameterization) are not specified\n- line 39 (As shown in Section 6, the loss curve is still fractal even when all things in the MDP are deterministic.): I could not find this result. Could you refer to the exact theorem or the experimental result?\n- line 58: Is the edge of stability work a relevant reference for this section? Such as Arora et al. (2022) Understanding Gradient Descent on Edge of Stability in Deep Learning?\n- line 73 (It also explains why the classical RL algorithms in [25] are provably efficient in finite space settings.): some references would be nice\n- lines 105: for this equation, it would be nice if you could point to the exact section of the reference [26] \n- line 110 (However, as we will see in Section 4 and 6, the existence of $\\nabla V$ may fail in many continuous MDPs even if $\\mathcal{S}$ is compact): I don't see this exact result anywhere. Could you please refer to the specific theorem or the experimental result?\n- lines 116-128: could you explicitly define $\\Delta Z_t$\n- line 128 (but is applicable to a broad class of continuous state-space MDPs having any one of these characteristics) what characteristics is the paper referring to?\n- line 133 (Definition 4.1) Should this be called local Holder continuity?\n- regarding the Assumptions (lines 146-148 / 93-97): it would be nice if they are written in the same style as a definition or theorem and you can reference to them in the paper.\n- line 141 (that the MLE of (1), say λ(θ)): I think the system here is not that given by Eq. 1, but rather $s_{t+1} = f(s_t, \\pi(s_t))$\n- line 146 (Assumption 4): This is assumption is the most non-trivial one and very critical to the analysis. Some explanation as to why this is reasonable could be useful.\n- None of the appendices are referenced in the main paper. Further, Appendices A and C seem highly tangential to the main paper, especially because chaos and fractal theory are never explicitly discussed.\n- line 154: this sentence is awkward, has a forward reference to Example 4.1, and doesn't describe that the following text will discuss a lower bound. An appropriate section heading here would be very useful.\n- lines 162-164: define $T_i$s\n- line 164 (Eq. 8): I am unable to get the factor of $e^{(\\log \\gamma) T_2}$ in the inequality.\n- line 176: should the inequality be (sign reversed): $(p-1)/(\\lambda(\\theta) + \\log \\gamma) \\geq p / \\log \\gamma$?\n- line 184: I don't understand how to get the equation following this line. Some more explanation could be helpful..\n- line 195 (Actually, the most important implication of this theorem is that $J(\\theta)$ is not guaranteed **(and is very unlikely)** to be Lipschitz continuous when): why is it very unlikely? Is this an analytical result, or an empirical observation? In either case, some references would be nice.\n- Figure 2-5: the figures and especially their font is extremely small to be read when the paper would be printed\n- line 213: define $\\omega, \\Omega$\n- line 214 (sending the variance to 0): variance of what, the policy?\n- Section 6: all the sub-figures should have axis labels. Also the figure captions could be more descriptive. \n- The text has some minor grammar / typographical issues which should be easy to fix. For instance, line 97 spelling of transitons, or line 133 (definition 4.1): suppose that some scalar $\\alpha > 0$. --- this sentence seems grammatically incorrect.\n\nQUESTIONS:\nI would really appreciate if the authors could clarify the following (please refer to the Weaknesses section for details):\n\n(1) What is the connection between Holder coefficient and the non-smoothness of the objective? And don't we need a bound on the \"max Holder coefficient\" for getting the non-smoothness result?\n\n(2a) Is this an actual flaw in the proof, or am I missing something?\n\n(2b) Is this an actual flaw in the proof, or am I missing something?\n\n(2c) Could you please provide more details?\n\n(2d) Am I missing something?\n\n(3) Does non-smoothness of the objective function directly imply that it is fractal in nature?\n\nLIMITATIONS:\nThe paper does not comprehensively discuss the limitations of the project.\n\nAssumptions, in particular A4, could have more motivation. \n\nThe case when $\\lambda(\\theta) < -\\log \\gamma$ could be further discussed; at the very least some comment on what the authors think would happen in that case could be helpful. (Maybe this reasoning is trivial, but I'll still encourage the authors to add that.)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}