Policy Gradient Algorithms Implicitly Optimize by Continuation
Authors: Adrien Bolland
Gilles Louppe
Damien Ernst
Venue: Accepted by TMLR ()

Abstract: Direct policy optimization in reinforcement learning is usually solved with policy-gradient algorithms, which optimize policy parameters via stochastic gradient ascent. This paper provides a new theoretical interpretation and justification of these algorithms. First, we formulate direct policy optimization in the optimization by continuation framework. The latter is a framework for optimizing nonconvex functions where a sequence of surrogate objective functions, called continuations, are locally optimized. Second, we show that optimizing affine Gaussian policies and performing entropy regularization can be interpreted as implicitly optimizing deterministic policies by continuation. Based on these theoretical results, we argue that exploration in policy-gradient algorithms consists in computing a continuation of the return of the policy at hand, and that the variance of policies should be history-dependent functions adapted to avoid local extrema rather than to maximize the return of the policy.

Published in Transactions on Machine Learning Research (10/2023)
Policy Gradient Algorithms Implicitly Optimize by Continuation
Adrien Bolland
adrien.bollend@hhleg.. be
Montefiore Institute, University of Liège
Gilles Louppe
Shateee be
Monteftore Institude, University of Lidge
Damien Ernst
dernst@rulege..b be
Montefiore Institute, University of Liëge
LTCI, Telecom Poris, Institut Polytechnique de Paris
Reviewed OpenReview: https: //ppre/veeee net/ sermfo id= 3BoHH3m225
Abstract
Direct policy optimization in reinforcement learning is usually solved with policy-gradient algorithms,, which optimize policy parameters via stochastic gradient ascent. This paper provides a new theorctical interpretation and justification of these algorithms. First, we formulate direct policy optimization in the optimization by continuation framework. The latter is a framework for optimizing nonconvex functions where a sequence of surrogate objoctive functions, called continuations, are locally optimized. Second, we show that op- timizing affine Gaussian policies and performing entropy regularization can be interpreted as implicitly optimizing deterministic policies by continuation. Based these theoretical results, argue that exploration in policy-gradient algorithms consists in computing continuation of the return of the policy at hand, and that the variance of policies should be history-dependent functions adapted to avoid local extrema rather than to maximize the return of the policy.
1 Introduction
A Applications where one has control an environment are numerous and solving these control problems efficiently is the preoccupation of many researchers and engineers. Reinforcement learning (RL) has emerged solution when the environments at hand have complex and stochastic dynamics (Sutton & Barto 2018).
Direct policy optimization and more particularly (on-policy) policy gradients are methods that have been successful in recent years. These methods, reviewed by Duan et al. (2016) and Andrychowicz et al. (2020). all consist in parameterizing a policy (most often with a neural network) and adapting the parameters with a local-search algorithm in order to maximize the expected sum of rewards received when the policy is executed, called the return of the policy. We distinguish two basic elements that determine the performance of these mothods. As first clement, we have the formalization of the optimization problem. It defined through two main choices: the (functional) parametrization of the policy and the learning objective function, which mostly relies adding entropy regularization term to the return. As second element, there is the choice of the local-scarch algorithm to solve the optimization problem we focus on stochastic gradient ascent methods in this study.
The policy parameterization the first formalization choice. In theory, there exists an optimal deterministic policy (Sutton & Barto 2018), which can be optimized by deterministic policy gradient (Silver al., 2014) with a guarantee of converging towards a stationary solution (Xiong al. 2022).. However, this approach may give poor results in practice it is subject to convergence towards local optima (Silver et al. 2014

---

Published in Transactions on Machine Learning Research (10/2023)
It is therefore usual to optimize stochastic policies where this problem is mitigated in practice Duan et al,
2016 Andrychowicz et al. 2020). For discrete state and action spaces, theoretical guarantees of global
convergence hold for softmax or direct policy parameterization (Bhandari & Russo 2019
Zhang et al,
2021
Agarwal ct al. 2020 In the general case of continuous spaces, these results no longer hold and only
convergence lowards stationarity can be ensured under strong hypotheses (Bhatt et al. 2019
Zhang et al,
2020b; Bodi et al.
2021). Recently, convergence under milder assumptions was established assuming that
the policy follows heavy-tailed distribution, which guarantees a sufficiently spread distribution of actions Nevertheless, most of the empirical works have focused on (light-tailed) Gaussian policies
[Bddi et al.]
2022
2016 Andrychowicz et al.] 2020) for which convergence is thus not ensured in the general case 2022). The importance of a sufficiently spread distribution in policy gradient had already been observed in early works and was loosely interpreted as exploration (Liliccap el al.) 2015; Mnih et. al.) 2016)
Duan et al.
Bedi et al.
This concept originally introduced in bandit theory and based RL, where it consists in selecting a
suboptimal action to exccute in order to refine a statistical estimate (Simon 1955 Sutton & Barto 2018
is to our knowledge not well defined for direct policy optimization. Other empirical works also showed that relying on Beta distributions when the set of actions is bounded within interval outperformed Gaussian policies (Chou et al. 2017 Fujita & Maeda 2018). As side note, another notable advantage of stochastic policies is the possibility to rely on information geometry and use efficient trust-region methods to speed up the local-search algorithms (Shani et al. 2020 Cen et. al. 2022). In summary, no consensus has yet been reached on the exact policy parameterization that should be used in practice.
The second formalization choice is the learning objective and more particularly the choice of entropy reg- ularization. Typically, bonus enforcing the uniformity of the action distribution is added to the rewards in the objective function (Williams & Peng 1991 Haarnoja et al.. 2019). Intuitively, it avoids converg- ing too fast towards policies with small spread, which are subject to being locally optimal. More general entropy regularizations were applied for encouraging high-variance policies while keeping the distribution sparse (Nachum et al. 2016) or enforcing the uniformity of the state-visitation distribution in addition to the action distribution Islam et al. 2019). Again, no consensus is reached about the best regularization to use in practice.
The importance of introducing sufficient stochasticity and regularizing entropy is commonly accepted in the community. Some preliminary research has been conducted to develop theoretical foundation for this observation. [Ahed et al. (019) proposed an empirical analysis of the impact of the entropy regularization term. They concluded that adding this term yields a smoothed objective function. A local-search algorithm will therefore be less prone to convergence to local optima. This problem was also studied by Husain et al. (022)) They proved that optimizing a policy by regularizing the entropy is equivalent to performing a robust optimization against changes in the reward function. This result was recently reinterpreted by Brekelmans et al. (2022) who deduced that the optimization is equivalent to game where one player adapts the policy while an adversary adapts the reward. The research papers that have been reviewed concentrate solely on learning objectives in the context of entropy regularization, leaving unanswered the question of the relationship between a policy's 's return and the distribution of actions. This question is of paramount imporlance for understanding how the formalization of the direct policy optimization problem impacts the resulting control strategy.
In this work, we propose a new thcoretical interpretation of the effects of the action distribution on the objective function. Our analysis is based on the theory of optimization by continuation (Allgower & Georg, 1980), which consists in locally optimizing a sequence of surrogate objective functions. The latter are called continuations and are often constructed by filtering the optimization variables in order to remove local optima. Our main contributions are twofold. First, we define a contimuation for the return of policies and formulate direct policy optimization in the optimization by continuation framework. Second, based on this framework, we study different formulations, i.e., policy parameterization and entropy regularization, of direct policy optimization. Several conclusions are drawn from the analysis. First, we show that the continuation of the return of deterministie policy is equal to the return of a Gaussian policy. Second, we show that the continuation of the return of Gaussian policy equals the return of another Gaussian policy with scaled variance. We then derive from the previous results that optimizing Gaussian policies using policy- gradient algorithms and performing regularization can be interpreted as optimizing deterministic policies by
2

---

Published in Transactions on Machine Learning Research (10/2023)
continuation. In this regard, exploration as it is usually understood in policy gradients, consists in computing the continuation of the return of the policy at hand. Finally, we show that for more general continuation, the continuation of the return of a deterministic policy equals the return of a Gaussian policy where the variance is a function of the observed history of states and actions. These results provide a new interpretation for the variance of a policy: can be seen as a parameter of the policy-gradient algorithm instead of an element of the policy parameterization. Morcover, to fully exploit the power of continuations, the variance of a policy should be a history-dependent function iteratively adapted to avoid the local extrema of the return.
Optimization by continuation provides aims to provide two main practical advantages to solve optimization problems. First, these methods smooth the objective function and allow the application of gradient- optimization methods, as discussed in the literature on variational optimization (Staines & Barber 2012) and applied by Murray & Ng (010)) to discrete optimization problems. Second, it enables to compute the global optimum of the optimization problem. This global optimum can be reached for example assuming that the optimum of the first continuation is simple to compute, and that the path of local optima of cach continuation leads to the global optimum of the problem (Allgower & Georg] 1980). Theoretical guarantees convergence are still scarce in the literature. Several works focus on Gaussian continuations where the contimuations are convolutions of the objective function by Gaussian kernels (Mobahi & Fisher III, 2015). It is particularly useful when the objective function is concave, ensuring smoothness of the surrogate optimization problems and providing convergence guarantees to the global optimal solution (Nesterov & Spokoiny 2017). Interestingly, a recent work links these continuations to concave envelopes (Mobahi & Fisher 2015). Another notable result holds for similar continuations relying on uniform kernels for building the surrogate problems where convergence to the global solution is guaranteed under certain assumptions on the objective function (Hazan et al. 2016). In the general case, finding the right sequence of continuations for achieving convergence to the global optimum remains heuristic and problem-dependent. The framework of optimization by continuation can nevertheless provide valuable insights, as will be further explored in this work.
Despite the lack of theoretical guarantees, optimization by continuation has found successful machine learning applications, including image alignment (Mobahi et al. 2012), greedy layerwise training of neural networks [Bengio| 2009), and neural network training by itcratively incrcasing the non-linearity of the activation functions (Pathak& & Paffenroth 2019 To our knowledge, optimization by continuation has never yet been applied to direct policy optimization. However, optimizing distribution over the policy parameters rather than directly optimizing the policy is reinforcement learning technique that has been used to perform direct policy optimization (Schnke et al. 2010 Salimans et al. 2017 Zhang et al.] 2020a). Among other things, it decreases the variance the gradient estimates in some cases. this distribution over policy parameters is a Gaussian, it is furthermore by definition equivalent to optimizing the policy by Gaussian continuation (Mobahi & Fisher III, 2015). Another method, called reinforcement learning with logistic rewihtdweighttd regression (Wierstra et al. 2008 Peters & Schaal 2007), consists in optimizing surrogate objective of the return. The surrogate is the expected utility of the sum of rewards. Originally justified relying on the field of decision theory (Chernoff & Moses 2012 it can equivalently be seen as an optimization by continuation method.
The paper is organized as follows. In Section 2 the background of direct policy optimization is reminded. The framework for optimizing policies by contimuaioon is developed in Section 3 and thcoretical results relating the return of policies to their continuations are presented in Section 4 In Section 5 these results are used for elaborating on the formulations of direct policy optimization. Finally, the results are summarized and further works discussed in Section G
2 Theoretical Background
In this section, we remind the background of reinforcement learning in Markov decision processes and discuss the direct policy optimization problem.
3

---

Published in Transactions on Machine Learning Research (10/2023)
2.1 Markov Decision Processes
We study problems in which an agent makes sequential decisions in stochastic environment in order to maximize an expected sum of rewards (Sutton & Barto 2018p. The environment is modeled with an infinite- time Markov Decision Process (MDP) composed of state space S, an action space A, an initial state distribution with density po: a transition distribution (dynamic) with conditional density p, a bounded reward function p. and a discount factor 7 E [0, 1[- When an agent interacts with the MDP (S,A, Po- P: P: 7), first, an initial state so po(-) is sampled, then, the agent provides at each time step an action as A leading to new state St+1 p(-|s,, at). Such sequence of states and actions h2 = (Su. ($u,00,-. ao. ................................ St-1, 4,-1, 81) E H is called a history and H is the set of all histories of any arbitrary length. In addition, al each time step I, reward = p(8,, 04) E IR is observed.
A (stochastic) distnre-depennent policy E P(A) is a mapping from the set of historics H to the set of probability measures on the action space P(A), where n(a(h) is the associated conditional probability density of action a given the history h. A (stochastic) Markov policy и E II= S - P(A) mapping from the state space S to the set of probability measures on the action space P(A), where T(ass) is the associated conditional probability density of action in state s. Finally, deterministic policies E M = S - A are functions mapping an action = pu() E A to each state E S. We note that for each deterministic policy l there exists an equivalent Markov policy, where the probability measure is a Dirac mcasure on the action a = ={(s) in each state 8. In addition, for each Markov policy, there exists an equivalent history- dependent policy only accounting for the last state in the history. We therefore write by abuse of notation that M ç II _. E.
The function J + R is defined the function mapping to any policy ny the expected discounted cumulative sum of rewards gathered by an agent interacting in the MDP by sampling actions from the policy nj. The value (1) is called the return of the policy v) and is computed follows:
Σ p() 1=
J(n) = E
(1)
sompol)) ( (Pe)) #t 1 (pp(((1)))))
An optimal agent follows an optimal policy 1y" maximizing the expected discounted sum of rewards J.
2.2 Direct Policy Optimization
Problem statement. Let (S,A, po. P. p. 7.7) be an MDP and let ne be a policy parameterized by the real vector 0 e de The objective of the optimization problem is to find the optimal parameter 6" Rdd such that the return of the policy is maximized:
0- argmax ((1o).
BERda
In this work, we consider on-policy policy-gradiee algorithms (Andrychowicz et al. 2020). These algorithms optimize differentiable policies with local-search methods using the derivatives of the policies. They itera- tively repeat two operations. First, they approximate an ascent direction relying on historics sampled from the policy, with the current parameters, in the MDP. Second, they update these parameters in the ascent direction.
Deterministic Policies. It is possible to approximate solution of the optimization problem equation (2) for deterministic policy parameterized with a function approximator E .M by stochastic gradient ascent using the deterministic policy gradient theorem (Silver et al.) 2014). Nevertheless, optimizing deterministic policies with inadequate exploration (i.c., without sufficient additional policy randomization) usually results in locally optimal policies with poor performance (Silver et al. 2014).
Gaussian Policies. In direct policy optimization, most of the works focus on learning Markov policies where the actions follow Gaussian distribution whose mean and covariance matrix parameterized
4

---

Published in Transactions on Machine Learning Research (10/2023)
function approximators (Duan et al. 2016 Andrychowicz et al, 2020 More precisely, parametrized Gaussian policy =/ E IT is a policy where the actions follow Gaussian distribution of mean pa(s) and covariance matrix [s(s) for each state and parameter 0, it thus has the following density:
(als) (s)
(3)
Affine Policies. parameterized policy (deterministic or stochastic) said to be affine, if the function approximators used to construct the functional form of the policy are affine functions of the parameter 0. Formally, each function approximator fo of history-dependent policy has the following form wh H:
fo(h) (h)" b(h);
(4)
where and b are functions building features from the histories. Such policies are often considered in theoretical studies Busoniu et al. 2017) and perform well on complex tasks in practice (Rajeswaran et al. 2017
3
Optimizing Policies by Continuation
In this section, introduce the optimization by continuation methods and formulate direct policy opti- mization in this framework.
3.1 Optimization by Continuation
Optimization by continuation (Allgower & Georg 1980 is a technique used to optimize nonconvex functions with the objective of avoiding local extrema. A sequence of optimization problems is solved iteratively using the optimum of the previous iteration. Each problem consists in optimixing a deformation of the original function and is typically solved by local search. Through the iterations, the function is less and less deformed. Such procedure is also sometimes referred to as graduated optimization (Blake & Zisserman, 1987) or optimization by homotopy (Watson & Haftka, 1989). as the homotopy of a function refers to its deformation in topology.
Formally, let f: X + the real-valued function to optimize. Let g y - be another real-valued function used for building the deformation of f. Finally, let the conditional distribution function p p: X P(Y) be the mapping from optimization variable that p(ylz) is the associated density function for any random event y E y given E X. The continuation of the function f under the distribution and deformation function is defined as the function f X + R such that Vz E X:
X' to the set of probability measures PP)),, such
fx
E y~p((-1))
[o(v)]
(5)
For the optimization by continuation described hereafter, there must exist conditional distribution p" for which fe equals f in the limit p approaches p". A typical example is to choose the function equal to J, and to use a Gaussian distribution with a constant diagonal covariance matrix for the distribution p. We then have so-called Gaussian continuations (Mobahi & Fisher III 2015).
Finally, optimizing a function f by continuation involves iteratively locally optimizing its continuation for a sequence of cona distributions approaching p" with decreasing spread. Formally, let po > Pt > ...> Pr-1 be sequence of conditional distributions (monotonically) approaching p* with strictly decreasing covariance matri Then, optimizing f by continuation consists in locally optimizing its continuation f with local-search algorithm initialized at x; for each ileration i. This general procedure is summarized in Algorithm [ Particular instances of this algorithm are described by Hazan et al. (006)) and Shao et al. (2019h for Gaussian continuations.
In this work, we consider the n.2 of functions and the Laewner order over of covariance matrices Siotani
1967
5

---

Published in Transactions on Machine Learning Research (10/2023)
In practice, the optimization process be approximated by performing a limited number of local-scarch iterations at each step of the optimization by continuation. In the following sections, we consider that each optimization of the contimuation fPi is approximated with a single gradient ascent step and that the contimuation distribution sequence Po > P1 >... > PI-1 is constructed by itcratively reducing the variance of the distribution Pt- Note that if this variance reduction is sufficienlly slow, and the stepsize well chosen, single gradient ascent step enables accurately approximate z--
Algorithm Optimization by Continuation
Provide a sequence of functions po > Pi > ..... > Pr-1 Provide an initial variable value z X for the local search
do 4: 2741 < Optimize the continuation f by local search initialized at 2!
for i = 0, 0.1.....,
5: end for
6: return 27
3.2
Continuation of the Return of a Policy
The direct policy optimization problem usually consists in maximizing a nonconvex function. Optimization by continuation is thus a good candidate for computing a solution. In this section, we introduce a novel continuation adapted to the return policies.
The return of a policy depends on the probability of a sequence of actions through the product of the density ne (e))ss)) of each action a for a given parameter 0, see equation ( We define the continuation of interest the expectation of the return where each factor in the product of densities depends vector. This expectation is taken according to distribution that disturbs these parameter vectors at each time step with a variance depending on the history. Formally, using the notations from Section 3.1 we optimize the function f that for all x = 0 equals the return, f()) = (TTO, over the set X' = Rde Let the covariance function A - Rd de be a function mapping a history he E H to covariance matrix A(h)). Let the continuation distribution q be a distribution such that (0, Ahh)) the density of 0g distributed with mean and covariance matrix A(ha). Then, let y (S x A Rd)) be the set of (infinite) sequences of states, actions and parameters and let and g, the two functions defining the continuation, be as follows:
different parameter
p(yl=) p(so) II 1=0
no, (1.:ba))))
(6)
g(v)= = Σ ((),
(7)
t=0
where (0c((())) (0,(0, A(h))) such that the spread of pe depends the function A. Taken logether, the continuation fA fr of the return of the policy no E corresponding to the distribution q and covariance
function A, is defined v0 e igde as:
F2(0)
Σr'(ss,aa)
(8)
sum 0.-0(-10A((h))
e (-1be)
Finally, the continuation equation { converges towards the return of no in the limit the covariance function A approaches zero, required in Section 3.1
This continuation is expected to be well-suited for removing local extrema of the return for three main reasons. First, marginalizing the variables a function as in our continuation is expected to smooth this function and therefore remove local extrema the particular case of Gaussian blurring has been widely studied in the literature (Mobahi & Fisher 2015; Nesterov & Spokoiny (017)). Second, we underline the interest of considering continuation in which the disturbance of the policy parameters may vary based

---

Published in Transactions on Machine Learning Research (10/2023)
the time step. Indeed, changing the parameter vector of the policy at different time steps (and changing the action distributions) may modily the objective function in significantly different ways. Third, we justify the factorization of the conditional distribution pe equation (6) by the causal effect of actions in the MDP. As the actions only inftuence the rewards to come, the past history is expected to provide a sufficinnt statistic for disturbing the parameters in order remove local optima. We therefore chose parameter probabilities conditionally independent given the past history. This history-dependencc is encoded through the covariance function in equation (8).
Maximizing f to solve the optimization problem from Algorithm is complicated task. A common local-search algorithm used in machine learning is stochastic gradient ascent, which is known for performing well on several complex functions depending on many variables (Bottou 2010). The gradient of S can be computed by Monte-Carie sampling applying the reparameterization trick Goodfellow et al., 2016) for simple continuation distributions or relying on the REINFORCE trick (Williams 1992] in the more general case. Due to the complex time dependencies of the random events, these vanilla gradient estimates have practical limitations: the estimates may have large variance, the infinite horizon shall be truncated, and the direction provided is computed in the Euclidean space of parameters rather than in space of distributions (Peters & Schaal 2008). Finally, the evaluation of the continuation and its derivatives require one to sample paramcter vectors, which may be computationally expensive for complex high-dimensional distributions. The study of different contimuation distributions and the application of the optimization procedure from Algorithm [ to practical problems left for further works. In this work, we rather rely on the continuation to study existing direct policy optimization algorithms. To this end, we show in the next section that maximizing the continuation defined by equation is equivalent to solving direct policy optimization problem for another policy, called mirror policy.
4 Mirror Policies and Continuations
This section is dedicated to the interpretation of the continuation of the return of a policy. We show it equals the return of another policy, called a mirror policy. The existence and closed form of mirror policies also discussed.
4.1 Optimizing by Continuation with Mirror Policies
Definition Let (S.A. A...A.ooPPP.. P.p.)) be an MDP and let ne E be a history distory-dependent policy parameterized with the vector 0 TRdo_ In addition, let f% be the continuation of the return of the policy no corresponding to continuation distribution q and covariance function A defined in equation (8) We call mirror policy of the original policy no, under the continuation distribution and covoriance function A, any history-dependent policy vo E E such that V0 E TRde
0)) ((16).
(9)
Let us assume we are provided with the continuation S of the return of an original policy he depending on the parameter 0 that shall be optimized. In addition, let us assume we can compute a mirror policy T/ for the original policy n7. By Definition [ the continuation of the original policy equals the return of the mirror policy for all 0. In addition, under smoothness assumptions, all their derivatives are equal too. Therefore, maximizing the continuation of original policy by stochastic gradient ascent can be performed by maximizing the return of its mirror policy by policy gradient. Applying ofate-of-the-art policy-gradies algorithms on the mirror policies for optimizing the continuations in Algorithm may alleviate several of the shortcomings of the optimization procedure described earlier.
4.2 Existence and Closed Form of Mirror Policies
In this section, we first show that there always exists a mirror policy. In addition, several closed forms are provided depending on the original policy, the continuation distribution, and the covariance function.
Theorem 1. For any original Distory-dependert policy no E E parameterized with the vector f E B and for any continuation distribution q and covoriance, function A, there erists mirror history depeeded policy
7

---

Published in Transactions on Machine Learning Research (10/2023)
No E & of the original policy no that writes
E r/ (a/ah) 0' 0'~a(100A(0)) (nor(a)h))
(10)
Theorem 1 guarantees the existence of mirror policies. Such mirror policy is a function depending on the same parameters as its original policy but that has a different functional form and may therefore provide actions following a different distribution compared to the original policy.
Theorem | leads to two important corollary results. First, as demonstrated in Theorem 2 in Appendix A let r/" be mirror policy of 1/ and let r/ be mirror policy of the original policy n of the form of equation (10). Then, there exists a continuation for which v/ is mirror policy of the original policy n. It follows that the return of the mirror policy of another mirror policy is itself equal to continuation of the original policy. Second, Theorem D also reveals that for a given original policy and continuation distribution, the variance of the mirror policy is defined through the continuation covariance function A. Furthermore, we remind that the variance of the continuation is an hyperparameter that shall be selected for each iteration of the optimization by continuation, see Section 3 This choice of hyperparameter is thus reflected as the choice of the variance of a mirror policy. The expert making this choice sees the effect of the disturbed parameters the enviromment through the variance of the mirror policy. From practical perspective, it is probably casier to quantify the effect on the local extrema depending on the variance of the mirror policy rather than depending on the variance of the continuation.
Property 4.1. Let the original policy To E II be a Markov policy and let the covariance function depend solely on the last state in the history. Then, there erists a mirror Markov policy m/ E IT.
4.1 is an intermediate result providing sufficient assumptions the continuation for having mir- ror Markov policies. Note that for this type of contimuation, the parameters of the policy are disturbed independently of the history followed by the agent.
Property
Property 4.2. Let the original policy GP Il be a Gaussian policy as defined in equation 13 with offine function approzimators. Let the covariance function depend solely on the last state in the history and let the distribution q be Gaussian distribution. Then, there cxists mirror Markov policy x/ E such that for all states s S, it converges towards a Gaussion policy in the limit as the affine coefficients of the covoriance matrix Eg(s) approaches zero (IVV))(())I + +0)
<ap(|ss) + N(aluo(ss, (()())
(11)
where - ==(s)=
+ Eo(s) and Ca(s) = (sp A(s) Vauo(s).
Under the assumptions of Property 4.2 a mirror policy can be approached by a policy that only differs from the original one by having a variance which is increased by the term Co(s) proportional to the variance of the continuation. In particular, when the variance of the original policy T T is solely dependent on the state, then [s)]ll = O and (a(sss = (aaluus(s) (s)). In this case, for any 0, the covariance matrix of this mirror policy is additionally bounded from below such that 2g(s) 2 Co(s).
Property 4.3. Let the original policy μe E M be an affine deterministic policy. Let the covariance function depend solely on the last state in the history and let the distribution g be a Gaussian distribution. Then, the Markov policy mg III is mirror policy:
CP' To (als) (odl))=[(appp)) -x(alpo(s), Eb(s),
(12)
where (x() (sp A(s) Vopo(s).
Therefore, under some assumptions, disturbing deterministic policy and optimizing it afterwards can be interpreted as optimizing the continuation of the return of this policy.
Property 4.4. Let the original policy P40 E M be an affine deterministic policy. Let the distribution be a Gaussian distribution. Then, the policy 1/e E is a mirror policy:
1/ h) = (alluo(s)) E%
(13)
8

---

Published in Transactions on Machine Learning Research (10/2023)
where (h) Veuu(s)) A(h) Volu(s).
Property 4.4 extends Property 4.3 to more general continuation distributions. This extension is used later to justify the interest of optimiziig history-dependent policies in order to optimize an underlying deterministic policy by continuation. The theorem and properties are shown in Appendix A
5 Implicit Optimization by Continuation
In this section, two formulations, ie., a parameterized policy and a learning objective each, used by several policy-gradient algorithms are analyzed relying original and mirror policies. In Section [..]] we show that optimizing each formulation by local search corresponds to optimizing continuation. The optimized policy is thus the mirror policy of an unknown original policy. We show the existence of the corresponding continuation and original policy and discuss their closed form. This analysis provides a novel interpretation of the state-of-the-ar algorithms for direct policy optimization. We discuss the role of stochastic policies in
light of this interpretation in Scction 5.2
5.1 Gaussian Policies and Regularization
The policy-gradient literature has mainly focused on optimizing two problem formulations by local search typically with stochastic gradient ascent and (approximate) trust-region methods. First, the vast majority of works focuses on optimizing the return of Gaussian policies (Duan et al. 2016 Andrychowiez et al. 2020). Second, in many formulations this objective function exlended by adding a bonus to the entropy of the optimized policy (Williams & Peng, [991] Haarnoja et al., 2019p. We show that when optimizing policy according to these formulations, there exists an (unknown) deterministic original policy and continuation under which the optimized policy is a mirror policy. Provided with the local-search algorithm from the policy-gradient method, we conclude that optimizing both formulations is equivalent to implicitly optimizing deterministic policy by continuation.
First, we remind that under Property 4.3 for any affine deterministic policy μe: there exists an alfine Gaussian mirror policy my GGP' as defined by equation (2).. In Property 5.1 the converse of Property 4.3 is stated, which answers to the question: under which conditions a Gaussian policy is the mérror policy of on (unknoun) determánistic policy. For this converse statement to be true, the transformation between covariance functions in Property 4.3 must. be surjective, which is guaranteed if d S de and Vona(() is full rank. The first assumption is always met in practice and the sccond is met when no action is a deterministic function of the others.
GCP' be an affine Gaussian policy with mean function μa, and with covariance function Property 5.1. Let no
Σg = E' constant with respect to the parameters of the policy (i.e., function depending solely on the state). If dd de and if Vopa(s) is full rank, then, there erists continuation, with covariance A proportional to E' for which Tg is mirror policy of the original policy plg
Entropy regularization ensures that the variance of the policy remains sufficiently large during the opti- mization process Similar objectives are pursued with maximum entropy reinforcement learning (Harrooj et al. 2019) or with (approximate) trust-region methods where the trust-region constraint is dualized (Schul- man et al. 2015 2017). Let us consider an affine Gaussian original policy = with constant covariance Eg = Σ. Under Property 4.2 there exists another affine Gaussian policy T GGP' that is a mirror policy of GP This mirror policy has the same mean function and a covariance function bounded from below by Co C. Property 5.2 provides the converse and answers to the question: under which conditions a Gaussian policy with sufficiently larye covoriance is the mirror policy of an (unknown and Gaussian) policy. Similar to the previous property, this is guaranteed when d S de and Vopo(s) is full rank.
FFormally, for two matrices A and B, have that A 2 B > |^] policy is a concave function of the determinant of the covariance matrix, bounded covariance matrix implics bounded entropy. The retrooy-regullaiiattn entropy-buunded optimization problem.
Siotani 1967 As the entropy of Caussian
learning objective can therefore be interpreted as the Lagrangian relaxation of the latter
9

---

Published in Transactions on Machine Learning Research (10/2023)
Property 5.2. Let & be an affine Gaussian policy with mean function μe; and with covariance function E = = - C constant with respect to the porameters of the policy (f.e., a function depending solely on the state) and bounded from bellow by C. If d S. de and if Vapa(s) is full rank, then, there erists a continuation, with covariance A proportional to C, for which Th is a mirror policy of an original Gaussian policy GP with the same mean function pe and with conslant covariance function Σ E/
The two previous properties indicate that a Gaussian policy is guaranteed to be a mirror policy of another policy, Gaussian deterministic, under some assumptions. If we furthermore guarantee that the continuation covariance decreases during the optimization, policy-gradient algorithms optimizing affine Gaussian policies can be interpreted as algorithms optimizing an original policy by continuation.
Let us consider two cases, each corresponding lo a problem formulation, where we optimize by policy gradient an affine Gaussian policy T with covariance function constant with respect to the parameters of the policy. First, we consider the case where its covariance matrix decrcases during the optimization through a manual scheduling. In this context, under property 5.I] there exists an original deterministic policy and the covariance of the continuation decreases through the optimization, such that the saaiiy-gradient algorithm optimizes this policy by continuation. Second, we consider the case where the entropy is regularized with a decrcasing regularization term (e.g., by scheduling the Lagrange multiplier). Then, as entropy regularization can be seen as a constraint on the covariance of the policy, under property 5.2, there exists an original Gaussian policy and the covariance of the continuation decreases through the optimization, such that the policy-gradient algorithm optimizes this stochastic policy by continuation. Finally, as stated previously and shown in Theorem 2 in Appendix A optimizing the return of the mirror policy of another mirror policy is equivalent to optimizing continuation of the original policy. Therefore, policy-gradient algorithms that optimize affine Gaussian policies with both discounted covariance and decreasing regularization by local search can also be interpreted as algorithms optimizing the mean function (i.e., a deterministic policy) of this policy by continuation.
We now illustrate how policy-gradient algorithms implicitly optimize by continuation. We take as example an environment in which a car moves in a valley and must reach its lowest point, (positioned in Ftarget) to maximize the expected sum of rewards gathered by the agent, see Appendix B We assume we want to find the best K-controller, i.c., deterministic policy po(x)) = 0 x (x #ttarget); where x is the position of the car. Directly optimizing such a policy is in practice subject to converging to a local extremum, explain hereafter. We thus consider the Gaussian policy T& (a(x) = [(a)po(z) o'), where po(x) and o' are the mean and variance of the policy, respectively. This policy is a mirror policy of the deterministic policy Hi under a continuttio of variance À = d'/(x - Itarget)?, see Property 4.3 As can be seen in Figure [. for each value of o', the return of the mirror policy equals the smoothed return of the original deterministic policy pa. Consequently, optimizing by policy gradient the Gaussian policy is equivalent to optimizing the deterministic policy by continuation. For well-chosen sequence of o', with fixed scheduling with adequate entropy regularization, the successive solutions found by local search will escape the basin of attraction of the suboptimal parameter for any initial parameter of the local scarch - whereas optimizing the deterministic policy directly would provide suboptimal solutions.
In this section, we have established an equivalence between the optimization of some policies by policy gradient and the optimization of an underlying policy by continuation. It opens up new questions about the hypothesis space of the (mirror) policy to consider in practice in order to exploit the properties of continuations at best. These considerations are made in the next section. We finally recall that central assumption in the previous results is the affinity of policies. Seemingly restrictive, such policies allow to optimally control complex environments in practice (Rajeswaran et al. 2017) and give first-order results for non-affine policies.
5.2 Continuations for Interpreting Stochastic Policies
In practice, we know that optimizing stochastic policies tends to converge to a final policy with low vari- ance and better performance than if we had directly optimized a deterministic policy. Practitioners often justify this observation by the need to explore through a stochastic policy. Nevertheless, to our knowledge, this concept inherited from bandit theory is not well defined for direct policy optimization. The previous
10

---

Published in Transactions on Machine Learning Research (10/2023)
70-
60-
Roturn
50-
40-
30
2
Figure 1: Illustration the return of the policies Maalpe (2)), a'), where uo(x)) = x (x Itarget), for different o' values. The darker the curve, the smaller o', and the darkest one is the return of the deterministic policy pg. The green dots represent the global maxima and the red dots the local maxima. For some sufficiently large value for o', the return of the policy has a single extremun. For a well chosen schedule of decreasing o', a local-search algorithm will track the sequence of global extrema and converge towards the optimal deterministic policy.
analysis establishes an equivalence between optimizing stochastic policics with aaiicy-gradient algorithms and optimizing deterministie policies by continuation. Furthermore, as explained in Section 3.2 the contin- uation equation (8) consists in smoothing the return of this deterministic policy through the continuation distribution. Local optima tend to be removed when the variance of the contimuation is sufficiently large. Optimizing stochastic policies and regularizing the entropy, as in most state-of-the-ard policy-gradient algo- rithms, is therefore expected to avoid local extrema before converging towards policies with small variance. We thus provide a theoretical motivation for the performance reached by algorithms applying exploration as understood in direct policy optimization.
The relationships between optimization by continuation and policy gradient in Section 5.1 have been estab- lished relying Property 4.2 and Property 4.3 They assume continuations where the covariance matrix depends only on the current state and not on the whole observed history. In the general case, Property 1.4 allows one to extend these results by performing an analysis similar to Section 5.1 To be more specific, let us assume an affine Gaussian policy GGP' where the mean μg a function of the state and where the covariance Σg Σ is a function of the history and constant with respect 0. Under this assumption, if dA S de and Velu((s) is full rank, the return of the policy =GP" is equal a (unknown) continuation of the mcan function (4g (i.c., deterministic policy). Furthermore, optimizing the Gaussian policy by policy gradient while discounting the covariance continuation. In practice, this result suggests to optimize history-dependent policies by policy gradient to take advantage of the most general regularization of the objective function through implicit continuation. A similar observation was recently made by Mutti et al. (2022) who argued that history-dependdnt policies are required when more complex regularizations are involved. In parallel, Patil et al.) (2022) also discussed the potential advantage using history-depend parameterized policies for approximating correctly optimal policies using little representation power.
be interpreted as optimizing the deterministic policy μi by
Finally, last point has been left open in the previous discussions, namely the update of the covariance matrix of the mirror policies. The latter is defined through the covariance of the continuation. Therefore, the covariance must decrease through the optimization and must be chosen to avoid local optima. One direction to investigate order to select a variance that removes local extrema is to update the parameters of the policy by following combination of two directions: the functional gradient of the optimized policy's return with respect the policy mean and the functional gradient of another measure (to be defined) with respect to the policy variance. An example of heuristic measure for hnoss might be the entropy of the
11

---

Published in Transactions on Machine Learning Research (10/2023)
actions and/or states encountered in histories. This strategy obviously does not follow the classical approach when optimizing stochastic policies where the covariance is adapted by the policy-gradient algorithı to locally maximize the return and the exact procedure for updating the variance will require future studies. However the empirical inefficiency of this classical approach has already been highlighted in previous works that improved the performance of policy-gradient algorithms by exploring alternative learning objective Papini et al. 2020).
functions (ouuhoofft et al., 2018
6 Conclusion
In this work, we have studied the problem formulation, i.c., policy parameterization and reward-shaping strategy, when solving direct policy optimization problems. More particularly, we established connections between formulations of state-of-the-art policy-gradient algorithms and the optimization by continuation framework (Allgower & Georg, 1980). We have shown that algorithms optimizing stochastic policies and regularizing the entropy inherit the properties of optimization by continuation and are thus less subject to converging towards local optima. In addition, the role of the variance of the policies is reinterpreted in this framework: it is a parameter of the optimization procedure to adapt in order to avoid local extrema. Additionally, to inherit the properties of generic continuations, may be beneficial to consider variances that are functions of the history of states and actions observed at each time step.
Our study leaves several questions open. Firstly, hold in practice. Specifically, it is unclear how our findings can be generalized to non-alline policies and alternative to Gaussian policies. Nonetheless, our results can be extended in cases where we can obtain an analytic expression for the mirror policy outlined in Theorem [ While finding such an expression may be challenging in general, we can casily extend our conclusions to non-affine policies by considering the first- order approximation. Additionally, our study is focused on Gaussian policies, which are commonly used continuous state-action spaces. However, for discrete action spaces, natural choice of policy is Bernoulli distribution over the actions (or categorical distribution for more than one action). If the state space is also discrete, this distribution may be parameterize by table providing the success probability of the Bernoulli distribution for each state. In the case of Beta continuation distribution, mirror policy can be derived where actions follow a Beta-binomial distribution in each state, & result known in Bayesian inference as the Beta distribution is a conjugate distribution of the binomial distribution (Bishop & Nasrabadi] 2006). An analysis of this mirror policy would allow us to draw conclusions equivalent to those of the continuous case studied in this paper. Sccondly, the study focused on entropy regularization of the policy only. Recent works have underlined the benefits of other regularization strategies that enforce the spread of other distributions the state visiting frequency the marginal state probability (Hazan al. 2019: Guo et al. 2021 Mutti ct al. 2022). Future rescarch is also needed to better understand the effect of these regularizations on the optimization procedure.
results rely several assumptions that may not
Finally, we give a new interpretation for the variance of policies that suggests shall be updated to avoid local extrema rather than to maximize the return locally. A first strategy for updating the variance is proposed in Section 5.2 which opens the door further research and new algorithm development.
7 Acknowledgments
The authors would like to thank Csaba Szepesvári for the discussion on some mathematical aspects that allowed to increase the quality of this study. We also thank our colleagues Gaspard Lambrechts, Arnaud Delaunoy, Pascal Leroy, and Bardhyl Miftari for valuable comments this manuscript. Adrien Bolland gratefully acknowledges the financial support of a research fellowship of the F. F.R.S.-FNRS.
12

---

Published in Transactions on Machine Learning Research (10/2023)
References
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory, pp. 64 66. PMLR, 2020.
Zafarali Ahmed, Nicolas Le Roux, Mohammnad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International Conference on Machine Learning, pp. 151-160. PMLR, 2019.
Eugene L Allgower and Kurt Georg, Numerical continuation methods: introduction, volume 13. Springer Scries in Computational Mathematics, 1980.
Marcin Andrychowicz, Anton Raichuk, Piotr Staúczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthien Gcist, Olivier Pictquin, Marcin Michalski, et al. What matters in on-policy reinforcemenl learning? large-scale empirical study. arXiv preprinl arXie:20060556
2020.
Amrit Singh Bedi, Anjaly Parayil, Junyu Zhang, Mengdi Wang, and Alec Koppel. On the sample complexity and metastability of heavy-tailed policy search in continuous control. orXiv preprint 0rXiv:2106.0844 2021.
Amrit Singh Bedi, Souradip Chakraborty, Anjaly Parayil, Brian M Sadler, Pratap Tokekar, and Alec Koppel. On the hidden biases of policy mirror ascent in continuous action spaces. In International Conference Machine Learning, pp. 1716 1731. PMLR, 2022.
Yoshua Bengio. Learning deep architectures for ai. Foundations and Trends in Machine Learning, 2(1): 1 127, 2009.
Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXie preprint arXiv:1906.1 01 186, 2019.
Sujay Bhatt, Alec Koppel, and Vikram Krishnamurthy. Policy gradient using weak derivatives for rein- forcement learning. In Conference on Decision and Control (CDC), volume 58, pp. 5531-5537. IEEE, 2019.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learnång, volume 4. Springer, 2006.
Andrew Blake and Andrew Zisserman. Visual reconstruction. MIT press, 1987.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMP- 2T1T pp. 177-186. Springer, 2010.
Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Grégoire Delétang, Markus Kunesch, Shane Legg, and Pedro Ortega. Your policy regularizer secretly an adversary. arXie preprint arXiv:2203. 12592, 2022.
Lucian Busoniu, Robert Babuska, Bart De Schutter, and Damien Frnst. Reinforcement learning and dynamic programming using function approzimators. CRC press, 2017.
Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 70(4):2563 2578, 2022.
Herman Chernoff and Lincoln E Moses. Elementory decision theory. Courier Corporation, 2012.
Po-Wei Chou, Daniel Maturana, and Sebastian Scherer. Improving stochastic policy gradients continuous control with deep reinforcement learning using the beta distribution. In International Conference on Machine Learning, pp. 834-843. PMLR, 2017.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Picter Abboel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329- -338,. PMLR, 2016.
13

---

Published in Transactions on Machine Learning Research (10/2023)
Yasuhiro Fujita and Shin-ichi Maeda. Clipped action policy gradient. In International Conference Machine Learning, pp. 1597-1606. PMLR, 2018.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Alaa Saade, Shantanu Thakoor, Bilal Piot, Bernardo Avila Pires, Michal Valko, Thomas Mesnard, Tor Lattimore, and Rémi Munos. Geometric entropic exploration. arXiv preprint arXiv:2101.0 02055,,,, 2021.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta; Picter Abbecl, et al. Soft actor-critic algorithms and applications. arXiv preprint orXio:1812.05900 2019.
Elad Hazan, Kfir Yehuda Levy, and Shai convex problems. In International Conference on Machine Learning, pp. 1833 1841. PMLR, 2016.
On graduated optimization for stochastic non-
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy explo- ration. In International Conference on Machine Learning, pp. 2681-2691. PMLR, 2019.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAI Jonathan Ho, and Picter Abbeel. Evolved policy gradients. Advances in Newral Information Processing Systems, 31, 2018.
Hisham Husain, Kamil Ciosek, and Ryota Tomioka. Regularized policies are reward robust. In International Conference Artificial Intelligence and Slalislics, pp. 64-72. PMLR, 2021.
Riashat Islam, Zafarali Ahmed, and Doina Precup. Marginalized state distribution entropy regularization in policy optimization. arXiv preprint 1rXit:1912.05123 2019.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.088 02091,, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Inter- national Conference on Machine Learning, pp. 1928-1937. PMLR, 2016.
Hossein Mobahi and John W Fisher. On the link between gaussian homotopy continuation and convex envelopes. In International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, pD. 43 56. Springer, 2015.
Hossein Mobahi and John Fisher III. A theoretical analysis of optimization by gaussian continuation. In Conference on Artificial Intelligence, volume 29, AAAL, 2015.
Hossein Mobahi, C Lawrence Zitnick, and Yi Ma. Seeing through the blur. Conference on Computer Vision and Pattern Recogrition, pp. 1736 1743, IEEE, 2012.
Walter Murray and Kien-Ming Ng. An algorithm for nonlinear optimization problems with binary variables. Computational optimization and applications, 47(2):257722) 2010.
Mirco Mutti, Riccardo De Santi, and Marcello Restelli. The importance of non-markovianity in maximum state entropy exploration. arXiv preprint arXiv:2202. 03060, 2022.
Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans. Improving policy gradient by exploring under- appreciated rewards. arXiv preprint arXiv:1611 1X1.09911 2016.
Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527-566, 2017.
Mattco Papini, Andrea Battistello, and Marcello Restelli. Balancing learning speed and stability in policy gradient via adaptive exploration. In International conference on artificial intelligence and statistics, pp. PMLR, 2020.
14

---

Published in Transactions on Machine Learning Research (10/2023)
Harsh Nilesh Pathak and Randy Paffenroth. Parameter continuation methods for the optimization of deep neural networks. In International Conference on Machine Learning And Applications (ICMLA), volume 18, pp. 1637 1643. IEEE, 2019.
Gandharv Patil, Aditya. Mahajan, and Doina Precup. On learning history based policies for controlling markov decision processes. arXiv preprint arXiv:2211.0011, 2022.
Jan Peters and Stefan Schaal. Reinforcement learning by reward- weeihtdd regression for operational space control. In International conference on Machine learning, volume 24, pp. 745 750, 2007.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4):682 697, 2008.
Aravind Rajeswaran, Kendall Lowrey, Emanuel V Todorov, and Sham M Kakade. Towards generalization and simplicity in continuous control. Adoances in Newrul Information Processing Systems, 30, 2017.
Calyampudi Radhakrishna Rao. Linear statistical inference and its applications, volume 2. Wiley New York, 1973.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint Xiv: 703.7003..8000 2017.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arxi 1eXX::117700000 70070007.3. 06747, 2017.
Frank Sehnke, Christian Osendorfer, Thomas RückstieBß Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients. Neural Networks, 23(4):551-559, 2010.
Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive trust region policy optimization: Global con-
vergence and faster rates for regularized mdps. In Conference on Artificial Intelligence, volume 34, pp. 5668-5675, AAAI, 2020.
Weijia Shao, Christian Geißler, and Fikret Sivrikaya. Graduated optimization black box functions. arXiv preprint XrXio:1906. 01279, 2019.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Determin- istic policy gradient algorithms. In International Conference on Machine Learning, pp. 397-395. PMLR, 2014.
Herbert A Simon. A behavioral model of rational choice. The quarterly journal of economics, 69(1):99 118, 1955.
Minoru Siotani. Some applications of loewner's ordering symmetric matrices. Annals of the Institute of Statistical Mathematics, 19:245-259, 1967.
Joe Staines and David Barber. Variational optimization. arXiv preprint orXi::1212.4507, 2012.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Layne T Watson and Raphael T Haftka. Modern homotopy methods in optimization. Computer Methods in Applied Mechanics and Engineering, 74(3):289-3055 1989.
Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Episodic reinforcement learning by logistic reward-weighted regression. In International Conference on Artificial Neural Networks, pp. 407- 416. Springer, 2008.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229 256, 1992.
15

---

Published in Transactions on Machine Learning Research (10/2023)
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algo-
rithms. Connection Science, 3(3):241-20) 1991.
Huaqing Xiong, Tengyu Xu, Lin Zhao, Yingbin Liang, and Wei Zhang. Deterministic policy gradient:
Convergence analysis. In Conference Uncertainly in Artificial Intellignnce volume 28, 2022.
Jiaxin Zhang, Hoang Tran, Dan Lu, and Guannan Zhang. novel evolution strategy with directional gaussian smoothing for blackbox optimization. arXiv preprint arXiv:2002. 03001, 2020a.
Junzi Zhang, Jongho Kim, Brendan O'Donoghue, and Stephen Boyd. Sample efficient reinforcement learning with reinforce. In Conference on Artificial Intelligence, volume 35, pp. 10887 10895. AAAI, 2021.
Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods (almost) locally optimal policies. SIAM Journal on Control and Optimization, 58(6):3586-300) -612-361,, 2020b.
16

---

over the Published in Transactions on Machine Learning Research (10/2023)
A Theoretical Results on Mirror Policies
Theorem 1 For any original history-dependent policy na E & parameterized with the vector 0 E gd and for any continuuaiion distribution and covariance function A, there erists mirror history-dependen policy to E of the original policy He that writes as:
Th (a)h)) 0' (0.(
E
[Tte (a]h)]
(14)
Proof. The continuation S is defined in equation distribution such that:
S as the expectation of the function
[A(0)
E so ppl
Σ Etppser 1==0
(15)
8, (..A(1
at mn, (:1ha)
# 1 rppt rupl-lse.as (s)))s)
- p((0))
/
20 Σ E''p(ss,ai dsodondOo....
II no, (a. (10
(16)
For the sake of simplifying the notations, let (so. ap, $1, 01....) E H be a history and let R(h) be the discounted of rewards computed from this history. Let us reorder the terms of the integral and change the order of integration such that:
(2(0) / p(00) II no (o ax) R(h) dhddo....
(17)
/
II 1==
Tie (oe (o,,(o))t))((000.A(()))))
p(so)
II ((sc+1132, R(h) dhddo....
(18)
r=
)
no,
p(50) Il I=
[1s441190, at) R(h) dh
(19)
t=
In the inner integral over the parameters, cach term of the product depends solely on the parameter at single time step such that the integral of the product simplifies to the product of the integrals as follows:
F2(0) = /
(0e11,))((00.AAA(())) d0, (no(so) II p( av) t=
R(h) dh
(20)
/ II (o((aels))) (1o(60)
II Ie( R((h) dh. 1=0
(21)
By definition, the latter equation is equal to the return J(H) of the policy %o for any parameter vector 0. Therefore, T/e is mirror policy of the original policy under the process q and covariance matrix A.
Theorem 2.
Let q be a contivuation distribution and let be covariance function defined in Section 3.2 In addition, let no, 16 and vi be three parameterized history-degendent policies such that:
(a(aah) / no (( A(h)) d8'
(22)
(a(h) = (alh(( A(h)))())))))))))) d8"
(23)
Then, 1// mûrror policy of the original policy no and n/ a mirror policy of the original policy vip under continuation distribution and covariance function A. In addition, there exists a continuation for which ni is márror policy of the original policy no-
17

---

Published in Transactions on Machine Learning Research (10/2023)
Proof. First, v/ is mirror policy of the original policy no and v/ is mirror policy of the original policy % under continuation distribution and covariance function A, see Theorem 1 Then, let us substitute equation (22 in equation (33
f - (box(al)))(000000(((0) do"
V/l (a]h)
(24)
/ ( N1 ( (alh)a(00000 .A(1))))'",A))))))))))))))))))") do') a(00)0, do"
(25)
f nor (alh) (f (fa1110 ((h))q(0010AA((H) (00 a0") do'
(26)
We thus have that:
1// (a(h) = / no no (alh))oo(0 de'
(27)
(0 (h) = / ((h)) d0"
(28)
The distribution pa is contimuation distribution with spread depending the history h through the covariance function A. By Theorem | v/ is mirror policy of the original policy ne-
Let the original policy ny E II be a Markov policy and let the covariance function depend solely on the last state in the history. Then, there exists a mirror Markoe policy v/p E
Property 4.1
Proof. By hypotheses, the covariance matrix only depends on the last state s, of the history hr, therefore:
((,,)),A(s)).
(29)
In addition, the original policy is Markov policy, therefore
no(as(he) =
(30)
The closed form of the mirror policy, provided by equation 14 can thus be simplified as:
(hoa) / no. (allh))))9 A(h,)) dA.
(31)
/ no. (all))))00000 A(x))) d0,-
(32)
The previous equation is independent of h knowing St, there thus exists Markov mirror policy T/ II respecting Theorem such that:
(33)
Let the original policy GP E II be a Gaussian policy as defined in equation 3 with offine function approzimators. Let the covariance function depend solely on the last slate in the history and let the distribution q be Gaussian distribuution. Then, there exists mirror Markov policy ni such that for all states S, it converges towards Gaussion policy in the limit the affine coefficients of the covoriance matrix Es(s) approaches zero ((V) 0):
Property 4.2
r'((als) + N(aluu((s) Eb)()))
(34)
where Ex(s) Cs(s) Eg(s) and Cols) VOpo(s)T A(s) (s)o
18

---

Published in Transactions on Machine Learning Research (10/2023)
Proof.
First, the existence of Markov mirror policy results from Property 4.1 and is provided by equation
33
=b(0)(1)) = f
d0,
(35)
In addition, no, and q are Gaussian distributions by hypotheses:
Te, (ce(ss) ((aelpo, (st); Eo, (8))
(36)
(00.00, N(0.)0,A(ss))
(37)
where Me, (s)) and (s)) are affine functions of 0.. Therefore, these functions be written as follows:
No, (s:) (Vo,Ho, (s:)) ) 0, (s()) Eg, (8+) = (Vo, Ex, (8x) 0, + (7))
(38)
(39)
For any state St, in the limit as affine coefficients of the covariance approaches zero, the covariance is such that:
lim V No, [a:]1=+00
Eo, (st) Es:))
(40)
In this limit, equation (35 consists in marginalizing a conditional linear Gaussian transition model with Gaussian prior and is such that (Bishop & Nasrabadi 2006
lim =o(ae(8e) N (ail (VoloVo(())))) (/(),, (Vopo(se))) M(x)) (VoHo((:)) E'(s)) (o))(s)
(41)
=N (alaalo(ss) Msse) (Vouo(si)) + Eo(st))
(42)
Let the original policy Pa E M be an affine deterministic policy. Let the covariance function depend solely on the last slate in the hislory and let the distribulion be a Gaussian distribution. Then, the Markov policy GPP E Il mirror policy:
Property 4.3
T TP = -N(alpo(s)
(43)
where Eg(() = Vo M(s) Vopo(s)
Proof. The statereent results from the particularization of Property 4.2 to the case deterministic policies.
Let Th GP II be an affine Gaussian policy with constant covariance matrix for any state Eg(st) C. In that case, have by Property 4.2 that nig E IT is mirror policy follows:
[s(() N (arlua(ss). (VeHo(se))) (V)))a())).eeepss)))) A(s:) (Veuo(s:)) Eo(si)
(44)
= (a (s) (Vopo(s:)) A(s:) (VoHo(s:)) + +C)
(45)
Taking the limit of T as the constant covariance matrix C approaches zero, we get that the original policy
from Property 4.2 converges to the one Property 4.3 namely the deterministic policy Pa- This implies
that the policy GP' E πI provided by the limit of the mirror policy from Property 4.2 see equation (45 is
mirror policy of the original policy pe from Property 1.3
GP' GP' (a, (c)]s))= lim (a((())) - (acluo(si), (Vopo(se))) A(se) (Vono(si))) T
(46)
C-0
19

---

Published in Transactions on Machine Learning Research (10/2023)
Property 4.4 Let the original policy pe M be affine deterministic policy. Let the distribution q be Gaussian distribution. Then, the policy 1h E & is mirror policy:
- (x(ao))))g()
(47)
where (h) = (o A(h) (oll(ss)
Proof. The policy pe, is an affine function of the parameter vector dr and can thus be written as follows:
(st) = Mo, (st)) 0, p(()))
(48)
In addition, the samples drawn from the process q are distributed according to a Gaussian distribution:
(00,)0,A(h)) ((0))),AA(h)))
(49)
The closed form of the density of the mirror policy, provided by equation 14 is thus simplified as:
(aelh)))) / no, (o..oo/h.)g(e.ee.A((b)))) d8:
(50)
/ no. ((000 dB,
(51)
where no, is the policy where each action respecting equation 48 has a probability one. The policy is degenerated Gaussian distribution (Rao 1973), it provides a dirac measure to each state, and its (generalized)
density function may be approached as follows:
nlo (aelhe)
lim (arl (Ve,Ho, (st)) 0g + v'(s))
(52)
|
By substitution, we therefore get that the mirror policy Th writes as follow:
[a(aalhe) = f
d0,
(53)
= ff lim N (axl (Vo,na, 02 + / E (Ob), d0, |
(54)
The product of the Gaussian prior parameters and the lincar Gaussian transition model of the actions
provides joint Gaussian distribution of actions and parameters (Bishop & Nasrabadi, 2006), which is degenerated but has a density for the (marginal) Gaussian distribution of actions (Rao 1973). The density
of the mirror policy v/ can thus be computed taking the limit of the marginalization:
(arl))) = lim / (acl |2|l=0
(s.). N ((N d0,
(55)
lim N |>]|=>0
(st), (Vopo(s))) A(he) (Vono(se)) + +E)
(56)
lim .V (allu((s) A(h))))))))))))))))))(()))))))() (VoHo(se)) + +2)
(57)
|2||=+0
=N (allpo(s)) (Vopo(s:)) A(ha) (VoHo(s:)))
(58)
We note that this result can be obtained without working on degenerated Gaussian distributions. The policy is an affine function of the parameters, which follow Gaussian distribution, the marginal distribution of actions is thus also a Gaussian distribution of the form of equation (58) This distribution is furthermore the one of mirror policy, see Theorem
20

---

Published in Transactions on Machine Learning Research (10/2023)
B Description of the Car Environment
In this section, we formalize the reinforcement learning environment that models the movement of a car a valley with two floors separated by a peak, as depicted in Figure 2 The car always starts at the topmost floor and receives rewards proportionally to its depth the valley. An optimal agent drives the car from the initial position the lowest floor in the valley by passing the peak. In the following, we describe each element composing the environment.
2-
Valley h(x)
0-
-1-
Ziditool
Searget
22
Position if
Figure 2: Valley in which the car moves.
State Space. The state E R² of the environment is composed of two scalar values, namely the position 24 E R the pointmass representing the car and its tangent speed 1/2 E R.
Action Space. At each time step, the agent controls the force applied on the car through the actions de R it exccutes.
The car always starts at the topmost floor Tinitial = - in the valley at rest. The initial state distribution thus provides a probability one to the state
Initial Position.
zgg Tinitia!
(59)
0g 0.
(60)
Transition Distribution. The contimuous motion of the car in the valley is derived for Newton's formula.
The valley's analytical description is provided by the function h, the car's mass is denoted by 0.5,
gravitational acceleration by g = 9.81, and damping factor by = 0.65. The position x and speed v of the car follow the subsequent continuous-time dynamics as a function of the force a:
i w
(61)
gh'(x) (l()))"(()
0 m(1 (2))² 1 (x)x²²
ev?
(62)
1+ 1+ 1'(x)2
The position and force are furthermore bounded intervals part of the dynamics such that
[Zmm FM 4,
(63)
E [0m, am] [-10, 10]
(64)
Clamped force values are therefore used in equation
62 Similarly, the position is clamped in equation
61
In discrete time, the state $t+1 computed through Euler integration of the continuous time dynamics, con- sidcring an initial position given by the current state 8,. The force remains constant during discrctization time A 0.1 and equal the action a,, with an additive noise drawn from AV--0, 1) and clamped before integration.
21

---

Published in Transactions on Machine Learning Research (10/2023)
Reward Function. The rewards correspond to the depth of the valley at the current position. The reward function thus solely depends on the position
p(s: -h(x))
(65)
Discount Factor. mumerical computation.
The discount factor equals 7 0.99 and the horizon is curtailed to T 100 in each
22


Reviews:
[{"review_id": "MP02ymc79U", "reviewer": "Reviewer_bwWD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- Interesting angle to tackle regularization in policy gradient optimization.\n- The work is well presented: good motivation, clear introduction, sufficient background, sound mathematical exposition.\n\nWeaknesses:\n- The work is limited to a specific model of the policies function class: the affine Gaussian policies.\n- I found that the submission was lacking empirical illustration of the theoretical findings: in practice what are the similarities with the main algorithms? In which kind of MDPs do we observe differences in policy optimization?\n\nAs I said before, the submission is remarkably typo-proof, and my only minor remark will be on the choice of \\eta to design policies while stochastic policies are denoted by \\pi, and deterministic policies by \\mu. I found that it unnecessarily overloaded the notations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission offers a novel theoretical perspective for policy-gradient algorithms (PGA). Initially, it casts PGA within the 'optimization by continuation' framework. This approach is tailored for optimizing nonconvex functions by progressively optimizing a series of surrogate objectives known as continuations. Subsequently, it demonstrates that the process of optimizing affine Gaussian policies combined with entropy regularization can be seen as an indirect optimization of deterministic policies through continuation. Drawing from these theoretical findings, it advances that exploration in PGA amounts to determining a continuation of the current policy's return. Finally, it argues that the policy variance should be based on history rather than being put in trade-off with the policy return maximization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I find the paper good enough as it is, but I am a bit frustrated by the lack of empirical studies to support it. So my only question would be, why not have some experiments that could both serve as didactic content and as validation of the theory?", "broader_impact_concerns": "I do not have concern on any potential negative impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yk2MViVe5w", "reviewer": "Reviewer_gGC6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n## Clarity\n\nThe paper is clear and well presented. \n\n## Insight\n\nLinks between RL methods and optimization are not new, but to my knowledge this specific link was not shown before in the literature. In general, I think that linking works form two different communities is often a valuable contribution, as it can provide new understanding on these methods.\n\nIn particular, this works proposes directions to understand better entropy regularization in RL. Entropy is many modern RL algorithms, but it is not always clear what its exact function is. This is a studied question in the RL literature, and while this work does not give a definitive answer to it, I think it can be interesting to many researchers in that area. For example, it fits nicely in the direction open by [1], arguing that entropy is useful because it smooth the optimization landscape of the policy.\n\n\n# Weakness\n\nI think the work is lacking a bit of context on why these results are interested. In particular, two questions are not addressed (that are not independent):\n\n## How and why optimization by continuation is used in classic optimization ?\n\nWhen introducing OC, it would be valuable to understand why this method would be used in non-convex optimization, for example if it has a inherent theoretical or practical advantage, and maybe give one example where it would be effective. This would make the contribution clearer, because right now it is hard to see the point of this method by itself.\n\n## What are the known convergence results on optimization by continuation, and how would they translate to RL ?\n\nI am not a specialist of optimization, but I think it would be interesting to state what are the known theoretical properties of OC. In particular, this could be discussed in relation to section 5.1. If some policy gradient algorithms can be analyzed through OC, it would be valuable to know if some analysis tools form optimization can be applied to this analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This works shows a link between direct policy optimization methods (in reinforcement learning) and optimization by continuation (in non-convex optimization).\n\nOptimization by continuation solves a non-convex optimization problem by solving a sequence of \"relaxed\" optimization problems, on fgunctions called continuation of the objective functions. This works shows that the continuation of the return of the policy is the return of another policy (the mirrror policy), and then that solving the return by continuation is the same as solving a policy optimization problem for the mirror policy.\n\nThen, the authors link more specific policy gradient methods to optimization by continuation. In particular, they show that in the case of a affine-parametrized Guassian policy with decreasing variance, policy gradient is equivalent to optimization by continuation on the deterministic policy (that just outputs the mean of the Gaussian).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Poposed adjustments are the answers to the Weakness section. I would suggest \n - extending the background on OC, \n-   discussing the theoretical and practical properties of this method, and how they can influence RL algorithm analysis adn design.", "broader_impact_concerns": "No broader impact concerns (theoretical work).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3PaMSCaouS", "reviewer": "Reviewer_EhvW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The final decision is left for after discussions. The scope of the paper is appropriate for TMLR. The connection to optimization by continuation is intuitive and is explained in an example environment. The proofs for two of the main results are not presented clearly (comments 1 and 2 in Correctness Section below) and I can only assess the correctness of the results if the authors clarify these proofs in a revision.\n\nCorrectness:\n1. There are multiple steps to obtain Eq 23 from Eq 22 in the appendix which should be laid out. In particular I do not understand how the order of the product over t and the expectation over theta are swapped to obtain Eq 23.\n\n2. The discussion for Gaussian policies in Section 5.1 should be stated formally, ideally as propositions. Why does the condition on dimensions and the rank of the gradient satisfy the requirement in this discussion? What does it precisely mean that the covariance is discounted and why does it guarantee full rank gradient?\n\n3. Section 2.2 mentions that an MDP always has an optimal deterministic policy. This is true for *finite* MDPs. A continuous-action MDP, the problem this paper focuses on, may not have an optimal policy in general (stochastic or deterministic). I recommend removing this statement.\n\n4. The same paragraph cites Silver et al (2014) to claim deterministic policies tend to get stuck in local optima. Silver et al in fact mention inadequate exploration as the problem with deterministic policies.\n\n5. The conclusion mentions that a categorical distribution is a natural policy parametrization for discrete state space. The action space, and not the state space, determines policy parametrization. A continuous-action discrete-state problem would still naturally use Gaussian or deterministic (rather than categorical) policies.\n\n6. Does the inner loop in the optimization by continuation algorithm continue until convergence or for one or a few steps? If it is until convergence, this difference between this algorithm and policy gradient algorithms should be clarified in the text.\n\nOrganization:\n1. The results that follow from Theorem 1 are presented as \"Property 1\", \"Property 2\", etc. except Theorem 2 which is called a theorem, described informally in the main paper, and proven in a dedicated section in the appendix before the proof for Theorem 1. I think presenting the result like the others would make the paper clearer. The paper is otherwise well organized.\n\nNotation and typos:\n1. Even if \"p\" is the conventional notation for continuation it's still better to use a different letter here since p is already used for start distribution and transition kernel.\n2. Extra phrase \"algorithm algorithmic\" before Algorithm 1.\n3. In all references to equations \"equation\" is repeated.\n4. (Below property 1) Independently *of* the history.\n5. From property 2 onwards the notation for covariance matrix changes to have a subscript of theta.\n6. (Section 5.1, Gaussian policies) \"does optimizing a Gaussian policy is\" -> \"optimizing a Gaussian policy is\". \"trough\" -> \"through\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission provides an interpretation of optimizing a Gaussian policy instead of a deterministic policy or adding entropy regularization as optimization by continuation (aka graduated optimization), a global search method that smoothens the objective function at monotonically decreasing levels to avoid bad local optima. An important difference between the analysis in this paper and common policy gradient algorithms is that here the covariance matrix of the Gaussian policy is not optimized but should be annealed independently to match the framework of optimization by continuation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The comments in the Correctness section are all critical.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are happy with the improvements to the work. They unanimously voted to accept.

I know that this is an Accept as is, but I would request one very minor thing. It would be great if you could consider breaking up the very large paragraphs that were added. I can see obvious subparagraphs within those large paragraphs, and they are currently hard to read.
Decision: Accept (poster)