Convergence of an actor-critic gradient flow for entropy regularised MDPs in general spaces
Authors: Denis Zorba
David Siska
Lukasz Szpruch
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Gradient Flow
Markov Decision Process
Entropy Regularization
Non-convex optimization
Mirror descent method
Fisher–Rao gradient flow
Global convergence
Function approximation
Actor Critic
Primary area: reinforcement learning

Abstract: We prove the stability and global convergence of a coupled actor-critic gradient flow for infinite-horizon and entropy-regularised Markov decision processes (MDPs) in continuous state and action space with linear function approximation under Q-function realisability.
We consider a version of the actor critic gradient flow where the critic is updated using temporal difference (TD) learning while the policy is updated using a policy mirror descent method on a separate timescale.
For general action spaces, the relative entropy regularizer is unbounded and thus it is not clear a priori that the actor-critc flow does not suffer from finite-time blow-up.
Therefore we first demonstrate stability which in turn enables us obtain a convergence rate of the actor critic flow to the optimal regularised value function.
The arguments presented show that timescale separation is crucial for stability and convergence in this setting.

Published as a conference paper at ICLR 2026
CONVERGENCE OF AN ACTOR-CRITIC GRADIENT FLOW FOR ENTROPY REGULARISED MDPs IN GEN- ERAL SPACES
Denis Zorba, David Šiška Lukasz Szpruch School of Mathematics University of Edinburgh
Edinburgh, United Kingdom (e.zorba, D.Siska, I.Sxpruch) @ed.ac.uk
ABSTRACT
We prove the stability and global convergence of a coupled actor-critic gradi- ent flow for infinite-horizon and entropy-regularised Markov decision processes (MDPs) in continuous state and action space with linear function approximation under Q-function realisability. We consider version of the actor critic gradient flow where the critic is updated using temporal difference (TD) learning while the policy is updated using policy mirror descent method For general action spaces, the relative entropy regularizer is unbounded and thus it is not clear priori that the actor-crite flow does not suffer from finite-time blow- up. Therefore we first demonstrate stability which in turn enables us obtain a convergence rate of the actor critic flow to the optimal regularised value function. The arguments presented show that timescale separation is crucial for stability and convergence in this setting.
separate timescale.
INTRODUCTION
In reinforcement learning (RL) an agent aims to learn an optimal policy that maximizes the expected cumulative reward through repeated interactions with its environment. Such methods typically in- volve two key components: policy evaluation and policy improvement. During policy evaluation, the advantage function corresponding to policy, or its function approximation, is updated using state, action and reward data generated under this policy. Policy improvement then uses this ap- proximate advantage function update the policy, most commonly through some policy gradient method. Algorithms that explicitly combine these two components are known as actor-critic (AC) methods Konda & Tsitsiklis (1999), where the actor corresponds to policy improvement and the critic to policy evaluation.
There many policy gradient methods choose from. In the last decade trust region policy opti- mization (TRPO) meethods[Schulman et al. (2015) and methods inspired by these like PPO Schulman al. (017) have become increasingly well-established due to their impressive empirical perfor- mance. Largely, this is because they alleviate the difficulty in choosing appropriate step sizes for the policy gradient updates: for vanilla policy gradient even a small change the parameter may result in large change in the policy, leading to instability, but TRPO prevents this by explicitly ensuring the KL divergence between successive updates is smaller than some tolerance. Mirror descent replaces the TRPO's hard constraint with penalty leading to a first order method which is also ameanable to analysis. Indeed, at least for direct parametrization. it is known to converge with lib-linear and even linear rate for entropy regularised problems (depending exact assumptions) Ju & Lan (2024): Lan 22033: Kerimkulov ct al. (2025a).
Due to the favourable analytical properties of mirror descent, in this paper consider version of the actor critic gradient flow where the policy is updated using policy mirror descent method while the critic is updated using temporal difference (TD) separate timescale.
MDPs are widely used in practice since the entropic regularizer leads to a num- ber of desirable properties: it has natural interpretation something that drives exploration, it

---

Published as a conference paper at ICLR 2026
ensures that there is unique optimal policy and it can accelerate convergence of policy gradient methods Mei et al. (2020) However, analysing the stability and convergence of actor-critic meth- ods in this cetropy-regularised setting with general state and action spaces remains highly non-trivial due to lack of priori bounds on the value functions.
To address the actor critic methods for entropy regularised MDPs in general action spaces, a careful treatment of tools from two timescale analysis, convex analysis over both Euclidean spaces and measure spaces must be deployed.
In this paper, we address precisely this challenge. We study the stability and convergence of a widely used actor-critic algorithm in which the critic is updated using Temporal Difference (TD) learning Sutton (1988), and the policy is updated through Policy Mirror Descent Ju & Lan (2024). Our analysis employs two-timescale update scheme Borkar & Konda (197),, where both the actor and critic are updated at each iteration with the critic updated on a faster timescale.
1.1 RELATED WORKS
We focus on the subset of RL literature that address the convergence of coupled actor-critic al- gorithms. In the unregularised setting, actor-critic methods have been studied extensively. The first convergence results in the two-timescale regime established asymptotic convergence in the continuous- time limit of coupled updates (Borkar & Konda (1997): Konda Tsitsiklis (1999). Most modern research employs linear function approximation for the critic, where linear conver- gence rates have been obtained under various assumptions on the step-sizes of the actor and critic Barakat al. 2022); Zhang et al. (2020); Hong et al. (2023)).
Closely related to our work is Zhang et al. (202I), which considers the same two-timescale actor- critic scheme in the continuous-time limit for unregularised MDPs, with overparameterized neu- ral network used for the critic. However, convergence to the optimal policy was not established, and restarting mechanism
required to ensure the stability of the dynamics.
In the entropy regularsee setting, Cayei et al. (202aa b) address the convergence of natural actor critic algorithm. However, the convergence and stability of these results rely the finite cardinality of the action space presence of entropy regularisation.
1.2 OUR CONTRIBUTION
Under linear Qr-realisability assumption, we address the following question:
"Is the actor-critic gradient flow for entropy-regularised MDPs in general action spaces stable and convergent, and if so, at what rate?"
There two main technical challenges one has to overcome when working with entropy- regularised MDPs in general action spaces.
Even in mirror descent with exact advantage, the rate of convergence depends on constant term Is (-(==((-ss) (oolsss(m[[-(())))) (=))s) (ds). See (023); Kerimkulov et al. (2025a). In general action spaces, without entropy regularisation almost impossible to choose my which would make this term finite, see Remark 2.1 Thus we need to include the regularisation in
the analysis.
Moreover, ensuring that the relative entropy does not blow up is difficult in general action spaces. In the finite action space setting, for any measure y E P(A) such that p(a;)) > 0 S it holds that (TT((((-1s)])
and for all divergence has no upper bound (can be +oo) even if has full support. Under mild assump- tions we show that the KL divergence does not blow up in finite time, see Corollary 5.1
S log |4]. In general action spaces the KT.
Our main contributions are as follows:
We study a common variant of actor-critic where the critic is updated using temporal dif- ference (TD) learning and the policy is updated using mirror descent. Similarly to Konda & Tsitsiklis (1999); Zhang et al. (202T), we analyse the coupled updates the continuous- time limit, resulting in dynamical system where the critic flow is captured by semi-
2

---

Published as a conference paper at ICLR 2026
gradient flow and the actor flow corresponds to an approximate Fisher-Reee gradient flow over the space probability kernels.
By combining convex analysis over the space of probability measures with classical Eu- clidean convex analysis, we develop a Lyapunov-basee stability framework that captures the interplay between entropy regularisation and timescale separation, and establish stabil- ity of the resulting dynamics.
We prove convergence of the actor-critic dynamics for entropy-regularised MDPs with in-
finite action spaces.
1.3 NOTATION
Let (E, d) denote a Polish space (i.e., complete separable metric space). We always equip Polish space with its Borel sigma-field B(E). Denote by (E) the space of bounded measurable functions f E + I endowed with the supremum norm [[B,(E) = supzeg e [x)). Denote by M(E) the Banach space of finite signed measures j on E endowed with the total variation norm (p)M(() |x((E), where [n] is the total variation measure. Recall that if μ ʃ dp, where (E') is nonnegative measure and f L' p), then [|MMEE P(E) c MME) the set of probability measures on E. Let ò₂ E P(E) denote the Dirac measure with mass at x E E. Moreover, we denote the Euclidean norm on IE- by |- with inner product (..). Given some A, B E NRNN by Amin(A) we denote the minimum eigenvalue of A and write - and only if A B is positive semidefinite. Finally, we denote by |Alop the operator norm Mar
(' Denote by
of A induced by the Euclidean norm, A op == 20 [n]
1.4 ENTROPY REGULARISED MARKOV DECISION PROCESSES
Consider an infinite horizon Markov Decision Process (S, A. P. 7), where the state space S and action space A are Polish, P E P(S x A) is the state transition probability kernel, is a bounded cost function and y E (0, is a discount factor. Let μ E P((A) denote refer- ence probability measure and r > denote regularisation parameter. Let p E PPS) an arbitrary initial state distribution. To ease notation, for each E P(A[S) we define the kernels P= (ds's(s) === [a (dds'ss, T (dals) and (dd'' (uu|s, a) == P(ds'|s [s. ())(dda)ss) s'). Due to (Bert- construct a probability measure Pp. expectation E. and stochastic processes (Sn) (an (eN,, with the conditional transition probabilities corre- sponding to those given by p" and respectively and with so p. Let BT == B3, For x E P(AS) define the regularised value function as
sekas & Shreve 1996 Proposition 7. 7.28)
15 E
Σ 7"h (c(Sn)ax) + (L(T(11s))))) {RU(
where KL (TL(T(-1))) (-((ss))) is the Kullback-Leible (KL) divergence of (-|ss) with respect to define as ( (-ss) == Sa In da infinity otherwise.
di (dls) if (-|ss) is absolutely continuous with respect to μ, and
For given initial distribution p P(S), the optimal value function defined as
min V (p), with V7 (p) = / V (s)p(ss)
(P) (EP()
and we refer to n' E P(ASS) as the optimal policy if (+) V= (p). The Bellman Principle for entropy regularised MDPs, see Theorem A. suggests that without loss of generality, it is sufficient to consider policies from the class given by Definition below...
Definition 1.1 (Admissible Policies). Let Hy denote the class of policies for which there exists I E BB x A) with T (dals) JA exp(f(s,a))uuda)) u(da).
exp(J(s,a)) a))
3

---

Published as a conference paper at ICLR 2026
For each E II the value function V is the unique bounded solution of the on-policy Bellman equation
(s) / (o + rIn du (a, (a. x (daass)
du
see e.g, (Kerimkulov al. 2025a Lemma B.2).
For each π E 1ly. we define the state-action value function 07 E BB(S x A) by
( a) = c(s,a) + (ddss)((
(2)
We see that Q :5 - is a fixed point of T B&(S x A) + B&(S x A), defined as
T ((s a) + J Sxx a(dd
da' d a) + m> (I(
(3)
As one can show this operator is a contraction, we see that O fact the unique fixed point.
MIRROR-DESCENT AND THE FISHER-RAO GRADIENT FLOW
Let the soft advantage function be defined as
da (s, V=).s.
(5 a) (s, + In
du
Then for some À > 0 and 11 the Policy Mirror Descent update rule reads
A7" (s, (m)((a))))) x" ="(dals)) 1 [T (1+))-
n+ (-1s) arg min (AA
Due to (Dupuis & Ellis 1997
Lemma 1.4.3) we know that this pointwise minimum is achieved by
dx=+1 (a,
exp (-AA3" (s, a)) Ja exp (-AA;" (s, a))
d
TTh (dals)
From 1 we note that for any π E P(AAS), it holds that SA (5 (s) = 0. Hence taking the
logarithm of we have
d+ (8,
dx
(s, = -AA; (s, a) e (aa)"
(e.a) (dalss)
du
dp
Interpolating in the time variable and letting À we expect retrieve the Fisher gradient flow for the policies
a In d dp
17 (8, (s,0)==(A(((000)))
/ AT ())a))(ddals)) A (8, a).
(s,a)=
(5)
Note that the soft advantage formally corresponds to the functional derivative of the value function with respect to the policy x" and thus (5) can be seen gradient flow of the value function over the space of kernels S)A(S) (see Kerimkulov et al. (2025a) for a detailed description of the functional derivative).
Remark 2.1. In the case where the advantage function fully accessible for all 2 0, Kerimkulov et al. (2025a)/Theome 2.8] shows that the entropy regularisation in the value function induces an exponential convergence to the optimal policy.
More specifically, their result shows that for all 2 0 we have
0 0 V (p) V (p) (1 (1-7)(rr-1)) ( (L((((Hs)) (da)
T
If the action space has finite cardinality and nn is chosen to be uniform we see that (T (1)))(7((1ss) log lAl for all E S, where |A] represents the cardinality of the action
4

---

Published as a conference paper at ICLR 2026
space. One then let T - 0 in the above estimate formally obtain convergence rate of order / for the unregularised problem.
In the setting of general action spaces [-I)(((\s][ is finite only if the density dan exists. However, by the dynamic programming principle for the unregularised problem (Hernández-Lenen mixture of Dirac distributions. Therefore, KT.(T" (-1s)[TO(( (-1s)) will be finite only if mo is also a mixture of Dirac distributions with support which contains the support of (.1s) for all E S. It is not realistic to assume that one can guess the initial policy To which will have the above property. However, in the
& Lasserre 7996 Theorem 4.2.3) shows that the optimal policies will have support
entropy regularised case, Theorem A.I tells us (-|s) has fill support on A and so one simply has to choose ro(- to have full support on the action space A for all S.
3 ACTOR CRITIC METHODS
Given some feature mapping ø S x A + RN,, parametrise the state-action value function as Q(s, a: = (A, Φ(s, a)). Moreover, we let the approximate soft advantage function defined as
A(s, a: 0) Q(,, lln dm (s, a) S (((s a; + Tln dm (s. [(dals). du
du
The Mean Squared Bellman Error (MSBE) is defined as
MSBE((, / (Q(s, TT) ds)
SxA
where for some fixed [ E PSS A), dī E P(S A) is the state-action occupancy measure defined in (O).. Given that E P(S A) has full support, by (3) holds that MSBE((, F) 0 if and only if ((s, a; 0) = Q7 a) for all s S and E A. Hence one approach to implementing the policy mirror descent updates is to calculate the optimal parameters for Qss, a; 0) by minimising the MSBE at each policy mirror descent iteration and then update the policy using variable steps {Nn Jnnn20 This reads
g = arg min MSBE(O, x"), BERN
n+
exp (- Ann 1n+1)) dn (a, 8) Ja (- a; (m+1)) xr" "" (dals)
(6)
To avoid fully solving the arg min in (6) for each policy update, semi-gradient descent on different timescale to the policy update. Let 1tnnoo be the step-sizes of the critic at iteration n 2 0. Let the semi-gradient RNN x P(A(S) + IRN of the MSBE with respect be
update the critic using
9(8, m) = -
|SXA (Q(s, a;) ()) ds).
The full arg min update in (6) is then replaced by
g=+1 = 0" - 10ng(0 T"),
where timescale separation Tho = An > ensures that the critic is updated on a much faster timescale than the policy to improve the local estimation the policy updates.
With general action spaces, which allow the KL term may be unbounded, may need go further and choose a scheme which does several (and possibly increasing) number of updates of the critic before doing actor update.
In this paper we focus on continuous time idealisation of the above which is presented in the next section.

---

Published as a conference paper at ICLR 2026
4 DYNAMICS
We study the stability and convergence the two-timescale actor-critic Mirror Descent scheme in the continuous-time limit. Let [0, oo) + (o) be non-decreasing function representing the timescale separation, then for some 8g E INN. To E II and 3 E P(S x A), we have the following coupled dynamics
d0: - (T), Oo 00 ERR.
(8)
0 = - A(s (d)ls) t 120, 0,
7 E Ellys
(9)
where 9 RN x P(A[S) is the semi-gradient of the MSBE defined in We refer to approximate Fisher-Rao Gradient flow.
as the
We perform our analysis under the following assumptions.
Assumption 4.1 (QT-realisability) For all 1ly there exists Oz IR such that (s, (0z, 6s, a)) for all (s, a) E S A.
simple example of when this holds the tabular case, where one can choose to be a one-hot encoding of the state-action space. Moreover, all linear MDPs are Q"-realisable. In linear MDP there exists exists & SS + RN, w E IRN and sequence {w with tb, MM()) such that for all (8, a) E S x A.
N
c(s, = (w, ((s,a)),
P(ds' ss,)) = Σ (s,,a)
In this case holds that (0x): = W/ Ig V (s') Assumption 4.1 can be seen as a con- vention to omit function approximation errors in the final convergence results. This assumption, or the presence of approximation errors in convergence results, are widely present in the actor-critic
literature (Cayci et al. (2024a). Zhang al. (2020). Zanette al. (2021. Hong al. (2023) Quu
et al. (0211).
More recently, Lin al. (2025) derives some weaker ordering conditions in the bandit case (empty state space) which guarantee the convergence of soft-max policy gradient in the tabular setting be- yond realisability. However as of now is unclear how this applies MDPs and also fundamentally depends on the finite cardinality of the action space.
Since for all E II, we know that Q E Bo(S A) we also have QF E L'(S A: 3), which is Hilbert space. By (Brezis) 2011 Theorem 5.11), Assumption 4. holds in the limit N - oo when Φ); are the basis functions of T.2SS x A; 3). However, analysis in such a Hilbert space becomes more involved and intricate and is the result of ongoing work. Combining this approach with careful truncation of the basis functions has demonstrated empirical success in Ma et al. (2024): 2023).
Ren et al.
Assumption 4.2. For all (s,a) a) E S x A it holds that (Ø(s, a)l 1
Assumption 4.2 is purely for convention and without loss of generality in the fiite-ddimensional case.
Assumption 4.3. Let B E P(S x A) be fixed. Then
Ag Amiu ( o(s a)o(s, T) B(ds da) 0.
Note that unlike the analogous assumptions imposed in Hong et al. (023), Assumption 4.3 is in- dependent of the policy. This property allows to remove any dependence on the continuity of eigenvalues.
Definition 4.1. For all TTT and Ç E P(S A), the squared loss with respect to Ç is defined as 1 (0, a)) - 2 SxA
1(0,T;
where Q is defined in

---

Published as a conference paper at ICLR 2026
A straightforward calculation given in Lemma B.3 shows that due to Lemma A.1 and Assumption 4.3 for any x E I it holds that L(,, T; dgg) is (1 7) D-g-stoongly convex.
The following result then connects the geometry of the semi-gradient of the MSBE and the gradient of L(:; т: (9), which can be seen an extension of Lemma 3 of BBhandari et al. 22021 to the current entropy regularised setting.
Lemma 4.1. Let Assumption 4.1 hold. Then for all 0 E R. and π E 11 it holds that
- (g(0, 0x) - - (V (B, 0x)
with
VoL(0, 1 (10,0(s, (s, 3) ds).
9xA
See Appendix B.1 for a proof.
5 STABILITY
In this section we analyse the stability of the coupled actor-critic flow. Let (02, TT) be given by the system (8) (9. Under mild assumptions, Corollary 5.1 shows that for all E S, KL( (a) does not blow up in finite time in the sense that there in no T > 0 and no 8 E S such that li (=x(===-ls))) the actor-critic dynamics.
+0. Existence of such time T > would result in singularity in
Throughout this section, to ease notation let
: Aa(1 7) V7), K, : sup (L)
sES
with A3 > 0 the constant from Assumption 4.3
Using Lemma [. Lemma [5. then establishes the effect of the coupling and timescale separation in the actor-critic flow and its effect on the stability of the critic parameters.
Lemma 5.1. Let Assumptions 4.2 and 4.3
hold. Then for all 2 0 it holds that
d 21/ di
r
T 222 + |c| (.(xA)))))
0 <
10.1 2
See Appendix C.I for proof. By connecting the result from Lemma 5.1 with the approximate Fisher-Rao gradient flow, we are able to establish a Grönwall-type inequality for the KI. divergence of the policies with respect the reference measure. Lemma [. also illustrates that the coupled actor-critic flow is forcing-damping system, where the damping comes from the strong convexity of the loss 0 L(0, kr; d. and the forcing coming from the policy updates manifesting as the K, term on the rindt-hand-side of the estimate. Here can that in the finite action space setting, the forcing K, term is upper bounded by a constant and thus we can arrive at stability straight away. the current setting this is not possible and we must perform analysis over P(ASS) on the approximate Fisher-Rao gradient flow to arrive at stability.
Theorem 5.1. Let Assumptions
4.2 and hold. Let no > F. Then there exists constants
dTo
ai
To 7, X3 (((xA))
>
du Ba(SxA)
and a2(T, no. 7, Aa) > such that for all 10 and 2 0 it holds that
K2 S 01 + 02 ' e ( dr.
See Appendix C.2 for proof. Through applications of Grönwall's Lemma (Lemma A.3 two direct corollaries of Theorem show that the KL divergence the policies with respect to the reference measure and the critic parameters do not blow up in finite time.

---

Published as a conference paper at ICLR 2026
Corollary 5.1 (Stability of Te). Under the same assumptions Theorem 5.1 for all E (0,1), 8 E S and t 2 holds that
(-s) I Sa,e
Corollary 5.2 (Stability of 81). Under the same assumptions as Theorem 5.7 suppose that there
exists 0 such that dt Th S Cm7, Then for all y E (0, 1) there exists r1, r2 O such that for all
t 1 0 it holds that
10cl S rietzt
See Appendix and for the proofs.
the MDP has sufficiently small effective time horizon due to a sufficiently small discounting factor and thus is in a sense regularised, the KL divergence of the policies with respect to the reference measure remains uniformly bounded along the flow, see Corollaries E. and E.2
CONVERGENCE
In section we will present final three key components before we get to final convergence result for the coupled actor-critic flow. First, we characterise the time derivative of the state-action value function along the approximate gradient flow for the policies.
Lemma 6.1. For all 2 0 and (s, a) E S A, holds that
d QT (s)
2 / -
(J (s" (s", d" da (da" (da" (ds")s")) P(ds'|s,
--
dt
See Appendix D.I for proof. Observe that in the exact setting, where d -A as in 5). we obtain the dissipative property (Q7) 1:20 along the flow
d 2 (s. dt
(/ 17: "f" P(ds'ls, SO.
Second, Theorem 6.1 shows that the actor-critic flow maintains the exponential convergence to the optimal policy induced by the r-regularisation up to error term arising from not solving the critic full accuracy.
Theorem 6.1. Let { 0r)200 be the trajectories of the actor critic flow. Let Assumptions 4.1 and
4.2 hold. Then for all > 0 holds that
min V (p) V= (p) S 2(1 7)(1 - -=- rE(0.4]
T
E 6 (-5) (ds)
[ e 1 00, - 0...1 Prr
2r
See Appendix D.2 for proof. Theorem 6.1 shows that the exponentially weighted error term determines the rate of convergence the actor-critic dynamics.
Third, Theorem 6.2 shows that this emor term decays exponentially up integral which now depends on the rate of change of the true state-action value function and the timescale separation.
Theorem 6.2. Let Assumptions 4.1 4.2 and 4.3 hold. Let no 1 and 0 I 2 O there exists constants by, by > such that
1. Then for all
3r 5' +bb ( -(t-r) 1 d
2
( 0 0x. 12r -
0x dr.
Tr dr
See Appendix D.3 for a proof.
Finally ready present the main result of the paper. Using Corollary 5.1 and by choosing Te such that the critic flows runs much faster than the actor, Theorem 6.3 below demonstrates an exponential convergence the optimal policy for all 7 E (0, 1).
8

---

Published as a conference paper at ICLR 2026
Theorem 6.3. Under the same assumptions as Theorem 6.2 and lo2 > such that for all y E (0,1) and t >> it holds that
there exists 2 with n nuo
(1s (+() (ds) + min V (p) V (p) S 2(1 - 7) $0 rE(0,t]
TC" gt
1-2
2T
See Appendix D.4 for proof.
Corollary E.3 then shows that if the MDP is sufficiently regularised through small discounting factor, one can arrive at convergence for much more general class of timescale separation functions - T)-
7 LIMITATIONS
In this work, we only study the continuous-time dynamics of the actor-critic algorithm. Although this formulation gives insights into the discrete counterpart, a rigorous treatment of the discrete-tima setting is more realistic for practical purposes and is left for future research.
Moreover, for the purposes of analysis our critic approximation linear while in practice non-linear neural networks are used to approximate the critic.
Finally, our work assumes all integrals are evaluated exactly, in particular the semi-gradient 7 In practice these would need to be estimated from samples leading to additional Monte-Carlo errors. To fully analyse this left for future work.
ACKNOWLEDGEMENTS
DZ was supported by the EPSRC Centre for Doctoral Training in Mathematical Modelling, Analysis and Computation (MAC-MIGS) funded by the UK Engineering and Physical Sciences Research Council (grant EP/S023291/I), Heriot Watt University and the University of Edinburgh. The work on this project by D$ was partially supported by grant from the Simons Foundation. D$ and L.S acknowledge funding from the UKRI Prosperity Partnerships grant APP43592: AI Assurance and Insurance for Artificial Intelligence, which supported this work. The authors would like to thank the Isaac Newton Institute for Mathematical Sciences, Cambridge, for support and hospitality during the programme "Bridging Stochastic Control And Reinforcement Learning", where work on this paper was undertaken. This work was supported by EPSRC grant EP/V521929/1.
Last but not least, the authors would like to thank the anonymous reviewers whose insightful com- ments have helped improve the paper.

---

wwtpaaeeccccccaaoroooooooaooeeee Published as a conference paper at ICLR 2026
REFERENCES
Anas Barakat, Pascal Bianchi, and Julien Lehmann. Analysis of a target-based actor-critic algorithm with linear function approximation. In Gustau Camps- Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 991-1040, PMLR, 28- 30 Mar 2022. sttp::ttpp/::seetdtll
m prwss wttpse//vll/baaraata22222
html
Dimitri Bertsekas and Steven E Shreve. Stochastic optimal control: the discrete-time case, volume 5. Athena Scientific, 1996.
Jalaj Bhandari, Daniel Russo, and Raghav Singal. finite time analysis of temporal difference learning with linear function approximation. Oper. Res., 69(3)950-973,, May 2021. ISSN 0030- 364X. doi: 10.12877/pre.2222.2222 URL / 2024
2020.
V. S. Borkar and V. R. Konda, The actor-critic algorithm as multi-time--cale stochastic approxima- tion. Sadhuna, 22(5):525-544 1997. doi: 10. N0.10077/F02744557
Haim Brezis.
Functional Analysis, Sobolev Spaces and Partial Differential Equations. Univer- sitext. Springer, New York, NY, I edition, 2011. ISBN 978-0-387-70913--0
doi: 10. 1007
778-03387-70914- URL htttp:: //doi wrg/10. 1007 7978 0--30008777788 -----77777777
Pub-
lished: 02 November 2010, eBook ISBN: 978 978-0-387-70914777 Series ISSN: 0172-5939, Series
E-ISSN: 2191-6675.
Semih Cayci, Niao He, and R. Srikant. Convergence of entropy-reeullarized natural policy gradient with linear function approximation. SIAM Journal on Optimizaation, 34(3):2729-277 2755, 2024a. doi: 10.1137/22M15401.5
/RLp:tts: wtt.///o..orgg///o 2ttp://0001137/222144155
Semih Cayei, Niao He, and R. Srikant. Finite-time analysis of entropy-regularived neural natural actor-critic algorithm. Transactions Machine Learning Research, 2024b. ISSN 2835-8856.
http:///pppeeeeeieeeeee /ttp:ttppp///eeeereeei...ot URL httph:://///
Paul Dupuis and Richard S. Ellis. A Weak Convergence Approach to the Theory of Large Deviations. Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., 1997. ISBN 9780471076728. doi: 10.1002/9781 118111111888 165904. URL https: org 10 1002/ 00002/978111816880
Onésimo Hernández-Lerma and Jean Bernard Lasserre. Discrete- Time Markov Control Processes: Basic Optimality Criteria. Stochastic Modelling and Applied Probability. Springer, New York, NY, 1996. ISBN 978-0-387-94579-. doi: 10.1007/978- 1-4612-0729-0. First edition.
Mingyi Hong, Hoi-To Wai, Zhaoran Wang. and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal Optimization, 33(1):147-180, 2023. doi: 10.1137/20M1387341. URL https://
doi.
Caleb Ju and Guanghui Lan. Policy optimization over general state and action spaces, 2024. URL https://arxiv. http://aarvvvooorr///2222122..
Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learn-
ing. In International Conference on Machine Learning, 2002. URL hhtpps://api.
Bekzhan Kerimkulov, James-Michael Leahy, David Šižka, Lukasz Szpruch, and Yufei Zhang A fisher-rao gradient flow for entropy-regularised markov decision processes in polish spaces. Foundations of Computational Mathematicss, 2025a. ISSN 1615-3383. doi: 10.1007/ s10208-023 -2008-925--972-3. URL /tttttts://di..org/// 101.:110100000000000000010000000 1000/2122000-0000722-2-222-
Bekzhan Kerimkulov, David Šiška, Lukasz Szpruch, and Yufei Zhang. Mirror descent for stochastic control problems with gecasure-valuee controls. Stochastic Processes and their Applications, 190: 104765, 2025b. ISSN 0304-4149, doi: http:h/://booogggo...oo........p
7/www. i encedi rect. com htt.a//ww.......c..a..aaaaaaaaaa
10

---

Published as a conference paper at ICLR 2026
Vijay Konda and John Tsitsiklis.
Actor-critic algorithms. In S. Solla, T. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https::/ppocceediiiiceeeo nwurpps........pp..o........... ccppaprr files/paper/199)) F1le/6449f440 4ot.44/4444444eaaaaaaaaaoaaaaaaa
Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming, 198(1):1059-110) 1106, 2023.
Max Qiushi Lin, Jincheng Mei, Matin Aghaei, Michael Lu, Bo Dai, Alekh Agarwal, Dale Schu- urmans, Csaba Szepesvari, and Sharan Vaswani. Rethinking global convergence of softmax policy gradient with linear function approximation, 2025. 25C5. 03155
abs/
L. Liu, M. B. Majka, and E. Szpruch. Polyak-Lojasievicc inequality on the space of measures and convergence of mean-field birth-death processes. Applied Matematics and Optimization, 87:48, 2023. doi: 10.1007/s00245-022 000-022445-2220962--00 s00245- 022 09962-0
H. Ma, Z. Ren, B. Dai, and N. Li. Skill transfer and discovery for sim-to-real learning: A representationn-assel viewpoint. In Proceedings of the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8603-8609, Abu Dhabi, United Arab Emirates, 2024. doi: 10. 00.109//005899222244.. 10880166...
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International conference on machine learning, pp- 6820-6829. PMLR, 2020.
Shuang Qiu, Zhaoran Yang, Jianfeng Ye, and Zhuoran Wang. On finite-time convergence of actor- critic algorithm. IEEE Journal on Selected Areas in Information Theory, 2(2):652-664, June 2021. doi: 10. 109/1SATT.2021 30777754.
T. Ren, Z. Ren, N. Li, and B. Dai. Stochastic nonlinear control via finiteddimensional spectral dynamic embedding. In Proceedings of the 2023 62nd IEEE Conference Decision and Control (CDC), pp. 795-800, Singapore, Singapore, 2023. doi: D0.100//DDC49953.2223.. 203333338 10383842.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3(1):9-44, 1988. doi: 10.1007/BF001 0000//BF00150099 15509.
Andrea Zanette, Martin Wainwright, and Emma Brunskill. Provable benefits of actor-critic meth- ods for offline reinforcement learning. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393
Shangtong Zhang, Bo Liu, Hengshuai Yao, and Shimon Whiteson. Provably convergent two- timescale off- off-policy actor-critic with function approximation. In Proceedings of the 37th In- ternational Conference Machine Learning, volume 119 of Proceedings Machine Learning Research, pp. 1 11204-11213. PMLR, 13-18 Jul 2020. URL press/ 1ttps//v11////agg00000 hhlp//aa.n2200. h
ngs mmlr.
Yufeng Zhang. Siyu Chen, Zhuoran Yang. Michael Jordan, and Zhaoran Wang. Wasserstein flow meets replicator dynamics: A mean-field analysis of representation learning in actor-critic. In Advances Neural Information Processing Systems, 2021. URL htttps://openreviiee
net fet/foormm2 (1t.:rrommmm11111.11111111
11

---

Published as a conference paper at ICLR 2026
A KNOWN PROPERTIES MDPS AND OTHER USEFUL RESULTS
The state-oceupancy kernel dx E S)) is defined by
df (ds' (s) (1 7)
P (ds') (ds''sss) [dssss).
=2=0
where Pa is the n-times product of the kernel Pa with (ds (s) := d, (ds'). Moreover, for each t E ((A[S) and (8, a) E A, we define the state-action occupancy kernel
d= (ds, dals, a) = (1 (1-1) 11-7)
(P*P)" dals.a)
n=0
where (PT))) is the n-times product of the kernel pa with (P")" da' da'|s, a= (ssa)) da'). Given some initial state-action distribution 3 E P(S x A) with initial state distribution given by p(ds) = - B(da, ds), we define the state-occupancy and state-action occupancy measures
(d)s)) = S d= (dsls')p(dss) dh(ds, (a)=,
- { dds, dals', a')3(da', ds').
(10)
JSXA
Note that for all E E B(S x A), by defining the linear operator Jn P(S x A) - P(S x A) as Jx3(E) J
JSXA (')s", da'),
(11)
it directly holds that
20
do ds) = -1 Σ ==0
r" J 3(da, ds),
with Jsh the n-fold product of the operator Jr with J I, the identity operator on P(S x A). The following lemma establishes properties of the state-action occupancy measure defined in (10) and which are useful in the proofs.
Lemma A.1. For all P(A[S), B P(S A) and E B(S A) it holds that dE) = (E).
Moreover, for all 7 E (0, 1) we have
dz(E) - - [ (1 -))3())
(12)
For any E P(S A), P(A[S) and E B(S A), it holds that Proof.
() = (1-7) Σ ())))
n=0
(E)
where we just used the associativity of the operator Jr. Furthermore by letting m = n + 1 it holds that
d (F) (1
J. ((F))
==0
(1 7)
7 m
Jm ((E)
m=1
86
2 1m ((E)
(d)(E) - (1 - 3))
Rearranging concludes the proof.
12

---

Published as a conference paper at ICLR 2026
Theorem A.1 (Dynamic Programming Principle). Let O. The optimal value function V is the unique bounded solution of the following Bellman equation:
/ exp (-1 u(da),
(²) (=()==
where Q E BB(S x A) is defined by
Q7(s, a(s, J V=(ss) Exx.
Moreover, there is an optimal policy x= E P(A(S) given by
(- p(da), Vs E S.
x" x"(aass) = eep
Finally, the value function V is the unique bounded solution of the following Bellman equation for all E S
(x) SA (T dr (a, 0) π (edaas)
+ +IIn du
The performance difference lemma, first introduced for tabular unregularised MDPs, has become fundamental in the analysis of MDPs it acts substitute for the strong convexity of the V if the Sate-occupancy measure dī is ignored (e.g Kakade & Langford (2002), Zhang et al. 2021 Ju & Lan (2024)). ). By virtue of Kerimkulov al. (2025a). we have the Tollowing pertormance difference for entropy regularised MDPs in Polish state and action spaces.
Lemma A.2 (Performance difference). For all E P(S) and n, n' E 11
(P( - V (p)
- S 1.
dx'
O (s.a)+ + (a, s) (I - (dals) + (L(((()))) de (ds)). du
Lemma A.3 (Grönwall). Let X(s)
a(s), b= b(s) and y = y(s) be locally integrable, real- valued functions functioos defined on [0, T) such that is also locally integrable and for almost all [0,
y(s) + a(s) s b(s) + S X(t)((t)dt.
Then
u(*) + a(s) a(t) s()))s(()))))))))))))) b(s)+ I' X(0) [[
a(r))drr di, As e E
Furthermore, if monotone increasing and is non-negative, then y(s)+ + a(s) S b(s)e Mr)dr, Vs E (0, T).
B AUXILIARY RESULTS
Lemma B.1. For some P(S x A), let dz E P(S A) be the state-action occupancy measure. Moreover let k(ds, da, ds', da') = pa (ds', da'|s, a)dg(ds, da). Then for any æ E TI and any integrable f S x R, holds that
1 /Sv [(s,a)2d((dsdda) [( ((s , da, do', da')= S J SXAXSxA
Proof. By Hölder's inequality, holds that
S (s o d')nsds, do, ds', da')
(13)
$ <VerAessA (18,0)2ndds) da, ds', ,dd'))
USSAASSA (('',a')nxdds, da, ds' ,da')
13

---

Published as a conference paper at ICLR 2026
Moreover, observe that
SSKANSRA [1, da,d'', dad) ISSA Voon pm
dd()o,)) pddddddd''ss (( da)
= S SXA
d( da),
hence 13 becomes
$ (JonAsss f(s', ('s,,d)'x((ss, da, ds', daf)
USeAASSA {(,,))k(dss da, ds', .dd))
s (
[(s a) ((s,,)9d3(daa (ds, da))
(S
f(d d'ddd') ds', ds', }
Now by the first part of Lemma A.I it holds that
1 JSSANSKA f(s Prosssdssssssss ndsss da, ds', da')
I SxAXSXA f(s',a') d' (ds', da' da'|s, (dd da)
J JSX f(s, a(s,a){dJ+ add dd),
where JR : P(S A + P(S A) is defined in have
11I. Then by the second part of Lemma A.1 we
da, da, ds' da')
(I as) (f
(J a)s,
< J SxA [(,, a) dF da),
which concludes the proof.
To alleviate notation let Qe(s, a) = Q(,, az 0c) and Ar(s, A(s, ((ssay a.).).
Lemma B.2. For some On E N and mn E II let (T,,0 0.h12 be the trajectory of coupled actor-
critic flow. Moreover let K; sup.es (L(T(())))l)) There exists Ci > 0 such that for all 2 0 holds that
sup .ES
|0 I |A B((xA))
|Ax (SxA)) 5 221( + In dite du ((((SA))
107 (x(
5 7- (IoBB(SSAA) +T7K.),
1
dite In dj Ba(SxA)
2
sup |0,1 + sup K,. re(0.c]
re(0)t)
Proof. The first claim sup.es (-1)) I |AA (Sx A) follows trivially from the definition of the approximate Fisher-Rao gradient flow defined in (9. Moreover, it holds that
ditt
SA (Q.l1.0)+ + dote (.)a) (dall))) dy
| AllB,(sA))
Q: + +n
dμ
(x(8x
di
2 Q: + + In
dp
Br(SxA)
d
2 Q2 (((B)(((x))
2r
In
du
(x())A)
14

---

Published as a conference paper at ICLR 2026
where we used the triangle inequality in the final inequality. Moreover, the state-action value func- tion Q a fixed point of the Bellman operator defined in (3. Hence, for all (s, a) E A, we have
JSXA +2 J Q (5', d') pr (ds', dal ( + my / (T(m(()))))l)) P(ds'ls, [s,a).
= (c,))
Taking absolute values and using the triangle inequality have
10 (x, 2 |c| Ba(SxA) 10 BA(SxA) + sup KL (L(
e'ES
= =|c ((xxA) + ! (x()A + TTK,
Taking the supremum over (s, E S x A on the left-hand side yields
10 (Sx S (|)x((xx)++ + + (n((xa) + TyKe.
Rearranging gives
11->) 1 (((SAA) S (cm,(sxa) + tTkK
which is the desired bound. Recall the approximate Ras gradient flow for the policies (m.) which for 2 0 and for all (s, a) E S given by
O2 In dmt (s, = (,((,,)) + Iln dj
d (a, - / (Q((s, + du
dis d (d', .)) Te (ddd((l))
du
Duhamel's principle yields for all t 2 0 that
In dit (s, = In doo (a,s) ( e-((-T)) (L (add Qr((,a) dr (14) dp
du
+T f (L(T-
Since To E 11. there exists C1 2 1 such that have that for all t t2 O.
In dug dp B((SxA)
S C1. Then by Assumption 4.2 we
In dite (s) du
sc1 (' e ( / Q (r( a))
+
-(((r))) KL(TT, (1()((l)))) dr
e (((-rr) 10, dr + + (6 e-((t-r) dr
S 01 + 2
2
S C1 +
|0,| + sup K,, (0,0.)
sup
7
re
where in the last inequality we used Jo e e-(((-r) dr < -. Taking the supremum over (s,a) ES A yields
dm
2
In
$ C1 + sup
|0 + sup K,,
dp Bs(SXA]
J rE(o rE(o.r]
which is the desired bound.
Lemma B.3. Let Assumption strongly convex.
4.3 hold. Then for all π E II it holds that (.,T:d)) is Ag 7)-
Proof. For any & E P(S x A), let E = SSXA 6(s a))(s, a) a)TT)) TE(ds, E RNNN Then by
Lemma A.I and Assumption 4.3 it holds that Edz 2 (1 7) z (1 - T and thus L(, n. dg)
(1 ? -stongly convex.
15

---

Published as a conference paper at ICLR 2026
B.1 PROOF OF LEMMA 4.1
Proof. Recall that Q(s, a) = (0, ((s, a)) for some 0 E RR and that for all π E IT, there exists
0 E IR such that Q (s, a) = (0,, (( a)) by Assumption 4.1 Then by definition of the semi- gradient of the MSBE R P(AS) R. in 77 holds that
(), 0m)
(Q(s,a) T Q(s Ø(s, a)d (da, ds), 0 On
8xA
-V 5xA
(Q(s. (=, a add ds), 0 0s
+ SxA (QT(s, - T ((. ((s. ds), 0z
(V 5xA (Q(s,a) (s,
0z
(VeAA - (s', (ds', da' (s, a) (d da),0 0 0z
where added and subtracted the true state-action value function O B, A) in the second equality and used the fact that is a fixed point of the Bellman operator defined 3 To ease notation, let e(s, a) Q(s, a) Q (s, Multiplying both sides by and using the associativity of the inner product, we have
(g(0 R), 0z)
-- ( c(s, a),s) a) (da ds), 0 0x
(1 SXA
+7
E(S': (ds', da' as da), 0x
-- SXA
c(s,a)
(o(s, 0z) dī ds)
I e(s',a')
+7
(d(s,a), 0z) Pa (ds' da'ls, (s, a)d da)
SXA
((),a)2dg(1)) (da,
SXA
+7
E(s, (ss,a)=(s',a')) p")) da' a)d) da)
SxAxSxA
r()) + 1())
Now applying Lemma B.1 (2)
have
((2)
c(8,a)E(s', d') (ds', da' Is, a)d (ds, da)
SxAxSXA
s - I
Vス SXA E(S, da).
Thus it holds that
- (g(0 0₂) $ (1) + (2)
$ -(1 v7)
E(s, dls.a))5((d,dd ds)
|SXA
SXA
-(1 v
(Q(s,a) - a)) dd (a ds)
SXA
=- /
(O 0z)
where the last inequality follows from the Assumption 4.1 and the definition of Q(s (0, Ø(s, a)).
16

---

Published as a conference paper at ICLR 2026
C PROOF OF STABILITY RESULTS
C.1 PROOF OF LEMMA 5.1
Proof. Consider the following equation
1 d 217 dt
1 d Pe, 0:
10.1
(15)
dt
(0,, 0.)
V - QT)(s.)))())(( o(s a) dz (da, ds), 0z
V SXA
()) (da, 0,
(1 SXA
T Qr(s dz
(da, (da,
1 + J
where we used the O2 dynamics from ( in the second equality and the definition of the semi-gradient in the third equality. For any π E 1L let 2 ERN be
Ex = / BxA
(1,8))((sa) dā(da,, ds).
Then by definition we have that Qc = (0,, (ss a)). hence for J( we have
[ Qi(s,a)o(s, a) di (da, ds), 0z
8xA
(0e, a)) ((s, add (da, ds), 0:
On (1SAA ((s, a) dg (da, ds) 0::
USA
(0,, E²)
(16)
Now dealing with J( expanding the Bellman operator defined in 3 we have
1
Qc(ss a) d (da, ds), 0,
SxA
c(s,a)δ(s, adda (da, ds), 02
SXA
(1 ((s' a (ds', da'|s, a)d
(da, ds).
(JS (/ILLT(t)))PP((ssssooos
00
< (101, [ +,4,0) 1
where we defined
1 (1 (0,, ((s' a) da'ls, o)d' (da, ds), 01
a))(( 1 (Son ( ([KL(T(((s)) (.)dssss (d)(aa,))dddddd 0,
Moreover, to notation let
K, = KL (L(
sES
17

---

Published as a conference paper at ICLR 2026
and temporarily let Kiz Ke(ds, da, ds', da')= pm pds'd', da'ls, (da, ds). Now focusing on 11 holds that
1000 (JocAssaa Oss, d')) (8, (dd ds', da, ds), 0r
J SxxXSXX (0,, 0s, o)) (0z, p(s', d))) kelds' da' ds, da).
Now using Lemma B.1 with f = (8, .0. :)) we have
10 S
(Or, (o(ssaa)" dz (ds, da))
(oon (Or, a)) d3 (ds, da)
EXA
1
(0r, H(s a))² (d (ds, da)
V7
SXA
(0,, (0))))))))
Thus all together it holds that
71,00 s (0,0000000)
Now focusing on n!2) we have
1. (S (KKL(((1))))PPassi a) ø(s. a)dz' (da, ds).
< Kr SSXA
als,a)d (ds, da) 0cl
$ [
where we used Assumption 4.2 in the final inequality. Hence along with (16). (15) becomes
217 dt 10,1 < JJ(2) + J(
1 d
(17)
S - -Oc, I + (c + 71,") + ryy1
S (0r, 5. -0 0.) + / (0,, E -T + (c(B,(Sx)) [01| T,KK 0.
(1 VF) (0,, 0x) + (Ic)B((SXA) + T.) 10cl-
Observe that by 12 and Assumption 4.2 Σ" NxN is positive definite for all P(A[S), hence it holds that
(0,, 0,) 2 7) 0,1
Therefore 17 becomes
1
10,12 < -(1 ~) 10,12 + (Icl,,(SxA) + 0,
217 dt
Let :- >s(1ps1111- 7) - vT). By Young's inequality, there exists > such that
1 d 00, 2 + .ク 10,11 + ((c),,(8AA) ?2
2m di
2c 10,12 + ((B..S k2g2K3
-r + 2
where we used the identity (a (a+ 6)2 < 20² 26. Choosing e we arrive at
1 d
r <
22?
+ lc
0
0,1
2n, dl
2
which concludes the proof.
18

---

Published as a conference paper at ICLR 2026
C.2
PROOF OF THEOREM 5.1
Proof. By Lemma 5.1
we have that for all r 2 0
1
r
7 T² K
[(B((SXA))
10,13
10,
+
+
2f, dr
2
Rearranging, it holds that for all t 2 0
Bl( + +2
1 d
10, 10.12
0, +
Th, dr
r2
Multiplying both sides by -((-r) and integrating over from to & we have that for all 2 0
f -(t-T) 10-2 1r,22r s - / (((-r) 1 d
B(xAA) r
f E e--(t-r)
10,12 +
(18)
1/r dr
2T272 po
r y e
K-.t-1))?ddr
+
/'"
T((-rr) 1 d
2r
[ e-r(t-r)
Thr 10 dr
Jo
r2r
T
where we used that J0 e-((t-r) S Integrating the first term by parts, we have
- (/' e ((--r)
T 1
10, .d = 10,
10u
2
((--r)
+ e
rt
10
dr
(19)
Th. dr
770
T/
1' 10,1
+(t
dr dr.
Since by definition have that for all
0, 176
and
de n it holds that
d
L. 10, T((-)) d
dr Tr dr 2 0.
n
Hence dropping the negative terms on the right hand side of 19 and using that me 2 for all t 2 0, we have
1 -
1 d 2 dr S e - 10o1
Twoo /
T
(00
+
The dr
Tno
Substituting this back into 18 for all 12 have that
1012
- 101 < ---
Tmp
Tro
2|c 2
(8)xA)
272
pt
e
2
2
0
Grouping like terms we have
1 (1- / e f Se
100
B)
2r2n2
1' (T(t-r)
Tmjo
T27
Recall that we have no > F to ensure that t 2 0 that
r Imo
> 0. Dividing through by T'o
T
gives for all
f e r(t-)) 1r Pdr 50, 02 I' (r((tr)
(20)
where we've set
10u 2
2/c|2
01 :
+
Pmpo
T
T2r 1
l'm
Tyu
19

---

Published as a conference paper at ICLR 2026
27
02 = 1 Tip
Recall the approximate Fisher-Rau gradient flow for the policies { 1120 which for all t 2 0 and for all 8 S. a E A is
0, In dm (s,a) (s.a)=-((((6 (0,((s))))) (=)()==((((s))))) + In die (a,
- - (Q((s,0) + + dae (a,
oee Te(dals)
du
du
du
Duhamel's principle yields for all t 2 0 that
In d du
(s.a) e- In d
(a. s) + ! e-(( (L, (ddl) (,(,,))) -r dr (21)
du
+ (L(T((ss)d
Observe that since E 14 there exists Ci 2 1 such that In
do
C₁. Using that
dp
Bo(SxA)
e- < 1 and assumption 4.2 gives that for all 2
dig In
(8, a) +
10 +
e (((-r) (TTT
du
SC +
c- 10,11 +
e-((t-)
Integrating over the actions with respect to Te (-15)) E P(A) gives for all / that
(L(T((-ss)l)) - Ci + / e 00 + f. (tr() Krdr
where we again use that K, = sup.es (T, (u). Following from the techniques in Liu et al. 2023 observe that from (21) and Assumption 4.2 similarly get for all 2 0 that
In dp (a, In du: (s, a) S Ci du
e -- 10,1dr - -r S - <d,
dm
Now integrating over the actions with respect to the reference measure μ E P(A) we have
(L(5) s C +2 2 J 10 - (tdr
Moreover, using the non-negativity of the KL divergence, it holds for all t 2 that
(+ss(()))) < (L((s(-)))))) + KL क. (+1s)) < 20, + 4 / e e- dr
Since this holds for any E S, it holds for all 2 0 that
K, S 201 + L. c c- dr
Now squaring both sides and using the Hölder's inequality, we have
K² < 20 +4 4 1 e (t-r) ( 0,
2
$ &8())² +
6 (((-r) 10,1dr
-(C))² + +32 (/"" e (((-r) e - 11,,111
(-((-r) dr (J" (T((--)) 10.12
2( + 32
<8
32
/"'
- 10.12s Pdr,
(22)
20

---

Published as a conference paper at ICLR 2026
where we again used So e T( dr 1
We can now substitute
20 into 22 to arrive
32
32
KJ S 8(C))2 +
01 +
02
T
7
e --((t-r (2 K2dr
= 01 + a2
with a1 8(C,) + 28 01 and 02 3202.
C.3 PROOF OF COROLLARY
Proof. By Theorem 5.1 it holds that
K3 S a: + 02 f K,((t-))
as Observe that by multiplying through by ett we can rewrite this
" S e + a2 /'' K2dr.
Hence after defining g(t) = ' and applying Grönwall's inequality (Lemma A.3 for y (0, 1) it holds for all f 2 0 that
K2 S 01e
C.4
PROOF OF COROLLARY 5.2
Proof. By Corollary 5.1 and Lemma
5.1 for all 7 E (0, 1) it holds that
d
r
00 2 <
77 0.1 + by
2 dt
2
such that
+2r27aa
and
b
r
Recall that there exists such that di 1t S 0T}, then another application of Grönwall's Lemma then concludes the proof.
D PROOF OF CONVERGENCE RESULTS
D.1 PROOF OF LEMMA 6.1
Proof. By the definition of the state-action value function 2 it holds that
d
2= (s, Q (s,a)
(8) lim
d
h-0
{ d v (s' (s))P(ds')ssa))
=7 7
s
Now observe that by Kerimkulov et al. (025a)((rooo) of Proposition 2.6], we have
1
V=((8)
(8 (8, ()s,a)))mmd((aa "dm (ds'
di
1 SxA
Thus we have
d
1-7 S (V (s", "da" P(ds'|s, (s)
(s,
dt
21

---

Published as a conference paper at ICLR 2026
D.2 PROOF OF THEOREM 6.1
Proof. Recall the performance difference Lemma (Lemmaa A.2 for all p P(S) and x' I1
V. (p) - V (p)
S [/A QT (s) + In
da' (a, .) (5 =')' (II((((16)) dd(ds)
du
Now let π n² and n' ne and multiply both sides by -1 we have
()) - V (p) - / (Js (0T (0m(0.)) + IIn
dae
(a, (x* :e)(dals) (23)
du
+ KL (-16) do (ds).
Recall the approximate Fisher-Rao dynamics, which we write
0, In dite dp
(s.a) a) + (c((s.)) + + ditt (a, 8) - L (R((s,0)) + du
dite
du (a', Te Te(dd'(s) = 0.
(24)
Observe that since the normalisation constant (enforcing the conservation of mass along the flow) La (Qe(s,a) a) + r In dr (a, xee re(dals) is independent of a E A, holds that du
S (u((s +
dit (((
(d', The (d(a'a)s)) (I=" ( (dals)) = O.
du
Hence adding 0 in the form of (24 into
it holds that for all 2 0
(Jend
Ō In due (a, s)(x" (d)(aa)) (ds)
(D) V. I- Usx
(25)
dp
+ #1 (Qi(s, Q (s,a)))) (d) (ds) KL (Iss (ds)
SXA
By Kerimkulov et al. 2025b Lemma 3.8) and Corollary 5.1 for any fixed v E NT the map 1 (I(());) is differentiable. Hence we have
/ 0. In di (s.a)(TT* (d)ss) = / O In du
dm
(8, a) (dals) - O In d (s, (s)a)
dj
du
/ a In
dite
as, (dals)
du
d (-Is))
dt
where we used the conservation of mass of the policy dynamics in the second equality. Substituting this into (55 we have
1-7 /
1 11(-100 d
(T) VF (p)
(-Iss (ds)
(26)
(Qi(s.a) - (s) md)(dalsdd (ds) / (L(T"((1s) (ds)
8xd
22

---

Published as a conference paper at ICLR 2026
Focusing the second term, we have
J (Qi(s,a) (Q((s,))-Q* (T, )d, -m (ds)
< a) (s. / TV( (-1s), (1ls), (ds)
V2 10-- 0zi1 / KI( KT( (10)) (ds)
1
V2 00 0z. ([
1
(ds)
where we used Pinsker's Inequality in the second inequality and Hölder's inequality in the final such that
inequality. Now applying Young's 's inequality, there exists
10: 102-0x.1 0z,1 (LL)
$ I 00 0x,1 2-
1
2 2 (1)) (ds).
(ds)
2c
Substituting this back into 26 and choosing
22 we have
(p) V (p) = 1-7 ( dt / KL (15)) (ds)
1
T
2 / KL(T (ds) + - 10: 0xT!
4T
Rearranging, we arrive at d
dt / (I,(T"(1s)) (ds) S (IL(T(++))) (ds)
-1 (z* - V (0) + 4r
00, 0m, 1.
Applying Duhamel's principle yields
| KL(T" (1)(T(((s))(((((ss)) (ds) s c 11 s /
(ds)
-(1-7) ( F(( (V= (p) - v (p))dr +
2r ( =((-r) 10, 0n. 1dr.
1
Now using that f0 e [(t-r) = 21-e i1 we have
/ KL( (L)) (ds) se 3' / (15))=(( (ds)
2(1 71 e
2T { e (t-r) 0, 00,0
min r rE(o,)
(v+ (p) v, (0)
Rearranging, we have
min V. - V. (p) 5 rE(0.s)
T
e- / KL( ((1s) (ds)
2(1 - 7) - =)
1 f c ((- 10, 0z. Prr
+ 2r
which concludes the proof.
23

---

Published as a conference paper at ICLR 2026
D.3 PROOF OF THEOREM 6.2
Proof. Using the chain rule and the critic dynamics in
8
we have that for all r 2 0
d
1 (1 d0, 7/
dl
21,, dr 00 0x.12=
dr 0,
0,
0TTT
dr
1
d8
= (g(0,, Ty) 0, 0x.)
0r A
7/
dr
Let r A3(1 (1)- Using Lemma 4.1 and the Ag-strong convexity of L(,, tI 8) and recalling that L(BN, TT.) = 0 for all r 2 0, it holds for all r 2 0 that
1 d 00 0x 12 (0( me); 8 az,)
1
d0=,
, m..
2m dt
T dt
1
d0 mc
-(1 (1 / o ((),, Te: 3) mm.)
0₁
0
T
dt
r
1
d0 m
-11 7) v m3;
00 m..
12
0
0.
2
1
dt
T
1
d0
-1 () (3)
00
m..
10:
0z
2
2TT
dt
(27)
T
d0,r
= =- (1 L mx
10:
0
2
277
2m
dt
where we used Hölder's and Young's 's inequalities in 27
Since no > and т) is non-decreasing
function, it holds that 7) > for all t. 2 0. Hence F-
2m 0 and thus can drop the second term.
Moreover the Ag-strong convexity of [[;,T;; for all π E I,, gives that
n;; (3) along with I(Om; n; B) and VoI.(0 x)
10, 22 A3
Hence for all r 2 we arrive
d 0r 1 S 2nr dr
d0 2
2 0r Oz,
2nr
dr
Rearranging, multiplying by e e(((rr) and integrating over from 0 to it holds for all t 2 that
5(1-)) 0 - 0n. 1n.12dr S r /' -- [(-rr) 1 d
1 /' e s-(((r))
d0=,
Tr dr 00 2r.1dr +
dr.
dt
Integrating the first term by parts (identically
from the proof of Theorem 5.1
we have
10. 0m,
- 10g
0xo 12
3(--ri
0, 0z.1 1er _ r
+ -
7
+
T
S.' c (t--r) 1 10, 0z. 0z. Pdr
(// 10, 0z
(t-rr d
de
nr
dr
2
+
/ e
(1--))
do,
dr
dr
Since for all 2 0 holds that 1/1 2 and d
1: 2 0,
we have that
1 10, 0x
((--))
dF //
dr
2 0.
1/
24

---

Published as a conference paper at ICLR 2026
Thus after dropping all negative terms and using that 7/ 2 7o for all t 2 0. we have
(1-
T no ! e- = 10,-0 - -0,, 10r,1er S e
100 0mo 12
/-" - (t-r) 1
d0
2
dr.
Tyjo
71 dr
Since 170 > 전 and < 1, it holds that
J >
0 and hence it holds that
27
10 0, Ir e e
10n 0zo 12
+
S 6 [(-rr)
1
d0
2
dr,
I'm (1 21
17/ dr
21
which concludes the proof.
D.4 PROOF OF THEOREM 6.3
Proof. By Theorem 6.2 we have
J 500 00, 0, dr se
100 fx 1
+
I 6
[t--r)
1
d0 2
dr.
I'm
(1
dr
vM
2T
Hence it remains characterise the growth of the final integral. Observe that for all E P(A[S),
0 E R. satisfies the least-squares optimality condition given by
0 = arg min L(0, :: = (Ssna (ss ())( a) B(da, do))
( ø(s, (x), B(ds,
Setting л = म, and differentiating time we arrive at
d0 dt
( ( ()s,)))(s,)) B(da, ds) )
(1
UxxA o(s, a) d Q*t (s, a) B(ds, da))
dt
Hence by Lemma 6.1 Assumption 4.2 and Assumption 4.3 for all 2 0 it holds that
d0 m
(1 SxA
a(s a(s, a))(s,))) B(da, ds)
(f
d(s,a) d dt Q. (s, B(ds, da)
dt
SxA
d
B((aa) ((s,a))(s,a) B(da,
SxA
dt Bs(SxA)
op
d
Q
As dt
Bs(SXA)
//(0 ( USA (s" a (dmm (d=""''))))))) ""00 p(dd'l..)
7
A1 --7) 7)
Ba(Sx
| B))SxA)
sup @mm ( sES
Aa(1
Now using Lemma
B.2
it holds that
A
(Sx
sup SES
$ |A7 B.(SxA) |A (llBA(SXA)
drp
2 21Qr(1,(Sxa) + 27 dac
<
2 2 (x(8x)) + 2r
In
djc
djc Bx(()))
B)(
Hence by Corollaries
5.1 and 5.2 and Lemma
B.2 there exists an a2 > 0 such that
d0
S ast
dt
Thus Theorem 6.2 becomes
S0 e (t-r)) 0r 0n, P < e
0g
Oz 12
Q'1
e 2 (t-r)
agr
Ino (1 20
J0
n0
25

---

Published as a conference paper at ICLR 2026
Let 7/ nee for any h
> 2 + ag. Then observe that
/
e - (tt-r)
ea2r
dr
1
e
2t
Ju
c(5ta=-ki)rd
F/r
770
13 -1
1
<
e it
e
770
7. + 02 - kn
2
e - =t
110
(3
+ 022 ki)
hence all together it holds that
[ (- - 8T, 1r S
100 0x 12
O
+ e
Dnjo
21 no
(10
21
(3
02 -
ki)
Substituting this into the result from Theorem
6.2 concludes the proof.
E
ADDITIONAL RESULTS
Corollary E.1 (Uniform boundedness). Under the same assumptions Theorem 5.1 for y (0,1)
such that
64y²
< 1 holds that 02 < t and for all t 2 it holds that
12 Er
KL(T (L((t((\s))p)² < T 02
E.1 PROOF OF COROLLARY
Proof. By Theorem 5.1 we have that
K2 < a1 02
Taking the supremum over [0, t] on the right hand side, we have
K2 S 01 + 02 sup K..
rE(u.4]
Since this holds for all 2 0, we have
K² < 01
sup K.
sup
rE(o.t]
[E
Now forcing
42 0, which is equivalent to the condition
64
<
r
Pr
No
Hence after rearranging have
K?
S sup
K² <
O1T
rE[U,]]
7
Remark E.1. Observe that if one does not apply the loose upper bound e - < in 21 from the proof of Theorem 5.1 it holds that
32 01
a1 a1 8c 2rt (Ct)
with Tro
1u
(SxA)
Then choosing Th0 + for any 0 that the
r2 [L- r
conditions of Theorem 5.1 holds, formally sending
we obtain (T.(Te) () for all
s S.
Corollary E.2. Under the conditions of Corollary E.1
there exists R such that for all t 2 0
holds that
0.1 R
26

---

Published as a conference paper at ICLR 2026
E.2
PROOF OF COROLLARY E.2
Proof. By Corollary E.
for sufficiently small 7 > 0 it holds that for all / 2 0,
K2 S T 02
017
Hence by Lemma 5.1 we have
2/c| ()x + 22
417 7-a2
1 d 1el²
T -1h 10 + t
dl
The uniform boundedness in time of [0:] then follows by Grönwall's Lemma (Lemma A.3
Corollary E.3. Under the same assumptions Theorem 6.2 for (0, 1) such that
2x2
there exists d1 0 such that for all 2 O.
min V (p) V= (p) S 2(1 7) - e -2-
7
c / (-1s(())) (ds)
rE[U,[]
I' e ((((r) 1
di 0
dr
f/r
E.3 PROOF OF THEOREM 6.3
Proof. Following completely identically to the proof of Theorem 6.3 we have
d0x, dt
5
(Sx A) sup AA
0eTr(-1s)
As(1 - 7)
SES
2
(1 - 7)² ((SAA) + K,) 2 +
+ C
sup rE(O.1]
|0,1 + sup K, rE(0,))
d0 dt
2 S d1. Hence by Theorem 6.2
Then by Corollaries E.1 and E.2 there exists b2 such that
have
min V (p) V (p) 2(1 7)(1 e- E (1g (-1)) (ds)
0
+ dl 1' - ()-r)) 1
Jo
dr
27


Reviews:
[{"review_id": "Q6Ll3NeEws", "reviewer": "Reviewer_o1x9", "summary": "This paper presents theoretical analyses of the Actor-Critic algorithm for continuous action spaces in entropy-regularized Markov Decision Processes (MDPs). In addition, it provides explicit gradient expressions for both the critic and actor updates.", "questions": "1. Assumption 4.1 appears to be quite strong and may not hold in practical scenarios, particularly in environments with continuous state and action spaces (potentially high-dimensional). Does the convergence guarantee in this work critically rely on this assumption?\n\n2. Could the authors elaborate on the technical challenges encountered when extending the analysis or algorithm to continuous state and action spaces? How were these challenges addressed in the proposed framework?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The theoretical results presented in this work appear to be sound and logically consistent with the proposed framework. The derivations are coherent, and the assumptions, although somewhat idealized, support the validity of the main conclusions. Overall, the theoretical analysis contributes to a solid understanding of the algorithm’s behavior and convergence properties.\n\n2. In addition, the definitions and problem descriptions are generally clear and well-structured. The authors provide sufficient background and notation, making the paper accessible to readers familiar with reinforcement learning and KL-regularized optimization. The clarity of exposition facilitates understanding of both the motivation and the technical development of the method.", "weaknesses": "1. Some assumptions, such as Assumption 4.1, seem overly strong and may not hold in practical continuous or high-dimensional settings. The convergence results appear to rely heavily on these assumptions.\n\n\n2. The technical novelty and improvements over prior work are unclear, and the contribution would benefit from a clearer explanation of how it advances existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vNs7gowdgB", "reviewer": "Reviewer_Va3v", "summary": "The paper proves the convergence of an actor-critic algorithm for entropy-regularized reinforcement learning (RL) in the setting with continuous states, actions and time dynamics. Concretely, the paper shows that the algorithm is stable and converges at an exponential rate.", "questions": "The authors choose to regularize on a fixed action distribution \\mu rather than on a full stochastic policy that selects a different action distribution in each state. Several existing algorithms such as TRPO regularize on full policies, so this seems like a limiting choice. Why do you not regularize on a full policy, and would it be difficult to extend the result to such policies?\n\nWhat do you mean by a policy being \"equivalent\" to the reference action distribution \\mu? My interpretation would be that the policy selects the same action distribution \\mu in each state, but I believe that this is not what you mean.\n\nWhy did you choose to minimize the MSBE rather than some other objective? There are other objectives that seem to have better properties for entropy-regularized RL, e.g. Logistic Q-learning.\n\nEquation (23) is a non-regularized objective, so Lemma 4.1 seems to imply that we can upper bound the regularized gradient by a non-regularized gradient. This is curious to me since I thought that bounding the regularized gradient directly would be easier. Why did you choose this particular proof strategy?\n\nLemma 5.1 looks familiar to me as a flow constraint, but I have a hard time interpreting Lemma 5.2, which looks like a differential equation. What is the interpretation of this lemma?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "If the proofs are correct and novel, I believe that the paper makes an important contribution.", "weaknesses": "I am well familiar with the theory of entropy-regularized RL, but I am not an expert on continuous-time dynamics, which makes some of the proofs difficult to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qIlPZgKXhd", "reviewer": "Reviewer_354V", "summary": "The paper analyzes an actor-critic method in continuous time with (i) TD learning as the critic, and (ii) policy mirror descent for policy optimization to solve entropy-regularized MDPs with general (Polish) state-action spaces. Under a realizability assumption on Q-functions throughout the trajectory, and for sufficiently small $\\gamma$, an exponential convergence rate up to a critic error was established.", "questions": "- An interesting connection between critic error and the policy optimization was established in (Kakade, 2001), which is known as the compatible function approximation. One can formulate the NPG updates as the minimizer of $L_\\theta(w) = E_{s,a}[|\\nabla \\log \\pi_\\theta(a|s) w - A^{\\pi_\\theta}(s,a)|^2]$. For linear function approximation, this may roughly be thought as minimizing $E|\\phi({s,a})\\cdot w - Q^{\\pi_\\theta}(s,a)|^2$. Since one also uses TD learning with linear function approximation using the same feature map $\\phi({s,a})$ to approximate $Q^{\\pi_\\theta}$, which gives us an exact solution under the realizability assumption, the insight from (Kakade, 2001) also applies here nicely. Is this idea used in the paper in an related/alternative form? The joint Lyapunov analysis that connects the actor and critic may possibly have a connection with this.\n- In conjunction with the above point, the realizability seems to be a significant contributor to the exponential convergence rate. What would happen without this realizability assumption? More clearly, how would the projection error propagate?\n- Is the small-$\\gamma$ assumption inevitable? Additional discussion on its cause would be useful.\n\n**References**\n\nKakade, S. M. \"A Natural Policy Gradient.\" Advances in Neural Information Processing Systems 14, 2001.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is very well-written and mathematically rigorous. The technical contribution is solid.\n- Convergence of an actor-critic method based on TD-critic and PMD-actor was analyzed for MDPs with general (Polish) state-action spaces. The existing works analyze entropy-regularized actor-critic methods for finite action spaces, where the optimality gap has a term $\\sqrt{\\frac{\\log |A|}{t}}$. Since this bound becomes vacuous for continuous action spaces, the contribution of this paper is significant.\n- The paper establishes a stability analysis that explicitly captures the interplay between entropy regularization and timescales.", "weaknesses": "- The paper makes a strong assumption that $\\gamma$ should be sufficiently small, given that the practical $\\gamma$ values are typically larger than 0.9. Indeed, the range of $\\gamma$ shrinks as $\\tau$ increases (e.g., in Corollary 6.1).\n- Equation (40) indicates that the decay rate of the optimality gap is $O(t^{-1/2})$. This makes the exponential convergence argument in the abstract a little confusing. I would recommend revising that statement in the abstract to indicate that it is exponential convergence up to an additional error term that stems from the critic.\n- The impact of stochasticity is not characterized as the analysis is fully deterministic. For TD learning, this impact can be particularly significant, and may have significant implications for policy optimization.\n- The realizability assumption is strong, but it can be relaxed by including an additional projection error term.\n- A continuous action space $A$ would make the computation of an exact softmax policy intractable due to the integration in Line 188-189. As such, in order to sample from this policy, one may either compute an inexact policy (via an approximate integral), or use Langevin dynamics to sample from $\\pi$ without explicitly computing it. In any case, this brings an additional error term. For this reason, in practice, this extension of finite-action softmax can be intractable and variants such as deterministic policy gradient (Silver et al., 2014) are used.\n- (Minor) There is a continual switching between \"regularisation\" and \"regularization\". I recommend making it consistent.\n\n**References**\n\nSilver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. \"Deterministic Policy Gradient Algorithms.\" In International Conference on Machine Learning (pp. 387-395), PMLR, 2014.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ueVybzxj2C", "reviewer": "Reviewer_N1Ri", "summary": "The paper studies infinite state-action space actor-critic with policy iteration and entropy regularization. It proves stability of the system and proves exponential convergence.", "questions": "Q1) Theorem 1.1 and Lemma 1.1 are insightful but I am not sure how novel it is. If it is established then I urge the authors to state it a proposition or property. And lemmas and theorems  be reserved for the results which are novel contribution of the paper.\n\nQ2)  Policy Mirror Descent requires updating all states at each iteration, how can it be extended to online setting where we have access to only samples?\n\nQ3) Corollary 5.1 (stabiliity): The constant $a_2$ >0, then how does the upper bound of $e^{a_2t}$ indicates stability, it is diverging very fast? Same for Corollary 5.2?\n\nQ4) Corollary 5.3: At the limit $\\tau \\to\\infty$, RHS converges to $a_1$. However, intuitively we expect that when the temperature becomes very high (so does the penality of deviating from the current policy) the policy should not change at all. In other words, at this limit $a_1$ should be zero. Is it the case?\n\nQ5)  Authors mention that entropy regularization boosts exploration but policy iteration requires updating all states at each iteration. What benefits  entropy regularization provides to policy iteration?\n\nQ6) In general, policy iteration conveniently convergence to the global optimal policy, and entropy regularization makes it even more stable ()", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The analysis is done for continuous state-action space.", "weaknesses": "Presentation of results is bit cluttered, its little difficult to understand the contribution. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)