SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dongsheng Ding*
University of Pennsylvania dongshed@seas.upenn.edu
Chen-Yu Wei*
University of Virginia chenyu.wei@virginia.edu
Kaiqing Zhang*
University of Maryland, College Park kaiqing@umd.edu
Alejandro Ribeiro
University of Pennsylvania  
aribeiro@seas.upenn.edu
We study the problem of computing an optimal policy of an infinite-horizon discounted constrained Markov decision process (constrained MDP). Despite the popularity of Lagrangian-based policy search methods used in practice, the oscillation of policy iterates in these methods has not been fully understood, bringing out issues such as violation of constraints and sensitivity to hyper-parameters. To fill this gap, we employ the Lagrangian method to cast a constrained MDP into a constrained saddle-point problem in which max/min players correspond to primal/dual variables, respectively, and develop two single-time-scale policy-based primal-dual algorithms with non-asymptotic convergence of their policy iterates to an optimal constrained policy. Specifically, we first propose a regularized policy gradient primal-dual (RPG-PD) method that updates the policy using an entropy-regularized policy gradient, and the dual variable via a quadratic-regularized gradient ascent, simultaneously. We prove that the policy primal-dual iterates of RPG-PD converge to a regularized saddle point with a sublinear rate, while the policy iterates converge sublinearly to an optimal constrained policy. We further instantiate RPG-PD in large state or action spaces by including function approximation in policy parametrization, and establish similar sublinear last-iterate policy convergence. Second, we propose an optimistic policy gradient primal-dual (OPG-PD) method that employs the optimistic gradient method to update primal/dual variables, simultaneously. We prove that the policy primal-dual iterates of OPG-PD converge to a saddle point that contains an optimal constrained policy, with a linear rate. To the best of our knowledge, this work appears to be the first non-asymptotic policy last-iterate convergence result for single-time-scale algorithms in constrained MDPs. We further validate the merits and the effectiveness of our methods in computational experiments.
Constrained Markov decision process (Constrained MDP) is the classical model for constrained dynamic systems in the early stochastic control literature (e.g., [1, 2, 3, 4, 5]) and the recent constrained reinforcement learning (RL) literature (e.g., [6, 7, 8, 9, 10, 11]). It is applicable to many constrained control problems by integrating other system specifications in constraints, and admits a natural extension of constrained optimization and Lagrangian in policy space. Lagrangian-based policy search methods, especially policy-based primal-dual methods that work simultaneously with
*Alphabetical order.
primal/dual variables, lie at the heart of recent successes of constrained MDPs, e.g., navigation [12], autonomous driving [13, 14], robotics [15], and finance [16]; see [17, 18, 19, 20] for more examples.
Despite the popularity of policy-based primal-dual algorithms, classical asymptotic convergence assumes that primal-dual updates are in two-time-scale $^1$  type [21, 6, 22, 16, 9] (and/or work in two nested loops $^2$ ), and considerable global non-asymptotic convergence guarantee is measured via an average of past objective/constraint functions [23, 24, 25, 26] or a mixture of past policies [27, 28]. These results are unfavorable in constrained dynamic systems, especially safety-critical ones, due to three reasons: (i) Average and mixture performance of non-asymptotic convergence conceals oscillating (or even overshooting) objective/constraint functions of immediate policy iterates [29, 30], and oscillation-incurred constraint violation impedes a policy iterate being optimal; (ii) Asymptotic convergence is not instructive, because arbitrarily slow convergence, and oscillation and overshoot in any finite time can happen; (iii) Two-time-scale algorithms including algorithms with nested loops are sensitive to hyper-parameters and are therefore typically difficult to tune [16, 9]. Thus, we ask the following question in constrained MDPs:
Can the policy iterates of a single-time-scale policy-based primal-dual algorithm converge to an optimal constrained policy with non-asymptotic rate?
By "single-time-scale", we refer to the classical methods [31, 32, 33] that iterate primal/dual variables concurrently (with the same constant stepsize). Only partial answers to this question are provided in recent studies [34, 35, 36] since they either do not work in the single-time-scale scheme or they do not have non-asymptotic convergence guarantees. In this work, we provide an affirmative answer in two methodologies. First, we initiate the design and analysis of single-time-scale policy-based primal-dual algorithms via regularization, while previous works [24, 28, 34] rely on two-time-scale schemes. Second, inspired by convex minimax optimization [37, 38, 39], we propose a new optimistic policy gradient for a single-time-scale policy-based primal-dual algorithm that solves a class of non-convex minimax problems. While preparing our work, we noticed a contemporaneous work [40], which has empirically validated the effectiveness of other optimistic methods in constrained MDPs, has further inspired the pursuit of our contributions, as outlined in detail below.
Contributions. To compute an optimal policy of an infinite-horizon discounted constrained MDP, we employ the Lagrangian method to cast it into a constrained saddle-point problem in which max/min players correspond to primal/dual variables, propose two single-time-scale policy-based primal-dual algorithms, and prove global non-asymptotic convergence of their policy iterates.
- Nearly dimension-free sublinear last-iterate policy convergence. We propose a regularized policy gradient primal-dual (RPG-PD) method that updates the policy using an entropy-regularized policy gradient, and the dual using a quadratic-regularized gradient ascent, simultaneously. We prove that the policy primal-dual iterates of RPG-PD converge to a regularized saddle point with a sublinear rate, and the policy iterates converge to an optimal constrained policy sublinearly.
- Sublinear last-iterate policy convergence with function approximation. We generalize RPG-PD for constrained MDPs with large state/action spaces by including function approximation in policy parametrization. We prove that the policy primal-dual iterates of an inexact RPG-PD converge to a regularized saddle point with a sublinear rate, but up to a function approximation error, and the policy iterates converge sublinearly to an optimal constrained policy when the error is small.
- Problem-dependent linear last-iterate policy convergence. We propose an optimistic policy gradient primal-dual (OPG-PD) method that employs the optimistic gradient method to update the primal/dual, simultaneously. We prove that the policy primal-dual iterates of OPG-PD converge to a saddle point that contains an optimal constrained policy with a problem-dependent linear rate.
While last-iterate convergence is of importance in its own right, by adding proper conservatism in the constraint, both methods can ensure no constraint violation for the last policy iterate, which perhaps is best for safety-critical tasks [10, 20]. As far as we know, this work shows the first non-asymptotic and policy last-iterate convergence for single-time-scale algorithms in the constrained MDP literature. We further exhibit the merits and the effectiveness of our methods in experiments.
1One update has relatively very large/small (fast/slow) stepsize than the other.
2We view an algorithm with two nested gradient-based loops as a two-time-scale algorithm.
Technical comparisons with prior art. Although global asymptotic last-iterate convergence has been established for single-time-scale algorithms very recently [36, 40], and value-average or policy-mixture non-asymptotic convergence have been established for other algorithms [23, 24, 25, 27, 28, 41, 42, 26], these studies did not investigate global non-asymptotic and last-iterate convergence for single-time-scale algorithms. Our results not only strengthen these prior guarantees, but also set up a new framework for analyzing policy-based primal-dual algorithms via the distance of primal-dual iterates to a saddle point that contains an optimal constrained policy. Our RPG-PD and OPG-PD keep the simplicity of single-time-scale primal-dual methods and output a nearly-optimal policy in the last iterate, which is more convenient than the history-average policies [28, 24] or the policies from subroutines [34, 35]. Compared with the policy-based methods [40], our OPG-PD is a projected policy gradient method that enjoys policy last-iterate convergence with linear rate. Compared with the constrained saddle-point problems [38, 39], our minimax optimization that results from constrained MDP is non-convex. Hence, our OPG-PD extends the last-iterate convergence guarantee from convex minimax optimization to a class of non-convex ones, while preserving a linear rate. Compared with the analysis in the two-player zero-sum Markov game [43, 44], there is no reduction from constrained MDPs to per-state bilinear games. Please see more details in Appendix A.
We consider an infinite-horizon discounted constrained Markov decision process [3, 5, 8] -  $\mathrm{CMDP}(S,A,P,r,u,b,\gamma,\rho)$  - where  $S$  and  $A$  are state/action spaces,  $P$  is a transition kernel that specifies the transition probability  $P(s^{\prime} \mid s,a)$  from state  $s$  to next state  $s^{\prime}$  under action  $a \in A$ ,  $r, u: S \times A \to [0,1]$  are reward/utility functions,  $b$  is a constraint threshold,  $\gamma \in [0,1)$  is a discount factor, and  $\rho$  is an initial state distribution. A stationary stochastic policy  $\pi: S \to \Delta(A)$  determines a probability distribution over the action space  $A$  based on current state, i.e.,  $a_{t} \sim \pi(\cdot \mid s_{t})$  at time  $t$ , where  $\Delta(A)$  is a probability simplex over  $A$ . Let  $\Pi$  be the set of all possible stochastic policies. A policy  $\pi \in \Pi$ , together with the initial state distribution  $\rho$ , induces a distribution over trajectories  $\tau = \{(s_{t},a_{t},r_{t},u_{t})\}_{t=0}^{\infty}$ , where  $s_{0} \sim \rho$ ,  $a_{t} \sim \pi(\cdot \mid s_{t})$ ,  $r_{t} = r(s_{t},a_{t})$ ,  $u_{t} = u(s_{t},a_{t})$  and  $s_{t+1} \sim P(\cdot \mid s_{t},a_{t})$  for all  $t \geq 0$ .
Given a policy  $\pi$ , the value functions  $V_r^\pi, V_u^\pi: S \to \mathbb{R}$  associated with the reward function  $r$  or the utility function  $u$  are given by the expected sums of discounted rewards or utilities under policy  $\pi$ :
$$
V _ {r} ^ {\pi} (s) := \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r (s _ {t}, a _ {t}) \mid s _ {0} = s \right] \text {a n d} V _ {u} ^ {\pi} (s) := \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} u (s _ {t}, a _ {t}) \mid s _ {0} = s \right]
$$
where the expectation  $\mathbb{E}$  is over the randomness of the trajectory  $\tau$  induced by  $\pi$ . Their expected values under  $\rho$  are  $V_r^\pi (\rho)\coloneqq \mathbb{E}_{s\sim \rho}[V_r^\pi (s)]$  and  $V_{u}^{\pi}(\rho)\coloneqq \mathbb{E}_{s\sim \rho}[V_{u}^{\pi}(s)]$ . It is useful to introduce the discounted state visitation distribution,  $d_{s_0}^\pi (s) = (1 - \gamma)\sum_{t = 0}^\infty \gamma^t\mathrm{Pr}(s_t = s\mid \pi ,s_0)$  which adds up discounted probabilities of visiting  $s$  in the execution of  $\pi$  starting from  $s_0$ . Denote  $d_{\rho}^{\pi}(s)\coloneqq \mathbb{E}_{s_0\sim \rho}[d_{s_0}^\pi (s)]$  and thus  $d_{\rho}^{\pi}(s)\geq (1 - \gamma)\rho (s)$  for any  $\rho$  and  $s$ . Furthermore, for the reward function  $r$ , we introduce the state-action value function  $Q_{r}^{\pi}\colon S\times A\to \mathbb{R}$  when the agent begins with a state-action pair  $(s,a)$  and follows a policy  $\pi$ , and the associated advantage function  $A_{r}^{\pi}\colon S\times A\to \mathbb{R}$ ,
$$
Q _ {r} ^ {\pi} (s, a) := \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, a _ {t}\right) \mid s _ {0} = s, a _ {0} = a \right] \text {a n d} A _ {r} ^ {\pi} (s, a) := Q _ {r} ^ {\pi} (s, a) - V _ {r} ^ {\pi} (s).
$$
Similarly, we define  $Q_u^\pi : S \times A \to \mathbb{R}$  and  $A_u^\pi : S \times A \to \mathbb{R}$  for the utility function  $u$ .
In this work, we aim to find a policy solution  $\pi^{\star}$  of a constrained policy optimization problem,
$$
\underset {\pi \in \Pi} {\text {m a x i m i z e}} V _ {r} ^ {\pi} (\rho) \quad \text {s u b j e c t t o} V _ {u} ^ {\pi} (\rho) \geq b \tag {1}
$$
where the objective is the reward value function  $V_{r}^{\pi}(\rho)$  and the constraint requires that the utility value function  $V_{u}^{\pi}(\rho)$  is above a given threshold  $b$ . For notational simplicity we assume a single constraint, but our algorithms are readily generalizable to the problems with multiple constraints, as well as our subsequent last-iterate convergence theory. Since  $V_{r}^{\pi}(\rho)$  and  $V_{u}^{\pi}(\rho) \in [0,1/(1-\gamma)]$ , we assume  $b \in (0,1/(1-\gamma)]$  to avoid trivial cases. Let  $g: S \times A \to [-1,1]$  be  $g \coloneqq u - (1-\gamma)b$ . We, equivalently, translate the constraint  $V_{u}^{\pi}(\rho) \geq b$  into  $V_{g}^{\pi}(\rho) \geq 0$  that is our focal constraint. The optimal constrained policy  $\pi^{\star}$  depends on the initial state distribution  $\rho$ ; see Appendix B.1.
By the method of Lagrange multipliers [45], we dualize the constraint in (1) and present a standard Lagrangian  $L(\pi, \lambda) \coloneqq V_r^\pi(\rho) + \lambda V_g^\pi(\rho)$ , where  $\pi \in \Pi$  is the primal variable and  $\lambda \in [0, \infty]$  is the dual variable or the Lagrangian multiplier. Introduction of the Lagrangian  $L(\pi, \lambda)$  interprets Problem (1) as a max-min problem: maximize  $\pi \in \Pi$  minimize  $\lambda \in [0, \infty]$ $L(\pi, \lambda)$ , and thus we can view the Lagrangian  $L(\pi, \lambda)$  as a value function with a composite function  $r + \lambda g$ ,
$$
\underset {\pi \in \Pi} {\text {m a x i m i z e}} \underset {\lambda \in [ 0, \infty ]} {\text {m i n i m i z e}} V _ {r + \lambda g} ^ {\pi} (\rho). \tag {2}
$$
However, it's defective to view Problem (2) as a standard MDP problem by fixing a dual variable  $\lambda$ , even the optimal one; also see [9, 46, 47, 30]. This is often referred to as the scalarization fallacy [46]; see Appendix B.2 for the detail. From the perspective of game theory, we instead view  $V_{r + \lambda g}^{\pi}(\rho)$ :  $\Pi \times [0,\infty] \to \mathbb{R}$  as a payoff function for a two-player zero-sum game in which max-player is the policy  $\pi \in \Pi$  and min-player is the dual variable  $\lambda \in [0,\infty]$ , and study its saddle points. To proceed, we assume feasibility for Problem (1) throughout our analysis.
Assumption 1 (Feasibility). There exists a policy  $\bar{\pi} \in \Pi$  and  $\xi > 0$  such that  $V_{g}^{\bar{\pi}}(\rho) \geq \xi$ .
Feasibility mirrors the Slater condition in the duality analysis of constrained optimization [45]. It can be verified by solving an unconstrained MDP problem with respect to  $V_{g}^{\pi}(\rho)$ .
A saddle point  $(\pi', \lambda')$  satisfies  $V_{r + \lambda'g}^{\pi}(\rho) \leq V_{r + \lambda'g}^{\pi'}(\rho) \leq V_{r + \lambda g}^{\pi'}(\rho)$  for all  $\pi \in \Pi$ ,  $\lambda \in [0, \infty]$ , or equivalently,  $\pi'$  is the max-min point, i.e.,  $\pi' \in \operatorname{argmax}_{\pi \in \Pi} V_{r + \lambda'g}^{\pi}(\rho)$  and  $\lambda'$  is the min-max point, i.e.,  $\lambda' \in \operatorname{argmin}_{\lambda \in [0, \infty]} V_{r + \lambda g}^{\pi'}(\rho)$ . To view Problem (2) as a saddle-point problem, we denote  $V_P^\pi(\rho) := \inf_{\lambda \in [0, \infty]} V_{r + \lambda g}^\pi(\rho)$  as the primal function which takes  $V_r^\pi(\rho)$  when  $V_g^\pi(\rho) \geq 0$  and  $-\infty$  otherwise, and  $V_D^\lambda(\rho) := \max_{\pi \in \Pi} V_{r + \lambda g}^\pi(\rho)$  as the dual function. Let an optimal dual variable be  $\lambda^* \in \operatorname{argmin}_{\lambda \in [0, \infty]} V_D^\lambda(\rho)$ . For Problem (1) under Assumption 1, strong duality holds in policy space [12, Theorem 3] and optimal dual variables are bounded [48, Lemma 3].
Lemma 1 (Strong duality/Saddle point existence and boundedness). Let Assumption 1 hold. Then, (i) strong duality holds for Problem (1), i.e.,  $V_{P}^{\pi^{\star}}(\rho) = V_{D}^{\lambda^{\star}}(\rho)$ ; (ii) optimal dual variables are bounded, i.e.,  $\lambda^{\star} \in [0, (V_r^{\pi^{\star}} - V_r^{\bar{\pi}}) / \xi]$ .
Let the set of max-min points be  $\Pi^{\star} := \operatorname{argmax}_{\pi \in \Pi} \min_{\lambda \in [0,\infty]} V_{r + \lambda g}^{\pi}(\rho)$  and the set of min-max points be  $\Lambda^{\star} := \operatorname{argmin}_{\lambda \in [0,\infty]} \max_{\pi \in \Pi} V_{r + \lambda g}^{\pi}(\rho)$ . From Lemma 1 (ii),  $\Lambda^{\star}$  is contained in a bounded interval  $\Lambda := [0,1 / ((1 - \gamma)\xi)]$ . Lemma 1 (i) shows that any pair  $(\pi^{\star},\lambda^{\star}) \in \Pi^{\star} \times \Lambda^{\star}$  solves the following constrained saddle-point problem,
$$
\underset {\pi \in \Pi} {\text {m a x i m i z e}} \underset {\lambda \in \Lambda} {\text {m i n i m i z e}} V _ {r + \lambda g} ^ {\pi} (\rho) = \underset {\lambda \in \Lambda} {\text {m i n i m i z e}} \underset {\pi \in \Pi} {\text {m a x i m i z e}} V _ {r + \lambda g} ^ {\pi} (\rho). \tag {3}
$$
Any saddle points associated with the set  $\Lambda^{\star}$  are captured by Problem (3) due to the invariance of saddle points, and searching for any pair  $(\pi^{\star},\lambda^{\star})\in \Pi^{\star}\times \Lambda^{\star}$  is sufficient by the interchangeability of saddle points; see Lemmas 8-9 in Appendix B.3 for the properties of saddle points. Thus, we view the policy (primal) as max-player and the dual as min-player in a zero-sum game.
Three structural properties from constrained MDPs distinguish Problem (3) from recent last-iterate convergence for learning in zero-sum games (e.g., [37, 38, 43, 39, 44]): (i) Two players are asymmetric. One plays a stochastic policy that affects the transition dynamics and the other selects an action in a continuous interval that only changes the payoff; (ii) Problem (3) is a non-convex game, because of the non-concavity of the payoff  $V_{r + \lambda g}^{\pi}(\rho)$  in policy  $\pi$  (e.g., [49, Lemma 1]); (iii) A saddle-point policy for Problem (3) cannot be uniformly max-min optimal, i.e., being optimal across all states, since an optimal policy often depends on the initial state distribution  $\rho$  in a constrained MDP; see Appendix B.1. Hence, known last-iterate results in zero-sum convex games or symmetric Markov games that admit uniformly optimal policies can't be applied and new techniques are required to address this non-standard saddle-point problem, which warrants our contributions in this work.
Warm-up: Indirect policy search in occupancy-measure space. Finding a saddle point of a nonconvex game is hard in general [50]. Nevertheless, Problem (1) can be rewritten as a linear program regarding the occupancy measure [5], which permits indirectly searching for a saddle point of a bilinear Lagrangian [51, 42, 36]. These asymptotic or average-iterate convergence results can be easily strengthened by applying last-iterate convergence results for bilinear games (e.g., [38, 39]) to be
non-asymptotic and last-iterate. By doing so, we state an optimistic primal-dual (OPD) method (18) in Appendix B.4. Compared with a contemporaneous work [40], OPD is free of projection to an occupancy measure set, and enjoys strengthened linear convergence.
OPD is an indirect policy search method that iterates using occupancy measure-based gradients, not policy-based gradients. It is crucial to develop direct policy search methods that are widely-used in RL, which is our focus. We propose two such methods in Section 3 and Section 4, respectively.
3 Policy Last-Iterate Convergence: Regularized Method
Towards achieving policy last-iterate convergence, a practical strategy is using regularization [52] to "convexify" Problem (3). We present a regularized method - Regularized Policy Gradient Primal-Dual (RPG-PD) - that converges to a saddle point that yields an optimal constrained policy.
3.1 Regularized policy gradient primal-dual method
We introduce a regularized Lagrangian  $L_{\tau}(\pi, \lambda) \coloneqq V_{r + \lambda g}^{\pi}(\rho) + \tau (\mathcal{H}(\pi) + \frac{1}{2}\lambda^{2})$  by adding a regularization term  $\mathcal{H}(\pi) + \frac{1}{2}\lambda^2$  onto the original Lagrangian  $V_{r + \lambda g}^{\pi}(\rho)$ , where  $\tau$  is a regularization parameter, and  $\mathcal{H}(\pi) \coloneqq \mathbb{E}[\sum_{t=0}^{\infty} -\gamma^{t}\log\pi(a_{t}|s_{t})]$  is an entropy-like regularization term [52]. We now introduce a regularized constrained saddle-point problem,
$$
\underset {\pi \in \Pi} {\text {m a x i m i z e}} \underset {\lambda \in \Lambda} {\text {m i n i m i z e}} L _ {\tau} (\pi , \lambda) = \underset {\lambda \in \Lambda} {\text {m i n i m i z e}} \underset {\pi \in \Pi} {\text {m a x i m i z e}} L _ {\tau} (\pi , \lambda). \tag {4}
$$
Problem (4) is well-defined, since there exists a saddle point for  $L_{\tau}(\pi ,\lambda)$  over  $\Pi \times \Lambda$  and it is unique; see Appendix C.1 for proof. A saddle point  $(\pi_{\tau}^{\star},\lambda_{\tau}^{\star})$ , i.e.,  $\pi_{\tau}^{\star} = \mathrm{argmax}_{\pi \in \Pi}\min_{\lambda \in \Lambda}L_{\tau}(\pi ,\lambda)$  and  $\lambda_{\tau}^{\star} = \mathrm{argmin}_{\lambda \in \Lambda}\max_{\pi \in \Pi}L_{\tau}(\pi ,\lambda)$ , satisfies a sandwich-like property,
$$
V _ {r + \lambda_ {\tau} ^ {\star} g} ^ {\pi} (\rho) - \tau \mathcal {H} \left(\pi_ {\tau} ^ {\star}\right) \leq V _ {r + \lambda_ {\tau} ^ {\star} g} ^ {\pi_ {\tau} ^ {\star}} (\rho) \leq V _ {r + \lambda g} ^ {\pi_ {\tau} ^ {\star}} (\rho) + \frac {\tau}{2} \lambda^ {2} \text {f o r a l l} (\pi , \lambda) \in \Pi \times \Lambda \tag {5}
$$
that states that  $(\pi_{\tau}^{\star},\lambda_{\tau}^{\star})$  is a saddle point of the original Lagrangian  $V_{r + \lambda g}^{\pi}(\rho)$ , up to two  $\tau$ -terms. We thus propose a regularized policy gradient primal-dual (RPG-PD) method by maintaining a sequence for policy and dual variables each:  $\{\pi_t\}_{t\geq 0}$  for the policy-player, and  $\{\lambda_t\}_{t\geq 0}$  for the dual-player,
$$
\pi_ {t + 1} (\cdot \mid s) = \underset {\pi (\cdot \mid s) \in \hat {\Delta} (A)} {\operatorname {a r g m a x}} \left\{\sum_ {a} \pi (a \mid s) Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {t}} (s, a) - \frac {1}{\eta} \mathrm {K L} (\pi (\cdot \mid s), \pi_ {t} (\cdot \mid s)) \right\} \tag {6a}
$$
$$
\lambda_ {t + 1} = \underset {\lambda \in \Lambda} {\operatorname {a r g m i n}} \left\{\lambda \left(V _ {g} ^ {\pi_ {t}} (\rho) + \tau \lambda_ {t}\right) + \frac {1}{2 \eta} \left(\lambda - \lambda_ {t}\right) ^ {2} \right\}, \tag {6b}
$$
where the gradient direction  $Q_{r + \lambda_t g + \tau \psi_t}^{\pi_t}(s, a)$  is the state-action value function under a composite function  $r + \lambda_t g + \tau \psi_t$  in which  $\psi_t(s, a) := -\log \pi_t(a|s)$ ,  $\mathrm{KL}(p, p') := \sum_a p_a \log \frac{p_a}{p'_a}$  is the Kullback-Leibler (KL) divergence,  $\hat{\Delta}(A) := \{\pi(\cdot|s) \in \Delta(A) | \pi(a|s) \geq \frac{\epsilon_0}{|A|}, a \in A\}$  is a restricted probability simplex set with parameter  $\epsilon_0 \in (0, 1)$ ,  $\eta$  is the stepsize, and  $(\pi_0(\cdot|s), \lambda_0) \in \hat{\Delta}(A) \times \Lambda$  is an initial point. Projecting the policy iterate to the simplex set  $\hat{\Delta}(A)$  ensures the boundedness of the gradient. Primal update (6a) works as the classical mirror descent with KL divergence [53] with a projection onto the set  $\hat{\Delta}(A)$ . Dual update (6b) performs typical projected gradient descent. Hence, RPG-PD is a single-time-scale method. RPG-PD simplifies the two-time-scale method [34] to be single-time-scale and generalize the single-time-scale methods [23, 54] with regularization.
3.2 Policy last-iterate convergence