SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this experiment, we continue our previous tabular constrained MDP with a random transition, a discount factor  $\gamma = 0.9$ , uniform rewards  $r \in [0,1]$  and utilities  $g \in [-1,1]$ , and an uniform initial state distribution  $\rho$ . Instead of the nominal constraint  $V_{g}^{\pi}(\rho) \geq 0$ , we use a conservative constraint  $V_{g'}^{\pi}(\rho) \geq 0$  in RPG-PD and OPG-PD, where  $g' := g - (1 - \gamma)\delta$  is a conservative utility function and  $\delta$  is the conservative parameter. To get zero constraint violation regarding the nominal constraint, we apply RPG-PD and OPG-PD to the conservative constraint  $V_{g}^{\pi}(\rho) \geq \delta$  where we take the conservative parameter  $\delta = 0.1$ . As above, we compare conservative RPG-PD and OPG-PD with three typical learning algorithms in the constrained MDP literature: (i) primal-dual methods [23, 29]; (ii) dual methods [28, 24, 34]; and (iii) primal approach [77]. We report our comparison for primal-dual methods in Figure 6, for dual methods [28, 24, 34] in Figure 7, and for primal approach [77] in Figure 8. We observe that conservative RPG-PD and OPG-PD achieve similar performance regarding the oscillation suppression and the optimality of reward values as shown in Figure 1, Figure 4, and Figure 5. Interestingly, in Figure 6, Figure 7, and Figure 8, RPG-PD and OPG-PD converge to a utility value that is strictly above zero, i.e.,  $V_{g}^{\pi t}(\rho) > 0$  for large  $t$ , which is not guaranteed in many of other methods. To sum up, we have confirmed that RPG-PD and OPG-PD can ensure zero constraint violation of instantaneous policy iterates in a finite number of training time.
Figure 6: Convergence performance of RPG-PD, OPG-PD, and primal-dual methods. Learning curves of our RPG-PD  $(- - )$  and OPG-PD  $(-)$ , and NPG-PD [23]  $(- - )$  and PID Lagrangian [29]  $(\dots)$  methods. The horizontal axes represent the policy iterations  $\{\pi_t\}_{t\geq 0}$  that are generated by each method and the vertical axes mean the value functions of the policy iterates  $\{\pi_t\}_{t\geq 0}$ : reward value  $V_{r}^{\pi_{t}}(\rho)$  (Left) and utility value  $V_{g}^{\pi_{t}}(\rho)$  (Right). In this experiment, we apply RPG-PD and OPG-PD to a conservative constraint  $V_{g}^{\pi}(\rho)\geq \delta$ , and we take the conservative parameter  $\delta = 0.1$ , the same stepsize  $\eta = 0.1$  for all methods, the regularization parameter  $\tau = 0.08$  for RPG-PD, and the uniform initial distribution  $\rho$ .
E.3 Sensitivity of RPG-PD (6) to regularization and stepsize
In this experiment, we use our previous tabular constrained MDP with a random transition, a discount factor  $\gamma = 0.9$ , uniform rewards  $r \in [0,1]$  and utilities  $g \in [-1,1]$ , and an uniform initial state distribution  $\rho$ . The constraint is  $V_g^\pi(\rho) \geq 0$ . We recall that when we set the regularization parameter  $\tau = 0$ , RPG-PD becomes a policy-based primal-dual method [23] that often suffers the oscillation issue as shown in Figure 1. However, larger regularization usually bias regularized methods more to sub-optimal solutions. Hence, it is important to reveal how the last-iterate convergence of RPG-PD depends on the regularization parameter  $\tau$  and the stepsize  $\eta$ .
We first repeat executing RPG-PD with a fixed stepsize  $\eta = 0.1$ , but varying three different regularization parameters  $\tau \in \{0.1, 0.05, 0.01\}$ . In Figure 9, RPG-PD damps initial oscillations successfully for  $\tau = 0.1$  and 0.05, and oscillates for  $\tau = 0.01$ . When we decrease  $\tau$  from 0.1 to 0.05, the reward value RPG-PD converges to become higher, and the utility value's oscillation is slightly amplified initially, but damped eventually. When  $\tau$  is further reduced to 0.01, although the reward value reaches a value around the optimal reward value 8.16, the utility value behaves oscillating.
Figure 7: Convergence performance of RPG-PD, OPG-PD, and dual methods. Learning curves of our RPG-PD  $(- - )$  and OPG-PD  $(- - )$ , and PMD-PD [24]  $(- - )$ , AR-CPO [28]  $(- - )$ , and Accelerated Dual [34]  $(\dots)$  methods. The horizontal axes represent the policy iterations  $\{\pi_t\}_{t\geq 0}$  that are generated by each method and the vertical axes mean the value functions of the policy iterates  $\{\pi_t\}_{t\geq 0}$ : reward value  $V_{r}^{\pi_{t}}(\rho)$  (Left) and utility value  $V_{g}^{\pi_{t}}(\rho)$  (Right). In this experiment, we apply RPG-PD and OPG-PD to a conservative constraint  $\tilde{V}_g^\pi (\rho)\geq \delta$ , and we take the stepsize  $\eta = 0.1$ , the conservative parameter  $\delta = 0.1$ , the regularization parameter  $\tau = 0.08$  for RPG-PD, and the initial distribution  $\rho$  is uniform. For PMD-PD, AR-CPO, and Accelerated Dual, we use the stepsize  $\eta = 0.1$  for the dual update, the regularized NPG stepsize  $\alpha = 1$ , and the regularization parameter  $\tau = 0.08$ , and the uniform initial distribution  $\rho$ .
Figure 8: Convergence performance of RPG-PD, OPG-PD, and primal methods. Learning curves of our RPG-PD  $(- - )$  and OPG-PD  $(-)$ , and CRPO [77]  $(- - )$  methods. The horizontal axes represent the policy iterations  $\{\pi_t\}_{t\geq 0}$  that are generated by each method and the vertical axes mean the value functions of the policy iterates  $\{\pi_t\}_{t\geq 0}$ : reward value  $V_{r}^{\pi_{t}}(\rho)$  (Left) and utility value  $V_{g}^{\pi_{t}}(\rho)$  (Right). In this experiment, we apply RPG-PD and OPG-PD to a conservative constraint  $V_{g}^{\pi}(\rho)\geq \delta$  and we take the conservative parameter  $\delta = 0.1$ , the stepsize  $\eta = 0.1$ , the regularization parameter  $\tau = 0.08$  for RPG-PD, and the initial distribution  $\rho$  is uniform. For CRPO, we update the policy via the NPG step with stepsize  $\eta = 0.1$  and the uniform initial distribution  $\rho$ .
A reason for this is that RPG-PD with a relatively small regularization parameter (compared to the stepsize) works as usual un-regularized single-time-scale primal-dual methods. Hence, increasing the regularization parameter,  $\tau = 0.1$  can accelerate the convergence and attenuate the oscillation more effectively, although it makes the reward value more sub-optimal, which is also clearly shown in Figure 10.
To demonstrate the convergence of RPG-PD's policy iterates, we measure the policy optimality gap via the squared norm distance of policy iterates to an optimal policy  $\pi^{\star}$  that is obtained from an occupancy-measure-based linear program. From the optimality gap in Figure 10, we see that three policy optimality gaps decrease sublinearly to some constants in the logarithmic scale plot, which verifies the sublinear last-iterate convergence of RPG-PD's policy iterates in Theorem 2. Hence, we
conjecture that it is impossible to improve the order of RPG-PD's sublinear rate, without introducing new algorithmic design.
To reduce the oscillation behavior, we repeat this experiment with a smaller stepsize  $\eta = 0.01$  as suggested by Corollary 3. We report our result in Figure 11 and Figure 12. The oscillation in the utility value previously happened for  $\tau = 0.01$  becomes less frequent, and the best reward value is achieved in this case. A noticeable loss is that the convergence has been slowed down, apparently in Figure 12. Hence, we have shown that balancing the stepsize and the regularization parameter (e.g., Corollary 3) is important for RPG-PD to achieve better oscillation attenuation and last-iterate convergence in practice.
Figure 9: Convergence performance of RPG-PD with regularization parameter  $\tau$ :  $(\tau = 0.1, -\infty)$ ,  $(\tau = 0.05, -\infty)$ ,  $(\tau = 0.01, \dots)$ . The horizontal axes represent the policy iterations  $\{\pi_t\}_{t \geq 0}$  that are generated by RPG-PD and the vertical axes mean the value functions of the policy iterates  $\{\pi_t\}_{t \geq 0}$ : reward value  $V_r^{\pi_t}(\rho)$  (Left) and utility value  $V_g^{\pi_t}(\rho)$  (Right). In this experiment, we fix the same stepsize  $\eta = 0.1$  and take the regularization parameter  $\tau$  among 0.1, 0.05, and 0.01, and the uniform initial distribution  $\rho$ .
Figure 10: Convergence performance of RPG-PD with regularization parameter  $\tau$ :  $(\tau = 0.1, -)$ ,  $(\tau = 0.05, -)$ ,  $(\tau = 0.01, \dots)$ . The horizontal axis represents the policy iterations  $\{\pi_t\}_{t \geq 0}$  that are generated by RPG-PD and the vertical axis means the policy optimality gap that measures the distance of the policy iterates  $\{\pi_t\}_{t \geq 0}$  to an optimal policy  $\pi^{\star}$ :  $\sum_{s} \| \pi_t(\cdot | s) - \pi^{\star}(\cdot | s) \|^2$ . In this experiment, we fix the stepsize  $\eta = 0.1$  and take the regularization parameter among 0.1, 0.05, and 0.01, and the uniform initial distribution  $\rho$ .
Figure 11: Convergence performance of RPG-PD with regularization parameter  $\tau$ :  $(\tau = 0.1, -)$ ,  $(\tau = 0.05, -)$ ,  $(\tau = 0.01, \dots)$ . The horizontal axes represent the policy iterations  $\{\pi_t\}_{t \geq 0}$  that are generated by RPG-PD and the vertical axes mean the value functions of the policy iterates  $\{\pi_t\}_{t \geq 0}$ : reward value  $V_r^{\pi_t}(\rho)$  (Left) and utility value  $V_g^{\pi_t}(\rho)$  (Right). In this experiment, we fix the same stepsize  $\eta = 0.01$  and take the regularization parameter  $\tau$  among 0.1, 0.05, and 0.01, and the uniform initial distribution  $\rho$ .
Figure 12: Convergence performance of RPG-PD with regularization parameter  $\tau$ :  $(\tau = 0.1, -)$ ,  $(\tau = 0.05, -)$ ,  $(\tau = 0.01, \dots)$ . The horizontal axis represents the policy iterations  $\{\pi_t\}_{t \geq 0}$  that are generated by RPG-PD and the vertical axis means the policy optimality gap that measures the distance of the policy iterates  $\{\pi_t\}_{t \geq 0}$  to an optimal policy  $\pi^{\star}$ :  $\sum_s \| \pi_t(\cdot | s) - \pi^{\star}(\cdot | s) \|^2$ . In this experiment, we fix the stepsize  $\eta = 0.01$  and take the regularization parameter among 0.1, 0.05, and 0.01, and the uniform initial distribution  $\rho$ .
E.4 Sensitivity of OPG-PD (9) to stepsize
In this experiment, we use our previous tabular constrained MDP with a random transition, a discount factor  $\gamma = 0.9$ , uniform rewards  $r \in [0,1]$  and utilities  $g \in [-1,1]$ , and an uniform initial state distribution  $\rho$ . The constraint is  $V_g^\pi(\rho) \geq 0$ . We repeat executing OPG-PD by varying three different step sizes  $\eta \in \{0.05, 0.1, 0.2\}$ . To demonstrate the optimality of OPG-PD's policy iterates, we measure the policy optimality gap via the squared norm distance of policy iterates to an optimal policy  $\pi^{\star}$  that is obtained from an occupancy-measure-based linear program. To demonstrate the optimality of OPG-PD's policy iterates, we measure the policy optimality gap via the squared norm distance of policy iterates to an optimal policy  $\pi^{\star}$  that is obtained from an occupancy-measure-based linear program. From the policy optimality gap in Figure 2, we see that three policy optimality gaps decrease linearly in the logarithmic scale plot, which verifies the linear last-iterate convergence of
OPG-PD's policy iterates in Theorem 6. Furthermore, in Figure 13 we show the optimality of OPG-PD's policy iterates by plotting the optimality gap  $|V_r^{\pi^*}(\rho) - V_r^{\pi_t}(\rho)|$  and the constraint violation  $|V_g^{\pi_t}(\rho)|$ , where  $V_r^{\pi^*}(\rho) = 8.16$ . We see that the optimality gap and the constraint violation both decay asymptotically in linear rates in spite of some oscillations, and larger stepsize enjoys faster convergence. It is worth mentioning that, these descending oscillations actually show that value functions are approaching the optimal ones. Hence, we have verified the linear last-iterate convergence of OPG-PD's policy iterates in Theorem 6 using a range of constant step sizes.
Figure 13: Convergence performance of OPG-PD with stepsize  $\eta$ :  $(\eta = 0.05, \dots)$ ,  $(\eta = 0.1, -\infty)$ ,  $(\eta = 0.2, -\infty)$ . The horizontal axes represent the policy iterations  $\{\pi_t\}_{t \geq 0}$  that are generated by OPG-PD and the vertical axes mean the optimality of the policy iterates  $\{\pi_t\}_{t \geq 0}$ : optimality gap of reward value  $|V_r^{\pi^*}(\rho) - V_r^{\pi_t}(\rho)|$  (Left) and constraint violation of utility value  $|V_g^{\pi_t}(\rho)|$  (Right). In this experiment, we take the stepsize  $\eta$  among 0.05, 0.1, and 0.2, and the uniform initial distribution  $\rho$ .
Last but not least, we extend this experiment for a variant of OPG-PD that is based on the multiplicative weights update (MWU). We call this variant as an optimistic multiplicative weights update primal-dual (OMWU-PD) method which maintains two sequences for policy and dual variables each:  $\{\pi_t\}_{t\geq 1}$  and  $\{\hat{\pi}_t\}_{t\geq 1}$  for the policy-player, and  $\{\lambda_t\}_{t\geq 1}$  and  $\{\hat{\lambda}_t\}_{t\geq 1}$  for the dual-player,
$$
\pi_ {t} (\cdot \mid s) = \underset {\pi (\cdot \mid s) \in \Delta (A)} {\operatorname {a r g m a x}} \left\{\sum_ {a} \pi (a \mid s) Q _ {r + \lambda_ {t - 1} g} ^ {\pi_ {t - 1}} (s, a) - \frac {1}{\eta} \mathrm {K L} \left(\pi (\cdot \mid s), \hat {\pi} _ {t} (\cdot \mid s)\right) \right\} \tag {41a}
$$
$$
\hat {\pi} _ {t + 1} (\cdot | s) = \underset {\pi (\cdot | s) \in \Delta (A)} {\operatorname {a r g m a x}} \left\{\sum_ {a} \pi (a | s) Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, a) - \frac {1}{\eta} \mathrm {K L} (\pi (\cdot | s), \hat {\pi} _ {t} (\cdot | s)) \right\}
$$
$$
\lambda_ {t} = \underset {\lambda \in \Lambda} {\operatorname {a r g m i n}} \left\{\lambda V _ {g} ^ {\pi_ {t - 1}} (\rho) + \frac {1}{2 \eta} \left(\lambda - \hat {\lambda} _ {t}\right) ^ {2} \right\} \tag {41b}
$$
$$
\hat {\lambda} _ {t + 1} = \underset {\lambda \in \Lambda} {\operatorname {a r g m i n}} \left\{\lambda V _ {g} ^ {\pi_ {t}} (\rho) + \frac {1}{2 \eta} (\lambda - \hat {\lambda} _ {t}) ^ {2} \right\}
$$
where  $\eta$  is the stepsize and  $(\hat{\pi}_0,\hat{\lambda}_0) = (\pi_0,\lambda_0)\in \Pi \times \Lambda$  is the initial point. OMWU-PD concurrently works with two primal iterates and two dual iterates, which is similar to OPG-PD. The only difference is that Primal update (41a) works as the NPG or MWU updates [23, 54], instead of the projected  $Q$ -ascent [58, 65], which is also different from the one-step optimistic MWU in policy-based ReLOAD [40].
To further investigate the applicability of optimistic gradient methods, we execute OMWU-PD in the same setting and report our result in Figure 14 and Figure 15. We see that the policy optimality gaps decay sublinearly in Figure 14, and the optimality gap and the constraint violation asymptotically decay in sulinear rates in Figure 15. As a comparison, we repeat the same experiment for policy-based ReLOAD [40], which is different from our OMWU-PD in using one-step optimistic gradient updates. In Figure 16 and Figure 17, we observe sublinear decay of policy optimality gap, optimality gap and constraint violation, where are similar as shown in Figure 14 and Figure 15.
The empirical results from two MWU-based optimistic primal-dual algorithms are suggestive. First, MWU-based optimistic algorithms can also have policy last-iterate convergence to an optimal policy. Second, convergence rates of MWU-based optimistic algorithms look slower than OPG-PD's under the same constant stepsize, which we leave as an immediate future work to establish sublinear convergence rates for MWU-based optimistic primal-dual algorithms.
Figure 14: Convergence performance of OMWU-PD with stepsize  $\eta$ :  $(\eta = 0.05, \dots)$ ,  $(\eta = 0.1, -\infty)$ ,  $(\eta = 0.2, -\infty)$ . The horizontal axis represents the policy iterations  $\{\pi_t\}_{t \geq 0}$  that are generated by OMWU-PD and the vertical axis means the policy optimality gap that measures the distance of the policy iterates  $\{\pi_t\}_{t \geq 0}$  to an optimal policy  $\pi^{\star}$ :  $\sum_{s} \mathrm{KL}(\pi^{\star}(\cdot | s), \pi_t(\cdot | s))$ . In this experiment, we take the stepsize  $\eta$  among 0.05, 0.1, and 0.2, and the uniform initial distribution  $\rho$ .
Figure 15: Convergence performance of OMWU-PD with stepsize  $\eta$ :  $(\eta = 0.05, \dots)$ ,  $(\eta = 0.1, \dots)$ ,  $(\eta = 0.2, \dots)$ . The horizontal axes represent the policy iterations  $\{\pi_t\}_{t \geq 0}$  that are generated by OMWU-PD and the vertical axes mean the optimality of the policy iterates  $\{\pi_t\}_{t \geq 0}$ : optimality gap of reward value  $|V_r^{\pi^*}(\rho) - V_r^{\pi_t}(\rho)|$  (Left) and constraint violation of utility value  $|V_g^{\pi_t}(\rho)|$  (Right). In this experiment, we take the stepsize  $\eta$  among 0.05, 0.1, and 0.2, and the uniform initial distribution  $\rho$ .
F Supporting Lemmas
In this section, we collect some lemmas that are helpful to our analysis.
Figure 16: Convergence performance of policy-based ReLOAD [40] with stepsize  $\eta$ :  $(\eta = 0.05, \dots)$ ,  $(\eta = 0.1, -\infty)$ ,  $(\eta = 0.2, \infty)$ . The horizontal axis represents the policy iterations  $\{\pi_t\}_{t \geq 0}$  that are generated by ReLOAD and the vertical axis means the policy optimality gap that measures the distance of the policy iterates  $\{\pi_t\}_{t \geq 0}$  to an optimal policy  $\pi^{\star}$ :  $\sum_{s} \mathrm{KL}(\pi^{\star}(\cdot | s), \pi_t(\cdot | s))$ . In this experiment, we take the stepsize  $\eta$  among 0.05, 0.1, and 0.2, and the uniform initial distribution  $\rho$ .
Figure 17: Convergence performance of policy-based ReLOAD [40] with stepsize  $\eta$ :  $(\eta = 0.05, \dots)$ ,  $(\eta = 0.1, -\infty)$ ,  $(\eta = 0.2, \infty)$ . The horizontal axes represent the policy iterations  $\{\pi_t\}_{t \geq 0}$  that are generated by ReLOAD and the vertical axes mean the optimality of the policy iterates  $\{\pi_t\}_{t \geq 0}$ : optimality gap of reward value  $|V_r^{\pi^*}(\rho) - V_r^{\pi_t}(\rho)|$  (Left) and constraint violation of utility value  $|V_q^{\pi_t}(\rho)|$  (Right). In this experiment, we take the stepsize  $\eta$  among 0.05, 0.1, and 0.2, and the uniform initial distribution  $\rho$ .
F.1 Lemmas in optimization
For any convex differentiable function  $\psi \colon X \to \mathbb{R}$ , the Bregman divergence of  $x$ ,  $x' \in X$  is given by  $D_{\psi}(x', x) \coloneqq \psi(x') - \psi(x) - \langle \nabla \psi(x), x' - x \rangle$ . When  $\psi$  is  $\sigma$ -strongly convex,  $D_{\psi}(x', x) \geq \frac{\sigma}{2} \|x' - x\|^2$  for any  $x'$ ,  $x \in X$ . Specifically, when  $\psi(x) = \frac{1}{2} \|x\|^2$ ,  $D_{\psi}(x', x) = \frac{1}{2} \|x' - x\|^2$ .
Lemma 24. Let  $X$  be a convex set. If  $x' = \operatorname{argmin}_{\bar{x} \in X} \langle \bar{x}, g \rangle + D_{\psi}(\bar{x}, x)$ , then for any  $x^{\star} \in X$ ,
$$
\langle x ^ {\prime} - x ^ {\star}, g \rangle \leq D _ {\psi} (x ^ {\star}, x) - D _ {\psi} (x ^ {\star}, x ^ {\prime}) - D _ {\psi} (x ^ {\prime}, x).
$$
Proof. See [38, Lemma 10].
Lemma 25. Assume that  $D_{\psi}(x,x^{\prime}) \geq \frac{1}{2}\| x - x^{\prime}\|_{p}^{2}$  for some  $\psi$  and  $p \geq 1$ . If
$$
x _ {1} = \underset {\bar {x} \in X} {\operatorname {a r g m i n}} \left\langle \bar {x}, g _ {1} \right\rangle + D _ {\psi} (\bar {x}, x) \text {a n d} x _ {2} = \underset {\bar {x} \in X} {\operatorname {a r g m i n}} \left\langle \bar {x}, g _ {2} \right\rangle + D _ {\psi} (\bar {x}, x)
$$
$$
\left\| x _ {1} - x _ {2} \right\| _ {p} \leq \left\| g _ {1} - g _ {2} \right\| _ {q}
$$
where  $\frac{1}{p} +\frac{1}{q} = 1$
Proof. See [38, Lemma 11].
Lemma 26. Let  $X \subset \mathbb{R}^m$  and  $Y \subset \mathbb{R}^n$  be polytopes and  $M \in \mathbb{R}^{m \times n}$  be a matrix. Then, there exists a problem-dependent constant  $c > 0$  such that
$$
\max _ {y ^ {\prime} \in Y} x ^ {\top} M y ^ {\prime} - V ^ {\star} \geq c \| x - \mathcal {P} _ {X ^ {\star}} (x) \|
$$
$$
V ^ {\star} - \min  _ {x ^ {\prime} \in X} \left(x ^ {\prime}\right) ^ {\top} M y \geq c \| y - \mathcal {P} _ {Y ^ {\star}} (y) \|
$$
where  $V^{\star}$  is the game value,
$$
V ^ {\star} := \underset {x \in X} {\text {m i n i m i z e}} \underset {y \in Y} {\text {m a x i m i z e}}   x ^ {\top} M y = \underset {y \in Y} {\text {m a x i m i z e}} \underset {x \in X} {\text {m i n i m i z e}}   x ^ {\top} M y
$$
and  $(X^{\star},Y^{\star})$  is the set of minimax optimal strategies.
Proof. See [38, Theorem 5].
Lemma 27. Let  $X \subseteq \Delta(A)$  be a convex set and  $g$  be a bounded vector in  $\mathbb{R}^{|A|}$ . If  $x' = \operatorname*{argmin}_{\bar{x} \in X} \langle \bar{x}, g \rangle + \frac{1}{\eta} KL(\bar{x}, x)$ , then for any  $x^{\star} \in X$ ,
$$
\langle x - x ^ {\star}, g \rangle \leq \frac {K L \left(x ^ {\star} , x\right) - K L \left(x ^ {\star} , x ^ {\prime}\right)}{\eta} + \eta \sum_ {a \in A} x _ {a} \left(g _ {a}\right) ^ {2}
$$
where  $\eta$  satisfies  $\eta g_{a} \geq -1$  for  $a \in A$ .
Proof. See [132, Theorem 2].
F.2 Properties of policy gradient
Lemma 28 (Performance difference lemma). For any two policies  $\pi$  and  $\pi'$ , and any state  $s_0$ ,
$$
V ^ {\pi} (s _ {0}) - V ^ {\pi^ {\prime}} (s _ {0}) = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {s _ {0}} ^ {\pi}} \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ A ^ {\pi^ {\prime}} (s, a) \right]
$$
Proof. See [49, Lemma 3.2].
Lemma 29 (Regularized PG and NPG under softmax parametrization). Let an entropy-regularized value function be  $V_{\tau}^{\pi}(\rho) \coloneqq V_r^{\pi}(\rho) + \tau \mathcal{H}(\pi)$ , and define  $Q_{\tau}^{\pi} \colon S \times A \to \mathbb{R}$  and  $V_{\tau}^{\pi} \colon S \to \mathbb{R}$  via Bellman equations,
$$
V _ {\tau} ^ {\pi} (s) = \mathbb {E} _ {a \sim \pi (\cdot , | s)} [ - \tau \log \pi (a | s) + Q _ {\tau} ^ {\pi} (s, a) ].
$$
Let a parametrized policy be  $\pi_{\theta}$  for some parameter  $\theta \in \mathbb{R}^d$ . If the policy  $\pi_{\theta}$  is differentiable and  $\sum_{a}\pi_{\theta}(a|s) = 1$ , then,
$$
\begin{array}{l} \frac {\partial V _ {\tau} ^ {\pi_ {\theta}} (\rho)}{\partial \theta_ {s , a}} = \frac {1}{1 - \gamma} d _ {\rho} ^ {\pi_ {\theta}} (s) \cdot \pi_ {\theta} (a \mid s) \cdot A _ {\tau} ^ {\pi_ {\theta}} (s, a) \\ = \frac {1}{1 - \gamma} d _ {\rho} ^ {\pi_ {\theta}} (s) \cdot \pi_ {\theta} (a \mid s) \cdot \left(Q _ {\tau} ^ {\pi_ {\theta}} (s, a) - \tau \log \pi_ {\theta} (a \mid s)\right) \\ \end{array}
$$
for all  $(s,a)\in S\times A$ , where  $A_{\tau}^{\pi_{\theta}}(s,a)\coloneqq Q_{\tau}^{\pi_{\theta}}(s,a) - \tau \log \pi_{\theta}(a|s) - V_{\tau}^{\pi_{\theta}}(s)$ . Moreover, if the policy  $\pi_{\theta}$  is in form of the softmax function  $\pi_{\theta}(a|s) = \frac{\exp(\theta_{s,a})}{\sum_{a'}\exp(\theta_{s,a'})}$  for all  $(s,a)\in S\times A$ , then
$$
\left[ \left(F _ {\rho} (\theta)\right) ^ {\dagger} \cdot \nabla_ {\theta} V _ {\tau} ^ {\pi_ {\theta}} (\rho) \right] _ {s, a} = \frac {1}{1 - \gamma} A _ {\tau} ^ {\pi_ {\theta}} (s, a) + c (s)
$$
where  $c(s)$  is an action-independent constant.
Proof. See [52, Lemma 6].