SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} f (y) \geq \langle x, y ^ {*} - y ^ {\prime} \rangle + \frac {1}{c} \left[ - c \langle x, \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] \rangle - \phi^ {*} ([ \nabla \phi (y ^ {\prime}) - c x ]) - \phi (y ^ {\prime}) + \langle \nabla \phi (y ^ {\prime}), y ^ {\prime} \rangle \right] \\ = \langle x, y ^ {*} - y ^ {\prime} \rangle - \langle x, \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] \rangle - \frac {1}{c} \left[ \phi^ {*} ([ \nabla \phi (y ^ {\prime}) - c x ]) + \phi (y ^ {\prime}) - \langle \phi (y ^ {\prime}), y \rangle \right] \\ = - \langle x, y ^ {\prime} \rangle + \langle x, \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] \rangle - \langle x, \nabla \phi^ {*} [ \nabla \phi (y ^ {\prime}) - c x ] \rangle - \frac {1}{c} \left[ \phi^ {*} ([ \nabla \phi (y ^ {\prime}) - c x ]) + \phi (y ^ {\prime}) - \langle \nabla \phi (y ^ {\prime}), y ^ {\prime} \rangle \right] \\ = - \langle x, y ^ {\prime} \rangle - \frac {1}{c} \left[ \phi^ {*} \left(\left[ \nabla \phi \left(y ^ {\prime}\right) - c x \right]\right) + \phi \left(y ^ {\prime}\right) - \left\langle \nabla \phi \left(y ^ {\prime}\right), y ^ {\prime} \right\rangle \right] \\ \end{array}
$$
$$
\begin{array}{l} \text {U s i n g} (4), \phi \left(y ^ {\prime}\right) = \left\langle \nabla \phi \left(y ^ {\prime}\right), y ^ {\prime} \right\rangle - \phi^ {*} \left(\nabla \phi \left(y ^ {\prime}\right)\right) \Longrightarrow \phi \left(y ^ {\prime}\right) - \left\langle \nabla \phi \left(y ^ {\prime}\right), y ^ {\prime} \right\rangle = - \phi^ {*} \left(\nabla \phi \left(y ^ {\prime}\right)\right), \\ \Rightarrow f (y) \geq - \langle x, y ^ {\prime} \rangle - \frac {1}{c} \left[ \phi^ {*} \left(\left[ \nabla \phi \left(y ^ {\prime}\right) - c x \right]\right) - \phi^ {*} \left(\nabla \phi \left(y ^ {\prime}\right)\right) \right] = - \frac {1}{c} \left[ c \langle x, y \rangle + \phi^ {*} \left(\left[ \nabla \phi \left(y ^ {\prime}\right) - c x \right]\right) - \phi^ {*} \left(\nabla \phi \left(y ^ {\prime}\right)\right) \right] \\ = - \frac {1}{c} \left[ c \langle x, y ^ {\prime} \rangle + \left[ \phi^ {*} \left(\left[ \nabla \phi \left(y ^ {\prime}\right) - c x \right]\right) - \phi^ {*} \left(\nabla \phi \left(y ^ {\prime}\right)\right) - \left\langle \nabla \phi^ {*} \left(\nabla \phi \left(y ^ {\prime}\right)\right), \nabla \phi \left(y ^ {\prime}\right) - c x - \nabla \phi \left(y ^ {\prime}\right) \right\rangle \right] \right. \\ \left. + \langle \nabla \phi^ {*} (\nabla \phi (y ^ {\prime})), \nabla \phi (y ^ {\prime}) - c x - \nabla \phi (y ^ {\prime}) \rangle \right] \\ \Rightarrow f (y) \geq - \frac {1}{c} \left[ c \langle x, y ^ {\prime} \rangle + D _ {\Phi} ^ {*} (\nabla \phi (y ^ {\prime}) - c x, \nabla \phi (y ^ {\prime})) + \langle y ^ {\prime}, - c x \rangle \right] = - \frac {1}{c} D _ {\Phi} ^ {*} (\nabla \phi (y ^ {\prime}) - c x, \nabla \phi (y ^ {\prime})) \\ \end{array}
$$
Using the definition of  $f(y)$
$$
\begin{array}{l} \langle x, y - y ^ {\prime} \rangle + \frac {1}{c} D _ {\Phi} (y, y ^ {\prime}) \geq - \frac {1}{c} D _ {\Phi} ^ {*} (\nabla \phi (y ^ {\prime}) - c x, \nabla \phi (y ^ {\prime})) \\ \Longrightarrow \langle x, y - y ^ {\prime} \rangle \geq - \frac {1}{c} \left[ D _ {\Phi} \left(y, y ^ {\prime}\right) + D _ {\Phi} ^ {*} \left(\nabla \phi \left(y ^ {\prime}\right) - c x, \nabla \phi \left(y ^ {\prime}\right)\right) \right] \\ \end{array}
$$
Lemma 2. If  $J + \frac{1}{\eta}\Phi$  is convex, then,  $J(\pi)$  is  $\frac{1}{\eta}$ -relatively smooth w.r.t to  $D_{\Phi}$ , and satisfies the following inequality,
$$
J (\pi) \geq J (\pi^ {\prime}) + \left\langle \nabla_ {\pi} J (\pi^ {\prime}), \pi - \pi^ {\prime} \right\rangle - \frac {1}{\eta} D _ {\Phi} (\pi , \pi^ {\prime})
$$
Proof. If  $J + \frac{1}{\eta}\Phi$  is convex,
$$
\begin{array}{l} \left(J + \frac {1}{\eta} \phi\right) (\pi) \geq \left(J + \frac {1}{\eta} \phi\right) \left(\pi^ {\prime}\right) + \left\langle \pi - \pi^ {\prime}, \nabla_ {\pi} \left(J + \frac {1}{\eta} \phi\right) \left(\pi^ {\prime}\right) \right\rangle \\ \Rightarrow J (\pi) \geq J \left(\pi^ {\prime}\right) + \left\langle \pi - \pi^ {\prime}, \nabla_ {\pi} J \left(\pi^ {\prime}\right) \right\rangle - \frac {1}{\eta} \left[ \phi (\pi) - \phi \left(\pi^ {\prime}\right) - \left\langle \nabla_ {\pi} \phi \left(\pi^ {\prime}\right), \pi - \pi^ {\prime} \right\rangle \right] \\ \Longrightarrow J (\pi) \geq J \left(\pi^ {\prime}\right) + \left\langle \pi - \pi^ {\prime}, \nabla_ {\pi} J \left(\pi^ {\prime}\right) \right\rangle - \frac {1}{\eta} D _ {\Phi} \left(\pi , \pi^ {\prime}\right) \\ \end{array}
$$
D Proofs for Sec. 4
Proof. As a warmup, let us first consider the tabular parameterization where  $\pi(\theta) = \theta \in \mathbb{R}^{SA}$ . In this case, the lower-bound in Prop. 1 is equal to,
$$
J (\pi) - J (\pi_ {t}) \geq \langle \hat {g} _ {t}, \theta - \theta_ {t} \rangle - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} (\theta , \theta_ {t}) - \frac {1}{c} D _ {\Phi^ {*}} \left(\nabla \Phi (\theta_ {t}) - c [ \nabla J (\theta_ {t}) - \hat {g} _ {t} ], \nabla \Phi (\theta_ {t})\right)
$$
We shall do a second-order Taylor expansion of the critic objective (blue term) in  $c$  around 0 and a second-order Taylor expansion of the actor objective (green term) around  $\theta = \theta_{t}$ . Defining  $\delta \coloneqq \nabla J(\theta_t) - \hat{g}_t$ ,
$$
\mathrm {R H S} = \langle \hat {g} _ {t}, \theta - \theta_ {t} \rangle - \frac {1}{2} \left(\frac {1}{\eta} + \frac {1}{c}\right) (\theta - \theta_ {t}) ^ {\top} [ \nabla^ {2} \Phi (\theta_ {t}) ] (\theta - \theta_ {t}) - \frac {c}{2} \langle \delta , \nabla^ {2} \phi^ {*} (\nabla \phi (\pi_ {t})) \delta \rangle + o (c) + o (\| \theta - \theta_ {t} \| _ {2} ^ {2}), \tag {Using Prop. 14}
$$
where  $o(c)$  and  $o(\| \theta -\theta_t\| _2^2)$  consist of the higher order terms in the Taylor series expansion. A necessary and sufficient condition for monotonic improvement is equivalent to finding a  $(\theta ,c)$  such that RHS is positive. As  $c$  tends to 0, the  $\theta$  maximizing the RHS is
$$
\theta^ {*} = \theta_ {t} + \frac {c \eta}{c + \eta} \left[ \nabla^ {2} \Phi (\theta_ {t}) \right] ^ {\dagger} \hat {g} _ {t}
$$
With this choice,
$$
\begin{array}{l} \mathrm {R H S} = \frac {1}{2} \frac {1}{\frac {1}{\eta} + \frac {1}{c}} \left\langle \hat {g} _ {t}, \left[ \nabla^ {2} \Phi (\theta_ {t}) \right] ^ {- 1} \hat {g} _ {t} \right\rangle - \frac {c}{2} \left\langle \delta , \nabla^ {2} \phi^ {*} (\nabla \phi (\pi_ {t})) \delta \right\rangle + o (c) \quad \left(o \left(\left\| \theta - \theta_ {t} \right\| _ {2} ^ {2}\right) \text {i s s u b s u m e d b y} o (c)\right) \\ = \frac {c}{2} \left\langle \hat {g} _ {t}, \left[ \nabla^ {2} \Phi (\theta_ {t}) \right] ^ {- 1} \hat {g} _ {t} \right\rangle - \frac {c}{2} \left\langle \delta , \nabla^ {2} \phi^ {*} (\nabla \phi (\pi_ {t})) \delta \right\rangle + \underbrace {\frac {1}{2} \left(\frac {1}{\frac {1}{\eta} + \frac {1}{c}} - c\right) \left\langle \hat {g} _ {t} , \left[ \nabla^ {2} \Phi (\theta_ {t}) \right] ^ {- 1} \hat {g} _ {t} \right\rangle} _ {o (c) \text {t e r m}} + o (c) \\ = \frac {c}{2} \left\langle \hat {g} _ {t}, \left[ \nabla^ {2} \Phi (\theta_ {t}) \right] ^ {- 1} \hat {g} _ {t} \right\rangle - \frac {c}{2} \left\langle \delta , \nabla^ {2} \phi^ {*} (\nabla \phi (\pi_ {t})) \delta \right\rangle + o (c) \quad \text {(S u b s u m i n g t h e a d d i t i o n a l o (c) t e r m)} \\ \end{array}
$$
If  $\langle \hat{g}_t, [\nabla^2\Phi (\theta_t)]^{-1}\hat{g}_t\rangle > \langle \delta, \nabla^2\phi^* (\nabla \phi (\pi_t))\delta \rangle$ , i.e. there exists an  $\epsilon > 0$  s.t.  $\langle \hat{g}_t, [\nabla^2\Phi (\theta_t)]^{-1}\hat{g}_t\rangle = \langle \delta, \nabla^2\phi^* (\nabla \phi (\pi_t))\delta \rangle + \epsilon$ , then,  $\mathrm{RHS} = \frac{c\epsilon}{2} + o(c)$ . For any fixed  $\kappa > 0$ , since  $o(c) / c \to 0$  as  $c \to 0$ , there exists a neighbourhood  $(0, c_{\kappa})$  around zero such that for all  $c$  in this neighbourhood,  $o(c) / c > -\kappa$  and hence  $o(c) > -\kappa c$ . Setting  $\kappa = \frac{\varepsilon}{4}$ , there is a  $c$  such that
$$
\mathrm {R H S} > \frac {c \varepsilon}{4} > 0
$$
Hence, there exists a  $c \in (0, \min\{\eta, c_{\kappa}\})$  such that the RHS is positive, and is hence sufficient to guarantee monotonic policy improvement.
On the other hand, if  $\langle \hat{g}_t,\left[\nabla^2\Phi (\theta_t)\right]^{-1}\hat{g}_t\rangle <  \langle \delta ,\nabla^2\phi^* (\nabla \phi (\pi_t))\delta \rangle$  , i.e. there exists an  $\epsilon >0$  s.t.  $\langle \hat{g}_t,\left[\nabla^2\Phi (\theta_t)\right]^{-1}\hat{g}_t\rangle =$ $\langle \delta ,\nabla^{2}\phi^{*}(\nabla \phi (\pi_{t}))\delta \rangle -\epsilon$  , then,  $\mathrm{RHS} = \frac{-c\epsilon}{2} +o(c)$  which can be negative and hence monotonic improvement can not be guaranteed. Hence,  $\langle \hat{g}_t,\left[\nabla^2\Phi (\theta_t)\right]^{-1}\hat{g}_t\rangle >\langle \delta ,\nabla^{2}\phi^{*}(\nabla \phi (\pi_{t}))\delta \rangle$  is a necessary and sufficient condition for improvement.
Let us now consider the more general case, and define  $m = SA$ ,  $\hat{g}_t \in \mathbb{R}^{m \times 1}$ ,  $\pi(\theta) \in \mathbb{R}^{m \times 1}$  is a function of  $\theta \in \mathbb{R}^{n \times 1}$ . Rewriting Prop. 1,
$$
J (\pi) - J (\pi_ {t}) \geq \langle \hat {g} _ {t}, \pi (\theta) - \pi (\theta_ {t}) \rangle - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} (\pi (\theta), \pi (\theta_ {t})) - \frac {1}{c} D _ {\Phi^ {*}} \bigg (\nabla \Phi (\pi_ {t}) - c [ \nabla J (\pi_ {t}) - \hat {g} _ {t} ], \nabla \Phi (\pi_ {t}) \bigg)
$$
As before, we shall do a second-order Taylor expansion of the critic objective (blue term) in  $c$  around 0 and a second-order Taylor expansion of the actor objective (green term) around  $\theta = \theta_{t}$ . Defining  $\delta \coloneqq \nabla J(\theta_t) - \hat{g}_t$ . From Prop. 14, we know that,
$$
\frac {1}{c} D _ {\Phi^ {*}} \bigg (\nabla \Phi (\pi_ {t}) - c [ \nabla J (\pi_ {t}) - \hat {g} _ {t} ], \nabla \Phi (\pi_ {t}) \bigg) = \frac {c}{2} \langle \delta , \nabla^ {2} \phi^ {*} (\nabla \phi (\pi_ {t})) \delta \rangle + o (c)
$$
In order to calculate the second-order Taylor series expansion of the actor objective, we define  $\nabla_{\theta}\pi (\theta_t)\in \mathbb{R}^{m\times n}$  as the Jacobian of the  $\theta \colon \to \pi$  map, and use  $\nabla_{\theta}[\pi (\theta_t)]_i\in \mathbb{R}^{1\times n}$  for  $i\in [m]$  to refer to row  $i$
$$
\langle \hat {g} _ {t}, \pi (\theta) - \pi (\theta_ {t}) \rangle = \sum_ {i = 1} ^ {m} \underbrace {[ \hat {g} _ {t} ] _ {i}} _ {1 \times 1} \underbrace {\nabla_ {\theta} [ \pi (\theta_ {t}) ] _ {i}} _ {1 \times n} \underbrace {(\theta - \theta_ {t})} _ {n \times 1} + \frac {1}{2} \underbrace {(\theta - \theta_ {t})} _ {1 \times n} \left[ \sum_ {i = 1} ^ {m} \underbrace {[ \hat {g} _ {t} ] _ {i}} _ {1 \times 1} \underbrace {\nabla_ {\theta} ^ {2} [ \pi (\theta_ {t}) ] _ {i}} _ {n \times n} \right] \underbrace {(\theta - \theta_ {t})} _ {n \times 1} + o (\| \theta - \theta_ {t} \| _ {2} ^ {2})
$$
where  $o(\| \theta -\theta_t\| _2^2)$  consist of the higher order terms in the Taylor series expansion. For expanding the divergence term, note that  $D_{\Phi}(\pi (\theta),\pi (\tilde{\theta}_t)) = \phi (\pi (\theta)) - \phi (\pi (\theta_t)) - \langle \nabla \phi (\pi (\theta_t)),\pi (\theta) - \pi (\theta_t)\rangle$
$$
\begin{array}{l} \phi (\pi (\theta)) - \phi (\pi (\theta_ {t})) = \underbrace {\nabla_ {\pi} \phi (\pi_ {t}) ^ {\intercal}} _ {1 \times m} \underbrace {\nabla_ {\theta} \pi (\theta_ {t})} _ {m \times n} \underbrace {(\theta - \theta_ {t})} _ {n \times 1} \\ + \frac {1}{2} \underbrace {\left(\theta - \theta_ {t}\right) ^ {\intercal}} _ {1 \times n} \left[ \underbrace {\nabla_ {\theta} \pi (\theta_ {t}) ^ {\intercal}} _ {n \times m} \underbrace {\nabla_ {\pi} ^ {2} \phi (\pi_ {t})} _ {m \times m} \underbrace {\nabla_ {\theta} \pi (\theta_ {t})} _ {m \times n} + \sum_ {i = 1} ^ {m} \underbrace {[ \nabla_ {\pi} \phi (\pi_ {t}) ] _ {i}} _ {1 \times 1} \underbrace {\nabla_ {\theta} ^ {2} [ \pi (\theta_ {t}) ] _ {i}} _ {n \times n} \right] \underbrace {\left(\theta - \theta_ {t}\right)} _ {n \times 1} + o (\| \theta - \theta_ {t} \| _ {2} ^ {2}) \\ \end{array}
$$
$$
\langle \nabla \phi (\pi (\theta_ {t})), \pi (\theta) - \pi (\theta_ {t}) \rangle = \sum_ {i = 1} ^ {m} \underbrace {[ \nabla \phi (\pi_ {t}) ] _ {i}} _ {1 \times 1} \underbrace {[ \nabla_ {\theta} \pi (\theta_ {t}) ] _ {i}} _ {1 \times n} \underbrace {(\theta - \theta_ {t})} _ {n \times 1} + \frac {1}{2} \underbrace {(\theta - \theta_ {t})} _ {1 \times n} \left[ \sum_ {i = 1} ^ {m} \underbrace {[ \nabla \phi (\pi_ {t}) ] _ {i}} _ {1 \times 1} \underbrace {\nabla_ {\theta} ^ {2} [ \pi (\theta_ {t}) ] _ {i}} _ {n \times n} \right] \underbrace {(\theta - \theta_ {t})} _ {n \times 1} + o (\| \theta - \theta_ {t} \| _ {2} ^ {2})
$$
Putting everything together,
$$
\begin{array}{l} \mathrm {R H S} = \left(\sum_ {i = 1} ^ {m} \underbrace {\left[ \hat {g} _ {t} \right] _ {i}} _ {1 \times 1} \underbrace {\nabla_ {\theta} \left[ \pi \left(\theta_ {t}\right) \right] _ {i}} _ {1 \times n}\right) \underbrace {\left(\theta - \theta_ {t}\right)} _ {n \times 1} \\ + \frac {1}{2} \underbrace {(\theta - \theta_ {t}) ^ {\intercal}} _ {1 \times n} \left[ \sum_ {i = 1} ^ {m} \left(\underbrace {[ \hat {g} _ {t} ] _ {i}} _ {1 \times 1} \underbrace {\nabla_ {\theta} ^ {2} [ \pi (\theta_ {t}) ] _ {i}} _ {n \times n}\right) - \left(\frac {1}{\eta} + \frac {1}{c}\right) \underbrace {\nabla_ {\theta} \pi (\theta_ {t}) ^ {\intercal}} _ {n \times m} \underbrace {\nabla_ {\pi} ^ {2} \phi (\pi_ {t})} _ {m \times m} \underbrace {\nabla_ {\theta} \pi (\theta_ {t})} _ {m \times n} \right] \underbrace {(\theta - \theta_ {t}) ^ {\intercal}} _ {n \times 1} \\ - \frac {c}{2} \left\langle \delta , \nabla^ {2} \phi^ {*} (\nabla \phi (\pi_ {t})) \delta \right\rangle + o (\| \theta - \theta_ {t} \| _ {2} ^ {2}) + o (c) \\ \end{array}
$$
Defining  $b_{t} \coloneqq \sum_{i=1}^{m} [\hat{g}_{t}]_{i} \nabla_{\theta}[\pi(\theta_{t})]_{i}$  and  $H_{t} \coloneqq \nabla_{\theta} \pi(\theta_{t})^{\top} \nabla_{\pi}^{2} \phi(\pi_{t}) \nabla_{\theta} \pi(\theta_{t}) - \frac{1}{\left( \frac{1}{\eta} + \frac{1}{c} \right)} \sum_{i=1}^{m} \left( [\hat{g}_{t}]_{i} \nabla_{\theta}^{2} [\pi(\theta_{t})]_{i} \right)$
$$
\mathrm {R H S} = \langle b _ {t}, \theta - \theta_ {t} \rangle + \frac {1}{2} \left(\frac {1}{\eta} + \frac {1}{c}\right) \langle (\theta - \theta_ {t}), H _ {t} (\theta - \theta_ {t}) \rangle - \frac {c}{2} \langle \delta , \nabla^ {2} \phi^ {*} (\nabla \phi (\pi_ {t})) \delta \rangle + o (\| \theta - \theta_ {t} \| _ {2} ^ {2}) + o (c)
$$
As a sanity check, it can be verified that if  $\pi (\theta) = \theta$ ,  $H_{t} = \left(\frac{1}{\eta} +\frac{1}{c}\right)\nabla^{2}\Phi (\theta_{t})$  and  $b_{t} = \hat{g}_{t}$ , and we recover the tabular result above. Notice that  $\left\langle (\theta -\theta_t),\frac{1}{\left(\frac{1}{\eta} + \frac{1}{c}\right)}\sum_{i = 1}^{m}\left([\hat{g}_t]_i\nabla_\theta^2 [\pi (\theta_t)]_i\right)(\theta -\theta_t)\right\rangle$  is  $o(c)$  as  $c$  goes to zero. Subsuming this term in  $o(c)$ ,
$$
\mathrm {R H S} = \left\langle b _ {t}, \theta - \theta_ {t} \right\rangle + \frac {1}{2} \left(\frac {1}{\eta} + \frac {1}{c}\right) \left\langle \left(\theta - \theta_ {t}\right), \tilde {H} _ {t} \left(\theta - \theta_ {t}\right) \right\rangle - \frac {c}{2} \left\langle \delta , \nabla^ {2} \phi^ {*} (\nabla \phi (\pi_ {t})) \delta \right\rangle + o \left(\left\| \theta - \theta_ {t} \right\| _ {2} ^ {2}\right) + o (c)
$$
where  $\tilde{H}_t\coloneqq \nabla_\theta \pi (\theta_t)^\top \nabla_\pi^2\phi (\pi_t)\nabla_\theta \pi (\theta_t)$ . As before, a necessary and sufficient condition for monotonic improvement is equivalent to finding a  $(\theta ,c)$  such that RHS is positive. As  $c$  tends to 0, the  $\theta$  maximizing the RHS is
$$
\theta^ {*} = \theta_ {t} + \frac {c \eta}{(c + \eta)} \left[ \tilde {H} _ {t} \right] ^ {\dagger} b _ {t}
$$
$$
\mathrm {R H S} = \frac {1}{2} \frac {1}{\frac {1}{\eta} + \frac {1}{c}} \left\langle b _ {t}, \left[ \tilde {H} _ {t} \right] ^ {\dagger} b _ {t} \right\rangle - \frac {c}{2} \left\langle \delta , \nabla^ {2} \phi^ {*} (\nabla \phi (\pi_ {t})) \delta \right\rangle + o (c) \quad \left(o \left(\| \theta - \theta_ {t} \| _ {2} ^ {2}\right) \text {i s s u b s u m e d i n} o (c)\right)
$$
As in the tabular case, since  $\frac{1}{\frac{1}{\eta} + \frac{1}{c}}$  is  $o(c)$ , we can subsume it, and we get that,
$$
\mathrm {R H S} = \frac {c}{2} \left\langle b _ {t}, \left[ \tilde {H} _ {t} \right] ^ {\dagger} b _ {t} \right\rangle - \frac {c}{2} \left\langle \delta , \nabla^ {2} \phi^ {*} (\nabla \phi (\pi_ {t})) \delta \right\rangle + o (c)
$$
Using the same reasoning as in the tabular case above, we can prove that
$$
\left\langle b _ {t}, \left[ \tilde {H} _ {t} \right] ^ {\dagger} b _ {t} \right\rangle > \left\langle \delta , \nabla^ {2} \phi^ {*} (\nabla \phi (\pi_ {t})) \delta \right\rangle
$$
is a necessary and sufficient condition for monotonic policy improvement. Finally, we use Gorni [19, Eq (1.5)] which shows that  $\nabla^2\phi^* (\nabla \phi (\pi_t)) = [\nabla_\pi^2\Phi (\pi_t)]^{-1}$  and complete the proof.
D.1 Proof of Prop. 3
The following proposition shows the convergence of inexact mirror ascent in the functional space.
Proposition 9. Assuming that (i)  $J + \frac{1}{\eta} \Phi$  is convex in  $\pi$ , for a constant  $c > 0$ , after  $T$  iterations of mirror ascent with  $\frac{1}{\eta'} = \frac{2}{\eta} + \frac{2}{c}$  we have
$$
\mathbb {E} \frac {D _ {\Phi} (\bar {\pi} _ {\mathcal {R} + 1} , \bar {\pi} _ {\mathcal {R}})}{\zeta^ {2}} \leq \frac {1}{\zeta T} \left[ [ J (\pi^ {*}) - J (\pi_ {0}) ] + \frac {1}{c} \sum_ {t = 0} ^ {T - 1} \mathbb {E} D _ {\phi^ {*}} \left(\nabla \phi (\bar {\pi} _ {t}) - c [ \nabla J (\bar {\pi} _ {t}) - \hat {g} (\bar {\pi} _ {t}) ], \nabla \phi (\bar {\pi} _ {t})\right) \right]
$$
where  $\zeta = \eta' / 2$  and  $\mathcal{R}$  is picked uniformly random from  $\{0,1,2,\ldots,T-1\}$ .
Proof. We divide the mirror ascent (MA) update into two steps:
$$
\nabla \phi (\tilde {\pi} _ {t + 1}) = \nabla \phi (\bar {\pi} _ {t}) + \eta_ {t} ^ {\prime} \hat {g} (\bar {\pi} _ {t}) \Rightarrow \hat {g} (\bar {\pi} _ {t}) = \frac {1}{\eta_ {t} ^ {\prime}} [ \nabla \phi (\tilde {\pi} _ {t + 1}) - \nabla \phi (\bar {\pi} _ {t}) ]
$$
$$
\bar{\pi}_{t + 1} = \operatorname *{arg  min}_{\pi \in \Pi}D_{\Phi}(\pi ,\tilde{\pi}_{t + 1}).
$$
We denote the above update as  $\bar{\pi}_{t + 1} = \mathrm{MA}(\bar{\pi}_t)$ . Using Prop. 1 with  $\pi = \bar{\pi}_{t + 1}$ ,  $\pi' = \bar{\pi}_t$ ,
$$
\begin{array}{l} J(\bar{\pi}_{t + 1})\geq J(\bar{\pi}_{t}) + \hat{g} (\bar{\pi}_{t})^{\top}(\bar{\pi}_{t + 1} - \bar{\pi}_{t}) - \left(\frac{1}{\eta} +\frac{1}{c}\right)D_{\Phi}(\bar{\pi}_{t + 1},\bar{\pi}_{t}) - \underbrace{\frac{1}{c}D_{\phi^{*}}\Bigl(\nabla\phi(\bar{\pi}_{t}) - c[\nabla J(\bar{\pi}_{t}) - \hat{g} (\bar{\pi}_{t})],\nabla\phi(\bar{\pi}_{t})\Bigr)}_{:= \epsilon_{t}^{c}} \\ \geq J (\bar {\pi} _ {t}) + \frac {1}{\eta_ {t} ^ {\prime}} \left\langle \nabla \phi \left(\tilde {\pi} _ {t + 1}\right) - \nabla \phi \left(\bar {\pi} _ {t}\right), \bar {\pi} _ {t + 1} - \bar {\pi} _ {t} \right\rangle - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} \left(\bar {\pi} _ {t + 1}, \bar {\pi} _ {t}\right) - \epsilon_ {t} ^ {c} \quad (U s i n g \\ \geq J \left(\bar {\pi} _ {t}\right) + \frac {1}{\eta_ {t} ^ {\prime}} \left\{D _ {\Phi} \left(\bar {\pi} _ {t + 1}, \bar {\pi} _ {t}\right) + D _ {\Phi} \left(\bar {\pi} _ {t}, \tilde {\pi} _ {t + 1}\right) - D _ {\Phi} \left(\bar {\pi} _ {t + 1}, \tilde {\pi} _ {t + 1}\right) \right\} - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} \left(\bar {\pi} _ {t + 1}, \bar {\pi} _ {t}\right) - \epsilon_ {t} ^ {c} \tag {using Lemma 4} \\ = J(\bar{\pi}_{t}) + \frac{1}{\eta^{\prime}}\underbrace{\{D_{\Phi}(\bar{\pi}_{t},\tilde{\pi}_{t + 1}) - D_{\Phi}(\bar{\pi}_{t + 1},\tilde{\pi}_{t + 1})\}}_{:= A} + \left(\frac{1}{\eta^{\prime}} -\frac{1}{\eta} -\frac{1}{c}\right)D_{\Phi}(\bar{\pi}_{t + 1},\bar{\pi}_{t}) - \epsilon^{c}_{t} \\ \geq J \left(\bar {\pi} _ {t}\right) + \left(\frac {1}{\eta_ {t} ^ {\prime}} - \frac {1}{\eta} - \frac {1}{c}\right) D _ {\Phi} \left(\bar {\pi} _ {t + 1}, \bar {\pi} _ {t}\right) - \epsilon_ {t} ^ {c} \quad (A \geq 0 \text {s i n c e} \bar {\pi} _ {t + 1} \text {i s t h e p r o j e c t i o n o f} \tilde {\pi} _ {t + 1} \text {o n t o I I}) \\ \geq J \left(\bar {\pi} _ {t}\right) + \underbrace {\left(\frac {1}{\eta} + \frac {1}{c}\right)} _ {:= \frac {1}{\zeta}} D _ {\Phi} \left(\bar {\pi} _ {t + 1}, \bar {\pi} _ {t}\right) - \epsilon_ {t} ^ {c} \quad \left(\text {S i n c} 1 / \eta_ {t} ^ {\prime} = 2 / \eta + 2 / c\right) \\ \end{array}
$$
Recursing for  $T$  iterations and dividing by  $1 / \zeta$ , picking  $\mathcal{R}$  uniformly random from  $\{0, 1, 2, \ldots, T - 1\}$  and taking expectation we get
$$
\begin{array}{l} \mathbb {E} \frac {D _ {\Phi} (\bar {\pi} _ {\mathcal {R} + 1} , \bar {\pi} _ {\mathcal {R}})}{\zeta^ {2}} = \frac {1}{\zeta^ {2} T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} D _ {\Phi} (\bar {\pi} _ {t + 1}, \bar {\pi} _ {t}) \\ \leq \frac {\mathbb {E} [ J (\bar {\pi} _ {T}) - J (\pi_ {0}) ]}{\zeta T} + \frac {\sum_ {t = 0} ^ {T - 1} \mathbb {E} \epsilon_ {t} ^ {c}}{T} \\ \leq \frac {\left[ J \left(\pi^ {*}\right) - J \left(\pi_ {0}\right) \right]}{\zeta T} + \frac {\sum_ {t = 0} ^ {T - 1} \mathbb {E} \epsilon_ {t} ^ {c}}{T} \\ = \frac {1}{\zeta T} \left[ [ J (\pi^ {*}) - J (\pi_ {0}) ] + \frac {1}{c} \sum_ {t = 0} ^ {T - 1} \mathbb {E} D _ {\phi^ {*}} \left(\nabla \phi \left(\bar {\pi} _ {t}\right) - c [ \nabla J (\bar {\pi} _ {t}) - \hat {g} (\bar {\pi} _ {t}) ], \nabla \phi (\bar {\pi} _ {t})\right) \right] \\ \end{array}
$$
Compared to Dragomir et al. [15], D'Orazio et al. [12] that analyze stochastic mirror ascent in the smooth, non-convex setting, our analysis ensures that the (i) sub-optimality gap (the LHS in the above proposition) is always positive, and (ii) uses a different notion of variance that depends on  $D_{\Phi^*}$ .
Similar to Vaswani et al. [57], we assume that each  $\pi \in \Pi$  is parameterized by  $\theta$ . In Algorithm 1, we run gradient ascent (GA) on  $\ell_t(\theta)$  to compute  $\pi_{t + 1} = \pi (\theta_{t + 1})$  and interpret the inner loop of Algorithm 1 as an approximation to the projection step in the mirror ascent update. We note that  $\ell_t(\theta)$  does not have any additional randomness and is a deterministic function w.r.t  $\theta$ . Note that  $\bar{\pi}_{t + 1} =$  where  $\pi_t = \pi (\theta_t)$ . Assuming that  $\ell_t(\theta)$  is smooth, and satisfies the PL condition [27], we get the following convergence guarantee for Algorithm 1.
$$
\mathbb {E} \left[ \frac {D _ {\Phi} \left(\bar {\pi} _ {\mathcal {R} + 1} , \pi_ {\mathcal {R}}\right)}{\zeta^ {2}} \right] \leq \frac {1}{\zeta T} \left[ \underbrace {J \left(\pi^ {*}\right) - J \left(\pi_ {0}\right)} _ {\text {T e r m (i)}} + \sum_ {t = 0} ^ {T - 1} \left(\underbrace {\frac {1}{c} \mathbb {E} D _ {\Phi^ {*}} \left(\nabla \Phi \left(\pi_ {t}\right) - c \delta_ {t} , \nabla \Phi \left(\pi_ {t}\right)\right)} _ {\text {T e r m (i i)}} + \underbrace {\mathbb {E} [ e _ {t} ]} _ {\text {T e r m (i i i)}}\right) \right]
$$
Proof. For this proof, we define the following notation:
$$
\pi_ {t} := \pi (\theta_ {t})
$$
$$
\ell_ {t} (\theta) := J (\pi_ {t}) + \hat {g} (\pi_ {t}) ^ {\top} (\pi (\theta) - \pi_ {t}) - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} (\pi (\theta), \pi_ {t})
$$
$$
\bar {\theta} _ {t + 1} := \arg \max  \ell_ {t} (\theta)
$$
$$
\bar {\pi} _ {t + 1} := \pi (\bar {\theta} _ {t + 1}) = \underset {\pi \in \Pi} {\arg \max } \left\{\left\langle \hat {g} _ {t} \left(\pi_ {t}\right), \pi - \pi_ {t} \right\rangle - \frac {1}{\eta^ {\prime}} D _ {\Phi} \left(\pi , \pi_ {t}\right) \right\} = \mathrm {M A} \left(\pi_ {t}\right)
$$
(Iterate obtained after running 1 step of mirror ascent starting from  $\pi_t$ )
$$
\theta_ {t + 1} := \operatorname {G r a d i e n t A s c e n t} \left(\ell_ {t} (\theta), \theta_ {t}, m _ {a}\right)
$$
$$
\pi_ {t + 1} := \pi \left(\theta_ {t + 1}\right),
$$