SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\| \theta_ {k} - \theta_ {k - 1} \| = \alpha \| g _ {k - 1} \| \stackrel {(i)} {=} \alpha \| h (\theta_ {k - 1}; \tau_ {k - 1} ^ {\mathrm {E}}) - h (\theta_ {k - 1}; \tau_ {k - 1} ^ {\mathrm {A}}) \| \leq \alpha \| h (\theta_ {k - 1}; \tau_ {k - 1} ^ {\mathrm {E}}) \| + \alpha \| h (\theta_ {k - 1}; \tau_ {k - 1} ^ {\mathrm {A}}) \| \stackrel {(i i)} {\leq} \frac {2 \alpha L _ {r}}{1 - \gamma}
$$
where (i) follows the definition of the reward gradient estimator defined in (16). The inequality (ii) follows the definition of  $h(\theta; \tau)$  in (16) and the bound gradient property  $\| \nabla_{\theta} r(s, a; \theta) \| \leq L_r$  in (18) from Assumption 3. Recall that we have defined the constant  $L_q \coloneqq \frac{L_r}{1 - \gamma}$ , then we obtain the following result:
$$
\left\| \theta_ {k} - \theta_ {k - 1} \right\| \leq 2 \alpha L _ {q}. \tag {57}
$$
Plugging (57) into (56), we can show that
$$
\left\| Q _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty} \leq \gamma \left\| Q _ {k - 1} - Q _ {\theta_ {k - 1}} \right\| _ {\infty} + 4 \alpha L _ {q} ^ {2} + \frac {2 \gamma \epsilon_ {\mathrm {a p p}}}{1 - \gamma}. \tag {58}
$$
Summing (58) from  $k = 1$  to  $k = K$  and dividing  $K$  on both sides, then we have the following result:
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \left\| Q _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty} \leq \frac {\gamma}{K} \sum_ {k = 0} ^ {K - 1} \left\| Q _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty} + 4 \alpha L _ {q} ^ {2} + \frac {2 \gamma \epsilon_ {\mathrm {a p p}}}{1 - \gamma}. \tag {59}
$$
By rearranging (59), we obtain the following inequality:
$$
\frac {1 - \gamma}{K} \sum_ {k = 1} ^ {K} \left\| Q _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty} \leq \frac {\gamma}{K} \left(\left\| Q _ {0} - Q _ {\theta_ {0}} \right\| _ {\infty} - \left\| Q _ {K} - Q _ {\theta_ {K}} \right\| _ {\infty}\right) + 4 \alpha L _ {q} ^ {2} + \frac {2 \gamma \epsilon_ {\mathrm {a p p}}}{1 - \gamma}. \tag {60}
$$
Assuming the initial error  $\| Q_0 - Q_{\theta_0}\|_{\infty}$  is bounded by a positive constant  $\Delta_0$  where  $\| Q_0 - Q_{\theta_0}\|_{\infty} \leq \Delta_0$  and dividing  $1 - \gamma$  on the both sides of (60), then the following inequality holds:
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \left\| Q _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty} \leq \frac {\gamma}{K (1 - \gamma)} \Delta_ {0} + \frac {4 \alpha L _ {q} ^ {2}}{1 - \gamma} + \frac {2 \gamma \epsilon_ {\mathrm {a p p}}}{(1 - \gamma) ^ {2}}. \tag {61}
$$
Then we subtract  $\frac{1}{K}\left\| Q_K - Q_{\theta_K}\right\|_\infty$  and add  $\frac{1}{K}\left\| Q_0 - Q_{\theta_0}\right\|_\infty$  on both sides on (61), we obtain the following result:
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \left\| Q _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty} \leq \frac {\gamma}{K (1 - \gamma)} \Delta_ {0} + \frac {1}{K} \Delta_ {0} - \frac {1}{K} \left\| Q _ {K} - Q _ {\theta_ {K}} \right\| _ {\infty} + \frac {4 \alpha L _ {q} ^ {2}}{1 - \gamma} + \frac {2 \gamma \epsilon_ {\mathrm {a p p}}}{(1 - \gamma) ^ {2}} \\ \leq \frac {1}{K (1 - \gamma)} \Delta_ {0} + \frac {4 \alpha L _ {q} ^ {2}}{1 - \gamma} + \frac {2 \gamma \epsilon_ {\mathrm {a p p}}}{(1 - \gamma) ^ {2}}. \tag {62} \\ \end{array}
$$
Plugging (62) into (55), then we obtain the convergence result of the policy estimates:
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \left\| \log \pi_ {k + 1} - \log \pi_ {\theta_ {k}} \right\| _ {\infty} \stackrel {(i)} {\leq} 2 \epsilon_ {\mathrm {a p p}} + \frac {2}{K} \sum_ {k = 0} ^ {K - 1} \left\| Q _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty} \\ \stackrel {(i i)} {\leq} \left(2 + \frac {4 \gamma}{(1 - \gamma) ^ {2}}\right) \epsilon_ {\text {a p p}} + \frac {2 \Delta_ {0}}{K (1 - \gamma)} + \frac {8 \alpha L _ {q} ^ {2}}{1 - \gamma} \tag {63} \\ \end{array}
$$
where (i) follows (55) and (ii) is from (62). Recall that the stepsize  $\alpha$  is defined to be  $\alpha = \alpha_0 \times K^{-\frac{1}{2}}$ , then we could obtain the convergence rate of the policy estimates as below:
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \| \log \pi_ {k + 1} - \log \pi_ {\theta_ {k}} \| _ {\infty} = \mathcal {O} \left(\epsilon_ {\mathrm {a p p}}\right) + \mathcal {O} \left(K ^ {- \frac {1}{2}}\right). \tag {64}
$$
The relation (20a) has been proven.
J.2 Proof of the convergence of reward parameter (20b)
Proof. In the section, we prove the convergence of the reward parameters  $\{\theta_k\}_{k\geq 0}$ . Recall that we have shown the Lipschitz property of the surrogate objective  $\widehat{L} (\theta)$  in (19b) of Lemma 4. Based on the Lipschitz smooth property, we are able to have following analysis:
$$
\begin{array}{l} \widehat {L} (\theta_ {k + 1}) \stackrel {(i)} {\geq} \widehat {L} (\theta_ {k}) + \left\langle \nabla \widehat {L} (\theta_ {k}), \theta_ {k + 1} - \theta_ {k} \right\rangle - \frac {L _ {c}}{2} \| \theta_ {k + 1} - \theta_ {k} \| ^ {2} \\ \stackrel {(i i)} {=} \widehat {L} (\theta_ {k}) + \alpha \left\langle \nabla \widehat {L} (\theta_ {k}), g _ {k} \right\rangle - \frac {L _ {c} \alpha^ {2}}{2} \| g _ {k} \| ^ {2} \\ = \widehat {L} (\theta_ {k}) + \alpha \left\langle \nabla \widehat {L} (\theta_ {k}), g _ {k} - \nabla \widehat {L} (\theta_ {k}) \right\rangle + \alpha \| \nabla \widehat {L} (\theta_ {k}) \| ^ {2} - \frac {L _ {c} \alpha^ {2}}{2} \| g _ {k} \| ^ {2} \tag {65} \\ \end{array}
$$
where (i) is due to the Lipschitz smooth property in (19b) and (ii) follows the update rule of the reward parameter in (17). Recall that the gradient estimator  $g_{k}$  is defined in (16). Due to the bound gradient of any reward parameter, we could obtain the following bound:
$$
\left\| g _ {k} \right\| \stackrel {(i)} {=} \| h \left(\theta_ {k}; \tau_ {k} ^ {\mathrm {E}}\right) - h \left(\theta_ {k}; \tau_ {k} ^ {\mathrm {A}}\right) \| \leq \| h \left(\theta_ {k}; \tau_ {k} ^ {\mathrm {E}}\right) \| + \| h \left(\theta_ {k}; \tau_ {k} ^ {\mathrm {A}}\right) \| \stackrel {(i i)} {\leq} \frac {2 L _ {r}}{1 - \gamma} \tag {66}
$$
where (i) is from (16) and (ii) follows the bounded gradient of reward parameter in (18) of Assumption 3. Recall that we have defined the constant  $L_{q} \coloneqq \frac{L_{r}}{1 - \gamma}$ , then the following result holds:
$$
\widehat {L} \left(\theta_ {k + 1}\right) \geq \widehat {L} \left(\theta_ {k}\right) + \alpha \left\langle \nabla \widehat {L} \left(\theta_ {k}\right), g _ {k} - \nabla \widehat {L} \left(\theta_ {k}\right) \right\rangle + \alpha \| \nabla \widehat {L} \left(\theta_ {k}\right) \| ^ {2} - 2 L _ {c} L _ {q} ^ {2} \alpha^ {2}. \tag {67}
$$
After taking expectation on both sides of (67), we have
$$
\mathbb {E} \left[ \widehat {L} \left(\theta_ {k + 1}\right) \right]
$$
$$
\geq \mathbb {E} \left[ \widehat {L} (\theta_ {k}) \right] + \alpha \mathbb {E} \left[ \left\langle \nabla \widehat {L} (\theta_ {k}), g _ {k} - \nabla \widehat {L} (\theta_ {k}) \right\rangle \right] + \alpha \mathbb {E} \left[ \| \nabla \widehat {L} (\theta_ {k}) \| ^ {2} \right] - 2 L _ {c} L _ {q} ^ {2} \alpha^ {2}
$$
$$
= \mathbb {E} \left[ \widehat {L} (\theta_ {k}) \right] + \alpha \mathbb {E} \left[ \left\langle \nabla \widehat {L} (\theta_ {k}), \mathbb {E} [ g _ {k} - \nabla \widehat {L} (\theta_ {k}) | \theta_ {k} ] \right\rangle \right] + \alpha \mathbb {E} \left[ \| \nabla \widehat {L} (\theta_ {k}) \| ^ {2} \right] - 2 L _ {c} L _ {q} ^ {2} \alpha^ {2}
$$
$$
\stackrel {(i)} {=} \mathbb {E} \left[ \widehat {L} (\theta_ {k}) \right] + \alpha \mathbb {E} \left[ \left\langle \nabla \widehat {L} (\theta_ {k}), \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {k}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {k}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {k + 1}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {k}) \right] \right\rangle \right]
$$
$$
+ \alpha \mathbb {E} \left[ \| \nabla \widehat {L} (\theta_ {k}) \| ^ {2} \right] - 2 L _ {c} L _ {q} ^ {2} \alpha^ {2}
$$
$$
\overset {(i i)} {\geq} \mathbb {E} \left[ \widehat {L} (\theta_ {k}) \right] - \underbrace {2 \alpha L _ {q} \mathbb {E} \left[ \left\| \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {k}} , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t} , a _ {t} ; \theta_ {k}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {k + 1} , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t} , a _ {t} ; \theta_ {k}) \right] \right\| \right]}
$$
$$
+ \alpha \mathbb {E} \left[ \| \nabla \widehat {L} \left(\theta_ {k}\right) \| ^ {2} \right] - 2 L _ {c} L _ {q} ^ {2} \alpha^ {2} \tag {68}
$$
where (i) is from the definitions of the reward gradient estimator in (16) and the reward gradient expression in (15) of Lemma 3; (ii) follows  $\| \nabla \widehat{L} (\theta_k)\| \leq 2L_q$  which could be proved according to (66). Then we analyze the error term due to policy mismatch as below:
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {k}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {k}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {k + 1}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {k}) \right] \right\| \right] \\ \stackrel {(i)} {=} \mathbb {E} \left[ \left\| \frac {1}{1 - \gamma} \mathbb {E} _ {(s, a) \sim d _ {\hat {P}} ^ {\pi_ {\theta_ {k}}} (\cdot , \cdot)} \left[ \nabla_ {\theta} r (s, a; \theta_ {k}) \right] - \frac {1}{1 - \gamma} \mathbb {E} _ {(s, a) \sim d _ {\hat {P}} ^ {\pi_ {k + 1}} (\cdot , \cdot)} \left[ \nabla_ {\theta} r (s, a; \theta_ {k}) \right] \right\| \right] \\ = \frac {1}{1 - \gamma} \mathbb {E} \left[ \left\| \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} \nabla_ {\theta} r (s, a; \theta_ {k}) \cdot \left(d _ {\widehat {P}} ^ {\pi_ {\theta_ {k}}} (s, a) - d _ {\widehat {P}} ^ {\pi_ {k + 1}} (s, a)\right) \right\| \right] \\ \leq \frac {1}{1 - \gamma} \mathbb {E} \left[ \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} \left| \left| \nabla_ {\theta} r (s, a; \theta_ {k}) \right| \right| \cdot \left| d _ {\widehat {P}} ^ {\pi_ {\theta_ {k}}} (s, a) - d _ {\widehat {P}} ^ {\pi_ {k + 1}} (s, a) \right| \right] \\ \stackrel {(i i)} {\leq} \frac {L _ {r}}{1 - \gamma} \mathbb {E} \left[ \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} \left| d _ {\widehat {P}} ^ {\pi_ {\theta_ {k}}} (s, a) - d _ {\widehat {P}} ^ {\pi_ {k + 1}} (s, a) \right| \right] \\ \stackrel {(i i i)} {=} \frac {2 L _ {r}}{1 - \gamma} \mathbb {E} \left[ \left\| d _ {\hat {P}} ^ {\pi_ {\theta_ {k}}} (\cdot , \cdot) - d _ {\hat {P}} ^ {\pi_ {k + 1}} (\cdot , \cdot) \right\| _ {\mathrm {T V}} \right] \tag {69} \\ \end{array}
$$
where (i) follows the definition of the visitation measures  $d_{\widehat{P}}^{\pi_{\theta_k}}(\cdot, \cdot)$  and  $d_{\widehat{P}}^{\pi_{k+1}}$ , (ii) follows the bound gradient of reward parameter in (18) of Assumption 3 and (iii) follows the definition of the total variation norm. Recall that we have defined the constant  $L_q \coloneqq \frac{L_r}{1 - \gamma}$ . Due to the fact that  $\pi_{\theta_k}$  and  $\pi_{k+1}$  are softmax policies parameterized by  $Q_{\theta_k}$  and  $\widehat{Q}_k$ , then we obtain the following result based
on Lemma 6:
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {k}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {k}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {k + 1}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {k}) \right] \right\| \right] \\ = 2 L _ {q} \mathbb {E} \left[ \left\| d _ {\widehat {P}} ^ {\pi_ {\theta_ {k}}} (\cdot , \cdot) - d _ {\widehat {P}} ^ {\pi_ {k + 1}} (\cdot , \cdot) \right\| _ {\mathrm {T V}} \right] \\ \stackrel {(i)} {\leq} 2 L _ {q} C _ {d} \mathbb {E} \left[ \left\| Q _ {\theta_ {k}} - \widehat {Q} _ {k} \right\| \right] \\ \stackrel {(i i)} {\leq} 2 L _ {q} C _ {d} \sqrt {| \mathcal {S} | \times | \mathcal {A} |} \mathbb {E} \left[ \left\| Q _ {\theta_ {k}} - \widehat {Q} _ {k} \right\| _ {\infty} \right] \\ = 2 L _ {q} C _ {d} \sqrt {| \mathcal {S} | \times | \mathcal {A} |} \mathbb {E} \left[ \left\| Q _ {\theta_ {k}} - Q _ {k} + Q _ {k} - \widehat {Q} _ {k} \right\| _ {\infty} \right] \\ \leq 2 L _ {q} C _ {d} \sqrt {| \mathcal {S} | \times | \mathcal {A} |} \mathbb {E} \left[ \left\| Q _ {\theta_ {k}} - Q _ {k} \right\| _ {\infty} + \epsilon_ {\text {a p p}} \right] \tag {70} \\ \end{array}
$$
where (i) follows (25) in Lemma 6 and (ii) follows the conversion between Frobenius norm and the infinity norm. Plugging (70) into (68), we have
$$
\mathbb {E} \left[ \widehat {L} \left(\theta_ {k + 1}\right) \right] \geq \mathbb {E} \left[ \widehat {L} \left(\theta_ {k}\right) \right] - 4 \alpha C _ {d} L _ {q} ^ {2} \sqrt {| \mathcal {S} | \times | \mathcal {A} |} \mathbb {E} \left[ \left\| Q _ {\theta_ {k}} - Q _ {k} \right\| _ {\infty} + \epsilon_ {\text {a p p}} \right] + \alpha \mathbb {E} \left[ \| \nabla \widehat {L} \left(\theta_ {k}\right) \| ^ {2} \right] - 2 L _ {c} L _ {q} ^ {2} \alpha^ {2} \tag {71}
$$
Rearranging (71) and dividing its both sides by  $\alpha$ , we could obtain the following result:
$$
\mathbb {E} \left[ \| \nabla \widehat {L} (\theta_ {k}) \| ^ {2} \right] \leq \frac {1}{\alpha} \mathbb {E} \left[ \widehat {L} \left(\theta_ {k + 1}\right) - \widehat {L} \left(\theta_ {k}\right) \right] + 4 C _ {d} L _ {q} ^ {2} \sqrt {| \mathcal {S} | \times | \mathcal {A} |} \mathbb {E} \left[ \left\| Q _ {\theta_ {k}} - Q _ {k} \right\| _ {\infty} + \epsilon_ {\mathrm {a p p}} \right] + 2 \alpha L _ {c} L _ {q} ^ {2}. \tag {72}
$$
Let us denote the constant  $C_0 \coloneqq 4C_dL_q^2\sqrt{|\mathcal{S}|\times|\mathcal{A}|}$ . Summing (72) from  $k = 0$  to  $k = K - 1$  and dividing  $K$  on the both sides, then we obtain
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla \widehat {L} \left(\theta_ {k}\right) \| ^ {2} \right] \leq \frac {\mathbb {E} \left[ \widehat {L} \left(\theta_ {K}\right) - \widehat {L} \left(\theta_ {0}\right) \right]}{\alpha K} + \frac {C _ {0}}{K} \sum_ {k = 0} ^ {K - 1} \left[ \left\| Q _ {\theta_ {k}} - Q _ {k} \right\| _ {\infty} + \epsilon_ {\mathrm {a p p}} \right] + 2 \alpha L _ {c} L _ {q} ^ {2} \tag {73}
$$
Recall that we bound the gap between the likelihood objective and the surrogate objective in (9). Then we have
$$
| L (\theta) - \widehat {L} (\theta) | \leq \frac {\gamma C _ {v}}{1 - \gamma} \cdot \mathbb {E} _ {(s, a) \sim d ^ {\mathrm {E}} (\cdot , \cdot)} \left[ \| P (\cdot | s, a) - \widehat {P} (\cdot | s, a) \| _ {1} \right] \leq \frac {2 \gamma C _ {v}}{1 - \gamma} \tag {74}
$$
where the last inequality is due to the fact that  $\| P(\cdot |s,a) - \widehat{P} (\cdot |s,a)\| _1\leq 2$  holds for any state-action pair  $(s,a)$ . Since  $L(\theta)$  is the log-likelihood function which is always negative, we can show that the surrogate objective  $\widehat{L} (\theta)$  is upper bounded under any reward parameter  $\theta$ . Denote a positive constant  $C_1\coloneqq \frac{2\gamma C_v}{1 - \gamma}$ , we have
$$
\widehat {L} (\theta) \leq L (\theta) + \frac {2 \gamma C _ {v}}{1 - \gamma} \leq \frac {2 \gamma C _ {v}}{1 - \gamma} = C _ {1}. \tag {75}
$$
Furthermore, considering  $\widehat{L}(\theta_0)$  is the initial value of the surrogate objective, we can simply denote  $C_2 := \widehat{L}(\theta_0)$ . After plugging (62) into (73), we obtain the following result:
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla \widehat {L} (\theta_ {k}) \| ^ {2} \right] \leq \frac {C _ {1} - C _ {2}}{\alpha K} + C _ {0} \epsilon_ {\mathrm {a p p}} + \frac {C _ {0} \Delta_ {0}}{K (1 - \gamma)} + \frac {4 \alpha C _ {0} L _ {q} ^ {2}}{1 - \gamma} + \frac {2 \gamma \epsilon_ {\mathrm {a p p}} C _ {0}}{(1 - \gamma) ^ {2}} + 2 \alpha L _ {c} L _ {q} ^ {2}.
$$
Recall that the stepsize  $\alpha$  is defined as  $\alpha = \alpha_0\cdot K^{-\frac{1}{2}}$ , then we can show the order of the convergence error as
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla \widehat {L} (\theta_ {k}) \| ^ {2} \right] = \mathcal {O} (K ^ {- \frac {1}{2}}) + \mathcal {O} (\epsilon_ {\mathrm {a p p}}).
$$
This completes the entire proof.
K Proof of Theorem 3
Proof. In this section, we prove the optimality guarantee when the reward function is linearly parameterized as  $r(s,a;\theta)\coloneqq \phi (s,a)^{\top}\theta$  where  $\phi (s,a)$  is the feature vector of the state-action pair  $(s,a)$ . Based on the definition of the surrogate objective in (7) and the definition of the soft value function in (3a), we can rewrite the formulation (12) as below:
$$
\begin{array}{l} \max  _ {\theta} \widehat {L} (\theta) := \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r \left(s _ {t}, a _ {t}; \theta\right) + U \left(s _ {t}, a _ {t}\right)\right) \right] \\ \left. - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r \left(s _ {t}, a _ {t}; \theta\right) + U \left(s _ {t}, a _ {t}\right) + \mathcal {H} \left(\pi_ {\theta} (\cdot | s _ {t})\right)\right) \right] \right. \tag {76a} \\ \end{array}
$$
$$
s. t. \pi_ {\theta} := \arg \max  _ {\pi} \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r \left(s _ {t}, a _ {t}; \theta\right) + U \left(s _ {t}, a _ {t}\right) + \mathcal {H} \left(\pi \left(\cdot \mid s _ {t}\right)\right)\right) \right]. \tag {76b}
$$
As a remark, there is a common term  $\mathbb{E}_{\tau^{\mathrm{A}}\sim (\eta ,\pi ,\hat{P})}\Big[\sum_{t = 0}^{\infty}\gamma^{t}\Big(r(s_{t},a_{t};\theta) + U(s_{t},a_{t}) + \mathcal{H}\big(\pi (\cdot |s_{t})\big)\Big)\Big]$  in both (76a) and (76b). Then we can obtain the following formulation which is equivalent to (76):
$$
\begin{array}{l} \max  _ {\theta} \min  _ {\pi} \mathcal {L} (\theta , \pi) := \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r \left(s _ {t}, a _ {t}; \theta\right) + U \left(s _ {t}, a _ {t}\right)\right) \right] \\ \left. - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r \left(s _ {t}, a _ {t}; \theta\right) + U \left(s _ {t}, a _ {t}\right) + \mathcal {H} \left(\pi (\cdot \mid s _ {t})\right)\right) \right]. \right. \tag {77} \\ \end{array}
$$
Based on the equivalence between (76) and (77), we further show that any stationary point  $\tilde{\theta}$  in (76) together with its corresponding optimal policy  $\pi_{\tilde{\theta}}$  consists of a saddle point  $(\tilde{\theta},\pi_{\tilde{\theta}})$  to the problem (77) when the reward is linearly parameterized. We summarize this statement in the following claim:
Claim 1: Assume the reward function is linearly parameterized, i.e.,  $r(s,a;\theta)\coloneqq \phi (s,a)^{\top}\theta$  where  $\phi (s,a)$  is the feature vector of the state-action pair  $(s,a)$ . Any stationary point  $\tilde{\theta}$  in (76) together with its optimal policy  $\pi_{\tilde{\theta}}$  consists of a saddle point  $(\tilde{\theta},\pi_{\tilde{\theta}})$  to the problem (77).
Here, we show the proof to Claim 1 as below.
Under the linearly parameterized reward function  $r(s,a;\theta)\coloneqq \phi (s,a)^{\top}\theta$ , we can further rewrite  $\mathcal{L}(\theta ,\pi)$  as below:
$$
\mathcal {L} (\theta , \pi) := \underbrace {\left\langle \theta , \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}} , P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \phi (s _ {t} , a _ {t}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \phi (s _ {t} , a _ {t}) \right] \right\rangle}
$$
Term A: a linear function of the reward parameter  $\theta$
$$
+ \underbrace {\mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}} , P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} U (s _ {t} , a _ {t}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \Big (U (s _ {t} , a _ {t}) + \mathcal {H} (\pi (\cdot | s _ {t})) \Big) \right]}.
$$
Term B: independent of the reward parameter  $\theta$
Note that the max-min objective  $\mathcal{L}(\cdot, \cdot)$  is linear (thus concave) in the reward parameter  $\theta$  given any fixed policy  $\pi$ . We can utilize this property to prove the statement in Claim 1.
Recall that a tuple  $(\tilde{\theta},\pi_{\tilde{\theta}})$  is called a saddle point of  $\mathcal{L}(\cdot ,\cdot)$  if the following condition holds:
$$
\mathcal {L} (\theta , \pi_ {\tilde {\theta}}) \leq \mathcal {L} (\tilde {\theta}, \pi_ {\tilde {\theta}}) \leq \mathcal {L} (\tilde {\theta}, \pi) \tag {79}
$$
for any other reward parameter  $\theta$  and policy  $\pi$ . To show that  $(\tilde{\theta}, \pi_{\tilde{\theta}})$  satisfies the condition (79), we prove the following conditions respectively:
$$
\tilde {\theta} \in \arg \max  _ {\theta} \mathcal {L} (\theta , \pi_ {\tilde {\theta}}), \tag {80a}
$$
$$
\pi_ {\tilde {\theta}} \in \arg \min  _ {\pi} \mathcal {L} (\tilde {\theta}, \pi). \tag {80b}
$$
Here, we first show that any stationary point  $\tilde{\theta}$  of the surrogate objective  $\widehat{L}(\cdot)$  satisfies the optimality condition (80a). Recall the gradient expression of  $\widehat{L}(\cdot)$  in (15), any stationary point  $\tilde{\theta}$  of the surrogate objective  $\widehat{L}(\cdot)$  satisfies the following first-order condition:
$$
\nabla \widehat {L} (\tilde {\theta}) = \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \tilde {\theta}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\tilde {\theta}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \tilde {\theta}) \right] = 0
$$
Moreover, when the reward function is linearly parameterized as  $r(s,a;\theta)\coloneqq \phi (s,a)^{\top}\theta$ , we obtain the following condition for any stationary point  $\tilde{\theta}$ :
$$
\nabla \widehat {L} (\tilde {\theta}) = \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \phi \left(s _ {t}, a _ {t}\right) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\tilde {\theta}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \phi \left(s _ {t}, a _ {t}\right) \right] = 0 \tag {81}
$$
Then we can go back to the formulation  $\mathcal{L}(\cdot, \cdot)$  in (78). Given any fixed policy  $\pi$ , we can show the gradient of  $\mathcal{L}(\theta, \pi)$  w.r.t. the reward parameter  $\theta$  as below:
$$
\nabla_ {\theta} \mathcal {L} (\theta , \pi) = \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \phi \left(s _ {t}, a _ {t}\right) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \phi \left(s _ {t}, a _ {t}\right) \right]. \tag {82}
$$
Due to the first-order condition we show in (81), we obtain the following result:
$$
\nabla_ {\theta} \mathcal {L} (\theta = \tilde {\theta}, \pi = \pi_ {\tilde {\theta}}) = 0. \tag {83}
$$
Recall that we have shown  $\mathcal{L}(\cdot, \cdot)$  is concave in terms of the reward parameter  $\theta$  given any fixed policy  $\pi$ . Due to the concavity as shown in (78) and the condition in (83), we have completed the proof of the optimality condition (80a).
Then we prove the optimality condition (80b). Recall that  $\pi_{\tilde{\theta}}$  is the optimal policy defined in (76b) under the reward parameter  $\theta$ . By observing the objective  $\mathcal{L}(\cdot, \cdot)$ , we obtain the following result: