SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {L} (\theta , \pi) := \underbrace {\mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}} , P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r (s _ {t} , a _ {t} ; \theta) + U (s _ {t} , a _ {t})\right) \right]} _ {\text {T e r m I} _ {1}: \text {i n d e p e n d e n t o f} \pi} \\ - \underbrace {\mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r \left(s _ {t} , a _ {t} ; \theta\right) + U \left(s _ {t} , a _ {t}\right) + \mathcal {H} \left(\pi \left(\cdot \mid s _ {t}\right)\right)\right) \right]} _ {\text {T e r m I} _ {2}: \text {a f u n c t i o n o f t h e p o l i c y} \pi}. \tag {84} \\ \end{array}
$$
Here, we can observe that the second term in (84) is exactly the objective in (76b). Moreover, since  $\pi_{\tilde{\theta}}$  is the optimal policy defined in (76b) under reward parameter  $\tilde{\theta}$ , we obtain the result that
$$
\pi_ {\tilde {\theta}} \in \arg \min  _ {\pi} \mathcal {L} (\tilde {\theta}, \pi)
$$
which completes the proof to show the optimality condition (80b). Hence, we obtain that any stationary point  $\tilde{\theta}$  of the surrogate objective  $\widehat{L}(\cdot, \cdot)$  together with its optimal policy  $\pi_{\tilde{\theta}}$  is a saddle point of  $\mathcal{L}(\cdot, \cdot)$ .
Therefore, we complete the proof of Claim 1.
Next, we show that for any saddle point of (77), its reward parameter is a globally optimal solution to (76). We summarize this statement in the following claim:
Claim 2: For any saddle point  $(\tilde{\theta}, \pi_{\tilde{\theta}})$  of the saddle point problem (77), the reward parameter  $\tilde{\theta}$  is a globally optimal solution to the surrogate objective  $\widehat{L}(\cdot)$  in (76).
Here, we start to show the proof to Claim 2 as below.
Given a saddle point  $(\tilde{\theta},\pi_{\tilde{\theta}})$  of  $\mathcal{L}(\cdot ,\cdot)$  defined in (77), we have the following property that
$$
\min  _ {\pi} \max  _ {\theta} \mathcal {L} (\theta , \pi) \leq \max  _ {\theta} \mathcal {L} (\theta , \pi_ {\tilde {\theta}}) \stackrel {(i)} {=} \mathcal {L} (\tilde {\theta}, \pi_ {\tilde {\theta}}) \stackrel {(i i)} {=} \min  _ {\pi} \mathcal {L} (\tilde {\theta}, \pi) \leq \max  _ {\theta} \min  _ {\pi} \mathcal {L} (\theta , \pi) \tag {85}
$$
where  $(i)$  follows the optimality condition (80a) and  $(ii)$  follows the optimality condition (80b). According to the minimax inequality, we always have the following condition that
$$
\max  _ {\theta} \min  _ {\pi} \mathcal {L} (\theta , \pi) \leq \min  _ {\pi} \max  _ {\theta} \mathcal {L} (\theta , \pi). \tag {86}
$$
Through combining the saddle point inequality (85) and the minimax inequality (86), we obtain the following equality:
$$
\min  _ {\pi} \max  _ {\theta} \mathcal {L} (\theta , \pi) = \max  _ {\theta} \mathcal {L} (\theta , \pi_ {\tilde {\theta}}) = \mathcal {L} (\tilde {\theta}, \pi_ {\tilde {\theta}}) = \min  _ {\pi} \mathcal {L} (\tilde {\theta}, \pi) = \max  _ {\theta} \min  _ {\pi} \mathcal {L} (\theta , \pi). \tag {87}
$$
Therefore, for any saddle point  $(\tilde{\theta},\pi_{\tilde{\theta}})$ , we obtain the following property of the reward parameter  $\tilde{\theta}$  and the corresponding policy  $\pi_{\tilde{\theta}}$ :
$$
\tilde {\theta} \in \arg \max  _ {\theta} \min  _ {\pi} \mathcal {L} (\theta , \pi), \tag {88a}
$$
$$
\pi_ {\tilde {\theta}} \in \arg \min  _ {\pi} \max  _ {\theta} \mathcal {L} (\theta , \pi). \tag {88b}
$$
Due to the expression of the surrogate objective  $\widehat{L} (\cdot)$  in (76) and the objective  $\mathcal{L}(\cdot ,\cdot)$  in (77), we have the following equality holds for any reward parameter  $\theta$
$$
\widehat {L} (\theta) = \min  _ {\pi} \mathcal {L} (\theta , \pi). \tag {89}
$$
Combining (88a) and (89), we obtain the following result:
$$
\tilde {\theta} \in \arg \max  _ {\theta} \min  _ {\pi} \mathcal {L} (\theta , \pi) = \arg \max  _ {\theta} \widehat {L} (\theta). \tag {90}
$$
According to the property we shown in (90), we obtain that for any saddle point  $(\tilde{\theta},\pi_{\tilde{\theta}})$  of  $\mathcal{L}(\cdot ,\cdot)$ , the reward parameter  $\tilde{\theta}$  is a globally optimal solution of the surrogate objective  $\widehat{L} (\cdot)$  in (76).
Therefore, we complete the entire proof of Claim 2.
Through combining Claim 1 - Claim 2, we obtain that when the reward function is linearly parameterized, any stationary point of the surrogate problem (76) is a global optimum.
Let's denote the optimal reward parameters associated with  $L(\cdot)$  and  $\widehat{L} (\cdot)$  as below, respectively:
$$
\theta^ {*} \in \arg \max  _ {\theta} L (\theta), \quad \hat {\theta} \in \arg \max  _ {\theta} \widehat {L} (\theta). \tag {91}
$$
Recall that in Theorem 1, we have shown for any  $\epsilon \in (0,2)$ , suppose there are more than  $N$  data points on each state-action pair  $(s,a) \in \Omega$  and the number of transition dataset  $\mathcal{D}$  satisfies:
$$
\# \text {t r a n s i t i o n} \geq | \Omega | \cdot N \geq \frac {4 \gamma^ {2} \cdot C _ {v} ^ {2} \cdot c ^ {2} \cdot | \Omega | \cdot | \mathcal {S} ^ {\mathrm {E}} |}{(1 - \gamma) ^ {2} \varepsilon^ {2}} \ln \left(\frac {| \Omega |}{\delta}\right) \tag {92}
$$
$$
L (\theta^ {*}) - L (\hat {\theta}) \leq \varepsilon .
$$
Through combining Claim 1 - Claim 2, we have already shown that any stationary point  $\tilde{\theta}$  is a global optimum of the surrogate problem (76) when the reward function is linearly parameterized. Therefore, we obtain that when the number of transition samples satisfies (92), any stationary point  $\tilde{\theta}$  of the surrogate problem (76) is an epsilon-optimal solution to the maximum likelihood estimation problem (2). For any stationary point  $\tilde{\theta}$  of the surrogate problem (76), with probability greater than  $1 - \delta$ , it holds that
$$
L (\theta^ {*}) - L (\tilde {\theta}) \leq \varepsilon .
$$
This completes the entire proof of Theorem 3.