SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \nabla \widehat {L} \left(\theta_ {1}\right) - \nabla \widehat {L} \left(\theta_ {2}\right) \right\| \\ \stackrel {(i)} {:=} \left\| \right.\left(\mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {1}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) \right]\right) \\ \left. \right. - \left(\mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {2}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {2}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {2}) \right]\right) \| \\ \leq \left\| \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) \right] - \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {2}) \right] \right\| \\ + \left\| \mathbb {E} _ {\tau^ {\mathrm {A}} \sim \left(\eta , \pi_ {\theta_ {1}}, \widehat {P}\right)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r \left(s _ {t}, a _ {t}; \theta_ {1}\right) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim \left(\eta , \pi_ {\theta_ {2}}, \widehat {P}\right)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r \left(s _ {t}, a _ {t}; \theta_ {2}\right) \right] \right\| \tag {38} \\ \end{array}
$$
where (i) follows the gradient expression in (15). For the first term in (38), the following series of relations holds:
$$
\begin{array}{l} \left\| \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) \right] - \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {2}) \right] \right\| \\ = \left\| \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(\nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) - \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {2})\right) \right] \right\| \\ \stackrel {(i)} {\leq} \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left\| \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) - \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {2}) \right\| \right] \\ \stackrel {(i i)} {\leq} \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} L _ {g} \left\| \theta_ {1} - \theta_ {2} \right\| \right] \\ = \frac {L _ {g}}{1 - \gamma} \left\| \theta_ {1} - \theta_ {2} \right\| \tag {39} \\ \end{array}
$$
where (i) follows Jensen's inequality and (ii) follows (18) from the Assumption 3. For the second term in (38), we have
$$
\left\| \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {1}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {2}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {2}) \right] \right\|
$$
$$
\leq \underbrace {\left\| \mathbb {E} _ {\tau^ {\Lambda} \sim (\eta , \pi_ {\theta_ {1}} , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t} , a _ {t} ; \theta_ {1}) \right] - \mathbb {E} _ {\tau^ {\Lambda} \sim (\eta , \pi_ {\theta_ {2}} , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t} , a _ {t} ; \theta_ {1}) \right] \right\|}
$$
T1: error term due to policy mismatch
$$
+ \underbrace {\left\| \mathbb {E} _ {\tau^ {\Lambda} \sim (\eta , \pi_ {\theta_ {2}} , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r \left(s _ {t} , a _ {t} ; \theta_ {1}\right) \right] - \mathbb {E} _ {\tau^ {\Lambda} \sim (\eta , \pi_ {\theta_ {2}} , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r \left(s _ {t} , a _ {t} ; \theta_ {2}\right) \right] \right\|}. \tag {40}
$$
T2: error term due to reward parameter mismatch
In (40), we decompose the difference between reward gradient trajectories into two error terms. The first error term is due to the policy mismatch between  $\pi_{\theta_1}$  and  $\pi_{\theta_2}$ . The second error term is due to
the reward parameter mismatch between  $\theta_{1}$  and  $\theta_{2}$ . Here, we first bound the error term T1 in (40):
$$
\begin{array}{l} \left\| \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {1}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {2}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) \right] \right\| \\ \stackrel {(i)} {=} \left\| \frac {1}{1 - \gamma} \mathbb {E} _ {(s, a) \sim d _ {\hat {P}} ^ {\pi_ {\theta_ {1}}} (\cdot , \cdot)} \left[ \nabla_ {\theta} r (s, a; \theta_ {1}) \right] - \frac {1}{1 - \gamma} \mathbb {E} _ {(s, a) \sim d _ {\hat {P}} ^ {\pi_ {\theta_ {2}}} (\cdot , \cdot)} \left[ \nabla_ {\theta} r (s, a; \theta_ {1}) \right] \right\| \\ = \frac {1}{1 - \gamma} \left\| \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} \nabla_ {\theta} r (s, a; \theta_ {1}) \cdot \left(d _ {\widehat {P}} ^ {\pi_ {\theta_ {1}}} (s, a) - d _ {\widehat {P}} ^ {\pi_ {\theta_ {2}}} (s, a)\right) \right\| \\ \leq \frac {1}{1 - \gamma} \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} \left\| \nabla_ {\theta} r (s, a; \theta_ {1}) \right\| \cdot \left| d _ {\widehat {P}} ^ {\pi_ {\theta_ {1}}} (s, a) - d _ {\widehat {P}} ^ {\pi_ {\theta_ {2}}} (s, a) \right| \\ \stackrel {(i i)} {\leq} \frac {2 L _ {r}}{1 - \gamma} \| d _ {\widehat {P}} ^ {\pi_ {\theta_ {1}}} (\cdot , \cdot) - d _ {\widehat {P}} ^ {\pi_ {\theta_ {2}}} (\cdot , \cdot) \| _ {\mathrm {T V}} \\ \stackrel {(i i i)} {\leq} \frac {2 L _ {r} C _ {d}}{1 - \gamma} \| Q _ {\theta_ {1}} - Q _ {\theta_ {2}} \| \\ \stackrel {(i v)} {\leq} \frac {2 L _ {r} C _ {d} \sqrt {| \mathcal {S} | \cdot | \mathcal {A} |}}{1 - \gamma} \| Q _ {\theta_ {1}} - Q _ {\theta_ {2}} \| _ {\infty} \\ \stackrel {(v)} {\leq} \frac {2 L _ {q} L _ {r} C _ {d} \sqrt {| \mathcal {S} | \cdot | \mathcal {A} |}}{1 - \gamma} \| \theta_ {1} - \theta_ {2} \| \tag {41} \\ \end{array}
$$
where (i) is from the definition of the visitation measure and (ii) follows the bounded gradient of reward function in (18) in Assumption 3. Moreover, (iii) is from (25) in Lemma 6 and (v) follows the Lipschitz property (19a) in Lemma 4. For the inequality (iv), it holds due to the fact  $|Q_{\theta_1}(s,a) - Q_{\theta_2}(s,a)| \leq \| Q_{\theta_1} - Q_{\theta_2}\|_\infty$  for any state-action pair  $(s,a)$  and thus  $\| Q_{\theta_1} - Q_{\theta_2}\| \leq \sqrt{|\mathcal{S}| \cdot |\mathcal{A}|} \| Q_{\theta_1} - Q_{\theta_2}\|_\infty$ .
Next, let us bound the second error term in (40):
$$
\begin{array}{l} \left\| \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {2}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {2}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {2}) \right] \right\| \\ = \left\| \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {2}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(\nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) - \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {2})\right) \right] \right\| \\ \stackrel {(i)} {\leq} \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {2}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left\| \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) - \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {2}) \right\| \right] \\ \stackrel {(i i)} {\leq} \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {2}}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} L _ {g} \| \theta_ {1} - \theta_ {2} \| \right] \\ = \frac {L _ {g}}{1 - \gamma} \| \theta_ {1} - \theta_ {2} \| \tag {42} \\ \end{array}
$$
where (i) follows Jensen's inequality and (ii) follows the Lipschitz property in (18) from Assumption 3. Plugging (41) and (42) into (40), we could show the following result:
$$
\left\| \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {1}}, \hat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {1}) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi_ {\theta_ {2}}, \hat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \nabla_ {\theta} r (s _ {t}, a _ {t}; \theta_ {2}) \right] \right\| \leq \frac {2 L _ {q} L _ {r} C _ {d} \sqrt {| S | \cdot | \mathcal {A} |} + L _ {g}}{1 - \gamma} \cdot \| \theta_ {1} - \theta_ {2} \|. \tag {43}
$$
We denote the constant  $L_{c} = \frac{2L_{q}L_{r}C_{d}\sqrt{|S|\cdot|\mathcal{A}| + 2L_{g}}}{1 - \gamma}$ . By plugging (39) and (43) into (38), we arrive at the desired Lipschitz property of the surrogate objective  $\widehat{L}(\theta)$ , as shown below:
$$
\left\| \nabla \widehat {L} \left(\theta_ {1}\right) - \nabla \widehat {L} \left(\theta_ {2}\right) \right\| \leq L _ {c} \| \theta_ {1} - \theta_ {2} \|.
$$
We completed the proof of the Inequality (19b).
H Proof of Proposition 1
Proof. Before starting the proof, first recall that we have defined the expert-visited state-action space  $\Omega := \{(s,a) \mid d^{\mathrm{E}}(s,a) > 0\}$  and the expert-visited state space  $\mathcal{S}^{\mathrm{E}} := \{s \mid \sum_{a \in \mathcal{A}} d^{\mathrm{E}}(s,a) > 0\}$ . Then we have the following analysis.
For all state-action pairs  $(s,a)\in \Omega$ , there are at most  $|\mathcal{S}^{\mathrm{E}}|$  active states in the distribution  $P(\cdot |s,a)$  such that  $P(s'|s,a) > 0$ . This is due to the fact that the next state  $s'$  of an expert-visited state-action pair  $(s,a)$  must belong to the expert-visited state space  $\mathcal{S}^{\mathrm{E}}$ . Then suppose there are  $N$  i.i.d. samples on each state-action pair  $(s,a)\in \Omega$ , we can obtain an empirical estimate of the transition probability as  $\widehat{P}(s'|s,a) = \frac{N(s,a,s')}{N}$  for any  $s' \in \mathcal{S}^{\mathrm{E}}$ , where  $N(s,a,s')$  is the number of observed transition samples  $(s,a,s')$ . For any  $(s,a)\in \Omega$  and  $s' \notin \mathcal{S}^{\mathrm{E}}$ , we know  $\widehat{P}(s'|s,a) = \frac{N(s,a,s')}{N} = 0$ , since the expert only visits states in  $\mathcal{S}^{\mathrm{E}}$ .
Therefore, at any state-action pair  $(s,a)\in \Omega$  and by following (24) in Lemma 5, the following result holds with probability greater than  $1 - e^{-N\epsilon^2}$ :
$$
\left\| P (\cdot \mid s, a) - \widehat {P} (\cdot \mid s, a) \right\| _ {1} \leq \sqrt {\left| \mathcal {S} ^ {\mathrm {E}} \right|} \cdot \left(\frac {1}{\sqrt {N}} + \epsilon\right). \tag {44}
$$
Let us denote  $\tilde{\delta} \coloneqq e^{-N\epsilon^2}$ , then we have  $\epsilon = \sqrt{\frac{1}{N}\ln\frac{1}{\tilde{\delta}}}$ . By plugging  $\epsilon$  into (44), we can show that the following result holds with probability greater than  $1 - \tilde{\delta}$ :
$$
\| P (\cdot | s, a) - \widehat {P} (\cdot | s, a) \| _ {1} \leq \sqrt {\left| \mathcal {S} ^ {\mathrm {E}} \right|} \cdot \left(\frac {1}{\sqrt {N}} + \sqrt {\frac {1}{N} \ln \frac {1}{\tilde {\delta}}}\right) = c \sqrt {\frac {\left| \mathcal {S} ^ {\mathrm {E}} \right|}{N} \ln \frac {1}{\tilde {\delta}}}. \tag {45}
$$
where we define  $c \coloneqq 1 + \frac{1}{\sqrt{\ln\frac{1}{\delta}}}$  as a positive constant dependent on the probability  $\tilde{\delta}$ . Then we can further show that
$$
P \Big (\max  _ {(s, a) \in \Omega} \| P (\cdot | s, a) - \widehat {P} (\cdot | s, a) \| _ {1} \geq c \sqrt {\frac {| \mathcal {S} ^ {\mathrm {E}} |}{N} \ln \frac {1}{\tilde {\delta}}} \Big)
$$
$$
\stackrel {(i)} {\leq} \sum_ {(s, a) \in \Omega} P \left(\| P (\cdot | s, a) - \widehat {P} (\cdot | s, a) \| _ {1} \geq c \sqrt {\frac {| \mathcal {S} ^ {\mathrm {E}} |}{N} \ln \frac {1}{\bar {\delta}}}\right)
$$
$$
\stackrel {(i i)} {\leq} | \Omega | \cdot \tilde {\delta}
$$
where (i) follows the union bound and (ii) follows (45). Therefore, with probability greater than  $1 - |\Omega| \cdot \tilde{\delta}$ , we have
$$
\max  _ {(s, a) \in \Omega} \| P (\cdot | s, a) - \widehat {P} (\cdot | s, a) \| _ {1} \leq c \sqrt {\frac {\left| S ^ {E} \right|}{N} \ln \frac {1}{\tilde {\delta}}}. \tag {46}
$$
Denoting  $\delta \coloneqq |\Omega| \cdot \tilde{\delta}$ , then we have  $\tilde{\delta} = \frac{\delta}{|\Omega|}$ . Therefore, with probability greater than  $1 - \delta$ , the following result holds
$$
\max  _ {(s, a) \in \Omega} \| P (\cdot | s, a) - \widehat {P} (\cdot | s, a) \| _ {1} \leq c \sqrt {\frac {| S ^ {\mathrm {E}} |}{N} \ln \left(\frac {| \Omega |}{\delta}\right)}. \tag {47}
$$
In order to control the estimation error  $\max_{(s,a) \in \Omega} \| P(\cdot | s, a) - \widehat{P}(\cdot | s, a) \|_1 \leq \varepsilon$ , the number of samples  $N$  at each state-action pair  $(s, a) \in \Omega$  should satisfy:
$$
c \sqrt {\frac {| \mathcal {S} ^ {\mathrm {E}} |}{N} \ln \left(\frac {| \Omega |}{\delta}\right)} \leq \varepsilon \Rightarrow N \geq \frac {c ^ {2} \cdot | \mathcal {S} ^ {\mathrm {E}} |}{\varepsilon^ {2}} \ln \left(\frac {| \Omega |}{\delta}\right).
$$
Suppose we uniformly sample each expert-visited state-action pair  $(s, a) \in \Omega$  and the total number of samples in the transition dataset  $\mathcal{D} = \{(s, a, s')\}$  satisfies:
$$
\# \text {t r a n s i t i o n} \geq | \Omega | \cdot N \geq \frac {c ^ {2} \cdot | \Omega | \cdot | \mathcal {S} ^ {\mathrm {E}} |}{\varepsilon^ {2}} \ln \left(\frac {| \Omega |}{\delta}\right). \tag {48}
$$
Then with probability greater than  $1 - \delta$ , the generalization error between  $P$  and  $\hat{P}$  could be controlled:
$$
\mathbb {E} _ {(s, a) \sim d ^ {\mathrm {E}} (\cdot , \cdot)} \left[ \| P (\cdot | s, a) - \widehat {P} (\cdot | s, a) \| _ {1} \right] \leq \max  _ {(s, a) \in \Omega} \| P (\cdot | s, a) - \widehat {P} (\cdot | s, a) \| _ {1} \leq \varepsilon . \tag {49}
$$
The theorem is proved.
Proof. First, we have the following decomposition of the error between the loss function evaluated at  $\theta^{*}$  and  $\hat{\theta}$ , respectively:
$$
\begin{array}{l} L (\theta^ {*}) - L (\hat {\theta}) \\ = \left(L \left(\theta^ {*}\right) - \widehat {L} \left(\theta^ {*}\right)\right) + \left(\widehat {L} \left(\theta^ {*}\right) - \widehat {L} (\hat {\theta})\right) + \left(\widehat {L} (\hat {\theta}) - L (\hat {\theta})\right) \\ \stackrel {(i)} {\leq} \frac {\gamma C _ {v}}{1 - \gamma} \cdot \mathbb {E} _ {(s, a) \sim d ^ {\mathbb {E}} (\cdot , \cdot)} \left[ \left\| \widehat {P} (\cdot | s, a) - P (\cdot | s, a) \right\| _ {1} \right] + \left(\widehat {L} (\theta^ {*}) - \widehat {L} (\widehat {\theta})\right) + \frac {\gamma C _ {v}}{1 - \gamma} \cdot \mathbb {E} _ {(s, a) \sim d ^ {\mathbb {E}} (\cdot , \cdot)} \left[ \left\| \widehat {P} (\cdot | s, a) - P (\cdot | s, a) \right\| _ {1} \right] \\ = \frac {2 \gamma C _ {v}}{1 - \gamma} \cdot \mathbb {E} _ {(s, a) \sim d ^ {\mathrm {E}} (\cdot , \cdot)} \left[ \| \widehat {P} (\cdot | s, a) - P (\cdot | s, a) \| _ {1} \right] + \left(\widehat {L} \left(\theta^ {*}\right) - \widehat {L} (\hat {\theta})\right) \tag {50} \\ \end{array}
$$
where (i) follows (9). Since we have defined  $\hat{\theta}$  as the optimal solution to  $\widehat{L}(\cdot)$ , we know that  $\widehat{L}(\theta) - \widehat{L}(\hat{\theta}) \leq 0$  for any  $\theta$ . Plugging this result into (50), the following result holds:
$$
\begin{array}{l} L (\theta^ {*}) - L (\widehat {\theta}) \leq \frac {2 \gamma C _ {v}}{1 - \gamma} \cdot \mathbb {E} _ {(s, a) \sim d ^ {\mathbb {E}} (\cdot , \cdot)} \left[ \left\| \widehat {P} (\cdot | s, a) - P (\cdot | s, a) \right\| _ {1} \right] + \left(\widehat {L} (\theta^ {*}) - \widehat {L} (\widehat {\theta})\right) \\ \leq \frac {2 \gamma C _ {v}}{1 - \gamma} \cdot \mathbb {E} _ {(s, a) \sim d ^ {\mathbb {E}} (\cdot , \cdot)} \left[ \left\| \widehat {P} (\cdot | s, a) - P (\cdot | s, a) \right\| _ {1} \right]. \\ \end{array}
$$
Based on (48) and (49), we assume that each expert-visited state-action pair is uniformly sampled and the total number of transition samples satisfies:
$$
\# \text {t r a n s i t i o n s a m p l e s} \geq \frac {c ^ {2} \cdot | \Omega | \cdot | \mathcal {S} ^ {\mathrm {E}} |}{\left(\frac {(1 - \gamma) \varepsilon}{2 \gamma C _ {v}}\right) ^ {2}} \ln \left(\frac {| \Omega |}{\delta}\right) = \frac {4 \gamma^ {2} \cdot C _ {v} ^ {2} \cdot c ^ {2} \cdot | \Omega | \cdot | \mathcal {S} ^ {\mathrm {E}} |}{(1 - \gamma) ^ {2} \varepsilon^ {2}} \ln \left(\frac {| \Omega |}{\delta}\right).
$$
Then with probability greater than  $1 - \delta$ , the generalization error between transition dynamics and the optimality gap between reward estimates could be controlled:
$$
\mathbb {E} _ {(s, a) \sim d ^ {\mathbb {E}} (\cdot , \cdot)} \left[ \left\| \widehat {P} (\cdot | s, a) - P (\cdot | s, a) \right\| _ {1} \right] \leq \frac {(1 - \gamma) \varepsilon}{2 \gamma C _ {v}},
$$
$$
L (\theta^ {*}) - L (\hat {\theta}) \leq \frac {2 \gamma C _ {v}}{1 - \gamma} \cdot \mathbb {E} _ {(s, a) \sim d ^ {\mathbb {E}} (\cdot , \cdot)} \left[ \left\| \widehat {P} (\cdot | s, a) - P (\cdot | s, a) \right\| _ {1} \right] \leq \varepsilon .
$$
J Proof of Theorem 2
Proof. In this section, we prove the convergence results (20a) - (20b) respectively.
J.1 Proof of convergence of the policy estimates in (20a)
In this section, we show the convergence result of the policy estimates  $\{\pi_{k + 1}\}_{k\geq 0}$ , which track the optimal solutions  $\{\pi_{\theta_k}\}_{k\geq 0}$ . Recall that each policy  $\pi_{k + 1}$  is generated from the soft policy iteration in (14), then we track the approximation error between  $\pi_{k + 1}$  and  $\pi_{\theta_k}$  as below:
$$
\begin{array}{l} \left| \log \pi_ {k + 1} (a | s) - \log \pi_ {\theta_ {k}} (a | s) \right| \\ \stackrel {(i)} {=} \left| \log \left(\frac {\exp \widehat {Q} _ {k} (s , a)}{\sum_ {\tilde {a} \in \mathcal {A}} \exp \widehat {Q} _ {k} (s , \tilde {a})}\right) - \log \left(\frac {\exp Q _ {\theta_ {k}} (s , a)}{\sum_ {\tilde {a} \in \mathcal {A}} \exp Q _ {\theta_ {k}} (s , \tilde {a})}\right) \right| \\ = \left| \left(\widehat {Q} _ {k} (s, a) - \log \left(\sum_ {\tilde {a} \in \mathcal {A}} \exp \widehat {Q} _ {k} (s, \tilde {a})\right)\right) - \left(Q _ {\theta_ {k}} (s, a) - \log \left(\sum_ {\tilde {a} \in \mathcal {A}} \exp Q _ {\theta_ {k}} (s, \tilde {a})\right)\right) \right| \\ \leq \left| \widehat {Q} _ {k} (s, a) - Q _ {\theta_ {k}} (s, a) \right| + \left| \log \left(\sum_ {\tilde {a} \in \mathcal {A}} \exp \widehat {Q} _ {k} (s, \tilde {a})\right) - \log \left(\sum_ {\tilde {a} \in \mathcal {A}} \exp Q _ {\theta_ {k}} (s, \tilde {a})\right) \right| \tag {51} \\ \end{array}
$$
where (i) follows (14) and (4). In order to analyze the second error term in (51), we first denote two  $|\mathcal{A}|$ -dimensional vectors  $\overrightarrow{a} \coloneqq [a_1, a_2, \dots, a_{|\mathcal{A}|}]$  and  $\overrightarrow{b} \coloneqq [b_1, b_2, \dots, b_{|\mathcal{A}|}]$ . Then we could obtain the following result:
$$
\begin{array}{l} \left| \log \left(\| \exp (\vec {a}) \| _ {1}\right) - \log \left(\| \exp (\vec {b}) \| _ {1}\right) \right| \stackrel {(i)} {=} \left| \langle \vec {a} - \vec {b}, \nabla_ {\vec {v}} \log \left(\| \exp (\vec {v}) \| _ {1}\right) \rangle \right| \\ \leq \left\| \overrightarrow {a} - \overrightarrow {b} \right\| _ {\infty} \cdot \left\| \nabla_ {\overrightarrow {v}} \log \left(\left\| \exp (\overrightarrow {v}) \right\| _ {1}\right) \right\| _ {1} \\ \stackrel {(i i)} {=} \| \overrightarrow {a} - \overrightarrow {b} \| _ {\infty} \tag {52} \\ \end{array}
$$
where (i) follows the mean value theorem and  $\vec{v}$  is a convex combination between vectors  $\vec{a}$  and  $\vec{b}$ . Moreover, (ii) is due to the equality that
$$
[ \nabla_ {\overrightarrow {v}} \log \big (\| \exp (\overrightarrow {v}) \| _ {1} \big) ] _ {i} = \frac {\exp (v _ {i})}{\| \exp (\overrightarrow {v}) \| _ {1}}, \quad \| \nabla_ {\overrightarrow {v}} \log \big (\| \exp (\overrightarrow {v}) \| _ {1} \big) \| _ {1} = 1, \quad \forall \overrightarrow {v} \in \mathbb {R} ^ {| \mathcal {A} |}.
$$
Based on the property we show in (52), we could further analyze (51) as below:
$$
\begin{array}{l} \left| \log \pi_ {k + 1} (a | s) - \log \pi_ {\theta_ {k}} (a | s) \right| \leq \left| \widehat {Q} _ {k} (s, a) - Q _ {\theta_ {k}} (s, a) \right| + \left| \log \left(\sum_ {\tilde {a} \in \mathcal {A}} \exp \widehat {Q} _ {k} (s, \tilde {a})\right) - \log \left(\sum_ {\tilde {a} \in \mathcal {A}} \exp Q _ {\theta_ {k}} (s, \tilde {a})\right) \right| \\ \leq \left| \widehat {Q} _ {k} (s, a) - Q _ {\theta_ {k}} (s, a) \right| + \max  _ {\tilde {a} \in \mathcal {A}} \left| \widehat {Q} _ {k} (s, \tilde {a}) - Q _ {\theta_ {k}} (s, \tilde {a}) \right|. \tag {53} \\ \end{array}
$$
If we take the maximum over all state-action pairs on the both sides of (53) and denote  $\| \log \pi \|_{\infty} \coloneqq \max_{s \in S, a \in A} |\log \pi(a|s)|$ , then we obtain the following property:
$$
\left\| \log \pi_ {k + 1} - \log \pi_ {\theta_ {k}} \right\| _ {\infty} \leq 2 \left\| \widehat {Q} _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty}. \tag {54}
$$
Recall that  $\widehat{Q}_k$  is an approximation to the soft Q-function  $Q_{k}$  where  $\epsilon_{\mathrm{app}}$  is the approximation error, then we have:
$$
\left\| \log \pi_ {k + 1} - \log \pi_ {\theta_ {k}} \right\| _ {\infty} \leq 2 \left\| \widehat {Q} _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty} = 2 \left\| \widehat {Q} _ {k} - Q _ {k} + Q _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty} \leq 2 \epsilon_ {\mathrm {a p p}} + 2 \left\| Q _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty}. \tag {55}
$$
In order to track the approximation error  $\| \log \pi_{k + 1} - \log \pi_{\theta_k}\|_{\infty}$ , we could analyze the convergence between the  $Q_{k}$  and  $Q_{\theta_k}$  according to (55). Here, we define an auxiliary sequence  $\{Q_{k + \frac{1}{2}}\}_{k\geq 0}$  in the conservative MDP where  $Q_{k + \frac{1}{2}}$  is the soft Q-function under the reward parameter  $\theta_{k}$  and the policy  $\pi_{k + 1}$  defined in (14). Then we have the following analysis:
$$
\begin{array}{l} \left\| Q _ {k} - Q _ {\theta_ {k}} \right\| _ {\infty} = \left\| Q _ {k} - Q _ {\theta_ {k}} + Q _ {\theta_ {k - 1}} - Q _ {\theta_ {k - 1}} + Q _ {k - \frac {1}{2}} - Q _ {k - \frac {1}{2}} \right\| _ {\infty} \\ \leq \left\| Q _ {\theta_ {k}} - Q _ {\theta_ {k - 1}} \right\| _ {\infty} + \left\| Q _ {\theta_ {k - 1}} - Q _ {k - \frac {1}{2}} \right\| _ {\infty} + \left\| Q _ {k} - Q _ {k - \frac {1}{2}} \right\| _ {\infty} \\ \stackrel {(i)} {\leq} \left\| Q _ {\theta_ {k - 1}} - Q _ {k - \frac {1}{2}} \right\| _ {\infty} + 2 L _ {q} \| \theta_ {k} - \theta_ {k - 1} \| \\ \leq^ {(i i)} \gamma \left\| Q _ {\theta_ {k - 1}} - Q _ {k - 1} \right\| _ {\infty} + 2 L _ {q} \| \theta_ {k} - \theta_ {k - 1} \| + \frac {2 \gamma \epsilon_ {\mathrm {a p p}}}{1 - \gamma} \tag {56} \\ \end{array}
$$
where (i) follows (19a) in Lemma 4 and (26) in Lemma 7. Moreover, the inequality (ii) follows (27b) in Lemma 8. Recall that the update rule of the reward parameter  $\theta$  is defined in (17), then the following result holds: