SlowGuess/NeurIPS_2023_OCR huggingface dataset:
According to Assumption 1, we have that
$$
\begin{array}{l} d _ {h} ^ {\mathrm {U}} (s, \pi_ {h} ^ {\mathrm {E}} (s)) = \eta d _ {h} ^ {\pi^ {\mathrm {E}}} (s, \pi_ {h} ^ {\mathrm {E}} (s)) + (1 - \eta) d _ {h} ^ {\pi^ {\beta}} (s, \pi_ {h} ^ {\mathrm {E}} (s)) \\ \stackrel {(a)} {\geq} \eta d _ {h} ^ {\pi^ {\mathrm {E}}} (s, \pi_ {h} ^ {\mathrm {E}} (s)) + \frac {(1 - \eta)}{\mu} d _ {h} ^ {\pi^ {\mathrm {E}}} (s, \pi_ {h} ^ {\mathrm {E}} (s)) \\ = \left(\eta + \frac {(1 - \eta)}{\mu}\right) d _ {h} ^ {\pi^ {\mathrm {E}}} (s, \pi_ {h} ^ {\mathrm {E}} (s)). \\ \end{array}
$$
Inequality  $(a)$  follows the definition of  $\mu$  in Theorem 3: for any  $(s,h)\in S\times [H]$ , we have  $d_h^{\pi^{\mathrm{E}}}(s,\pi_h^{\mathrm{E}}(s)) / d_h^{\pi^\beta}(s,\pi_h^{\mathrm{E}}(s))\leq \mu$ . Then we obtain that
$$
\begin{array}{l} \mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {I S W - B C}}) \right] \leq H \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} d _ {h} ^ {\pi^ {\mathrm {E}}} (s) (1 - d _ {h} ^ {\mathrm {U}} (s, \pi_ {h} ^ {\mathrm {E}} (s))) ^ {N _ {\mathrm {t o t}}} \\ \leq \left(\frac {1}{\eta + (1 - \eta) / \mu}\right) H \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} d _ {h} ^ {\mathrm {U}} (s, \pi_ {h} ^ {\mathrm {E}} (s)) \mathbb {P} \left((s, \pi_ {h} ^ {\mathrm {E}} (s)) \notin \mathcal {D} _ {h} ^ {\mathrm {U}}\right). \\ \end{array}
$$
For each  $(s,h)\in S\times [H]$  , we observe that
$$
d _ {h} ^ {\mathrm {U}} (s, \pi_ {h} ^ {\mathrm {E}} (s)) \mathbb {P} \left((s, \pi_ {h} ^ {\mathrm {E}} (s)) \notin \mathcal {D} _ {h} ^ {\mathrm {U}}\right) = d _ {h} ^ {\mathrm {U}} (s, \pi_ {h} ^ {\mathrm {E}} (s)) \left(1 - d _ {h} ^ {\mathrm {U}} (s, \pi_ {h} ^ {\mathrm {E}} (s))\right) ^ {N _ {\mathrm {t o t}}} \leq \frac {4}{9 N _ {\mathrm {t o t}}}.
$$
Here the last inequality follows Lemma 5. Consequently, we can derive that
$$
\sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} d _ {h} ^ {\mathrm {U}} (s, \pi_ {h} ^ {\mathrm {E}} (s)) \mathbb {P} \left((s, \pi_ {h} ^ {\mathrm {E}} (s)) \notin \mathcal {D} _ {h} ^ {\mathrm {U}}\right) \leq \frac {4 H | \mathcal {S} |}{9 N _ {\mathrm {t o t}}},
$$
$$
\mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {I S W - B C}}) \right] \leq \left(\frac {1}{\eta + (1 - \eta) / \mu}\right) \frac {4 H ^ {2} | \mathcal {S} |}{9 N _ {\mathrm {t o t}}} = \frac {4 H ^ {2} | \mathcal {S} |}{9 \left(N _ {\mathrm {E}} + N _ {\mathrm {S}} / \mu\right)}.
$$
D.5 An Example Corresponding to Theorem 3
In this section, we provide an example that illustrates the required feature design in Theorem 3 can hold.
Example 1. To illustrate Theorem 3, we consider an example in the feature space  $\mathbb{R}^2$ . In particular, for time step  $h \in [H]$ , we have the expert dataset and supplementary dataset as follows.
$$
\mathcal {D} _ {h} ^ {\mathrm {E}} = \left\{\left(s ^ {(1)}, a ^ {(1)}\right), \left(s ^ {(4)}, a ^ {(4)}\right) \right\}, \mathcal {D} _ {h} ^ {\mathrm {S}} = \left\{\left(s ^ {(2)}, a ^ {(2)}\right), \left(s ^ {(3)}, a ^ {(3)}\right) \right\},
$$
$$
\mathcal {D} _ {h} ^ {\mathrm {S}, 1} = \left\{\left(s ^ {(2)}, a ^ {(2)}\right) \right\}, \mathcal {D} _ {h} ^ {\mathrm {S}, 2} = \left\{\left(s ^ {(3)}, a ^ {(3)}\right) \right\}.
$$
The corresponding features are
$$
\phi_ {h} \left(s ^ {(1)}, a ^ {(1)}\right) = (0, 1) ^ {\top}, \phi_ {h} \left(s ^ {(2)}, a ^ {(2)}\right) = \left(- \frac {1}{2}, 0\right) ^ {\top},
$$
$$
\phi_ {h} \left(s ^ {(3)}, a ^ {(3)}\right) = \left(0, - \frac {1}{2}\right) ^ {\top}, \phi_ {h} \left(s ^ {(4)}, a ^ {(4)}\right) = (- 1, 0) ^ {\top}.
$$
Notice that the set of expert-style samples is  $\mathcal{D}_h^{\mathrm{E}}\cup \mathcal{D}_h^{\mathrm{S},1} = \{(s^{(1)},a^{(1)}),(s^{(2)},a^{(2)}),(s^{(4)},a^{(4)})\}$  and the set of non-expert-style samples is  $\mathcal{D}_h^{\mathrm{S},2} = \{(s^{(3)},a^{(3)})\}$ . It is direct to calculate that the ground-truth parameter that achieves the maximum margin among unit vectors is  $\bar{\theta}_{h} = (-\sqrt{2} /2,\sqrt{2} /2)^{\top}$  and the maximum margin is  $\Delta_h(\bar{\theta}_h) = \sqrt{2} /2$ . According to Eq. (6), for  $\theta_{h} = (\theta_{h,1},\theta_{h,2})^{\top}$ , the optimization objective is
$$
\begin{array}{l} \mathcal {L} _ {h} (\theta_ {h}) \\ = \sum_ {(s, a)} \widehat {d _ {h} ^ {\mathrm {E}}} (s, a) [ \log (1 + \exp (- \langle \phi_ {h} (s, a), \theta_ {h} \rangle)) ] + \sum_ {(s, a)} \widehat {d _ {h} ^ {\mathrm {U}}} (s, a) [ \log (1 + \exp (\langle \phi_ {h} (s, a), \theta_ {h} \rangle)) ] \\ = \frac {1}{2} \left(\log \left(1 + \exp \left(- \theta_ {h, 2}\right)\right) + \log \left(1 + \exp \left(\theta_ {h, 1}\right)\right)\right) \\ \left. + \frac {1}{4} \left(\log \left(1 + \exp \left(\theta_ {h, 2}\right)\right) + \log \left(1 + \exp \left(- \frac {1}{2} \theta_ {h, 1}\right)\right)\right)\right) \\ + \frac {1}{4} \left(\log \left(1 + \exp \left(- \frac {1}{2} \theta_ {h, 2}\right)\right) + \log \left(1 + \exp \left(- \theta_ {h, 1}\right)\right)\right). \\ \end{array}
$$
We apply CVXPY [12] to calculate the optimal solution  $\theta_h^\star \approx (-0.310, 0.993)^\top$  and the objective values  $\mathcal{L}_h(\theta_h^\star) \approx 1.287$ ,  $\mathcal{L}_h(\bar{\theta}_h) \approx 1.309$ . Furthermore, we calculate the Lipschitz coefficient  $L_h$  appears in Lemma 1.
$$
(s ^ {(2)}, a ^ {(2)}) = \operatorname * {a r g m i n} _ {(s, a) \in \mathcal {D} _ {h} ^ {\mathrm {E}} \cup \mathcal {D} _ {h} ^ {\mathrm {S}, 1}} \langle \theta_ {h} ^ {\star}, \phi_ {h} (s, a) \rangle , (s ^ {(3)}, a ^ {(3)}) \in \operatorname * {a r g m a x} _ {(s, a) \in \mathcal {D} _ {h} ^ {\mathrm {S}, 2}} \langle \theta_ {h} ^ {\star}, \phi_ {h} (s, a) \rangle ,
$$
$$
L _ {h} = \left\| \phi_ {h} (s ^ {(2)}, a ^ {(2)}) - \phi_ {h} (s ^ {(3)}, a ^ {(3)}) \right\| = \frac {\sqrt {2}}{2}.
$$
Then we calculate the parameter of strong convexity  $\tau_h$  appears in Lemma 2. Based on the proof of Lemma 2, our strategy is to calculate the minimal eigenvalue of the Hessian matrix.
First, for  $\theta_h = (\theta_{h,1},\theta_{h,2})^\top$  , the gradient of  $\mathcal{L}_h(\theta_h)$  is
$$
\begin{array}{l} \nabla \mathcal {L} _ {h} (\theta_ {h}) \\ = - \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \widehat {d _ {h} ^ {\mathrm {E}}} (s, a) \sigma (- \langle \phi_ {h} (s, a), \theta_ {h} \rangle) + \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \widehat {d _ {h} ^ {\mathrm {U}}} (s, a) \sigma (\langle \phi_ {h} (s, a), \theta_ {h} \rangle) \\ \end{array}
$$
$$
= \left(\frac {1}{2} \sigma \left(\theta_ {h, 1}\right) - \frac {1}{4} \sigma \left(- \theta_ {h, 1}\right) - \frac {1}{8} \sigma \left(- \frac {1}{2} \theta_ {h, 1}\right), \frac {1}{4} \sigma \left(\theta_ {h, 2}\right) - \frac {1}{2} \sigma \left(- \theta_ {h, 2}\right) - \frac {1}{8} \sigma \left(- \frac {1}{2} \theta_ {h, 2}\right)\right) ^ {\top}.
$$
Here  $\sigma(x) = 1 / (1 + \exp(-x))$  for  $x \in \mathbb{R}$  is the sigmoid function. Then the Hessian matrix at  $\theta_h$  is
$$
\nabla^ {2} \mathcal {L} _ {h} (\theta_ {h}) = \left( \begin{array}{c c} \frac {3}{4} f (\theta_ {h, 1}) + \frac {1}{1 6} f \left(\frac {1}{2} \theta_ {h, 1}\right) & 0 \\ 0 & \frac {3}{4} f (\theta_ {h, 2}) + \frac {1}{1 6} f \left(\frac {1}{2} \theta_ {h, 2}\right) \end{array} \right),
$$
where  $f(x) = \sigma (x)(1 - \sigma (x))$  and  $f(x) = f(-x)$ . For any  $t\in [0,1]$ , the eigenvalues of the Hessian matrix at  $\theta_h^t = \bar{\theta}_h + t(\theta_h^\star -\bar{\theta}_h)$  are
$$
\frac {3}{4} f \left(\theta_ {h, 1} ^ {t}\right) + \frac {1}{1 6} f \left(\frac {1}{2} \theta_ {h, 1} ^ {t}\right), \frac {3}{4} f \left(\theta_ {h, 2} ^ {t}\right) + \frac {1}{1 6} f \left(\frac {1}{2} \theta_ {h, 2} ^ {t}\right).
$$
Now, we calculate the minimal eigenvalues of  $\nabla^2\mathcal{L}_h(\theta_h^t)$ . We consider the function
$$
g (x) = \frac {3}{4} f (x) + \frac {1}{1 6} f \left(\frac {1}{2} x\right), \forall x \in [ a, b ].
$$
The gradient is
$$
g ^ {\prime} (x) = \frac {3}{4} \sigma (x) (1 - \sigma (x)) (1 - 2 \sigma (x)) + \frac {1}{3 2} \sigma \left(\frac {1}{2} x\right) \left(1 - \sigma \left(\frac {1}{2} x\right)\right) \left(1 - 2 \sigma \left(\frac {1}{2} x\right)\right).
$$
We observe that  $\forall x\leq 0,g^{\prime}(x)\geq 0$  , and  $\forall x\geq 0,g^{\prime}(x)\leq 0$  . Thus, we have that the minimum of  $g(x)$  must be achieved at  $x = a$  or  $x = b$  . Besides, we have that  $g(x) = g(-x)$  . With the above arguments, we know that the minimal eigenvalue is  $g(0.993)\approx 0.163$  and  $\tau_h\approx 0.163$  . Then we can calculate that
$$
\sqrt {\frac {2 \left(\mathcal {L} _ {h} \left(\bar {\theta} _ {h}\right) - \mathcal {L} _ {h} \left(\theta_ {h} ^ {\star}\right)\right)}{\tau_ {h}}} \approx 0. 5 2 0, \frac {\Delta_ {h} (\bar {\theta} _ {h})}{L _ {h}} = 1.
$$
The inequality in Theorem 3 holds.
In the main text, we focus on the tabular representations for policies. Furthermore, we consider a trajectory sampling procedure for behavior policy in collecting the supplementary dataset. We present two possible extensions in this section.
E.1 Function Approximation of Policies
In the main text, the theoretical analysis for BC-based algorithms considers the tabular setting in policy learning where a table function represents the policy. Here we provide an analysis of BC with general function approximation in policy learning. Notice that the algorithms considered in this paper (i.e., BC, NBCU and ISW-BC) can be unified under the framework of maximum likelihood estimation (MLE)<sup>8</sup>. Therefore, the theoretical results in the main text can also be extended to the setting of general function approximation by a similar analysis.
Assume that the learner is access to a finite function class  $\Pi = \{\pi = (\pi_1, \pi_2, \dots, \pi_h)\}$ , where  $\pi_h: S \to \Delta(\mathcal{A})$  could be any function (e.g., neural networks). For simplicity of analysis, we assume that  $\Pi$  is a finite class. Notice that the algorithms considered in this paper are BC and its variants, which all take the principle of maximum likelihood estimation (MLE). The theoretical analysis of these algorithms is based on the following inequality:
$$
V \left(\pi^ {\mathrm {E}}\right) - V (\pi) \leq H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \operatorname {T V} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} (\cdot | s)\right) \right].
$$
Therefore, the key is to upper bound the TV distance. Take BC as an example (i.e.,  $\pi = \pi^{\mathrm{BC}}$ ). By using the concentration inequality in [1, Theorem 21], we obtain that for any  $\delta \in (0,1)$ , when
Among these algorithms, the main difference is the weight function in the MLE objective; see Equations (1), (2) and (7).
$\left|\mathcal{D}^{\mathrm{E}}\right| \geq 1$ , with probability at least  $1 - \delta$  over the randomness within  $\mathcal{D}^{\mathrm{E}}$ ,
$$
\mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \mathrm {T V} ^ {2} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} ^ {\mathrm {B C}} (\cdot | s)\right) \right] \leq 2 \frac {\log (| \Pi | / \delta)}{| \mathcal {D} ^ {\mathrm {E}} |}. \tag {19}
$$
With additional efforts (by using union bound and Jensen's inequality), we have the following result.
Theorem 4 (BC with Function Approximation). Under Assumption 1. In the general function approximation setting, additionally assume that  $\pi^{\mathrm{E}}\in \Pi$ . If we apply BC on the expert data, we have
$$
\mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {B C}}) \right] = \mathcal {O} \left(H ^ {2} \sqrt {\frac {\log (| \Pi | H N _ {\mathrm {E}})}{N _ {\mathrm {E}}}}\right),
$$
where the expectation is taken over the randomness in the dataset collection.
The detailed proof is deferred to Appendix F. Compared with Theorem 1, we notice that the change in theoretical bound is that  $\mathcal{O}(|S| / N_{\mathrm{E}})$  is replaced by  $\mathcal{O}(\sqrt{\log(|\Pi|HN_{\mathrm{E}}) / N_{\mathrm{E}}})$ . Such a change is expected for other algorithms (e.g., NBCU and WBCU), so our theoretical implications still hold. We leave the detailed analysis for future works.
E.2 Supplementary Data with Corruption
In the main text, we consider the trajectory sampling procedure in Assumption 1. However, in some cases, the supplementary data can be poisoned and corrupted by an adversary. For example, although the human expert demonstrates an optimal trajectory, the recorder or the recording system possibly corrupts the data by accident or on purpose. Data corruption is one of the main security threats to imitation learning methods [31]. Therefore, it is valuable to investigate the robustness of the presented algorithms in this poison setting. Supplementary data with corruption is partially investigated in our experiments under the noisy expert setting, which we argue have a large state-action distribution shift.
Assumption 3 (Poison Setting). The supplementary dataset  $\mathcal{D}^{\mathrm{S}}$  and expert dataset  $\mathcal{D}^{\mathrm{E}}$  are collected in the following way: each time, with probability  $\eta$ , we rollout the expert policy to collect a trajectory. With probability  $1 - \eta$ , we still rollout the expert policy to collect a trajectory but with probability  $1 - \eta'$ , the actions along the sampled trajectory are replaced with actions uniformly sampled from the action space. Such an experiment is independent and identically conducted by  $N_{\mathrm{tot}}$  times.
Theorem 5 (NBCU in the Poison Setting). Under Assumption 3. In the tabular case, for any  $\eta \in (0,1]$ , we have
$$
\mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {N B C U}}) \right] = \mathcal {O} \left((1 - \eta) (1 - \eta^ {\prime}) H ^ {2} \left(1 - \frac {1}{| \mathcal {A} |}\right) + H ^ {2} \sqrt {\frac {| S | | \mathcal {A} |}{N _ {\mathrm {t o t}}}}\right),
$$
Theorem 6 (ISW-BC in the Poison Setting). Under Assumptions 2 and 3, if the feature is designed such that  $\sqrt{\frac{2\left(\mathcal{L}_h(\bar{\theta}_h) - \mathcal{L}_h(\theta_h^*)\right)}{\tau_h}} < \frac{\Delta_h(\bar{\theta}_h)}{L_h}$  holds, we have the imitation gap bound
$$
\mathbb {E} [ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {I S W - B C}}) ] = \mathcal {O} \left(\frac {H ^ {2} | \mathcal {S} |}{N _ {\mathrm {E}} + N _ {\mathrm {S}} \eta^ {\prime}}\right).
$$
Proofs of Theorem 5 and Theorem 6 can be found in Appendix F. Compared with the imitation gap of NBCU, there is no non-vanishing gap due to the corrupted actions in the imitation gap of ISW-BC. This means that ISW-BC is still robust in this setting.
F Proof of Results in Section E
F.1 Proof of Theorem 4
According to [42, Lemma 4.3], we have
$$
V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {B C}}) \leq H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \mathrm {T V} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} ^ {\mathrm {B C}} (\cdot | s)\right) \right].
$$
With [1, Theorem 21], when  $|\mathcal{D}^{\mathrm{E}}| \geq 1$ , for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  over the randomness within  $\mathcal{D}^{\mathrm{E}}$ , we have that
$$
\mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \mathrm {T V} ^ {2} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} ^ {\mathrm {B C}} (\cdot | s)\right) \right] \leq 2 \frac {\log (| \Pi | / \delta)}{| \mathcal {D} ^ {\mathrm {E}} |}.
$$
With union bound, with probability at least  $1 - \delta$ , for all  $h \in [H]$ , it holds that
$$
\mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \mathrm {T V} ^ {2} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} ^ {\mathrm {B C}} (\cdot | s)\right) \right] \leq 2 \frac {\log (| \Pi | H / \delta)}{| \mathcal {D} ^ {\mathrm {E}} |},
$$
$$
\begin{array}{l} V \left(\pi^ {\mathrm {E}}\right) - V \left(\pi^ {\mathrm {B C}}\right) \leq H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \operatorname {T V} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} ^ {\mathrm {B C}} (\cdot | s)\right) \right] \\ \stackrel {(a)} {\leq} H \sum_ {h = 1} ^ {H} \sqrt {\mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}} (\cdot)}} \left[ \mathrm {T V} ^ {2} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s) , \pi_ {h} ^ {\mathrm {B C}} (\cdot | s)\right) \right]} \\ \leq \sqrt {2} H ^ {2} \sqrt {\frac {\log (| \Pi | H / \delta)}{| \mathcal {D} ^ {\mathrm {E}} |}}. \\ \end{array}
$$
Inequality  $(a)$  follows Jensen's inequality. Taking expectation over the randomness within  $\mathcal{D}^{\mathrm{E}}$  yields that
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} ^ {\mathrm {E}}} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {B C}}) \right] \leq \delta H + (1 - \delta) \sqrt {2} H ^ {2} \sqrt {\frac {\log (| \Pi | H / \delta)}{| \mathcal {D} ^ {\mathrm {E}} |}} \\ \stackrel {(a)} {=} \frac {H}{2 | \mathcal {D} ^ {\mathrm {E}} |} + \left(1 - \frac {1}{2 | \mathcal {D} ^ {\mathrm {E}} |}\right) \sqrt {2} H ^ {2} \sqrt {\frac {\log (2 | \Pi | H | \mathcal {D} ^ {\mathrm {E}} |)}{| \mathcal {D} ^ {\mathrm {E}} |}} \\ \leq \left(\sqrt {2} + 1\right) H ^ {2} \sqrt {\frac {\log (2 | \Pi | H | \mathcal {D} ^ {\mathrm {E}} |)}{| \mathcal {D} ^ {\mathrm {E}} |}} \\ \leq 4 H ^ {2} \sqrt {\frac {\log (4 | \Pi | H | \mathcal {D} ^ {\mathrm {E}} |)}{| \mathcal {D} ^ {\mathrm {E}} |}}. \\ \end{array}
$$
Equation (a) holds due to the choice that  $\delta = 1 / (2|\mathcal{D}^{\mathrm{E}}|)$ . For  $|\mathcal{D}^{\mathrm{E}}| = 0$ , we directly have that
$$
\mathbb {E} _ {\mathcal {D} ^ {\mathrm {E}}} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {B C}}) \right] \leq H.
$$
Therefore, for any  $|\mathcal{D}^{\mathrm{E}}| \geq 0$ , we have that
$$
\mathbb {E} _ {\mathcal {D} ^ {\mathrm {E}}} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {B C}}) \right] \leq 4 H ^ {2} \sqrt {\frac {\log (4 | \Pi | H \max \{| \mathcal {D} ^ {\mathrm {E}} | , 1 \})}{\max \{| \mathcal {D} ^ {\mathrm {E}} | , 1 \}}}.
$$
We consider a real-valued function  $f(x) = \log (cx) / x$  for  $x\geq 1$ , where  $c = 4|\Pi |H > 4$ . Its gradient function is  $f^{\prime}(x) = (1 - \log (cx)) / x^{2}\leq 0$  when  $x\geq 1$ . Then we know that  $f(x)$  is decreasing as  $x$  increases. Furthermore, we have that  $\max \{|D^{\mathrm{E}}|,1\} \geq (|D^{\mathrm{E}}| + 1) / 2$  when  $|\mathcal{D}^{\mathrm{E}}|\geq 0$ . Then we obtain
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} ^ {\mathrm {E}}} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {B C}}) \right] \leq 4 H ^ {2} \sqrt {\frac {\log (4 | \Pi | H \max \{| \mathcal {D} ^ {\mathrm {E}} | , 1 \})}{\max \{| \mathcal {D} ^ {\mathrm {E}} | , 1 \}}} \\ \leq 4 H ^ {2} \sqrt {\frac {2 \log (4 | \Pi | H (| \mathcal {D} ^ {\mathrm {E}} | + 1))}{| \mathcal {D} ^ {\mathrm {E}} | + 1}}. \\ \end{array}
$$
Taking expectation over the random variable  $|\mathcal{D}^{\mathrm{E}}| \sim \mathrm{Bin}(N_{\mathrm{tot}}, \eta)$  yields that
$$
\mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {B C}}) \right] \leq 4 H ^ {2} \mathbb {E} \left[ \sqrt {\frac {2 \log (4 | \Pi | H (| \mathcal {D} ^ {\mathrm {E}} | + 1))}{| \mathcal {D} ^ {\mathrm {E}} | + 1}} \right]
$$
$$
\stackrel {(a)} {\leq} 4 H ^ {2} \sqrt {\mathbb {E} \left[ \frac {2 \log (4 | \Pi | H (| \mathcal {D} ^ {\mathrm {E}} | + 1))}{| \mathcal {D} ^ {\mathrm {E}} | + 1} \right]}.
$$
Inequality  $(a)$  follows Jensen's inequality. We consider the function  $g(x) = -x\log (x / c)$  for  $x\in (0,1]$ , where  $c = 4|\Pi |H$
$$
g ^ {\prime} (x) = - \left(\log (x / c) + 1\right) \geq 0, g ^ {\prime \prime} (x) = - \frac {1}{x} \leq 0, \quad \forall x \in (0, 1 ].
$$
Thus,  $g(x)$  is a concave function. By Jensen's inequality, we have that  $\mathbb{E}[g(x)] \leq g(\mathbb{E}[x])$ . Then we can derive that
$$
\begin{array}{l} \mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {B C}}) \right] \leq 4 H ^ {2} \sqrt {\mathbb {E} \left[ \frac {2 \log (4 | \Pi | H (| \mathcal {D} ^ {\mathrm {E}} | + 1))}{| \mathcal {D} ^ {\mathrm {E}} | + 1} \right]} \\ = 4 \sqrt {2} H ^ {2} \sqrt {\mathbb {E} \left[ g \left(\frac {1}{| \mathcal {D} ^ {\mathrm {E}} | + 1}\right) \right]} \\ \leq 4 \sqrt {2} H ^ {2} \sqrt {g \left(\mathbb {E} \left[ \frac {1}{| \mathcal {D} ^ {\mathrm {E}} | + 1} \right]\right)} \\ \stackrel {(a)} {\leq} 4 \sqrt {2} H ^ {2} \sqrt {g \left(\frac {1}{N _ {\mathrm {E}}}\right)} \\ \leq 4 \sqrt {2} H ^ {2} \sqrt {\frac {\log (4 | \Pi | H N _ {\mathrm {E}})}{N _ {\mathrm {E}}}}. \\ \end{array}
$$
In inequality  $(a)$ , we use the facts that  $g^{\prime}(x) \geq 0$  and  $\mathbb{E}\left[1 / (|\mathcal{D}^{\mathrm{E}}| + 1)\right] \leq 1 / N_{\mathrm{E}}$  from Lemma 3. We complete the proof.
F.2 Proof of Theorem 5
We first analyze the data distribution in  $\mathcal{D}^{\mathrm{U}}$ . According to Assumption 3, we summarize the sampling procedure of trajectories in  $\mathcal{D}^{\mathrm{U}}$  as follows. Each time, we rollout the expert policy to collect a trajectory. Furthermore, with the probability of  $(1 - \eta)(1 - \eta')$ , the actions along the sampled expert trajectory are replaced with actions uniformly sampled from the action space. Then we put this poisoned expert trajectory into  $\mathcal{D}^{\mathrm{U}}$ . Otherwise, with the probability of  $1 - (1 - \eta)(1 - \eta')$ , we directly put the original expert trajectory into  $\mathcal{D}^{\mathrm{U}}$ . Therefore, we can formulate the marginal distribution of the state-action pairs in time step  $h$  in  $\mathcal{D}^{\mathrm{U}}$ . For each  $(s,a,h) \in S \times \mathcal{A} \times [H]$ ,
$$
d _ {h} ^ {\mathrm {U}} (s, a) = (1 - (1 - \eta) (1 - \eta^ {\prime})) d _ {h} ^ {\pi^ {\mathrm {E}}} (s, a) + (1 - \eta) (1 - \eta^ {\prime}) d _ {h} ^ {\pi^ {\mathrm {E}}} (s) \frac {1}{| \mathcal {A} |},
$$
$$
d _ {h} ^ {\mathrm {U}} (s) = \sum_ {a \in \mathcal {A}} d _ {h} ^ {\mathrm {U}} (s, a) = d _ {h} ^ {\pi^ {\mathrm {E}}} (s).
$$
Then we proceed to analyze the imitation gap. Similar to the proof of Theorem 2, according to [42, Lemma 4.3], we have
$$
V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {N B C U}}) \leq H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \operatorname {T V} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} ^ {\mathrm {N B C U}} (\cdot | s)\right) \right].
$$
Again, we introduce the definition of the policy  $\pi^{\mathrm{mix}}$ .
$$
\forall (s, a) \in \mathcal {S} \times \mathcal {A}, \forall h \in [ H ], \pi_ {h} ^ {\mathrm {m i x}} (a | s) = \left\{ \begin{array}{l l} \frac {d _ {h} ^ {\mathrm {U}} (s , a)}{d _ {h} ^ {\mathrm {U}} (s)} & \text {i f} d _ {h} ^ {\mathrm {U}} (s) = d _ {h} ^ {\pi^ {\mathrm {E}}} (s) > 0, \\ \frac {1}{| \mathcal {A} |} & \text {o t h e r w i s e .} \end{array} \right.
$$
In particular, if  $d_h^{\mathrm{U}}(s) > 0$ , we have that
$$
\pi_ {h} ^ {\operatorname {m i x}} (a | s) = \frac {d _ {h} ^ {\mathrm {U}} (s , a)}{d _ {h} ^ {\mathrm {U}} (s)} = (1 - (1 - \eta) (1 - \eta^ {\prime})) \pi_ {h} ^ {\mathrm {E}} (a | s) + (1 - \eta) (1 - \eta^ {\prime}) \frac {1}{| \mathcal {A} |}.
$$
Then we decompose the imitation gap into two parts.
$$
\begin{array}{l} V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {N B C U}}) \\ \leq H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \operatorname {T V} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} ^ {\mathrm {N B C U}} (\cdot | s)\right) \right] \\ \leq H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \operatorname {T V} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} ^ {\operatorname {m i x}} (\cdot | s)\right) \right] + H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \operatorname {T V} \left(\pi_ {h} ^ {\operatorname {m i x}} (\cdot | s), \pi_ {h} ^ {\mathrm {N B C U}} (\cdot | s)\right) \right]. \\ \end{array}
$$
We first analyze the first term in RHS. For certain  $(s,h)$  such  $d_h^{\mathrm{U}}(s) = d_h^{\pi^{\mathrm{E}}} (s) > 0$ , we have that