SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[52] Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. In Proceedings of the 38th International Conference on Machine Learning, pages 10961-10970, 2021.
[53] Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger. Inequalities for the 11 deviation of the empirical distribution. Hewlett-Packard Labs, Technical Report, 2003.
[54] Steven Euijong Whang, Yuji Roh, Hwanjun Song, and Jae-Gil Lee. Data collection and quality challenges in deep learning: A data-centric ai perspective. The VLDB Journal, pages 1-23, 2023.
[55] Grover J Whitehurst and Ross Vasta. Is language acquired through imitation? Journal of Psycholinguistic Research, 4:37-59, 1975.
[56] Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In Proceedings of the 36th International Conference on Machine Learning, pages 6818-6827, 2019.
[57] Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in Neural Information Processing Systems 34, pages 27395-27407, 2021.
[58] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. arXiv preprint arXiv:2302.03169, 2023.
[59] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. In Advances in Neural Information Processing Systems 33, pages 15737-15749, 2020.
[60] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[61] Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weighted offline imitation learning from suboptimal demonstrations. In Proceedings of the 39th International Conference on Machine Learning, pages 24725-24742, 2022.
[62] Tian Xu, Ziniu Li, Yang Yu, and Zhi-Quan Luo. Understanding adversarial imitation learning in small sample regime: A stage-coupled analysis. arXiv preprint arXiv:2208.01899, 2022.
[63] Tian Xu, Ziniu Li, Yang Yu, and Zhi-Quan Luo. Provably efficient adversarial imitation learning with unknown transitions. In Proceedings of the 39th Conference on Uncertainty in Artificial Intelligence, pages 2367–2378, 2023.
[64] Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303.10158, 2023.
This study delves into the theoretical aspects of offline imitation learning with supplementary data, and we verify our findings through experiments on established benchmarks. While this paper does not present any immediate, direct social impacts, the potential practical applications of our research could bring about positive change. By expanding the reach of imitation learning algorithms, our work may facilitate the development of more efficient and effective solutions in fields such as robotics, autonomous vehicles, and healthcare. However, we must also acknowledge that the misuse of such technology could have negative consequences, such as the manipulation of information to influence people's behavior. Therefore, it is crucial to remain vigilant in ensuring that the benefits of imitation learning are harnessed in a responsible and ethical manner.
B.1 Imitation Learning from Imperfection
The problem considered in this paper is related to IL with a single imperfect dataset [56, 6, 51, 52, 45, 31]. In particular, the supplementary dataset in our set-up can also be viewed as imperfect demonstrations. However, our problem setting differs from IL with imperfect demonstrations in two key aspects. First, in IL with imperfect demonstrations, they either pose strong assumptions [51, 45, 31] or require auxiliary information (e.g., confidence scores on imperfect trajectories) on the imperfect dataset [56, 6]. In contrast, we assume access to a small number of expert trajectories to identify in-expert-distribution data. Second, most works [56, 6, 51, 52] in IL with imperfect demonstrations require online environment interactions while we focus on the offline setting. Additionally, in a related study, [7] employed imperfect data to infer a reward function. Under the assumption of the expert being strictly sub-optimal, [7] demonstrated that it is possible to learn a policy that surpasses the performance of the demonstrator by utilizing the recovered reward function.
Our research also bears relevance to the recent works by [29, 47], which aimed to unify offline Reinforcement Learning (RL) and IL. Different from our approach, these studies adopt a principle akin to adversarial imitation learning [50]. For a more detailed discussion, we refer readers to [29, 47]. We also observe a connection between the objective of importance sampling weighted behavior cloning (ISW-BC) and the reward-weighted regression (RWR) framework introduced in [38, 37]. Specifically, the training objective of ISW-BC can be likened to reward-weighted regression (RWR) with  $\gamma = 0$  and a reward function denoted as  $r(s,a) = \log \frac{d^{\mathrm{E}}(s,a)}{d^{\mathrm{U}}(s,a)}$ . Nevertheless, there exist two distinctions between the two approaches. First, RWR primarily focuses on the online setting, while our work centers around the offline setting. Second, RWR finds its application in the context of RL, where rewards are readily available. Conversely, in our imitation learning setup, the reward (or the importance sampling ratio) needs to be inferred.
B.2 Difference with DemoDICE and DWBC
Our work builds upon previous research in IL with supplementary data, specifically the algorithms DemoDICE [25] and DWBC [61]. A significant distinction arises between ISW-BC and these two methods in terms of the weighting rule design. While DemoDICE and DWBC employ regularized weighting rules, our method directly estimates the importance sampling ratio. This fundamental difference can be critical as regularized weighting rules may struggle to recover the expert policy exactly even with infinite samples. We provide further elaboration on this point below.
First, DemoDICE also uses the weighted BC objective in Eq. (3). But, DemoDICE uses the weighting rule of  $\widetilde{w}(s,a) \propto d^{\star}(s,a)/d^{\mathrm{U}}(s,a)$  (refer to the formula between Equations (19)-(20) in [25]), where  $d^{\star}(s,a)$  is computed by the expert's state-action distribution matching objective regularized by a divergence to the union data distribution (refer to [25, Equations (5)-(7)]):
$$
d ^ {\star} = \underset {d} {\operatorname {a r g m i n}} D _ {\mathrm {K L}} (d \| d ^ {\mathrm {E}}) + \alpha D _ {\mathrm {K L}} (d \| d ^ {\mathrm {U}})
$$
$$
\begin{array}{l l} \text {s . t .} & d (s, a) \geq 0 \quad \forall s, a. \end{array}
$$
For a moment, we use the notations in [25] and present their results under the stationary and infinite-horizon MDPs. Same as the discussion of DWBC [61].
$$
\sum_ {a} d (s, a) = (1 - \gamma) \rho (s) + \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} P (s | s ^ {\prime}, a ^ {\prime}) d (s ^ {\prime}, a ^ {\prime}) \quad \forall s.
$$
where  $\gamma \in [0,1)$  is the discount factor,  $\alpha > 0$  is a hyper-parameter. Due to the regularization term in the objective, it holds that  $d^{\star}(s,a) \neq d^{\pi^{\mathrm{E}}}(s,a)$ , resulting in a biased weighting rule  $\widetilde{w}(s,a)$ .
Second, DWBC considers a different policy learning objective (refer to [61, Equation (17)]):
$$
\begin{array}{l} \min  _ {\pi} \alpha \sum_ {(s, a) \in \mathcal {D} ^ {\mathrm {E}}} [ - \log \pi (a | s) ] - \sum_ {(s, a) \in \mathcal {D} ^ {\mathrm {E}}} \left[ - \log \pi (a | s) \cdot \frac {\lambda}{c (1 - c)} \right] \\ + \sum_ {(s, a) \in \mathcal {D} ^ {\mathrm {S}}} \left[ - \log \pi (a | s) \cdot \frac {1}{1 - c} \right], \tag {7} \\ \end{array}
$$
where  $\alpha > 0, \lambda > 0$  are hyper-parameters, and  $c$  is the output of the discriminator that is jointly trained with  $\pi$  (refer to [61, Equation (8)]):
$$
\begin{array}{l} \min  _ {c} \lambda \sum_ {(s, a) \in \mathcal {D} ^ {\mathrm {E}}} [ - \log c (s, a, \log \pi (a | s)) ] + \sum_ {(s, a) \in \mathcal {D} ^ {\mathrm {S}}} [ - \log (1 - c (s, a, \log \pi (a | s))) ] \\ - \lambda \sum_ {(s, a) \in \mathcal {D} ^ {\mathrm {E}}} [ - \log (1 - c (s, a, \log \pi (a | s))) ]. \\ \end{array}
$$
Since its input additionally incorporates  $\log \pi$ , the discriminator is not guaranteed to estimate the state-action distribution. Thus, the weighting in Eq. (7) loses a connection with the importance sampling ratio.
C Proof of Results in Section 4
Recall the objective of BC in Eq. (1):
$$
\pi^{\mathrm{BC}}\in \operatorname *{argmax}_{\pi}\sum_{h = 1}^{H}\sum_{(s,a)\in \mathcal{S}\times \mathcal{A}}\widehat{d_{h}^{\mathrm{E}}} (s,a)\log \pi_{h}(a|s),
$$
where  $\widehat{d_h^{\mathrm{E}}}(s,a) = n_h^{\mathrm{E}}(s,a) / N_{\mathrm{tot}}$  is the empirical state-action distribution in the expert dataset, and  $n_h^{\mathrm{E}}(s,a)$  is the number of expert trajectories such that their state-action pairs are equal to  $(s,a)$  in time step  $h$ . With the tabular representations, we can obtain a closed-formed solution to the above optimization problem.
$$
\pi_ {h} ^ {\mathrm {B C}} (a | s) = \left\{ \begin{array}{l l} \frac {n _ {h} ^ {\mathrm {E}} (s , a)}{n _ {h} ^ {\mathrm {E}} (s)} & \text {i f} n _ {h} ^ {\mathrm {E}} (s) > 0 \\ \frac {1}{| \mathcal {A} |} & \text {o t h e r w i s e} \end{array} \right. \tag {8}
$$
where  $n_h^{\mathrm{E}}(s) \triangleq \sum_{a'} n_h^{\mathrm{E}}(s, a')$ . Analogously, we also have a closed-form solution for NBCU in the tabular setting:
$$
\pi_ {h} ^ {\mathrm {N B C U}} (a | s) = \left\{ \begin{array}{l l} \frac {n _ {h} ^ {\mathrm {U}} (s , a)}{n _ {h} ^ {\mathrm {U}} (s)} & \text {i f} n _ {h} ^ {\mathrm {U}} (s) > 0 \\ \frac {1}{| A |} & \text {o t h e r w i s e} \end{array} \right. \tag {9}
$$
We will discuss the generalization performance of NBCU later.
In the proof, we frequently use the notation  $\lesssim$  and  $\gtrsim$ . In particular,  $a(n) \lesssim b(n)$  means that there exist  $C, n_0 > 0$  such that  $a(n) \leq C b(n)$  for all  $n \geq n_0$ . In our context,  $n$  usually refers to the number of trajectories. For any two distributions  $P$  and  $Q$  over a finite set  $\mathcal{X}$ , we define the total variation distance as
$$
\operatorname {T V} (P, Q) = \frac {1}{2} \sum_ {x \in \mathcal {X}} | P (x) - Q (x) | = \| P - Q \| _ {1}.
$$
When  $|\mathcal{D}^{\mathrm{E}}| \geq 1$ , by [42, Theorem 4.2], we have the following imitation gap bound for BC:
$$
V (\pi^ {\mathrm {E}}) - \mathbb {E} _ {\mathcal {D} ^ {\mathrm {E}}} \left[ V (\pi^ {\mathrm {B C}}) \right] \leq \frac {4 | \mathcal {S} | H ^ {2}}{9 | \mathcal {D} ^ {\mathrm {E}} |}.
$$
When  $|\mathcal{D}^{\mathrm{E}}| = 0$ , we simply have that
$$
V \left(\pi^ {\mathrm {E}}\right) - \mathbb {E} _ {\mathcal {D} ^ {\mathrm {E}}} \left[ V \left(\pi^ {\mathrm {B C}}\right) \right] \leq H.
$$
Therefore, we have the following unified bound.
$$
V \left(\pi^ {\mathrm {E}}\right) - \mathbb {E} _ {\mathcal {D} ^ {\mathrm {E}}} \left[ V \left(\pi^ {\mathrm {B C}}\right) \right] \leq \frac {| \mathcal {S} | H ^ {2}}{\max  \{| \mathcal {D} ^ {\mathrm {E}} | , 1 \}} \leq \frac {2 | \mathcal {S} | H ^ {2}}{| \mathcal {D} ^ {\mathrm {E}} | + 1}.
$$
The last inequality follows that  $\max \{x,1\} \geq (x + 1) / 2$  for any  $x\geq 0$ . Finally, notice that  $|\mathcal{D}^{\mathrm{E}}|$  follows a binomial distribution by Assumption 1, i.e.,  $|\mathcal{D}^{\mathrm{E}}|\sim \mathrm{Bin}(N_{\mathrm{tot}},\eta)$ . By Lemma 3, we have that  $\mathbb{E}[1 / (|\mathcal{D}|^E +1)]\leq N_{\mathrm{tot}}\eta$ , so
$$
V (\pi^ {\mathrm {E}}) - \mathbb {E} \left[ V (\pi^ {\mathrm {B C}}) \right] \leq \mathbb {E} \left[ \frac {2 | \mathcal {S} | H ^ {2}}{| \mathcal {D} ^ {\mathrm {E}} | + 1} \right] \leq \frac {2 | \mathcal {S} | H ^ {2}}{N _ {\mathrm {t o t}} \eta} = \frac {2 | \mathcal {S} | H ^ {2}}{N _ {\mathrm {E}}},
$$
For analysis, we first define the mixture state-action distribution as follows.
$$
d _ {h} ^ {\operatorname {m i x}} (s, a) \triangleq \eta d _ {h} ^ {\pi^ {\mathrm {E}}} (s, a) + (1 - \eta) d _ {h} ^ {\pi^ {\beta}} (s, a),
$$
$$
d _ {h} ^ {\operatorname {m i x}} (s) \triangleq \sum_ {a \in \mathcal {A}} d _ {h} ^ {\operatorname {m i x}} (s, a), \forall (s, a) \in \mathcal {S} \times \mathcal {A}, \forall h \in [ H ].
$$
By Assumption 1, in the population level, the marginal state-action distribution of union dataset  $\mathcal{D}^{\mathrm{U}}$  in time step  $h$  is exactly  $d_h^{\mathrm{mix}}$ . That is,  $d_h^{\mathrm{U}}(s,a) = d_h^{\mathrm{mix}}(s,a)$ ,  $\forall (s,a,h)\in S\times \mathcal{A}\times [H]$ . Then we define the mixture policy  $\pi^{\mathrm{mix}}$  induced by  $d^{\mathrm{mix}}$  as follows.
$$
\pi_ {h} ^ {\operatorname {m i x}} (a | s) = \left\{ \begin{array}{l l} \frac {d _ {h} ^ {\operatorname {m i x}} (s , a)}{d _ {h} ^ {\operatorname {m i x}} (s)} & \text {i f} d _ {h} ^ {\operatorname {m i x}} (s) > 0, \\ \frac {1}{| \mathcal {A} |} & \text {o t h e r w i s e .} \end{array} \forall (s, a) \in \mathcal {S} \times \mathcal {A}, \forall h \in [ H ]. \right. \tag {10}
$$
From the theory of Markov Decision Processes, we know that (see, e.g., [41])
$$
\forall h \in [ H ], \forall (s, a) \in \mathcal {S} \times \mathcal {A}, \quad d _ {h} ^ {\pi^ {\operatorname {m i x}}} (s, a) = d _ {h} ^ {\operatorname {m i x}} (s, a).
$$
Therefore, we can obtain that the marginal state-action distribution of union dataset  $\mathcal{D}^{\mathrm{U}}$  in time step  $h$  is exactly  $d_h^{\pi_{\mathrm{mix}}}^{\mathrm{min}}$ . Then we have the following decomposition.
$$
\begin{array}{l} \mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {N B C U}}) \right] = \mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {m i x}}) + V (\pi^ {\mathrm {m i x}}) - V (\pi^ {\mathrm {N B C U}}) \right] \\ = \mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {m i x}}) \right] + \mathbb {E} \left[ V (\pi^ {\mathrm {m i x}}) - V (\pi^ {\mathrm {N B C U}}) \right] \\ = V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {m i x}}) + \mathbb {E} \left[ V (\pi^ {\mathrm {m i x}}) - V (\pi^ {\mathrm {N B C U}}) \right]. \\ \end{array}
$$
For  $V(\pi^{\mathrm{E}}) - V(\pi^{\mathrm{mix}})$ , we have that
$$
\begin{array}{l} V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {m i x}}) = \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \left(d _ {h} ^ {\pi^ {\mathrm {E}}} (s, a) - d _ {h} ^ {\pi^ {\mathrm {m i x}}} (s, a)\right) r _ {h} (s, a) \\ = \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \left(d _ {h} ^ {\pi^ {\mathrm {E}}} (s, a) - d _ {h} ^ {\mathrm {m i x}} (s, a)\right) r _ {h} (s, a) \\ = (1 - \eta) \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \left(d _ {h} ^ {\pi^ {\mathrm {E}}} (s, a) - d _ {h} ^ {\pi^ {\beta}} (s, a)\right) r _ {h} (s, a) \\ = (1 - \eta) \left(V \left(\pi^ {\mathrm {E}}\right) - V \left(\pi^ {\beta}\right)\right). \tag {11} \\ \end{array}
$$
The last equation follows the dual formulation of policy value (see, e.g., [41]), i.e.,  $V(\pi) = \sum_{h=1}^{H} \sum_{(s,a)} d_h^\pi(s,a) r_h(s,a)$  for any policy  $\pi$ . Besides, notice that  $\mathbb{E}\left[V(\pi^{\mathrm{mix}}) - V(\pi^{\mathrm{NBCU}})\right]$  is exactly the imitation gap of BC when regarding  $\pi^{\mathrm{mix}}$  and  $\mathcal{D}^{\mathrm{U}}$  as the expert policy and expert dataset, respectively. Note that  $\pi^{\mathrm{mix}}$  may be a stochastic policy. By [42, Theorem 4.4], we have the following imitation gap bound
$$
\mathbb {E} \left[ V \left(\pi^ {\text {m i x}}\right) - V \left(\pi^ {\mathrm {N B C U}}\right) \right] \lesssim \frac {| S | H ^ {2} \log \left(N _ {\text {t o t}}\right)}{N _ {\text {t o t}}}. \tag {12}
$$
Combining Eq. (11) and Eq. (12) yields that
$$
\mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {N B C U}}) \right] \lesssim (1 - \eta) \left(V (\pi^ {\mathrm {E}}) - V (\pi^ {\beta})\right) + \frac {| \mathcal {S} | H ^ {2} \log (N _ {\mathrm {t o t}})}{N _ {\mathrm {t o t}}}.
$$
C.3 Proof of Proposition 1
The hard instance in Proposition 1 builds on the Standard Imitation MDP proposed in [60]; see Figure 4 for illustration. For this MDP, each state is an absorbing state, i.e.,  $P_{h}(s|s,a) = 1$  for any  $s$  and  $a$ . This property is mainly used to facilitate probability calculation and does not change the nature of our analysis. Furthermore, by only taking the action  $a^1$  (shown in green), the agent can obtain a reward of  $+1$ . Otherwise, the agent obtains a reward of 0 for the other action  $a \neq a^1$ . The initial state distribution is a uniform distribution, i.e.,  $\rho(s) = 1 / |S|$  for any  $s \in S$ .
Figure 4: The Standard Imitation MDP in [60] corresponding to prove Proposition 1.
0
Green arrow: expert action
Blue arrow: non-expert action
We consider that the expert policy  $\pi^{\mathrm{E}}$  always takes the action  $a^1$  (shown in green) while the behavioral policy  $\pi^{\beta}$  always takes another action  $a^2$  (shown in blue). Formally,  $\pi_h^{\mathrm{E}}(a^1 |s) = 1$  and  $\pi_h^\beta (a^2 |s) = 1$  for any  $s\in S$  and  $h\in [H]$ . It is direct to calculate that  $V(\pi^{\mathrm{E}}) = H$  and  $V(\pi^{\beta}) = 0$ . The supplementary dataset  $\mathcal{D}^{\mathrm{S}}$  and the expert dataset  $\mathcal{D}^{\mathrm{E}}$  are collected according to Assumption 1. The mixture state-action distribution (introduced in Appendix C.2) can be calculated as for any  $s\in S$  and  $h\in [H]$ :
$$
d _ {h} ^ {\operatorname {m i x}} (s, a ^ {1}) = \eta d _ {h} ^ {\pi^ {\mathrm {E}}} (s, a ^ {1}) + (1 - \eta) d _ {h} ^ {\pi^ {\beta}} (s, a ^ {1}) = \eta d _ {h} ^ {\pi^ {\mathrm {E}}} (s, a ^ {1}) = \eta \rho (s),
$$
$$
d _ {h} ^ {\mathrm {m i x}} (s, a ^ {2}) = \eta d _ {h} ^ {\pi^ {\mathrm {E}}} (s, a ^ {2}) + (1 - \eta) d _ {h} ^ {\pi^ {\beta}} (s, a ^ {2}) = (1 - \eta) d _ {h} ^ {\pi^ {\beta}} (s, a ^ {2}) = (1 - \eta) \rho (s).
$$
Note that in the population level, the marginal distribution of the union dataset  $\mathcal{D}^{\mathrm{U}}$  in time step  $h$  is exactly  $d_h^{\mathrm{mix}}$ . The mixture policy induced by  $d^{\mathrm{mix}}$  (introduced in Appendix C.2) can be formulated as
$$
\pi_ {h} ^ {\operatorname {m i x}} \left(a ^ {1} | s\right) = \eta , \pi_ {h} ^ {\operatorname {m i x}} \left(a ^ {2} | s\right) = 1 - \eta , \forall s \in \mathcal {S}, h \in [ H ].
$$
Just like before, we have  $d_h^{\pi^{\mathrm{mix}}}(s,a) = d_h^{\mathrm{mix}}(s,a)$ . The policy value of  $\pi^{\mathrm{mix}}$  can be calculated as
$$
V (\pi^ {\operatorname {m i x}}) = \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} d _ {h} ^ {\operatorname {m i x}} (s, a) r _ {h} (s, a) = \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} d _ {h} ^ {\operatorname {m i x}} (s, a ^ {1}) = \eta H.
$$
Recall from Eq. (9) that  $\pi^{\mathrm{NBCU}}$  can be formulated as
$$
\forall h \in [ H ], \quad \pi_ {h} ^ {\mathrm {N B C U}} (a | s) = \left\{ \begin{array}{l l} \frac {n _ {h} ^ {\mathrm {U}} (s , a)}{\sum_ {a ^ {\prime}} n _ {h} ^ {\mathrm {U}} (s , a ^ {\prime})} & \text {i f} \sum_ {a ^ {\prime}} n _ {h} ^ {\mathrm {U}} (s, a ^ {\prime}) > 0 \\ \frac {1}{| A |} & \text {o t h e r w i s e} \end{array} \right. \tag {13}
$$
We can view that the BC's policy learned on the union dataset mimics the mixture policy  $\pi^{\mathrm{mix}}$ . In the following part, we analyze the lower bound on the imitation gap of  $\pi^{\mathrm{NBCU}}$ .
$$
\begin{array}{l} \mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {N B C U}}) \right] = V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {m i x}}) + \mathbb {E} \left[ V (\pi^ {\mathrm {m i x}}) - V (\pi^ {\mathrm {N B C U}}) \right] \\ = H - \eta H + \mathbb {E} \left[ V (\pi^ {\mathrm {m i x}}) - V (\pi^ {\mathrm {N B C U}}) \right] \\ \end{array}
$$
$$
= (1 - \eta) \left(V \left(\pi^ {\mathrm {E}}\right) - V \left(\pi^ {\beta}\right)\right) + \mathbb {E} \left[ V \left(\pi^ {\operatorname {m i x}}\right) - V \left(\pi^ {\mathrm {N B C U}}\right) \right].
$$
Then we consider the term  $\mathbb{E}\left[V(\pi^{\mathrm{mix}}) - V(\pi^{\mathrm{NBCU}})\right]$ .
$$
\begin{array}{l} V (\pi^ {\mathrm {m i x}}) - V (\pi^ {\mathrm {N B C U}}) \\ = \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \left(d _ {h} ^ {\pi^ {\operatorname {m i x}}} (s, a) - d _ {h} ^ {\pi^ {\mathrm {N B C U}}} (s, a)\right) r _ {h} (s, a) \\ = \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \rho (s) \left(\pi_ {h} ^ {\operatorname {m i x}} (a | s) - \pi_ {h} ^ {\mathrm {N B C U}} (a | s)\right) r _ {h} (s, a) \\ = \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \rho (s) \left(\pi_ {h} ^ {\operatorname {m i x}} (a | s) - \pi_ {h} ^ {\mathrm {N B C U}} (a | s)\right) r _ {h} (s, a) \mathbb {I} \{n _ {h} ^ {\mathrm {U}} (s) > 0 \} \\ + \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \rho (s) \left(\pi_ {h} ^ {\operatorname {m i x}} (a | s) - \pi_ {h} ^ {\mathrm {N B C U}} (a | s)\right) r _ {h} (s, a) \mathbb {I} \{n _ {h} ^ {\mathrm {U}} (s) = 0 \}. \\ \end{array}
$$
We take expectation over the randomness in  $\mathcal{D}^{\mathrm{U}}$  on both sides and obtain that
$$
\begin{array}{l} \mathbb {E} \left[ V \left(\pi^ {\text {m i x}}\right) - V \left(\pi^ {\mathrm {N B C U}}\right) \right] (14) \\ = \mathbb {E} \left[ \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \rho (s) \left(\pi_ {h} ^ {\operatorname {m i x}} (a | s) - \pi_ {h} ^ {\mathrm {N B C U}} (a | s)\right) r _ {h} (s, a) \mathbb {I} \{n _ {h} ^ {\mathrm {U}} (s) > 0 \} \right] \\ + \mathbb {E} \left[ \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \rho (s) \left(\pi_ {h} ^ {\operatorname {m i x}} (a | s) - \pi_ {h} ^ {\mathrm {N B C U}} (a | s)\right) r _ {h} (s, a) \mathbb {I} \left\{n _ {h} ^ {\mathrm {U}} (s) = 0 \right\} \right]. (15) \\ \end{array}
$$
For the first term in RHS, we have that
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \rho (s) \left(\pi_ {h} ^ {\operatorname {m i x}} (a | s) - \pi_ {h} ^ {\mathrm {N B C U}} (a | s)\right) r _ {h} (s, a) \mathbb {I} \{n _ {h} ^ {\mathrm {U}} (s) > 0 \} \right] \\ = \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \rho (s) r _ {h} (s, a) \mathbb {E} \left[ \left(\pi_ {h} ^ {\operatorname {m i x}} (a | s) - \pi_ {h} ^ {\mathrm {N B C U}} (a | s)\right) \mathbb {I} \{n _ {h} ^ {\mathrm {U}} (s) > 0 \} \right] \\ = \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \rho (s) r _ {h} (s, a) \mathbb {P} \left(n _ {h} ^ {\mathrm {U}} (s) > 0\right) \mathbb {E} \left[ \pi_ {h} ^ {\text {m i x}} (a | s) - \pi_ {h} ^ {\mathrm {N B C U}} (a | s) \mid n _ {h} ^ {\mathrm {U}} (s) > 0 \right] \\ = 0. \\ \end{array}
$$
The last equation follows the fact that  $\pi_h^{\mathrm{NBCU}}(a|s)$  is an unbiased estimation of  $\pi_h^{\mathrm{mix}}(a|s)$ , so  $\mathbb{E}[\pi_h^{\mathrm{mix}}(a|s) - \pi_h^{\mathrm{NBCU}}(a|s) \mid n_h^{\mathrm{U}}(s) > 0]$ . For the second term in Eq. (15), we have that