SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {T V} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} ^ {\text {m i x}} (\cdot | s)\right) = \sum_ {a \neq \pi_ {h} ^ {\mathrm {E}} (s)} \pi_ {h} ^ {\text {m i x}} (a | s) \\ = \sum_ {a \neq \pi_ {h} ^ {\mathrm {E}} (s)} \left(1 - (1 - \eta) \left(1 - \eta^ {\prime}\right)\right) \pi_ {h} ^ {\mathrm {E}} (a | s) + (1 - \eta) \left(1 - \eta^ {\prime}\right) \frac {1}{| \mathcal {A} |} \\ = (1 - \eta) \left(1 - \eta^ {\prime}\right) \left(1 - \frac {1}{| \mathcal {A} |}\right). \\ \end{array}
$$
$$
H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \operatorname {T V} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} ^ {\operatorname {m i x}} (\cdot | s)\right) \right] \leq (1 - \eta) (1 - \eta^ {\prime}) H ^ {2} \left(1 - \frac {1}{| \mathcal {A} |}\right).
$$
Now we analyze the second term of
$$
H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \operatorname {T V} \left(\pi_ {h} ^ {\operatorname {m i x}} (\cdot | s), \pi_ {h} ^ {\mathrm {N B C U}} (\cdot | s)\right) \right].
$$
Recall the formula of  $\pi^{\mathrm{NBCU}}$ .
$$
\pi_ {h} ^ {\mathrm {N B C U}} (a | s) = \left\{ \begin{array}{l l} \frac {n _ {h} ^ {\mathrm {U}} (s , a)}{n _ {h} ^ {\mathrm {U}} (s)} & \text {i f} n _ {h} ^ {\mathrm {U}} (s) > 0 \\ \frac {1}{| A |} & \text {o t h e r w i s e} \end{array} \right.
$$
Notice that  $\pi^{\mathrm{NBCU}}$  is the maximum likelihood estimation of  $\pi^{\mathrm{mix}}$ . According to the concentration inequality of total variation [53], for each  $(s,h)\in S\times [H]$ , for any fixed  $\delta \in (0,1)$ , when  $n_h^{\mathrm{U}}(s) > 0$  with probability at least  $1 - \delta$ , we have
$$
\mathrm {T V} \left(\pi_ {h} ^ {\mathrm {m i x}} (\cdot | s), \pi_ {h} ^ {\mathrm {N B C U}} (\cdot | s)\right) \leq \sqrt {\frac {| \mathcal {A} | \log (3 / \delta)}{n _ {h} ^ {\mathrm {U}} (s)}}.
$$
When  $n_h^{\mathrm{U}}(s) = 0$ , we have that
$$
\operatorname {T V} \left(\pi_ {h} ^ {\operatorname {m i x}} (\cdot | s), \pi_ {h} ^ {\mathrm {N B C U}} (\cdot | s)\right) \leq 1 \leq \sqrt {| \mathcal {A} | \log (3 / \delta)}.
$$
By combining the above two inequalities, for each  $(s,h)\in S\times [H]$ , with probability at least  $1 - \delta$ , we have
$$
\operatorname {T V} \left(\pi_ {h} ^ {\operatorname {m i x}} (\cdot | s), \pi_ {h} ^ {\mathrm {N B C U}} (\cdot | s)\right) \leq \sqrt {\frac {| \mathcal {A} | \log (3 / \delta)}{\max  \left\{n _ {h} ^ {\mathrm {U}} (s) , 1 \right\}}}.
$$
Applying union bound yields that with probability at least  $1 - \delta /2$ , for all  $(s,h)\in \mathcal{S}\times [H]$
$$
\mathrm {T V} \left(\pi_ {h} ^ {\mathrm {m i x}} (\cdot | s), \pi_ {h} ^ {\mathrm {N B C U}} (\cdot | s)\right) \leq \sqrt {\frac {| \mathcal {A} | \log (6 | \mathcal {S} | H / \delta)}{\max \{n _ {h} ^ {\mathrm {U}} (s) , 1 \}}}.
$$
$$
H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \mathrm {T V} \left(\pi_ {h} ^ {\mathrm {m i x}} (\cdot | s), \pi_ {h} ^ {\mathrm {N B C U}} (\cdot | s)\right) \right]
$$
$$
\begin{array}{l} \leq H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \sqrt {\frac {| \mathcal {A} | \log (6 | \mathcal {S} | H / \delta)}{\max \left\{n _ {h} ^ {\mathrm {U}} (s) , 1 \right\}}} \right] \\ = H \sqrt {| \mathcal {A} | \log (6 | \mathcal {S} | H / \delta)} \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \sqrt {\frac {1}{\max \{n _ {h} ^ {\mathrm {U}} (s) , 1 \}}} \right] \\ = H \sqrt {| \mathcal {A} | \log (6 | \mathcal {S} | H / \delta)} \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} \sqrt {d _ {h} ^ {\pi^ {\mathrm {E}}} (s)} \sqrt {\frac {d _ {h} ^ {\pi^ {\mathrm {E}}} (s)}{\max \left\{n _ {h} ^ {\mathrm {U}} (s) , 1 \right\}}} \\ \leq H \sqrt {| \mathcal {A} | \log (6 | \mathcal {S} | H / \delta)} \sum_ {h = 1} ^ {H} \sqrt {\sum_ {s \in \mathcal {S}} \frac {d _ {h} ^ {\pi^ {\mathrm {E}}} (s)}{\max \left\{n _ {h} ^ {\mathrm {U}} (s) , 1 \right\}}}. \\ \end{array}
$$
Here the last inequality follows Cauchy-Swartz inequality. Notice that  $n_h^{\mathrm{U}}(s)$  is the number of times that the state  $s$  appears in  $\mathcal{D}^{\mathrm{U}}$  in time step  $h$  and thus follows the Binomial distribution of  $\operatorname{Bin}(N_{\mathrm{tot}}, d_h^{\pi^{\mathrm{E}}} (s))$ . By applying Lemma 4, for each  $(s,h)$ , with probability at least  $1 - \delta$ , we have
$$
\frac {d _ {h} ^ {\pi^ {\mathrm {E}}} (s)}{\max  \left\{n _ {h} ^ {\mathrm {U}} (s) , 1 \right\}} \leq \frac {8 \log (1 / \delta)}{N _ {\mathrm {t o t}}}.
$$
By union bound, with probability at least  $1 - \delta /2$ , for all  $(s,h)\in S\times [H]$
$$
\frac {d _ {h} ^ {\pi^ {\mathrm {E}}} (s)}{\max  \left\{n _ {h} ^ {\mathrm {U}} (s) , 1 \right\}} \leq \frac {8 \log (2 | \mathcal {S} | H / \delta)}{N _ {\mathrm {t o t}}}.
$$
Then, with probability at least  $1 - \delta$ , we have
$$
H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \mathrm {T V} \left(\pi_ {h} ^ {\operatorname {m i x}} (\cdot | s), \pi_ {h} ^ {\mathrm {N B C U}} (\cdot | s)\right) \right] \leq H ^ {2} \sqrt {\frac {8 | \mathcal {S} | | \mathcal {A} | \log^ {2} (6 | \mathcal {S} | H / \delta)}{N _ {\mathrm {t o t}}}}.
$$
Finally, we upper bound the imitation gap. With probability at least  $1 - \delta$ , we have
$$
V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {N B C U}}) \leq (1 - \eta) (1 - \eta^ {\prime}) \left(1 - \frac {1}{| \mathcal {A} |}\right) + H ^ {2} \sqrt {\frac {8 | \mathcal {S} | | \mathcal {A} | \log^ {2} (6 | \mathcal {S} | H / \delta)}{N _ {\mathrm {t o t}}}}.
$$
We set  $\delta = H / N_{\mathrm{tot}}$  and obtain that
$$
\begin{array}{l} \mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {N B C U}}) \right] \\ \leq \delta H + (1 - \delta) \left((1 - \eta) (1 - \eta^ {\prime}) \left(1 - \frac {1}{| \mathcal {A} |}\right) + H ^ {2} \sqrt {\frac {8 | \mathcal {S} | | \mathcal {A} | \log^ {2} (6 | \mathcal {S} | H / \delta)}{N _ {\mathrm {t o t}}}}\right) \\ \leq \frac {H ^ {2}}{N _ {\mathrm {t o t}}} + (1 - \eta) (1 - \eta^ {\prime}) \left(1 - \frac {1}{| \mathcal {A} |}\right) + H ^ {2} \sqrt {\frac {8 | \mathcal {S} | | \mathcal {A} | \log^ {2} (6 | \mathcal {A} | N _ {\mathrm {t o t}})}{N _ {\mathrm {t o t}}}} \\ \leq (1 - \eta) (1 - \eta^ {\prime}) \left(1 - \frac {1}{| \mathcal {A} |}\right) + 4 H ^ {2} \sqrt {\frac {2 | \mathcal {S} | | \mathcal {A} | \log^ {2} (6 | \mathcal {A} | N _ {\mathrm {t o t}})}{N _ {\mathrm {t o t}}}}. \\ \end{array}
$$
On the other hand, we directly have  $\mathbb{E}[V(\pi^{\mathrm{E}}) - V(\pi^{\mathrm{NBCU}})]\leq H$ . We complete the proof.
F.3 Proof of Theorem 6
In the poison setting, we can conduct the same analysis as in the proof of Theorem 3 and demonstrate that  $\pi^{\mathrm{ISW - BC}}(\pi_h^{\mathrm{E}}(s)|s) = 1$ ,  $\forall (s,\pi_h^{\mathrm{E}}(s))\in \mathcal{D}_h^{\mathrm{E}}\cup \mathcal{D}_h^{\mathrm{S},1}$ , where  $\mathcal{D}_h^{\mathrm{E}}$  is the set of state-action pairs in  $\mathcal{D}^{\mathrm{E}}$  in time step  $h$  and  $\mathcal{D}_h^{\mathrm{S},1} = \{(s,a)\in \mathcal{D}_h^{\mathrm{S}}:d_h^{\pi^{\mathrm{E}}} (s) > 0,a = \pi_h^{\mathrm{E}}(s)\}$ . According to [42, Lemma 4.3], we have
$$
V \left(\pi^ {\mathrm {E}}\right) - V \left(\pi^ {\mathrm {I S W - B C}}\right) \leq H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \mathrm {T V} \left(\pi_ {h} ^ {\mathrm {E}} (\cdot | s), \pi_ {h} ^ {\mathrm {I S W - B C}} (\cdot | s)\right) \right].
$$
Since the expert policy is assumed to be deterministic, we can obtain
$$
\begin{array}{l} V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {I S W - B C}}) \leq H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \mathbb {E} _ {a \sim \pi_ {h} ^ {\mathrm {I S W - B C}} (\cdot | s)} \left[ \mathbb {I} \left\{a \neq \pi_ {h} ^ {\mathrm {E}} (s) \right\} \right] \right] \\ \leq H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \mathbb {I} \left\{\left(s, \pi_ {h} ^ {\mathrm {E}} (s)\right) \notin \mathcal {D} _ {h} ^ {\mathrm {E}} \cup \mathcal {D} _ {h} ^ {\mathrm {S}, 1} \right\} \right]. \\ \end{array}
$$
Let  $\mathcal{D}^{\mathrm{S,clean}}$  denote the non-corrupted dataset in  $\mathcal{D}^{\mathrm{S}}$ . Then we can obtain that
$$
\begin{array}{l} V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {I S W - B C}}) \stackrel {(a)} {\leq} H \sum_ {h = 1} ^ {H} \mathbb {E} _ {s \sim d _ {h} ^ {\pi^ {\mathrm {E}}} (\cdot)} \left[ \mathbb {I} \left\{\left(s, \pi_ {h} ^ {\mathrm {E}} (s)\right) \notin \mathcal {D} _ {h} ^ {\mathrm {E}} \cup \mathcal {D} _ {h} ^ {\mathrm {S , c l e a n}} \right\} \right] \\ = H \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} d _ {h} ^ {\pi^ {\mathrm {E}}} (s) \mathbb {I} \left\{\left(s, \pi_ {h} ^ {\mathrm {E}} (s)\right) \notin \mathcal {D} _ {h} ^ {\mathrm {E}} \cup \mathcal {D} _ {h} ^ {\mathrm {S , c l e a n}} \right\}, \\ \end{array}
$$
where  $\mathcal{D}_h^{\mathrm{S,clean}}$  denotes the set of state-action pairs in  $\mathcal{D}^{\mathrm{S,clean}}$  in time step  $h$ . Inequality  $(a)$  follows that  $\mathcal{D}_h^{\mathrm{S,clean}} \subseteq \mathcal{D}_h^{\mathrm{S,1}}$  since  $\mathcal{D}^{\mathrm{S,clean}}$  is collected by the expert policy. Taking expectation over the randomness in  $\mathcal{D}^{\mathrm{E}}$  and  $\mathcal{D}^{\mathrm{S,clean}}$  on both sides yields that
$$
\mathbb {E} _ {\mathcal {D} ^ {\mathrm {E}}, \mathcal {D} ^ {\mathrm {S}, \mathrm {c l e a n}}} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {I S W - B C}}) \right] \leq H \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} d _ {h} ^ {\pi^ {\mathrm {E}}} (s) \mathbb {P} \left((s, \pi_ {h} ^ {\mathrm {E}} (s)) \notin \mathcal {D} _ {h} ^ {\mathrm {E}} \cup \mathcal {D} _ {h} ^ {\mathrm {S}, \mathrm {c l e a n}}\right).
$$
Notice that both  $\mathcal{D}^{\mathrm{E}}$  and  $\mathcal{D}^{\mathrm{S,clean}}$  are collected by the expert policy. Then if  $|\mathcal{D}^{\mathrm{E}}| + |\mathcal{D}^{\mathrm{S,clean}}| \geq 1$ , we can calculate that for each  $(s,h) \in S \times [H]$ ,
$$
\begin{array}{l} d _ {h} ^ {\pi^ {\mathrm {E}}} (s) \mathbb {P} \left(\left(s, \pi_ {h} ^ {\mathrm {E}} (s)\right) \notin \mathcal {D} _ {h} ^ {\mathrm {E}} \cup \mathcal {D} _ {h} ^ {\mathrm {S , c l e a n}}\right) = d _ {h} ^ {\pi^ {\mathrm {E}}} (s) \left(1 - d _ {h} ^ {\pi^ {\mathrm {E}}} (s)\right) ^ {| \mathcal {D} ^ {\mathrm {E}} | + | \mathcal {D} ^ {\mathrm {S , c l e a n}} |} \\ \leq \frac {4}{9 (| \mathcal {D} ^ {\mathrm {E}} | + | \mathcal {D} ^ {\mathrm {S , c l e a n}} |)}, \\ \end{array}
$$
where the last inequality follows Lemma 5. If  $|\mathcal{D}^{\mathrm{E}}| + |\mathcal{D}^{\mathrm{S,clean}}| = 0$ , we directly have that
$$
d _ {h} ^ {\pi^ {\mathrm {E}}} (s) \mathbb {P} \left((s, \pi_ {h} ^ {\mathrm {E}} (s)) \notin \mathcal {D} _ {h} ^ {\mathrm {E}} \cup \mathcal {D} _ {h} ^ {\mathrm {S , c l e a n}}\right) \leq 1 = \frac {1}{\max \{| \mathcal {D} ^ {\mathrm {E}} | + | \mathcal {D} ^ {\mathrm {S , c l e a n}} | , 1 \}}.
$$
We unify the above two inequalities and get that
$$
d _ {h} ^ {\pi^ {\mathrm {E}}} (s) \mathbb {P} \left((s, \pi_ {h} ^ {\mathrm {E}} (s)) \notin \mathcal {D} _ {h} ^ {\mathrm {E}} \cup \mathcal {D} _ {h} ^ {\mathrm {S , c l e a n}}\right) \leq \frac {1}{\max  \{| \mathcal {D} ^ {\mathrm {E}} | + | \mathcal {D} ^ {\mathrm {S , c l e a n}} | , 1 \}}.
$$
Now we proceed to upper bound the imitation gap.
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} ^ {\mathrm {E}}, \mathcal {D} ^ {\mathrm {S}, \text {c l e a n}}} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {I S W - B C}}) \right] \leq H \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} d _ {h} ^ {\pi^ {\mathrm {E}}} (s) \mathbb {P} \left((s, \pi_ {h} ^ {\mathrm {E}} (s)) \notin \mathcal {D} _ {h} ^ {\mathrm {E}} \cup \mathcal {D} _ {h} ^ {\mathrm {S}, \text {c l e a n}}\right) \\ \leq \frac {| \mathcal {S} | H ^ {2}}{\operatorname* {m a x} \left\{| \mathcal {D} ^ {\mathrm {E}} | + | \mathcal {D} ^ {\mathrm {S , c l e a n}} | , 1 \right\}}. \\ \end{array}
$$
Note that  $|\mathcal{D}^{\mathrm{E}}| + |\mathcal{D}^{\mathrm{S,clean}}| \sim \mathrm{Bin}(N_{\mathrm{tot}}, \eta + (1 - \eta)\eta')$ . Taking expectation with respect to  $|\mathcal{D}^{\mathrm{E}}| + |\mathcal{D}^{\mathrm{S,clean}}|$  yields that
$$
\begin{array}{l} \mathbb {E} \left[ V (\pi^ {\mathrm {E}}) - V (\pi^ {\mathrm {I S W - B C}}) \right] \leq \mathbb {E} \left[ \frac {| \mathcal {S} | H ^ {2}}{\max \{| \mathcal {D} ^ {\mathrm {E}} | + | \mathcal {D} ^ {\mathrm {S , c l e a n}} | , 1 \}} \right] \\ \leq \mathbb {E} \left[ \frac {2 | \mathcal {S} | H ^ {2}}{| \mathcal {D} ^ {\mathrm {E}} | + | \mathcal {D} ^ {\mathrm {S , c l e a n}} | + 1} \right] \\ \stackrel {(a)} {\leq} \frac {2 | \mathcal {S} | H ^ {2}}{N _ {\mathrm {t o t}} (\eta + (1 - \eta) \eta^ {\prime})} \\ = \frac {2 | S | H ^ {2}}{N _ {\mathrm {E}} + \eta^ {\prime} N _ {\mathrm {S}}}. \\ \end{array}
$$
Inequality  $(a)$  follows Lemma 3. We finish the proof.
G Technical Lemmas
Lemma 3. For any  $N \in \mathbb{N}_+$  and  $p \in (0,1)$ , if the random variable  $X$  follows the binomial distribution, i.e.,  $X \sim \operatorname{Bin}(N,p)$ , then we have that
$$
\mathbb {E} \left[ \frac {1}{X + 1} \right] \leq \frac {1}{N p}.
$$
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{X + 1} \right] = \sum_ {x = 0} ^ {N} \left(\frac {1}{x + 1}\right) \frac {N !}{x ! (N - x) !} p ^ {x} (1 - p) ^ {N - x} \\ = \frac {1}{(N + 1) p} \sum_ {x = 1} ^ {N + 1} \left(\frac {(N + 1) !}{x ! (N + 1 - x) !}\right) p ^ {x} (1 - p) ^ {N + 1 - x} \\ = \frac {1}{(N + 1) p} (1 - (1 - p) ^ {N + 1}) \leq \frac {1}{N p}. \\ \end{array}
$$
Lemma 4 (Binomial concentration (Lemma A.1 in [57])) For any  $N \in \mathbb{N}_+$  and  $p \in (0,1)$ , suppose  $X \sim \mathrm{Bin}(N,p)$ . Then with probability at least  $1 - \delta$ , we have
$$
\frac {p}{\operatorname* {m a x} \{X , 1 \}} \leq \frac {8 \log (1 / \delta)}{N}.
$$
Lemma 5. For any  $N\in \mathbb{N}_{+}$  and  $x\in [0,1]$ , consider the function  $f(x)\coloneqq x(1 - x)^{N}$ , then we have
$$
\forall x \in [ 0, 1 ], f (x) \leq \frac {4}{9 N}.
$$
Proof. We calculate that  $f'(x) = (1 - x)^{N-1}(1 - (N + 1)x)$ . It is direct to have that  $f(x)$  achieves its maximum at  $x^{\star} = 1 / (N + 1)$ . Furthermore, we have
$$
f \left(\frac {1}{N + 1}\right) = \frac {1}{N} \left(1 - \frac {1}{N + 1}\right) ^ {N + 1} \stackrel {(a)} {\leq} \frac {1}{e N} \leq \frac {4}{9 N}.
$$
Inequality  $(a)$  follows that  $(1 + x / N)^{N}\leq \exp (x),\forall N\geq 1,|x|\leq N$  . We complete the proof.
H Experiments Details and Additional Results
H.1 Experiment Details
In this section, we present the experiment details to facilitate the replication of our results. The experiments are conducted on a machine comprising 48 CPU cores and 4 V100 GPU cores. We repeat each experiment 5 times using different random seeds (2021, 2022, 2023, 2024, and 2025).
H.1.1 Robotic Locomotion Control
In this study, we evaluate the performance of various imitation learning algorithms on four locomotion control tasks from the MuJoCo suite: Ant-v2, HalfCheetah-v2, Hopper-v2, and Walker2d-v2. These tasks are widely used in the literature and are considered challenging benchmarks.
To train the expert policy, we use the online Soft Actor-Critic (SAC) algorithm [19] with 1 million training steps. We implement the algorithm using the rlkit codebase, which is available at https://github.com/rail-berkeley/rlkit. The training curves of the online SAC agent are shown in Figure 5. We treat the resulting policy as the expert policy and use it to generate expert trajectories.
Figure 5: Training curves of online SAC on 4 locomotion control environments.
In our experimental setup, we utilize an expert dataset comprising of 1 expert trajectory collected by the trained SAC agent. Additionally, all algorithms are provided with a supplementary dataset. There are two settings of the supplementary data.
- Full Replay. The supplementary dataset is obtained from the replay buffer of the online SAC agent, which has over one million samples, equivalent to  $1000+$  trajectories. The rapid convergence of online SAC, as illustrated in Figure 5, implies that the replay buffer is enriched with a substantial number of expert-level trajectories. As a result, we expect that utilizing the supplementary data without any modification may lead to desirable results.
- Noisy Expert. The supplementary dataset comprises of 10 clean expert trajectories and 5 noisy expert trajectories. In this case, we replace the action labels in the noisy trajectories with random actions drawn from  $[-1,1]$ . This replacement creates noisy action labels for the expert states, leading to a significant distribution shift at the state-action level, as noted in Remark 1. The high degree of distribution shift makes it challenging for using the supplementary data.
We use a 2-hidden-layer multi-layer perceptron (MLP) with hidden size 256 and ReLU activation for all algorithms, as the state information in locomotion control tasks is informative by design. The codebase of DemoDICE is based on the original authors' work, which can be accessed at https://github.com/KAIST-AILab/imitation-dice. For DWBC, we also use the authors' codebase, which is available at https://github.com/ryanxhr/DWBC. We experimented with different hyper-parameters for both algorithms but found that the default parameters provided by the authors work well. We normalize state observations in the dataset before training all algorithms, following [25]. This is crucial for achieving satisfactory performance.
In training the discriminator of ISW-BC, we use the gradient penalty (GP) regularization, as recommended by [25]. We add the following loss to the original loss (4) to enforce 1-Lipschitz continuity:
$$
\min  _ {\theta} \sum_ {(s, a) \in \mathcal {B}} \left(\left\| g (s, a; \theta) \right\| - 1\right) ^ {2},
$$
where  $g$  is the gradient of the discriminator  $c(s, a; \theta)$ , and  $\mathcal{B}$  is a mini-batch. This promotes the learning of smooth features and can improve generalization performance.
In our implementation of ISW-BC, we employ 2-hidden-layer MLPs with 256 hidden units and ReLU activation for both the discriminator and policy networks. We use a batch size of 256 and Adam optimizer with a learning rate of 0.0003 for training both networks. The training objective is to maximize the log-likelihood. We set  $\delta$  to 0 and use a gradient penalty coefficient of 8 by default, unless otherwise stated. The training process is carried out for 1 million iterations. We evaluate the performance every 10k iterations with 10 episodes. The normalized score in the last column of Table 2 is computed in the following way:
$$
\text {N o r m a l i z e d} = \frac {\text {E x p e r t p e r f o r m a n c e} - \text {A g e n t p e r f o r m a n c e}}{\text {E x p e r t p e r f o r m a n c e} - \text {R a n d o m p o l i c y p e r f o r m a n c e}}. \tag {20}
$$
H.1.2 Atari Video Games
We evaluate algorithms on a set of 5 Atari video games from the standard benchmark: Alien, MsPacman, Phoenix, Qbert, and SpaceInvaders. We preprocess the game environments using a standard set of procedures, including sticky actions with a probability of 0.25, grayscale, downsampling to an image size of [84, 84], and stacking frames of 4. These procedures follow the instructions provided by the dopamine codebase, which is available at https://github.com/google/dopamine/blob/master/dopamine/discrete_domains/atari_lib.py. The final image inputs are of shape (84, 84, 4).
We use the replay buffer data from an online DQN agent, which is publicly available at https://console.cloud.google.com/storage/browser/atari-replay-datasets, thanks to the work of [2]. The dataset consists of 200 million frames, divided into 50 indexed buckets (ranging from 0 to 49). However, using the entire dataset is computationally infeasible and unnecessary for our task. Therefore, we select specific buffer buckets for imitation learning.
We choose the expert data from bucket index 49, using only the first 400K frames for training. This makes the task challenging (we find that BC performs well with 1M frames of expert data). For the full replay setting, we select supplementary data from buffer indices 45 to 48, using the first 400K frames from each bucket. This yields a supplementary dataset that is 4 times larger than the expert data. In the noisy task setting, we follow the same procedure for selecting supplementary data, but replace the action labels with random labels on buffer index 45.
All agents employ the same convolutional neural network (CNN) architecture as the DQN agent, consisting of three convolutional blocks. The first block applies a filter size of 8, a stride of 4, and has a channel size of 32. The second block uses a filter size of 4, a stride of 4, and a channel size of 64, while the third block applies a filter size of 3, a stride of 4, and has a channel size of 64. All blocks use the ReLU activation function. The feature representations are flattened to a vector, on which a 1-hidden-layer MLP with a hidden size of 512 and ReLU activation function is applied. Finally, the outputs are passed through a softmax function to obtain a probability distribution.
Atari games are not considered in [25, 61] and public implementations of DemoDICE and DWBC for Atari games are not available. To use these methods in the Atari environment, we extend their original implementation by replacing the MLP used in locomotion control with the CNN described earlier. Implementing ISW-BC is a little more complicated. We use the same CNN policy network as in the other methods, but find that directly training the discriminator from scratch is less effective. This is because the discriminator tends to focus on irrelevant background information instead of the decision-centric part. To overcome this issue, we build the discriminator upon the feature extractor of the policy network, leveraging its ability to extract useful information. The discriminator is an MLP with ReLU activation and a hidden size of 1024: the image feature representation has a dimension 512 and the action feature representation also has a dimension 512 (we randomly project one-hot discrete actions to a 512-dimension space). We find that the depth of the MLP is crucial for performance, using a depth of 1 for the full replay setting and 3 for the noisy expert setting. We clip the importance sampling ratio for numerical stability, using a minimum value of 0 and a maximum value of 5 for the full replay setting, and a minimum value of 0.2 and a maximum value of 5 for the noisy expert setting. We provide ablation studies of these hyperparameters in Appendix H.2.2.
All methods were optimized using the Adam optimizer with a learning rate of 0.00025 and a batch size of 256. The training objective is to maximize the log-likelihood. The training process consisted of  $200\mathrm{K}$  gradient steps. Every  $2\mathrm{K}$  gradient steps, the algorithms were evaluated by running 10 episodes and computing the raw game scores. The normalized score in the last column of Table 3 is computed by Eq. (20).
H.1.3 Image Classification
We utilize the publicly available DomainNet dataset [36] for our experiments, which can be accessed at http://csr.bu.edu/ftp/visda/2019/multi-source. This dataset comprises six sub-datasets: clipart, infographic, painting, quickdraw, real, and sketch, with 2103, 2626, 2472, 4000, 4864, and 2213 images, respectively. Our task involves recognizing objects from 10 different classes: bird, feather, headphones, ice_cream, teapot, tiger, whale, windmill, wine_glass, and zebra. We divided the images into training and test sets, with  $80\%$  for training and  $20\%$  for testing.