- Can our method discriminate expert and non-expert trajectories in the task-agnostic dataset? In Sec. F.1, we visualize the change of  $R(s)$  and behavior cloning weight  $W(s, a)$  along the trajectory, as well as the average  $R(s)$  and  $W(s, a)$  for expert and non-expert trajectories in the task-agnostic dataset  $D_{\mathrm{TA}}$ . We find that our method successfully discriminates expert and non-expert trajectories in the task-agnostic dataset across multiple experiments.
- How does our method perform compared to other offline RL/IL methods? In addition to LobsDICE and SMODICE, in the appendix we compare our performance to the following offline RL/IL methods: a) a very recent DICE method, ReCOIL; b) methods with extra access to expert actions or reward labels, such as DWBC [68], model-based RL methods MOReL [26] and MOPO [70]; c) offline adaption for Advantage-Weighted Regression(AWR) [45], MARWIL [62]; d) a recent Wasserstein-based offline IL method, OTR [40]. We found our method to be consistently better than all these methods. See Sec. F.2 for details.
- How does our method and baselines perform on the kitchen environment with "KitchenComplete-v0"? The kitchen testbed adopted in SMODICE uses an expert trajectory from the "kitchen-complete-v0" environment in D4RL, which has a different subtask sequence from the ones evaluated in the environment and the ones in the task-agnostic data ("kitchen-mixed-v0"); the two environments also have different state spaces. When changed to "kitchen-complete-v0," all baselines except BC fail, because the state space between task-agnostic data and evaluation environment is different. Nonetheless our method still works. See Sec. F.3 for details.
- How sensitive is our method to hyperparameters, such as  $\alpha$ ,  $\beta_{1}$ ,  $\beta_{2}$ ,  $\eta_{p}$ , and  $\gamma$ ? We conduct a sensitivity analysis in Sec. F.4, and find our method to be generally robust to a reasonable selection of hyperparameters.
- Our method and baselines use different batch sizes. Is this a fair comparison? In Sec. F.4.6, we test our method with batch size 512 and SMODICE/LobsDICE with batch size 8192. We find that our method is more stable with a larger batch size for a few settings, but SMODICE and LobsDICE do not generally benefit from an increased batch size. In addition, our method with batch size 512 is still better than the baselines. Thus, the comparison is fair to SMODICE and LobsDICE.
- How important is the Lipschitz regularizer for training of  $R(s)$ ? A Lipschitz regularizer is used in many methods tested in this work, including our method, SMODICE, LobsDICE, and ORIL. In Sec. F.4.7, we found this regularizer to be crucial for the generalization of the discriminator and  $R(s)$ .
- Our standard imitation from observation setting is different from that of SMODICE. Is this fair? Compared with the SMODICE's original setting, in Sec. 4.3 we focused on a more challenging experiment setting with less expert trajectories in the task-agnostic dataset for the four Mujoco environments (hopper, halfcheetah, ant, and walker2d). In the appendix, we also provide experimental evaluation under the SMODICE's original setting, and find our method still outperforming baselines. See Sec. F.5 for details.