SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We visualize this process for the  $|S||\mathcal{A}| = 2$  case in Fig. 2. Assuming we are able to recover  $c^*$ , we can guarantee that our learners will be able to act safely, regardless of the task they are asked to do. However, the assumptions required to do so are quite strong: we are effectively asking for our expert policies to form a basis for the space of occupancy measures, which means we must see expert data for a large set of diverse tasks. Furthermore, we need the experts to be reward-optimal.
Identifiability (the goal of prior works like Chou et al. [2020], Amin et al. [2017]) is too strong a goal as it requires us to estimate the value of the constraint everywhere in the state-action space. If we know the learner will only be incentivized to go to a certain subset of states (as is often true in practice), we can guarantee safety without fully identifying  $c^*$ . Therefore, we now consider how, by making distributional assumptions on how tasks are generated, we can generalize to novel tasks.
3.6 A Statistical Condition for Generalization
Assume that tasks  $\tau$  are drawn i.i.d. from some  $P(\tau)$ . Then, even if we do not see a wide enough diversity of expert policies to guarantee identifiability of the ground-truth constraint function, with enough samples, we can ensure we do well in expectation over tasks. For some constraint  $c$ , let us define
$$
V (c) = \mathbb {E} _ {\tau \sim P (\tau)} [ J (\pi^ {\tau}, c) - J \left(\pi_ {E} ^ {\tau}, c\right) ], \tag {14}
$$
where  $\lambda^{\tau},\pi^{\tau} = \mathrm{CRL}(r^{\tau},c)$  denote the solutions to the inner optimization problem. We begin by proving the following lemma.
Lemma 3.3. With
$$
K \geq O \left(\log \left(\frac {\left| \mathcal {F} _ {c} \right|}{\delta}\right) \frac {(2 T) ^ {2}}{\epsilon^ {2}}\right) \tag {15}
$$
samples, we have that with probability  $\geq 1 - \delta$ , we will be able to estimate all  $|\mathcal{F}_c|$  population estimates of  $V(c)$  within  $\epsilon$  absolute error.
Note that we perform the above analysis for finite classes but one could easily extend it [Sriperumbudur et al., 2009]. The takeaway from the above lemma is that if we observe a sufficient number of tasks, we can guarantee that we can estimate the population loss of all constraints, up to some tolerance.
Consider the learner being faced with a new task they have never seen before at test time. Unlike in the single task case, where it is clear how to set the cost limit passed to CRL, it is not clear how to do so for a novel task. Hence, we make the following assumption.
Assumption 3.4. We assume that  $\mathbb{E}_{\tau}[J(\pi_{E}^{\tau},c^{*})]\leq 0$  , and that  $\forall c\in \mathcal{F}_c,\exists \pi \in \Pi$  s.t.  $J(\pi ,c)\leq 0$
This (weak) assumption allows us to a) use a cost limit of 0 for our CRL step and b) search over a subset of  $\mathcal{F}_c$  that the expert is safe under. Under this assumption, we are able to prove the following:
Theorem 3.5. Let  $c_{1:N}$  be the iterates produced by Algorithm 3 with  $K(\epsilon, \delta)$  chosen as in Lemma 3.3 and let  $\bar{\epsilon} = \frac{1}{N} \sum_{i}^{N} \epsilon_{i}$  denote their time-averaged regret. Then, w.p.  $\geq 1 - \delta$ , there exists a  $c \in c_{1:N}$  such that  $\pi(r) = CRL(r, c, \delta = 0)$  satisfies
$$
\mathbb {E} _ {\tau \sim P (\tau)} [ J (\pi (r ^ {\tau}), c ^ {*}) - J (\pi_ {E} ^ {\tau}, c ^ {*}) ] \leq \bar {\epsilon} T + 3 \epsilon T a n d \mathbb {E} _ {\tau \sim P (\tau)} [ J (\pi (r ^ {\tau}), r ^ {\tau}) - J (\pi_ {E} ^ {\tau}, r ^ {\tau}) ] \geq - 2 \epsilon T.
$$
In short, if we observe enough tasks, we are able to learn a constraint that, when optimized under, leads to policies that approximately Pareto-dominate those of the experts on average.
We now turn our attention to the practical implementation of these algorithms.
4 Practical Algorithm
We provide practical implementations of constrained reinforcement learning and inverse constraint learning and benchmark their performance on several continuous control tasks. We first describe the environments we test our algorithms on. Then, we provide results showing that our algorithms learn policies that match expert performance and constraint violation. While it is hard to guarantee constraint recovery in theory, we show that we can recover the ground-truth constraint empirically if we search over a restricted enough function class.
We focus on the ant environment from the PyBullet [Coumans and Bai, 2016] and MuJoCo [Todorov et al., 2012] benchmarks. The default reward function incentivizes progress along the positive  $x$  direction. For our single-task experiments, we consider a velocity and position constraint on top of this reward function.
1. Velocity Constraint:  $\frac{\|q_{t+1} - q_t\|_2}{\mathrm{d}t} \leq 0.75$  where  $q_t$  is the ant's position
2. Position Constraint:  $0.5x_{t} - y_{t} \leq 0$  where  $x_{t}, y_{t}$  are the ant's coordinates
For our multi-task experiments, we build upon the D4RL [Fu et al., 2020] AntMaze benchmark. The default reward function incentivizes the agent to navigate a fixed maze to a random goal position:  $\exp(-\|q_{\mathrm{goal}} - q_t\|_2)$ . We modify this environment such that the walls of the maze are permeable, but the agent incurs a unit step-wise cost for passing through the maze walls.
Our expert policies are generated by running CRL with the ground-truth constraint. We use the Tianshou [Weng et al., 2022] implementation of PPO [Schulman et al., 2017] as our baseline policy optimizer. Classical Lagrangian methods exactly follow the gradient update shown in Algorithm 1, but they are susceptible to oscillating learning dynamics and constraint-violating behavior during training. The PID Lagrangian method [Stooke et al., 2020] extends the naive gradient update of  $\lambda_{i}$  with a proportional and derivative term to dampen oscillations and prevent cost overshooting. To reduce the amount of interaction required to solve the inner optimization problem, we warm-start our policy in each iteration by behavior cloning against the given expert demonstrations. We used a single NVIDIA 3090 GPU for all experiments. Due to space constraints, we defer all other implementation details to Appendix B.
4.2 ICL Results
We begin with results for the single-task problem, before continuing on to the multi-task setup.
4.3 Single-Task Continuous Control Results
As argued above, we expect a proper ICL implementation to learn policies that perform as well and are as safe as the expert. However, by restricting the class of constraints we consider, we can also investigate whether recovery of the ground-truth constraint is possible. To this end, we consider a reduced-state version of our algorithm where the learned constraint takes a subset of the agent state as input. For the velocity constraint, the learned constraint is a linear function of the velocity, while for the position and maze constraints, the learned constraint is a linear function of the ant's position.
Using this constraint representation allows us to visualize the learned constraint over the course of training, as shown in Figure 3. We find that our ICL implementation is able to recover the constraint,
Figure 3: Over the course of training, the learned ICL constraint recovers the ground-truth constraints for the velocity and position tasks. The learned policy matches expert performance and constraint violation. Standard errors are computed across 3 seeds.
Figure 4: As ICL training progresses, the learned position constraint (red line) converges to the ground-truth constraint (blue line) and the policy learns to escape unsafe regions (red region).
as the learned constraint for both the velocity and position tasks converges to the ground-truth value. Our results further show that over the course of ICL training, the learned policies match and exceed expert performance as their violations of the ground-truth constraint converge towards the expert's. Figure 4 provides a direct depiction of the evolution of the learned constraint and policy. The convergence of the red and blue lines shows that the learned position constraint approaches the ground truth, and the policy's behavior approaches that of the expert in response to this.
Figure 5: We re-conduct our position velocity constraint experiments using suboptimal experts to generate demonstrations. While, because of the ill-posedness of the problem, we do not exactly recover the ground truth constraint, we are able to use it to learn a policy that is higher performance than the expert while being just as safe. Standard errors are computed across 3 seeds.
To measure the robustness of our method to sub-optimal experts, we repeat the above experiments using demonstrations from an expert with i.i.d. Gaussian noise added to their actions at each timestep. We are still able to learn a safe and performant policy, which matches what our theory predicts.
4.4 Multi-Task Continuous Control Results
We next consider an environment where, even with an appropriate constraint class, recovering the ground-truth constraint with a single task isn't feasible due to the ill-posedness of the inverse constraint learning problem. Specifically, we use the umaze AntMaze environment from D4RL [Fu et al., 2020], modified to have a more complex maze structure. As seen in Figure 7, the goal of each task is to navigate through the maze from one of the starting positions (top/bottom left) to one of the grid cells in the rightmost column. We provide expert data for all 10 tasks to the learner.
As we can see in Figure 6, multi-task ICL is, within a single iteration, able to learn policies that match expert performance and constraint violation across all tasks, all without ever interacting with the ground-truth maze. Over time, we are able to approximately recover the entire maze structure.
We visually compare several alternative strategies for using the multi-task demonstration data in the bottom row of Figure 7. The  $0/1$  values in the cells correspond to querying the deep constraint network from the last iteration of ICL on points from each of the grid cells and thresholding at some confidence. We see that a single-task network  $(d)$  learns spurious walls that would prevent the learner from completing more than half of the tasks. Furthermore, learning 10 separate classifiers and then aggregating their outputs  $(e) / (f)$  also fails to produce reasonable outputs. However, when we use data from all 10 tasks to train our multi-task constraint network  $(g) / (h)$ , we are able to approximately recover the walls of the maze. These results echo our preceding theoretical argument about the importance of multi-task data for learning constraints that generalize to future tasks.
Figure 6: We see that over ICL iterations, we are able to recover the ground-truth walls of the anti-maze, enabling the learner to match expert performance and constraint violations. Results for the second two plots are averaged across all 10 tasks. Standard errors are computed across 3 seeds.
(g)
Figure 7: We consider the problem of trying to learn the walls of a custom maze (a) based on the AntMaze environment from D4RL [Fu et al., 2020]. We consider both a single-task (b) and multi-task (c) setup. We see that the single-task data is insufficient to learn an accurate constraint (d). Averaging or taking the max over the constraints learned from the data for each of the ten goals (e)-(f) also doesn't work. However, if we use the data from all 10 tasks to learn the constraint (g)-(h), we are able to approximately recover the ground-truth constraint with enough constraint learning iterations.
We release the code we used for all of our experiments at https://github.com/konwook/mticl.
In this work, we derive an algorithm for learning safety constraints from multi-task demonstrations. We show that by replacing the inner loop of inverse reinforcement learning with a constrained policy optimization subroutine, we can learn constraints that guarantee learner safety on a single task. We then give statistical and geometric conditions under which we can guarantee safety on unseen tasks by planning under a learned constraint. We validate our approach on several control tasks.
Limitations. In the future, we would be interested in applying our approach to real-world problems (e.g. offroad driving). Algorithmically, the CRL inner loop can be more computationally expensive than an RL loop - we would be interested in speeding up CRL using expert demonstrations, perhaps by adopting the approach of Swamy et al. [2023]. We also ignore all finite-sample issues, which could potentially be addressed via data-augmentation approaches like that of Swamy et al. [2022].
We thank Drew Bagnell for edifying conversations on the relationship between ICL and IRL and Nan Jiang for connecting us to various references in the literature. We also thank an anonymous reviewer for pointing out that our method does not actually require the expert to be the optimal safe policy, a fact we did not fully appreciate beforehand. ZSW is supported in part by the NSF FAI Award #1939606, a Google Faculty Research Award, a J.P. Morgan Faculty Award, a Facebook Research Award, an Okawa Foundation Research Grant, and a Mozilla Research Grant. KK and GS are supported by a GPU award from NVIDIA.
Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1-6, 2022. URL http://jmlr.org/papers/v23/21-1127.html.
Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn. Learning a prior over intent via meta-inverse reinforcement learning. In International conference on machine learning, pages 6952-6962. PMLR, 2019.
Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo Li, and Ding Zhao. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint arXiv:2209.08025, 2022.
Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152, 2020.
Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilistic context variables. Advances in neural information processing systems, 32, 2019.
Brian Ziebart, Anind Dey, and J Andrew Bagnell. Probabilistic pointing target prediction via inverse optimal control. In Proceedings of the 2012 ACM international conference on Intelligent User Interfaces, pages 1-10, 2012.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaii, volume 8, pages 1433-1438. Chicago, IL, USA, 2008a.
Brian D Ziebart, Andrew L Maas, Anind K Dey, and J Andrew Bagnell. Navigate like a cabbie: Probabilistic reasoning from observed context-aware behavior. In Proceedings of the 10th international conference on Ubiquitous computing, pages 322-331, 2008b.
Matt Zucker, Nathan Ratliff, Martin Stolle, Joel Chestnutt, J Andrew Bagnell, Christopher G Atkesson, and James Kuffner. Optimization and learning for rough terrain legged locomotion. The International Journal of Robotics Research, 30(2):175-191, 2011.
Proof. Let  $\Pi(c)$  denote the set of policies such that
$$
J (\pi , c) - J \left(\pi_ {E}, c\right) \leq 0. \tag {16}
$$
First, note that  $\forall c\in \mathcal{F}_c,\pi_E\in \Pi (c)$ . CRL is therefore trying to maximize reward over a set of policies that contains  $\pi_{E}$ . Thus, the reward condition is trivially true. We therefore focus on the safety condition. First, we note that by the definition of regret,
$$
\frac {1}{N} \sum_ {i} ^ {N} \left[ \left(J \left(\pi_ {i}, \hat {c}\right) - J \left(\pi_ {E}, \hat {c}\right)\right) - \left(J \left(\pi_ {i}, c _ {i}\right) - J \left(\pi_ {E}, c _ {i}\right)\right) \right] = \frac {T}{N} \sum_ {i} ^ {N} \ell_ {i} (\hat {c}) - \ell_ {i} (c _ {i}) \leq \bar {\epsilon} T. \tag {17}
$$
$$
\frac {1}{N} \sum_ {i} ^ {N} \left[ \left(J \left(\pi_ {i}, c ^ {*}\right) - J \left(\pi_ {E}, c ^ {*}\right)\right) - \left(J \left(\pi_ {i}, c _ {i}\right) - J \left(\pi_ {E}, c _ {i}\right)\right) \right] \leq \bar {\epsilon} T, \tag {18}
$$
as  $\hat{c}$  is the best-in-hindsight constraint. We then note that  $J(\pi_i, c_i) - J(\pi_E, c_i) \leq 0$  by the fact that  $\pi_i$  is produced via a CRL procedure, which means we can drop the former term from the above sum, giving us
$$
\frac {1}{N} \sum_ {i} ^ {N} \left(J \left(\pi_ {E}, c ^ {*}\right) - J \left(\pi_ {i}, c ^ {*}\right)\right) \leq \bar {\epsilon} T. \tag {19}
$$
Because this equation holds on average, there must be at least one  $\pi \in \pi_{1:N}$  for which it holds. Now, we recall that  $\pi_i = \mathrm{CRL}(r,c_i)$  to complete the proof.
A.2 Proof of Lemma 3.3
Proof. For a single  $c$ , a standard Hoeffding bound tells us that
$$
P \left(\left| \frac {1}{K} \sum_ {i = 0} ^ {K} V _ {k} (c) - \mathbb {E} [ V (c) ] \right| \geq \epsilon\right) \leq 2 \exp \left(\frac {- 2 K \epsilon^ {2}}{(2 T) ^ {2}}\right), \tag {20}
$$
where  $V_{k}(c)$  denotes the value of the payoff using data from the  $k$ th task. We have  $|\mathcal{F}_c|$  constraints and want to be within  $\epsilon$  of the population mean uniformly. We can apply a union bound to tell us that w.p. at least
$$
1 - 2 \left| \mathcal {F} _ {c} \right| \exp \left(\frac {- 2 K \epsilon^ {2}}{(2 T) ^ {2}}\right), \tag {21}
$$
we will do so. If we want to satisfy this condition with probability at least  $1 - \delta$ , simple algebra tells us that we must draw
$$
K \geq O \left(\log \left(\frac {\left| \mathcal {F} _ {c} \right|}{\delta}\right) \frac {(2 T) ^ {2}}{\epsilon^ {2}}\right) \tag {22}
$$
A.3 Proof of Theorem 3.5
Proof. For each  $c \in \mathcal{F}_c$ , define the set of safe policies as  $\Pi(c) = \{\pi \in \Pi | J(\pi, c) \leq 0\}$ . This set is non-empty by assumption. Define
$$
u (\tau , c) = \mathbb {1} \left\{\pi_ {E} ^ {\tau} \in \Pi (c) \right\}. \tag {23}
$$
We prove each of the desired conditions independently.
Reward Condition. Let  $\hat{c} \in c_{1:N}$ . Recall that we want to prove that
$$
\mathbb {E} _ {\tau \sim P (\tau)} \left[ J \left(\pi \left(r ^ {\tau}\right), r ^ {\tau}\right) - J \left(\pi_ {E} ^ {\tau}, r ^ {\tau}\right) \right] \geq - 2 \epsilon T, \tag {24}
$$
where  $\pi^{\tau} = \mathrm{CRL}(r^{\tau},\hat{c},\delta = 0)$ . Observe that
$$
\pi_ {E} ^ {\tau} \in \Pi (\hat {c}) \Rightarrow J \left(\pi^ {\tau}, r ^ {\tau}\right) = \max  _ {\pi \in \Pi (\hat {c})} J \left(\pi , r ^ {\tau}\right) \geq J \left(\pi_ {E} ^ {\tau}, r ^ {\tau}\right). \tag {25}
$$
Thus, if
$$
\mathbb {E} _ {\tau} [ u (\tau , \hat {c}) ] \geq 1 - \epsilon , \tag {26}
$$
$$
\begin{array}{l} \mathbb {E} _ {\tau} \left[ J \left(\pi^ {\tau}, r ^ {\tau}\right) - J \left(\pi_ {E} ^ {\tau}, r ^ {\tau}\right) \right] (27) \\ \leq \mathbb {E} _ {\tau} \left[ (1 - u (\tau , \hat {c})) \left(J \left(\pi^ {\tau}, r ^ {\tau}\right) - J \left(\pi_ {E} ^ {\tau}, r ^ {\tau}\right)\right) \right] (28) \\ \leq \mathbb {E} _ {\tau} \left[ \left(1 - u (\tau , \hat {c})\right) \right] \left(\sup  _ {\tau} J \left(\pi^ {\tau}, r ^ {\tau}\right) - J \left(\pi_ {E} ^ {\tau}, r ^ {\tau}\right)\right) (29) \\ \leq \mathbb {E} _ {\tau} \left[ (1 - u (\tau , \hat {c})) \right] 2 T (30) \\ \leq - 2 \epsilon T (31) \\ \end{array}
$$
We now prove that with  $K$  large enough, we can guarantee Eq. 26 holds true w.h.p. Define
$$
\tilde {\mathcal {F}} _ {c} = \{c \in \mathcal {F} _ {c} | \forall k \in [ K ], \pi_ {E} ^ {K} \in \Pi (\hat {c}) \}. \tag {32}
$$
We now argue that if
$$
K \geq O \left(\log \frac {\left| \mathcal {F} _ {c} \right|}{\delta} \frac {1}{\epsilon^ {2}}\right), \tag {33}
$$
w.p.  $\geq 1 - \delta$ , Eq. 26 holds true  $\forall c \in \widetilde{\mathcal{F}}_c$ . This means that as long as we pick  $c_{1:N} \in \widetilde{\mathcal{F}}_c$ , our desired condition will be true. Note that this is fewer than the number of samples we assumed in the theorem statement.
For a single constraint, a Hoeffding bound tells us that
$$
P \left(\left| \frac {1}{K} \sum_ {k} ^ {K} \mathbb {1} \left\{\pi_ {E} ^ {k} \in \Pi (c) \right\} - \mathbb {E} _ {\tau} [ u (\tau , c) ] \right| \geq \epsilon\right) \leq 2 \exp \left(- 2 K \epsilon^ {2}\right). \tag {34}
$$
Union bounding across  $\mathcal{F}_c\supseteq \widetilde{\mathcal{F}}_c$  , we get that the probability that  $\exists c\in \widetilde{\mathcal{F}}_c$  s.t. Eq. 26 does not hold is upper bounded by
$$
1 - 2 \left| \mathcal {F} _ {c} \right| \exp \left(- 2 K \epsilon^ {2}\right). \tag {35}
$$
To have this quantity be  $\geq 1 - \delta$ , we need
$$
K \geq O \left(\log \frac {\left| \mathcal {F} _ {c} \right|}{\delta} \frac {1}{\epsilon^ {2}}\right). \tag {36}
$$
Safety Condition. We begin by considering the infinite sample setting. We therefore desire to prove that
$$
\mathbb {E} _ {\tau} \left[ J \left(\pi^ {\tau}, c ^ {*}\right) - J \left(\pi_ {E} ^ {\tau}, c ^ {*}\right) \right] \leq \bar {\epsilon} T. \tag {37}
$$
Define the per-round loss of the constraint player as
$$
\ell_ {i} (c) = \frac {1}{T} \mathbb {E} _ {\tau} \left[ J \left(\pi_ {i} ^ {\tau}, c\right) - J \left(\pi_ {E} ^ {\tau}, c\right) \right] \in [ - 1, 1 ], \tag {38}
$$
the best-in-hindsight comparator as  $\hat{c} = \operatorname{argmax}_{c\in \mathcal{F}_c}\sum_i^N\ell_i(c)$ , instantaneous regret as  $\epsilon_{i} = \ell_{i}(\hat{c}) - \ell_{i}(c_{i})$ , and average regret as  $\bar{\epsilon} = \frac{1}{N}\sum_{i}^{N}\epsilon_{i}$ . Proceeding similarly to the single-task case,
$$
\begin{array}{l} \bar {\epsilon} T = \frac {1}{N} \sum_ {i} ^ {N} \mathbb {E} _ {\tau} \left[ J \left(\pi_ {i} ^ {\tau}, \hat {c}\right) - J \left(\pi_ {E} ^ {\tau}, \hat {c}\right) \right] - \mathbb {E} _ {\tau} \left[ J \left(\pi_ {i} ^ {\tau}, c _ {i}\right) - J \left(\pi_ {E} ^ {\tau}, c _ {i}\right) \right] (39) \\ \geq \frac {1}{N} \sum_ {i} ^ {N} \mathbb {E} _ {\tau} \left[ J \left(\pi_ {i} ^ {\tau}, c ^ {*}\right) - J \left(\pi_ {E} ^ {\tau}, c ^ {*}\right) \right] - \mathbb {E} _ {\tau} \left[ J \left(\pi_ {i} ^ {\tau}, c _ {i}\right) - J \left(\pi_ {E} ^ {\tau}, c _ {i}\right) \right] (40) \\ \end{array}
$$
We now argue that the second term in the above sum must be non-positive. Consider an arbitrary task  $\tau$ . Then, because  $\pi_E^\tau \in \Pi(c_i)$  and CRL is optimizing over  $\Pi(c_i)$ , this term must be negative. As it is
negative per-task, it must be negative in expectation. Thus, we are free to drop the second term in the above expression which tells us that
$$
\bar {\epsilon} T \geq \frac {1}{N} \sum_ {i} ^ {N} \mathbb {E} _ {\tau} \left[ J \left(\pi_ {i} ^ {\tau}, c ^ {*}\right) - J \left(\pi_ {E} ^ {\tau}, c ^ {*}\right) \right] \tag {41}
$$
Because this equation holds on average, there must be at least one  $\pi \in \pi_{1:N}$  for which it holds. Now, we recall that  $\pi_i^\tau = \mathrm{CRL}(r^\tau, c_i)$  to complete the infinite-sample proof.
We now consider the error induced by only observing a finite set of tasks. There are two places finite-sample error enters: in estimating the value of  $\ell_i(c)$  and in estimating  $\widetilde{\mathcal{F}}_c$ .
By Lemma 3.3, the maximum error we can induce by estimating  $\ell_{i}$  from finite samples is upper bounded w.h.p by  $\epsilon$ . Thus, the extra error induced on the average regret is also bounded by  $\epsilon$ . Observe that our losses are scaled by  $\frac{1}{T}$  in comparison to difference of  $Js$ . Therefore, we need to add an  $\epsilon T$  to our bound for the infinite-sample setting.
By our argument in the reward section,  $\pi_E^\tau \notin \Pi(c_i)$  w.p.  $\leq \epsilon$ . When this is true,  $J(\pi_i^\tau, c_i) - J(\pi_E^\tau, c_i)$  can be as big as  $2T$ . Thus, the term we dropped in Eq. 40 ( $V(c_i)$ ) can be as big as  $2\epsilon T$  instead of 0. In the worst case, this adds an additional  $2\epsilon T$  to our bound.
Combining both of the above, when we transition from the infinite sample setting to the finite sample setting, our bound degrades by  $3\epsilon T$
An overview of the key experimental details is presented below. The exact code for reproducing our experiments is available at https://github.com/konwook/mticl.
B.1 CRL