Conformalized Interactive Imitation Learning: Handling Expert Shift and Intermittent Feedback
Authors: Michelle D Zhao
Henny Admoni
Reid Simmons
Aaditya Ramdas
Andrea Bajcsy
Venue: ICLR 2025 Poster (2025)
Keywords: conformal prediction
interactive imitation learning
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We contribute an interactive imitation learning approach wherein the robot uses prediction intervals calibrated by online conformal prediction as a reliable measure of deployment-time uncertainty to actively query for more expert feedback.

Abstract: In interactive imitation learning (IL), uncertainty quantification offers a way for the learner (i.e. robot) to contend with distribution shifts encountered during deployment by actively seeking additional feedback from an expert (i.e. human) online. Prior works use mechanisms like ensemble disagreement or Monte Carlo dropout to quantify when black-box IL policies are uncertain; however, these approaches can lead to overconfident estimates when faced with deployment-time distribution shifts. Instead, we contend that we need uncertainty quantification algorithms that can leverage the expert human feedback received during deployment time to adapt the robot's uncertainty online. To tackle this, we draw upon online conformal prediction, a distribution-free method for constructing prediction intervals online given a stream of ground-truth labels. Human labels, however, are intermittent in the interactive IL setting. Thus, from the conformal prediction side, we introduce a novel uncertainty quantification algorithm called intermittent quantile tracking (IQT) that leverages a probabilistic model of intermittent labels, maintains asymptotic coverage guarantees, and empirically achieves desired coverage levels. From the interactive IL side, we develop ConformalDAgger, a new approach wherein the robot uses prediction intervals calibrated by IQT as a reliable measure of deployment-time uncertainty to actively query for more expert feedback. We compare ConformalDAgger to prior uncertainty-aware DAgger methods in scenarios where the distribution shift is (and isn't) present because of changes in the expert's policy. We find that in simulated and hardware deployments on a 7DOF robotic manipulator, ConformalDAgger detects high uncertainty when the expert shifts and increases the number of interventions compared to baselines, allowing the robot to more quickly learn the new behavior.

Published as a conference paper at ICLR 2025
CONFORMALIZED INTERACTIVE IMITATION LEARNING: HANDLING EXPERT SHIFT & INTERMITTENT FEEDBACK
Michelle Zhao Reid Simmons", Henny Admoni", Aaditya Ramdas" Andrea Bajesy** " Robotics Institute, School of Computer Science, Carnegie Mellon University
b Departments of Statistics and Machine Learning, Carnegie Mellon University (mzhao2, hadmoni, rsimmons, aramdas, abajcsy) (andrew. cmu. edu
ABSTRACT
interactive imitation learning (IL), uncertainty quantification offers a way for the learner (i.e. robot) to contend with distribution shifts encountered during de- ployment by actively seeking additional feedback from an expert (i.e. human) online. Prior works use mechanisms like ensemble disagreement Monte Carlo dropout to quantify when black-box II. policies are uncertain; however, these approaches can lead to overconfident estimates when faced with deployment-time distribution shifts. Instead, we contend that we need uncertainty quantification algorithms that can leverage the expert human feedback received during deploy- ment time to adapt the robot's 's uncertainty online. To tackle this, we draw upon online conformal prediction, a distribution-fre method for constructing prediction intervals online given stream of ground-truth labels. Human labels, however, are intermittent in the interactive II. setting. Thus, from the conformal prediction side, introduce novel uncertainty quantification algorithm called intermittent quantile tracking (IQT) that leverages a probabilistic model of intermittent labels, maintains asymptotic coverage guarantees, and empirically achieves desired cov- erage levels. From the interactive IL side, we develop ConformalDAgged a new approach wherein the robot uses prediction intervals calibrated by IQT as reliable measure of deployment-time uncertainty to actively query for more expert feedback. We compare ConformalDAgger prior uncertainty-awart DAgger methods in scenarios where the distribution shift is (and isn't 't) present because of changes in the expert's policy. We find that in simulated and hardware deployments on a 7DOF robotic manipulator, ConformalDAgee the expert shifts and increases the number of interventions compared to base- lines, allowing the robot to more quickly learn the new behavior. Project page at c.u intenttaab.. github. wt/ ctt//oofooraaaa zed interactive ii/
detects high uncertainty when
INTRODUCTION
End-to-end robot policies trained via imitation learning (IL) have proven to be an extremely powerful way to learn complex robot behaviors from expert human demonstrations (Schaal) 1996 Price &
Boutilier 2003 Argall et al. 2009. Levine et al. 2016 J Jang et al. 2022 Chi et al. 2023) Kim et al.
2024). At the same time, distribution shift is core challenge in this domain, hampering the reliability of deploying such robot policies in the real world (Chang et al. 202I).
One way to combat this is via uncertainty quantification. By training an ensemble of policies (Mendal et al., 2019) or via monte-carlo dropout during training (Cui et al. 2019), the robot learner can detect uncertain states and actively elicit additional action labels from the human expert online via an interactive II. framework (such DAgger (Ross et al. (01T)). At their core, these prior uncertainty- aware II. methods look to the training demonstration data as proxy for deployment-ting uncertainty. Any human expert labels requested at deployment time using this uncertainty estimate are simply stored for later re- re-training; the uncertainty estimate itself is not adapted online to the expert data nor does it inform any subsequent queries during the deployment episode.
Instead, we contend that the human feedback requested and received during deployment time is a valuable uncertainty quantification signal that should be leveraged to update the robot's 's uncertainty estimate online. If properly accounted for, the updated uncertainty estimate will influence when the robot asks for more help, enabling it targetedly probe the human expert improve policy

---

Published as a conference paper at ICLR 2025
human feedback
nusemboo's planned action
ConformalDAgee uncertainty
Stationary Expert
Expert Shift
El leedback
feedbback
R aoks help
Rasks for help
Racting independently
acting independendly
0.1
AogfermalDAgeer
ConformalDAgge uncurtaiity 0.0
uncortaiidy 0.0
Time tnijectry
Time in trajeciory
Figure 1: Conformalized Interactive Imitation Learning. The robot learns an initial policy via imitation learning to wipe line drawing by following straight path. When the initial policy is deployed, AongormalDAgger calibrates the robot's uncertainty (represented by red prediction interval boxes) based on feedback received from the human during the interactive IL. loop. (left) The robot refrains from asking questions when its uncertainty is low because the expert policy at deployment time is aligned to the demonstrations on which the learner was trained. (right) When the human shifts in their task strategy, uncertainty increases and the robot starts to query for more expert labels.
performance. The challenge is how to do this online uncertainty update in the presence of the end-to-end "black box" policies underlying modern imitation learning.
To tackle this, we take inspiration from online conformal prediction (Gibbs & Candes) 202 1) which a distribution-fre way to represent uncertainty via prediction sets constructed on the output of black-box model. Our first contribution is extending online conformal prediction to the case where labels are observed intermittently, as is the case interactive II. with an expert. Specifically, we instantiate Intermittent Quantile Tracking (IQT), an algorithm which adjusts prediction intervals online to ensure that the true label lies within the predicted interval with high-probability despite probabilistic access to labels. On standard conformal time series datasets, empirically find that IQT achieves empirical coverage close to the desired level by boosting the size of the calibrated intervals based on the likelihood of observing feedback.
With our intermittent conformal algorithm in hand, we develop AonformalDAgger, a new interactive II. approach wherein the robot uses prediction intervals calibrated by IQT reliable measure of deployment-time uncertainty to actively query for more expert feedback. We instantiate Con- formalDAgger in a simulated 4D robot goal-reaching task and in hardware on 7 fregree-of-freedon robotic manipulator that uses a state-of-the-arr Diffusion Policy (Chi et al. 2023) learned via IL to perform a sponging task (Figure M. We study how ConformalDAgger compares prior uncertainty- aware DAgger methods when the deployment-time and training-time datasets are from the same distribution (left, Figure well as shifted one (right, Figure D Specifically, we instantiate a po- tential source of distribution shift as experi policy shift: the training-time expert wipes line-drawing with straight-line path while the deployment-time expert refines their strategy to be a zigzag path (Figure [T. We find that ConformalDAgger automatically increases uncertainty online when the expert shifts, resulting in more expert labels queries compared to EnsembleDAgger and allowing our approach rapidly learn policy aligned with the expert's intentions.
2 RELATED WORK
Interactive Imitation Learning (IL) with Online Experts. Interactive IL is a branch of imitation learning wherein a robot learner can query a (human) expert to receive additional labels either during or after task execution (Celemin et al. 2022). The utility of in- in-the-loop expert interventions is mitigating distribution shift between training and deployment of the policy. Distribution shift can stem from multitude of reasons, including covariate shift (Spencer et al. 2021) (e.g.. encountering new observations of the environment during policy execution), or expert shift: wherein the expert's latent strategy changes over time (Hong et al. 2024 Sagheb et al. 2023 Xie et al. (02T)) A foundational approach for interactive IL is DAgger (Dataset Aggregation) (Ross et al.] 2011). which iteratively augments the training dataset by aggregating data from the expert and learner policies and assuming the expert is stationary. In the online case, the robot learner can cede control to the expert at any time to get additional state-action data (also known as robot- gated feedback) the expert can actively
2

---

Published as a conference paper at ICLR 2025
intervene at any time (also known as uuman-gated feedback) Kelly et al. 2019). From the learner's perspective, a key question is when to request feedback from the expert so that it minimizes expert effort but also minimizes negative events caused by an erroncous policy (e.g.. running into a wall). On one hand, prior works focus on minimizing human effort by constraining robot requests via a limited human attention model (Hoque al.
2023) or a budget of human interventions (Hoque
et al. 202la bb. On the other hand, prior works prioritize deployment-tine safety by classifying safe versus unsafe states (SafeDAgger (Zhng Cho, 2017). Replay Estimation (Swamy al. 2022)), estimating uncertainty via ensemble disagreement (EnsembleDAggee (Menda et al. 2019)), or Monte Carlo dropout (Cui et al. 2019). We present ConformalDAgges which uses our novel uncertainty quantification method grounded in conformal prediction to adaptively increase the learner requests for help under uncertainty and decrease the number of requests when confident.
Online Conformal Prediction. Conformal prediction is distribution-free uncertainty quantification method for constructing prediction intervals for both classification and regression problems (An gelopoulos & Bates 2023 Romano al. 2019 2020 Zaftran et al., 2022), , as well as for offiine or online data. We focus on the online setting (e.g.. timeseries) where uncertainty quantification is performed on streaming pairs of input-label data that are not necessarily i.i.d. (Gibbs & Candes) 2021I. Broadly speaking, there are two predominant algorithms in this setting: adaptive conformal inference (ACT) (Gibbs & Candes 202T Gibbs & Candès 2024 Bhatnagar al., 2023 Zaffran et 2022) and quantile tracking (QT) (Angelopoulos et al. (202). Both are online gradient descent-based methods which guarantee asymptotic coverage in the online setting. Unlike ACI which is prone to infinitely sized intervals after a series of miscoverage events, QT directly estimates the value of the empirical quantile itself, ensuring coverage with finite intervals. In this work, we relax the assumption that labels must be observed at each time point in the streaming data and extend the online conformal paradigm to ensure coverage in the intermittent label regime.
ONLINE CONFORMAL PREDICTION WITH INTERMITTENT LABELS
From the uncertainty quantification side, our core technical contribution is extending online conformal prediction to settings where ground truth labels are intermittently observed. We present our algorithm in the context of quantile tracking for relevance to our interactive II. experiments. However, we also derive an extension of ACI (Gibs & Candes 202T) to intermittent labels in the Appendix Section D
Setting. We focus on online conformal prediction in the adversarial setting, such as time-series forecasting. This considers an arbitrary sequence of data points (#t, 31) E X' x )) for t = 1,2, ..., that are not necessarily I.LD. Our goal is to produce prediction sets on the output of any base prediction model such that the sets contain the true label with a specified coverage probability. Mathemati- cally, at each time f, we observe z; and seek to cover the true label y, with a set (xx)), which depends on a base prediction model, f X + V. The base model takes as input the current oe and outputs prediction 9e; in the non-intermittent case, we observe the ground-truth label yt after each prediction.
obs
obs,
CTr
9.
Figure 2: Graphical Model of IQT. IQT introduces random variable obs,, which represents the receiving of ground truth observations at f. obs; is distributed ac- cording to Pt. which may depend on æt history through t - 1.
Background: Quantile Tracking. The quantile tracking (QT) algorithm from Angelopoulos et al. 2024) implicitly seeks to track the value of the 1 quantile via online gradient descent on the quantile loss (Koenker& & Bassett Jr. 1978). As in Angelopoulos et al. (2024). we leverage a bounded nonconformity score function: $ ) - [0, BJ where B < co, to quantify the error made by the initial prediction of the base model. We assume the nonconformity score function s(2,, ye) is negatively oriented (lower values indicate less nonconformity or greater prediction accuracy). Let 4: represents the estimated 1 quantile of the score sequence st, f E N. Prediction intervals are constructed using the nonconformity score function:
(x.) = {y E y s v1.) S $
To expand or contract the prediction intervals, the level is adjusted via the online update:
gt+1 qi + Yr (err, - -a),
(2)
where 7t > 0 is time-varying step size and err; (C.x)) is the empirical miscoverage
3

---

Published as a conference paper at ICLR 2025
Intuitively, this update increases the quantile threshold when the model continuously miscovers and decreases it when the model coverage is performant. For arbitrary step size " with no assumptions on the sequence of data points 1 31). (22, (22, y2),... and an initial quantile threshold q1 [0, BJ, the quantile tracking update satisfies Equation (Theorem of Angelopoulos et al. (024)):
1
B + maxisIsT Tt
Σ 1 EC.(())
(1 at)
-
||A1:111
T
T
where A is defined A 71 and A, 7o - for t 2 2, and A (Ai...., AT). We can
T see that limy- T 1 1 - err, approaches - a. This guarantees quantile tracking gives the 1 a long-term empirical coverage frequency.
Intermittent Quantile Tracking (IQT). Our paradigm lifts the assumption that the ground truth label yr is observed constantly. Instead, it is observed with some probability each timestep. Let the binary random variable obs, E {0, 1 represent whether the robot observes label y/ at timestep I:
f1, if Y/ observed obs; == {
otherwise
We introduce a probabilistic observation model. Pt. == Pobs, = =1 zt), where Pt represents the probability of receiving feedback, which can be dependent possibility is demonstrated by the dotted line in Figure D A key observation under the paradigm of intermittent labels is that enr, may not every timestep be accessible to the algorithm if y, is not provided, but the value of err, exists at every timestep even if unobservable.
input z; (and past history, this
Our quantile tracking update under probabilistic observations, which we call Intermittent Quantile Tracking (IQT), is defined in Equation 5 as:
+ Lt (err, - a.jobs;
qu+1 = q
(5)
Pt
Proposition 1. Let (I1, 31), (z2, (22, v2..... be an arbitrary sequence of data points, and let s X y [0, B]. Let "It be an arbitrary positive sequence, and fix an initial threshold qı E B). Then Intermittent Quantile Tracking (IQT) satisfies for all T 2 1:
T
B maxIsIST St. Pt ||Arr1
em a
(6)
T
T
This guarantees IQT gives the 1 a long-term empirical coverage frequency, even with intermittent ground truth access, regardless of the underlying data generation process. While these asymptotic coverage guarantees are sound in theory, we believe it is important to acknowledge that typical robot deployment conditions are finite-horizon. Nevertheless our empirical findings in Section5 the practical utility of IQT's s uncertainty estimates to obtain extra human feedback during deployment. Refer full proof in Appendix Section Note that when feedback is constant (i.e. Pt- = 1), obs; = for all timesteps, and IQT reduces to online quantile tracking with arbitrary step sizes.
Practical Note: Choosing Gamma In practice, an important decision is the choice of % Prior works in quantile tracking with constant labels (Angelopoulos et al. 2023) choose "t- = Irß, where B, := max, - s(,,, yr)), & represents a look-back window across the previous timesteps with observed ground truth labels, and Ir is small constant. On our domains, find that choosing % such that it contains B, improves empirical coverage over the choice of a constant 7t = 7.
3.1 EXPERIMENTS: INTERMITTENT QUANTILE TRACKING ON STANDARD TIMESERIES DATA
We first empirically evaluate our approach for intermittent conformal quantile tracking on time series benchmarks used in the online conformal prediction literature. The goal of testing on standard conformal datasets is to evaluate (1) how different choices of step size 7/t affect the empirical coverage of IQT, and (2) validate that under probabilistically intermittent observations, IQT maintains coverage close to the desired level. We use the findings from these experiments inform how we leverage intermittent quantile tracking in the interactive imitation learning domain.
Setup. We test on three benchmark datasets from Angelopoulos et al. 2023): (1) Amazon stock
prices, (2) Google stock prices (Nguyen 2018), and the (3) Flec2 dataset (Haries 1999). We test.
4

---

Published as a conference paper at ICLR 2025
+=
0.1
Ir 0.01
Cansag
Comnag
1.0
M M1 MMM
IQT-pi
500
0.0
1.0
4//
1000
IQT-pd
500
0.0
1000
2000
3000
1000
2000
3000
1000
2010
3000
1000
2000
3000
1000
2000
3000
1000
2000
3000
Figure 3: IQT Amazon Stocks with AR base model: Coverage & Interval Visualization. We set Pa = 0.1, vt to see true price labels only 10% of the time and show the prediction interval for 1 seed and coverage averaged over seeds (shaded error on coverage plots show std deviation). IQT-pd is prone to large intervals when Ir is large, and boosts coverage over IQT-pi when Ir is small.
four base prediction models, f. all trained via darts (Herzen et al. 2022) to see if IQT consistently maintains coverage close the desired level. We present the Autoregressive (AR) model with 3 lags for brevity in the main text and defer the other model results to the Appendix. Our nonconfornity score is the asymmetric (signed) residual score. We measure (1) marginal coverage over the time series, (2) longest miscoverage error sequence, and (3) mean prediction interval size.
Observation Models. We test three different observation frequencies: (1) infrequent pt = 0.1, Vt. (2) partial pt = 0.5, Vt, and (3) frequent p: 0.9, Yt. We focus in this section the results for infrequent observations, and discuss further results in the Appendix Section
Methods. We compare two variants of our IQT algorithm by controlling if time-varying step-size Y in Equation 5 depends or does not depend on pt- For each variant, we test Ir E [1, (1,0.1,0.01] 0.01]. IQT with a pr-independent update (IQT-pi) uses 7 Ir Brpept. The quantile tracking update reduces to g++1 qr + Be - jobs,, Since the p: term cancels out in the update, refer to this model as the p-independent update. IQT with pe-dependent update (IQT-pd) uses 7/- Ir., Since the pe term remains in the update Q++1 = qe + IrL (err, - aoobs,,, we refer to this model as the p-dependent update. For both variants, we set our desired coverage to
0.9 for all experiments. We set lookback window h 100 timesteps for the Amazon stock price data.
Results. We center our discussion on the Amazon stock price results for IQT-pd and IQT-pi with AR base model under ground truth labels observed with Pt 0.1 frequency, as the intermittent setting is our focus. See Appendix Section E for further results on other base models and datasets. Inversely scaling the quantile tracking step size by Pt. causes IQT-pd pd to construct larger intervals than IQT-pi. Figure 3 shows the prediction coverage as a moving average (window=50) for 5 random seeds and interval sizes for one seed. When Ir is high (1.0), IQT-pi and IQT-pd achieve comparable coverage, but IQT-pd is prone constructing much larger prediction intervals than IQT-pi. Smaller and midsized learning rates (Ir = 0.01, 0.1) regulate the size of IQT-pd intervals, leading to tight prediction intervals which maintain desired coverage levels.
4 CONFORMALDAGGER A CALIBRATED APPROACH TO ASKING FOR EXPERT FEEDBACK
Since intermittent quantile tracking enables us to rigorously quantify the learner's uncertainty despite the fact that the labels are only revealed intermittently (i.e., when the expert intervenes), we can develop ConformalDAgger: new way for the robot to tradeoff between acting autonomously and strategically asking for help when uncertainty increases.
Setup. ConformalDAgger treats the robot's policy as the base model f n on which we perform intermittent quantile tracking. The initial novice policy के X' + A is trained on initial demonstration data Do of task T performed by the expert क Here, X' represents the policy's 's inputs (e.g., image observations, proprioception), y = A are the labels representing the robot's actions (e.g.. future end-effector positions). During deployment, the robot policy generates a sequence of input- predicted action pairs (æ,, a)) X x ) for t 1,2, .... that temporally correlated. Relatedly, for each input x, the learner observes, there is corresponding expert action (æ,, a. ) E X x ) that is the ground-truth label we seek to cover via IQT prediction intervals (xz)).
5

---

Published as a conference paper at ICLR 2025
Offline
Deployment Episode
State-
Intermittent Quantile Tracking
Human
Buffer
π
aj
axcersiey
Interval
Plabs
π
Update AgT
Robot policy
Chosce Lusnon Plotec
ConformalIAAgeer
Figure 4: ConformalDAgger Framework. After obtaining an initial learner policy x" (left), ConformalDAgger calibrates uncertainty during the interactive deployment episode with the expert via intermittent quantile tracking (center). When the size of the uncertainty intervals is high, the robot actively queries the user for feedback. When uncertainty is low, the rohot executes its predicted action and the human may independently intervene with some low probability. After deployment episode ends, the data aggregated and the learner retrained (arrow from right to left).
Algorithm 1 ConformalDAgger (changes from DAgger (Ross et al. 2011) highlighted)
1: Collect initial demonstration data Do from expert Tú and train initial learner policy mp. for interactive deployment episode = 0 .M do
2:
3: Initialize d6 4
4: for deployment timestep / = 1: H do
5:
Get predicted action label: of <= (24)
6:
Construct calibrated uncertainty interval: C,(x,)= = [a] o a. gf
7:
Compute robot query likelihood (e.g. P(obs; | (;; = u(x;; > T, else 0.)
8:
if robot queries (w.p. P(obs; (OGT)) or human intervenes (w.p. P(obs o.)) then
9:
to ja,hi + ps IQT update: 9(+1 {- 4t else
(err, - a:oos, w/ expert action of < a (2.).
10:
IQT update: le+1 + qf Jo.hi
11:
Aggregated dataset with observed state and expert action pairs: Det1 + Dj U (1x, o (x))
13:
Retrain learner: T/+1 + arg min, ((D,+1)
Observation Model. A key component of IQT is the observation likelihood model, pt = P(obs; = 1 xz). A nice byproduct of this model is that can naturally derive feedback model that is simultaneously human- and aobor-gated by decomposing the observation model into the combination of both gating functions. Intuitively, the likelihood of observing the expert feedback t is given by the probability that the human chooses to give feedback or the robot asks the expert for feedhack. Let obs, E {0, 1} be a random variable representing observing a numan-gated feedback (if obs, = 1) where the expert initiates providing an action label at n/ (Ix)) for Ht. Let obs,, (0, be random variable representing robot-gated feedback, where the robot asks the expert for a label if obs, 1. We assume that if the robot elects ask a question, the human will respond with probability Our observation model takes the form: P: = Ioobss;
xx) F(obs 1 V obs, = x.))
P(obs) 1 2x) + Pobbs," 1 xx)) P(obs, not prescriptive about these models, in our experiments we model (obss x;)) = c as a small constant to represent infrequent human interventions. We describe the robot-gated likelihood model P(obss = xx) below, informed by our uncertainty estimates.
2, P (obs,
x2)). While our approach is
Quantifying Uncertainty: IQT. In the interactive IL. setting, IQT begins with initial upper and lower quantiles in the action (i.e. "label") space, ob .ggi E A. The nonconformity score is residual on the predicted (a") versus expert (a") action. Let (a, a²)) a. a be the lower residual (referred to as s/" for brevity), and (a a²) at at be the upper residual (referred as s/i),. If the expert action at is observed, we compute the upper and lower miscoverage of d! g.tt as two indicator vectors: err!" (st0 < glo) and enr! (x²* < " IQT then updates the quantile estimates online to obtain qla 1 1 via the update rule from equation 5 the next timestep, the adjusted prediction
interval is constructed with C++1 (Fx-1) = [o[+1 gf ai- + g/i 1. If the expert action is not observed, ohs, O, then 4 de and hi, q: and the prediction interval size remains the same.

---

Published as a conference paper at ICLR 2025
Note that although err, is not known in the case where the expert does not provide action label, IQT does not require it; IQT simply makes no change to the quantile estimate. In our simulated and hardware experiments, IQT for continuous vectors, but the approach extends to discrete- action spaces.
action space, qb and u6" are vector-valued. Our experiments instantiate
Leveraging Uncertainty: Asking for Help. Finally, the calibrated intervals C++1 (x++1) constructed by IQT enable us to design new robot- robot-gated feedback mechanism. Specifically, the model P(obs 1z) is informed by the calibrated interval size, u(xe+1) == [|C
In simulation, we use P(obs 2x) = (ff((I12 r²) - r]) where acts as an uncertainty threshold on u(2;; (ex;;x)) 8 is temperature hyperparameter, and a is the sigmoid function. hardware experiments, we set T as a hard threshold on on u(x;;"")) above which P((bbs,
xx)) = 1, and below which Pobbs, xxx) = 0.
ConformalDAgger Algorithm. We summarize ConformalDAgger in Algorithm D highlighting the difference from traditional DAgger (Ross et al.] 2011). The learner interacts with the intermittent expert during M interactive episodes. At the start of each, g² ght and are reset. Each episode lasts H time steps and the internittently-obsrvve the prior training data in a fixed size training buffer to form the updated dataset, Di+1, which enables us to retrain the learner Tf+1 for the next deployment episode.
expert state-action pairs (xz, af) are aggregated with
5 SIMULATED INTERATIVE IMITATION LEARNING EXPERIMENTS
To evaluate ConformalDAgged we run a series of simulated experiments with access to an oracle expert. We ground our experiments in simulated robot goal-reaching task (left. Figure D and study scenarios where the distribution shift occurs due to the expert's 's changing preference in goal location.
5.1 EXPERIMENTAL SETUP
Task & Initial Learner Policy. The robot learns neural network policy n²: X + A to move cup from a start to the expert's s desired goal location. We model input x E X' C 12 as the xyz position of the robot well its binary gripper state (open or closed) across the previous 3 timesteps. The labels are actions, a E A C R represented as next xyz position and gripper state. The robot always starts at an initial x with the gripper state closed (holding a cup) and must keep holding the cup while moving it to the unknown goal location that the expert prefers; let [1 be two such goals. Before the first interactive deployment = 0), we simulate the expert as initially giving demonstrations placing the cup at 9o. where (x) = x + W Qo =z maxaallo =z]] where maxa(-) is maximum over dimensions. The step size is regulated to be at most w = 0.01. The initial robot policy र. is trained on a dataset Do of 10 expert trajectories with synthetically injected noise drawn from (11, 0.5) for robustness as in (Laskey et al. 2017). See Appendix |P for policy implementation details.
Simulated Expert Policies. To induce controlled distribution shift, we study three expert policies (left, column of Figure D. (1) Stationary: the expert has a fixed goal, go. across all deployment episodes. (2) Shift: the expert goal shifts from 9o to 91 at deployment episode i = 5. For example, the expert may have decided that different cup location is easier to reach. (3) Drift: expert's goal slowly drifts from 9u to 91 over the course of deployment episodes (in Figure the drift from gu - 9ie occurs at episode i 5, 91a + 915 occurs at episode = 8, and 91b + 91 occurs at episode i 11). For example, the expert may start with a conservative goal location initially (e.g., goal nearby) and incrementally move the cup closer and closer to their target goal that may be further out of reach. (4) Environment Shift: Under environment shift, the policy experiences covariate shift, where the deployment-tim state distribution differs from the training distribution. Because the policy is trained on inputs which capture the robot's 's position, we instantiated environment shift by changing the starting position of the robot in the first episode, maintaining this new position for all latter episodes.
Interactive Deployment Episodes & Learner Re-training. We consider M 15 deployment episodes before re- re-training the learner. Each deployment episode has two interactive task executions. The task ends when the cup has reached the correct goal position, g". or when the maximum timesteps (100) have been reached. The expert answers queries with optimal actions under their current policy a = n² (z). Following DAgger (Ross et al., 1), after each deployment episode, the state-action pairs where the expert provided action labels are aggregated into the training dataset, forming aggregated buffer D for the next deployment episode i + 1. We constrain the size of the replay buffer to 300 datapoints, dropping the old experiences.
Methods. We compare ConformalDAgger to EnsembleDA gger Menda et al., 2019). LazyDAgger (Hoque et al. 2021b), and SafeDA gger (Zhang & Cho 2017). We performed a hyperparameter
7

---

Published as a conference paper at ICLR 2025
No Shill,yy dg
Miscovenge (I)
of Epionde Intervened
Expert Decision Deviation (1) Export Trajectury Deviation (I)
10
10-
C3
Suiithige
Pxpert Ny
0.0
08
10--
05.
Expert Deitt
00-
70-
Environment
10-
10
115
0.-
Interactive Deployment Episode
Figure 5: Simulated Robot Results. When the expert shifts or drifts (middle rows), ConformalDAg ger increases the number of requests for expert feedback compared to EnsembleDAgger and gger but has less interventions than Lagrr This decreases miscoverage and expert deviation after re-training. With stationary expert (top) or environment shift (bottom), both Ensem- bleDAgger and ConformalDAgger perform similarly. Shading is std. error across 5 seeds.
sensitivity analysis for all methods (see B and selected thresholds such that all approaches ask infrequent questions in the first few interactive deployment episodes, where shift has occurred. ConformalDA gger uses an uncertainty threshold T = 0.06, temperature 3 = 100, lookback window
k = 100, Ir = 0.6, and initial 9o lo = 0.01. The uncertainty threshold is heuristically tuned to ask few, but infrequent questions in the first interactive deployment episode. EnsembleDA queries an expert online when there is high action prediction variance across an ensemble of learner policies, and when safety classifier detects dissimilarity between expert and robot actions. We use ensemble members and uncertainty threshold of 0.06 for the ensemble disagreement and safety classifier threshold of 0.03. LazyDAgger uses 8 = 0.03 to begin human intervention, and only switches back to autonomous mode when the deviation between the lcarner's prediction and expert's 's action are below a sontext-switching threshold, 0.1 * s. To make SafeDAgger's number of initial interventions comparable, we decrease the safety classifier threshold to 0.01.
Metrics. We measure the quality of uncertainty quantification via the miscoverage rate and human effort via the intervention percentage of the deployment trajectory. We compute the miscoverage for EnsembleDAgger using three times the standard deviation as the prediction interval. We measure the quality of the learned policy via two metrics. Decision deviation simulates the learner's 's behavior under its current policy and queries the expert at each learner state to obtain an expert action label. We measure the average L2 distance between the predicted and expert action. Trajectory deviation forward simulates both the expert and the learner acting independently under their policies, starting from the same initial æ. We compare the L2 distance between the trajectories.
5.2 EXPERIMENTAL RESULTS
We focus results on infrequent human-gated feedback where IP(obs = xx) 0.2, VI. In Appendix F. present experiments with frequent (=0.9) and partial (=0.5) feedback. Irrespective of the human-gated likelihood, the learner can always actively ask for feedback based on P(bbs; xx).
When the expert shifts, ConformalDAgger asks for more help immediately compared to EnsembleDA the expert with fewer human interventions than LazyDAgge DAgger. Consider deployment episode i where the expert shifts from goal 0 to goal (center row, Figure 5. ConformalDAgge immediately increases the number of expert feedback requests: before shift, the expert intervened ~20% of the time but in the 5th episode they intervene ~60% of the time. In contrast, EnsembleDAgger remains close to 30%. Relatedly, ConformalDAgger has consistently lower miscoverage rate (max 0.4 at shift; converges 0.2) compared to the baseline (max- 1.0 at shift, converges to 0.4). Due to
Takeaway 1:
Aggeeade ggerand AgeeDA enabling the algorithm to more quickly align to
8

---

Published as a conference paper at ICLR 2025
the extra solicited feedback, ConformalDAgger' ultimate decision and trajectory deviations minimized in the subsequent retrained policies.
Takeaway 2: Agger automatically asks for more help each time the expert drifts. In the third row of Figure S we see ConformalDAggeer maintain a similar level of queries as EnsembleDAgger during the first shift (at episode 5) and last shift (episode 11) but increases feedback during the intermediate shift (at episode 8). Despite these similarities, EnsembleDAgger miscoverage rate is consistently higher than ConformalDAgger's and EnsembleDAgget 's re- re-trained policy on average does not adapt quickly (with higher expert decision and trajectory deviation). We hypothesize that this is because AonformalDAgger asks frequent questions consistently across all seeds compared to EnsembleDAgger (i.e. lower variance in Fig
Takeaway 3: With stationary expert, ConformalDAgge and baselines are similar. We find that without distribution shift (top row, Figure 5 all methods ask for minimal help (staying near the 20% human- gated probability). with LazyDAgge asking slightly more questions, and achieve similar performance in alignment to the expert.
Takeaway 4: Under environment shift, ConformalDAgger is comparable to baselines. In the fourth row of Figure 5 we see that EnsembleDA and ConformalDAgger balance intervention frequency with policy learning quality marginally better than LazyDAgger or gggeD.
6 HARDWARE EXPERIMENTS
Finally, we deployed ConformalDA in hardware 7 degree-of sare--offfffeee freedom robotic manipulator that uses a state-of-the-art Diffusion Policy (Chi et al. (023) trained via IL to perform a sponging task (Figure D.. The goal of our hardware experiments to a high-dimensional, real-world policy and understand how ConformalDAgger enables the robot learner to query a real human teleoperator. Our goal is to train the robot to perform a real-world cleaning task where it wipes up line drawn by an Expo marker on whiteboard with sponge.
demonstrate how our approach can scale
Human Expert. The human expert teleoperates the robot via a Meta Quest 3 remote controller. They initially provide 50 demonstrated trajectories moving to construct the initial training dataset, Do. We model human-gate interventions with probability P(obs; = 1 xx) = 0.2, Vt. To control this rate, our interface queries the user with 20% probability at the any timepoint along deployment to simulate that consistent independent intervention rate. We induce potential distribution shift during interactive deployment episodes by changing the expert's wiping strategy from a straight line to zig-zag pattern.
straight-line path along the marker line
Learner Policy. We represent the robot's 's policy CNN-based diffusion policy (Chi et al. 2023). The policy predicts a" E A = y as 16 future actions, where each action an end-effector position and quaternion orientation. The inputs the wrist and third-person camera. Image observations are encoded using a ResNet-18 visual encoder (trained end-to-end with the diffusion policy) and the action-generating process is conditioned on encoded observation features with FiLM (Perez et al. 2018). The initial policy Tī is trained for 50K iterations (training parameters in Appendix See H.I. The robot is deployed to interactively execute the task 25 times after which its policy is retrained. This is an interactive deployment where the robot queries the user via our ConformalDA gger algorithm, or the user can independently intervene intermittently. During expert feedback, the human teleoperates the robot for a sequence of 16 timesteps, after which, if the robot doesn't ask for feedback, control is handed back over to the robot. After the interactive deployment, the initial learner policy is fine-tuned for an additional 35K iterations. We reset the learning-rate schedule, aggregate datapoints at which the expert gave feedback, and cap the training buffer size at 6k most recent input-action pairs.
X' are the current and previous image observations from
ConformalDA Hyperparameters. We apply IQT on only the predicted end-effector position, because reason about nonconformity via the signed residual in Fuclidean space. For quaternion rotation, we calibrate only the positional output to avoid the complexity of multiple IQT processes for positional and rotational nonconformities.
We initialize di ti 0.01, our desired coverage level is a = 0.1, and 7 = 0.15B,, where the lookback window for B, is 20. We use a simple model for PP(oss" (x)) = 1 that asks for help if the L2 norm of the uncertainty interval Cz(zz) exceeds threshold
0.07.
Baseline. Because the diffusion policy implicitly represents the action distribution present in the training data, we sample from the policy N=3 times and evaluate variance over the predicted action in

---

Published as a conference paper at ICLR 2025
Interactive Deployment
Covlomatiiggg
Frurskiliigg
S460Mgge
Sood Aag
Expert Stationary
LayDigger
SunDD
Export
Figure 6: Hardware Results. (bottom, left) Conformal enables the robot significantly increase the number of questions when the expert shifts their strategy while Safe and Ensemble only increase slightly. (bottom, right) Qualitatively, Conformal learns higher quality zigzag strategy. Lazy exhibits a similar an intervention profile to Conformal, but because the algorithm doesn's 't ask for help consistently in each episode, it does not receive enough additional data to learn the shifted policy.
order to capture the notion of epistemic uncertainty. We consider this an implicit EnsembleDAgger: we use the implicit ensemble approach from (Wolleb et al., 2022) by sampling multiple times from the same diffusion model. The safety classifier is a fully connected network that takes as input the ResNet-18 88 embeddings of the camera images concatenated with robot proprioceptive state. We set the classifier threshold 8 to be the mean of the distribution of expert-learner divergence over the training datapoints. EnsemblelAAgerrs uncertainty threshold is 0.005, at approximately the mean of the distribution of ensembles variances (over position and orientation predicted by the policy) over the training data, tuned such that the learner asks occasional but not excessive questions, We also compared with LazyDAgger (threshold of 0.25 s, informed by Hoque al. (2021a)), and SafeDAgger. All models use the same initial learner policy.
Results. The left of Figure 6 shows quantitative results for both methods. ConformalDA increases the cumulative number of robot-requested interventions up to about 35 timesteps of human feedback when interacting with the shifted expert (who switches to a zig-zag pattern) compared to the stationary expert (who always wipes in a straight line) which does not induce as frequent of questions. To better understand this, we analyzed Figure D which shows ConformalDA Aggeer engaging with each type of expert in one interactive deployment episode. Here, we see how ConformalDAgger has uncertainty when interacting with the stationary expert (due to inherent noise in the VR- VR-teleoperated intermittent human interventions) but the robot's uncertainty never rises above the threshold the robot initiates expert feedback. This is in stark contrast the expert shift scenario (right of Figure D where half-way through the deployment episode the uncertainty intervals become large enough to trigger robot asking for help. On the other hand, with the shifted expert, EnsembleDAgger and SageDAgger do not request as many interventions as LazyDAgger and ConformalDA gger, (Figure 6). When DA DAgger asks for help the queries follow a similar profile as ConformalDAgger, but it does not reliably ask every episode, leading to less data for learning the shifted strategy. We hypothesize that this may be because the training demonstrations have low variance while the sponge is in contact with the whiteboard, but higher variance as the robot approaches the table. Because EnsemblelD Ager'ss and ggger's uncertainty is uncalibrated to the deployment-ting expert's 's data, it unable to identify the need for additional feedback when it reaches the whiteboard. Qualitative results are shown on the right of Figure [ As expected, all approaches remain aligned with the stationary expert when retrained. However, under the shifted expert, rollouts from the retrained ConformalDAgger policy exhibit a more distinct zig-zas pattern, compared to EnsembleDAgger and SafeDAgger, likely because the approaches did not ask for much help.
LazyDAgger
7 CONCLUSION
We first extend uncertainty quantification via online conformal prediction to handle intermittent labels, such those observed in interactive imitation learning. We then propose ConformalDAgger. a unification of our online conformal prediction algorithm with interactive imitation learning. Our ap- proach provides asymptotic coverage guarantees for deployed end-to-end policies, uses the calibrated uncertainty measure to detect expert distribution shifts and actively query for more feedback, and empirically enables the robot learner update its policy to better align with the expert.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like thank Gokul Swamy for insightful conversations and the detailed review, Yilin Wu for help with diffusion policy and robot hardware setup. MZ is supported by an NDSEG fellowship.
REFERENCES
Anastasios Angelopoulos and Stephen Bates, Conformal prediction: A gentle introduction. Founda- lions and Trends in Machine Learning, 16(4):494-391 2023.
Anastasios Angelopoulos, Emmanuel Candes, and Ryan J Tibshirani. Conformal pid control for time series prediction. Advances in neural information processing systems, 36:23047-23074, 2023.
Anastasios N Angelopoulos, Rina Foygel Barber, and Stephen Bates. Online conformal prediction with decaying step sizes. In Proceedings of the 41 International Conference on Machine Learning. 2024.
Brenna D Argall, Sonia Chermova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-45) 483, 2009.
Vassilis Assimakopoulos and Konstantinos Nikolopoulos. The theta model: decomposition approach to forecasting. International Journal of Forecasting. 16(4):521-55) 2000.
Aadyot Bhatnagar, Huan Wang, Caiming Xiong, and Yu Bai. Improved online conformal prediction via strongly adaptive online learning. In International Conference Machine Learning, pp- 2337-2363. PMLR, 2023.
Carlos Celemin, Rodrigo Pérez-Dattary Eugenio Chisari, Giovanni Franzese, Leandro de Souza Rosa, Ravi Prakash, Zlatan Ajanovié, Marta Ferraz, Abhinav Valada, Jens Kober, et al. Interactive imitation learning in robotics: A survey. Foundations and Trends in Robotics, 10(1-2):1-197, 2022.
Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Advances in Neural Information Processing Systems, 34:965-979, 2021.
Yuxiao Chen, Ugo Rosolia, Chuchu Fan, Aaron Ames, and Richard Murray. Reactive motion planning with probabilistic safety guarantees. Conference on Robot Learning, pp. 1958-1970. PMLR, 2021.
Cheng Chi, Siyuan Feng. Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), 2023.
Yuchen Cui, David Isele, Scott Niekum, and Kikuo Fujimura. Uncertainty- aware data aggregation for deep imitation learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 761-767. IEEE, 2019.
Thomas G Dietterich and Jesse Hostetler. Conformal prediction intervals for Markov decision process trajectories. arXiv preprini arXiv:2206.04488 2022.
Anushri Dixit, Lars Lindemann, Skylar Wei, Matthew Cleaveland, George Pappas, and Joel W Burdick. Adaptive conformal prediction for motion planning among dynamic agents. In Learning for Dynamics and Control Conference, pp. 300-314. PMLR, 2023.
Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34:1660-1672, 2021.
Isaac Gibbs and Emmanuel J Candès. Conformal inference for online prediction with arbitrary distribution shifts. Journal of Machine Learning Research, 25(162):1-36 2024.
Michael Harries. Splice-2 comparative evaluation: Electricity pricing. 1999.
11

---

Published as a conference paper at ICLR 2025
Julien Herzen, Francesco Lässig, Samuele Giuliano Piazzetta, Thomas Neuer, Léo Tafti, Guillaume Raille, Tomas Van Pottelbergh, Marek Pasicka, Andrzej Skrodzki, and Nicolas Huguenin. Darts: User-friendly modern machine learning for time series. Journal of Machine Learning Research, 23 (124):1-6, 2022.
Joey Hong, Sergey Levine, and Anca Dragan. Learning to influence human behavior with offline reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S Brown, and Ken Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. arXiv preprint arXiv:2109.08273, 2021a.
Ryan Hoque, Ashwin Balakrishna, Carl Putterman, Michael Luo, Daniel S Brown, Daniel Seita, Brijen Thananjeyan, Ellen Novoseller, and Ken Goldberg. Lazydagger: Reducing context switching in interactive imitation learning. In 2021 IEEE 17th international conference automation science and engineering (case), pp. 502-509. IEEE, 2021b.
Ryan Hoque, Lawrence Yunliang Chen, Satvik Sharma, Karthik Dharmarajan, Brijen Thananjeyan, Pieter Abbeel, and Ken Goldberg. Fleet-dagger: Interactive robot fleet learning with scalable human supervision. In Conference Robot Learning, pp. 368-380. PMLR, 2023.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 991-1002. PMLR, 2022.
Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbelt and Mykel J Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference Roboties and Automation (ICRA), pp. 8077-8083. IEEE, 2019.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, al. OpenVLA: An open-source vision-language-action model. arXiv preprint woXiv:2406.0924: 2024.
Roger Koenker and Gilbert Bassett Jr. Regression quantiles. Econometrica: journal of the Economet- ric Society, pp. 33-50, 1978.
Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learming, pp. 143-156. PMLR, 2017.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
Justin Lidard, Hang Pham, Ariel Bachman, Bryan Boateng, and Anirudha Majumdar, Risk-calibrated human-robot interaction via set-valued intent prediction. In Robotics: Science and Systems, 2024.
Albert Lin and Somil Bansal. Verification of neural reachable tubes via scenario optimization and conformal prediction. In 6th Annual Learning for Dynamics & Control Conference, pp. 719-731. PMLR, 2024.
Lars Lindemann, Matthew Cleaveland, Gihyun Shim, and George J Pappas. Safe planning in dynamic environments using conformal prediction. IEEE Robotics and Automation Letters, 2023.
Rachel Luo, Shengjia Zhao, Jonathan Kuck, Boris Ivanovic, Silvio Savarese, Edward Schmerling, and
Marco Pavone. Sample-efficient safety assurances using conformal prediction. In International Workshop on the Algorithmic Foundations of Robotics, pp. 149-169. Springer, 2022.
Kunal Menda, Katherine Trigg-CCampbele and Mykel J Kochenderfer. EnsembleDAgger: A
Bayesian approach to safe imitation learning. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 5041-5048. IEEE, 2019.
Anish Muthali, Haotian Shen, Sampada Deglurkar, Michael H Lim, Rebecca Roelofs, Aleksandra Faust, and Claire Tomlin. Multi-agent reachability calibration with conformal prediction. arXiv preprint ar arXiv:2304.004. 2023.
12

---

Published as a conference paper at ICLR 2025
Cam Nguyen. S&P 500 stock data, 2018. URL /ttppt/e//////..aaaaaaaaaao///aa
Camnugent/ Eamtugent/sandp5500 sandp500
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models.
In International Conference on Machine Learning, pp. 8162-88 8171. PMLR, 2021.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Bob Price and Craig Boutilier.
Accelerating reinforcement learning through implicit imitation. Journal of Artificial Intelligence Research, 19:569-629, 2003.
Allen Z Ren, Amushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Pen Xu, Leila Takayama Takayama, Fei Xia, Jake Varley, al. Robots that ask for help: Uncertainty align- ment for large language model planners. In Conference on Robot Learning (CoRL). Proceedings of the Conference on Robot Learning (CoRL), 2023.
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019.
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems, 33:3581-3591, 2020.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, JMLR Workshop and Conference Proceedings. 2011.
Shahabedin Sagheb, Ye-Ji Mun, Neema Ahmadian, Benjamin A Christie, Andrea Bajesy, Katherine and Dylan P Losey. Towards robots that influence humans over long-term interaction. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp- 7490-7496. IEEE, 2023.
Stefan Schaal. Learning from demonstration. Advances in Neural Information Processing Systems, 9, 1996.
Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J Andrew Bag- nell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2102.02872, 2021.
Gokul Swamy, Nived Rajaraman, Matt Peng, Sanjiban Choudhury, J Bagnell, Steven Z Wu, Jiantao Jiao, and Kannan Ramchandran. Minimax optimal online imitation learning via replay estimation. Advances in Neural Information Processing Systems, 35:7077-7088, 2022.
Muhammad Faaiz Taufiq, Tean-Francod Ton, Rob Cornish, Yee Whye Teh, and Amnad Doucet. Conformal off-policy prediction in contextual bandits. Advances in Neural Information Processing Systems, 35:31512-315524 2022.
Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 72(1):37-45, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C Cattin. Diffusion models for implicit image segmentation ensembles. International Conference on Medical Imaging with Deep Learning, pp. 1336-1348. PMLR, 2022.
Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representa- tions to influence aulti-agent interaction. Conference on robot learning, pp. 575-588. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
Margaux Zaffran, Olivier Féron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In International Conference on Machine Learning, pp. PMLR, 2022.
Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to-end autonomous driving. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017.
14


Reviews:
[{"review_id": "eBlmEPWnKB", "reviewer": "Reviewer_YA6C", "summary": "Interactive imitation learning with queryable experts (e.g. Dagger) are a powerful approach to robot learning from demonstration. Unlike supervised-based offline approaches like behavioral cloning, interactive IL can avoid the distribution shift between the training distribution (i.e. the expert’s state distribution) and the test-time distribution (i.e. the learner’s state distribution). Prior work in uncertainty quantification-based IL has developed methods of actively seeking feedback from the expert (robot-gated), in contrast to the original Dagger-based approach, where the expert dictated the intervention (human-gated). The paper proposes leveraging online conformal prediction as a method for uncertainty quantification during deployment time. The paper extends quantile tracking to the intermittent feedback setting, which is what is encountered in the interactive IL setting the paper considers for robot learning. Using the proposed intermittent quantile tracking (IQT), the paper proposes a new approach to uncertainty quantification-based IL that uses IQT’s prediction intervals and evaluates on simulation and physical robot tasks.", "questions": "The phrasing of Line 7 in Algorithm 1 is confusing, as it suggests two of the exact same samplings.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper motivates the problem well and presents an improvement to existing uncertainty-aware interactive IL methods (specifically Dagger-based) that incorporates uncertainty estimates in deployment time. The figures are well-done, and the experimental setup is mostly understandable (see points below).", "weaknesses": "Overall, the limitations in the paper’s experimental results are the most significant weaknesses of the paper, which I discuss in detail below.\n\nThe paper only compares to one baseline, and they do not provide justification for why other baselines are not considered. Either incorporating additional baselines or providing a discussion of why other Dagger-based or Monte Carlo-based (e.g. Cui et al. 2019) methods were not considered and would strengthen the empirical results.\n\nContinuing with the limitations of the baselines, in the Hardware Experiments section (Sec. 6), the paper implements EnsembleDagger without a safety classifier. Rather than train a safety classifier to detect dissimilarity between the expert’s actions and the robot’s actions, the diffusion policy is sampled 3 times in order to “capture the notion of epistemic uncertainty.” The justification for this decision is unclear. Is replacing the safety constraint with multiple samples from the diffusion policy (which will be stochastic) better than implementing a safety classifier? Or is implementing a safety classifier unfeasible? (Additionally, I was unable to find the relevant discussion of an *implicit* EnsembleDagger-type approach in the referenced Wolleb et al. 2022.)\n\nThere are no details of the parameter tuning that was done to select EnsembleDagger’s uncertainty threshold. This is quite an important parameter that can affect its performance. EnsembleDagger is the only baseline compared against, so the lack of discussion of the uncertainty parameter undermines the relevance of the baseline. \n\nMore broadly, the empirical results only consider distribution shift due to expert shift. The paper does not test on environments with environment shift (e.g. changing dynamics). \n\nOn a different but important dimension, Dagger is also known to be a resource-intensive training procedure, requiring a queryable expert. The paper currently lacks a discussion of the computational efficiency of ConformalDagger and how it compares to EnsembleDagger. (e.g. measuring the wall-clock time to establish a comparison, discussing the IQT procedure’s efficiency when integrated into Algorithm 1, etc). \n\nAs a further extension, the original Dagger is known to fail in the setting where the expert policy is unrealizable. Currently, the paper has no discussion on what setting they consider with respect to the expert policy and the learner’s policy class, and whether the ConformalDagger approach would fare better or worse than prior work in the misspecified setting. \n\n**Minor Comments:**\n- Section 3 would benefit from further explanation (e.g. explaining the significance of Proposition 1) would improve the clarity of the section. \n- Appendix section numbers are listed as ???.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mYNLs6Jj6O", "reviewer": "Reviewer_YNNu", "summary": "The authors propose ConformalDAgger that uses runtime prediction interval to actively query expert’s feedback in the context of imitation learning for robot manipulation. ConformalDAgger introduces Intermittent Quantile Tracking, an uncertainty quantification method that provably minimizes regret about error of following expert’s behavior based on Angelopoulos et al., 2024, but also extends it to situations where the feedback is intermittent. Robot asks for human feedback whenever the prediction interval size exceeds some defined threshold, and the threshold also affects the regret guarantee. ConformalDAgger is validated on both typical time-series prediction settings and also robot imitation learning with human correction, showing improved coverage of expert behavior when the expert behavior distribution shifts, compared to EnsembleDAgger, a Bayesian uncertainty quantification method that can be miscalibrated.", "questions": "My question for the rebuttal is about the choice of hyperparameters. Can the authors comment on the choices in Sec. 5 and Sec. 6? If it is not possible to run new experiments, what would the effect of varying the threshold and temperature on the comparison to EnsembleDAgger look like?\n\nI also suggest adding the training details of EnsembleDAgger and e.g., how the action variance is computed, in the appendix.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper addresses a very important and timely problem in training robot manipulation policies — simple behavior cloning suffers from covariate shift, and rigorously quantifying the policy’s uncertainty and using it to query additional feedback from humans are an intuitive and potentially effective way to address covariate shift. Proper UQ ensures strong coverage while minimizing the amount of feedback needed.\n\nConformalDAgger is based on the online conformal prediction theory, which provides rigorous guarantees for the policy error, albeit being a regret-based one. The proposed extension to intermittent feedback setting is very novel.\n\nConformalDAgger shows clear advantage in coverage compared to EnsembleDAgger, which can be miscalibrated (although I have concerns about the setup of the baseline below). I also appreciate the experiments in the appendix varying p_t.\n\nThe paper is very well-written. Most experimental details are presented in the main text. All the illustrations are nicely done and convey the central ideas and experimental results cleanly.", "weaknesses": "I have mainly two concerns about the paper and proposed approach. The first one is about the experiment setup comparing ConformalDAgger to EnsembleDAgger. There is no discussion about how some of the hyperparameters, e.g., uncertainty threshold and safety classifier threshold used in EnsembleDAgger, are chosen in practice. I understand that these numbers can be difficult to choose in a rigorous manner, but some discussion will be greatly appreciated. Also, to strengthen the overall results, I suggest trying out a few different sets of thresholds, or even sweeping them, and comparing the two methods across the choice of these parameters. My understanding is that, with rigorous online conformal prediction theory, some of these parameters in ConformalDAgger will be easier to tune, and it would be good to discuss this too (or not if I misunderstand). \n\nMy second concern lies in the overall approach. IQT is a nice theory and provides guarantees, but in practice, if the feedback rate (p_t) is not high, which probably should be the case given human intervention can be tedious, the regret guarantee can be quite loose. The relationship between interval size and robot asking for help also confuses me: if robot doesn’t ask for help, the prediction interval is not updated since there is no new information, and intuitively this leads to less information for the robot deciding whether to ask for help. From my intuition, it makes sense to consider additional contextual information to adjust the prediction set size. I am curious how the authors think of this concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xTUaKxtPvK", "reviewer": "Reviewer_jhY9", "summary": "This paper presents an interactive imitation learning method (ConformalDAgger) that has the learner (robot) actively ask for additional supervision when it is uncertain during deployment. As in prior methods, it uses uncertainty quantification to do this, but unlike other methods, it updates the uncertainty quantification online using the expert feedback during deployment. The paper does this by introducing a novel online conformal prediction method called IQT, which is able to deal with intermittent expert labels, unlike prior approaches in online conformal prediction. The proposed method is shown to be effective at dealing with situations where there is Expert Shift, i.e. the expert policy changes between train and test time.", "questions": "- IQT is only applied to the end effector position (bottom of page 9) -- is this a limitation? There might be robot manipulation tasks that require more rotation-heavy actions.\n- In the hardware robot experiment, why is the EnsembleDAgger baseline different from the one used in the simulation experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and mostly easy to follow. The methods presented in the paper are interesting and appear to be novel. There is empirical evidence showing the ability of the proposed method to deal with \"Expert Shift\" in both simulated and real-world experiments.", "weaknesses": "**Weak and Unclear Motivation.** The proposed method seems to be exclusively useful for the setting of \"Expert Shift\" where the expert policy changes between train-time and test-time. This does not seem to be a very practical, commonplace issue. Are there other motivating examples for the proposed method as well, and corresponding settings where the proposed method would outperform other baselines? This aligns with Takeaway 3 in Section 5 too, where performance is similar to the baseline in the absence of expert shift. On a related note, a change in the expert policy can be arbitrary - it would be useful to provide some more intuition on what the best behavior should be for an ideal method in sensing and dealing with such a shift.\n\n**Thoroughness and Reproducibility of Experiments.** The paper could benefit from some more thorough simulation experiments. For example, including a simulation experiment with high-dimensional observations (e.g. images) similar to the real-world experiments could be beneficial. Also running such experiments on some standard benchmark tasks in existing simulation frameworks that go beyond goal-reaching would greatly improve the reproducibility of the work and the experiments. Some suggestions include [robomimic](https://robomimic.github.io/), [RLBench](https://sites.google.com/view/rlbench), [ManiSkill](https://www.maniskill.ai/home), or some of the tasks presented in [Diffusion Policy](https://diffusion-policy.cs.columbia.edu/). Also, the paper only compares against EnsembleDAgger but it seems like there are other suitable baselines that could also be appropriate, such as [ThriftyDAgger](https://sites.google.com/view/thrifty-dagger/home).\n\n**Other Issues.**\n\n- Algorithm 1, line 11 is potentially misleading. The expert is only selectively annotating states unlike DAgger where all states observed by the agent would be relabeled. \n- several undefined Sec refs (line 154, line 212, line 252, line 471)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)