ASkDAgger: Active Skill-level Data Aggregation for Interactive Imitation Learning
Authors: Jelle Luijkx
Zlatan Ajanović
Laura Ferranti
Jens Kober
Venue: Accepted by TMLR ()

Abstract: Human teaching effort is a significant bottleneck for the broader applicability of interactive imitation learning. To reduce the number of required queries, existing methods employ active learning to query the human teacher only in uncertain, risky, or novel situations. However, during these queries, the novice’s planned actions are not utilized despite containing valuable information, such as the novice’s capabilities, as well as corresponding uncertainty levels. To this end, we allow the novice to say: “I plan to do this, but I am uncertain.” We introduce the Active Skill-level Data Aggregation (ASkDAgger) framework, which leverages teacher feedback on the novice plan in three key ways: (1) S-Aware Gating (SAG): Adjusts the gating threshold to track sensitivity, specificity, or a minimum success rate; (2) Foresight Interactive Experience Replay (FIER), which recasts valid and relabeled novice action plans into demonstrations; and (3) Prioritized Interactive Experience Replay (PIER), which prioritizes replay based on uncertainty, novice success, and demonstration age. Together, these components balance query frequency with failure incidence, reduce the number of required demonstration annotations, improve generalization, and speed up adaptation to changing domains. We validate the effectiveness of ASkDAgger through language-conditioned manipulation tasks in both simulation and real-world environments. Code, data, and videos are available at https://askdagger.github.io.

Published in Transactions Machine Learning Research (08/2025)
ASKDAGGER: Active Skill-level Data Aggregation for Interactive Imitation Learning
Jelle Luijkx
j.d. J.S.kaikkodWudellf..
Department of Coguétive Robolics Delft University of Technology
Zlatan Ajanovié
slaton, www.:oviiimmm....teeeeeeee..eeee
Department of Computer Science
RWTH Aachen University
Laura Ferranti
Lferronti@itudefff..t
Department of Cognitive Robotics
Delft University of Technology
Jens Kober
j.kober@tudelff.om al
Department of Cogmitive Robotics Delft University of Technology
Reviewed on OpenReview: https: openreveew net/ forum? id= 987Ax0fffT
Abstract
Human teaching effort is significant bottleneck for the broader applicability of interactive imitation learning. reduce the number of required queries, existing methods employ active learning to query the human teacher only in uncertain, risky, novel situations. However, during these queries, the novice's planned actions are not utilized despite con- taining valuable information, such as the novice's capabilities, as well corresponding uncertainty levels. To this end, allow the novice to say: "I plan to do this, but I am certain." " We introduce the Active Skill-level Data Aggregation (ASKDAGGER) framework, which leverages teacher feedback on the novice plan in three key ways: (1) S S-waare Gating (SAG): Adjusts the gating threshold to track sensitivity, specificity, or a minimun success rate; (2) Foresight Interactive Experience Replay (FIER), which recasts valid and relabeled novice action plans into demonstrations; and (3) Prioritized Interactive Experience Replay (PIER), which prioritizes replay based on uncertaint, novice success, and demonstration age. Together, these components balance query frequency with failure incidence, reduce the number of required demonstration annotations, improve generalization, and speed up adaptation to changing domains. We validate the effectiveness of ASKDAGGER through language-conddtionee manipulation tasks in both simulation and real-world environments. Code, data, and videos are available at https://askdagge: seatllggggrgggghub....
Introduction
The promise of imitation learning is to enable individuals to teach robots to perform desired lasks, all without the need for specialized knowledge in coding robotics. This learning paradigm is beneficial when humans possess the knowledge to solve a lask but prefer not to do it themselves due to its repetitive, risky nature, when automation is more efficient. Specifically, demonstrating correct robot behavior in unstructured environments can be simpler than engineering a controller. Imitation learning has demonstrated succoss across various domains, such as autonomous driving [Pomerleaul
1988), helicopter acrobatics (Abbeel et al.) 2010). language-conditioned robotie manipulation [Jang al. 2022 Shridhar et. al. 2021). and generalist.

---

Published in Transactions on Machine Learning Research (08/2025)
robot policies (Brohan et al. 2022 Reed al. 2022 Octo Model Team et al. 2024). Despite these successes of behawioral clowing (BC), it can suffer from covoriate shift. This issue arises when imitation learning is naively posed as a standard supervised learning problem. In imitation learning, the data is not independent and identically distributed because past predictions can influence future states (Ross et al., 2011). As a result, prediction error can lead to encountering states unseen in the training data, causing a cascade of mistakes since errors in these unfamiliar states are even more likely,
Covariate shift issues can be alleviated with Interactive Imitation Learning (IIL) methods (Celemin ot al. 2022). These methods involve obtaining human demonstrations, corrections, and reinforcements interac- tively. In seminal work, Ross et al. (201T) introduced the Dataset Aggregation (DAGGER) algorithm. This approach alleviates the covariate shift problem by aggregating human input while executing the novice policy. This enables the novice to learn to recover from failures, for instance. While having favorable per- formance guarantees, the DAGGER algorithm requires continuous teacher input and can have safety issues as the novice policy is executed while learning to perform the task.
To overcome these limitations of the DAGGER algorithm, various extensions allow the novice to actively query the teacher in risky (Hoque et al. 2022) or uncertain situations (Menda et al., 2019 2017 Zhang & Cho 2017 Hoque et al. 2023). We refer to these data aggregation methods as active DAGGER approaches, they integrate data aggregation with active learning. The benefit of this active learning strategy is twofold. First, possible failures can be prevented during the interactive training phase since uncertainly is assumndd
to be correlated with failures. Second, this stralegy minimizes the number of demonstrations nceded by maximizing their meaningfulness. That is to say, it is waste of resources if humans demonstrate behaviors already mastered by novices. Instead, the teacher should only demonstrate what the robot novice can not do to enable learning from as few demonstrations as possible.
Existing active DAGGER methods hand over control when querying the human teacher. Instead, we allow the novice to also communicate their planned action when they are uncertain. This allows the teacher to validate or correct the novice plan, providing valuable feedback that can be leveraged in several ways. First, it reveals the levels of uncertainty where the novice succeods or fails. This information can be used to improve gating by allowing dynamic threshold adjustments to maintain a desired sensitivity, specificity, or minimum system success rate. This extends cxisting methods, which either require constant supervision (Kelly et al.) 2019), rely on heuristics (Zhang & Cho 2017), or use a fixed query rate independent of novice performance (Hoque et al. 2022). Second, validated novice actions can be aggregated into the demonstration dataset, reducing the need for teacher demonstrations, Additionally, invalid plans may be useful demonstrations for alternative goals the teacher relabels them accordingly. Third, since not all demonstrations are created equally, we can prioritize replay by considering the validity of the novice plan, the corresponding uncertainty level, and the demonstration ago. For example, the novice might learn more from a recent demonstration in situation where they failed rather than one where they acted successfully.
To this end, we introduce the Active Skill-level Data Aggregation (ASKDAGGER) framework, a novel II method where the robot novice actively communicates its planned actions when uncertain. An overview is shown in Fig, 1 and it is built three key contributions: i) S-Aware Gating (SAG): Adjusts the gating threshold to maintain a user-specified metric - sensitivity (true positive rate), specificity (true negative rate), or minimum system success rate. ii) Foresight Interactive Experience Replay (FIER): aggrogates valid and relabcled novice action plans into the demonstration datasct. iii) Prioritized Interactive Experience Replay (PIER): prioritizes replay based on uncertainty, novice success, and demoustration age.
of
Since ASKIAAGERR relies on the novice communicating its planned actions for teacher feedback, the hod is most practical for moderate feedback frequencies. ASKDAGGER therefore largets mid- to high-level control tasks rather than end-to-end policy learning. It is most applicable in scenarios where a robot has access to predefined parancterizable skills such as grasping, walking, pushing, door opening, screwing, or inserting. In such cases, the robot novice needs to learn the parameters and affordances of these skills given user-specified command. When querying the teacher, the robot novice can specify which skill they plan to use, along with the parameterization of that skill. If the teacher deems the novice's plan invalid, they can provide a demonsttation by annotating the appropriate skill and its parameters. For example, a pick skill can be parameterized by Cartesian pick position and orientation.

---

Published in Transactions Machine Learning Research (08/2025)
PIER
FIER Validate (o. ox 1)
Expert action og
Prioritize replay based on (a,r,
Place
Relabel (o, o...g.g.
Paper
Bio
k
Annotate (0, a.g.g.r
Pick
SAG
observation
novice action ax uncertainty u
goal g
Adaptive thresholding using history u,r,
Put the paper an the poper hine
Figure I: The Active Skill-level Data Aggregation (ASKDAGGER) framework consists of three main components: S-Aware Gating (SAG, detailed in Sec. 4.1h, Foresight Interactive Experience Replay (FIER, detailed in Sec. 4.2 and Priorilized Interactive Experience Ruplay (PIER, detailed in Sec. (.3). In this interactive imitation learning framework, we allow the novice to say: "I plan to do this, but / am uncertain." " 'The uncertainty gating threshold is set by SAG to track user-specified metric: sensitivity, specificity, or minimum system success rate. This facilitates trade-off between queries and failures. Teacher feedback is obtained with FIER, enabling demonstrations through validation, relabeling, or annotation demonstrations. Lastly, PIER prioritizes replay based on novice success, uncer- lainly, and demonstration age.
We make the following claims, considering an active data aggregation setting where the teacher can validate novice action plans:
C1 SAG balances query count and system failures by tracking a user-specified metric value: desired sensitivity, specificity, or minimum system success rate.
C2 FIER reduces the number of annotations needed to achieve given success rate by recasting novice actions to demonstrations.
C3 FIER enhances generalization to unseen scenarios by recasting failures to demonstrations.
C4 PIER improves the success rate and roduces the required annotations under domain shift compared to uniform sampling.
The remainder of this paper is structured as follows. Sec. 2 reviews related work. The problem formulation is presented in Sec. 3 Our method is introduced in See. followed by its experimental evaluation in Sec.
Sec. (6 discusses the results and limitations, and Sec. 7 concludes the paper.
2 Related Work
Uncertainty-aware III: In a seminal work on III. with active learning, Chernova & Veloso 2007) in- troduced the Confidence-Bann Based Autonomy (CBA) algorithm that combined the prediction confidence of a Gaussian Mixture Model (GMM) with the nearest neighbor distance from demonstration data to quantify the confidence the novice policy. Based on this confidence measure, control is then gated between the novice policy and the human expert. Several related strategies exist, primarily as safety- and uncertainty-awarr variants of the DAGGER algorithm (Ross et al. 2011), which we refer to active DAGGER approaches. Like CBA, these methods apply form active learning, i.e., actively querying in situations deemed informative and/or risky. Such techniques include confidence mensures based prediction confidence (Grollman &
Jenkins 2007), maximum mean discrepancy (Kim & Pineau 2013 Laskey et al. 2016), predicted proximity
novice actions expert actions (Zhang & Cho (017). Monte Carlo dropout (Menda et al. 2017 Cui
al. 2019), ensembles (Menda et al.) 2019 [i & Silver 2023 Li & Zhang] 2023). variational autoencoder
reconstruction error (Liu et al. [2024; Wong et al. 2002]] value estimates Hoque et al., 2022 Gokmen
al.
2023), ambiguity (Franzese
al.
2020
Luijkx
al. 2022). divergence (Datta al. 2023 and
diliusion policy training loss (Lee & Kuo,
2024
Outside the scope of IIL, robot-gating based on conformal
prediction theory was introduced as well
[Ren et al. 2023). In contrast to these robot-gated techniques,
human-gated hothods have also been proposed, requiring continuous human supervision, where the teacher

---

Published in Transactions on Machine Learning Research (08/2 (08/2025)
2024 There are also combinations actively intervenes Spencer et al., 2020, Kelly ct 2019 Luo et al.
of robot and human gating (Celemin & Kober 2023 Hoque et al,, 2022]. Our approach differs from ex- isting methods by considering the novice's actions during active queries, which we loverage in three ways. gating strategy to balance query frequency with
First, it enables a http://iww//pppeeieeee///eeeeeee error incidence while maintaining the desired metric value. Second, it allows novice actions to be recast as demonstrations by validating the novice's plan or relabeling the goal, inspired by Hindsight Experience Replay (HER) (Andrychowicz et al., 2017). Third, it enables replay prioritization based novice success, drawing inspiration from Prioritized Experience Replay (PER) (Schaul et al. 2015). Affordance Learning:
As ASKDAGGER Iearns goal-conditioned affordances the form of skill parameters, it relates to affordance learning (Mo et al. 2021: Mazzaglia et al. 2024 Wang et al. 2022 Ning et al. 2023 Geng et al. 2023), which focuses on predicting where and how a robol can successfully apply a skill, aiming for genera lization to un- seen objects. Key differences are as follows. First, these affordance learning methods typically require reward signals for all interactions, whereas AGKEAGGER queries for demonsstationns and requires leacher feedback only for robot- gated queries. Second, ASkDACGER learns a policy providing goal-conditioned affordances, avoiding distribution mismatch by aligning training and policy obscrvations. In contrast, cxisting affordance- learning methods primarily focus on efficiently exploring the large affordance statc-action space rather than directly learning policies. Although these methods can learn from failures, they require extensive exploration and may experience distribution mismatch during deployment. Reinforcement Learning from Human Feedback (RLHF): As ASKDAGGER queries teacher demonstration for uncertain novice actions, it is related to RLHF (Kaufmann et al., 2025). The most, closely related approaches also use uncertainy--basod querying, e-g., via information gain (Cui et al., 2019 Lindner et al., 2021). However, ASkDAGGER differs in that it requests new demonstrations when the teacher rejects a novice action, regulates the query rate through SAG, and collects relabeled demonstrations. In contrast, these RLHF methods typically learn a reward model and optimize the policy via reinforcement learning, whereas ASKDAGGER directly imitates expert actions, Another line of RLHF work focuses on human- -ated methods, where teacher interventions interpreted negative rewards (Kahn et al.. 2021 Luo et al. 2024). The main difference is that these methods require continuous human supervision during the data collection phase. Finally, Kumar et al. 2022 compare learning policics via behavioral cloning and offline RL from demonstrations. They conclude that the better choice depends on demonstration quality and the presence of critical states. Given our assump- tion of high-quality expert data and safety-critical decisions (e.g., skill misuse may damage the robot or its environment), our setting is better suited for a behavioral cloning approach. Research gap: Existing uncertainty- III, approaches request expert queries based solely on uncertainty estimates. These gal- ing methods do not take the novice's performance into account, as they discard the novice's actions when querying. In contrast, ASKDAGGER incorporates teacher feedback on queried actions and leverages this information for adaptive gating, demonstration collection, and prioritized replay for more efficient learning. Affordance learning methods aim to generalize to unseen objects but require constant reward information and are usually trained off-policy, which can lead to distribution mismatch at deployment. ASKDAGGER instead learns goal-conditioned affordances while collecting demonstrations on-policy, aligning training and execution distributions, while also not requiring constant reward information. RLHF methods often rely on learning a reward model or require continuous human supervision. ASkDAGGER avoids both by imitating expert actions and querying the teacher only when necessary, using validated or relaboled novice actions to enhance generalization and allowing for user-specified gating strategies.
3 Problem Statement
We consider an III. problem, where a (robot) novice is learning interactively from (hurnan) teacher feedback. The novice and teacher are denoted with subscripts and T, respectively. The novice learns a policy TN: O G + AA that maps observations of O and goals g² E g to actions of E A during episode k at time step L. We focus on mid- to high-level control lasks, where the novice has access to a set of predefined skills such as walking, grasping, or inserting. The novice learns from a demonstration dataset D = (r*) o consisting of trajectories r. These trajectorics consist of the parameters of the demonstrated skills provided by the teacher. Contrasting with existing works, we lot the teacher optionally provide a reward r indicating whether the novice's is actions were appropriate considering a goal ge and the observation of. Therefore, a

---

Published in Transactions Machine Learning Research (08/ (08/2025)
Algorithm Active Skill-level DAGGER (ASRDAGGER)
Input:
BC dataset PBC, BC policy T&. teacher policy Fy
Parameters: Gating mode mode {sensitivity, sppciiiiiiii.ssiiiiiiiit sprccfiicttyy sucoess], desired gating mode value ddes random query rate Prand, maximum number of episodes kmax
Output: kmnns
1. a += [].r+ [1:k+ 11.D+ Dgc
2 opisode k == Ninax do
r² og observa(), g command(), dona Palse, while not done do
- * g0)
", + quantiEy_uncertaing wwwertaintyllyt..... of g9)
- saG(u, r, des, Prand, mode)
// Set threshold to track ddes (Alg.
U10.1)
if w 7 ore Prand then
// Query actively and with probability Prand
a + FIER(O)) aal 441 act(a/)
// Collect demonstration (Alg. 5
13
else
13
o done act(aa))
1 ++1
D D
P(i), r.
// Prioritize replay (Alg.
++1 <- updato update_ P(().w)
"x
return "N
Amax
trajectory consists of tuples t² H(ot gt ,r))) where reward
if the teacher validates novice action;
-1 if the teacher rejects novice action and provides annotation; otherwise.
(I)
is worth noting that of is a Leacher reward obtained during queries related to the novice's 's actions, these actions may be different than at. In our interactive approach, we colloct data while exccuting the novice policy "N and iteratively update it with the dataset D, aggregating new demonstrations. Optionally, the policy can be pro-trained with a BC dataset DBC- During this update, we aim to find the policy л within between the novice's actions and the teacher's actions,
policy space II that minimizes a loss given the distribution of observations in D: x" = argmin L(x, D). Since we generally do not have full state information, we consider of to result from an observation mapping O : + and we observe the state of O(st). We define the goal state set S C to be the sel of states that satisfy the constraints of Therefore, an action af leads to success if 8i1 E Sa, The set of states that result in achieving some goal is the union of all possible goal sets, i.c., S = Useg S", We assume the goal g is constant throughout episode, i.c., independent of the dynamics and actions taken. Therefore, a failure described by transition (sf, at, #f.1. success success = 0) can be "relabeled" to success (sf at sf 1' auccess = 1) if st11 E 8 i.e., if the action resulted achieving some other goal g' E G. So, if the teacher can observe sf and (predict) &i,1, and can infer whether sh. SG, the teacher can relabel failure transitions to successes. Furthermore, we consider an active approach based on the policy's 's prediction uncertainty. Therefore, we require an uncertainty operator U I xx + Roo. that provides the prediction uncertainty of the novice policy, given the current observation and goal, i..., u. U(TN. of,g)) Optionally, one also take D into account when quantifying uncertainty, e.g., to quantify the proximity of an observation to those in D.
Active skill-level Data Aggregation(ASSDAgger Framework
ASKDAGGER is interactive imitation learning framework where the teacher is actively queried based on S-Aware Gating (SAG). The teacher can provide foedback through Foresight Interactive Experience Replay (FIER) in three modalities: validation, relabeling, annotation demonstrations. We update the novice

---

Published in Transactions on Machine Learning Research (08/2025)
policy using the demonstration datasct while we perform Prioritized Interactive Experience Roplay (PIER). The main training procedure of ASKDAGCER, summarized in Alg. follows these steps: In each episode k at every time step t, the novice policy o selects an action of based on observation of and goal g². Besides inferring its policy, the novice also quantifies the corresponding uncertainty ut (Alg. [ lines 2-6). SAG then sets a gating threshold ry to track the nser-defined level of sensitivity, specificity or minimum system success rate des (line 7). The teacher is queried if the novice uncertainty exceeds the gating threshold. Additional queries are obtained with probability Praud to enhance the performance of the SAG algorithm (detailed in Sec. 4.1h. During these queries the novice presents its planned action at, allowing the teacher to validate, relabel, and/or provide an annotation demonstration (lines 9-11). If the teacher is not queried, the novice acts autonomously (lines 12-13). When the episode is done, e-g., because the goal constraints are satisfied or a time-out is reached, the demonstration trajectory - is added to D. Finally, a model update can be performed using PIER (lines 16-17). Note that we keep track of the update/episode counts =0,,.. [0,..., [0,,..,Kr.] uncertainties u [ug.... ..... .ukk). and rewards r
rfkk] during training with ASKDAGGER, The next sections will provide detailed descriptions of the subroutines from Alg. [ i.e. SAG for gating (Alg, 21, FIER
for demonstration collection (Alg. 3) and PIER for replay prioritization (Alg.
4.1 S-Aware Gating (SAG)
At each time step, the uncertainty of the novice policy determines whether it should act autonomously or re- quest teacher feedback. We introduce SAG for this gating problem. It dynamically adjusts the gating thresh- old to maintain a user-specified larget: sensitivily (mode = sensitivity). specificily (mode = specificity), or minimum system success rate (mode = success). In this context, queries are treated as positives and autonomous actions as negatives. A false positive occurs when the teacher is queried despite the novice's 's action being valid, which reduces autonomy. A false negative occurs when the teacher is not queried despite an invalid novice action, which results system failure. Different gating modes are provided to suit varying task requirements. If system failures (false negatives) are costly, one can choose high desired sensitivity. If unnecessary expert queries (false positives) are more costly, one can choose to ensure a high specificity. Finally, when the overall system success rale is the primary concern, the success mode allows specifying a minimum desired success rate. SAG continuously adjusts y to ensure the success rate meets this target. If the novice's success rate falls below the threshold, more queries are issued to increase reliability via expert interventions. If the success rate exceeds the threshold, queries are kept to a minimum, as the constraint is already satisfied.
We formalize the gating problem as superveue logistic regression, using uncertainty u as the indepen- dent variable and reward as the indicator variable. The logistic regression assumption (the log- liheeoo ratio of class distributions is linear in the observations) holds for various exponential distributions, such as normal, beta, and gamma distributions (Amini & Gallinari 2002).
We summarize SAG in Alg. 2 and provide a visualization for more intuitive understanding in Fig. 2 For computing the gating threshold 7, we maintain a window of the most recent values in uw, and kw Wo do this because, with each model update, the uncertainty and reward information becomes more outdated (line 1 of Alg. 2 The window size is adjusted adaptively to ensure that the window contains at least Nmin relevant labels. Relevant labels correspond
r = 1 in specificity mode, r == -1 in sensitivity mode, and both are relevant in success mode. This is because, for instance, known failures (true positives) are essential when approximating sensitivity (true positive rate). Since the failure distribution over uncertainty shifts over time due to model updates, we normalize uw (lincs 2-4 of Alg. D to match the expected uncertainty at the current episode K. This is achieved by performing linear regression on kw and uw, then adjusting for the expected difference in uncertainty between episode and K. This is visualized in Fig. P A. Next, we fit logistic function to -Tw and uw (line 5 of Alg. [ and Fig, B-C). We negate the rewards because, in the context of sensitivity, positive cases typically represent costly events (novice failures). We capture these failures by negating rw. If the teacher was not queried, we do not know whether the novice acted successfully at (ke f). Since the uncertainty už is known, we can obtain pscudo-label by sampling from the fitted logistic model at uf (line 11 and Fig. 2 D). This enables us to approximate true and false positive rates for different threshold values.

---

Published in Transactions Machine Learning Research (08/20) (08/2025)
Algorithm S-Aware Gating (SAG)
Input: Uncertainties u, rewards т; update specificityysuceeee desired value dees random query Prand:
Parameters: Gating mode mode (sensitivity, seeeeiiiityysseeffcitty..
minimum number of negative labels Nuia: number of imputation repetitions Neep
Output: Gating threshold
,W, AW < (et_vindow( &. "minsmode)
2 Win: bin + LLinRerres () fit(ww; uw)
// Linear regression for normalizing
for u. aw do
L u² - win(K
// Visualized in Fig.
wlogsbiog Mng= (agtegrea)).ittuurr rw)
Logistic regression for inputation where
[]
7 for Nrep do
Jw -
// Visualized in Fig.
for r rw do
if r == then
// If teacher not queried
1t (-1,1)
P( = sigmold( blog)
// Visualized in Fig.
"K chr I'r dees Prand mode)
// Visualized Fig.
median(7)
return 7
Queries are not only made when uncertainty exceeds the threshold but also with probability Prand- These random queries ensure that labels collected across the entire uncertainly range, not just in the high- uncertainty regime. If random queries are not accounted for when setting the threshold, it will be too conservative, as more querics are triggered than intended. Since Prand is uscr-defined, we can incorporate these random queries when determining the gating threshold.
The overall SAG procedure identical across modes, with the only difference being how the threshold set (line 12). Here, we describe threshold selection for the sensitivity mode; for specificity- and success- aware gating, see App. [. and App. A.2 respectively. The total sensitivity rees can be computed by considering both the true positives TP, and false negatives FN, resulting from active gating, along with the true positives TP rand from random gating:
TP,, + TTP TP, FN,
The number of TP rand are determined by which of FN,, are queried. These queries occur with probability Praud (sce lines 8-9 of Alg. []. Therefore, its expected value is FN, Prand- This leads to:
TP, + FN, + FN,, Prand EnNUion (o. sene
TP,
(3)
TP, + FN, Prand (1 --0 o.
0.
seus
where ozee is the sensitivily for galing disregarding the random queries. Thus, we interpolate between
threshold values that best satisfy the desired sensitivity level dáes 0. sens and Fig. D).
+ Prand (1 (1- (1-o of (line 12 of Alg. 2
For each mode, the process of sampling pseudo labels is repeated Nrep times, and the final threshold y is set as the median of the thresholds obtained across repetitions.
4.2 Foresight Interactive Experience Replay (FIER)
When the novice's uncortainty exceeds the threshold set by SAG, feedback is requested from the teacher through FIER. This method enhances demonstration collection by considering the novice's planned actions during queries and presenting them to the human teacher. The FIER procedure is summarized in Alg, 3 FIER reduces the number of required teacher annotations by enabling the human teacher to provide two additional feedback modalities. First, we allow the teacher to validate the proposed actions. If the teacher

---

Published in Transactions on Machine Learning Research (08/ (08/2025)
1.00
0.8
Negated reward -r
Probability P
0.75
P(f(a)
Uncertainty
0.6
Not queried
Fallure f
Wlin D
0.50
Fit on data in B
0.4
0.25
0.2
0.00
92
93
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
Update count Is
Uncertainty
Uncertainty
Uncertainty
r=
LinRegres
P( 1/u)
P( 1/u)
75
Figure 2: Visualization of the SAG algorithm (mode sensitivity) with experimental data from Sec. 5.1 First, we normalize the uncertainty values in uw using linear regression, as the uncertainty distribution shifts with the number of updates (A). By negating the rewards rw: we obtain labels for novice failures, denoted as fw (B). Labels are mavailable when the teacher was not queried (r 0). However, since uncertainties at those time steps are known,
we generate pseudo labels by sampling from logistic distribution fit to uw and fw (C). then compute gating threshold " using both the labels and pseudo labels (D). This sampling and threshold calculation process is repeated Neop times.
considers the plan valid (r == 1 in line 2 in Alg. 3 we exccute and add the novice's planned actions to the demonstration dataset. The novice plan can be valuable even if is invalid. Inspired by HER Andrychowizz et al. (2017), we can utilize invalid novice plans as demonstrations if they achieve another goal (line 7 in Alg. 3 In this case, we can obtain a new demonstration by letting the teacher relabel the goal. The benefit of relabeling demonstrations is twofold. First, there can be situations where relabeling the goal is less demanding for the teacher than providing an annotation. In that case, it is possible to collect additional demonstrations at a reduced cost. Secondly, it allows for the collection of demonstrations for goals induced by the novice policy instcad of collecting demonstrations only for goals induced by the distribution of commands. This way, the novice can learn to perform tasks beyond the instructed commands and possibly generalize better to novel scenarios. After providing the option to relabel the novice's actions, the teacher is asked to provide annotation demonstration (line 6). To summarize, can collect three types of demonstrations with FIER:: validation, relabeling, and annotation demonstrations.
4.3 Prioritized Interactive Experience Replay (PIER)
At the end of the episode, the demonstrations collected through FIER are aggregated into the demonstration dataset, allowing the novice policy to be updated. Efficiently performing policy updates is particularly important in interactive imitation lcarning, as this paradigm involves human teacher providing online feedback. Taking inspiration from PER (Schaul et al. 2015); we introduce an interactive equivalent, which we call PIER (summarized in Alg. PIER prioritizes the replay of the demoustration dataset based on uncertainty, novice success, and demonstration age. We prioritize demonstrations where the novice fails over successes. Those with low uncertainty receive the highest priority among failures, as they suggest confident yet mistaken actions. While among successes, those with high uncertainty are prioritized to reduce the novice's uncertainty for those situations. Successes with low uncertainty, indicating proficient performance, are given the lowest priority. Since the uncertainty and novice success information become outdated with each model update, we diminish the prioritization based on the age of the demonstration. Alg. 4 shows how these desired properties are integrated into our prioritized replay scheme. Similar to PER (Schaul et al.) 2015), we define the probability of sampling demonstration tuple of episode limestep t to be:
(x/)a
F(k,t) = EEE,
Here of is the priority of the demonstration tuple from episode & at time step t and a 2 O. Increasing a results in more prioritization, while corresponds uniform sampling. We define the prioritization exponent.

---

Published in Transactions Machine Learning Research (08/2025)
Algorithm 3: Foresight Interactive Experi-
Algorithm 4: Prioritized Interactive Experience Replay (PIER)
ence Replay (FIER)
Input: Observation novice action a, leacher policy ær, trajectory goal gf
Input: Uncertainties rewards r, upilate counts &
Parameters: Scale A, base b, exponents a
Parameters: Goal set G
Output: Sampling priorities p. weights wr
Output: Action a, trajectory 7
1 g < queryío,a) a)
for
0:: do
2 if r then
Validation tuple
for t 0: do
a g
of = Au + (1
2)
A
else
pt 1 rr (0))a
T( T a r= 11 Annotation tuple o' E then
mr(o)
P(k, t) E, 2, 10330
Relabeled tuple
7 (o, 0 =0)
" ((k) P(ke [)) /maxi wj
return a, T.
return ((),
as a linear combination of the uncertainty and the number of model updates since the demonstration was added to the dalaset, i.c,, ot = Auy + (1- ( Here, S < 1 scales prioritization based on the uncertainty versus novelty of the sample. Finally, the priorities are:
2 =
(6)
b
where >> 1 the base. In this way, the priorities of old demonstrations with high novice uncertainty depend little novice success. In contrasi, the priorities for recent demonstrationn with low novice uncertainly depend greatly on novice success. Having defined the priorities, we need to compensate for the bias that prioritization introduces when minimizing the expectation of loss function 1, i.c., P(,,e (#N. (o af)). instead of sampling k, t according to the distribution of the datasct-i.c., k, f D(k, t). Again, following PER (Schaul al. 20155. we can mitigate the bias by introducing importance-sampling weights w = [uβ:.... (see line of Alg. Here, 8 determines the level of bins compensation. The weights are normalized to prevent numerical instabilities.
Experimental Evaluation
To support claims C1-4 from Sec. 1 we evaluate ASKDAGGER and its components in four sets of ex- periments. First, we performed active dataset aggregation on the MNIST dataset (LeCun al. 1998) using Torchuncertainty (Lafage & Laurent 2024) to validate SAG extensively. Second, we interactively trained CLIPort agents on simulated 2021). Third, conducted experiments extend beyond simulation. Finally, we showease ASKDAGGER's applicability by integrating it with built-in primitive actions on a Spot robot to perform a sorting task.
conditioned tabletop manipulation tasks (Shridhar et al., real-world assembly setup to demonstrate that these claims
5.1 MNIST Dataset Aggregation
To support claim C1 SAG balances query count and system failures by tracking a user-specified metric value: desired sensitinity, specificity. or minimum system success rafe we conducted experiments in which we interactively trained digit classification models on the MNIST dataset (LeCun et al. 1998) We selected this setup due to its low computational requirements, enabling extensive ablations and easy reproducibility. Additionally, existing packages such as TorchUncertannty (Lafage & Laurent, (024) facilitate uncertainty quantification in this setting. Since we focus on the SAG component, in Alg. I but without demonstration collection via relabeling or replay prioritization. To validate whether SAG can track desired sensitivity, performed interactive training for nine different scnsitivity, specificity and success rate values, i.e, ddes E {0.11)) repeating the procedure ten times for each value of des.
follow the procedure described
The code and from these experiments are available at htttp::/ httpa////g.ttubb.cccaaaaaeggegge http./////.aaagggerr/idagggorrra

---

Published in Transactions Machine Learning Research (08/2025)
These experiments proceod as follows. We sample a batch of 128 handwritten digit images from the MNIST dataset at each timestep without replacement. This allows for 468 timesteps, the dataset contains 60,000 samples. Although MNIST provides ground truth labels, we simulate an active learning scenario where labels queried if the prediction uncertainty exceeds the threshold y set. by SAG randomly with probability Prand = 0.1. Uncertainty quantification is performed using Monte Carlo Dropout (MC Dropout) (Gal
Ghahramani 2016) with a dropout rate of 0.4 and 16 stochastic forward passes, forming an ensemble C= (1 ,11). For sample with label y: prediction uncertainty is computed where Pe(vll)) Te Li=1 (y)a).) Ground truth labels are obtained for the queried samples and added to the training dataset. The model is updated every five timesteps using the aggregated dataset.
1 maxy Po(y)
Sensitivity-Aware Gating
Specificity-Awarr Gating
Success-Aware Gating
1.0
1.0
1.0
System Success Rate
0.8
0.8
0.8
Sensifivity
Specificity
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
200
400
200
400
200
400
1.0
1.0
1.0
0.8
0.8
0.8
Query Rate
Query Rate
Query Rate
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
200
400
200
400
200
400
1.0
1.0
1.0
Novice Success Raie
Novice Success Rate
Novice Success Rate
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
200
400
200
400
200
400
A
Step
B
Step
C
Step
des 0.1
dee 0.3
ddes 0.5
ddee
0.7
ddes 0.9
des 0.2
dee 0.4
ddes 0.6
ddee
0.8
Desired (top row)
Figure 3: Results for various levels of dées for sensitivity-aware (A). specificity-aware (B), and success-aware (C) gating active dataset aggregation with SAG on the MNIST' (LeCun al. 1998) dataset. Sensitivity in A and specificity in B are calculated over moving window of 1000 failures and successes, respectively. Novice and system success rates are calculated over window of 1000 samples (approximately eight steps). Mean and standard deviation are shown for ten repetitions.
The results of these experiments are summarized in Fig. 3 The sensitivity and specificity plots in Fig. 3 A and B show that SAG successfully tracks the desired levels for all nine values of ddes- In success-aware mode, Fig. 3 C shows that when the novice success rate is low, SAG issucs enough queries to maintain the desired system succoss rate des. As the novice success rate increases, the query rate decreases, reaching a minimum once the novice success rate exceeds ddes. The query rate plots in Fig. also indicate that each
10

---

Published in Transactions on Machine Learning Research (08/2025)
mode requires a different query pattern to track its respective metric. The success rate plots show that, all modes, the novice ultimately learns to perform the task.
5.2 CLIPort Benchmark Tasks
To support claims C1-3, we conducted experiments using AGGER to train CLIPort (hridhar et al. 2021) agents interactivcly - CLIPort is a danguage-conditionce imitation-leariing agent for vision-based manipulation. The observations in these experiments consist of an RGB-D image and natural language lext command. CLIPort employs a Lwo-stream architecture: a spatial stream and a semantic stream. The semantic stream uses frozen CLIP encoders (Radford et al. 2021 extract features from the RGB image and the language command. The spatial stream is an untrained Transporter network (Zeng cl al.) 2021 whose decoder layers are fused with features from the semantic stream. The model outputs pixel wise value estimates for both picking and placing. The demonstration dataset includes RGB-I) images, language commands, and expert actions in the form of Cartesian pick and place poses, represented as a pixel location with discretized orientation. We selected this set up because it allows novices to communicate their actions image alongside language command, making it well-
by indicating planned pick-and-place locations suited for A SSKDAGGER
We compared ASKDAGGER'S performance against an active DAGGER bascline without both PIER and FIER to provide evidence for claims C2-3. We also compare ASKDAGGER against SaGEDAGGER (Zhang & Cho 2017) and ThriftyDAcors (Hoque et al. 2022), which are also DAGGER approaches that incorporate active learning. Furthermore, we performed ablations with ASkDAGGER without PIER and ASKDAGGER without FIER to isolate the elfects of the individual components. The models trained with ASKDAGGER and active DAGGER use SAG for gating and rely on prediction entropy to quantify uncertainty for a fair comparison.
The comparison was conducted across a subset of tasks by [Shridhar et al. [2021 These tasks are visualized in Fig. 4 The CLIPort benchmark includes seen and unseen task settings. In the unseen sctting, dest-time commands involve different objects, shapes, or colors than during training. For example, Tween colors = [yellow, brown, gray, cyan} and Tunsson colors (purple, pink, white, black]. Some colors appear in both settings, i.e., Tall colors = [red, green, blue}. A complete list of seen and unseen objects is provided in Tab. f in App. [A.8] We modified the tasks involving Google objects and shapes by sometimes introducing unseen objects
distractors during training, as real-world scenarios also involve varying distractor objects. Therefore, it is possible for ASKDAGGER to acquire demonstrations for the unseen set via relabeling with FIER when the novice fails in a meaningful way. This setup allows us to provide evidence for claim C3 by evaluating whether failures can be relabeled and whether this improves generalization to the unseen setting.
The following hyperparameters were used for both ASKDAGGER and the active DAGGER baseline if applicable. For SAG we used mode sensitivity, ddes 0.9, Namin 15. Prand = 0.2 and for PIER = 1.5,b b- = 10, 3- = 1 and À = 0.5. Each setting involved training ten CLIPort agents without BC pretraining, collecting 300 interactive demonstrations, and evaluating checkpoints every 100 demonstrations. Model updates occur at the end of an episode if demonstration is collected. This way, ensure that ASKDAGGER and the baselines had the same mumber of updates.
The cumulative rewards for evaluating checkpoints on lasks with seen and unseen objects are shown in Fig. 5 While ASKDAGGER performs equally or better on all tasks, the number of teacher annotations is significantly lower, as shown in Fig. G The performance gain of ASKDAGGER can be attributed to the composition of the demonstration datasct. For the active DAGGER baselincs, all demonstrations consist of annotation tuples, whereas ASK D) collects many through validation and relabeling. The relabeled demonstrations explain ASKDAGGER 's superior performance on unseen tasks: agents sometimes obtained demonstrrations by relabeling novice failures, where the intended pick was a distractor object from the unseen set. Fig, aalso shows that ASKDAGGER requires fewer annotation demonstrations than the active DAGGER baselines. The corresponding sensitivity curves for ASKDAGGER in Fig. confirm that SAG maintains the desired scnsitivity level across all tasks.
The code, data videos, and a notebook are available http:://ghhhe...oo/aaaaarr/aaasg
11

---

Published in Transactions Machine Learning Research (08/2025)
"pack alarm in the brown box"
paac all the plush objects in the brown hox"
"pack triangle in the brown box"
"pul the blocks in a yellow how!"
Figure The CI.IPort. benchmark tasks from to right: prrkiing-oooll--bieee booeagg-oooooosss----eeo packing-shapes and put-blocks- in-bowls.
packing-seen--oone objects-sed
packing-seen-google Sjjects-group
put-block-in bowi-seen-colors
packing- seaiiigg-eee--haees
100
80
80
80
Seen: Reward
75
60
40
40
50
20
20
25
100
200
300
200
100
300
100
200
300
nacking-unseen-google
packing-unseen-google
put-block-in bowl-unseen-colors
objects-seq
objects-group
packing-unseen-shapss
Unseen: Reward
60
60
40
40
20
20
20
0
100
200
300
100
200
300
100
200
300
100
200
300
E
H
Demonstrations
Demonstrations
Demonstrations
Demonstrations
ActiveDAgger
ThriftyDAgger
SafaDAgger
ASkDAgger
Figure 5: Cumulative rewards for evaluating checkpoints over 100 episodes tasks with seen and unseen objects. Mean and standard deviation shown for ten policies using moving window of episodes. The results show clear improvements with AGGRR on the unsee acenarios.
To support C4, we performed ablations with ASKDAGGER under domain shifts. We trained agents using ASKDAGGER, ASKDAGGER without PIER (w/o PIER), and ASKDAGGER without FIER (w/o FIER) on a sequence tasks with increasing domain shifts: packing-sees seen- seenng--ee---sppss,,, packing- wecking-usseen- shapes, and packing seen- google-objects-sqq. As shown in Fig. & ASKDAGGER and ASKDAGGER w/o PIER perform similarly the initial task, while ACCER w/o FIER performs slightly worse, likely due to the benefits relabeling demonstrations. Moreover, ASKDAGGER w/o FIER requires more annotation demonstrat tions (Fig. S D)- After transitioning to mucking-unsenn-sskuee though performance is similar after 300 demonstrations across all set settings. This improved adaptation likely stems from FIER's relabeling and PIER's replay prioritization. Upon shifting to packing-seen www.seeeee-----nneeeee oojoee--bjjees- seg, AGGERGGER and ASKDAGGER w/o FIER outperform ASKDAGGER w/o PIER, highlighting the benefits of replay prioritization with PIER. This effect is more pronounced after the second transition, as the shift from unseen-shapes to seen-google objjetss-seee is larger than from to shapes. Additionally
ASKDAGGER adapts slightly faster to the domain shift initially,

---

Published in Transactions Machine Learning Research (08/20 (08/2025)
packing-google objects-seq
packing-google objects-group
put-block in-bowl
packing-shapes
Demonstration Tuples
400
400
500
200
250
0
A
100
200
300
B
100
200
300
C
100
200
300
D
100
200
300
Demonstrations
Demonstrations
Demonstrations
Demonstrations
ActiveDAgger: Annotation
SafeDAgger: Annotation
ASkDAgger: Relabeled
ThrityDAgger: Annotation
ASKDAgger: Annotation
ASKDAgger: Validation
Figure Composition the demonstration datasets, showing mean values for tem policies. ASKDAGGER relies on fewer annotation demonstrations and benefits from relabeling and validation demonstrations.
packing-google objects- seq
packing-google objects goroupt-groou group
put-block in-bowl
packing- shapes
Sensirivity
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
A
100
200
300
B
0
100
200
300
C
0
100
200
300
D
0
100
200
300
Demonstrations
Demonstrations
Demonstrations
Demonstrations
Sensitivity
Desired
Figure Sensitivity during training for ASKDAGGER the CLIPort tasks. Sensitivity calculated over moving window of 50 failures. Mean and standard deviation are shown for ten policies.
the demonstration dataset grows, the probability of sampling a specific demonstration decreases uniform sampling, further increasing PIER's effect. Finally, ASKDAGGER'S improved performance the third task requires fewer annotation demonstrations, as more validation demonstrations are collected (Fig. FP.
5.3 World Engine Assembly
We conducted experiments real-world assembly task to demonstrate that our claims extend beyond simulation and showcase ASKDAGGER's applicability in real-world scttings M This task is a simplified version of diesel engine assembly using 3D-prinnedd models. The procedure is illustrated in Fig. g As shown in Fig. g A, the setup includes a Franka Panda robot equipped with an in-hand RealSense D405 RGB-D camera and a Franka. hand with custom- ceiton-printed fingers for grasping bolts. The control scheme implemented using the EAGERRx framework (van der Heijden al. 2024). The objective to pick bolts from holder and insert them into specific locations on the engine block. We use pick-and-place primitives that roly on 2D Cartesian positions, assuming a given height for picking and placing.
The task involves four bolt colors (red, yellow, green, and black) and seven insortion locations. The bolts are randomly ordered and placed in a holder. The human operator interacts with the robot via interface shown in Fig. B. This Gradio {Abid al. 2019 interface allows command input via speech or text. In our experiments, we generate random commands in the form: "Insert the (color) boft at location number [location number]]" Upon receiving command, a top-down RGB-D reconstruction is obtained following Zeng et al. (2021), and the novice policy is evaluated. We
CLIPort agent with the same hyperparameters in Sec. 5.2 but with a smaller batch size three due to GPU memory constraints on our laptop. If the prediction uncertainty exceeds the threshold 7 set by SAG, randomly with probability Prand 0.2, the robot queries the human operator. This is shown in Fig. g C. The robot's planned action is visualized in the
A video of this experimental evaluation avsilable https:: httpps//askdagger...m iithub.io

---

Published in Transactions Machine Learning Research (08/22) (08/2025)
packing-seen-shapes
packing-unseen--haaps
packingssee--googll-abieeee
Novice Success Rate
0.6
0.0
0.4
0.6
0.2
0.4
150
200
250
300
C
300
350
400
450
150
150
300
Ann. Tuples
100
100
200
50
100
150
200
300
400
450
Demonstrations
Demonstrations
Demonstrations
ASkDAgger w/o PIER
ASkDAgger w/o FIER
ASkDAgger
Figure 8: Novice success rate during training (A-C) and the number of collected annotation tuples (D-F) under domain shifts. The shift. The first 150 demonstrations are collected on packing-setd seeinng-eee--iee meen-shapes, the next 150 on packing-are packing-anseen--eee -n--en-sesppps,, and seennnng-eee--geee the final 150 on packing seen gogle-objects-sey Mean and standard deviation are shown for ten policies.
is computed
moving window of 50 upisodes, reinitialized alter each domain
interface, allowing the operator validate, relabel, or reject the plan. If validated, the robot executes the plan and it is added to the dataset. If relabeling is required, as in Fig, C, the operator is prompted to provide corrections by selecting the correct bolt and location. This is shown in Fig. g D. The robot then executes the annotation demonstration (Fig. g and aggregates the relabeling and annotation demonstrations into the dataset. When demonstration is collected, model update is performed while the robot executes the demonstration.
summarized in Fig, 10 The novice and system success rates in Fig. I0 A show that while the novice is learning to perform the task, the system can maintain a high success rate by querying based on SAG. The composition of the dataset Fig, [0] B shows a similar trend as in the simulateed experiments. This figure shows that ASKDAGGER can learn from mostly validation demonstrations in the later stages of training. Also, it shows that relabeling demonstrations is possible not only in simulation but also in real-world scenarios. The sensitivity in Fig. 10 shows that SAG track the desired scnsitivity level across the task.
The results for collecting 150 demonstrations
5.4 Real-World Sorting Task
To further demonstrate ASKDAGGER's applicability in real-world scenarios, integrated it with Boston Dynamics Spot quadruped's built-in primilive skills to perform a sorting lask A Since ASKDAGGER is designed to work with any robot with one or more skills, we selected Spot for its built-in grasping, walking, and placing capabilities. The task involves sorting objects into paper and organic waste bins, as illustrated Fig. I For this demonstration, we trained a CLIPort agent using an interface similar to Fig. g with the command format: " "Put the [object] in the [bin type] bin." Since CI.Ioort requires top-down RGB-D projection, we use the hand RGB-D camera to scan the environment from different perspectives. With these images, we can oblain a top-down projection using robot's 's joint states, adometry, and camera intrinsics. The teacher issues the command through Gradio (Abid et al. 2019) interface at the beginning of an episode. Similar to See. 5.3 the Leacher is queried if the novice's uncertainty exceods the gating
A video of this demonstraaion available at. htttps://askdaggerr httpe////aacbaggergiinn...oo github.io

---

Published in Transactions Machine Learning Research (08/2025)
What do you want me to do?
I'm not sure what to do, can provide feedback on plan?
Arond
Plo
Fllk
plan
image provide demonstration
verete sades te wester sunder
Polii
n Chee
Figure 9: Real-world implementation of ASKDAGGER on an engine assembly task. (A) The setup includes Franka Panda robot, an RGB-D camera, and 3D-printed parts. (B) The interface allows the operator to issue commands. (C) When queried, the operator validate, relabel, reject the plan. (D) The operator relabels the plan. (E)
The operator provides an annotation demonstration. (F) The robot executes the demonstration.
1.0
Demonstration Tuples
Ann.
1.0
0.8
Rel.
Success Rate
Val.
Sensifivity
0.9
0.6
wM
100
0.8
0.4
Novice
50
Sensitivity
System
Desired
0.0
50
50
100
100
150
A
C
Demonstrations
Demonstrations
Demonstrations
Figure 10: Results for the engine ussembly task. (A) Success rates calculated window of 50 episodes. Composition of the demonstration dataset. (C) Sensitivity calculated over a moving window 50 failures.
(B)

---

Published in Transactions Machine Learning Research (08/20) /2025)
B
Figure 11: Real-world implementation ASKDAGGER on a sorting task with Spot (A). This demonstrates AS&DAG- CER's applicability in scenarios where a robot has access to one or more skills, such as grasping (B), walking, and placing (C).
Uhreshold, and the interface shows the planned pick and place locations. The teacher can choose to validate relabel the novice's actions. The teacher can provide annotation demonstrations by clicking where to pick and where to place in the top-down projected image. Demonstrations are aggregated to the demonstration dataset the end of the episode, and the policy updated. In the meantime, the robot walks back to its initial position and rescans its environment.
Discussion
This section discusses the results from Sec. 5 by revisiting the claims Sec. limitations of the ASKDAGGER framework and its experimental evaluation.
We also examine the
6.1 Claims
Our experiments were designed to provide evidence for claims C1-4 from Sec. 1 First, we claim that SAG balances query count and system failures by tracking user-specified metric value: desired sensitivrity, specificity, or minómam syslem sensitivity, specificity, and system success rate tracking for multiple values of ddes an MNIST dataset aggregation lask Fig. 3 Moreover, we exlend these results for sensitivity-aware mode simulated and real-worie robot tasks in Sec. 7 and Fig. 10 C. Secondly,
role (C1). We provide evidence for this claim by showing accurate
claim that FIER reduces the number of annotations needed to achieve given success rate (C2). We trained CLIPort agents on a set of benchmark tasks simulation for 300 demonstrations and ovaluated the performance of chockpoints saved every demonstrations. The results from these experiments support this claim. The evaluation results in Fig. show that ASKDAGGER performs equivalent or better compared to SafeDAGGER (Zhang Cho 2017]. Thrifty (Hoque al. 2022), and AGGER,, while Fig. shows that ASKIAAGER requires significantly fewer leacher annotations. The results in Fig. 10 provide evidence that these results generalize to real-world tasks, as we see similar composition the dataset and similar performance improvements. Thirdly, claim that FIER enhonces generalization to unseen scenarios by recusting failures to demonstra- tions (C3). The results in Fig. provide evidence for this claim, as the AGGER checkpoints outperform the active baselines on the unseen scenarios. This can be attributed to relabeling failures involving the dis- tractor objects, resulting in demonstrations for the unseen scenarios. Finally, claim that PIER improves the success rate and reduces the required annotations under domain shift compared to uniform sampling. Exidence for this claim is provided by the results in Fig. g where PIER outperforms uniform sampling under domain shift regarding success rate while requiring fewer annotations.
6.2 Limitations
AGGEAGGER is designed for tasks with sparse rewards and learning mid- to high-level control. While this covers many problems, does not extend applications requiring high-rate feedback from the teacher. Thus, ASKDAGGER is best suited for sccnarios where robot has access to predefinod skills and can learn higher-level plans for those skills. Additionally, ASKDAGGER can be integrated with methods that are better suited for learning low-level control horizon reasoning. While FIER significantly improves
16

---

Published in Transactions on Machine Learning Research (08/2025)
performance, it relics on recasting failures as successes, which can be challenging in some applications. Additionally, although AGKER reduces the number of teacher annotations, it assumes the teacher can validate or relabel actions before execution. This may not always be feasible. In such cases, relabeling still occur post-execution. The impact of ASKDR on teaching effort depends on both the task and the teaching interface, Nonetheless, we provide an in-depth discussion in Sec. A..6] including timing measurements from Sec. and cost scenarios for Sec. 5.2 Tuning hyperparameters in real- world scttings is challenging. However, our experiments show that a single set of hyperraaaaeeeess without extensive tuning, generalizes effectively across different tasks, suggesting that hyperparameter sensitivity is not strongly dependent on task descriptions. Finally, our ovaluations primarily usod policies with a CLIPort (Shridhar et al. 2021 architecture, but AGKDAGGER is not limited to this choice. It can be applied to any policy architecture where a teacher can determine the success of actions and provide demonstrations..
and validation
Conclusion
We have introduced the ASKIAAGGR framework, which consists of three components: gating procedure SAG that can track a user-specilied metrie; FIER for recasting novice actions to demonstrations; and PIER to prioritize replay experience based on uncertainty novice success, and age. Our experiments show that ASKDAGGER allows for effective balance between queries and failures based on user-specified metric sensitivity, specificity, or minimum system success rate. It significantly reduces the number of required annotated demonstrations while also improving its generalization capabilitics to unseen scenarios by utilizing demoustrations acquired through interactive relabeling and validation. Additionally, it improves adaptation to domain shifts by prioritizing replay.
For future work, several promising directions emerge for further enhancing the capabilities of AGGER Extending this methodology to scenarios involving longer horizons and non-sparse rewards could improve its applicability. Leveraging gencralivv/louudation foundation models and simulators for visualizing action plans and generating artificial demonstrations could also expand the application domain of ASkDACGER, potentially improving training efficiency and outcome predictability. Additionally, applying ASkDAGGER to higher- frequency, low-level control tasks is another interesting future direction. Although relabeling and validation may become less beneficial with higher query frequencies, SAG could still ellectively query human teachers when robot execution slowed, combining robot-driven queries with human interventions. Integrating robol-gated AGKEAGGER for high-level affordance or skill learning with human-gated interventions for low- level control could create a hierarchical structure. This integrated approach would allow simultaneous on- policy demonstration collection across multiple control levels at different frequencies. Moreover, conducting participant studies would be vital to assess the mental load associated with ASKDAGGER and refine user interfaces accordingly. Finally, adapting ASKDAGGGE to environments characterized by heterogeneous or imperfect teachers could widen its applicability and effectiveness in real-world settings.
8 Acknowledgments
Research reported in this work was partially completely facilitated by computational resources and support of the Delft AI Cluster (DAIC) (Delft AI Cluster (DAIC) 2024) at TU Delft (RRID: SCR. 025091), but remains the sole responsibility of the authors, not the DAIC team.
References
P. Abbeel, A. Coales, and A. Ng. Autonomous helicopter aerobatics through apprenticeship learning. Intl. Journal of Robotics Research (IJRR), 29(13):1608-160) 1639, 2010.
A. Abid, A. Abdalla, A. Abid, D. Khan, A. Alfozan, and J. Zou. Gradio: Hassle-free sharing and testing of ML models in the wild. arxiv preprinl, 2019.
M.R. Amini and P. Gallinari. Semi-supervised logistic regression. In Proc. of the Europ. Conf. on Artificial Intelligence (ECAI), pp. 11, 2002.
17

---

Published in Transactions on Machine Learning Research (08/2025)
M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, Fong, P. Welinder, B. McGrow, J. Tobin, P. Abbeel, OpenAT, and W. Zaremba. Hindsight experience replay. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), 2017.
A. Brohan, N. Brown, J. Carbajal,
Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. RT-1: Robotics transformer for real-world control scale. arXiv preprint, 2022.
C. Celemin and J. Kober. Knowledge-and ambiguity-aware robot learning from corrective and evaluative feedback. Neural Computing and Applications, pp. 1-19, 2023.
C. Celemin, R. Pároz-Dattari, E. Chisari, G. Franzese, L. de Souza Rosa, R. Prakash, Z. Ajanović, M. Forraz, A. Valada, J. Kober, et al. Interactive imitation learning robotics: A survey. Poundations and Trends in Robotics, 10(1-2):1-197, 2022.
S. Chernova and M. Veloso. Confidenco-based policy learning from demonstration using gaussian mixture models. In Proc. of the Inil. Conf. on Autonomous Agents and Mullingent Systems ( (AAMAS), pp- 1 8, 2007.
Y. Cui, D. Isele, S. Nickum, and K. Fujimura. Uncertainty-awan aware data aggregation for deep imitation learning. In Proc. of the IEEE Inll. Conf. on Robolics & Automation (ICRA), pp. 761-767. IEEE, 2019.
G. Datta, R. Hoque, A. Gu, E. Solowjow, and K. Goldberg. IIFI: Implicit interactive fleet learning from heterogeneous human supervisors. In Proc. of the Conf. on Robot Learning (CoRL), volume 229, pp. 2340 2356. PMLR, 2023.
Delft. Al Cluster (DAIC). The Delft Al Cluster (DAIC), BRUD:SCCR_002009 2024. URL https://doc. daic. tudelf..lll.
L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T. B. McHugh, and V. Vanhoucke. Google scanned objects: high-quality datasel of 31) scanned household items. Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), pp. 3553-2560. IEEE, 2022.
G. Franzese, C. Celemin, and J. Kober. Learning interactively to resolve ambiguity in reference frame selection. In Proc. of the Conf. on Robot Learning (CoItL). PMER, 2020.
Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proc. of the Intl. Conf. on Machine Learning (ICML), pp. 1050-1059. PMLR, 2016.
Y. Geng, B. An, H. Geng, Y. Chen, Y. Yang, and H. Dong, Rlafford: End-to-end affordance learning for robolic manipulation. In Proc. of the IEEE Inll. Conf. on Robolics & Aulomation (ICRA), pp. 5880- IEEE, 2023.
C. Gokmen, D. Ho, and M. Khansari. Asking for help: Failure prediction in behavioral cloning through value approximation. In Proc. of the IEEE Inll. Conf. on Robotics & Aulomation (ICRA), pp. 5821-5828, 2023.
D. H. Grollman and O. C. Jenkins. Dogged learning for robots. In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), pp. 9483-2488 IEEE, 2007.
R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg. ThriltyDgger.: Budget- aware novelty and risk gating for interactive imitation learning. In Proc. of the Conf. on Robot Learning (CoRL), volume 164, pp. 598 608. PMLR, 2022.
R. Hoque, L. Y. Chen, S. Sharma, K. Dharmarajan, B. Thananjeyan, P. Abbeel, and K. Goldberg. Fleet- DAgger: Interactive robot floct learning with scalable human supervision. In Proc. of the Conf. on Robot Learning (CoRI.), pp. 368 380. PMLR, 2023.
E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. BC-Z: Zcro-shot task generalization with robotic imitation learning. In Proc. of the Conf. on Robot Learning (CoRL), pp. 991 1002. PMLR, 2022.
18

---

Published in Transactions Machine Learning Research (08/2025)
E. Johns. Back to reality for imitation learning. In Proc. of the Conf. on Robot Learning (CoRL), pp. 1764 1768. PMLR, 2022.
G. Kahn, P. Abbeel, and S. Levine. LaND: Learning to navigate from disengagements. IEEE Robolics and Automation Letters (RA-L), 6(2):1872 1879, 2021.
T. Kaufmann, P. Weng, V. Bengs, and Hüllermeier. A survey reinforcement learning from human loodback. Trans. Machine Learning Research, 2025. ISSN 2835-8856.
M. Kelly, C. Sidrane, K.
and M. J. Kochenderfer. HG-DAgger Interactive imitation learning with human experts. In Proc. of the IEEE Inll. Conf. on Robolics & Aulomation (ICRA), pp- 8077 8083, 2019.
B. Kim and J. Pineau. Maximum mean discrepancy imitation learning. In Proc. of Robolics: Science and Systems (RSS), 2013.
A. Kumar, J. Hong, A. Singh, and S. Levine. Should run offine reinforcement learning or behavioral cloning? In Proc. of the Bull. Conf. on Learning Representations (ICLR), 2022.
A. Lafage and O. Laurent. TorchUncertainty github. htt.://0000000020///////r--aaaaa
2024.
M. Laskey, S. Staszak, W. Y.-S. Hsieh, J. Mahler, F. T. Pokorny, A. D. Dragan, and K. Goldberg. SHIV: Reducing supervisor burden in dagger using support vectors for efficient. learning from remonstrations in high dimensional state spaces. In Proc. of the IEEE Intl. Conf. on Robotics Automation (ICRA), pp. 462 469. IEEE, 2016.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. tradient-based learning applied to document recognition. Proc. of the IEEE, 86(11):22288 2324, 1998.
S.-W. Lee and Y.-L. Kuo. Diff-DAgger: Uncertainty estimationn with diffusion policy for robotic manipulation. arXiv preprint, 2024.
A. Li and T. Silver. Embodied active learning of relational state rastioctions for bilevel planning. In Proc. of the Conf. on Lifelong Learning Agents (CoLLAs), pp. 358-375. PMLR, 2023.
Y. Li and C. Zhang. Ensemble-based interactive imitation learning. Computing Research Repository (CoRR), 2023.
D. Lindner, M. Turchetta, S. Tschiatschek, Ciosck, and A. Krause, Information directed reward learning for reinforcement learning. In Proc. of the Advances in Neural Information. Processing Systems (NewrIPS), volume 34, pp.
2021.
H. Liu, S. Dass, R. Martin-Martin, and Y. Zhu. Model-based runtime monitoring with interactive imitation learning. In Proc. of the IEEE Inil. Conf. on Robotics & Automation (ICRA), 2024.
J. Luijkx, Z. Ajanović, L. Ferranti, and J. Kober. PARTNR: Pick and place ambiguity resolving by trust- worthy interactive learning. In NewrIPS Rohot Learning Workshop: Trustworthy Itbootcs, 2022.
Luo, P. Dong, Y. Zhai, Y. Ma, and S. Levine. RLIF: Interactive imitation learning as reinforcement learning. In Proc. of the Inll. Conf. Learning Representations (ICLR), 2024.
P. Mazzaglia, T. Cohen, and D. Dijkman. Information-driven affordance discovery for efficient robotic manipulation. In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), pp. 7780-7787. IEEE, 2024.
K. Menda, K. Driggs-Camplul and M. Kochenderfer. DrogoutDAggor: A bayesian approach safe imita- tion learning. ar Xiv preprint, 2017.
K. Menda, K. Driggs-Caan Campbell, and M. J. Kochenderfer. EnsembleDAgger: A bayesian approach to safe imitation learning. In Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pp. 5041 5048, 2019.
19

---

Published in Transactions on Machine Learning Research (08/2025)
K. Mo, L. Guibas, M. Mukadam, A. Gupta, and Tulsiani. Where2Act: From pixels to actions for articulated 3D objects. In Proc. of the IEEE Intl. Conf. Computer Vision (ICCV), pp- 6813 6823, 2021.
C. Ning, R. Wu, H. Lu, K. Mo, and H. Dong. Where2Explore: Few-shot affordance learning for unseen novel categories of articulated objects. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS), pp. 4585 4596, 2023.
Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y.-L. Tan, L. Y. Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. Proc. of Robolies: Science and Systems (RSS), 2024.
D. A. Pomerleau. ALVINN: An autonomous land vehicle in a neural network. Proc. of the Advonces in Neural Information Processing Systems (NewrIPS), 1, 1988.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transfcrable visual models from natural language supervision. In Proc. of the Intl. Conf. on Machine Learning (ICML), pp. 8748 8763. PMLR, 2021.
S. Reed, K. Zolna, E. Parisolto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint, 2022.
A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. In Proc. of the Conf. on Robot Lenrning (CoRL), pp. 661 682. PMIR, 2023.
Ross, G. Gordon, and D. Bagnell. A reduction imitation learning and structured prodiction no-regret online learning. In Proc. of the Intl. Conf. Artificial Intelligence and Statistics (AISTATS), volume 15, pp. 627-635. PMLR, 2011.
T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. orXie preprint, 2015.
B. Settles. Active learning litcrature survey. Computer Seiences Technical Report 1648, University of Wisconsin Madison, 2009.
M. Shridhar, h. Manuelli, and D). Fox. CLiPort: What and where pathways for robotic manipulation. In Proc. of the Conf. on Robot Learning (CoRL). PMLR, 2021.
Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang, P. Ramadge, and S. Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. In Proc. of Robotics: Science and Systems (RSS), 2020.
B. van der Heijden, J. Luijkx, L. Ferranti, Kober, and R. Babuska. Engine Agnostic Graph Environments for Robotics (EAGERx): A graph-based framework for sim2real robot learning. IEEE Robotics ond Automation Magazine (RAM), pp. 2-15, 2024.
Y. Wang, R. WI, K. Mo, J. Ke, Q. Fan, I. J. Guibas, and H. Dong. AdaAfford: Learning adapt manip- ulation affordance for 3D articulated objects via sew-shot interactions. In Proc. of the Europ. Conf. on Computer Vision (ECCV), pp. 90 107. Springer, 2022.
J. Wong, A. Tung. A. Kurenkov, A. Mandlekar, L. Fei-Fei, S. Savarese, and R. Martin-Martin. Error- aware imitation learning from teleoperation data for mobile manipulation. In Proc. of the Conf. on Robot Learning (CoRL). PMIRR, 2021.
C. Wu. SLIPort-Batchif github. cwm ctmm//Chenwuss//ppprttbbaaa
2024.
A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, 1. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Proc. of the Conf. on Robot Learning (CoRLL), pp. 726-747. PMLR, 2021.
Zhang and K. Cho. Query-efficict imitation learning for end-to-end simulated driving. In Proc. of the Conference on Advancements of Artificial Intelligence (AAAI), 2017.
20

---

Published in Transactions Machine Learning Research (08/2 (08/2025)
A Appendix
A.1 Specificit--Aware Gating
For Speeilicit--Aware Gating, the threshold in line 12 of Alg. set follows. The total specificity afpec can be computed by considering the true negatives TN, resulting from active queries, the false positives Fraadd resulting from random queries, along with the false positives FPy from active querying.
spec TN, FPrand rand
(7)
TN, + FP,
The number of rand are determined by which of TN,, are queried. These queries occur with probability Prand (see lines 8-9 of Alg. []. Therefore, its expected value is TN, Prand- This leads to:
TN,
TN, Prand
Be-Uoo. [o TN, FP, TN, FP,
Sxper (1 Prand),
(9)
where appec is the specificity for gating disregarding the random queries. Thus, we interpolate between threshold values that best satisfy the desired specificity level ddes = oppec(1 Prand) (line 12 of Alg. 2
A.2 Success-Awar Aware Gating
The goal of Success-Award Gating (SAG) is minimize expert queries while maintaining minimum system success rate. If the novice's success rate is below the desired success rate ddes, SAG increases expert queries to ensure that the combined success rate of expert and novice moets dee- We assume that expert actions are always correct for this mode. If the novice's success rate is already at or above des: SAG does not actively query the expert to avoid redundant queries. The only case when the system fails is when the novice's action invalid, while the expert not actively queried and not randomly queried. Therefore, the system success rate is:
FN, TP rand succ =1 TP, + FP, + TN, + FNN,
(10)
The number of TP rand are determined by which of FN, queried. These queries occur with probability Prand (see lines 8-9 of Alg. T Therefore, its expected value is PN, Prand- This leads to:
FN,
FN. Prand
Eu-Uja l0 Auco
= TP, + FP, + IN, FN, + TP, + FP, + TN, +NN, suce + Prana (1 - o000))
(11)
0
(12)
where Jeuce is the system success rate disregarding the random queries, Thus, we interpolate between
Uhreshold values that best satisly the desired system success rate ddes ou + Prana succ)) (line 12
Alg. 2
A.3 SAG Ablations
The first step linear regression normalize uw, and the second is logistic regression to impute pseudo labels for cases without teacher feedback (r= 0). To evaluate the impact of these steps, we performed ablations in the same experimental setup as described in Sec. 5.1 (mode sensitivity). We conducted dataset aggregation on the MNIST handwritten dataset under two conditions: one without normalization of uw (w/o Normalization) and one without pseudo-label imputation (w/o Imputation). Fig. 12 and Tab. compare these results to SAG with both normalization and imputation.
SAG involves two regression steps, as shown in Alg. 2 and Fig. D
The sensitivity plots and table show a clear performance dogradation for SAG w/o Imputation. Without imputing pseudo labels where success is unknown, the threshold set too high, as gating results in labels

---

Published in Transactions Machine Learning Research (08/200 (08/2025)
SAG
SAG w/o Imputation
SAG w/o Normalization
1.0
1.0
1.0
0.8
0.8
0.8
Senslfiviyy
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
100
200
300
400
100
200
300
400
100
200
300
400
A
C
Step
Step
Step
Ode 0.1
dde.
0.3
ddea 0.5
dde 0.7
Odea 0.9
dda
0.2
dde. 0.4
ddes = 0.6
des 0.8
Desired
Figure 12: Comparison of SAG, SAG without pseudo-label imputation (w/o Imputation), and SAG without uncer- lainly normalization (w/o Normalization). Seusitivity is calculated over moving window of 1000 failures. Mean and standard deviation
shown for ten repetitions. 'Tee plots show that SAG w/o Imputation sets the threshold too high, as it primarily uses labels from high-uncertaing, regions. SAG w/o Normalization performs poorly in the early training stages due to rapidly decreasing uncertainty, which also results in overly high threshold. A threshold that too high will result too low sensitivity, as not enough failures prevented through querying.
Table 1: Comparison of SAG, SAG without pseudo-label imputation (w/o Imputation), and SAG without uncertainly normalization (w/o Normalization). Sensitivity calculated over the complete training duration. Mean and standard deviation are shown for ten repetitions. Boldface is used to highlight the values corresponding to the best sensitivity tracking.
dees
SAG
SAG w/o Imputation
SAG w/o Normalization
0.1
0.104 # 0.003
0,114 = 0.003
0.103 + 0.002
0.2
0.207 + 0.005
0.141 + 0.005
0.190 + 0.004
0.3
0.304 + 0.009
0.171 = O. 0.003
0.272 + 0.006
0.4
0.397 + 0.007
0.200 = 0.005
0.365 + 0.010
0.5
0.497 + 0.008
0.236 = 0.005
0.448 + 0.010
0.6
0.598 + 0.007
0.283 + 0.009
0.548 + 0.012
0.7
0.695 + 0.005
0.350 = 0.009
0.655 + 0.013
0.8
0.793 + 0.010
0.447 + 0.008
0.764 + 0.005
0.9
0.899 + 0.004
0.628 + 0.008
0.850 + 0.007
for only the high-uncertainty regions. This lcads to overly low sensitivity values. Tab. D also indicates performance degradation when uw is not normalized. Fig. 12 suggests that this primarily results from poor sensitivity tracking in the early training stages. Uncertainty drops rapidly during early training. Therefore, failing to normalize uw causes the threshold to be set too high. As a result, sensitivity remains too low in these early slages.
We also performed an ablation to study the influence of the value of the random query probability Prand- We compared the experiments from Sec. 5_ with Prand 0.1 training with Prand = 0.05 and Prand 0.2. The results of this comparison are shown in Fig. 13 Tab. 2 and Tab. 3 The scnsitivity plots in Fig. 13 A-C
show that SAG can track multiple sensitivity lovels for different values Prand However, for Prand = 0.2 and
Jdes 0.1 it fails. This results from SAG being unable track sensitivities lower than Prand- This follows
22

---

Published in Transactions Machine Learning Research (08/2025)
from Eq. 3
BenUUo,,1 [0
= o + Prandd (1 o
(13)
2 Prand:
(14)
since o7 and Prand S 1. The specificity plots in Fig. 13 D-F show that increasing Prand results
in lower specificity values. This is also reflected in Tab. 3 This table shows the informedness values for the different values Prand- The informedness is also known as the Youden's statistic and calculated as:
informedness sensitivity specificity 1.
(15)
quantifies the performance of a dichotomous diagnostic test:
below zero means worse than random;
equals random performance;
bove indicates some degree of informed decision-making.
In Tab.
that increasing Prand results in lower informedness, higher percentage of queries results from random quorying, rather than from qucrying based on excceding the uncortainty threshold.
SAG Prand
SAG Prand
SAG Prand=O4
1.0
1.0
1.0
0.8
0.8
0.8
Sensifvity
0.4
0.4
0.4
0.2
0.2
0.2
100
200
300
400
100
200
300
400
100
200
300
A
C
1.0
0.8
0.8
0.8
Specificity
0.6
0.6
0.6
0.2
0.2
0.0
0.0
0.0
0
100
200
300
400
100
200
300
400
0
100
200
300
400
D
F
Step
Step
Step
die 0.1
Odea 0.3
Dioc 0.5
Gdea
Gáe 0.9
die 0.2
Gdea = 0.4
aáaa 0.6
Odea 0.8
Desired (A-C)
Figure 13: Comparison of SAG with different values of Prand (0.05 shown, 0.1 and 0.2). Sensitivity in A-C and specificity in E-F caleulated standard deviation are shown for ten repetitions. SAG can track desired sensitivity for different values Prand However, fails for daes 0.1 when Prand 0.2. This happens because the minimum trackable sensitivity is equal to Prand- The plots E-F show that increasing Prand leads to lower specificity values.
moving window of 1000 failures aud successes, respectively. Mean and

---

Published in Transactions Machine Learning Research (08/201 (0822025
Table 2: Comparison of SAG with different values of Prand- Sensitivity is calculated over the complete training duration. Mean and standard deviation shown for ten repetitions. Boldface is used highlight the valies corresponding to the best sensilivity tracking.
ddes
sensitivity Prand 0.05
sensitivity Prand 0.1
sensitivity Prand 0.2
0.1
0.113 0.006
0.104 0.003
0.200 0.003
0.2
0.198 + 0.012
0.207 + 0.005
0.202 + 0.003
0.3
0.298 + 0.010
0.304 + 0.009
0.309 = 0.005
0.4
0.399 0.006
0.397 + 0.007
0.406 0.007
0.5
0.493 + 0.009
0.497 + 0.008
0.507 = 0.007
0.6
0.591 + 0.006
0.598 = 0.007
0.599 + 0.005
0.7
0.696 0.010
0.695 + 0.005
0.699 + 0.005
0.8
0.799 + 0.010
0.793 + 0.010
0.799 + 0.006
0.9
0.897 0.003
0.899 = 0.004
0.900 + 0.006
Table 3: Comparison of SAG with different values of Pran- Informeduess (Youden's statistic sensilivity + specificity 1) is calculated over the complete training duration. Mean and standard deviation are shown for ten repetitions. Boldface is used highlight the values corresponding the best informedness.
informedness Prand 0.05
informedness Prand 0.1 informedness Prand 0.2
0.1
0.055 + 0.006
0.003 = 0.004
-0.000 + 0.004
0.2
0.126 + 0.011
0.092 + 0.004
0.002 + 0.004
0.3
0.209 + 0.012
0.170 0.010
0.094 + 0.007
0.4
0.284 + 0.007
0.245 + 0.007
0.171 = 0.010
0.5
0.349 + 0.013
0.311 + 0.015
0.249 = 0.010
0.6
0.410 + 0.010
0.375 + 0.013
0.308 + 0.010
0.7
0.464 + 0.011
0.427 = 0.014
0.365 0.011
0.8
0.483 + 0.025
0.460 0.019
0.407 + 0.015
0.9
0.456 + 0.027
0.445 0.014
0.403
24

---

Published in Transactions Machine Learning Research (08/ (08/2025)
A.4 Prioritized Interactive Experience Replay (PIER) Intuition
Prioritized Interactive Experience Replay (PIER) prioritizes the replay of demonstrations based novice success, uncertainly, and demonstration age. In Fig. [4] we visualize Eq. (6) to provide a more intuitive understanding of the prioritization scheme. The red lines correspond novice failures (r =-1, while the green lines correspond to novice successes (r 1). The black lines indicate r 0, which occurs, for example, for relabeled demonstrations where the goal was relabeled, and we do not know whether the novice would correctly. In this figure, observe that for values of b close 1, the priorities converge almost. linearly to 1, while for higher values of b, they converge exponentially.
2.00
1.75
1.01
1.50
b-10
1.25
1000
Priorities
1.00
1-1000
0.75
0.50
b=1.001
0.25
0.0
0.2
0.4
0.6
0.8
1.0
Prioritization Exponent
Figure 14: Visualization of Eq.
6
showing priorities p prioritization exponents for various values of hase b.
In summary, prioritize demonstrations in the following order (from highest to lowest):
Novice failure demonstrations that are recent and have low uncertainty.
Novice failure demonstrations that are either recent or have low uncertainty.
Novice failure demonstrations that are neither recent nor have low uncertainty.
Offline or relabeled demonstrations.
Novice success demonstrations that neither recent nor have low uncertainty.
Novice success demonstrations that are either recent or have low uncertainty.
Novice success demonstrations that both recent and have low uncertainty.
A.5 Foresight Interactive Experience Replay (FIER) Oracle
The oracle used in the experiments from Sec. 5.2 is based on (Shridhar al. (2021). We extend this oracle to provide relabeling demonstrations. Instead of qucrying human teacher about the validity of the novice's s planned actions, we exccute these actions in simulation to verify the plan and then reset the environment to its previous state.

---

Published in Transactions on Machine Learning Research (08/2025)
A.6 Considerations on Demonstration Types, Times, and Effort
The ASKDAGGER framework supports data collection through three modalities: annotation, relabeling, and validation. Each modality poses distinct challenges and places different. demands on the user in terms of time and physical or cognitive effort. As in other imitatioo learning approaches in robotics, time efficiency is a more meaningful metric than data efficiency for evaluating real-world robot learning performance (Johns) 2022). A large portion of the overall data collection time is spent on environment setup, action exccution, and resetting. We illustrate this with quantitative results from the experiments in Sec. 5.3 All reported timings are approximate and based on a 45-minute video recording of one experiment.
We measured the time required to provide each demonstration type in the assembly task. On average, validation look 3.7 + 1.9 seconds, relabeling 7.5 + 1.9 seconds, and annolation 7.0 + 3.8 seconds. Based on these values, one might conclude that relabeling is the most time-consuming, while validation is the fastest. However, following Johns (2022), time-efficiency analysis should also account for the full process involved in producing a demonstration, including environment resetting, issuing robot commands, model inference, and action execution. In our setup: Resetting the environment took 8 seconds and was required every four actions, averaging 2 seconds per action. Command generation was automated and took about 1 second to execute. CLIPort model inference (including sensor processing, preprocessing, and interface delays) took roughly 5 seconds. Action execution look approximately 25 seconds. Accounting for these overheads, validation demonstration takes about 37 seconds in total, and an annotation demonstration takes about 40 seconds. In contrast, a relabeling demonstration can be provided when prompted during an annotation query and does not require additional resets, executions, or inference. Thus, its effective cost remains close to 8 seconds, making it the most time-efficient modality overall. Morcover, since relabcled demonstrations are not executed, relabeling does not block robot execution and can be done in parallel. We emphasize that these limings are highly task- and interface-eppendent and are provided here as illustrative examples. Furthermore, time is only one aspect of demonstration cost; cognitive load and mental effort of the human demonstrator are also important factors.
As noted, the cost (teaching effort) of each demonstration type is task- and interfaco-depeudent Nonotheless, we can consider different scemarios for the experiments described in Sec. [5.2] and evaluate them. We discuss three such scenarios (these are not exhaustive and may differ in likelihood):
Sconario 1: Annotation demonstrations dominate the demonstration cost.
Scenario 2: Annotation and validation demonstrations are equally costly and dominate the overall cost.
Scenario 3: All demonstration types are equally costly.
Results for these scenarios are shown in Fig. 15 In Scenario 1, ASKER shows a clear advantage over other methods, achieving higher success rates during training for the same number of annotation demonstra- lions. In Scenario performance differences during training still outperforms other methods on the unseen tasks, as shown in Fig. 5 In Scenario 3, where all demon- stration types incur the same cost, ASKDAGGER has the highest total demonstration cost during training. Nonethcless, since it still generalizes better to unscen tasks, whether to collect relabeling demonstrations becomes a trade-off between generalization performance and demonstration cost.
less pronounced. However, ASKDRGGER
A.7 CLiPort Implementation
Our implementation of CLIPort agents builds on Shridhar et al. (2021) with modifications for interactive imitation learning. Efficient model updates are essential this setting, as saving multiple checkpoints and performing evaluation rollouts after each update is impractical. Thus, training stability is a priority. We replace rectified linear units (ReLUs) with leaky ReLUs to prevent vanishing gradients. We also use a larger batch size (8 in simulation, in real-world experiments) than the original (1) (Shridhar et al. 2021 Our batch training implementation is based on Wu (2024). Since batch training increases memory requirements, we reduce the model size by removing the two middle layers from the ResNet streams with lateral connections, the first language fusion layer, and its associated upscaling and lateral fusion layers.
26

---

Published in Transactions Machine Learning Research (08/20 (08/2025)
packing-googe objects-seq
packing-gooss objects-group
put-block in-bowl
packing-shapes
Scenario 1 Novice Success Rate
1.0
0.75
0.75
0.75
0.50
0.50
0.50
0.5
0.25
0.25
0.25
0.00
0.00
0.00
0.0
0
50
100
0
50
100
C
0
25
50
D
25
50
Ann. Tuples
Ann. Tuples
Ann. Tuples
Ann. Tuples
Novice Success Rate
1.0
Scenario 2
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0:25
0.00
0.00
0.00
0.0
E
0
200
F
100
200
G
50
100
H
100
200
Ann. + Val. Tuples
Ann. Val. Tuples
Ann. Val. Tuples
Ann. Val. Tuples
Novice Success Rate
1.0
Scenario 3
0.75
0.75
0.75
0.50
0.50
0.50
0.5
0.25
0.25
0.25
0.00
0.00
0.00
0.0
200
400
J
200
400
K
100
200
L
0
100
Ann. Rel. + Val. Tuples
Ann. + Rel. + Val. Tuples
Ann. + Rel. + Val. Tuples
Ann. + Rel. Val. Tuples
ActiveDAgger
ThrityDAgger
SafeDAgger
ASkDAgger
Figure 15: Novice success rate during training as function of the number of collected annotation tuples (top row), annotation + validation tuples (middle row), and annotation + validation + relabeling tuples (bottoom row). Mean and standard deviation are computed over ten policies using moving window of 50 episodes. The top shows (scenario 1), ASKDAGGER achieves a
that when annotation demonstrations clear performance gain on both seen and unseen tasks, benefiting from validation and relabeling demonstrations, The middle row shows that when annotation and validation demonstrations are equally costly and dominate the total (scenario 2), all methods perform similarly relative to demonstration cost, with ASKDAGGER offering a performanc gain only on unsen lasks. Finally, the boltom row shows that when all demonstration types are equally costly (scenario 3), ASKDAGGER performss better wsseen tasks but requires more costly training on seen tasks.
the dominant demonstration
A.8 CLiPort Tasks
The CLIPort (Shridhar et al. 2021 benchmark defines seen and unseen settings. In the see setting, mands are identical to those used during training. In the unseen setting, lest-time commands involve different objects, shapes, or colors. We use the same seen and unseen sets as in the experiments by [Shrdhar et al. (2021). The only difference is that, for tasks involving shapes and Google objects, include shapes and objects from the unseen set as distractors during training. Tab. provides a complete overview of the seen and unseen
A.9 THRIFTYDAGGER Implementation
Similar to Hoque al. 2022), , gating is based on novelty and riskiness in our implementations of ThriftyDAgger. For our novelty measure, take the mean entropy of the pixel-wisc value estimates over 5 dropout ovaluations with a dropout rate of 0.3. We define riskiness as one minus the maximum value of the pixel-wise value estimates during evaluation without dropouts.

---

Published in Transactions Machine Learning Research (08/2022 (08/2025)
Table 4: Specifications of the seen and unseen sets in the CLIPort benchmark tasks
Shridhar et al. 2021
Shapes
from Zeng al. (0021) and Google scanned objects are from Downs al. 2022
Seen
Unseen
All
Colors
cyan, yellow, brown, gray
pink, orange, purple, white
green, red, blue
Shapes
letter R shape, letter A shape, triangle, square, plus, letter ' shape, diamond, pentagon, rectangle, flower, star, circle, letter G shape, letter V shape
letter E shape, letter I. shape, ring, hexagon, heart, letter M shape
Google Scanned
alarm clock, android toy, black boot with leopard print, black fedora, black razer mouse, black sandal, black shoe with orange stripes, bull ligure, butterfinger chocolate, clamp, opener, crayon box, dog statue, frypan, green and white striped towel, grey soccer shoe with cleats, hard drive, honey dipper, magnifying glass, mario figure, nintendo 3ds, nintendo cartridge, office depot box, orea plush toy, pepsi gold caffeine free box, pepsi wild cherry box, porcelain cup, purple tape, red and white flashlight, rhino figure, rocket racoon figure, scissors, silver tape, spatula with purple head, spiderman figure, tablet, toy school bus
ball puzzle, black and blue sneakers, black shoe with green stripes, brown fedora, dinosaur figure, hammer, light. brown boot with golden laces, lion figure, pepsi max box, pepsi next box, porcelain salad plate, porcelain spoon, red and white striped towel, red cup, screwdriver, toy train, unicorn toy, white razer mouse, yoshi figure
Objects
A.10 SAFEDAGGER Implementation
In our experiments, SafeDAgger (Zhang & Cho 2017 is implemeentd follows. In the original formulation of SafeDAgger, a safety classifier is learned that takes as imput the observation and novice policy and outputs binary prediction whether the novice will deviate from the teacher policy. Later, more general definitions of SafcDAgger have been introduced, c.g., by Menda et al. (2019), who formalize SafcDAgger as decision rule based on a discrepancy measure between novice and teacher actions. In our implementation, the decision rule basod on the Q-value of the pixel-wise value estimates of the CLIPort model, since [Zhang & Cho (017)) describe that the safety policy is similar to value function in the case of sparse rewards.
A.11 CLIPort Ablation
The comparison of ASKDAGcER with the baselines in Fig, showws clear performance improvement on the unseen tasks. We attribute this gain to the recasting of failures as demonstrations, enabled by FIER (claim C3). To support this explanation, we conducted an ablation study where ASKDAGGER is run without relabeling (ASKI)AGEER w/o relabeling), the only change being the omission of relabeling. The results,
28

---

Published in Transactions Machine Learning Research (08/2025)
shown Fig. 16 confirm that the performance improvement on unseen tasks indeed due relabeling, thus supporting claim C3. With relabeling, ASKDAGGER achieves average evaluation reward improvement of 62% across the four unscen tasks.
Breking-seen-ggogle objects-seq
packing-seen-ggoolee objects-group
put-block-in bowi-seen-colors
packing-seen-shapes
100
80
Saen: Raward
75
60
60
60
40
40
40
25
A
100
200
300
B
100
200
300
100
200
300
D
100
200
300
packing-unss packing-unseen-googlee objects-seq
put-block-in
objects-group
packing-unsse suapiigg-nneee-spppes
bawl-unseen-color
60
Unseen: Reward
40
40
20
20
20
20
E
100
200
300
H
Demonstrations
Demonstrations
Demonstrations
Demonstrations
ASkDAgger w/o relabel
ASkDAgger
Figure 16: Cumulative rewards evaluating checkpoints over 100 episodes on tasks with seen and unseen objects. Mean and standard deviation are shown for len policius using moving window of 50 episodes. The results show that relabeling demonstrations improve the performance of ASkDAGGER the scenarios.
A.12 Comparison between ASkDAgger and Affordance Learning with Where2Act
In this section, we highlight the difference between ASKDAGGER, data aggregation strategy, and affordance learning methods (Mo et al.) 2021 Mazzaglia et al. 2024 Wang el al., 2022 Ning el al. 2023 Geng el al. 2023), which are designed to efficiently explore the large state-action space of affordances. ASKDAGGER collects on-policy, goal-conditioned data and is tailored to perform well on the distribution of states and goals encountered by the novice policy. In contrast, affordance learning methods such as Where2Act (Mo) al. 2021h aim not to provide accurate predictions for specific but to cover broad range of the state-action space and typically require constant reward information. In short. ASKDAGGER optimizes for on-policy performance, while affordance learning methods like Where2Act prioritize coverage and generalization. To illustrate these differing objectives, compare ASkDAGGER with Where2Act on a task where we finctune Where2Act model for the skill pushing, trained on the 10 training categorics from Mo al. (2021), the single category StoragePurniture (sce Fig. 18 For comparison, AGGDAGGER uses the same model architecture Where2Act, and we do not collect expert demonstrations of successful poses or relabeling demonstrations. As we use the Where2Act model for the novice policy, also learn from failures and aggregate those into the dataset. well. Its policy is based the prediction-bias biased adaptive data sampling strategy from Where2Act.
distribution induced
policy,
Therefore, the main differences between the two methods in this setting are: ASkDACGER uses the PIER replay prioritization method (while Where2Act samples successes and failures with equal probability), ASKDAGGER includes the SAG gating mechanism, and Where2Act explores using random actions 50% of the time.

---

Published in Transactions Machine Learning Research (08/2025)
We used the following scttings and parameters. For both methods, we collect data from the same mum- ber of interactions (800) with the same update-to-data ratio (4 update steps every 16 interactions). For ASKDAGGER, we use the scnsitivity mode, with ddes = -.9. For ASKDAGGER, we quantify the uncer- lainty u based
least. confidence approach (Settles 2009 based on the state-action value function Q, i.e. = [1{((o, a) > 0.5] Q(o, a)). For PIER, we used a = 0.5, X = 0.5, b = 10, and = 0. Please note that these parameters were not tuned.
The results for these experiments are shown in Fig. 17 These plots clearly show the different objectives of ASKDAGGER and Where2Aet. While ASKDAGGER performs better on-policy and on the seen shapes, Where2Act explores and performs better the unseen shapes.
Training Sample Success Rate
Training F-Score
Evaluation Sample Success Rate
Evaluation F-Score
Sample Success Rate
Sample Success Rate
0.75
0.6
0.6
0.4
F-Score
0.4
F-Score
0.50
0.4
0.2
0.2
0.25
0.2
0.0
A
20
B0
20
Seen
Unseen
Seen
Unseen
Updates
Updates
Where2Act
ASKDAgger
Figure 17: Training and evaluation sample success rates and F-scores (balance between precision and recall) are shown for finetuiig single object calegory EStorageFurrittrry for pushing skill. We also show evaluation results on an Storage/urmddureele shapes. Mean and slandard deviation shown for live repetitions. Random actions from Where2Act are excluded when caleulating sample success rates and F-scores during training. For comparison, AGEER was trained without annotation or relabeled demonstrations. ASKDAGGER is designed for on-policy data collection and performs better the designed to efficiently explore the state-action space, leading to better generalization anseen shapes.
of shapes. On the other hand, Where2Act (Mo et al. 2021
AGKERGGER
Where2Act
Figure 18: Visualization of value maps and high-value proposals from ASKDAGGER (left) and Where2Act MM et al. 2021) (right). Results are shown after finetuning on single object category for pushing skill and evaluating on an unseen shape from that category. ASKDAGGER is designed for on-policy data collection and performs better during policy execution. On the other hand, Where2Act (Mo et al. 2021) performs active exploration, leading to better gizatinization unseen shapes, as qualitatively shown in the higure.
A.13 Compute Resources
The MNIST experiments (Sec. 5.1 and real-world experiments (subsection 5.3 and subsection 5.4] were performed using an RTX 3080 Mobile graphics card. The CLIPort simulation benchmark experiments (Sec. 5.2 were performed using multiple A40 graphics cards high-performance computing cluster.


Reviews:
[{"review_id": "xQMMyTODTi", "reviewer": "Reviewer_C8RH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n* **Clear writing**: the paper clearly contextualizes the problem and states the claims to be verified in the Introduction. Then, it proceeds to effectively present the problem settings and method. Finally, it presents the experimental results while recalling the claims from the introduction. Overall, the flow is linear and the work is easy to follow.\n* **Novelty**: FIER and PIER techniques are heavily inspired by techniques often applied in Reinforcement Learning (HER [2] and PER [3], respectively). However, their application is less common in the IIL context and the authors fit them elegantly within their framework. The idea behind SAG also contains novelty, despite uncertainty quantification methods being common for active learning in robotics.\n* **Open**: the authors released code and assets for their work, demonstrating high-reproducibility and transparency in their work\n\n## Weaknesses:\n* **Complexity**: the introduced components make the method more complex and relying on different design choices, such as the method to estimate uncertainty in SAG, and additional hyperparameters for SAG and PIER. E.g. the probability of randomly sampling a query and the desired sensitivity in SAG, sampling hyperparameters in PIER. Nonetheless, the authors already ablate most of these components and provide at least an explanation on how they should be selected.\n* **Demonstrator's effort**: it is not clear whether the labelling methods introduced in FIER reduce the workload of the demonstrator. The authors only compare the number of annotations, e.g. in Figure 6 and 11. However, it would be advisable to quantify if there's any time or effort saved by reducing the number of annotations, in favor of relabeling and validation.\n* **Evaluation focussed on CLIPort**: given the multiple robotics experiments presented, I would have expected at least one of them to not be heavily relying on CLIPort for policy learning. See proposed changes for further details.\n\n\n[2] Hindsight Experience Replay, Andrychowicz et al, 2017\n\n[3] Prioritized Experience Replay, Schaul et al, 2015", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work presents AIDA (Action Inquiry DAgger), a framework for interactive imitation learning (IIL).\nAIDA aims to reduce human effort and improve performance in IIL by introducing three major distinctive features:\n* SAG (Sensitivity-Aware Gating): which tracks a sensitivity level to decide when it's necessary to query the demonstrator(s) for information;\n* FIER (Foresight Interactive Experience Replay): expands the demonstrator-provided information with trajectory relabeling and validation of the action plans from the IIL novice;\n* PIER (Prioritized Interactive Experience Replay): biases the sampling process from the replay buffer based on uncertainty, novice success and demonstration age.\nAIDA is evaluated in toy settings, and simulated and real-world high-level actions robotic setups, using CLIPort [1].\n\n[1] CLIPort: What and Where Pathways for Robotic Manipulation, Shridhar et al, 2021", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Major\n* **Details about the relabeling and validation time and effort**: see related Weakness\n* **Limited applicability?**: it is unclear to me why the method is not applicable in higher-frequency control problems. While it is generally more convenient to provide a feedback or an annotation on higher-level problem, it should be possible to apply AIDA also on more complex low-level tasks. For instance, modern teleoperation frameworks allow humans to intervene and provide demonstrations when the agent requires it. \n* **Comparison with affordance-based / reinforcement learning methods**: with the addition of SAG and FIER, the method resembles affordance-based methods using online learning techniques, such as the adaptive sampling in [5] and the information-driven sampling in [6]. Compared to IIL methods, these approaches learn better from failures, relying on supervised or reinforcement learning. A comparison with such methods could be a nice way to expand the paper beyond the CLIPort policy learning setup. Alternatively, I would recommend the authors to discuss more in details what differentiates their method from affordance supervised learning / reinforcement learning / goal-conditioned reinforcement learning techniques.\n### Minor\n* It would be interesting to see what's the performance of a Dagger (or Active Dagger) approach when the number of annotations is similar to the sum of (annotations + relabeled + validation) for AIDA\n* Figure 1 is quite generic and not particularly helpful to understand the method better at the moment. It would be best if they authors provided a more detailed illustration of the method(s) presented\n\n[4] Teleoperation with Immersive Active Visual Feedback, Cheng et al, 2024\n\n[5] Where2Act: From Pixels to Actions for Articulated 3D Objects, Mo et al, 2021\n\n[6] Information-driven Affordance Discovery for Efficient Robotic Manipulation, Mazzaglia et al, 2024", "broader_impact_concerns": "The work presents no ethical risks. Potential misuse of robotic systems in real-world can be unsafe, if no safeguard mechanisms are applied to the system and the hardware. Adding a short statement in the paper would be appropriate.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZyOx8FOcyx", "reviewer": "Reviewer_Uc41", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. I find the paper well-written and easy to follow. The authors made it very clear in the introduction what the scope and key contributions of this paper are.\n2. The paper shows strong results in support of its claims. The MNIST experiment is specifically designed to test whether SAG can accurately track and maintain a desired sensitivity level over time. CLIPort tasks involve language-conditioned, vision-based robotic manipulation representing complex, real-world challenges such as manipulation involving unseen objects.\n\n**Weaknesses:**\n1. For a large portion of this paper, it is unclear what is the application space is for this paper esp. in robotics. While the authors eventually make it clear that their action space is the space of mid-to-high level actions, one could be confused if this method was trying to improve an end-to-end visuomotor policy, which it isn’t. I think that point can be made clearer in the introduction or in Section 3 what the exact output/action spaces are in the experiments. \n2. In the CLIPort experiments (Section 5.2), it is unclear what a dataset entry is (input and output of the model). Is it an image and two 6-DoF actions? What is the space of language instructions that can be provided for each table-top setup?\n3. In the CLIPort experiments, what is the space of “unseen” objects? Can the authors explain why AIDA generalizes better than the baseline on unseen objects. As far as I understand, no technical contribution directly helps achieve better generalization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "AIDA (Action Inquiry DAgger) is a novel IIL framework that allows a novice policy to communicate its intended action alongside its uncertainty to a teacher policy. The authors introduce the following three modules in their IIL framework:\n1. Sensitivity-Aware Gating (SAG): a mechanism to dynamically adjust the threshold for querying the teacher based on a user-defined sensitivity level (true positive rate).\n2. Foresight Interactive Experience Replay (FIER): for aggregating valid and relabeled novice action plans into the demonstration dataset.\n3. Prioritized Interactive Experience Replay (PIER): for prioritizing replay based on uncertainty, novice success, and demonstration age.\n\nAuthors show experiments demonstrating the efficacy of each component, through learning a classifier on MNIST, keyframe actions on CLIPort tasks, and two real-world sorting tasks. Through their experiments, the authors showcase their different proposed components work and help train decision-making models with greater sample efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weaknesses in the above section. In summary, below are the requested changes for strengthening the paper in my opinion:\n1. More clarity about action space in problem setting.\n2. More clarity around CLIPort experiments - description of the input and output of the model, details on generalization experiments, and discussion on technical contribution to object generalization.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4INapGA7ae", "reviewer": "Reviewer_1FdY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The research addresses an important problem, particularly in tackling the challenges of data inefficiency and poor generalization in robotic learning.\n\n2. The algorithm is presented clearly and in a well-structured manner.\n\n3. The empirical validation includes both simulation and real-world robotic experiments, enhancing the credibility of the results.\n\n4. The performance on unseen tasks is impressive, demonstrating strong generalization capabilities.\n\n### Weaknesses\n\n1. The research gap relative to existing work is not clearly articulated, which weakens the motivation behind the algorithmic design.\n\n2. The three proposed components are largely heuristic in nature—for instance, using thresholds for uncertainty estimation and assigning weights for prioritized replay. As a result, the novelty of the approach appears limited.\n\n3. The baseline comparisons are primarily ablated versions of the proposed method, with only one external baseline (Active DAgger). Including a broader set of baselines would strengthen the evaluation.\n\n4. It remains unclear why the proposed algorithm should lead to improved generalization. A deeper theoretical or empirical justification would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an enhanced Interactive Imitation Learning (IIL) framework building on DAgger. Unlike existing active DAgger approaches that transfer control to the human teacher during queries, their method enables the novice to also communicate its intended action when uncertain. This allows the teacher to either validate or correct the novice's plan, offering richer and more targeted feedback. They present three key algorithmic contributions addressing: (1) when to initiate interaction, (2) how to structure the interaction, and (3) how to effectively utilize the resulting demonstrations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please check the weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)