Policy Optimization under Imperfect Human Interactions with Agent-Gated Shared Autonomy
Authors: Zhenghai Xue
Bo An
Shuicheng YAN
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Human-in-the-loop Learning
Imperfect Human Interaction
Human Feedback
Primary area: reinforcement learning
TL;DR: A new agent-gated shared autonomy framework that safely and efficiently trains RL policies from imperfect human interactions.

Abstract: We introduce AGSA, an Agent-Gated Shared Autonomy framework that learns from high-level human feedback to tackle the challenges of reward-free training, safe exploration, and imperfect low-level human control. Recent human-in-the loop learning methods enable human participants to intervene a learning agent’s control and provide online demonstrations. Nonetheless, these methods rely heavily on perfect human interactions, including accurate human-monitored intervention decisions and near-optimal human demonstrations. AGSA employs a dedicated gating agent to determine when to switch control, thereby reducing the need of constant human monitoring. To obtain a precise and foreseeable gating agent, AGSA trains a long-term gating value function from human evaluative feedback on the gating agent’s intervention requests and preference feedback on pairs of human intervention trajectories. Instead of relying on potentially suboptimal human demonstrations, the learning agent is trained using control-switching signals from the gating agent. We provide theoretical insights on performance bounds that respectively describe the ability of the two agents. Experiments are conducted with both simulated and real human participants at different skill levels in challenging continuous control environments. Comparative results highlight that AGSA achieves significant improvements over previous human-in-the-loop learning methods in terms of training safety, policy performance, and user-friendliness.

Published as a conference paper at ICLR 2025
POLICY OPTIMIZATION UNDER IMPERFECT HUMAN INTERACTIONS WITH AGENT-GATED SHARED AUTON- OMY
Zhenghai Xue+
Bo An Shuicheng Yan 2.3
Nanyang Technological University, Singapore Skywork AI 3 National University of Singapore
ABSTRACT
We introduce AGSA, Agent-Gated Shared Autonomy framework that learns from high-level human feedback to tackle the challenges of reward-free training, safe exploration, and imperfect low-level human control. Recent numan-in-the- loop learning methods enable human participants to intervene learning agent's s control and provide online demonstrations. Nonetheless, these methods rely heavily perfect human interactions, including accurate human-monitored intervention decisions and near-optimal human demonstrations. AGSA employs a dedicated gating agent to determine when to switch control, thereby reducing the need of constant human monitoring. To obtain a precise and foresceable gating agent, AGSA trains long-term gating value function from human evaluative feedback on the gating agent's intervention requests and preference feedback on pairs of human intervention trajectories. Instead of relying on potentially suboptimal hu- man demonstrations, the learning agent is trained using control-switchin signals from the gating agent. We provide theoretical insights on performance bounds that respectively describe the ability of the two agents. Experiments are conducted with both simulated and real human participants different skill levels challeng- ing continuous control environments. Comparative results highlight that AGSA achieves significant improvements over previous human-in- -aman-in-the-loopp learning meth- ods in terms of training safety, policy performance, and user- ussr-friendllnes. Project webpage is at https://agsa4rrl.giin githuu..i
INTRODUCTION
Human in-the- Learning (HL) methods (Kelly et al. 2019; Celemin et al. 2022) integrate human participants in the training process RI. and facilitate safe-guarded RL training without relying on environment rewards. Existing HI. methods leverage low-level human involvement in two main aspects: (1) Monitoring the agent training process for potential safety violations (Peng et al. 2021 Luo et al. 2024) and intervening agent control when necessary: (2) Providing online demonstrations during intervention (Li et al., 2022b Peng et al. 2023). However, human participants may exhibit suboptimal behaviors (Xue et al. 2023c when either monitoring providing demonstrations. For example, human participants can be unfamiliar with the task requirements or the interface for shared autonomy. They may get tired as training goes on and fail figure out whether the learning agent is in dangerous situation. Network latency may also occur stochastically when human participants perform remote operations (Mandlekar et al. 2020)). When interacting with embodied agents, human participants may struggle to control all joints and carry out a high-level policy instead (Li et al. 2022a). Therefore, one critical challenge of HI how to improve training safety and efficiency in face of unpredictable imperfections of human interactions?.
To address the challenge of imperfect human monitoring, recent methods have shifted from human- gated training to egent-gated approaches, where a separate gating agent oversees the environment interaction of the learning agent and calls for human intervention when necessary. EnsembleDAg- ger (Menda et al. 2019) uses high uncertainties in decision making the trigger of human interven- tion. But as we demonstrate in Sec. 3.1 dangerous regions may have low uncertainty after they are visited for few times, such heuristic criteria often fail to detect dangerous regions and safeguard the learning agent. To handle imperfect human demonstrations, some approaches attempt to model human behavior through environment reward and request intervention only when the learning agent is

---

Published as a conference paper at ICLR 2025
o
W
Gating Agent and Intervention Signal
O
Learning Agent and Control
Human Participant, Feedback and Control
The learning agent (in purple) interacts with the environment under the monitoring of the gating agent (in black). The gating agent decides when request human intervention. Learning agent trajectories are in green and human trajectories are in red and yellow. Human feedbacks are denoted with thumbs up and down. Feedbacks at 1 and 13 are human evaluations on whether the gating agent triggers control switch at proper timesteps. Feedbacks at 82 and & are human preferences on whether the current intervention trajectory better than the previous one. For example, the trajectory between is and t4 is better than that between i and 12, so human may provide positive feedback on 44
Figure I:
likely to act incorrectly (Xue et al. rewards which are not available in Swardd-free settings.
2023d: Liu et al., 2023b). But they assume access environment
In this work, we propose novel Agent- Gated Shared Autonomy (AGSA) framework that simultane- ously addresses both aspects of imperfect human interactions, as shown in Fig. [ AGSA is built upon the agent-gatee gated training pipeline and learns from human feedback, both on whether the gating agent proposes intervention proper timesteps and on whether the current human intervention trajectory is better than the previous one. Conceptually, rather than fully relying on low-level human monitoring or human demonstrations--bohh of which can be imperfect-we assume the accuracy of high-level human feedback, as it is easier for humans to make relative judgements that compare trajectories as better worse, than provide absolute optimal decisions (Helson 1964; Kahneman & Tversky 2013). The reliance on human feedback empowers recent success of applying RL from Human Feedback (RLHF) to train large language models (Ouyang et al. 2022), but has not been thoroughly investigated in HL for continuous control tasks. As in RLHF, we train reward models that capture human preferences, which are further used train gating value functions that estimate the long-term effect of the gating actions. In this way, the gating agent can provide more accurate, human-aligned and forseeable intervention signals than previous methods. To train the learning agent without environment reward, we regard states that require intervention as undesirable, assigning negative proxy rewards to state-action pairs that precede human intervention. Since the gating agent fully controls human interventions, the learning agent is insulated from imperfect human demonstrations.
Theoretical analyses show that optimizing human feedback provides performance and safety bounds for the mixed behavior policy that interacts with the environment, demonstrating the effectiveness of the gating agent. Meanwhile, training with negative proxy rewards ensures lower-bound performance guarantee for the learning agent. For empirical evaluations, we select two challenging continuous control tasks of robotic locomotion and autonomous driving, using the MuloCo (Todoroy ett 2012) and MetaDrive (Li et al. 2023) simulator. We employ neural policies with varying performance levels, along with human participants inexperienced in evaluation tasks, to provide imperfect human involvement. Comparative results demonstrate that AGSA learns efficiently from imperfect data while maintaining overall training safety. Our contributions in this paper can be summarized as follows: (I) We identify the challenges posed by imperfect low-level human control and propose to utilize high-level human feedback instead. (2) We design novel framework for ggent-gated shared autonomy, where the gating agent is trained with human feedback and the learning agent trained with intervention decisions from the gating agent.
(3) We provide both theoretical and empirical evidence to support the efficiency and safety of the proposed framework.
BACKGROUND
2.1 PRELIMINARIES
To model agent- gated shared autonomy, we consider two Markov Decision Processes (MDPs) for the learning agent and the gating agent. The learning MDP is defined by the tuple Mi (S, Ti,
2

---

Published as a conference paper at ICLR 2025
including state space S, action space A,, a transition function T, a discount factor 7, and an
initial state distribution dg The gating MDP is defined by the tuple M₁ = Ag, Tg. 7/ rg. do) with the same state space S, discount factor 7, and initial state distribution with M,. A, {0, 1] is the binary indicator of whether to let human policy TV, intervene. Tg is the learned reward function for training the gating agent. Policies of the gating MDP (s(s) have deterministic binary outputs and are regarded as gating functions, i.c., Ty (s) = denotes human intervention and control, and (s) = 0 denotes learning agent's is control. The overall behavior policy, or the data collection policy, can be defined as (-(s) = (1 Ty (=)s)) + -Tg (5) (-Is), where gi is the policy in the learning MDP. The goal of agent- gated shared autonomy is to optimize the learning policy xę and maximize 7'r(st, (t)], where r is the inaccessible environment reward function. Therefore, there is no form of human involvement during testing. The state-action value function for क is defined as Qi a) = E [Er=o 7'((s, ae)]s0 = 8, ag a). Qg for Ty is
its expected return ()))) = BT, 1oo
defined as Qs(s,a) = Em,.T [Ei= rrr (r'((t (8t. as 8, 30 = a).
2.2 RELATED WORK
Human-Gated Shared Autonomy Human-gated HL algorithms rely on human participants to monitor the environment interaction of the learning agent and intervene on dangerous or repetitive states. HG-DAg (Gggeg (Kelly al., (019) and IWR (Mandlekar al. 2020) let human participants provide corrective demonstrations after intervention and perform Imitation Learning (IL) on hu- man sampled trajectories. CEILING (Chisari al. 2022) takes evaluative feedback with human demonstrations, assigning different weights in the imitation loss. Other algorithms combine HL with RL under human monitoring. Under the reward-free setting, RL agents resort human-gated gated intervention for proxy feedback. HACO (Li et al. 2022b) and PVP (Peng et al.] (023) train the Q-value function by maximizing
(s, a) pairs from human generated trajectory and minimizing on (s, a) pairs from the agent. RLIF (Luo et al. 2024) assigns reward of - to state action pairs that are one-step prior to numan-gated intervention. While some human- human-gated shared autonomy methods, such as CEILING and RIIF, take suboptimal human demonstrations into consideration, they can still be negatively influenced by inaccurate human monitoring. Instead, this paper introduces a separate gating agent and no longer relies on humans to monitor the training process.
Agent-Gated Shared Autonomy Existing agent-gated methods include EnsembleDAgger (Menda) jct al. 2019) which estimates uncertainty in decision making and asks for human intervention when the uncertainty level is high. But uncertainty is only an empirical criterion and cannot be aligned with human instructions. Liu et al. (2023a) introduce model-based failure prediction that foresees potential danger in few steps. But the prediction still learns from human interventions that can be inaccurate. EGPO (Peng et al. 2021) lets human intervene if the learning agent has low action likelihood under human's policy distribution. ThriftyDAgger (Hoque et al., 202T) and BCVA (Gokmen al. 2023) use goal reaching rewards to learn proxy value functions. Human intervention will be triggered if the proxy value drops below pre-defined thresholds. TS2C (Xue al., 2023d) and AdapMen (Liu et al. 2023b) compare the value functions of the agent action and human action, and only let human intervene their actions are guaranteed to have better outcomes. But human policy distribution or environment reward are hardly accessible in many real-world applications. The reliance on such information hinders broader applications of these methods. We request additional human feedback to train the gating agent. The feedback is collected when the human intervention starts and terminates, which is easy to implement and adds minimal burden to human participants.
We leave relevant researches on reward-free RL in Appendix A where we mainly discuss the advantage of our framework over Preference-bee based RL (PbRL) methods.
POLICY OPTIMIZATION WITH AGENT-GATED SHARED AUTONOMY
In this section, we first provide motivating examples in Sec. 3.1 and discuss the drawbacks of previous agent-gated methods. Then we discuss our approach of training long-term gating value function from human feedback in Sec. 3.2 In Sec. 3.3 the learning agent is trained from proxy reward signals based on the intervention decisions of the gating agent. We conduct theoretical analysis the presented training framework in Sec. 3.4 and conclude the section with practical algorithm pipeline in Sec. 3.5
3

---

Published as a conference paper at ICLR 2025
Safe
ty:
Unsafe
kg Unsafe
fy. Safe
Uncertainty Estimation
Failure Detection
agla
5 f
fs t
Probabilities of the gating agent requesting human intervention, with the uncertainty estimation method and the failure detection method. Timesteps highlighted in red have problematic intervention probabilities, either failing to recognize danger or being overly conservative.
Figure 2:
3.1 MOTIVATING EXAMPLE
Among existing agent-gated shared autonomy algorithms,, the uncertainty estimation method (Menda) al. (019)) triggers human intervention when state-action uncertainty exceeds a pre-defined threshold.
The failure detection method (Liu et al., 2023a) imitates human intervention decisions. In Fig. 2 we illustrate the probabilities of both methods requesting human intervention along a trajectory in the MetaDrive (Li al.) 2023) simulator. In the presented trajectory, and 44 safe steps which should exhibit low intervention tendencies, but the failure detection method assigns a high intervention probability at 44. This is likely due to human participants being overly conservative in the presence of nearby dangerous zones, leading to unnecessary intervention even when the learning agent operates correctly. f/2 and t3 are dangerous steps due incorrect vehicle direction, but the uncertainty estimation method assigns a low intervention probability at t2. This is because state-action uncertainty is related to the complexity of environment components that is poorly aligned with the actual dangerous zones.
These examples highlight the limitations of current agent- gated algorithms, which cannot ensure appropriate timing to switch to human control. Instead of learning from potentially inaccurate human intervention decisions, in AGSA the gating agent first makes intervention decisions itself and then learns from human evaluative feedback whether the intervention decisions are appropriate. AGSA also learns from human preference feedback on subsequent trajectories influenced by intervention decisions, getting rid of the heuristic criterion in the uncertainty estimation method.
3.2 TRAINING GATING AGENT FROM HUMAN FEEDBACK
The motivating examples demonstrate that to train an effective gating agent, relying solely on heuristics or step-wise imitation of human instructions insufficient, mainly because these metries are loosely connected to the training process of the learning agent. Overall, the central role of the gating agent is to help train the learning agent safely and effectively. Such role can be characterized by the performance of the mixed behavior policy ma, as it is in charge of collecting meaningful training data in the environment and avoid safety violations. Therefore, we analyse the impact of policy switching the long-term performance of the mixed behavior policy To. We employ the gating value function Q, to quantify such long-term effect. Q, takes environment states s and the binary intervention decisions dy input. The gating policy can derived from Qg by selecting the gating action with higher long-term value:
(5) {
if Qy 1) (s(s,
(Human Intervention)
otherwise.
(Agent Control)
To properly train the gating value function Q, for optimal intervention timing, human participants follow three steps, as illustrated in Fig. (upper)
Providing a binary signal (s;) that assesses whether the current environment state is indeed worth intervention. Such human evaluation provides direct feedback whether gating agent's intervention decisions successfully indicates dangerous or unexplored areas. Its advantage over directly imitating human intervention decisions is that humans have time to examine the intervention quality, rather than making real-time decisions that can be influenced by tiredness, carelessness, or network latency.
4

---

Published as a conference paper at ICLR 2025
Gating Agent:
Identify when to let human intervene
Human feedback
O
r1
2
Arp
Arp
1(s+)
1 = Arp
a
ag
d
22
S3
S4
55
5
aT
02
a
a 1g
O
Human demonstrations
Learning Agent: Interact with the environment
The gating agent generates actions of E {0,1} to determine whether human intervene is required, receiving rewards rf based on human feedback. Lower: The learning agent generates actions a" interact with the environment. Its rewards y" are set to -1 on states preceding human intervention and to 0 on other states.
Figure 3: The framework for training both the gating agent and the learning agent. Upper:
2. Interacting with the environment for T steps and offering online demonstration segment (st, dz quman ...... - - O++T ,), aiming at guiding the learning agent out of the region that is dangerous or no longer needs exploration.
3. Providing preference signal pt Palo > o'] (0,0.5,1) indicating whether current segment is better than the previous segment o'. As human participants are familiar with recent trajectories
themselves, this way of online preference pair construction saves the burden for humans of reviewing previously sampled trajectories, as shown by the user study in Sec. 4.2 Bad human samples can happen due to imperfect human behaviors or untimely intervention decision of the gating agent. By assigning low human preference on these samples, the gating agent can learn from human demonstrations at all performance levels and mistakes in the intervention decision made by itself.
As RL environments usually have high-frequency actions, we allow humans to continuously intervene for T steps to provide more accurate preference feedback, where T a predefined hyperparameter. The preference reward model rgg is trained from human preference signals with the Bradley-Tere
model (Bradley Terry
1952
Lee et al. 202I):
exp Σ [(s.))E (y(ss
Ps Pa exp E E(s.a)oo (s, a) + exp E E(st (s),a))e T (s')
[Reward (o, vx) [(1 log P (o' a) + pe log Pe o'll
The overall reward function rg to train the gating value function Q, is the linear combination of the evaluation feedback ((s)) and the preference reward: ry rs af) 7(s,) + > (s 0,in), where À is the hyperparameter for reward balancing and n² denotes actions from the learning agent or human, depending on the control switching decision. The gating value function can therefore be trained with standard value-based RL methods with rg. Besides being able to measure long- term performance, this training procedure does not require human to monitor or provide feedback during the learning agent's 's control. Compared to numan-gated methods that require constant human oversight, this approach achieves more efficient utilization of human involvements and more user-friendly, as demonstrated by the human study in Sec.
3.3 TRAINING LEARNING AGENT FROM INTERVENTION SIGNALS
When training the learning agent in the reward-free setting, direct imitation will lead to degraded policy performance due to potentially suboptimal human demonstrations. So we need to design proxy reward model r- on the training data. One straightforward approach is to use the learned reward model ré as mainly because of the instability of Tv that keeps updating. Instead, we propose to set Tx based on the binary actions of the gating agent a/. which are generated through comparisons of gating values and filter out most of the noisy signals. As shown in Fig. 3 (lower), state- state-action pairs that precede control switching, such as ($2, a5) in Fig. 3 are likely to result in suboptimal outcomes and are assigned with a negative reward r= (82, az)
But as shown in Tab. [ the learning agent cannot benefit much from re-
1. Other agent- generated state-action pairs, such as ($), af)
and (s5; ag ) receive zero reward n= 0. Therefore, T. be set as follows:
5

---

Published as a conference paper at ICLR 2025
f-1 if 14 =1,
Tx ( o7)
otherwise.
While the learning agent may have erroneous actions far before the intervention, these mistakes cannot be identified without accurate human monitoring and access to environment reward. We instead rely on the ability of RI perform implicit credit assignment (Pignatelli al. 2024), allowing the agent to correct mistaken actions. This credit assignment problem is less challenging than that in tasks with sparse rewards (Rengarajan et al. 2022) thanks to the relatively dense intervention signals.
Human-generated samples, such as (33, dg algman) and (54, 01 a in Fig. are expected to guide the agent out of the dangerous or stagnant regions. However, given the potential suboptimality of human demonstrations, the learning agent is not directly trained on human-generatee generated samples and proxy rewards r5 are undefined for these samples. From the perspective of the learning agent, the trajectory temporarily terminates at when / + 1 is the human intervention step. Agent control will resume at 81+ (85 in Fig. (3) if the human successfully navigates through dangerous or unexplored areas. Otherwise, the environment will be reset to the initial state. In this way, the learning agent benefits from human-guided state recovery while remaining unaffected by imperfect human demonstrations.
3.4 THFORETICAL ANALYSIS
We provide theoretical justifications for the proposed training framework of AGSA. One important evaluation criteria of the gating agent is the ability of the mixed behavior policy T=(- = (1 To (+)) m (-(s) + TT (*) (())))(()))))))))))))))(())))( (+), which used to interact with the environment and collect training samples] A well-performing Ty facilitates efficient exploration and safety protection for the learning agent. In the following theorem, we show that the gating agent of AGSA can secure performance lower bound of the behavior policy.
Theorem 3.1. With the gating policy "g defined in Eq. ! and Qg trained with rug, the behavior policy Te defined in Eq. (*) has the following performance lower-bound 17( m7(ma) 2 max {π((*)), ((x)) (1-7)2, where E, max,,a (r(s.a) [w a) is the error of preference-basee reward modelling.
The bound contains the higher performance among the human and learning policy. This demonstrates that the gating agent facilitates efficient exploration by leveraging human demonstrations. Meanwhile, when human policies are suboptimal, the performance of the behavior policy will be lower-bounded by the learning policy itself, which deals with the issue of imperfect human demonstrations. In safety-critical scenarios, the step-wise training cost c(s, a), i.e., the penalty on the safety violation during training, can be regarded as a negative reward. The gating agent can also provide safety guarantee for Ti, discussed in Appendix B.2 We further show in the following theorem that the learning policy has a lower-bound performance guarantee when optimized with the proxy reward function V. demonstrating the effectiveness of the proxy reward from the intervention signal.
Theorem 3.2. Let ã be the optimal policy trained with proxy rewards Ta (s, T. has the following performance lower bound: v)( > TT(TT))
4e. (I-
Similar performance lower-bounds are derived in previous numan-in-the-lop, methods with human- gated training (Luo et al. (024) or with access to environment rewards (Liu et al. 2023b). AGSA obtains such lower bound with a milder assumption on the bounded error of preference- based reward modelling. Meanwhile, thanks to the gating agent that measures long-term intervention outcome, AGSA does not have performance upper bound and may outperform imperfect human participants, as shown by the results in Sec.
3.5 PRACTICAL ALGORITHM
Summarizing previous analysis, we present the detailed workflow of AGSA in Alg. I Line 5 and Line 10 construct the replay buffer D, for training the learning agent, assigning rewards based on gating agent outputs. Line corresponds to three kinds of human interactions, including human demonstrations, human evaluative feedback on the intervention decision, and human preference feedback on the demonstrations. Line constructs the replay buffer D, for training the gating value Q2 and the dataset Dp for training the reward model rv. Line 8 denotes that the preference pair
While human generated samples are mostly excluded when training the learning agent, the state 8:4T at intervention termination will influence the subsequent agent-generated samples. Prooff to the theorems in Appendix B.1

---

Published as a conference paper at ICLR 2025
Algorithm The practical workflow of AGSA.
1: Input:
Gating value function Qg: Learning agent policy ny; Human policy Tai Human preference model pæ: Reward model rw: Learning agent replay buffer Dú; Preference Replay buffer Dp: Gating agent replay buffer Dg: Preference reward ratio X; Human intervention steps T.
2: for epoch = 0, 2....... do
3: for timestep 1. do
4: if (x,, 1)
and not Append (8t-1, 0,-11, 11:1))11111 -1) to Dj.
then
5:
6:
Apply human policy Th for I steps, getting trajectory segment Ø; Query human for intervention evaluation (ss)) and preference feedback pt Pylo > o²).
7:
Append (st, 8,41 (st) + A (o at+n)) ) to Dg- Append (a, o', pc) to Dz-
8:
Set o' 0,
o, pr.prvious www.oooosintteveeeeeee
9:
else
Dj and (st, s1+1, Ar (st, at)) Dg- Apply learning agent policy for step; Set previous intervee=raaass..... Train Ti, rv. Qg on D, Dp. p. gg. respectively.
Append (5t-1, 1r,
11:
Table 1: Results of experiments with different performance levels neural policies. Numbers are normalized scores according to D4RL (Fu et al. 2020). Numbers after + are standard deviations across trials with four different seeds.
Expert Level
Ensemble- DAgger
Failure
DAgger
AGSA (Ours)
Prediction
Hopper
Low
19.54 +: 2.14
11.91 + 6,43
33.27 + 7.36
-29.69 + 0.59
89.39 + 9.76
94.18 1.54
Medium
38.70 = 3.70
10.30
6.68
50.02 + 10.85
-17.95
21.13
2.82
92.44 3.83
High
70.58 + 9.74
39.44 + 1.04
65.28 + 12.32
55.27 + 18.56
94.71 + 1.04
95,79
0.90
Average
42.94 + 5.19
20.35 + 4.72
49.52 + 10.18
2.55 + 13.43
92.16
4.54
94.14 =: 2.76
Walker2d
Low
12.37 1: 2.96
-9.15 1: 3.30
23.50 1: 5.34
-19.11
9.63
115.24
=: 12.09
114.16 1: 2.17
Medium
20.49 + 3.15
23.93
12.33
31.29 + 8.07
-14.53 + 10.56
69.50
38.44
109.38 2.29
High
57.94 +: 8.69
51.57
1,22
50.82 +: 3.28
7.85 45.88
65.53
35.63
129.09 2.98
Average
30.27 1: 4.93
22.12
5.62
35.20 1: 5.56
.8.60 :: 22.02
83.43
28.72
117.55 2.48
HallCheetah
Low
18.19 1. 1.97
11,42
8.89
11.53 1. 1.29
47.45 1. 5.44
20.54
2.93
83.01 0.80
Medium
31.53 +: 2.32
24.18
+ 0.35
15.91 +: 5,44
60.62
+ 7.40
15.79 +: 2.38
88.63 = 0.20
High
52.67
5.77
28.99
0.67
25.05
71.99
1.48
12.16
3.62
89.66 0.69
Average
34.13 3.35
3.30 3,30
17.50
60.02 4.77
2.98
87.10 0.56
(a, o²) is constructed with the current and the previous human generated trajectory. Line 13, m and Q, can be trained with any value-based RL algorithms, such TD3 (Fujimoto et al. 2018) and SAC (Haarnoja et al. 2018), and riy is trained with Eq. 3
EXPERIMENTS
In this section, we conduct experiments investigate the following questions: (1) Can AGSA facilitate efficient training and safe exploration in various challenging tasks, compared to previous -ntta-----th--ooop training methods? (2) How does the components of AGSA, such as the evaluative and preference feedback to train the gating agent, contribute to its overall performance? (3) How do human participants evaluate AGSA terms of performance alignment and interacting workload, compared with other algorithms? To answer these questions, locomotion and autonomous driving, as shown by Fig. ii Appendix [.1 We conduct comparative analysis and ablation studies, as well as designing questionnaires for human-centered studies.
consider the task of robotic
Experiments that involve real human participants are usually expensive and cost-sensitive. Their interaction can also exhibit large variance in different trials. Therefore, existing literature highly depend on trained neural policies as proxies for human policies (Peng et al.] 2021 Xue et al.) 2023d Luo al. 2024). We follow this setting in the robotics simulator MuloCo (Todorov et al.) 2012) and use neural policies with different performance levels to simulate imperfect human policies. We also conduct experiments with real human participants in the autonomous driving simulator MetaDrive (Li ct al. 2023). Though neural policies human participate involved in the training process, all reported metrics in this section are obtained by the learning agent alone in separate evaluation rollouts.
4.1 EXPERIMENTS WITH NEURAL POLICIES AS PROXY HUMAN POLICIES
Setup To obtain neural experts with different performance levels as proxy human policies, we use RLPPD (Ball al. 2023) to train RL policies and load checkpoints at different training steps. We
7

---

Published as a conference paper at ICLR 2025
Walker2ddLLow
Walker2d Medium
Walker2d- Hiig
5000
5000
4000
4030
3000
1000
2000
2000
2000
1000
1000
20
20
30
50
20
20
30
50
20
20
30
40
50
Hummnn Steps (1w(3)
Steppe (1w(3)
Hummn Stepp (1w1))
ACSA wi Fallure Prodiction
ACSA wio Human Preferance Feodback
ACSA (Ours)
Ensemble
Human Evaluative Feedback
Figure 4: Learning curves of methods in ablation study. We consider the Walker2d environment. Full results are in Appendix C areas denote the standard deviation.
The lines are average return across four different trials and the shadow
Table 2: Results of ablation studies on different module combinations. The results are averaged and normalized in the same way as in Tab.
Domain
Expert
AGSA w/ Failure Prediction
AGSA
AGSA wlo Human
AGSA w/o Human
AGSA w/
AGSA (Ours)
Level
Ensemble
Preference Feedback
Evaluntive Feedback
Tip
Hopper
Low
94.97 + 3.46
88.29 +: 12,60
80.48 + 14.35
83,28 + 13.53
12.21 +: 20.47
94.18
3.54
Medium
91.79 + 3.89
77.66 +: 13.93
80.03 1: 4.97
93.13 +: 1.71
4.64
1: 32.08
92.44 1: 3.83
High
83.99 + 5.08
77.47 + 12.87
56.72
2,70
70,86 30.45
-16.07
25.62
95.79
0.90
Average
90.25 = 4.14
81.14 + 13.13
72.41
7.34
82,42 15.23
-2.83 - 26.06
94.14 2.76
Walker2d
58.90 47.85
16.87
5.66
38.14 + 52.61
70.522337.57
103.50
16.98
114.16 + 2.17
Medium
-17.37 + 17.04
40.84
+ 33.34
-4.74 + 23.45
111.51 + 14.94
83.06 + 31.73
109.38
+ 2.29
Hlgh
23.06
6.52
4.47
9.30
75.26
1. 16.73
118.94 1.
101.37
38.82
129,09
2.98
Average
6.16 23,80
17.75 1: 16.10
36.22 1: 30.93
100.33 19.75
95.98
29.18
117.55 1: 2.48
HalfCheetah
Law
64.69 +: 4.61
d: 8.71
78.76
0.55
84.31 0.32
70.82
83.01
0.80
Medium
82,38 = 0.13
25.18 + 13,70
85.69 = 0.00
86.29 + 0.82
85,17
5.07
88.63 + 0.20
High
83.96 +: 1.34
32,75
1: 8.15
86.01 +: 0.92
85.36 + 0.99
93.93
0.74
89.66 0.69
Average
77.01 + 2.03
22. =: 10.19
83.49 + 0.49
85.32 + 0.71
83.31 = 3.22
87.10 = 0.56
use policies at around 20%, 50%, and 100% performance levels compared with the optimal policy and term them as "low", "medium", and "high" policies, respectively. Detailed discussions on neural policies are in Appendix C.T] We follow previous preference-based RL methods (Lee et al. 2022 Xue 2023b) and use comparisons of environment rewards to simulate human preferences feedback Pc Po o o'I. Human evaluative feedback [(s,) is simulated by value function comparisons (Luo it al,, (024)) with details discussed in Appendix C.T For baseline algorithms, we mainly select
iet al.
previous agent-gated methods, including DAgger (Ross al. 2011), EnsembleDAgge (Menda et al., 2019), Failure Detection (Liu et al. (2023a). and BCVA (Gokmen et al., 2023). RLIF (Luo et al. 2024) is also considered as the sfate-of-the-art human-gated algorithm in robotics locomotion. We use SAC (Haarnoja et al. (018)) to train both agents with rw and rx Detailed descriptions on the baseline algorithms are in Appendix C.2
Comparative Results As shown in Tab. [ our AGSA achieves the highest performance across all tasks and performance levels, demonstrating its ability of efficient learning from both optimal and imperfect simulated human policies. Imitation Learning- based methods, including DAgger, EnsembleDAgger, and failure detection, cannot outperform neural policies at each performance level and show suboptimal results. BCVA has poor performance in Hopper and Walker2d, due to the high variance in trajectory terminating signals. RLIF can outperform imperfect neural policies as value-gated intervention introduces additional information related to environment reward. But its performance drops significantly when value functions cannot provide accurate intervention signal, especially on HalfCheetah tasks.
Ablation Study The results of ablation studies on different module combinations are demonstrated in Tab. We present performance curves for these methods in the Walker2d environment in Fig. Full performance curves are left in Appendix C.5 The x-axis of Fig. 5 illustrates in the Walker2d environment, the experiments require 50k human intervention steps. AGSA takes an average of 30k human intervention steps to converge and is faster than all ablation methods. "AGSA w/ Failure Prediction" and "w/ Ensemble" are alternative approaches for constructing the gating agent. Both methods keep the learning agent training unchanged. "AGSA w/o Human Preference Feedback" and "w/o Human Evaluative Feedback" remove T (St, az) and [(::) respectively when computing the gating agent reward rG. "AGSA w/ ry as rx" " refers to using the reward model Ty trained from human preference feedback as the proxy reward T to train the learning agent, in the same way as PbRL algorithms (Lee et al. 2021
8

---

Published as a conference paper at ICLR 2025
Table 3: Results of ablation studies on different values of hyperparameters À and T. The results are averaged and normalized in the same way in Tab. [
Domain
Expert Level
AGSA w/
AGSA w/
AGSA w/
AGSA w/
AGSA (Ours) w/
0.01
T 2
T 10
0.03,
Hopper
Low
85.73
92.47 3.49
84.70 + 0.50
17.48 +: 33.65
94.18
3.54
Medium
84.24 1: 14.62
93.14 1: 1.84
96.43
1: 0.83
39.43 1: 20.90
92.44 1: 3.83
High
85.18 +: 12.62
59.31 d: 47.68
53,22
4: 8.74
35.80 d:
45.79 1:
Average
85.05 + 11.79
81.64 17.67
78.12
3.35
30.91 4: 22.26
94,14 +: 2.76
Walker2d
Low
111.03 +: 4.49
90.48 +: 36.48
119.76
1.27
55.92 +: 62.15
114.16 2.17
Medium
83.42 45.98
102.30 + 20.67
110.53 + 14.93
85.65 + 32,98
109.38
2.29
High
127.71 +: 5.13
130.58 +: 2.33
120.72 + 11.34
50.05 4: 22.41
129.09
2.98
Average
107.39 18.53
107.79 1: 19.83
117.00
9.18
63.87 1: 39.18
117.55
2.48
HallCheetah
Low
83.33 1 0.82
83.12 0.90
86.86 1: 0.56
67.78 1. 2.28
83.01 1. 0.80
Medium
0.36
0.61
89.09 +
85.83 0.50
+
Hlghh
88.66 1: 0,23
88.49 1: 0.52
88.72
1: 1.22
87.49 1: 0.32
89.66 1: 0.69
Average
86.66 1 0.47
86,32 1: 0.68
68.22 1: 0.67
80.37 1: 1.03
87.10 1: 0.56
Compared with the gating agent of AGSA that optimizes long-term performance, failure prediction and ensembled-basee gating agent have comparable performance in the Hopper environment which is relatively simple to solve, but fail to achieve good performance in Walker2d and HalfCheetah environments. Compared with failure prediction and EnsembleDAAger in Tab. thht involve imitation learning, AGSA uses the proxy reward function In to train the learning agent and obtains better performance in the Hopper and HalfCheetah environment. But rs is less effective when the gating agent is highly suboptimal, such as in the Walker2d environment with failure prediction.
According the ablation results, AGSA also have degraded overall performance without either human preference feedback Pt or human evaluative feedback (s;) where preference feedback leads to larger performance gaps. As shown in Fig. + evaluative feedback is helpful to stabilize the training process with neural policies that have poorer performance. Meanwhile, the learning agent will not benefit from the preference reward model re as the proxy reward r.,, is employed in PbRL. This is because T which is trained on human generated samples cannot accurately generalize to (8, a) pairs that are more likely to be sampled by the learning agent. While ri effective to train Qa with binary action space, such noisy reward signal may ruin the more complicated training of the learning agent.
We also conduct ablation studies on the hyperparameters in Tab. 3 including the preference reward ratio À and human intervention steps T. AGSA is robust with different scales of À and maintains superior performance compared with baseline algorithms. AGSA also fits well to fewer steps of continual intervention, but will have degraded performance if human demonstrations are extended to 10 steps. Large numbers of human control will increase distribution shift (Xu et al. 2022) of training samples and may lead to early termination due to imperfect interactions.
4.2 EXPERIMENTS WITH REAL HUMAN PARTICIPANTS
Setup We select the MetaDrive simulator (Li et al. 2023) to conduct experiments with real human participants that provide both low-level human demonstrations and high-level human feedback. Human participants are college students that are familiar with keyboard control but have little or no knowledge of the MetaDrive simulator. The instruction they receive in Appendix C33 Random control latency and environmen speedup are inserted during training to simulate remote operation. Therefore, human participants are likely to provide imperfect interactions during training. For baseline algorithms, apart from Ensemble (Menda et al. 2019), Failure Detection (L.iu et al. 2023a).
BCVA (Gokmen et al. 2023), and RLIF (Luo et al. 2024) that are used in MuJoCo experiments, we consider imitation learning algorithms BC and GAIL (Ho & Ermon 2016), well as PVP (Peng et al. 2023) which is the state-of-the-ar human gated algorithm. We use TD3 (Fujimoto et al. 2018)
to train the agents.
For more accurate algorithm evaluation, we utilize the feature of procedure generation in MetaDrive and make split of training and test environments with different maps and traffic. For the training process, we report the total human involvement steps that include steps of human monitoring and human taking actions in the simulator, total environment interaction steps of the learning agent and the human participants, and total safety cost which reflects the number of potential dangers exposed to the autonomous vehicle during training. We also report the episodic return, episodic safety cost of the learning agent, and the success rate as the test performance of the algorithms. The safety cost is metric used to evaluate the safety performance of driving agents. It is incurred when the agent's

---

Published as a conference paper at ICLR 2025
Table 4: Comparison of different human- somu--in-thee in-the-loop methods in the MetaDrive environment. The human attention rate is given besides the human attention steps. We run all algorithms with three different seeds and report their average score as well as standard deviation.
Training
Testing
Human
Favironment Interaction Steps
Total
Episodic Salety Cost
Episodic Return
Success Rate
Method
Involvement Steps
Safety Cost
BC GAIL
30K (1.0)
113.32 + 10.21
2.17 + 0.65
0.07 + 0.02
30K (0.015)
M
25.90 8.15
81.51 = 9.43
1.31 + 0.23
0.0 + 0.0
EnsembbeeAAgger
17.3K (0.865)
20K
55 :: 3.09
38.44 1: 3.98
8.38 1: 1.73
0.00 1: 0,00
Failure Detection
9.4K (0.47)
20K
d: 5.72
71.37 1: 15.24
1.92 di 0.34
0.00 1: 0.00
BCVA
12.9K (0.645)
20K
4 4.55
143.19
5.04 d: 1,16
+
PVP
20K (1.0)
20K
64 = 2.05
174.71 + 8.41
6.05 + 0.85
0.17 + 0.01
RLIF
20K (1.0)
20K
# 1.25
169.54 +: 6.39
3,90 4: 1.22
0.19 + 0.02
AGSA (Ours)
7.9K().795)
20K
51 + 2.94
263.56 d 8.22
5.78 +: 1.63
0.40 +: 0.02
vehicle collides with other objects or deviates from the designated road. This metric is crucial for assessing the agent's 's ability to navigate complex driving scenarios without accidents.
Performance Comparison Tab. 4 shows the performance comparison of the baseline algorithms and AGSA. AGSA requires the least human attention steps that is helpful for reducing human stress during training. Human-in-the-loog methods all have much lower training safety cost compared with the online imitation learning algorithm GAIL, with AGSA encountering the fewest safety violations. AGSA also obtains the highest test episodic return and test success rate, demonstrating its ability to train generalizable policies with imperfect human interactions. PVP and RLIF benefit from human monitoring and outperform agent-gated baseline algorithms. GAIL has the lowest test safety cost, mainly because of its poor performance and truncated trajectory.
Survey on Human Participants We de- sign user study to analyse the feelings of human participants during training. Detailed instructions are in Appendix C.3 We con- sider three metrics: devotion which is the de- gree of mental concentration, anxiety which measures the level of human stress and ten- sion, and performance which is the human evaluation on agent behaviors. As shown in
The result of user study. The maximum score is for each metric. Metrics with (t) are better with higher scores and vise versa.
Table 5:
PVP
RLIF
Failure Detection
AGSA
Devotion (L)
4.5 1: 0.5
4.7 1.5.
2.0 1 0.9
1.6 1: 0.7
Anxiety (L)
3,5
0.6
22
Performance (f)
0.8
222 + 0.6
19 0.7
4.5 0.7
Tab. 5 AGSA exhibits more user-friendliness compared with baseline algorithms. The agent-gated framework frees human participants from constant monitoring and reduces the amount of human devotion to the experiment. It also leads to less human stress because humans are not directly responsible for safety violations and only in charge of providing feedback. AGSA also has the highest human rated performance level, in line with numerical evaluations.
CONCLUSION
In this paper, we present novel Agent-Gated Shared Autonomy (AGSA) framework for human-in- the-loop RL from imperfect human interactions, achieving reward-fres sample-efficient, and safe training of RL agents. Unlike previous approaches that rely accurate human monitoring optimal human demonstrations, we propose to learn from human evaluative and preference feedback. The gating agent is trained with both types of feedback to accurately model the long-term influence of control switch decisions. The learning agent is directly trained with the intervention decisions of the gating agent, mitigating the issue of suboptimal human demonstrations. We also provide both theoretical and empirical analysis to verify the effectiveness of AGSA.
Limitations AGSA only considers the interaction between one human participant, one learning agent and one environment. It will be interesting to scale AGSA up for interactions between N human participants and M learning agents, where M > N. In the MetaDrive experiment with real human participants, the action space has 2 dimensions, which makes human involvement relatively easy. Potential human interaction interfaces to solve tasks with higher-dimensioaal action spaces are discussed in in Appendix D
10

---

Published as a conference paper at ICLR 2025
Ethics Statement To conduct human-in si--tt-h-loopp training, we pay human participants to invite them joining in the experiments. We ensure transparency by informing all participants about the aim of the experiments and how their interactions will be used. Every participant provide written consent, confirming they are fully aware and in agreement. As experiments are conducted in the simulator that supports pausing, human participants can pause or stop the experiment at any time, or temporarily refuse to intervene when the gating agent asks so. No human participants are injured and no real-world assets are effected by the experiments because all tasks are conducted through simulated environments. Experiments last no longer than thirty minutes and participants rest at least three hours after an experiment. During training and data processing, no personal information is collected in the trained models. The Human-in-the-lowe experiments are conducted under YTU-IRB-20244 1004-1100.
Acknowledgements This research is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISGAward No: AISG2-GC- 2023-009), and NUS Start-up Grant A-0010106-00-00.
REFERENCES
Philip J. Ball, Laura M. Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In ICML, 2023.
Erdem Biyik, Nima Anari, and Dorsa Sadigh. Batch active learning of reward functions from human preferences. ACM Trans. Hum. Robot Interact., 2024a.
Erdem Biyik, Nicolas Huynh, Mykel J. Kochenderfer, and Dorsa Sadigh. Active preference-based gaussian process regression for reward learning and optimization. Int. J. Robotics Res., 2024b.
Ralph Allan Bradley and Milton Terry. Rank analysis of incomplete block designs: I. The Method of Paired Comparisons. Biometrika, 1952.
Carlos Celemin, Rodrigo Pérez-Dattari, Eugenio Chisari, Giovanni Franzese, Leandro de Souza Rosa, Ravi Prakash, Zlatan Ajanovic, Marta Ferraz, Abhinav Valada, and Jens Kober. Interactive imitation learning in robotics: A survey. Found. Trends Robotics, 2022.
Eugenio Chisari, Tim Welschehold, Joschka Boedecker, Wolfram Burgard, and Abhinav Valada. Correct me if am wrong: Interactive learning for robotic manipulation. IEEE Robotics Autom. Lett., 2022.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In ICLR, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In ICML, 2018.
Cem Gokmen, Daniel Ho, and Mohi Khansari. Asking for help: Failure prediction in behavioral cloning through value approximation. In ICRA, 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive prefence learning: Learning from human feedback without rl. In ICLR, 2024.
Harry Helson. Adaptation-level theory: An experimental and systematic approach behavior: New York, 1964.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NIPS, 2016.
Ryan Hoque, Ashwin Balakrishna, Ellen R. Novoseller, Albert Wilcox, Daniel S. Brown, and Ken Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. In CoRI., 2021.
11

---

Published as a conference paper at ICLR 2025
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part I, pp. 99-127. World Scientific, 2013.
Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In ICMI, pp. 267-274. Morgan Kaufmann, 2002.
Michael Kelly, Chelsea Sidrane, Katherine Rose Driggs-Campbell, and Mykel J. Kochenderfer. HG-DAgger: Interactive imitation learning with human experts. In ICRA, 2019.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transforer: Modeling human preferences using transformers for RL. In ICLR, 2023.
Kimin Lee, Laura M. Smith, and Pieter Abbeel. PEBBLE: feecback-efffcient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In ICMI., 2021.
Quanyi Li, Zhenghao Peng. Haibin Wu, Lan Feng, and Bolei Zhou. Human-AI shared control via policy dissection. In NeurIPS, 2022a.
Quanyi Li, Zhenghao Peng, and Bolei Zhou. Efficient learning of safe driving policy via human-AI copilot optimization. In ICLR, 2022b.
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Trans. Pattern 2023.
Anal. Mach. Intell., 45(3):3461-333
Huihan Liu, Shivin Dass, Roberto Martín-Martín, and Yuke Zhu. Model-based runtime monitoring with interactive imitation learning. CoRR, abs/2310.17552, 2023a.
Xu-Hui Liu, Feng Xu, Xinyu Zhang, Tianyuan Liu, Shengyi Jiang, Ruifeng Chen, Zongzhang Zhang, and Yang Yu. How to guide your learner: Imitation learning with active adaptive expert involvement. In AAMAS, 2023b.
Jianlan Luo, Perry Dong, Yuexiang Zhai, Yi Ma, and Sergey Levine. RLIF: Interactive imitation learning reinforcement learning. In ICLR, 2024.
Ajay Mandlekar, Danfei Xu, Roberto Martin-Martin, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Human-in-the-loop imitation learning using remote teleoperation. CoRR, abs/2012.06733, 2020,
Connor Mattson, Anurag Aribandi, and Daniel S. Brown. Representation alignment from human feedback for cross-embodiment reward learning from mixed-quality demonstrations, RLJ, 2024.
Kunal Menda, Katherine Rose Camgs--a and Mykel J. Kochenderfer. EnsembleDAgger: A bayesian approach safe imitation learning. In IROS, 2019.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
Zhenghao Peng, Quanyi Li, Chunxiao Liu, and Bolei Zhou. Safe driving via expert guided policy optimization. In CoRL, 2021.
Zhenghao Peng, Wenjie Mo. Chenda Duan, Quanyi Li, and Bolei Zhou. Learning from active human involvement through proxy value propagation. In NeurlPS, 2023.
Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning. Trans. Mach. Learn. Res., 2024, 2024.
12

---

Published as a conference paper at ICLR 2025
Desik Rengarajan, Gargi Vaidya, Akshay Sarvesh, Dileep M. Kalathil, and Srinivas Shakkottai. Reinforcement learning with sparse rewards using guidance from offline demonstration. In ICLR, 2022.
Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In ICLR, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023.
Nils Wilde, Erdem Biyik. Dorsa Sadigh, and Stephen L. Smith. Learning reward functions from scale feedback. CoRL, 2021.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds imitating policies and environments for reinforce- ment learning. IEEE Trans. Puttern Anal. Mach. Intell., 2022.
Wanqi Xue, Bo An, Shuicheng Yan, and Zhongwen Xu. Reinforcement learning from diverse human preferences. CoRR, abs/2301.11774 2023a.
Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. Prefree: Recommender systems with human preferences for reinforcing long-term user engagement. In KDD, 2023b.
Zhenghai Xue, Qingpeng Cai, Tianyou Zuo, Bin Yang. Lantao Hu, Peng Jiang, Kun Gai, and Bo An. Adarec: Adaptive sequential recommendation for reinforcing long-term user engagement. CoRR, abs/2310.0398. 2023c.
Zhenghai Xue, Zhenghao Peng, Quanyi Li, Zhihan Liu, and Bolei Zhou. Guarded policy optimization with imperfect online demonstrations, In ICLR, 2023d.
13

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RELATED WORK
Reward-free RL To train RL policies without environment rewards, unsupervised skill discovery methods (Eysenbach et al. 2019 [Sharma al., 2020) aim to maximize policy diversity and coverage, Inverse RL methods learn the reward model by maximizing it on human-generated samples and minimizing on agent-generated samples. But the it can be hard for learned reward models to generalize due to insufficient and suboptimal human demonstrations. Recently, preference- based RL (PbRL) methods (Lee et al. 202T; Kim al., 2023) that learn the reward model from human preference pairs have achieved success in aligning large language models with human intentions (Ouyang et al. 2022; OpenAl 2023 Touvron al. 2023), primarily due to the relatively
low cost of constructing large-scale human preference datasets. PbRL has also shown promising capabilities in continuous control tasks (Mattson et al. 2024; Hejna et al. [024 Biyik et al.) 2024a Wilde et al. 2021; Biyik et al., 2024b). However, PbRL methods can be meffficiett in training RL policies from scratch (Lee et al. 2021), as poor performing policies can hardly generate informative preference pairs. They may also encounter safety issues when policies trained from inaccurate reward models are interacting with the environment. Our AGSA method proposes to train gating agent from human preferences that guide and safeguard the learning agent.
B THEORY
B. PROOFS
Theorem B.1 (Restatement of Thm. 3.1, With the gating policy Tg defined in Eq. ④ the behavior policy ny defined Eq. (*) has the following performance lower-bound:
Er
n(Tb) > max (n(T)), m()T)
(5)
71-
where E, max,,a [r(s a) - (s, a) is the error of preference- reward modelling.
We first show that by learning from preference-basee reward rw. the gating value function Q, has bounded discrepancy with the value function QT under the behavior policy Tb.
Proof.
Qg(s, Ba-ma(l))) (r)(s,a) + +. (1(aa)))))))) QQg('' (s( (1 (r)(s,a) + (10001
(6)
So we have
Qg(s, 1) Bowza (5 B (16) (8,( r(s,a)]
(7)
+ 7TB., (Ba' a') Ba'nnr(tle) (
Qs(s,0) Barm (11)) Q (s, a) Bowz(( (s, -r(s,a)
(8)
+ +B [E (1s 2o a') En' (T(s',
E (1 Qg a) can be computed by linearly combining Eq. and Eq.
Boox, Qg(o, a) E QT) (s, (s,
= =Rowns(1 (no(s,,) a) - r(s, a)] + 7B. [Baraa (a)) a')- 1 (9)
= =Eawro(+bb) (r)(s, - r(s, a)] + 7Banss(11)) (r - -(s(,a))
+ 2Ep [Ea'~ro((]""] Q, (s" ,a)) - Ea'nr Q8 a')]
Iteratively computing the last term in Eq. p.. we have
(1s) Ea (s, a)) = Berna (B -((s,, (10)
Er
Combining Eq. 10 with Eq. 7 and Eq. 8 we have
1Q,(s, 1) Ea (s,
E
7e,
Er
1-7
1-7
(11)
(g, 0) Bewny(-1s)) (s,
& <
7Er
Er
1 1-
1-9
14

---

Published as a conference paper at ICLR 2025
When the gating action ag 0, we have Qg(s, > (g(s,, so
B (11)) QTh (8, a) - (1am (8) & Qg(s, 1) - Qg(s, 0) + 2 <
22r
(12)
1-9 1-7
for all state s. Similarly, when the gating action Og = 1, we have Qo(s, 0) < Qg 1), so
22r
(0 a) (- Q (s,a)
(13)
1 1-7
According to the performance difference lemma (Kkkade & Langford 2002), we have
(TTT)) (TT) = Bandry [ENTT (1.)
1
Bonden v (1) 1. (5)
1 7
1
E Eondyy Q1)) - [s) (10))
1
(1 (s) as.
1
Eondey 1(1 Tg (s)) Ea (118) QT - (1)) 9 (1, (14)
1 7
2
Bandan 111- -
[s))]
(1 (1-7)2
21 (3)
(1 71-
28,
(1 7--
Rearranging terms, we have
2
(TT)) > (TT)
(15)
(1 7)2
A similar bound can be derived from Eq. 13 as
22,
(T(=)) > n(T))
(16)
(1 7)2
So we have
25r
(mm)) > max (n(Tm) [()]]
(17)
(1-7)2
which concludes the proof.
Theorem B.2 (Restatement of Thm. 3.2 Let it be the optimal policy trained with proxy rewards To (s. a). ăt has the following performance lower bound:
46,
nf(t) n)((m))
(18)
(1 7)-
Proof. The following proof borrows the main idea from RLIF (uoo et al. 2024). Since we assign negative rewards for human intervention steps with (, 1) > Q (s. 0). in order maximize the cumulative proxy rewards, म should make Qy 1) & Qg 0). According to Eq. (1]). we have
25,
25r
Ea QT (s, a) Ea Q (s, Q (s, 0) (s 1-
(19)
1-7
According to the performance difference lemma, we have
1
nT (T(TT))
7 Bende
(5
Exnds QT0 a) ()a)
(20)
7
2
(1 7)
15

---

Published as a conference paper at ICLR 2025
Figure 5: Environment visualizations of the robotics locomotion tasks Hopper, HalfCheetah, Walker2d, well the autonomous driving task.
Demonstrations
Preferences
Evaluations [(se)
Value-hased Intervention
Ground Truth Reward
DAgger
EnserbbleDAgger
Failure
Prediction
RCVA
V (partially)
RLJF
x
x
AGSA
Failure Prediction
x
AGSA w/
Ensemble
AGSA w/o
Preference Feedback
Evaluative Feedback
x
AGSA
To
AGSA
(Ours)
Table 6: Comparison of environment information required by baseline and ablation methods.
Combining with Eq. (14, we have
7)+)) - (T(a)) = ,1 - 1)(=))) + (7({))))) (T()
2E,
22,
4E,
(21)
(1 - 7}-
(1 7)-
(1 7)2
which concludes the proof.
B.2 SAFETY BOUND
Corollary B.3. With the gating policy my defined in Eq. D. the behavior policy Ta defined in Eq. (*)
has the following safety bound:
C(Ts) min (C(m))
(22)
(1-
where E, = max,,a (re(s, a) (s, a)l is the error of preference-basse reward modelling, re(s, a) is the cost function, and ((T)) = Erody [>a y/re(#t, az)] is the expected total cost of a trajectory.
Proof. The proof can be obtained by replacing the r(st, 0e) in the proof of Thm. 3.1 with re(St, at).
C ADDITIONAL EXPERIMENT DETAILS
C.1
SETUP
The training tasks are visualized in Fig. 5 Neural policies used the Hopper and Walker2d envi- ronment are the same as those in RLIF (Luo et al. 2024) experiments. For the Hopper environment, neural policies have about 20%, 70%. and 110% performance level compared with the optimal policy in D4RL (Fu et al. 2020). For the Walker2d environment, neural policies have about 15%, 40%, and 110% performance level compared with the optimal policy in D4RL (Fu et al. 2020). For the train with RLPD (Ball et al. 2023) and use policies trained at 20k, 40k, 60k steps as neural policies with "low", "medium" and "high" policies. They have about 40% 60%, and 100% relative performance, respectively. In MuloCo experiments, we follow the approach in
HalfCheetah environment,
16

---

Published as a conference paper at ICLR 2025
RLIF (Luo al. 2024 and obtain the simulated human intervention decision x zummann from the value function of the expert policy:
1, if Q (s. IT human (s)) Q (8, T(()))
human
T
0 otherwise.
I (8x is then obtained by comparing the gating agent's decision Fg with "g muuman when To = 1. If the gating agent proposes human intervention but human finds it unnecssary, i.e., "g human 0, (st) will be set to indicating bad gating action. Different from RLIF that queries Tg human on each timestep, we only query " human for evaluative feedhack when Tg = 1. We will add this discussion in the revision.
C.2 BASELINES
We consider the following methods as baselines:
BC: Use supervised learning to train the learning agent with the suman--gnneated dataset.
GAIL (Ho & Ermon 2016): Use trajectory matching to train the learning agent. The learning agent needs full control to interact with the environment.
DAgger (Ross et al., 2011;: No gating agent is involved, with random control switches between the learning agent and the neural policy. The learning agent trained by imitating the neural policy.
EnsembleDAgger (Menda et al., 2019): The gating agent uses the output variance of the ensembled learning policy to determine when to let neural policies intervene.
Failure Detection (Liu et al. 2023a]: The gating agent is trained by imitating human gating behaviors. We follow previous approaches (Luo et al. 2024 Xue et al. (023d) and use value function comparisons as proxy of human gating. The learning agent is trained with imitation loss and state reconstruction loss.
BCVA (Gokmen et al.) 2023): y Use goal reaching rewards to learn proxy value functions. In robotics locomotion tasks, set goal reaching rewards to -1 if the trajectory terminates. For the HalfCheetah environment without termination, we use the reward of the last step as the goal reaching reward. In the autonomous driving task, we send the goal reaching reward when the agent reaches the last checkpoint of the trajectory.
RLIF (Luo al., 2024h: robotics locomotion tasks with simulated human interactions, the learning agent is trained with human intervention signals that are generated by comparing the environment state-action value function Qen (s, a) between actions from the neural policy and the learning agent. As Qenv itself is learned from fixed replay buffer, such value- value-gated intervention can inaccurate, which simulates the imperfect human intervention. In the autonomous driving tasks, rely human monitoring for human-gated training.
PVP (Peng et al., 2023): Use human-gated training and directly optimizes the Q-value function of the learning agent to be close to +1 on human generated samples and close to -1 on agent generated samples.
In Tab. 6 we make detailed comparison which information is required by baselines and ablation methods Tab. and Tab. 2] All involved methods require expert demonstrations. Only DAgger, EnsembleDAgger, and AGSA with Ensemble (one of the ablation methods) do not require access to the ground truth reward, all of which have poor performances according to the experiment results.
C.3 HUMAN STUDY
In human- gated training, human participants are instructed to perform active intervention whenever they identify that the learning agent is in dangerous or under-explored regions. The order of experiments with different approaches following questionnaire to conduct user studies. Among the three metrics, "Performance" was chosen as a straightforward indicator of how effectively the agent completed tasks with minimal safety violations. "Anxiety" included to evaluate the level of stress and fatigue experienced by participants. This measure captures the emotional and psychological responses to the agent's s oscillations, unexpected behaviors, or limitations in intervention timing. It allows capture
randomized for each human participant. We use the
17

---

Published as a conference paper at ICLR 2025
Table 7: Hyperparameters for the training algorithms.
Algurithm
Hyperparameter
Values
Algorithm
Hyperparameter
Values
Common
Batch Size
256
Uncertainty
EnsembleDAgger
0.03 (Hopper)
Learning Rate
3e-4
Threshold
Weight Decay
le:3
(Walker2d)
0.05 (HaliCheetah)
Discount Factor
0.99
Hidden Dims
(256,256)
0.01 (MetaDrive)
for Targel Network Update
0.005
Roward Balancing
ACSA
0.03
Ratio
DAgger
Pretrain Steps
60,000
Human Intervention
Steps Per Iteraction
2500
Steps T
range of participant experiences and their immediate reactions agent behavior. This stress-related metric offers insights beyond raw task performance, highlighting how smoothly the participants feel they can monitor the agent without stress from unpredictable events. "Devotion" was intended to measure the concentration required from participants. indicating whether the agent demanded continuous attention or if participants could rely on the agent to function independently. This feedback helps assess the cognitive workload AGSA imposes, reflecting whether participants feel the need to remain vigilant throughout training, thus directly correlating with workload.
Performance:
Do you think the agent performs well with little safety violations when solving the task? The higher score the better.
Choices: 1, 2, 4,
Anxiety: Do you think training with this agent is stressed? The higher score the more fatigue and stress. A lower score means you more relaxed. Anxiety might come from many sources: Oscillating trajectory, unexpected behaviors, being unable to intervene on time, etc.
Choices: 1, 3, 5.5
Devotion: Do you think you have to keep focused when training with this agent? The higher score the more concentrated. lower score means you do not need to take special care of the training agent.
Choices: 1, 2, 3, 4,
C.4 HYPERPARAMETERS
We present the hyperparameters of the training algorithms in Tab. "Common" refers to common hyperparameter settings shared by all algorithms. In EnsembleDAgger (Menda et al. 2019), human intervention will be triggered if the variance in proposed actions exceeds the uncertainty threshold. The thresholds need to be tuned in different environments. À and T in AGSA keeps the same across all environments.
C.5 RESULTS
We present the full learning curves of ablation studies in Fig. In the Hopper environment, alternative gating algorithms facilitate more efficient training and achieve comparable overall performance. In the HalfCheetah environment, the "AGSA w/ Failure Prediction" method is also more efficient the early stage of training. This because alternative methods are either free of training or trained with more stable imitation loss, more quickly obtaining gating agents that have relatively good performance. Their performances also demonstrate the effectiveness of the proxy reward ra to train the learning agent from diverse gating agents. But in the Walker2d environment where these alternative methods have respective drawbacks, the performance of the learning agent will be degraded. Alternative methods cannot achieve higher asymptotic performance than AGSA either. We also illustrate the probability of AGSA requesting human intervention along the trajectory in the motivating example in Fig. Although AGSA still assigns little higher intervention probability on ta than on t, successfully detects the potential danger in to and t3 and correctly assigns high intervention probabilities.
18

---

Published as a conference paper at ICLR 2025
Hoppers.cw
Hoppenhiig
7503
2502
1003
2003
1503
1002
30
Wallker2e
Walker2s Mediuum
Walker
5002
6000
1003
3000
2003
1030
2002
33
40
50
10
23
30
60
1.0
20
50
Hallcheetsh--..o
Haltchheetah-Mediine
Hallcheeta--
10003
10020
10000
1003
8030
2003
5002
6000
4003
2030
20
60
50
102
0
20
40
90
200
40
60
300
Pirminn Staze Gcdl1
Mumman
Prominn
ACSA wol Fallure Prediction
ACSSA
Human Preference Feedback
AGSA
AGSA
Figure 6: Learning curves of methods in ablation study, The lines are average return across four different trials and the shadow denote the standard deviation.
Safe
Unsafe
Unsafe
Safe
Uncertainty Estimation
AGSA
Mofa = 1/s)
t t
t t
Figure Probabilities of the gating agent requesting human intervention, with the uncertainty estimation method and AGSA.
D FUTURE WORK
We briefly discussed the limitation of AGSA in high-dimensional action spaces the main paper. Such limitation is in fact longstanding challenge in human-in-the--oone learning. Existing researches largely focus on tasks with small action spaces, such as simple robotic arms and autonomous driving. Therefore, believe that generalization higher-dimensional action spaces orthogorical the scope of this paper. Our method focuses on learning from imperfect human interaction and may still be helpful when human- -nn--ii--ttttee tto- RL is applied nigher-dimensione action spaces.
Nevertheless, we share our thoughts on potential challenges and solutions of this generalization. One of the major challenge of nigh-dimensional action spaces is that human participants may be able to handle all dimensions simultancously. Learning policies may also need an exponential increase in human interaction data maintain the same demonstration coverage. This requires to design
19

---

Published as a conference paper at ICLR 2025
more scalable and efficient training algorithms. One potential solution is the hierarchical structure. A high-level planner agent with low-dimensional action space can be designed to guide the behavior of the low-level controller agent with high-dimensional action space. Human participants may only intervene the planner agent and provide demonstrations. Another potential solution is Policy Dissection (Li et af.) 2022a), which is simple yet effective frequency-base approach that aligns the intermediate representation of the learned neural controller with the kinematic attributes of the agent behavior. It has the potential of becoming novel human interaction interface for tasks with nigh-dimensional action spaces.
20


Reviews:
[{"review_id": "HofFiyhuBM", "reviewer": "Reviewer_ZseZ", "summary": "The problem domain of this paper is human-in-the-loop learning where the human provides feedback to an AI agent by providing interventions. However, as opposed to human-gated interventions where the human decides when to intervene, this paper considers the setup where an external AI agent decides when the human should intervene. In addition, the paper attempts to address the problem of imperfect human feedback, i.e., the human demonstrations when they intervene may be suboptimal. To this end, the paper introduces AGSA, an Agent-Gated Shared Autonomy framework. Through experiments in robotics and autonomous driving tasks, AGSA shows improved training efficiency, safety, and user-friendliness compared to prior methods, even with variable-quality human input.", "questions": "Please see the questions in the weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The proposed method achieves more efficient and user-friendly learning from interventions compared to existing works.\n- The paper provides both theoretical analyses and empirical evidence supporting the effectiveness of the proposed method.\n- The proposed method enables learning from humans with varying level of expertise.\n- The paper includes a human subject study.", "weaknesses": "- The proposed method requires the human to compare an intervention segment with the previous intervention segment. This brings the assumption that two segments are comparable. Since their initial states will be different, this is difficult to guarantee. This should be discussed.\n- The paper says RLHF has not been thoroughly investigated in HL for continuous control tasks. This is not true. Even the original RLHF paper (Christiano et al. 2017) uses continuous control tasks. Similarly, I recommend the authors to check the works by Daniel S. Brown, Scott Niekum, Erdem Biyik, Nils Wilde, Dorsa Sadigh for further studies of RLHF and preference-based learning for continuous control tasks.\n- The paper says \"uncertainty cannot be aligned with human instructions.\" It is not clear what this means. Plus, there is no reference or study that supports this statement. Some clarification is needed.\n- In line 141, \"compares\" should be \"compare\".\n- The theoretical analyses presented in the paper are nice but the bounds do not seem useful in practice as they are very loose bounds. This should be discussed as a limitation.\n- Line 290 has a broken reference to an equation.\n- What is eta in Theorem 3.1. Is it properly defined in the paper?\n- I understand the motivation behind using preference data. I agree it is a good way to mitigate the problems due to human suboptimality. But the imitation data could still be used in addition to preferences.\n- The human subject studies should report the number of subjects and demographics information.\n- Table 5 reports variances but I suggest adding statistical significance tests as well.\n- Why is future tense used in line 547?\n- The paper reports authors have applied for IRB. Is it not approved yet? It is not acceptable to conduct human subject studies before IRB approval.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NCtxLvFtXJ", "reviewer": "Reviewer_o6YB", "summary": "The paper presents an interactive imitation learning algorithm. The key idea is to learn a gating agent based on a ground truth intervention signals and a preference-based learned reward. The authors present experiments showing that their method achieves higher reward / success rates than comparison algorithms while being more convenient for human demonstrators.", "questions": "The following questions were ultimately answered in Algorithm 1, but until I got there I didn't find a clear answer in the main text of the paper:\n* It says that humans provide a signal $I(s_t)$ which indicates whether $s_t$ is worthy of an intervention. Does this happen online or do humans go back and relabel the rollouts?\n* When the humans interact with the environment (line 213) is that happening online based on some gating signal or is that a post-processing step as well?\n* Line 230: what do 'current' and 'previous' segment mean? Is this temporal (ie {s_1, s_}, {s_3, s_4}) or current and previous versions of the policy?\n* I don't see anywhere that it says how the gating agent is trained. Is it binary cross entropy on the GT gating values? Is it equivalent to the gating policy $\\pi_g$?\n\nAnother quick question:\n* In algorithm 1, are Q_G and Q_g the same thing?\n\nExperiment Questions:\n* How do you get the $I(s_t)$ values from the RL experts?\n* What are the parameters of the experiment going into Table 1? How much data do all the policies get? What kind of reward do the expert policies get?\n* How do the low and medium experts do so well for RLIF and AGSA?\n* It seems to me that your method has much more information available to it than comparison methods (demos + preferences + GT $I(s_t)$ values. Are there any possible baselines that can use a similar amount of data, even possibly ablations on your method?\n* I'm confused about what \"safety cost\" is in the human experiment. It says \"the number of potential dangers exposed to the agent\" but that's pretty vague", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n* The paper seems to be the first to use preferences in this context\n* The reward formulation combining the ground truth intervention signal with the preference reward is interesting\n\nQuality:\n* The paper seems to achieve some impressive results \n* The paper presents ablation results which verify the components of their method\n\nSignificance:\n* Human in the loop IL is an important research direction and", "weaknesses": "Unfortunately there are several key weaknesses in the paper that make me think it needs some more work before it is ready for publication.\n* As I mention below there are some missing details in the experiment section that make it difficult for me to understand the results. Most notably, I'm not sure how much data any of the algorithms were trained on or where the $I(s_t)$ values came from for AGSA.\n* I'm concerned that the experiment in Table 1, which is pretty important to the paper, is not fair. As far as I can tell AGSA is the only algorithm which has access to the intervention signals $I(s_t)$ and the preferences (and by extension the ground truth reward)\n* Some of the writing is confusing. Algorithm 1 helps a decent bit but I think there are a lot of missing details in the main text.\n\nI'm concerned about the paper's fit for ICLR. As far as I can tell, the paper's key contributions are not so much about the learning method, since no part of their pipeline is very novel. Instead, my takeaway from this paper is that asking for preferences and GT intervention signals does not place much additional burden on supervisors and can help to improve HL learning performance. I think this paper is maybe a better fit for a robotics conference such as CoRL, ICRA or RSS than a learning conference like ICLR. Ultimately I think this issue is better decided by the AC so I am not factoring this into my score, but I feel it's an issue to raise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AxizCyGJfb", "reviewer": "Reviewer_p1os", "summary": "The paper introduces AGSA (Agent-Gated Shared Autonomy), a novel framework designed to optimize reinforcement learning (RL) policy training in environments with imperfect human interaction. AGSA addresses key issues in human-in-the-loop learning, including reward-free training, safe exploration, and the challenges associated with suboptimal human control. Unlike existing methods, which often rely on perfect human intervention, AGSA uses a gating agent that determines when to involve human intervention, thus reducing the need for constant human oversight. This gating agent learns from human evaluative feedback and preferences regarding intervention timings and trajectories, aiming to minimize reliance on direct human monitoring or demonstrations. Theoretical insights provide performance bounds for the gating and learning agents, while experiments in continuous control environments with both simulated and real human participants highlight AGSA's advantages in safety, performance, and user-friendliness over previous approaches​.", "questions": "See above sections.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Innovative Framework for Human-in-the-Loop RL: AGSA introduces a unique approach to shared autonomy in reinforcement learning (RL) by using an agent-gated model that minimizes the need for perfect human intervention, enhancing training efficiency in reward-free settings.\n- Rigorous Theoretical Proof: The paper offers theoretical guarantees, providing performance bounds for both the gating and learning agents, supported by clear proofs and analyses. Extensive experiments across diverse continuous control environments (e.g., robotic locomotion, autonomous driving) demonstrate AGSA’s advantages in training safety, efficiency, and robustness over other approaches.\n- Potential Real-World Impact: AGSA’s design addresses the practical challenges of imperfect human feedback in RL, enabling more reliable and safe training in applications like autonomous driving and robotics. By ensuring safe exploration and learning efficiency even with suboptimal human intervention, AGSA broadens the scope of feasible real-world RL applications, especially in unpredictable, dynamic environments.\n- Clear and Well-Written Presentation: The paper is well-organized, with detailed explanations of AGSA’s components, including gating agent training and feedback processing, making complex ideas accessible to readers.", "weaknesses": "- Reliance on Predefined Feedback Structures: AGSA depends on specific types of human feedback, namely evaluative and preference feedback, which may not always be feasible or scalable in real-world applications. A discussion on how to adapt AGSA for more passive or implicit feedback—where direct input from humans is minimal—could make the framework more versatile and user-friendly in broader applications.\n- Generalization to Higher-Dimensional Action Spaces: The current evaluations are performed on tasks with relatively low-dimensional action spaces, such as robotic locomotion and simplified driving tasks. While these are challenging environments, it remains unclear how AGSA would perform in domains with more complex action requirements (e.g., humanoid robotics or high-degree-of-freedom manipulators). Testing or discussing potential challenges and solutions in these more complex settings would strengthen the generalizability of the approach.\n- Evaluation of Human Burden: While AGSA claims to reduce the human workload, the metrics used for the user study might not be the most relevant. In the paper, authors used Performance, Anxiety and Devotion. Performance is a good metric and easy to understand. However, anxiety and devotion are comparably abstract. From the explanation in Appendix C.3, the explanation for the questionnaire is very vague. There would be some better and quantitative metrics for the user study. (Minor issue (line 890): \"Choces\" should be \"Choices\").\n- Absence of Alternative Feedback Models for Gating: AGSA’s gating mechanism relies on a binary, evaluative feedback model that may not capture the nuances of continuous human feedback. Incorporating or discussing alternative feedback types, such as graded or probabilistic feedback, could further enhance the framework's adaptability to real-world conditions where binary feedback may be insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTKVg0h5dK", "reviewer": "Reviewer_1nyz", "summary": "The paper presents AGSA, a shared autonomy framework for learning from human feedback and gating when the robot passes control the human expert. The agent regards states that require intervention as undesirable, assigning negative rewards to state-action pairs that precede human intervention. AGSA trains a gating agent, and a learning agent. The gating agent requires human teachers to provide a binary signal on whether the current state is worth intervention, control for T steps when intervening, and provide a preference signal for whether the current segment is better than the previous segment. The learning agent is trained to optimize avoiding states that precede control switching to the human agent. The simulated results show that AGSA achieves high performance compared to baselines across multiple task and simulated performance levels. The ablation results indicate that the components are necessary for overall performance.", "questions": "In Section 3.1 Motivating example, what is the performance of the AGSA gating agent when handling the failure cases of uncertainty estimation and failure detection? It would helpful to show how AGSA addresses the issues in the motivating example.\n\nIt would be helpful to discuss the potential challenges of the assumptions AGSA places on the human. The human participant needs to have a very good understanding of the task, the robot’s policy, and how to change the robot’s policy in order to provide the 3 steps AGSA requires. For instance, is it possible the user could intervene but not know if the state necessarily required intervention? \n\nHow much time do humans have time to examine the intervention quality? How long did this take? How is it that they don’t have to “make real-time decisions that can be influenced by tiredness, carelessness, or network latency” when they still need to teleoperate the agent for T steps during intervention? If this occurs offline, then it makes sense how the human’s decisions would be free from latency. It would be great to comment on practical considerations and suitable tasks for a framework like AGSA.\n\nCan state-action pairs that precede control switching by more than 1 timestep warrant avoiding? Currently the learning agent learns to avoid the single previous state, did you also consider larger windows?\n\nFor the real-human participant experiments, what was the total number of participants, and what was their breakdown? Was the study IRB approved? No statistical testing is done on the human subjects.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscript is clear and the approach is well-described. I appreciate the motivating example, as it offers intuition about the limitations of ensemble-based uncertainty quantification and failure detection. While the experiments could use more detail about the quantity and type of data each method had access to, the results seem strong and indicate good performance of AGSA.", "weaknesses": "The paper presents a motivating example to demonstrate the limitations of ensemble-based uncertainty quantification and failure detection. However, it lacks an evaluation of how effectively the AGSA gating agent addresses these specific failure cases. The framework assumes a high level of expertise and understanding from the human operator, including familiarity with the robot's policy and decision-making criteria for interventions. Subsequently, these challenges are not addressed by the user study, which does not examine statistical testing nor provide sufficient details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)