Shared Autonomy with IDA: Interventional Diffusion Assistance
Authors: Brandon J McMahan
Zhenghao Peng
Bolei Zhou
Jonathan Kao
Venue: NeurIPS 2024 poster (2024)
Keywords: Shared Autonomy
Diffusion Models
copilots
intervention reinforcement learning
reinforcement learning
lunar lander
Mujoco
Primary area: human-AI_interaction
TL;DR: We develop an intervention function that dynamically shares control between a human operator and assistive AI agent by choosing the control that maximizes expected future returns.

Abstract: The rapid development of artificial intelligence (AI) has unearthed the potential to assist humans in controlling advanced technologies. Shared autonomy (SA) facilitates control by combining inputs from a human pilot and an AI copilot. In prior SA studies, the copilot is constantly active in determining the action played at each time step. This limits human autonomy that may have deleterious effects on performance. In general, the amount of helpful copilot assistance varies greatly depending on the task dynamics. We therefore hypothesized that human autonomy and SA performance improves through dynamic and selective copilot intervention. To address this, we develop a goal-agnostic intervention assistance (IA) that dynamically shares control by having the copilot intervene only when the expected value of the copilot’s action exceeds that of the human’s action. We implement IA with a diffusion copilot (termed IDA) trained on expert demonstrations with goal masking. We prove that IDA performance is lower bounded by human performance, so that IDA does not negatively impact human control. In experiments with simulated human pilots, we show that IDA achieves higher performance than both pilot-only and traditional SA control in variants of the Reacher environment and Lunar Lander. We then demonstrate with human-in the-loop experiments that IDA achieves better control in Lunar Lander and that human participants experience greater autonomy and prefer IDA over pilot-only and traditional SA control. We attribute the success of IDA to preserving human autonomy while simultaneously offering assistance to prevent the human from entering universally bad states.

Shared Autonomy with IDA: Interventional Diffusion Assistance
Brandon J. McMahan', Zhenghao Peng', Bolei Zhou', Jonathan C. Kao University of California, Los Angeles bmcmahan20250g. ucla. edu pzh@cs ucla. edu boleiêcs. ucla. edu kaošseas ucla.lllc.ooooooo edu
Abstract
The rapid development of artificial intelligence (AI) has unearthed the potential assist humans in controlling advanced technologies. Shared autonomy (SA) facilitates control by combining imputs from a human pilot and an Al copilot. In prior SA studies, the copilot is constantly active in determining the action played at each time step. This limits human autonomy and may have deleterious effects performance. In general, the amount of helpful copilot assistance can vary greatly depending on the task dynamics. We therefore hypothesize that human autonomy and SA performance improve through dynamic and selective copilot intervention. To address this, we develop a goal-agnostic intervention assistance (IA) that dynamically shares control by having the copilot intervene only when the expected value of the copilot's 's action exceeds that of the human's 's action across all possible goals. We implement IA with diffusion copilot (termed IDA) trained expert demonstrations with goal masking. We prove a lower bound on the performance of IA that depends on pilot and copilot performance. Experiments with simulated human pilots show that IDA achieves higher performance than pilot-only and traditional SA control in variants the Reacher environment and Lunar Lander. We then demonstrate that IDA achieves better control in Lunar Lander with human-in-tth-llog experiments. Human participants report greater autonomy with IDA and prefer IDA over pilot-only and traditional SA control. We attribute the success of IDA to preserving human autonomy while simultaneously offering assistance prevent the human pilot from entering universally bad states.
Introduction
As technology advances, humans continuously seek to operate more sophisticated and complex devices (Cascio and Montealegre 2016).. However, more sophisticated technologies typically involve complicated operational dynamics and high-dimensional control systems that restrict their use to narrowly defined environments and highly specialized operators (Schulman et 2018 While fully autonomous AI agents can be trained to perform these tasks, this approach has three key limitations. First, the user's goal is internalized and not easily deducible in most real-world environments. Second, removing the user from the control loop reduces their autonomy, potentially leading poor performance and decreased engagement (Wilson and Daugherty] 2018a (b). Third, as the capabilities of Al advance, it is important to consider how to create technologies that assist and empower humans instead of replacing them (Wilson and Daugherty 2018a b).
Shared autonomy (SA) addresses these limitations by blending human (pilot) actions with assistive agent (copilot) actions in closed-loop setting. Prior studies demonstrate SA can increase human task performance in robotic arm control (Laghi et al. (018)). drone control (Reddy et al.) (2018). and navigation (Peng al.) 2023). A critical component of prior work is an empirically tuned control- sharing hyperparameter that trades off copilot assistance with human user autonomy (Reddy et al.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Shared autonomy, prior works
Interventional Assist (IA), this work
Environment
Environment
Goal masking
masking
Interventional Assist
ae
ag
ap
Copilot
Copilot
8
app
Intervention
Human @mplicggoai
Human
(mmp
Qg
Figure 1: Overview of Interventional Assist Framework for control sharing. (a) Prior works perform
shared autonomy by passing human actions to a copilot (Javdani et al. [0135] Yoneda et al. 2023 Reddy et al., [2018] Jeon et al. 2220). The copilot then plays action, e.g.. by selecting feasible
action closest to the user suggestion Reddy et al., 2018) or through diffusion (Yoneda et al. 2023).
(b) In this work, we design intervention function that plays either the human pilot's s action, ap. the copilot's 's action, ac, based on their goal-agnostic advantages.
2018 Jeon et al. 2020; Yoneda et al. 2023). Excessive assistance can hinder goal achievement, while insufficient assistance can lead to poor control and performance. Prior work has established multiple methods of control sharing, but limitations remain, such as requiring control sharing hyperparameters be tumed empirically or limiting the copilot to general, non-goal-specife assistance (Schaff' and
Walter 2020: Du et al.) 2021; Tan et al. 2022). We will discuss them further in Section 2.
Imagine driving a car with assistive copilot. The majority of the time, the human driver should remain in control. However, the copilot should intervene in certain scenarios to prevent collisions and ensure safe driving. By dynamically adjusting the level of assistance based on the situation and needs of the driver, the copilot simultaneously preserves driver autonomy and engagement while ensuring safety. This example reflects an important open problem: how do we construct an optimal behavior policy from the human and copilot's individual policies? This problem is conceptually similar to the one posed by the Teacher-Studers Framework (TSF) (Zimmer et al. 2014), where teacher agent helps the student learn good policy by intervening to prevent the student from visiting deleterious states and providing online guidance (Kelly et al. 2019 Li al. 2022 Peng et al. 2021). A recent work (Xue al. 2023) has developed methods that improve the TSF framework by using trajectory-bass value estimate to decide when the teacher intervenes student learning. We propose that similar value estimate could be used to determine when the copilot should "intervene" in the user's control, providing the user with assistance. Therefore, we develop an intervention function that estimates the expected value of the copilot and human action in goal-agnostic fashion. Because formulation is goal-agnostic the human pilot with improved flexibility and generalization.
can generalize to goals not seen during the training process, endowing
Our main contribution is a shared control system that leverages a value-based intervention func- tion that can interface with many different copilot architectures while simultaneously improving performance and preserving pilot autonomy. In this work, we build on an existing diffusion-based copilot architecture (Yoneda et al.) 2023). These copilots are desirable because they can be trained using supervised methods which helps mitigate the sample complexity required to train effective copilots. While proceed with the diffusion copilot of (Yoneda al. (223)) we emphasize that our intervention function can be applied to any copilot capable of generating assistive, alternative, or corrective actions to aid the human in complex tasks.
2 Related Work
We have a brief discussion on three related works (Du et al. 2021 Yoneda et al. 2023 Tan et al. 2022).
Assistance via Empowerment (Du et al. 2021): It proposes method that increases human's ability to control the environment and mitigate the need to infer any goals. It defines an information

---

theoretic quantity that captures the number of future states accessible to a human from the current state. An assistive agent is then trained to maximize this quantity while the human performs the task. While this allows agents to assist in goal-agnostic fashion, it typically leads to lower performance than methods that infer the goal, since the assistive agent does not directly help the human achieve goal. In contrast, other methods, including this work, leverage copilot that implicitly infers the goal from human actions. While goal inference can lead to lower performance when the goal is incorrectly inferred, we mitigate this by restricting the copilot to only intervene when the human action is deleterious, i.e., worse than the copilot action across all possible goals.
To the Noise and Back: Diffusion for Shared Autonomy (Yoneda et al. 2023): It develops a copilot that uses diffusion to map a human action closer to an expert's action. They train diffusion process to generate actions from a distribution of expert actions conditioned on roal-agnostic state observations. At inference, the human action first pushed towards a Gaussian distribution by adding noise in the forward diffusion process. The reverse diffusion process is then run on this noised human action to transform it into a sample from the expert action distribution. This action is played in the environment, illustrated in Figure Da. The fraction of the forward diffusion process applied to the human action is the diffusion ratio y E [0, 1] and trades off action conformity (how similar the action is to expert actions) and human autonomy. This copilot therefore requires experimentally tuned hyperparameter to influence how much control the human has on the eventual action. Because the amount of assistance needed may vary greatly depending on task dynamics, using fixed amount of assistance throughout the task may limit performance and autonomy. Our intervention function presented in Section 3 addresses this limitation by allowing the copilot to dynamically intervene based on the human's 's actions.
On Optimizing Interventions in Shared Autonomy TTan et al. 2022): It proposes a method where a copilot is trained with penalty for intervention. This encourages the copilot to limit its intervention and preserve human autonomy. However, intervention is not inherently bad and need not be sparse. Instead of uniformly penalizing all intervention, only unnecessary intervention should be penalized. Additionally, this method is not inherently hyperparameter free as it does require setting penalty hyperparameter to determine how the copilot should trade off assistance and autonomy, although they demonstrate this hyperparameter can be solved via optimization. Another limitation is this method assumes access to the human policy during training so that the copilot can learn when best to intervene for a particular human (e.g., expert the task would likely have less copilot intervention intervention function based on how good
than a novice). In contrast, we define a general-purpon actions are in the environment (irrespective of the specific human pilot playing the actions), enabling one-time training of the intervention function. In our intervention function, an expert would still experience less intervention than a novice because the expert generally plays better actions in the environment. Empirically we find that with our approach the same intervention function can boost the performance of eight different human participants in the Lunar lander environment.
3 Method
Our goal is to introduce a general-purpost intervention function that increases the performance and human autonomy of an SA system. The inputs to the intervention function are: (1) the goal-agnostic environment state, 5, (2) a pilot action, Up, and (3) a copilot action, aes illustrated in Figure Ip. The intervention function then plays either the pilot action (a, = ap) or copilot action (ay = (z)) in the environment. We define this intervention function and describe its implementation. We also prove lower bound on the expected return associated with the policy using the intervention function proportional to the pilot and copilot performance.
We develop an intervention assistance called Interventional Diffusion Assistance (IDA). First, an expert policy is trained to maximize returns in the environment (Section B.2). Second, the expert is used to perform policy rollouts and generate demonstrations. Goal information is removed from the demonstration data. We then train a diffusion copilot from these demonstrations (Section 3.3) Third, we define a trajectory-based intervention function that decides whether to play the human or copilot action (Section 3.4). All training was performed on workstation with single 3080Ti and took approximately 48 hours to complete all three steps for our tasks.
3

---

3.1 Notation and Problem Formulation.
We assume the environment can be modeled as an anfinite-horizon Markov Decision Process (MDP) defined by the tuple M = (S, A, R. 7, P. do). S is the space of all possible environment states, and A the space of all possible actions. R: S A + Rmin Raax] is a scalar reward received for playing an action E A in state E S. P: S x A S + [0. 1] are the transition dynamies for the environment, y is the discount factor, and do the distribution of initial states. We define the state-action value func- tion induced by policy π to be (s((sa)) a) = E ==0==,,0=..0 - ( ppp(-)p--) (s,0a)) [Et= 7'r (8z, dt)],
where π. 5 x A + [0, 1] is the action distribution conditioned on the state.
We additionally introduce the notion of goal that encodes the task objective intention of the human. We can decompose any state (8lğ) into partial goal-agnostic state observation 8, which does not contain any goal specific information, and goal g E G", where G* is the space of all possible goals. Then Q" (s, a) = ("((3 g).a) Q" (8, alg) is the state-action value function under the goal-agnostic state $ and goal g.
We model the human behavior by a human pilot policy, Tps which generates human action ap ~ T-)+ according to the full state observation. The human observes and therefore has access to the goal, 9 E G However, the goal is not assumed to be accessible to the copilot. In this paper, assume the availability of an expert policy Te (A))s). that observes the full state and solves the environment. We also assume that we can query the state-action value of the expert policy Q*= a), with which we actions based on the pilot action and the goal-agnoste state de ~ Te(-|ap; 8). The ultimate goal of this paper is to derive goal-agnostic intervention function T(8, de: ap) E {0, 1] from the expert policy so that the SA system can achieve better performance than the pilot alone. The behavior policy that shares autonomy between the pilot and the copilot can be represented as π = TT. + (1 TT
evaluate the quality of an action We will train copilot policy n. that generates
3.2 Training an Expert Policy
We train a soft actor-critic (SAC) expert to solve the environment (Haarnoja et al. 2018) because it allows us to (1) later query Q* (x, a) for intervention and (2) generate demonstrations in the environment that can be used to train the copilot. In general, other methods of obtaining a Q-value estimator and for training copilot are compatible with IA. We choose SAC for computational convenience. We parameterize our SAC model with a four-layer MLP with 256 units in each layer and the ReLU non- non-lincarity. We use a learning rate of 3 10 and a replay buffer size of 10 The expert fully observes the environment including the goal, and is trained for 3 million time steps or until the environment is solved. We found that training with exploring starts (randomized initial state) produced a more robust Q function with better generalization abilities. Without exploring starts, the Q function performed poorly on unseen states, limiting the effectiveness of IDA.
3.3 Training a Diffusion Copilot
Following Yoneda et al. (2023), j, we trained diffusion copilot (a Ap; 8) using a denoising diffusion probabilistic model (DDPM) (Ho et al. (020). For each environment, we collected 10 million state- action pairs from episodes using the SAC expert. All goal information was removed from this demonstration data. The copilot learned to denoise expert actions perturbed with Gaussian noise, conditioned on the goal-agnostic state 5 and the pilot's is action Up.
Formally, the forward diffusion process is a Markov chain that iteratively adds noise ~ 10, according to a noise schedule (ag, ai,...,000 ar. 3 to an expert action ag. via
Vata:-1 + v1 04-16
Following the forward diffusion process, the diffusion copilot is then trained to predict the noise added by the forward process by minimizing the following loss (Ho al. 2020
CDDPM E,,,,r,,o~\(,.)) [||e - Ea 5, t.))
where eg is a neural network parameterized by 0 that approximates the noise conditioned on the noisy action ae, the goal-agnostic state 8, and the diffusion timestep I. 7 is the distribution of states in the demonstration data. The reverse diffusion process is modeled by a four-layer MLP to iteratively refine at toward ag.

---

Algorithm SA with IDA
1: Initialize environment
2: for each timestep in episode do
3: Sample state 8 and goal-masked state & from environment.
4: Sample human action Op Tpl- (s).
5: Sample diffusion copilot action a, Tel Tep-lap, 8).
6: Compute the copilot advantage score A55 des ap)
7:
if A acs ap) then
8: Play copilot action environment, 01 Gg-
9: else
Play pilot action in environment as ap.
11: end if
12:
end for
13:
environment reset
3.4
Trajectory-basee Goal-agnostic Value Intervention
IDA allows the copilot to intervene in pilot control when they take actions that are consistently bad for all possible goals. We therefore play the copilot's action ae instead of the pilot's action ap when the copilot's action has higher expected return under the expert Q-value, that is,
Q" (s, ae) 2 Q" (s,
However, we can not directly assess Equation 3 since in a SA system the goal is internal to the pilot. Instead, we only have access the goal agnostic state s, such that, Equation becomes,
2 ae) 2 QT a5,
We can define an intervention score [( ā. [g] which considers the value of (5 ae) under the assumption that ğ E G" is the goal, where G* is the space of all possible goals,
( āzlg) = Q" (5 2olg).
(5)
By marginalizing the difference in the intervention scores between the copilot and pilot over the entire goal space define a copilot advantage A(5, ac ap)
J gEG-
F (I(5, - I(5,, aplg) dg.
A aes ap) = -
(6)
where F is a function that maps the difference in intervention scores {-1,+1} to ensure all possible goals are weighted equally. Here we choose F(.) = =iign-). a normalization constant for the integral,
G= G=
max F [(3, ((3, I(5, aplg) dg.
JEG 8,0c.0p
When F(-) sign(-), then A(5, ae ap) [-1,+1]] is proportional to the fraction of the goal space over which the copilot action superior to the pilot action. Also, when F is the sign function, the normalization constant reduces to G = Sgea- dğ and if the goal space is discrete then G = |G* is the number of goals.
We adapt the value based intervention function proposed by Xue al. 2023 to use for shared autonomy by allowing intervention to occur when A(3, ac, a,) = 1. The copilot therefore intervenes when its action has a higher expected return compared the pilot action for all possible goals. Formally, we let
[1 if A(3, ac, ap) =1
T(š, Oc; ap)
otherwise
with intervention policy, m, = Tw. + (1 Ty The process for performing Shared Autonomy (SA) with IA highlighted in Algorithm The copilot advantage is computed at every timestep.
The behavioral (IA) policy is then determined by Equation g
5

---

3.5 Theoretical Guarantees on the Performance of IA
We prove that the return associated with IA is guaranteed have the following safety and performance guarantees.
Theorem 1. Let J = Esgrdds.ann 8.4. P 1 7rr(s,, at)] be the expected dis-
counted return of following a policy Then, the performance the Interventional Assistance policy (or behavior policy) Ti has the following guarantees:
1. For a near-optimal pilot, (Q" (8,ap) Rs max.. (8 a² )), my is lower bounded by upi
J(TI) 2 ((T))
2. For a low performing pilot. (QT ap) xe min, (s, Ti is low bounded by Tea ((TT) 2 (TT)).
The proof of Theorem is in Appendix A. Intuitively, the copilot will only intervene when the pilot attempts play actions from the current state that have expected future returns less than that of the copilot's 's action across all possible goals. The IA policy therefore does not degrade performance of a high-performing pilot, and when the pilot is poor, guarantees performance worse than the copilot.
Results
4.1
Experimental Setup
Baselines. In the experiments that follow, we compared three different control methods. The first method is pilot-only control. The second method is copilot control, where the behavior policy is equal to the copilot policy Te(aal up, 8). Copilot control is the standard practice in SA (Reddy et al. 2018. Yoneda al. 2023 Schaff and Walter, 2020 Jeon et al. 2020) it allows a copilot to improve the human action before it is played in the environment. For copilot control, the action played is the action generated by the diffusion copilot using forward diffusion ratio of y = 0.2, the setting that obtained the best control in (Yoneda et al,, (023). Our third method is IDA, which involves
dynamically setting the behavior policy based on Equation g
Environments. The first environment we use is Reacher, a 2D simulation environment that models two-jointed robotic arm with inertial physics. In this environment, torques are applied to the two joints of the robotic arm to position the arm's fingertip at a randomly spawned goal position within the arm's plane of motion. The state of the environment is 11 dimensional observation containing information about the position and velocities of the joints and goal location. Rewards are given for making smooth trajectories that move the fingertip close to the goal. In each episode, the arm's 's position is reset to a starting location and a new goal is sampled uniformly across the range of the arm's 's reach. Following previous works (Reddy et al. 2018 Schaff and Walter 2020 Yoneda et al.) 2023 Tan et al.) 2022)), we also use Lunar Lander, 2D continuous control environment in which a rocket ship must be controlled with three thrusters to land
desired goal location on the ground.
We modify the environment as described in (Yoneda et al. 2023) to make the landing location spawn randomly at different locations along the ground. On each episode the landing zone is indicated by two flags. The states are 9 dimensional observations of the environment containing information about the rocket ship's position, angular velocity, and goal landing zone. We define the success rate as the fraction of episodes that ended with a successful landing between the landing zone flags. We define the crash rate as the fraction of episodes that terminated due to a crash or flying out of bounds.
Pilots. We use simulated surrogate pilots (Reacher, Lunar Lander) and eight human pilots (Lunar Lander) to benchmark the performance of pilot-only, copilot, and IDA control (see Appendix D for details about human participants). All human experiments were approved by the IRB and participants were compensated with gift card. Surrogate control policies are designed to reflect some suboptimalities in human control policies. We consider noisy and laggy surrogate control policies. Surrogate policies are constructed by drawing actions from either an expert policy or a comrupted policy. We use a switch that controls if actions are drawn from the expert or corrupt policies. Actions are initially sampled from the expert policy. At every time step there is probability of corruption being turned on. Once corruption is turned on, actions sampled from the corrupt

---

Continuous
Linear
Quadrant
Framework
Expert
Noisy
Laggy
Expert
Noisy
Laggy
Expert
Noisy
Laggy
Pilot-Only
18.8
1.6
8.5
17.7
2.07
8.94
19.05
2.07
8.38
Copilot
0
0
0
0
0
0
0
0.03
IDA (FG)
18.0
2.9
8.5
17.3
3.1
9.0
18.5
2.9
8,4
IDA (DS)
18.5
2.4
8.8
18.5
2.7
9.5
17.9
2.8
8.7
IDA
18.8
3.3
8.7
20.0
3.1
9.4
19.7
3.6
9.2
Table 1: Target hit rate (per minute) of surrogate pilots in Reacher environment. "Continuous" uses the default environment goal space where targets spawn anywhere in the arm's plane motion. "Linear" uses goals that are restricted to line 100cm in front of the arm. "Quadrant" uses goals that are restricted to the top right quadrant of the workspace. "Copilot" is the pilot with a diffusion copilot (Yneda et al., 2023) with y 0.2. "IDA" is the pilot with interventional diffusion assistance. IDA (FG) is inferenced with faux goals obtained via monte Carlo sampling and IDA (DS) is inferenced with expert Q function trained on different goal distribution of five discrete goals.
control policy. At every time step while corruption on, there is a probability of turning corruption off. Once corruption is turned off, actions are sampled again from the expert control policy. The noisy surrogate policy is constructed by sampling actions uniformly randomly with 30% corruption probability. The laggy surrogate policy actions are drawn by repeating the action at the previous time step with 85% corruption probability.
4.2 Reacher Experiments
We compared the performance of IDA to pilot-only and the copilot SA method of (Yoneda et al. 2023) in the Reacher environment with targets that could randomly appear anywhere ("Continuous" in Table D We introduce two additional goal spaces to probe the generalization abilities of IDA: "Linear" and "Quadrant." In the linear goal space, goals spawned uniformly random along a horizontal line located 100em in front of the arm. In the quadrant goal space, goals spawned uniformly random in the upper right quadrant of the workspace. To use IDA without making the goal space known to the advantage computation (Equation 6 we constructed a "faux goal" space (FG) by assuming uniform distribution over potential next positions as goals. We then estimated the copilot advantage through Monte-Carlo sampling. Furthermore, we examined IDA's performance when the goal space unknown during Q function training by using a domain shift (DS) environment where goals appear randomly at one of five locations during training. In lieu of using humans, employed laggy and noisy surrogate control policies to emulate imperfect pilots in the Reacher environment across these goal spaces.
We evaluated performance by quantifying hit rate, the number of targets acquired per minute. In the continuous goal space we found that IDA always achieved performance greater than or equal to pilot-only control and outperformed the copilot (Table The expert control policy was optimal, and IDA therefore rarely intervened with copilot action, leading to similar performance. We also found the laggy pilot performed relatively well because laggy actions do not significantly impair target reaches, although it may delay target acquisition. When the policy was noisy, IDA improved hit
a Reacher environment
b
Laggy pilot
Noisy pilot
IDA
IDA
Pilot
Pilot
Coplict
Copiiot
1030
dds
Numbor Goals
Figure 2: Reacher experiments. (a) Continuous Reacher environment. (b) Laggy pilot experiments as the number of possible goals varies. IDA performance slightly decreases the number of possible goals increase, but it never significantly underperforms the pilot. The copilot is significantly worse than both the laggy pilot and IDA. (c) Noisy pilot experiments. IDA outperforms the pilot and copilot.
7

---

rate from 1.6 to 3.3 targets per minute, approximately doubling performance. In contrast, we found the copilot was unable acquire any largets. This may be because the copilot is unable infer the precise goal location from surrogate pilot actions. Together, these results demonstrate that IDA is simultaneously capable of preserving optimal control for high performing pilots while improving hit rate for sub-optimal surrogate control policies, consistent with Theorem
Additionally, we found that even without knowing the goal space during inference (IDA (FG)) or training (IDA (DS)), IDA still increased or maintained the hit rate of the noisy and laggy pilots in the continuous, linear, and quadrant goal spaces (Table I]. We also investigated how the performance of IDA varies with goal space size when goals can appear anywhere (Figure 21). For each evaluation, we randomly sampled specified number of candidate goal positions (ranging from as 100 and 1000) from the continuous goal space and then evaluated IDA with goals restricted to these sampled positions. IDA consistently outperformed copilot and the noisy pilot while maintining the performance of the laggy pilot for all goal space sizes (Figure c). Collectively, these results demonstrate that IDA performance is robust even when the true goal space is unknown.
10, as well
4.3 Lunar Lander
We next evaluated the performance of IDA in Lunar Lander with the noisy and laggy surrogate control policies. Consistent with Yoneda et al. (023), we modified Lunar Lander so that the landing zone appears randomly in one of nine different locations. IDA always achieved performance greater than or equal to the performance of the pilot-only control policy (Table Under the expert policy, which achieves 100% success rate and 0% crash rate, IDA does not degrade its performance, although copilot does. We observed that both copilot and IDA improved the performance of noisy and laggy surrogate control policies, with IDA consistently outperforming copilot in successful landings. Copilots and IDA also reduced crash rates for surrogate pilots. Additionally. we compared the performance of IDA to the penalty-based intervention approach proposed by Tan al. (022). Because the penalty-basee intervention in Tan et al. (2022) used an MLP, we compare it to both IDA and IA with an MLP based copilot. We found that IA consistently achieved a higher rate of successful landings for both the noisy and laggy surrogate pilots than penalty based intervention for both copilot architectures. We further found that IA (MLP and IDA) yielded a lower crash rate than penalty-based intervention.
Next we examined when and why copilot intervention occurred for surrogate pilots. Because these control policies were constructed by corrupting expert's control policy, we were able characterize intervention during periods of expert control versus corruption. We examined the distribution of copilot-human advantage scores, which measures the fraction of the goal space over which the copilot's
sss action has higher expected return than the pilot's action. For both the noisy and laggy pilots, we found the distribution of copilot advantage scores were different during expert actions vs corrupted actions (Figure b)..). When corrupted actions were played, there were a greater number of states where the copilot advantage was equal 1, indicating the copilot's action had a greater expected return over the entire goal space. Consistent with this, we see that intervention was more common during periods of corruption.
Success Rate
Crash Rate
Shared Autonomy
Expert
Noisy
Laggy
Expert
Noisy
Laggy
Pilot-only Copilot (MLP)
100%
21%
61%
0%
76%
39%
92%
54%
85%
0%
1.6%
0.7%
Copilot (diffusion)
93.3%
75%
92%
0.3%
3%
0.7%
Intervention-Penale (MLP)
100 %
58%
76%
0%
4%
8%
IA (MLP) IDA (diffusion)
99% 83%
96%
0%
0.7%
0%
100 %
83%
100 %
0%
3.3%
0%
Table 2: Performance of surrogate pilots in the Lunar Lander environment. IDA is interventional assist with a diffusion copilot. IA (MLP) is interventional assist with an MLP copilot. Intervention-Penne Penalty (MLP) refers to the intervention penalty approach proposed by Tan al. (2022)
8

---

a
Noisy pilot intervention characterization
C IDA intervention in bad states
pilot action
comrupted action (random)
290
IDA prevents flipping
copilal thrust: 86.3%
pilot thrust=56.1 1%
corrupted Action unintarvaned states
pilot comppted
21
Copilat Advantage
intervened abales
IDA aids graceful landing
b
Laggy pilot
cornupted action (lagged)
120
pilot action
28.7%
58.8%
100%
Figure 3: Analysis of copilot advantages during intervened states. (a) Characterization of intervention during the noisy pilot control. The left plot shows the copilot advantage, which is generally higher for corrupted (random) actions compared to pilot actions. When quantifying the number of intervened states, IDA intervenes
when the corrupted actions taken. (b) Same (a) but for the laggy pilot. (c) Example intervened states, In the top panel, the copilot prevents flipping. In the bottom panel, the copilot action helps to make graceful landing.
We also visualized example states of copilot intervention for both the noisy and laggy pilots (Fig- ure 3))) In two representative examples, we see that intervention generally occurs to stabilize and prevent the rocket ship from crashing or flipping over. In Figure 3 (top), we observe that during intervention, the copilot applies lateral thrust to the rocket to prevent it from rotating sideways despite the pilot operator attempting to fire the main rocket, which would result in a destabilizing rotation of the rocket ship. In Figure 3 (bottom), we observe that intervention occurred during landing where the copilot increased the amount of thrust the main thruster to soften the touch down while simultaneously applying thrust on the right rocket to level the rocket ship for landing. both instances, the copilot action prevented the rocket ship from entering universally low-value state.
4.4 Lunar Lander with Hemman-nn-thelloee Control
Given IDA improved the performance surrogate pilots in Lunar Lander, we performed experiments with eight human participants. Participants played Lunar Lander using pilot-only, copilot, or IDA. Participants used a Logitech game controller with two joysticks to control the rocket ship. The left joystick controlled the lateral thrusters and the right joystick controlled the main thruster. Each participant performed three sequences of 3 experimental blocks (pilot, copilot, IDA) for a total of blocks (see Appendix D for experiment block details). Each block consisted of 30 trials (episodes). Participants were blind to what block they were playing. The game was rendered at 50 fps.
Human pilots experienced considerable difficulty playing Lunar Lander, successfully landing at goal locations only 14% of the time (Table Copilot control allowed humans participants to successfully land the rocket ship at the goal locations 68.2% of the time. However, the copilot also frequently prevented any landing, resulting in a timeout in 13.8% of trials (0% pilot only and 0.1% in IDA). While this led to a lower crash rate (3.6%), it also reduced user autonomy and overall
Human-in-the-Lopg Lunar Lander
Success Rate
Crash Rate
Timeout Rate
Out of Goal Landing
Human Pilot
14.0 (16.8) %
85.8 (16.7) %
0.0 (0.0)
0.2 (0.6) %
w/ Copilot
68.2 (9.1) %
3.6 (2.4) %
13.8 (5.5)
14.4 (3.6)%
w/ IDA
91.7 (4.9) %
8.2 (5.1) %
0.1 (0.4)
0.0 (0.0) %
Table 3: Results of human pilots playing Lunar Lander. Mean and (standard error of the mean) presented for each metric.

---

lowered the success rate. In contrast, IDA achieved the highest performance, enabling participants to successfully land at the goal location 91.7% of the time which was significantly higher than pilot (p < 0.01, Wilcoxon signed-rank) and copilot (p < 0.01, Wilcoxon signed-rank) control. IDA also resulted in a significant reduction of crash rate when compared to human pilot control (p < 0.01, Wilcoxon signed-rank).
To quantify the level of ease, control, and autonomy human participants felt in each block (pilot only, copilot, IDA), we asked participants to provide a subjective rating in response to multiple questions. rate which blocks felt easiest, most controllable, and most autonomous.
We asked participants To assess ease of control, we asked: "How easy was the task of landing the rocket ship at the goal location?" To assess how much control the participants had, we asked: "How in control did you feel when performing this task?" However, participants may not have felt much control even in pilot-only mode, so assess autonomy, we asked: "How much do you believe your inputs affected the trajectory of the rocketship?" All questions were answered on scale of 1 to 5 with 5 indicating the easiest, most in control, or most autonomy, respectively. We found that humans subjectively prefer IDA to baseline copilot assistance in terms of ease of use, controllability, and preserving autonomy (p < 0.01, Wilcoxon signed-rank).
Conclusion and Discussion
Our primary contribution is Interventional Assistance (IA): a hyperparameter-ffe free and modular framework that plays a copilot action when it is better than the pilot action across all possible goals. We find that IA outperforms previous methods for intervention based shared autonomy proposed
IDA
Participant rating
Copilot
4.6
4.3
Pilot
by TTa et al. (2022) as well as traditional copilot-only
3.0
based methods for control sharing (Yoneda al.] [0233
Reddy et al. 2018). Furthermore, we empirically demon- strated IDA (IA with Diffusion copilot) improves both objective task performance and subjective satisfaction with real human pilots Lunar Lander (Figure f While prior SA systems may degrade pilot performance, particularly when copilots incorrectly infer the pilot's s goal (Tan al.) 2022
23
1.5
Ease
Control
Autonomy
Figure 4: Participants rated IDA as the casiest and most controllable condition. Participants subjectively rated IDA as achieving a similar level of autonomy to pilot only control but significantly better than copilot control.
Du et al. (021). IA does not degrade human perfor- mance (Theorem 1) and often improves
One limitation of our approach is that we must train an autonomous agent in a simulated environment obtain an expert Q function. However, this is not a fundamental requirement to learn an intervention function. straightforward extension the current work may use an offline dataset of expert demonstrations to train an ensemble of Q-networks (Chen et al. (021). In general, while IA requires access to an expert Q function it makes no assumptions about how that Q function obtained and we leave various methods of obtaining Q function as directions for future work.
Additionaly, IDA demonstrated resilience across changes in goal space and be easily adapted to real world settings where the goal space is unknown by construction these faux goal spaces. Of course, in many settings task structure can be leveraged further constrain the goal space and improve the assistance IA is able to provide. In these settings, another direction for future work is an implementation of IA that leverages belief system to differentially weight candidate goals. Future work could potentially improve IA by removing unlikely goals from the advantage computation.
Acknowledgments This project was supported by NSF RI-2339769 (BZ), NSF CPS-2344955 (BZ), the Amazon UCLA Science Hub (JCK), and NIH DP2NS 122037 (JCK). ZP is supported by the Amazon Fellowship via UCLA Science Hub.
10

---

References
Wayne F Cascio and Ramiro Montealegre. How technology is changing work and organizations. Annual review of organizational psychology and organizational behavior, 3(1):349-375, 2016.
Xinyue Chen, Che Wang. Zijian Zhou, and Keith Ross. Randomized ensembled double q- q-learning: Learning fast without a model, 2021.
Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan, Ave: Assistance via empowerment, 2021.
Tuomas Haarnoja. Aurick Zhou, Kristian Hartikainen, George Tucker. Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18200599, 2018.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 1116840-68511 2020.
Shervin Javdani, Siddhartha Srinivasa. J Andrew Bagnell, Shared autonomy via hindsight optimization. Robotics science and systems: online proceedings, 2015, 2015.
Hong Jeon, Dylan Losey, and Dorsa Sadigh. Shared autonomy with learned latent actions. arXiv preprint arXiv:2005.032100 2020.
Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell and Mykel Kochenderfer. Hg-dagger: Interactive imitation learning with human experts, In 2019 International Conference on Robotics and Automation (ICRA), pages 8077 8083. IEEE, 2019.
Marco Laghi, Michele Maimeri, Mathieu Marchand, Clara Leparoux, Manuel Catalano, Arash Ajoudani, and Antonio Bicchi. Shared-autonomy control for intuitive bimanual mal-paanipllaiio. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), pages 1-9. IEEE, 2018
Quanyi Li, Zhenghao Peng, and Bolei Zhou. Efficient learning of safe driving policy via human-ai copilot optimization. In International Conference Learning Representations, 2022.
Zhenghao Peng, Quanyi Li, Chunxiao Liu, and Bolei Zhou. Safe driving via expert guided policy optimization. In 5th Annual Conference Robot Learning, 2021.
Zhenghao Peng. Wenjie Mo, Cheuda Duan, Quanyi Li, and Bolei Zhou. Learning from active human involvement through proxy value propagation. Advances in Neural Information Processing Systems, 2023.
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning, 2018.
Charles Schaff and Matthew R. Walter. Residual policy learning for shared autonomy. 2020.
John Schulman, Sergey Levine, Pieter Abbrel, Michael I. Jordan, and Philipp Moritz, Trust region policy optimization. In JCMI., pages 1889-1897. JMr.R 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018
Weihao Tan, David Koleczek, Siddhant Pradhan, Nicholas Perello, Vivek Chettiar, Vishal Rohra, Aaslesha Rajaram, Soundararajan Srinivasan, H M Sajjad Hossain, and Yash Chandak. On optimizing interventions in shared autonomy. 2022.
H. J. Wilson and P. R. Daugherty. Collaborative intelligence: Humans and ai are joining forces. Harvard Business Review, 2018a.
H. J. Wilson and P. R. Daugherty. Human Machine: Reimagining Work in the Age of Al. Harvard Business Review Press, 2018b.
Zhenghai Xue, Zhenghao Peng, Quanyi Li, Zhihan Liu, and Bolei Zhou. Guarded policy optimization with imperfect online demonsirations, 2023.
Takuma Yoneda, Luzhe Sun, Ge Yang, Bradly Stadie, and Matthew Walter. To the noise and back: Diffusion for shared autonomy, 2023.
Matthicu Zimmer, Paolo Viappiani, and Paul Weng. Teacher-student framework: a reinforcement learning approach. 2014.
11

---

A Proof of Theorem 1
In this section, we will prove the theoretical guarantees the performance of the interventional We first introduce several useful lemmas.
assistance (IA) in Theorem |
First, we introduce a lemma for later use by following the Theorem 3.2 in (Xue et al. 2023).
Lemma 1. For any behavior policy æ deduced by a copilot policy nee pilot policy Ty. and an intervention f T(s, ap, oc), the state distribution discrepancy between Ti and Te is bounded by the policy discrepancy and intervention rate:
|R TTolll (1- (1-8)7 B B ||Te(-1)) Tp((1s)]l
(9)
1-7
where B Beenty Teetttiiieeeeee ( 1l (0))))))0000000/000000))))-- - is the weighted expected intervention
1 (Pc rate. Tay and THe are the corresponding state visitation foriiitttoonsollloliiii ni and ne respectively.
Proof. We begin with the result Theorem 3.2 in (Xue et al. 2023
?
|| Txall1 $ 1 BONTH, || =c(ss))l
? E ae M B
Bewiny ae [TTo(alss) + (1 (Tpp)(pls)) ne(aa|=)]l
7
[1(1 - T(s, ap)) (Tp(+ - =c(+1s]]l
1
(1-8)7
887
EYNTAY [|=( (-(()))l
1-7
To prove the theorem, the key lemma we use is the policy difference lemma in (Schulman al. 2013) introduced below. It introduces one policy's 's advantage function computed on states and actions sampled from another policy's generated trajectory. Here, the advantage function defined as A (s. a) = Q" (s, a) V= (s) and V (s) = ((s, a) is the state value function. J(x) = Begrdo -at s1411 (s)) E. ('r(SE, a:)] is the expected return following policy π.
Lemma 2 (Policy difference lemma). Let ı and xe' be two policies. The difference in expected returns can be represented as follows:
8 Σ An (sc, ai) [=
J(T) - J(T') = E., aa
(10)
We introduce two lemmas that exploits the intervention function we proposed in Section 3.4
Lemma 3. The Q value of the behavior action under the expert's Q estimate is greater or equal to the Q value of the pilot action.
E (1=0 2 (a,
(11)
Proof. According to the intervention function T Equation g intervention happens when the copilot advantage function A(s, de, ap) 1. If consider F to be the sign function, then according to Equation 6) A(3, de, ap) = 1 means for all goals we will always have I(3 az () > I(3, aplg), Vg.
Recall I (() az g) TT (5t. ()) when intervention happens, will have:
QT= (5t, acl@)= > =5 apll), v9
(12)
Therefore,
T(s, des ap) (5,, ag) > T(s, acs ap) (,,, aplg), Vg. where the equality holds when intervention does not happen and T = 0. Now we introduce the expectation over the behavior policy. Ea (113 Q* (51, alg)
(13)
Eeg WT, Ba, NT. TQ (Rt, acl 10) (1- TT, (14)
BayNE, QT (Zc. (g1)), Vg
(15)
The above equation holds for arbitrary g. Therefore E. Ea QT (s, a) > E Earra Q== (s,
0
12

---

Similar to Lemma we have:
Lemma 4. The Q value of the behavior action under the expert s Q estimate is greater or equal to the Q value the copilot action.
E Ea (-1s) Q" (s,a) 2 Ea.-x, Q"s (s, ae)
(16)
8 intervention happens when the copilot advantage function A(3, ac. ap) 1. If we consider F to be the sign function, then according to Equation 6 A(s, de, ap) means for all goals we will always have I(3, a. () > [($, apl9), Vg.
Proof. According the intervention function T in Equation
Recall St () = Q" ($,, drl)), when the intervention does not happen, we will have:
Q (5t, aolg)= S Q (5c, apl)),
(17)
Therefore,
(1 T(s, Oes ap)) (3,, apli) 2 (1 aer (3r Vg.
(18)
Now introduce the expectation over the behavior policy.
(- (5,, alg) = Eay BaENa. TQ (3 azld) + (1-1T)Q apl) (19)
2 Enown. Q= (3e, a (c))), V4
(20)
The above equation holds for arbitrary g. Therefore E. Ra QT (8, E Borxx QT (8, ac)-
We introduce another useful lemma.
Lemma (State Distribution Difference Bound). Let g and n' be two policies, and let T. :S - and Txx S + [0, 1J be the corresponding state visitation distributions. For any state-dependent function f() that is bounded by M (i.e. [ff(s) & M for all s), the difference in expectations of f() under these two distributions is bounded follows:
LE EENT [f())] - E [f(3)]] I M Talll1:
(21)
where
I T- Σ (T)()) (s))-
(22)
is the total variation distance between two distributions and Tz
Proof. The expectation of [()) under the state distribution Tx can written as:
E [f(s)] Σ (s)
(23)
Similarly, for policy n':
E B [f(s)] Σ (s)
(24)
The difference in these expectations is:
E [f(s)] E E [f())] (T) $ >Σ [s) (s)) (25)
Given that [[(s)] S M, we have:
Σ (s) - Tur (s) s M > (Tx))) - -Tx( = M] - Txr 111.
(26)
Thus, combining these bounds, we have:
EsNr. [f(s] [[ss)]_Ewrp
E [f(s)]| & M - -
(27)
This completes the proof.
13

---

A.1 The relationship between the behavior policy and the pilot policy
In this section, we will lower bound the performance of the IDA policy ((TT) by the performance of the pilot policy TTT) under certain conditions.
Theorem 2.
Let J = Beo-do,aurat counted return of following a policy Then for any behavior policy as deduced by a copilot policy To a pilot policy wy, and an intervention, function ((8, ap, Ge).
Pl ac) Ei= 7r1(st, az)] be the expected dis-
? 1 1 Eanda [TTel 1 8) - Tp( |
J(TI) J(mp) 3R
(28)
wherein R Ryaax Rynin is the range of the reward, 3 B (1) is the Esntay
weighted expected intervention rate, and the L1-norm of output difference [|Te(-)s) Tp (-1s))111
LA (als) [a)ss)) da is the discrepancy between Te and Ty state s.
Proof. By following Lemma
we have
((TT) (
(29)
Baa
Ae (8,a)
(30)
1
Bea-Ta
2' (s) VT (s))
(31)
= BgNtst
r (Pa (0-0) vs)
(32)
By following Lemma
we continue to have:
ESNTer
(33)
E Eg
7 (1s -
(34)
EgNTer
Σ 72 (1)
(35)
1=0
At the same time, we have
J(T,) - [xx.) = EaNTey
ndt (T
(36)
We want to relate Equation 33 with Equation 36 which has the expectation over state distribution Try, instead of Tzy-
To solve the mismatch, we apply Lemma
Here, we let
f(s) E B (1s)) (T ap).
(37)
1=0
7 R M, where R Rmax Rmin is the range of the reward. Lemma
We have [ < bounds the difference in expectations of f(s) under the state distributions Tx, and Ta, as follows:
1-9
EgNTo, [s(s)] - E E [[()]] S M|[Ta, - T2ylla
(38)
14

---

Substituting f(s) into this inequality, we get:
00 r
20
BENTTP
E
(11)) (s, (s)
BoNTas
Σ 1=0
2 E Bappaa((1)) (1)
Se. (39)
1=0
where M T|T Taillı- 111.
Thus, we can write:
(1)) (s, 2 BXNTEP
E Bag - xp 1m (8. (40)
BIUTEJ
1=
From the deduction above, we have J(TT) [Txe 2 Bantey the right hand side except is J ( J(Te). Therefore.
E BB (1s) A (5, (s) and
~
((TT) ((Te) 2 (TTp [TTe -E.
(41)
According to the Theorem 3.2 in (Xue et al. 2023) (Lemma D.. we have
By
Tmp Taz 1 5 - Bandr, [[ [s) 1))1s
(42)
where / Euseryy (1 can be upper bounded by:
ExNTT, [=<((10))xp((11)]] is the weighted expected intervention rate. Therefore, the
2 Bendey [[To[. (s) Tpl. s)]l
6= M| Txrll1 S. 3BR
(43)
Therefore, we have:
2 ((Tp) - -3R ?
Eanda, [TTel. 8) - xp(. (1)s))
(44)
This is the lower limit for IDA in terms of the pilot.
A.2 The relationship between the behavior policy and the copilot policy
In this section, will lower-bound the performance of the IDA policy J(TT) by the performance of the copilot policy J(xz)-
Theorem 3. Let (TT
(ss) a.) Er rrr(st, az)l be the expected dis- counted return of following a policy π. Then for any behavior policy deduced by copilot policy To a pilot policy Tp. and an intervention function T(s, ap, ac).
J( 2 - -1
B
7
E Eanday || 1 8) - Tp(. |s)]
(45)
1 7
Proof. According to Lemma 4
J(zi) [(me)
(46)
=Egaaxi
7 AATe(s,a)
(47)
=B,
TQ= (s (=)
(48)
1=0
>B
Bagwze((ss)
Σ
(1Q=== v
(49)
1=/
15

---

With Lemma 2 we have
20
( (TTO Buntre
Σ
ra (Ba)) A (s,
(50)
Now we will use Lemma letting
fc(s) =
"B
(51)
t=
We will have ||f((s)] bounds the difference in expectations of f() under the state distributions Tx. and Tx, as follows:
R = M, where R = Rmax Rmin is the range of the reward. Lemma
1-7
Egwt, [fc(s)] Es [fc(s]] s M Tay
(52)
Substituting f(s) into this inequality, we get:
E BEnTTe
7 E (1)) ao)
E B
7 Ba
S Ecs
(53)
where 6. TT Tma l1a
Thus, we can write:
E BRNTRI
Σ
AT 2 BeNtue
Bu nn (16) Ec.
(54)
By substituting Equation 49 and Equation 50 into Equation 54 we have:
[(T]) (TT)) 2 ((To) ( ec.
(55)
By using Lemmal I we have &c M THy ll upper-bounded by:
E (1--38)
Eanda, ||To(. ( -Tp( sp]l1))ll
(56)
Therefore, we have:
J(z1) 2 J(Te) B) 7 7
Banday || 1 8) - (1)1
(57)
This is the lower bound of IA terms of the copilot.
A.3
Safety and Performance Guarantees of IA
We now prove main theorem the performance guarantees of following the IA policy, restated below:
Theorem Let ( Esgodd atl st) at) Er ('((s,, at)] be the expected dis-
counted return of following policy т. Then, the performance following the Interventional Assistance policy (or behavior policy) ny has the following guarantees:
1. For a near-optimal pilot, (QT= (Qm= ap) se max. Q (s. my is lower bounded by Rz
((TT) 2 (TTp).
2. For a low performing pilor, (QT ap) ss min, (s, (8,a)); my low bounded by ne:
((TT) 2 (TT.)
16

---

Proof. From equation 44 and equation 57 performance of IA,
we have obtained the following lower bounds on the
J(TI) 2 J(Te) -11-10 3)R
Benddn || (6)) -Tp(- ()]l
(58)
7
1 - ((TI) 2 (TTp) - 3R
Eanday Tell (8) - =pp- |s]ll
(59)
For near optimal pilot,
Q(s, = max (s,
(60)
and therefore,
Q(s, ap) 2 (s. az).
(61)
This means that (s, ap: ao) R 0 and therefore 3 0. When we take the limit of the pilot lower bound as 3 - O, we have:
2 E Banda, ||=o(. s) -pp. (
lim B 8-0
((TT) 2 ((Tp) 33
(62)
which simplifies to
J(T1) 2 J(Tp).
(63)
Similarly for pilot that is far from optimal,
Q(s,ap) min Q (s,aa
(64)
and therefore
Q(s, ap) S Q(s ac).
(65)
This means that (ss, ap. ae) 1 and therefore 3 z 1. When we take the limit of the copilot lower bound as B + 1, we have:
7 ((TI) 2 - B
lim B+1
-
Bandyy || 8) Tpl. |8]]|]
(66)
7
which simplifies to
( 2 ((rc).
(67)
have therefore shown that for near optimal pilot, IA is guaranteed to not degrade performance, and for very poor pilot, IA is guaranteed to do at least as good as the copilot.
B Computational Cost of IDA
We found that the computation time for IDA inference only slightly increase as the size of the goal space increases (Table B.I. In all our experiments, we approximate continuous goal spaces by sampling 1,000 candidate goals which costs about ms. This is sufficiently fast for most real-time control applications. Performance could further be improved by using Monte-Carlo estimates as done in the Faux-Gor experiments.
Time
1
2
3
4
5
10
100
1000
10000
100000
(ms)
2.0
2.1
2.2
2.3
2.3
2.0
2.1
3.1
2.5
11.7
Table B.I: Computation time of Advantage Function v. Number of Goals on single RTX 3080Ti
17

---

C Examples of Real Human Intervention
We found that, for human experiments in Lunar Lander, interventions by IDA are most common at the start of episodes and when touching down (Figure C.Ip. Each episode starts with random disruptive force applied to the rocket so it makes sense that intervention should occur initially to ensure the human maintains stable flight. Additionally, the rocket ship is very sensitive to the speed and orientation when touching down. Interventions near touch down likely serve to prevent collisions with the ground.
Intervention Frequency
Intervention Frequency
Aftitude
Rocketship Altitude (m)
Horizontal Position
Figure C.I: Intervention tends to occur near the start of trajectories to stabalize the rocket and then again near the end of trajectories assist the touch down.
D Human-in-the-Loog Lunar Lander Experimental Design
Eight (2 female, 6 male) participants with no prior experience performed 3 sequences of Lunar Lander experiments, All participants were compensated with a gift card for their time and involvement in the study. Each sequence consisted of 3 experimental blocks (30 episodes per block) of pilot-only, copilot, and IDA control (Figure D.I). An episode of lunar lander timed out after 30 seconds. In total, each participant played 270 episodes of Lunar Lander, 90 for each condition of pilot-only, copilot, and IDA.
30 episodes
IDA copilot pilot
Blook
Blook
Block
Block 4
Block 5
Block
Block 7
Block 6
Block
Sequence
Sequence
Figure D.1: All participants completed 3 sequences of three blocks. Each sequence was composed of a pilot-only, copilot, and IDA control block. Each block consisted 30 episodes.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We rigorously develop the IDA SA framework as described in the introduction
in sections 3.1-3.6. We further empirically demonstrate the utility of our framework in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our approach in the discussion. In particular, we discuss the limitations of training a fully autonomous agent to perform the task and the assumptions that space of possible goals is known.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide a proof of theorem in appendix A. All the preliminaries for Theorem are defined in sections 3.1-3.4
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide a description of our methods in section 3.1-3.5 as well as pro- vided references for methods we adopted. We also provide descriptions the gymnasium environments that we use which are all publically available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
20

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code as supplementary zip file. Polished code will be released with the paper along with set up and installation instructs allow others to reproduce results as well as build future work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
public/guides/ httpeeeeeoeeeeeabaiiooooooooeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurIPS code and data submission guidelines (https
//wips.ccppubllii/ wttpc//pppppgppu.aeeeeeeep guwdee/deeeeee http///oodbbmssiioooooo soiooiiiisiiioiccci....oooooiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Parameters for training provided in sections 3 and 4.1. We specify the agents and environments that we use to perform training.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars corresponding to standard error of the mean are included on plots in figures 2, 3, and
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Training hardware and time is described in Section 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the Code of Ethics to ensure our paper complies with them.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss in the introduction the broader impacts of developing shared autonomy systems and the overall need to develop AT that assists humans and does not repalce humans.
22

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., denloymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Or paper leverages publically available models and datasets and builds a model for performing shared human-Al control. No new foundation models or datasets are released with our research.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the methods we adapt for training various parts of IDA cited in section 3.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
23

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Experimental set-up for human experiments described in section 4.4. Com- pensation is also reported in section 4.1
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We obtained consent from all subjects and all experiments were IRB approved.
Guidelines:
24

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "1x9oFdiCz3", "reviewer": "Reviewer_Y2yZ", "summary": "This paper presents an intervention assistance (IA) method, IDA, that dynamically decides whether the co-pilot should take over the control. The decision is determined by comparing the expected values of the co-pilot’s action versus the pilot’s action. The experiments with human surrogates showed that the proposed IA could improve task performance in the Reacher and Lunar Lander. The human study shows that human subjects prefer IDA.", "questions": "- How is the estimation of Q-value learned in the experiment? Using the same SAC expert or a separate Q-network on the collected data? It is unclear how it is trained and what data it requires.\n- If the estimated Q-value is too optimistic or too pessimistic, how does it affect the performance of shared autonomy?\n- In Reacher, the laggy action doesn’t affect much about the performance, one explanation can be the domain doesn’t require high-precision control, thus less useful when considering shared control.", "limitations": "- The experiments are done with simulated experts, even in real human experiments. In the real world, some domains do not have such simulated/RL experts. How the method transfers or applies to those domains when no simulated/RL is available is not clear.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method can dynamically adjust whether the copilot policy should intervene to improve ease of use and human autonomy.\n- The proposed method follows a goal-agnostic copilot policy, thus can generalize to unseen goal locations.", "weaknesses": "- The proposed method still seems to rely on offline datasets to estimate the Q-value. So, the quality of the estimated Q-value is important to the intervention strategy. There is no evidence or discussion on how the performance or impact on users when the expected Q-value is not good enough.\n- In the experiments with simulated human surrogates and real humans, the baseline is using the pilot-only or always with a copilot. These two are the most basic baselines. However, since the paper proposes intervention strategies, a better comparison should be the paper such as Tan et al. 2022 where an intervention budget is set. Without such comparison, it is unclear how the proposed intervention advantage is compared with an existing or naive strategy to limit intervention numbers.\n   - Weihao Tan, David Koleczek, Siddhant Pradhan, Nicholas Perello, Vivek Chettiar, Vishal Rohra, Aaslesha Rajaram, Soundararajan Srinivasan, H M Sajjad Hossain, and Yash Chandak. On optimizing interventions in shared autonomy, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to carefully read through and understand our paper, and provide constructive feedback. We’ve made important changes in response to your feedback (including new experiments) that we believe have significantly improved the manuscript.\n\n> The proposed method still seems to rely on offline datasets to estimate the Q-value. So, the quality of the estimated Q-value is important to the intervention strategy. There is no evidence or discussion on how the performance or impact on users when the expected Q-value is not good enough.\n\nWe want to clarify our method does not impose any requirements on how an expert Q function is obtained. In our experiments, it is computationally easy to train a SAC agent in the environment to obtain an expert Q function. However, learning in online simulation or offline data are both equally viable options. We also clarify that IDA depends on obtaining a good Q-value estimator and the performance of IDA will degrade as the quality of the Q-value estimator degrades. We have further incorporated discussion of these factors in the manuscript.  \n\n> However, since the paper proposes intervention strategies, a better comparison should be the paper such as Tan et al. 2022 where an intervention budget is set. Without such comparison, it is unclear how the proposed intervention advantage is compared with an existing or naive strategy to limit intervention numbers.\n\nThank you for raising this point. We performed additional experiments comparing to Tan et al., 2022 (“Intervention-Penalty (MLP)”), as well as Reddy et al., 2018 (“Copilot (MLP”) in the Rebuttal pdf, Table 3, which will be added to the manuscript. We find that IA is always higher performing than the penalty-based assistance proposed by Tan et al., 2022.\nRegarding what comparison is best, we want to highlight some important differences with Tan et al., 2022. \n1) **IA does not need access to the human control policy.** Penalty-based intervention assumes we can query the human control policy while training the copilot. IA does not. \n2) **IA better generalizes to continuous action spaces** because the penalty-intervention proposes a penalty whenever the copilot and human actions are different. However, for continuous action spaces the actions are unlikely to ever be equal, necessitating an additional hyper-parameter to decide if the copilot and human actions are sufficiently different to warrant a penalty. \n3) **IA has no hyperparameters.** The vast majority of previous work in shared autonomy relies on tuning hyper-parameters to optimize control sharing or requiring the copilot to implicitly learn to limit its own assistance (Reddy et al., 2018, Yoneda et al., 2023, Tan et al., 2022, Jeon et al., 2020, Schaff et al., 2020). Tan et al., 2022 requires setting a hyperparameter penalty or budget to determine how the copilot should trade off assistance and autonomy, although it can be solved via optimization.  \n\nWe will incorporate these new analyses and discussion into the manuscript \n\n> How is the estimation of Q-value learned in the experiment? Using the same SAC expert or a separate Q-network on the collected data? It is unclear how it is trained and what data it requires.\n\nIt is learned using the same SAC expert on the collected data. We emphasize our method does not require the Q-value estimator be learned in a particular way. One could also train a Q value from demonstrations collected by a human or heuristic control policy in an online setting or using offline RL methods like Conservative Q Learning [1] or preference-based RL methods like Conservative Preference Learning [2]  to estimate the Q value from the offline data. We chose SAC because it is straightforward in our environments. We will clarify this in the manuscript.\n[1]: Conservative Q-Learning for Offline Reinforcement Learning\n[2]: Contrastive Preference Learning: Learning from Human Feedback without RL\n\n> If the estimated Q-value is too optimistic or too pessimistic, how does it affect the performance of shared autonomy?\n\nIA computes an advantage score that is the difference in Q-values between the copilot and  pilot. Since we consider only relative differences we are resilient to any biases in the Q-value.\n\n> In Reacher, the laggy action doesn’t affect much about the performance, one explanation can be the domain doesn’t require high-precision control, thus less useful when considering shared control.\n\nIn Table 1 in our manuscript, the laggy action does degrade performance from 18.8 targets/min to 8.5 targets/min, which is 45% of the initial performance. Given this, we believe the reviewer may have been referring to IDA performance for laggy actions in Reacher. We believe this squares with the intuition that, in Reacher, most of the time laggy actions are not universally bad actions. However, even in this case where IDA offers little assistance we find that IDA is safe to use and does not degrade the laggy pilot’s performance. In Lunar Lander, where laggy actions can be more detrimental, IDA more significantly increases performance.\n\n> The experiments are done with simulated experts, even in real human experiments. In the real world, some domains do not have such simulated/RL experts. How the method transfers or applies to those domains when no simulated/RL is available is not clear.\n\nWe wish to clarify that there is no explicit need in our methodology to use simulated experts. Simulated experts were easily trainable for our environments so we made the choice to use them. We follow the methods of Yoneda et al., 2023 which used simulated experts due to computational ease, however, they could have been trained from demonstration data without a simulated RL expert. Depending on the use case, demonstration data collected from real humans, a heuristic control policy, simulated experts, or a combination of the three are all reasonable options for obtaining a Q-value estimator. We will clarify this in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8RFni2ALio", "reviewer": "Reviewer_yfMr", "summary": "This work presents the *intervention diffusion assistance* (IDA) framework for shared autonomy between a human \"pilot\" and an AI \"co-pilot\".  The IDA framework is designed to be goal agnostic, and does not attempt to infer the pilot's current goal.  This work extends Yoneda et al. (2023), using the same diffusion-based denoising policy, but now only applying the policy in states where the underlying expert policy is believed to be superior to the human pilot's policy.  Their experimental results (with both learned and human pilots) show that this selective denoising policy dominates both the pilot policy and the pilot's policy when corrected using the denoising policy at each step.\n\nReferences:\n1. Yoneda, Takuma, et al. \"To the noise and back: Diffusion for shared autonomy.\" arXiv preprint arXiv:2302.12244 (2023).", "questions": "1. It was somewhat surprising that in the reacher environment (figure 2) the original denoising policy performs extremely poorly when applied to every state, particularly given its success in the aparently similar box-pushing task in the original work (Yondea et al. 2023).\n2. Equation 6 suggests that the advantage must be exactly equal to 1 in order for an intervention to occur, which would seem to be a very rare occurence.  Is this correct, or just a typo?", "limitations": "A key limitation of this work that deserves more attention is the need to provide a prior goal distribution, and a mechanism for \"masking\" goal information in the observations.  Both the prior and the masking function would seem to require significant domain knowledge, and might be difficult to implement in real-world tasks.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strength of this work is the apparent effectiveness of the selection mechanism that chooses when the co-pilot is allowed to intervene and modify the human pilot's actions.  Experimental results suggest that this mechanism is able to restrict interventions to those states where the co-pilot's policy is most competent, and avoids failures due to \"gaps\" in the expert's policy.", "weaknesses": "One potential weakness with this work is that the intervention selection mechanism depends on knowledge of the goal distribution for the task (as well as a function for \"masking\" goal information contained in the observations).  Thes would need to be implemented separately for each environment we wish to apply IDA to, and require significant domain expertise.\n\nAnother issue is Theorem 1.  The assumptions under which Theorem 1 holds are unclear.  In particular, there appears to be no assumption that the intervention policy is superior to the human policy.  This would seem to be difficult to ensure in general, as the intervention policy is trained over a distribution of goals.  It seems possible that the bounds in Equation 7 would not hold for \"low probability\" goals.\n\nAdditionally, as it was explicitly mentioned that the proposed approach is compatible with any shared autonomy architecture, it would have been nice to see evaluations with the selection mechanism applied to other architectures besides the diffusion-based policy of Yoneda et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to carefully read our paper and provide detailed feedback. We have performed additional experiments in response to your invaluable feedback, which we believe has further enhanced and strengthened our work.\n\n> One potential weakness with this work is that the intervention selection mechanism depends on knowledge of the goal distribution for the task (as well as a function for \"masking\" goal information contained in the observations). Thes would need to be implemented separately for each environment we wish to apply IDA to, and require significant domain expertise.\n\nWe first hope to clarify that goal masking and assuming a fixed discrete set of goals is a common practice in the shared autonomy literature (Yoneda et al., Dragon et al., Reddy et al., 2018, Tan et al., 2022). While a recent study (Du et al., 2020) avoids goal inference, it does not perform as well. \nThis said, IDA has important advantages over previous work that we hope address your concerns over limiting goal assumptions in the shared autonomy literature:\n\n**IDA works for continuous (infinite) goal spaces.** Although we demonstrated this in our initial submission by testing Reacher with continuous targets, we re-emphasize it here.\n\n**For IDA, the goal space does not need to be known in advance and can be provided at inference.** IDA does not need to know the goal space to be trained. To demonstrate this, we show new experiments where IDA extends to goal spaces it was not trained on. We specifically test a domain shift (DS) environment where the agent was only trained on 5 discrete goals, but generalized to continuous goals.\n\n**IDA works when the goal space is unknown.** To achieve this, we construct a “faux goal” space (FG) by assuming a uniform distribution over potential next positions as goals and achieve a Monte-Carlo estimate of the advantage through sampling. We demonstrate in new experiments that IDA increases performance even in this setting.   \n\nWe perform additional evaluations of Reacher  (Table 1 in rebuttal pdf) in a continuous goal space, where targets can appear anywhere, as well as goal spaces not observed in training (Linear: goals appear continuously on a line; Quadrant: goals appear in a subset of the space). These results are shown in Table 1 on the attached pdf. IDA increased performance of the noisy pilot in all scenarios, demonstrating that the goal space does not need to be known in advance and that IDA improves performance (1) without a priori knowledge of the goal space (IDA (DS)) and (2) when the goal space is not known (IDA (FG)). We will update the manuscript to include these analyses and include discussion on these advantages in IDA.\n\nAlthough our experiments show IDA works when the goal space is not prior known and in continuous goal spaces, we do emphasize that in many real world applications, the goal space can be constrained (e.g. location of all graspable objects for a robotic arm with a gripper).\n\n**On goal masking:** In general, when using environments which provide full observations (such as these gym environments) goal masking will be necessary, otherwise the copilot can “cheat” to perform the task goal without inferring it from the user. However, for environments where only partial observations are provided, as is the case in many real world environments, goal masking is no longer necessary.\n\n> Another issue is Theorem 1. The assumptions under which Theorem 1 holds are unclear. \n\nWe greatly appreciate your attention to detail here and have decided to restate our theorem and parts of our proof for clarity. In addition to clarifying our theorem, we have revised our theorem to provide two lower bounds on the performance guarantees of IA (instead of just the one that uses the return of the pilot). We have restated the theorem in the attached pdf with assumptions more clearly stated.\n\n> it would have been nice to see evaluations with the selection mechanism applied to other architectures besides the diffusion-based policy of Yoneda et al.\n\nWe added additional baselines using an MLP-based copilot that is similar in architecture to Reddy et al., 2018. We have also compared our approach (IA) with the penalty-based intervention proposed by Tan et al., 2022. (Table 3 in rebuttal pdf)\n\n> It was somewhat surprising that in the reacher environment (figure 2) the original denoising policy performs extremely poorly when applied to every state, particularly given its success in the aparently similar box-pushing task in the original work (Yondea et al. 2023).\n\nThis performance difference is because the Block Pushing environment used by Yoneda et al., 2023 used only two large discrete goal locations that never changed. We use a continuous distribution of goals in a 2D plane. \n\n> Equation 6 suggests that the advantage must be exactly equal to 1 … Is this correct…?\n\nThis is a correct. The advantage is  the fraction of goals where the copilot’s action is superior to the pilot action. Intervention should only occur when the human takes actions that are bad for all possible goals. We will further clarify this in the manuscript.\n\n**Limitations:**\n\n> A key limitation of this work that deserves more attention is the need to provide a prior goal distribution, and a mechanism for \"masking\" goal information in the observations…  [which] might be difficult to implement in real-world tasks.\n\nTo summarize, we have demonstrated that IDA does not require a prior goal distribution. The goal distribution can be changed at inference, and does not even have to be explicitly known (IDA (FG)). Goal masking is necessary in fully observed environments (such as gymnasium) to ensure the copilot does not cheat and infers the user’s goal, but would not be necessary in partially observed environments where the goal is only in the user’s mind (such as reaching to a particular object amongst several).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tsOgeO43XI", "reviewer": "Reviewer_zqQF", "summary": "In the Shared Autonomy context, the authors propose a value-based intervention assistance method that aims to only have a copilot intervene only when the value of the assistant’s actions exceed that of the human. The proposed method first trains an expert policy for the task using privileged goal information, whose actions are then used to train a diffusion copilot for shared autonomy with the goal information removed. To determine if the copilot's actions should be used instead of the pilot, the method proposes a copilot advantage score, which computes the difference in q-value between the copilot and pilot actions marginalized across all possible goals in the environment. The efficacy of the method is demonstrated with both simulated human pilots and real human-in-the-loop experiments.", "questions": "- Can you clarify what \"exploring starts\" means for training the SAC expert?\n- Have the authors studied how increasing or decreasing the goal space size affects the usability of the method? For example, if we decrease / increase the number of total goals in the reacher environment, are there measurable impacts on the performance of the method? It would be insightful to see a plot showing change in computational efficiency as the number of goals change.", "limitations": "The authors do address the limitations of the work. Further discussion on potential societal impact would be helpful (e.g. what are the implications if agents are unable to predict real human goals accurately?)", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- To the best of my knowledge, the proposed method is novel and differs sufficiently from relevant prior work with the proposal of the value-based intervention system. Related methods that the paper builds upon are cited, and the experiments compare against the relevant baseline method.\n- The experiment setup follows the standard established by similar prior work in deep RL for shared autonomy, including both simulated and human experimenents. \n- Generally the submission is well written and clear to read.", "weaknesses": "- While Section 2 (Related Works) currently covers more recent relevant work, it would benefit from also providing a more general overview of the shared autonomy literature (e.g. providing an introduction to the shared autonomy and deep RL setting, such as described in \"Shared autonomy via deep reinforcement learning\" (Reddy et al., 2018) and relevant robotic teleoperation work, such as \"A policy-blending formalism for shared control\" (Dragan & Srinivasa, 2013)).\n- While the proposed method is intuitive, practically it seems difficult to implement since (1) an expert policy needs to be trained beforehand, and (2) the advantage computation at every timestep requires knowledge of the full goal space. A general challenge of the shared autonomy setting is that it is often difficult to capture the full goal space a human has in mind, and it's not clear how the proposed method will work if the agent does not know the full goal space beforehand. \n- Having to marginalize out all goals at every timestep for the advantage computation seems quite expensive, especially for higher dimensional goal spaces.\n- Figure 3c is quite interesting! it would be strengthened by adding more quantitative evaluation of the findings in Figure 3c (i.e. that the IDA copilot actively intervenes against negative behaviours). For example, computing the dot product between the copilot action and the pilot action vectors to see how much the copilot action differs from the pilot during intervention actions.\n- While the user study results show the improvement provided by the method, the sample size of five participants is quite small. I would encourage expanding this sample size and reporting the significance (e.g. p-values) of each of the corresponding metrics (see the similar comparison done in prior work, https://arxiv.org/pdf/1802.01744) .\n- Minor comment: renaming $Q^{\\pi_e}$ to $\\mathbf{I}$ seems unecessary, the former is more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your effort to thoroughly review our paper and for your feedback. In response to your feedback, we have made meaningful improvements that have strengthened the study. \n\n> While Section 2 (Related Works) currently covers more recent relevant work, it would benefit from also providing a more general overview of the shared autonomy literature.\n\nWe will rewrite text to include a more general overview of shared autonomy.\n\n> While the proposed method is intuitive, practically it seems difficult to implement since (1) an expert policy needs to be trained beforehand, and (2) the advantage computation at every timestep requires knowledge of the full goal space. A general challenge of the shared autonomy setting is that it is often difficult to capture the full goal space a human has in mind, and it's not clear how the proposed method will work if the agent does not know the full goal space beforehand.\n\nWe have performed new experiments to show that knowledge of the full goal space **is not required**.  In Reacher experiments, we implemented IDA (FG), using a “Faux Goal” (FG) space where we assume a uniform distribution of goals around the Reacher’s fingertip. We randomly sample goals from this distribution at each inference step. This assumption reflects that in real world scenarios, while the goal locations may be unknown, the user must move the arm in a continuous fashion and therefore any future state near the end effector is a reasonable short term candidate goal. We show that, even without goal space knowledge, IDA improves performance on the Reacher environment for the noisy pilot and maintains the performance of the laggy surrogate control policy. We added this in Table 1 of the rebuttal pdf and will also update the manuscript to include results using this faux goal space.\n\n>Having to marginalize out all goals at every timestep for the advantage computation seems quite expensive, especially for higher dimensional goal spaces. Have the authors studied how increasing or decreasing the goal space size affects the usability of the method? For example, if we decrease / increase the number of total goals in the reacher environment, are there measurable impacts on the performance of the method? It would be insightful to see a plot showing change in computational efficiency as the number of goals change.\n\nWe added Table 2 in the rebuttal pdf showing computation time on a single RTX 3080Ti GPU as a function of the goal space size. We find that the computation time only slightly increases as we increase the goal space and it remains under 12ms for goal spaces up to 100,000 candidate goals. This provides up to 80 Hz control, sufficient for most real time experiments. In our experiments we approximate continuous goal spaces by sampling 1,000 candidate goals and we never exceed 1,000. Computation time could be accelerated through using fewer samples, or, as we showed in FG experiments, using Monte-Carlo estimates. We will add this table to the Appendix.\n\n> Figure 3c is quite interesting! it would be strengthened by adding more quantitative evaluation of the findings in Figure 3c (i.e. that the IDA copilot actively intervenes against negative behaviours). For example, computing the dot product between the copilot action and the pilot action vectors to see how much the copilot action differs from the pilot during intervention actions.\n\nThank you for these suggestions! We have added an additional appendix figure that shows intervention as a function of rocketship altitude and rocketship spatial position (Figure 1 in the rebuttal pdf). This figure shows that interventions typically occur at the very beginning of episodes to stabilize the flight and then increase in frequency at the end of episodes to stabilize the touch down. \n\n> While the user study results show the improvement provided by the method, the sample size of five participants is quite small. I would encourage expanding this sample size and reporting the significance (e.g. p-values) of each of the corresponding metrics (see the similar comparison done in prior work, https://arxiv.org/pdf/1802.01744) .\n\nAt the current time we have run additional human experiments and now have a sample size of eight human subjects. We found that humans subjectively prefer IDA to baseline copilot assistance (p < 0.001, Wilcoxon signed-rank) and we also find that humans have a significantly higher success rate with IDA (p < 0.001, Wilcoxon signed-rank) than with baseline copilot control. We have added this additional data along with p-values to our manuscript and figure legends. \n\n> Can you clarify what \"exploring starts\" means for training the SAC expert?\n\nExploring starts means that the initial state of the environment is randomized for each training episode, and is a common way to improve exploration in reinforcement learning. We use exploring starts as it improves SAC training. We will clarify this in the manuscript.\n\n> The authors do address the limitations of the work. Further discussion on potential societal impact would be helpful (e.g. what are the implications if agents are unable to predict real human goals accurately?)\n\nIA does not directly infer human goals but instead marginalizes over the space of all possible goals. In cases where no constraints about the goal space are available faux goal spaces can be constructed which essentially just enables IA to prevent universally bad states from occurring. We will add this to our discussion to emphasize that IA can generalize to virtually any real world setting without restricting the users actions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their attention and thoughtful comments. We believe this has helped lead to a cleaner and improved manuscript.

We have responded to each of the reviewer's comments individually. Please see the attached pdf for tables and a figure that we refer to in reviewer-specific responses. 

We do wish to highlight here that we have performed two additional experiments:

1. We have added baselines from Reddy et al., 2018 and Tan et al., 2022 and compared them to IA in Lunar Lander and shown that IA is always the highest performing. 

2. We have conducted additional experiments in the Reacher environment where the goal space is different between training and inference time (domain shift) or unknown at inference time (faux goals).

Both of these additional results will be updated in our manuscript.

Meta-review: This paper introduces a novel and effective method for the value-based intervention system.
The experimental validation confirms the main claims. The submission is clearly written and organized. 
Moreover, the human study shows that human subjects prefer the proposed method.
Decision: Accept (poster)