Title: Shared Autonomy with IDA: Interventional Diffusion Assistance
Venue: NeurIPS 2024
Accepted: yes
Keywords: Shared Autonomy

Diffusion Models

copilots

intervention reinforcement learning

reinforcement learning

lunar lander

Mujoco
Abstract: The rapid development of artificial intelligence (AI) has unearthed the potential to assist humans in controlling advanced technologies. Shared autonomy (SA) facilitates control by combining inputs from a human pilot and an AI copilot. In prior SA studies, the copilot is constantly active in determining the action played at each time step. This limits human autonomy that may have deleterious effects on performance. In general, the amount of helpful copilot assistance varies greatly depending on the task dynamics. We therefore hypothesized that human autonomy and SA performance improves through dynamic and selective copilot intervention. To address this, we develop a goal-agnostic intervention assistance (IA) that dynamically shares control by having the copilot intervene only when the expected value of the copilot’s action exceeds that of the human’s action. We implement IA with a diffusion copilot (termed IDA) trained on expert demonstrations with goal masking. We prove that IDA performance is lower bounded by human performance, so that IDA does not negatively impact human control. In experiments with simulated human pilots, we show that IDA achieves higher performance than both pilot-only and traditional SA control in variants of the Reacher environment and Lunar Lander. We then demonstrate with human-in the-loop experiments that IDA achieves better control in Lunar Lander and that human participants experience greater autonomy and prefer IDA over pilot-only and traditional SA control. We attribute the success of IDA to preserving human autonomy while simultaneously offering assistance to prevent the human from entering universally bad states.

Shared Autonomy with IDA: Interventional Diffusion Assistance

Abstract

The rapid development of artificial intelligence (AI) has unearthed the potential to assist humans in controlling advanced technologies. Shared autonomy (SA) facilitates control by combining inputs from a human pilot and an AI copilot. In prior SA studies, the copilot is constantly active in determining the action played at each time step. This limits human autonomy and may have deleterious effects on performance. In general, the amount of helpful copilot assistance can vary greatly depending on the task dynamics. We therefore hypothesize that human autonomy and SA performance improve through dynamic and selective copilot intervention. To address this, we develop a goal-agnostic intervention assistance (IA) that dynamically shares control by having the copilot intervene only when the expected value of the copilot's action exceeds that of the human's action across all possible goals. We implement IA with a diffusion copilot (termed IDA) trained on expert demonstrations with goal masking. We prove a lower bound on the performance of IA that depends on pilot and copilot performance. Experiments with simulated human pilots show that IDA achieves higher performance than pilot-only and traditional SA control in variants of the Reacher environment and Lunar Lander. We then demonstrate that IDA achieves better control in Lunar Lander with human-in-the-loop experiments. Human participants report greater autonomy with IDA and prefer IDA over pilot-only and traditional SA control. We attribute the success of IDA to preserving human autonomy while simultaneously offering assistance to prevent the human pilot from entering universally bad states.

1 Introduction

As technology advances, humans continuously seek to operate more sophisticated and complex devices (Cascio and Montealegre, 2016). However, more sophisticated technologies typically involve complicated operational dynamics and high-dimensional control systems that restrict their use to narrowly defined environments and highly specialized operators (Schulman et al., 2018). While fully autonomous AI agents can be trained to perform these tasks, this approach has three key limitations. First, the user's goal is internalized and not easily deducible in most real-world environments. Second, removing the user from the control loop reduces their autonomy, potentially leading to poor performance and decreased engagement (Wilson and Daugherty, 2018a,b). Third, as the capabilities of AI advance, it is important to consider how to create technologies that assist and empower humans instead of replacing them (Wilson and Daugherty, 2018a,b).

Shared autonomy (SA) addresses these limitations by blending human (pilot) actions with assistive agent (copilot) actions in a closed-loop setting. Prior studies demonstrate SA can increase human task performance in robotic arm control (Laghi et al., 2018), drone control (Reddy et al., 2018), and navigation (Peng et al., 2023). A critical component of prior work is an empirically tuned controlsharing hyperparameter that trades off copilot assistance with human user autonomy (Reddy et al.,

s

Environment

Goal

masking

s

~

Copilot

Human

(implicit goal)

Intervention

ap

ac

aI

Interventional Assist

Interventional Assist (IA), this work

b

s

Environment

Goal

masking

s

~

Copilot

Human

(implicit goal)

ap

ac

Shared autonomy, prior works a

2018; Jeon et al., 2020; Yoneda et al., 2023). Excessive assistance can hinder goal achievement, while insufficient assistance can lead to poor control and performance. Prior work has established multiple methods of control sharing, but limitations remain, such as requiring control sharing hyperparameters be tuned empirically or limiting the copilot to general, non-goal-specific assistance (Schaff and Walter, 2020; Du et al., 2021; Tan et al., 2022). We will discuss them further in Section 2.

Imagine driving a car with an assistive copilot. The majority of the time, the human driver should remain in control. However, the copilot should intervene in certain scenarios to prevent collisions and ensure safe driving. By dynamically adjusting the level of assistance based on the situation and needs of the driver, the copilot simultaneously preserves driver autonomy and engagement while ensuring safety. This example reflects an important open problem: how do we construct an optimal behavior policy from the human and copilot's individual policies? This problem is conceptually similar to the one posed by the Teacher-Student Framework (TSF) (Zimmer et al., 2014), where a teacher agent helps the student learn a good policy by intervening to prevent the student from visiting deleterious states and providing online guidance (Kelly et al., 2019; Li et al., 2022; Peng et al., 2021). A recent work (Xue et al., 2023) has developed methods that improve the TSF framework by using a trajectory-based value estimate to decide when the teacher intervenes in student learning. We propose that a similar value estimate could be used to determine when the copilot should 'intervene' in the user's control, providing the user with assistance. Therefore, we develop an intervention function that estimates the expected value of the copilot and human action in a goal-agnostic fashion. Because our formulation is goal-agnostic, it can generalize to goals not seen during the training process, endowing the human pilot with improved flexibility and generalization.

Our main contribution is a shared control system that leverages a value-based intervention function that can interface with many different copilot architectures while simultaneously improving performance and preserving pilot autonomy. In this work, we build on an existing diffusion-based copilot architecture (Yoneda et al., 2023). These copilots are desirable because they can be trained using supervised methods which helps mitigate the sample complexity required to train effective copilots. While we proceed with the diffusion copilot of (Yoneda et al., 2023) we emphasize that our intervention function can be applied to any copilot capable of generating assistive, alternative, or corrective actions to aid the human in complex tasks.

2 Related Work

We have a brief discussion on three related works (Du et al., 2021; Yoneda et al., 2023; Tan et al., 2022).

Assistance via Empowerment (Du et al., 2021): It proposes a method that increases a human's ability to control the environment and mitigate the need to infer any goals. It defines an information

theoretic quantity that captures the number of future states accessible to a human from the current state. An assistive agent is then trained to maximize this quantity while the human performs the task. While this allows agents to assist in a goal-agnostic fashion, it typically leads to lower performance than methods that infer the goal, since the assistive agent does not directly help the human achieve a goal. In contrast, other methods, including this work, leverage a copilot that implicitly infers the goal from human actions. While goal inference can lead to lower performance when the goal is incorrectly inferred, we mitigate this by restricting the copilot to only intervene when the human action is deleterious, i.e., worse than the copilot action across all possible goals.

To the Noise and Back: Diffusion for Shared Autonomy (Yoneda et al., 2023): It develops a copilot that uses diffusion to map a human action closer to an expert's action. They train a diffusion process to generate actions from a distribution of expert actions conditioned on goal-agnostic state observations. At inference, the human action is first pushed towards a Gaussian distribution by adding noise in the forward diffusion process. The reverse diffusion process is then run on this noised human action to transform it into a sample from the expert action distribution. This action is played in the environment, as illustrated in Figure 1a. The fraction of the forward diffusion process applied to the human action is the diffusion ratio γ ∈ [0 , 1] and trades off action conformity (how similar the action is to expert actions) and human autonomy. This copilot therefore requires an experimentally tuned hyperparameter to influence how much control the human has on the eventual action. Because the amount of assistance needed may vary greatly depending on task dynamics, using a fixed amount of assistance throughout the task may limit performance and autonomy. Our intervention function presented in Section 3 addresses this limitation by allowing the copilot to dynamically intervene based on the human's actions.

On Optimizing Interventions in Shared Autonomy (Tan et al., 2022): It proposes a method where a copilot is trained with a penalty for intervention. This encourages the copilot to limit its intervention and preserve human autonomy. However, intervention is not inherently bad and need not be sparse. Instead of uniformly penalizing all intervention, only unnecessary intervention should be penalized. Additionally, this method is not inherently hyperparameter free as it does require setting a penalty hyperparameter to determine how the copilot should trade off assistance and autonomy, although they demonstrate this hyperparameter can be solved via optimization. Another limitation is this method assumes access to the human policy during training so that the copilot can learn when best to intervene for a particular human (e.g., an expert at the task would likely have less copilot intervention than a novice). In contrast, we define a general-purpose intervention function based on how good actions are in the environment (irrespective of the specific human pilot playing the actions), enabling one-time training of the intervention function. In our intervention function, an expert would still experience less intervention than a novice because the expert generally plays better actions in the environment. Empirically we find that with our approach the same intervention function can boost the performance of eight different human participants in the Lunar lander environment.

3 Method

Our goal is to introduce a general-purpose intervention function that increases the performance and human autonomy of an SA system. The inputs to the intervention function are: (1) the goal-agnostic environment state, ˜ s , (2) a pilot action, a p , and (3) a copilot action, a c , illustrated in Figure 1b. The intervention function then plays either the pilot action ( a I = a p ) or copilot action ( a I = a c ) in the environment. We define this intervention function and describe its implementation. We also prove a lower bound on the expected return associated with the policy using the intervention function proportional to the pilot and copilot performance.

We develop an intervention assistance called Interventional Diffusion Assistance (IDA). First, an expert policy is trained to maximize returns in the environment (Section 3.2). Second, the expert is used to perform policy rollouts and generate demonstrations. Goal information is removed from the demonstration data. We then train a diffusion copilot from these demonstrations (Section 3.3). Third, we define a trajectory-based intervention function that decides whether to play the human or copilot action (Section 3.4). All training was performed on a workstation with a single 3080Ti and took approximately 48 hours to complete all three steps for our tasks.

3.1 Notation and Problem Formulation.

We assume the environment can be modeled as an infinite-horizon Markov Decision Process (MDP) defined by the tuple M = ⟨ S, A, R, γ, P, d 0 ⟩ . S is the space of all possible environment states, and A is the space of all possible actions. R : S × A → [ R min , R max ] is a scalar reward received for playing an action a ∈ A in state s ∈ S . P : S × A × S → [0 , 1] are the transition dynamics for the environment, γ is the discount factor, and d 0 is the distribution of initial states. We define the state-action value function induced by policy π to be Q π ( s, a ) = E s 0 = s,a 0 = a,a t ∼ π ( ·| s t ) ,s t +1 ∼ p ( ·| s t ,a t ) [ ∑ ∞ t =0 γ t r ( s t , a t )] , where π : S × A → [0 , 1] is the action distribution conditioned on the state.

We additionally introduce the notion of a goal that encodes the task objective or intention of the human. We can decompose any state s = ⟨ ˜ s | ˆ g ⟩ into a partial goal-agnostic state observation ˜ s , which does not contain any goal specific information, and a goal ˆ g ∈ G ∗ , where G ∗ is the space of all possible goals. Then Q π ( s, a ) = Q π ( ⟨ ˜ s | ˆ g ⟩ , a ) = Q π (˜ s, a | ˆ g ) is the state-action value function under the goal-agnostic state ˜ s and goal ˆ g .

We model the human behavior by a human pilot policy , π p , which generates human action a p ∼ π p ( ·| s ) according to the full state observation. The human observes and therefore has access to the goal, ˆ g ∈ G ∗ . However, the goal is not assumed to be accessible to the copilot. In this paper, we assume the availability of an expert policy π e ( a e | s ) , that observes the full state and solves the environment. We also assume that we can query the state-action value of the expert policy Q π e ( s, a ) , with which we use to evaluate the quality of an action a . Wewill train a copilot policy π c that generates actions based on the pilot action and the goal-agnostic state a c ∼ π c ( ·| a p , ˜ s ) . The ultimate goal of this paper is to derive a goal-agnostic intervention function T (˜ s, a c , a p ) ∈ { 0 , 1 } from the expert policy so that the SA system can achieve better performance than the pilot alone. The behavior policy that shares autonomy between the pilot and the copilot can be represented as π I = T π c +(1 -T ) π p .

3.2 Training an Expert Policy

We train a soft actor-critic (SAC) expert to solve the environment (Haarnoja et al., 2018) because it allows us to (1) later query Q π e ( s, a ) for intervention and (2) generate demonstrations in the environment that can be used to train the copilot. In general, other methods of obtaining a Q -value estimator and for training a copilot are compatible with IA. We choose SAC for computational convenience. We parameterize our SAC model with a four-layer MLP with 256 units in each layer and the ReLU non-linearity. We use a learning rate of 3 × 10 -4 and a replay buffer size of 10 6 . The expert fully observes the environment including the goal, and is trained for 3 million time steps or until the environment is solved. We found that training with exploring starts (randomized initial state) produced a more robust Q function with better generalization abilities. Without exploring starts, the Q function performed poorly on unseen states, limiting the effectiveness of IDA.

3.3 Training a Diffusion Copilot

Following Yoneda et al. (2023), we trained a diffusion copilot π c ( a c | a p , ˜ s ) using a denoising diffusion probabilistic model (DDPM) (Ho et al., 2020). For each environment, we collected 10 million stateaction pairs from episodes using the SAC expert. All goal information was removed from this demonstration data. The copilot learned to denoise expert actions perturbed with Gaussian noise, conditioned on the goal-agnostic state ˜ s and the pilot's action a p .

Formally, the forward diffusion process is a Markov chain that iteratively adds noise ϵ ∼ N (0 , I ) according to a noise schedule { α 0 , α 1 , ..., α T } to an expert action a 0 , via

Following the forward diffusion process, the diffusion copilot is then trained to predict the noise added by the forward process by minimizing the following loss (Ho et al., 2020):

where ϵ θ is a neural network parameterized by θ that approximates the noise ϵ conditioned on the noisy action a t , the goal-agnostic state ˜ s , and the diffusion timestep t . τ is the distribution of states in the demonstration data. The reverse diffusion process is modeled by a four-layer MLP to iteratively refine a t toward a 0 .

Algorithm 1 SA with IDA

1: Initialize environment

2: for each timestep in episode do

3: Sample state s and goal-masked state ˜ s from environment.

4: Sample human action a p ∼ π p ( · | s ) .

5: Sample diffusion copilot action a c ∼ π c ( ·| a p , ˜ s ) .

6: Compute the copilot advantage score A (˜ s, a c , a p )

7: if A (˜ s, a c , a p ) = 1 then

8: Play copilot action in environment, a I = a c .

9: else

10: Play pilot action in environment a I = a p .

11: end if

12: end for

13: environment reset

3.4 Trajectory-based Goal-agnostic Value Intervention

IDA allows the copilot to intervene in pilot control when they take actions that are consistently bad for all possible goals. We therefore play the copilot's action a c instead of the pilot's action a p when the copilot's action has a higher expected return under the expert Q -value, that is,

However, we can not directly assess Equation 3, since in a SA system the goal is internal to the pilot. Instead, we only have access to the goal agnostic state ˜ s , such that, Equation 3 becomes,

We can define an intervention score I ( ˜ s t , ˜ a t | ˆ g ) which considers the value of ( ˜ s t , ˜ a t ) under the assumption that ˆ g ∈ G ∗ is the goal, where G ∗ is the space of all possible goals,

By marginalizing the difference in the intervention scores between the copilot and pilot over the entire goal space we can define a copilot advantage A (˜ s, a c , a p )

where F is a function that maps the difference in intervention scores to {-1 , +1 } to ensure all possible goals are weighted equally. Here we choose F ( · ) = sign ( · ) . G is a normalization constant for the integral,

When F ( · ) = sign ( · ) , then A (˜ s, a c , a p ) ∈ [ -1 , +1] is proportional to the fraction of the goal space over which the copilot action is superior to the pilot action. Also, when F is the sign function, the normalization constant reduces to G = ∫ ˆ g ∈ G ∗ d ˆ g and if the goal space is discrete then G = | G ∗ | is the number of goals.

We adapt the value based intervention function proposed by Xue et al. (2023) to use for shared autonomy by allowing intervention to occur when A (˜ s, a c , a p ) = 1 . The copilot therefore intervenes when its action has a higher expected return compared to the pilot action for all possible goals. Formally, we let

with intervention policy, π I = T π c +(1 -T ) π p . The process for performing Shared Autonomy (SA) with IA is highlighted in Algorithm 1. The copilot advantage is computed at every timestep. The behavioral (IA) policy is then determined by Equation 8.

3.5 Theoretical Guarantees on the Performance of IA

Weprove that the return associated with IA is guaranteed to have the following safety and performance guarantees.

Theorem 1. Let J ( π ) = E s 0 ∼ d 0 ,a t ∼ π ( ·| s t ) ,s t +1 ∼ P ( ·| s t ,a t ) [ ∑ ∞ t =0 γ t r ( s t , a t )] be the expected discounted return of following a policy π . Then, the performance following the Interventional Assistance policy (or behavior policy) π I has the following guarantees:

For a near-optimal pilot, ( Q π e ( s, a p ) ≈ max a ∗ Q π e ( s, a ∗ )) , π I is lower bounded by π p : J ( π I ) ≥ J ( π p ) .

For a low performing pilot, ( Q π e ( s, a p ) ≈ min a Q π e ( s, a )) , π I is low bounded by π c : J ( π I ) ≥ J ( π c ) .

The proof of Theorem 1 is in Appendix A. Intuitively, the copilot will only intervene when the pilot attempts to play actions from the current state that have expected future returns less than that of the copilot's action across all possible goals. The IA policy therefore does not degrade performance of a high-performing pilot, and when the pilot is poor, guarantees performance no worse than the copilot.

4 Results

4.1 Experimental Setup

Baselines. In the experiments that follow, we compared three different control methods. The first method is pilot-only control. The second method is copilot control, where the behavior policy is equal to the copilot policy π c ( a c | a p , ˜ s ) . Copilot control is the standard practice in SA (Reddy et al., 2018; Yoneda et al., 2023; Schaff and Walter, 2020; Jeon et al., 2020) as it allows a copilot to improve the human action before it is played in the environment. For copilot control, the action played is the action generated by the diffusion copilot using a forward diffusion ratio of γ = 0 . 2 , the setting that obtained the best control in (Yoneda et al., 2023). Our third method is IDA, which involves dynamically setting the behavior policy based on Equation 8.

Environments. The first environment we use is Reacher , a 2D simulation environment that models a two-jointed robotic arm with inertial physics. In this environment, torques are applied to the two joints of the robotic arm to position the arm's fingertip at a randomly spawned goal position within the arm's plane of motion. The state of the environment is an 11 dimensional observation containing information about the position and velocities of the joints and goal location. Rewards are given for making smooth trajectories that move the fingertip close to the goal. In each episode, the arm's position is reset to a starting location and a new goal is sampled uniformly across the range of the arm's reach. Following previous works ((Reddy et al., 2018; Schaff and Walter, 2020; Yoneda et al., 2023; Tan et al., 2022)), we also use Lunar Lander , a 2D continuous control environment in which a rocket ship must be controlled with three thrusters to land at a desired goal location on the ground. We modify the environment as described in (Yoneda et al., 2023) to make the landing location spawn randomly at different locations along the ground. On each episode the landing zone is indicated by two flags. The states are 9 dimensional observations of the environment containing information about the rocket ship's position, angular velocity, and goal landing zone. We define the success rate as the fraction of episodes that ended with a successful landing between the landing zone flags. We define the crash rate as the fraction of episodes that terminated due to a crash or flying out of bounds.

Pilots. We use simulated surrogate pilots (Reacher, Lunar Lander) and eight human pilots (Lunar Lander) to benchmark the performance of pilot-only, copilot, and IDA control (see Appendix D for details about human participants). All human experiments were approved by the IRB and participants were compensated with a gift card. Surrogate control policies are designed to reflect some suboptimalities in human control policies. We consider noisy and laggy surrogate control policies. Surrogate policies are constructed by drawing actions from either an expert policy or a corrupted policy. We use a switch that controls if actions are drawn from the expert or corrupt policies. Actions are initially sampled from the expert policy. At every time step there is a probability of corruption being turned on. Once corruption is turned on, actions are sampled from the corrupt

control policy. At every time step while corruption is on, there is a probability of turning corruption off. Once corruption is turned off, actions are sampled again from the expert control policy. The noisy surrogate policy is constructed by sampling actions uniformly randomly with a 30% corruption probability. The laggy surrogate policy actions are drawn by repeating the action at the previous time step with an 85% corruption probability.

4.2 Reacher Experiments

We compared the performance of IDA to pilot-only and the copilot SA method of (Yoneda et al., 2023) in the Reacher environment with targets that could randomly appear anywhere ('Continuous' in Table 1). We introduce two additional goal spaces to probe the generalization abilities of IDA: 'Linear' and 'Quadrant.' In the linear goal space, goals spawned uniformly random along a horizontal line located 100cm in front of the arm. In the quadrant goal space, goals spawned uniformly random in the upper right quadrant of the workspace. To use IDA without making the goal space known to the advantage computation (Equation 6) we constructed a 'faux goal' space (FG) by assuming a uniform distribution over potential next positions as goals. We then estimated the copilot advantage through Monte-Carlo sampling. Furthermore, we examined IDA's performance when the goal space is unknown during Q function training by using a domain shift (DS) environment where goals appear randomly at one of five locations during training. In lieu of using humans, we employed laggy and noisy surrogate control policies to emulate imperfect pilots in the Reacher environment across these goal spaces.

We evaluated performance by quantifying hit rate, the number of targets acquired per minute. In the continuous goal space we found that IDA always achieved performance greater than or equal to pilot-only control and outperformed the copilot (Table 1). The expert control policy was optimal, and IDA therefore rarely intervened with a copilot action, leading to similar performance. We also found the laggy pilot performed relatively well because laggy actions do not significantly impair target reaches, although it may delay target acquisition. When the policy was noisy, IDA improved hit

2

10

1000

100

6

Pilot

0

Reacher environment

a

b

Laggy pilot

Noisy pilot

c

Copilot

Pilot

Copilot

Hit rate

(targs/min)

10

2

10

1000

100

0

Number of Goals

rate from 1 . 6 to 3 . 3 targets per minute, approximately doubling performance. In contrast, we found the copilot was unable to acquire any targets. This may be because the copilot is unable to infer the precise goal location from surrogate pilot actions. Together, these results demonstrate that IDA is simultaneously capable of preserving optimal control for high performing pilots while improving hit rate for sub-optimal surrogate control policies, consistent with Theorem 1.

Additionally, we found that even without knowing the goal space during inference (IDA (FG)) or training (IDA (DS)), IDA still increased or maintained the hit rate of the noisy and laggy pilots in the continuous, linear, and quadrant goal spaces (Table 1). We also investigated how the performance of IDA varies with goal space size when goals can appear anywhere (Figure 2a). For each evaluation, we randomly sampled a specified number of candidate goal positions (ranging from 1 to 10, as well as 100 and 1000) from the continuous goal space and then evaluated IDA with goals restricted to these sampled positions. IDA consistently outperformed copilot and the noisy pilot while maintining the performance of the laggy pilot for all goal space sizes (Figure 2b, c). Collectively, these results demonstrate that IDA performance is robust even when the true goal space is unknown.

4.3 Lunar Lander

We next evaluated the performance of IDA in Lunar Lander with the noisy and laggy surrogate control policies. Consistent with Yoneda et al. (2023), we modified Lunar Lander so that the landing zone appears randomly in one of nine different locations. IDA always achieved performance greater than or equal to the performance of the pilot-only control policy (Table 2). Under the expert policy, which achieves 100% success rate and 0% crash rate, IDA does not degrade its performance, although copilot does. We observed that both copilot and IDA improved the performance of noisy and laggy surrogate control policies, with IDA consistently outperforming copilot in successful landings. Copilots and IDA also reduced crash rates for surrogate pilots. Additionally, we compared the performance of IDA to the penalty-based intervention approach proposed by Tan et al. (2022). Because the penalty-based intervention in Tan et al. (2022) used an MLP, we compare it to both IDA and IA with an MLP based copilot. We found that IA consistently achieved a higher rate of successful landings for both the noisy and laggy surrogate pilots than penalty based intervention for both copilot architectures. We further found that IA (MLP and IDA) yielded a lower crash rate than penalty-based intervention.

Next we examined when and why copilot intervention occurred for surrogate pilots. Because these control policies were constructed by corrupting an expert's control policy, we were able to characterize intervention during periods of expert control versus corruption. We examined the distribution of copilot-human advantage scores, which measures the fraction of the goal space over which the copilot's action has a higher expected return than the pilot's action. For both the noisy and laggy pilots, we found the distribution of copilot advantage scores were different during expert actions vs corrupted actions (Figure 3a,b). When corrupted actions were played, there were a greater number of states where the copilot advantage was equal to 1, indicating the copilot's action had a greater expected return over the entire goal space. Consistent with this, we see that intervention was more common during periods of corruption.

Table 2: Performance of surrogate pilots in the Lunar Lander environment. IDA is interventional assist with a diffusion copilot. IA (MLP) is interventional assist with an MLP copilot. Intervention-Penalty (MLP) refers to the intervention penalty approach proposed by Tan et al. (2022)

.

a

Noisy pilot intervention characterization

Laggy pilot

b

Histogram frequency

Copilot Advantage

-1

+1

pilot action

corrupted action

(random)

corrupted action

(lagged)

pilot action

Histogram counts

Action during

unintervened

states

pilot corrupted

0

250

0

120

Action during

intervened

states

pilot corrupted

c IDA intervention in bad states

100%

28.7%

58.8%

pilot thrust=56.1%

copilot thrust: 66.3%

IDA prevents flipping

IDA aids a graceful landing

-1

+1

We also visualized example states of copilot intervention for both the noisy and laggy pilots (Figure 3c). In two representative examples, we see that intervention generally occurs to stabilize and prevent the rocket ship from crashing or flipping over. In Figure 3c (top), we observe that during intervention, the copilot applies lateral thrust to the rocket to prevent it from rotating sideways despite the pilot operator attempting to fire the main rocket, which would result in a destabilizing rotation of the rocket ship. In Figure 3c (bottom), we observe that intervention occurred during landing where the copilot increased the amount of thrust on the main thruster to soften the touch down while simultaneously applying thrust on the right rocket to level the rocket ship for landing. In both instances, the copilot action prevented the rocket ship from entering a universally low-value state.

4.4 Lunar Lander with Human-in-the-loop Control

Given IDA improved the performance of surrogate pilots in Lunar Lander, we performed experiments with eight human participants. Participants played Lunar Lander using pilot-only, copilot, or IDA. Participants used a Logitech game controller with two joysticks to control the rocket ship. The left joystick controlled the lateral thrusters and the right joystick controlled the main thruster. Each participant performed three sequences of 3 experimental blocks (pilot, copilot, IDA) for a total of 9 blocks (see Appendix D for experiment block details). Each block consisted of 30 trials (episodes). Participants were blind to what block they were playing. The game was rendered at 50 fps.

Human pilots experienced considerable difficulty playing Lunar Lander, successfully landing at the goal locations only 14% of the time (Table 3). Copilot control allowed humans participants to successfully land the rocket ship at the goal locations 68 . 2% of the time. However, the copilot also frequently prevented any landing, resulting in a timeout in 13 . 8% of trials ( 0% in pilot only and 0 . 1% in IDA). While this led to a lower crash rate ( 3 . 6% ), it also reduced user autonomy and overall

lowered the success rate. In contrast, IDA achieved the highest performance, enabling participants to successfully land at the goal location 91 . 7% of the time which was significantly higher than pilot ( p < 0 . 01 , Wilcoxon signed-rank) and copilot ( p < 0 . 01 , Wilcoxon signed-rank) control. IDA also resulted in a significant reduction of crash rate when compared to human pilot control ( p < 0 . 01 , Wilcoxon signed-rank).

To quantify the level of ease, control, and autonomy human participants felt in each block (pilot only, copilot, IDA), we asked participants to provide a subjective rating in response to multiple questions. We asked participants to rate which blocks felt easiest, most controllable, and most autonomous. To assess ease of control , we asked: 'How easy was the task of landing the rocket ship at the goal location?' To assess how much control the participants had, we asked: 'How in control did you feel when performing this task?' However, participants may not have felt much control even in pilot-only mode, so to assess autonomy , we asked: 'How much do you believe your inputs affected the trajectory of the rocketship?' All questions were answered on a scale of 1 to 5 with 5 indicating the easiest, most in control, or most autonomy, respectively. We found that humans subjectively prefer IDA to baseline copilot assistance in terms of ease of use, controllability, and preserving autonomy ( p < 0 . 01 , Wilcoxon signed-rank).

5 Conclusion and Discussion

Our primary contribution is Interventional Assistance (IA): a hyperparameter-free and modular framework that plays a copilot action when it is better than the pilot action across all possible goals. We find that IA outperforms previous methods for intervention based shared autonomy proposed by Tan et al. (2022) as well as traditional copilot-only based methods for control sharing (Yoneda et al., 2023; Reddy et al., 2018). Furthermore, we empirically demonstrated IDA (IA with a D iffusion copilot) improves both objective task performance and subjective satisfaction with real human pilots in Lunar Lander (Figure 4). While prior SA systems may degrade pilot performance, particularly when copilots incorrectly infer the pilot's goal (Tan et al., 2022; Du et al., 2021), IA does not degrade human performance (Theorem 1) and often improves it.

One limitation of our approach is that we must train an autonomous agent in a simulated environment to obtain an

Pilot

Copilot

Ease

Control

Autonomy

Participant rating

5

0

4.6

3.0

1.5

2.3

2.2

4.3

4.3

2.8

4.2

expert Q function. However, this is not a fundamental requirement to learn an intervention function. A straightforward extension of the current work may use an offline dataset of expert demonstrations to train an ensemble of Q -networks (Chen et al., 2021). In general, while IA requires access to an expert Q function it makes no assumptions about how that Q function is obtained and we leave various methods of obtaining a Q function as directions for future work.

Additionaly, IDA demonstrated resilience across changes in goal space and can be easily adapted to real world settings where the goal space is unknown by construction of these faux goal spaces. Of course, in many settings task structure can be leveraged to further constrain the goal space and improve the assistance IA is able to provide. In these settings, another direction for future work is an implementation of IA that leverages a belief system to differentially weight candidate goals. Future work could potentially improve IA by removing unlikely goals from the advantage computation.

Acknowledgments This project was supported by NSF RI-2339769 (BZ), NSF CPS-2344955 (BZ), the Amazon UCLA Science Hub (JCK), and NIH DP2NS122037 (JCK). ZP is supported by the Amazon Fellowship via UCLA Science Hub.

References

Wayne F Cascio and Ramiro Montealegre. How technology is changing work and organizations. Annual review of organizational psychology and organizational behavior , 3(1):349-375, 2016.

Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model, 2021.

Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. Ave: Assistance via empowerment, 2021.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Shervin Javdani, Siddhartha S Srinivasa, and J Andrew Bagnell. Shared autonomy via hindsight optimization. Robotics science and systems: online proceedings , 2015, 2015.

Hong Jun Jeon, Dylan P Losey, and Dorsa Sadigh. Shared autonomy with learned latent actions. arXiv preprint arXiv:2005.03210 , 2020.

Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8077-8083. IEEE, 2019.

Marco Laghi, Michele Maimeri, Mathieu Marchand, Clara Leparoux, Manuel Catalano, Arash Ajoudani, and Antonio Bicchi. Shared-autonomy control for intuitive bimanual tele-manipulation. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids) , pages 1-9. IEEE, 2018.

Quanyi Li, Zhenghao Peng, and Bolei Zhou. Efficient learning of safe driving policy via human-ai copilot optimization. In International Conference on Learning Representations , 2022.

Zhenghao Peng, Quanyi Li, Chunxiao Liu, and Bolei Zhou. Safe driving via expert guided policy optimization. In 5th Annual Conference on Robot Learning , 2021.

Zhenghao Peng, Wenjie Mo, Chenda Duan, Quanyi Li, and Bolei Zhou. Learning from active human involvement through proxy value propagation. Advances in Neural Information Processing Systems , 2023.

Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning, 2018.

Charles Schaff and Matthew R. Walter. Residual policy learning for shared autonomy, 2020.

John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In ICML , pages 1889-1897. JMLR.org, 2015.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018.

Weihao Tan, David Koleczek, Siddhant Pradhan, Nicholas Perello, Vivek Chettiar, Vishal Rohra, Aaslesha Rajaram, Soundararajan Srinivasan, H M Sajjad Hossain, and Yash Chandak. On optimizing interventions in shared autonomy, 2022.

J. Wilson and P. R. Daugherty. Collaborative intelligence: Humans and ai are joining forces. Harvard Business Review , 2018a.

J. Wilson and P. R. Daugherty. Human Machine: Reimagining Work in the Age of AI . Harvard Business Review Press, 2018b.

Zhenghai Xue, Zhenghao Peng, Quanyi Li, Zhihan Liu, and Bolei Zhou. Guarded policy optimization with imperfect online demonstrations, 2023.

Takuma Yoneda, Luzhe Sun, , Ge Yang, Bradly Stadie, and Matthew Walter. To the noise and back: Diffusion for shared autonomy, 2023.

Matthieu Zimmer, Paolo Viappiani, and Paul Weng. Teacher-student framework: a reinforcement learning approach. 2014.

A Proof of Theorem 1

In this section, we will prove the theoretical guarantees on the performance of the interventional assistance (IA) in Theorem 1. We first introduce several useful lemmas.

First, we introduce a lemma for later use by following the Theorem 3.2 in (Xue et al., 2023).

Lemma 1. For any behavior policy π I deduced by a copilot policy π c , a pilot policy π p , and an intervention function T ( s, a p , a c ) , the state distribution discrepancy between π I and π c is bounded by the policy discrepancy and intervention rate:

where β = E s ∼ τπ I ,ac ∼ πc,ap ∼ πp ∥ T ( s,a p ,a c )[ π c ( a c | s ) -π p ( a p | s )] ∥ 1 E s ∼ τπ I ,ac ∼ πc,ap ∼ πp ∥ π c ( a c | s ) -π p ( a p | s ) ∥ 1 is the weighted expected intervention rate. τ π I and τ π c are the corresponding state visitation distributions following π I and π c , respectively.

Proof. We begin with the result of Theorem 3.2 in (Xue et al., 2023),

To prove the theorem, the key lemma we use is the policy difference lemma in (Schulman et al., 2015) introduced below. It introduces one policy's advantage function computed on states and actions sampled from another policy's generated trajectory. Here, the advantage function is defined as A π ′ ( s, a ) = Q π ′ ( s, a ) -V π ′ ( s ) and V π ′ ( s ) = E a ∼ π ′ Q ( s, a ) is the state value function. J ( π ) = E s 0 ∼ d 0 ,a t ∼ π ( ·| s t ) ,s t +1 ∼ P ( ·| s t ,a t ) [ ∑ ∞ t =0 γ t r ( s t , a t )] is the expected return following policy π .

Lemma 2 (Policy difference lemma) . Let π and π ′ be two policies. The difference in expected returns can be represented as follows:

We introduce two lemmas that exploits the intervention function we proposed in Section 3.4.

Lemma 3. The Q value of the behavior action under the expert's Q estimate is greater or equal to the Q value of the pilot action.

Proof. According to the intervention function T in Equation 8, intervention happens when the copilot advantage function A (˜ s, a c , a p ) = 1 . If we consider F to be the sign function, then according to Equation 6, A (˜ s, a c , a p ) = 1 means for all goals we will always have I (˜ s, a c | ˆ g ) > I (˜ s, a p | ˆ g ) , ∀ ˆ g . Recall I ( ˜ s t , ˜ a t | ˆ g ) = Q π e (˜ s t , ˜ a t | ˆ g ) , when intervention happens, we will have:

Therefore, where the equality holds when intervention does not happen and T = 0 .

Now we introduce the expectation over the behavior policy.

The above equation holds for arbitrary ˆ g . Therefore E a ∼ π I ( ·| s ) Q π e ( s, a ) ≥ E a p ∼ π p Q π e ( s, a p ) .

Similar to Lemma 3, we have:

Lemma 4. The Q value of the behavior action under the expert's Q estimate is greater or equal to the Q value of the copilot action.

Proof. According to the intervention function T in Equation 8, intervention happens when the copilot advantage function A (˜ s, a c , a p ) = 1 . If we consider F to be the sign function, then according to Equation 6, A (˜ s, a c , a p ) = 1 means for all goals we will always have I (˜ s, a c | ˆ g ) > I (˜ s, a p | ˆ g ) , ∀ ˆ g . Recall I ( ˜ s t , ˜ a t | ˆ g ) = Q π e (˜ s t , ˜ a t | ˆ g ) , when the intervention does not happen , we will have:

Therefore,

Now we introduce the expectation over the behavior policy.

The above equation holds for arbitrary ˆ g . Therefore E a ∼ π I ( ·| s ) Q π e ( s, a ) ≥ E a c ∼ π c Q π e ( s, a c ) .

We introduce another useful lemma.

Lemma5 (State Distribution Difference Bound) . Let π and π ′ be two policies, and let τ π : S → [0 , 1] and τ π ′ : S → [0 , 1] be the corresponding state visitation distributions. For any state-dependent function f ( s ) that is bounded by M (i.e., ∥ f ( s ) ∥ 1 ≤ M for all s ), the difference in expectations of f ( s ) under these two distributions is bounded as follows:

where is the total variation distance between two distributions τ π and τ π ′ .

Proof. The expectation of f ( s ) under the state distribution τ π can be written as:

The difference in these expectations is:

Given that ∥ f ( s ) ∥ 1 ≤ M , we have:

Thus, combining these bounds, we have:

This completes the proof.

Similarly, for policy π ′ :

A.1 The relationship between the behavior policy and the pilot policy

In this section, we will lower bound the performance of the IDA policy J ( π I ) by the performance of the pilot policy J ( π p ) under certain conditions.

Theorem 2. Let J ( π ) = E s 0 ∼ d 0 ,a t ∼ π ( ·| s t ) ,s t +1 ∼ P ( ·| s t ,a t ) [ ∑ ∞ t =0 γ t r ( s t , a t )] be the expected discounted return of following a policy π . Then for any behavior policy π I deduced by a copilot policy π c , a pilot policy π p , and an intervention function T ( s, a p , a c ) ,

wherein R = R max -R min is the range of the reward, β = E s ∼ τπ I ∥ T ( π c ( ·| s ) -π p ( ·| s )) ∥ 1 E s ∼ τπ I ∥ π c ( ·| s ) -π p ( ·| s ) ∥ 1 is the weighted expected intervention rate, and the L 1 -norm of output difference ∥ π c ( ·| s ) -π p ( ·| s ) ∥ 1 = ∫ A | π c ( a | s ) -π p ( a | s ) | d a is the discrepancy between π c and π p on state s .

Proof. By following Lemma 2, we have

By following Lemma 3, we continue to have:

At the same time, we have

We want to relate Equation 35 with Equation 36, which has the expectation over state distribution τ π p , instead of τ π I .

To solve the mismatch, we apply Lemma 5. Here, we let

We have ∥ f ( s ) ∥ ≤ γ 1 -γ R = M , where R = R max -R min is the range of the reward. Lemma 5 bounds the difference in expectations of f ( s ) under the state distributions τ π p and τ π I as follows:

Substituting f ( s ) into this inequality, we get:

where ϵ = M ∥ τ π p -τ π I ∥ 1 .

Thus, we can write:

From the deduction above, we have J ( π I ) -J ( π e ) ≥ E s ∼ τ π I [∑ ∞ t =0 γ t E a p ∼ π p ( ·| s ) A π e ( s, a p ) ] and the right hand side except ϵ c is J ( π c ) -J ( π e ) . Therefore,

According to the Theorem 3.2 in (Xue et al., 2023) (Lemma 1), we have

where β = E s ∼ τπ I ∥ T ( π c ( ·| s ) -π p ( ·| s )) ∥ 1 E s ∼ τπ I ∥ π c ( ·| s ) -π p ( ·| s ) ∥ 1 is the weighted expected intervention rate . Therefore, the ϵ can be upper bounded by:

Therefore, we have:

This is the lower limit for IDA in terms of the pilot.

A.2 The relationship between the behavior policy and the copilot policy

In this section, we will lower-bound the performance of the IDA policy J ( π I ) by the performance of the copilot policy J ( π c ) .

Theorem 3. Let J ( π ) = E s 0 ∼ d 0 ,a t ∼ π ( ·| s t ) ,s t +1 ∼ P ( ·| s t ,a t ) [ ∑ ∞ t =0 γ t r ( s t , a t )] be the expected discounted return of following a policy π . Then for any behavior policy π I deduced by a copilot policy π c , a pilot policy π p , and an intervention function T ( s, a p , a c ) ,

Proof. According to Lemma 4,

Now we will use Lemma 5, letting

We will have ∥ f c ( s ) ∥ ≤ γ 1 -γ R = M , where R = R max -R min is the range of the reward. Lemma 5 bounds the difference in expectations of f ( s ) under the state distributions τ π c and τ π I as follows:

Substituting f ( s ) into this inequality, we get:

Thus, we can write:

By substituting Equation 49 and Equation 50 into Equation 54, we have:

By using Lemma 1, we have ϵ c = M ∥ τ π c -τ π I ∥ 1 upper-bounded by:

Therefore, we have:

This is the lower bound of IA in terms of the copilot.

A.3 Safety and Performance Guarantees of IA

We now prove our main theorem on the performance guarantees of following the IA policy, restated below:

Theorem 1. Let J ( π ) = E s 0 ∼ d 0 ,a t ∼ π ( ·| s t ) ,s t +1 ∼ P ( ·| s t ,a t ) [ ∑ ∞ t =0 γ t r ( s t , a t )] be the expected discounted return of following a policy π . Then, the performance following the Interventional Assistance policy (or behavior policy) π I has the following guarantees:

For a near-optimal pilot, ( Q π e ( s, a p ) ≈ max a ∗ Q π e ( s, a ∗ )) , π I is lower bounded by π p : J ( π I ) ≥ J ( π p ) .

For a low performing pilot, ( Q π e ( s, a p ) ≈ min a Q π e ( s, a )) , π I is low bounded by π c : J ( π I ) ≥ J ( π c ) .

Proof. From equation 44 and equation 57 we have obtained the following lower bounds on the performance of IA,

For a near optimal pilot, and therefore,

and therefore

This means that T ( s, a p , a c ) ≈ 0 and therefore β ≈ 0 . When we take the limit of the pilot lower bound as β → 0 , we have:

which simplifies to

Similarly for a pilot that is far from optimal,

This means that T ( s, a p , a c ) ≈ 1 and therefore β ≈ 1 . When we take the limit of the copilot lower bound as β → 1 , we have:

which simplifies to

We have therefore shown that for a near optimal pilot, IA is guaranteed to not degrade performance, and for a very poor pilot, IA is guaranteed to do at least as good as the copilot.

B Computational Cost of IDA

We found that the computation time for IDA inference only slightly increase as the size of the goal space increases (Table B.1). In all our experiments, we approximate continuous goal spaces by sampling 1,000 candidate goals which costs about 3 ms. This is sufficiently fast for most real-time control applications. Performance could further be improved by using Monte-Carlo estimates as done in the Faux-Goal experiments.

C Examples of Real Human Intervention

We found that, for human experiments in Lunar Lander, interventions by IDA are most common at the start of episodes and when touching down (Figure C.1). Each episode starts with a random disruptive force applied to the rocket so it makes sense that intervention should occur initially to ensure the human maintains a stable flight. Additionally, the rocket ship is very sensitive to the speed and orientation when touching down. Interventions near touch down likely serve to prevent collisions with the ground.

Altitude

Intervention Frequency

Horizontal Position

Intervention Frequency

Rocketship Altitude (m)

0

1.4

D Human-in-the-Loop Lunar Lander Experimental Design

Eight ( 2 female, 6 male) participants with no prior experience performed 3 sequences of Lunar Lander experiments. All participants were compensated with a gift card for their time and involvement in the study. Each sequence consisted of 3 experimental blocks ( 30 episodes per block) of pilot-only, copilot, and IDA control (Figure D.1). An episode of lunar lander timed out after 30 seconds. In total, each participant played 270 episodes of Lunar Lander, 90 for each condition of pilot-only, copilot, and IDA.

...

30 episodes

copilot

pilot

Block 1

Block 2

Block 3

Block 4

Block 5

Block 6

Block 7

Block 8

Block 9

Sequence 1

Sequence 2

Sequence 3

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We rigorously develop the IDA SA framework as described in the introduction in sections 3.1-3.6. We further empirically demonstrate the utility of our framework in Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our approach in the discussion. In particular, we discuss the limitations of training a fully autonomous agent to perform the task and the assumptions that space of possible goals is known.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a proof of theorem 1 in appendix A. All the preliminaries for Theorem 1 are defined in sections 3.1-3.4

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a description of our methods in section 3.1-3.5 as well as provided references for methods we adopted. We also provide descriptions of the gymnasium environments that we use which are all publically available.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide code as a supplementary zip file. Polished code will be released with the paper along with set up and installation instructs to allow others to reproduce results as well as build future work.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Parameters for training are provided in sections 3 and 4.1. We specify the agents and environments that we use to perform training.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars corresponding to standard error of the mean are included on plots in figures 2, 3, and 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Training hardware and time is described in Section 3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the Code of Ethics to ensure our paper complies with them. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss in the introduction the broader impacts of developing shared autonomy systems and the overall need to develop AI that assists humans and does not repalce humans.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Or paper leverages publically available models and datasets and builds a model for performing shared human-AI control. No new foundation models or datasets are released with our research.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the methods we adapt for training various parts of IDA are cited in section 3.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: Experimental set-up for human experiments is described in section 4.4. Compensation is also reported in section 4.1

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: We obtained consent from all subjects and all experiments were IRB approved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the Shared Autonomy context, the authors propose a value-based intervention assistance method that aims to only have a copilot intervene only when the value of the assistant’s actions exceed that of the human. The proposed method first trains an expert policy for the task using privileged goal information, whose actions are then used to train a diffusion copilot for shared autonomy with the goal information removed. To determine if the copilot's actions should be used instead of the pilot, the method proposes a copilot advantage score, which computes the difference in q-value between the copilot and pilot actions marginalized across all possible goals in the environment. The efficacy of the method is demonstrated with both simulated human pilots and real human-in-the-loop experiments.\n\nSTRENGTHS:\n- To the best of my knowledge, the proposed method is novel and differs sufficiently from relevant prior work with the proposal of the value-based intervention system. Related methods that the paper builds upon are cited, and the experiments compare against the relevant baseline method.\n- The experiment setup follows the standard established by similar prior work in deep RL for shared autonomy, including both simulated and human experimenents. \n- Generally the submission is well written and clear to read.\n\nWEAKNESSES:\n- While Section 2 (Related Works) currently covers more recent relevant work, it would benefit from also providing a more general overview of the shared autonomy literature (e.g. providing an introduction to the shared autonomy and deep RL setting, such as described in \"Shared autonomy via deep reinforcement learning\" (Reddy et al., 2018) and relevant robotic teleoperation work, such as \"A policy-blending formalism for shared control\" (Dragan & Srinivasa, 2013)).\n- While the proposed method is intuitive, practically it seems difficult to implement since (1) an expert policy needs to be trained beforehand, and (2) the advantage computation at every timestep requires knowledge of the full goal space. A general challenge of the shared autonomy setting is that it is often difficult to capture the full goal space a human has in mind, and it's not clear how the proposed method will work if the agent does not know the full goal space beforehand. \n- Having to marginalize out all goals at every timestep for the advantage computation seems quite expensive, especially for higher dimensional goal spaces.\n- Figure 3c is quite interesting! it would be strengthened by adding more quantitative evaluation of the findings in Figure 3c (i.e. that the IDA copilot actively intervenes against negative behaviours). For example, computing the dot product between the copilot action and the pilot action vectors to see how much the copilot action differs from the pilot during intervention actions.\n- While the user study results show the improvement provided by the method, the sample size of five participants is quite small. I would encourage expanding this sample size and reporting the significance (e.g. p-values) of each of the corresponding metrics (see the similar comparison done in prior work, https://arxiv.org/pdf/1802.01744) .\n- Minor comment: renaming $Q^{\\pi_e}$ to $\\mathbf{I}$ seems unecessary, the former is more clear.\n\nQUESTIONS:\n- Can you clarify what \"exploring starts\" means for training the SAC expert?\n- Have the authors studied how increasing or decreasing the goal space size affects the usability of the method? For example, if we decrease / increase the number of total goals in the reacher environment, are there measurable impacts on the performance of the method? It would be insightful to see a plot showing change in computational efficiency as the number of goals change.\n\nLIMITATIONS:\nThe authors do address the limitations of the work. Further discussion on potential societal impact would be helpful (e.g. what are the implications if agents are unable to predict real human goals accurately?)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents the *intervention diffusion assistance* (IDA) framework for shared autonomy between a human \"pilot\" and an AI \"co-pilot\".  The IDA framework is designed to be goal agnostic, and does not attempt to infer the pilot's current goal.  This work extends Yoneda et al. (2023), using the same diffusion-based denoising policy, but now only applying the policy in states where the underlying expert policy is believed to be superior to the human pilot's policy.  Their experimental results (with both learned and human pilots) show that this selective denoising policy dominates both the pilot policy and the pilot's policy when corrected using the denoising policy at each step.\n\nReferences:\n1. Yoneda, Takuma, et al. \"To the noise and back: Diffusion for shared autonomy.\" arXiv preprint arXiv:2302.12244 (2023).\n\nSTRENGTHS:\nThe main strength of this work is the apparent effectiveness of the selection mechanism that chooses when the co-pilot is allowed to intervene and modify the human pilot's actions.  Experimental results suggest that this mechanism is able to restrict interventions to those states where the co-pilot's policy is most competent, and avoids failures due to \"gaps\" in the expert's policy.\n\nWEAKNESSES:\nOne potential weakness with this work is that the intervention selection mechanism depends on knowledge of the goal distribution for the task (as well as a function for \"masking\" goal information contained in the observations).  Thes would need to be implemented separately for each environment we wish to apply IDA to, and require significant domain expertise.\n\nAnother issue is Theorem 1.  The assumptions under which Theorem 1 holds are unclear.  In particular, there appears to be no assumption that the intervention policy is superior to the human policy.  This would seem to be difficult to ensure in general, as the intervention policy is trained over a distribution of goals.  It seems possible that the bounds in Equation 7 would not hold for \"low probability\" goals.\n\nAdditionally, as it was explicitly mentioned that the proposed approach is compatible with any shared autonomy architecture, it would have been nice to see evaluations with the selection mechanism applied to other architectures besides the diffusion-based policy of Yoneda et al.\n\nQUESTIONS:\n1. It was somewhat surprising that in the reacher environment (figure 2) the original denoising policy performs extremely poorly when applied to every state, particularly given its success in the aparently similar box-pushing task in the original work (Yondea et al. 2023).\n2. Equation 6 suggests that the advantage must be exactly equal to 1 in order for an intervention to occur, which would seem to be a very rare occurence.  Is this correct, or just a typo?\n\nLIMITATIONS:\nA key limitation of this work that deserves more attention is the need to provide a prior goal distribution, and a mechanism for \"masking\" goal information in the observations.  Both the prior and the masking function would seem to require significant domain knowledge, and might be difficult to implement in real-world tasks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an intervention assistance (IA) method, IDA, that dynamically decides whether the co-pilot should take over the control. The decision is determined by comparing the expected values of the co-pilot’s action versus the pilot’s action. The experiments with human surrogates showed that the proposed IA could improve task performance in the Reacher and Lunar Lander. The human study shows that human subjects prefer IDA.\n\nSTRENGTHS:\n- The proposed method can dynamically adjust whether the copilot policy should intervene to improve ease of use and human autonomy.\n- The proposed method follows a goal-agnostic copilot policy, thus can generalize to unseen goal locations.\n\nWEAKNESSES:\n- The proposed method still seems to rely on offline datasets to estimate the Q-value. So, the quality of the estimated Q-value is important to the intervention strategy. There is no evidence or discussion on how the performance or impact on users when the expected Q-value is not good enough.\n- In the experiments with simulated human surrogates and real humans, the baseline is using the pilot-only or always with a copilot. These two are the most basic baselines. However, since the paper proposes intervention strategies, a better comparison should be the paper such as Tan et al. 2022 where an intervention budget is set. Without such comparison, it is unclear how the proposed intervention advantage is compared with an existing or naive strategy to limit intervention numbers.\n   - Weihao Tan, David Koleczek, Siddhant Pradhan, Nicholas Perello, Vivek Chettiar, Vishal Rohra, Aaslesha Rajaram, Soundararajan Srinivasan, H M Sajjad Hossain, and Yash Chandak. On optimizing interventions in shared autonomy, 2022.\n\nQUESTIONS:\n- How is the estimation of Q-value learned in the experiment? Using the same SAC expert or a separate Q-network on the collected data? It is unclear how it is trained and what data it requires.\n- If the estimated Q-value is too optimistic or too pessimistic, how does it affect the performance of shared autonomy?\n- In Reacher, the laggy action doesn’t affect much about the performance, one explanation can be the domain doesn’t require high-precision control, thus less useful when considering shared control.\n\nLIMITATIONS:\n- The experiments are done with simulated experts, even in real human experiments. In the real world, some domains do not have such simulated/RL experts. How the method transfers or applies to those domains when no simulated/RL is available is not clear.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-20"}