Shared Autonomy with IDA: Interventional Diffusion Assistance

Authors: brandon j mcmahan
zhenghao  peng
bolei  zhou
jonathan c kao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:33:50

Abstract: The rapid development of artificial intelligence (AI) has unearthed the potential to assist humans in controlling advanced technologies. Shared autonomy (SA) facilitates control by combining inputs from a human pilot and an AI copilot. In prior SA studies, the copilot is constantly active in determining the action played at each time step. This limits human autonomy that may have deleterious effects on performance. In general, the amount of helpful copilot assistance varies greatly depending on the task dynamics. We therefore hypothesized that human autonomy and SA performance improves through dynamic and selective copilot intervention. To address this, we develop a goal-agnostic intervention assistance (IA) that dynamically shares control by having the copilot intervene only when the expected value of the copilot’s action exceeds that of the human’s action. We implement IA with a diffusion copilot (termed IDA) trained on expert demonstrations with goal masking. We prove that IDA performance is lower bounded by human performance, so that IDA does not negatively impact human control. In experiments with simulated human pilots, we show that IDA achieves higher performance than both pilot-only and traditional SA control in variants of the Reacher environment and Lunar Lander. We then demonstrate with human-in the-loop experiments that IDA achieves better control in Lunar Lander and that human participants experience greater autonomy and prefer IDA over pilot-only and traditional SA control. We attribute the success of IDA to preserving human autonomy while simultaneously offering assistance to prevent the human from entering universally bad states.

Main paper: 
 introduction: 
 As technology advances, humans continuously seek to operate more sophisticated and complex devices (Cascio and Montealegre, 2016) . However, more sophisticated technologies typically involve complicated operational dynamics and high-dimensional control systems that restrict their use to narrowly defined environments and highly specialized operators (Schulman et al., 2018) . While fully autonomous AI agents can be trained to perform these tasks, this approach has three key limitations. First, the user's goal is internalized and not easily deducible in most real-world environments. Second, removing the user from the control loop reduces their autonomy, potentially leading to poor performance and decreased engagement (Wilson and Daugherty, 2018a,b) . Third, as the capabilities of AI advance, it is important to consider how to create technologies that assist and empower humans instead of replacing them (Wilson and Daugherty, 2018a,b) .Shared autonomy (SA) addresses these limitations by blending human (pilot) actions with assistive agent (copilot) actions in a closed-loop setting. Prior studies demonstrate SA can increase human task performance in robotic arm control (Laghi et al., 2018) , drone control (Reddy et al., 2018) , and navigation (Peng et al., 2023) . A critical component of prior work is an empirically tuned controlsharing hyperparameter that trades off copilot assistance with human user autonomy (Reddy et al., 38th Conference on Neural Information Processing Systems (NeurIPS 2024). (Javdani et al., 2015; Yoneda et al., 2023; Reddy et al., 2018; Jeon et al., 2020) . The copilot then plays an action, e.g., by selecting a feasible action closest to the user suggestion (Reddy et al., 2018) or through diffusion (Yoneda et al., 2023) . (b) In this work, we design an intervention function that plays either the human pilot's action, a p , or the copilot's action, a c , based on their goal-agnostic advantages.2018; Jeon et al., 2020; Yoneda et al., 2023) . Excessive assistance can hinder goal achievement, while insufficient assistance can lead to poor control and performance. Prior work has established multiple methods of control sharing, but limitations remain, such as requiring control sharing hyperparameters be tuned empirically or limiting the copilot to general, non-goal-specific assistance (Schaff and Walter, 2020; Du et al., 2021; Tan et al., 2022) . We will discuss them further in Section 2.Imagine driving a car with an assistive copilot. The majority of the time, the human driver should remain in control. However, the copilot should intervene in certain scenarios to prevent collisions and ensure safe driving. By dynamically adjusting the level of assistance based on the situation and needs of the driver, the copilot simultaneously preserves driver autonomy and engagement while ensuring safety. This example reflects an important open problem: how do we construct an optimal behavior policy from the human and copilot's individual policies? This problem is conceptually similar to the one posed by the Teacher-Student Framework (TSF) (Zimmer et al., 2014) , where a teacher agent helps the student learn a good policy by intervening to prevent the student from visiting deleterious states and providing online guidance (Kelly et al., 2019; Li et al., 2022; Peng et al., 2021) . A recent work (Xue et al., 2023) has developed methods that improve the TSF framework by using a trajectory-based value estimate to decide when the teacher intervenes in student learning. We propose that a similar value estimate could be used to determine when the copilot should 'intervene' in the user's control, providing the user with assistance. Therefore, we develop an intervention function that estimates the expected value of the copilot and human action in a goal-agnostic fashion. Because our formulation is goal-agnostic, it can generalize to goals not seen during the training process, endowing the human pilot with improved flexibility and generalization.Our main contribution is a shared control system that leverages a value-based intervention function that can interface with many different copilot architectures while simultaneously improving performance and preserving pilot autonomy. In this work, we build on an existing diffusion-based copilot architecture (Yoneda et al., 2023) . These copilots are desirable because they can be trained using supervised methods which helps mitigate the sample complexity required to train effective copilots. While we proceed with the diffusion copilot of (Yoneda et al., 2023) we emphasize that our intervention function can be applied to any copilot capable of generating assistive, alternative, or corrective actions to aid the human in complex tasks. 
related work: 
 We have a brief discussion on three related works (Du et al., 2021; Yoneda et al., 2023; Tan et al., 2022) . (Du et al., 2021) : It proposes a method that increases a human's ability to control the environment and mitigate the need to infer any goals. It defines an information theoretic quantity that captures the number of future states accessible to a human from the current state. An assistive agent is then trained to maximize this quantity while the human performs the task. While this allows agents to assist in a goal-agnostic fashion, it typically leads to lower performance than methods that infer the goal, since the assistive agent does not directly help the human achieve a goal. In contrast, other methods, including this work, leverage a copilot that implicitly infers the goal from human actions. While goal inference can lead to lower performance when the goal is incorrectly inferred, we mitigate this by restricting the copilot to only intervene when the human action is deleterious, i.e., worse than the copilot action across all possible goals.To the Noise and Back: Diffusion for Shared Autonomy (Yoneda et al., 2023) : It develops a copilot that uses diffusion to map a human action closer to an expert's action. They train a diffusion process to generate actions from a distribution of expert actions conditioned on goal-agnostic state observations. At inference, the human action is first pushed towards a Gaussian distribution by adding noise in the forward diffusion process. The reverse diffusion process is then run on this noised human action to transform it into a sample from the expert action distribution. This action is played in the environment, as illustrated in Figure 1a . The fraction of the forward diffusion process applied to the human action is the diffusion ratio γ ∈ [0, 1] and trades off action conformity (how similar the action is to expert actions) and human autonomy. This copilot therefore requires an experimentally tuned hyperparameter to influence how much control the human has on the eventual action. Because the amount of assistance needed may vary greatly depending on task dynamics, using a fixed amount of assistance throughout the task may limit performance and autonomy. Our intervention function presented in Section 3 addresses this limitation by allowing the copilot to dynamically intervene based on the human's actions.On Optimizing Interventions in Shared Autonomy (Tan et al., 2022) : It proposes a method where a copilot is trained with a penalty for intervention. This encourages the copilot to limit its intervention and preserve human autonomy. However, intervention is not inherently bad and need not be sparse. Instead of uniformly penalizing all intervention, only unnecessary intervention should be penalized. Additionally, this method is not inherently hyperparameter free as it does require setting a penalty hyperparameter to determine how the copilot should trade off assistance and autonomy, although they demonstrate this hyperparameter can be solved via optimization. Another limitation is this method assumes access to the human policy during training so that the copilot can learn when best to intervene for a particular human (e.g., an expert at the task would likely have less copilot intervention than a novice). In contrast, we define a general-purpose intervention function based on how good actions are in the environment (irrespective of the specific human pilot playing the actions), enabling one-time training of the intervention function. In our intervention function, an expert would still experience less intervention than a novice because the expert generally plays better actions in the environment. Empirically we find that with our approach the same intervention function can boost the performance of eight different human participants in the Lunar lander environment.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of our approach in the discussion. In particular, we discuss the limitations of training a fully autonomous agent to perform the task and the assumptions that space of possible goals is known.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Our goal is to introduce a general-purpose intervention function that increases the performance and human autonomy of an SA system. The inputs to the intervention function are: (1) the goal-agnostic environment state, s, (2) a pilot action, a p , and (3) a copilot action, a c , illustrated in Figure 1b . The intervention function then plays either the pilot action (a I = a p ) or copilot action (a I = a c ) in the environment. We define this intervention function and describe its implementation. We also prove a lower bound on the expected return associated with the policy using the intervention function proportional to the pilot and copilot performance.We develop an intervention assistance called Interventional Diffusion Assistance (IDA). First, an expert policy is trained to maximize returns in the environment (Section 3.2). Second, the expert is used to perform policy rollouts and generate demonstrations. Goal information is removed from the demonstration data. We then train a diffusion copilot from these demonstrations (Section 3.3). Third, we define a trajectory-based intervention function that decides whether to play the human or copilot action (Section 3.4). All training was performed on a workstation with a single 3080Ti and took approximately 48 hours to complete all three steps for our tasks.We assume the environment can be modeled as an infinite-horizon Markov Decision Process (MDP) defined by the tuple M = ⟨S, A, R, γ, P, d 0 ⟩. S is the space of all possible environment states, and A is the space of all possible actions. R : S×A → [R min , R max ] is a scalar reward received for playing an action a ∈ A in state s ∈ S. P : S × A × S → [0, 1] are the transition dynamics for the environment, γ is the discount factor, and d 0 is the distribution of initial states. We define the state-action value function induced by policy π to be Qπ (s, a) = E s0=s,a0=a,at∼π(•|st),st+1∼p(•|st,at) [ ∞ t=0 γ t r (s t , a t )],where π : S × A → [0, 1] is the action distribution conditioned on the state.We additionally introduce the notion of a goal that encodes the task objective or intention of the human. We can decompose any state s = ⟨s|ĝ⟩ into a partial goal-agnostic state observation s, which does not contain any goal specific information, and a goal ĝ ∈ G * , where G * is the space of all possible goals. Then Q π (s, a) = Q π (⟨s | ĝ⟩, a) = Q π (s, a|ĝ) is the state-action value function under the goal-agnostic state s and goal ĝ.We model the human behavior by a human pilot policy, π p , which generates human action a p ∼ π p (•|s) according to the full state observation. The human observes and therefore has access to the goal, ĝ ∈ G * . However, the goal is not assumed to be accessible to the copilot. In this paper, we assume the availability of an expert policy π e (a e |s), that observes the full state and solves the environment. We also assume that we can query the state-action value of the expert policy Q πe (s, a), with which we use to evaluate the quality of an action a. We will train a copilot policy π c that generates actions based on the pilot action and the goal-agnostic state a c ∼ π c (•|a p , s). The ultimate goal of this paper is to derive a goal-agnostic intervention function T(s, a c , a p ) ∈ {0, 1} from the expert policy so that the SA system can achieve better performance than the pilot alone. The behavior policy that shares autonomy between the pilot and the copilot can be represented as π I = Tπ c + (1 -T)π p .We train a soft actor-critic (SAC) expert to solve the environment (Haarnoja et al., 2018) because it allows us to (1) later query Q πe (s, a) for intervention and (2) generate demonstrations in the environment that can be used to train the copilot. In general, other methods of obtaining a Q-value estimator and for training a copilot are compatible with IA. We choose SAC for computational convenience. We parameterize our SAC model with a four-layer MLP with 256 units in each layer and the ReLU non-linearity. We use a learning rate of 3 × 10 -4 and a replay buffer size of 10 6 . The expert fully observes the environment including the goal, and is trained for 3 million time steps or until the environment is solved. We found that training with exploring starts (randomized initial state) produced a more robust Q function with better generalization abilities. Without exploring starts, the Q function performed poorly on unseen states, limiting the effectiveness of IDA.Following Yoneda et al. (2023) , we trained a diffusion copilot π c (a c |a p , s) using a denoising diffusion probabilistic model (DDPM) (Ho et al., 2020) . For each environment, we collected 10 million stateaction pairs from episodes using the SAC expert. All goal information was removed from this demonstration data. The copilot learned to denoise expert actions perturbed with Gaussian noise, conditioned on the goal-agnostic state s and the pilot's action a p .Formally, the forward diffusion process is a Markov chain that iteratively adds noise ϵ ∼ N (0, I) according to a noise schedule {α 0 , α 1 , ..., α T } to an expert action a 0 , viaEQUATIONFollowing the forward diffusion process, the diffusion copilot is then trained to predict the noise added by the forward process by minimizing the following loss (Ho et al., 2020) :EQUATIONwhere ϵ θ is a neural network parameterized by θ that approximates the noise ϵ conditioned on the noisy action a t , the goal-agnostic state s, and the diffusion timestep t. τ is the distribution of states in the demonstration data. The reverse diffusion process is modeled by a four-layer MLP to iteratively refine a t toward a 0 .Algorithm 1 SA with IDA Sample state s and goal-masked state s from environment.Sample human action a p ∼ π p (• | s).Sample diffusion copilot action a c ∼ π c (•|a p , s).Compute the copilot advantage score A(s, a c , a p )7: if A(s, a c , a p ) = 1 then 8:Play copilot action in environment, a I = a c . 9:else 10:Play pilot action in environment a I = a p .11:end if 12: end for 13: environment resetIDA allows the copilot to intervene in pilot control when they take actions that are consistently bad for all possible goals. We therefore play the copilot's action a c instead of the pilot's action a p when the copilot's action has a higher expected return under the expert Q-value, that is,Q πe (s, a c ) ≥ Q πe (s, a p ).(3) However, we can not directly assess Equation 3, since in a SA system the goal is internal to the pilot. Instead, we only have access to the goal agnostic state s, such that, Equation 3 becomes,EQUATIONWe can define an intervention score I( st , ãt |ĝ) which considers the value of ( st , ãt ) under the assumption that ĝ ∈ G * is the goal, where G * is the space of all possible goals,EQUATIONBy marginalizing the difference in the intervention scores between the copilot and pilot over the entire goal space we can define a copilot advantage A(s, a c , a p )EQUATIONwhere F is a function that maps the difference in intervention scores to {-1, +1} to ensure all possible goals are weighted equally. Here we chooseEQUATIONWhenF (•) = sign(•), then A(s, a c , a p ) ∈ [-1, +1] is proportional to the fraction of the goal space over which the copilot action is superior to the pilot action. Also, when F is the sign function, the normalization constant reduces to G = ĝ∈G * dĝ and if the goal space is discrete then G = |G * | is the number of goals.We adapt the value based intervention function proposed by Xue et al. (2023) to use for shared autonomy by allowing intervention to occur when A(s, a c , a p ) = 1. The copilot therefore intervenes when its action has a higher expected return compared to the pilot action for all possible goals. Formally, we letEQUATIONwith intervention policy, π I = Tπ c + (1 -T)π p . The process for performing Shared Autonomy (SA) with IA is highlighted in Algorithm 1. The copilot advantage is computed at every timestep.The behavioral (IA) policy is then determined by Equation 8.We prove that the return associated with IA is guaranteed to have the following safety and performance guarantees.Theorem 1. Let J(π) = E s0∼d0,at∼π(•|st),st+1∼P (•|st,at) [ ∞ t=0 γ t r(s t , a t )] be the expected discounted return of following a policy π. Then, the performance following the Interventional Assistance policy (or behavior policy) π I has the following guarantees:1. For a near-optimal pilot, (Q πe (s, a p ) ≈ max a * Q πe (s, a * )), π I is lower bounded by π p :J(π I ) ≥ J(π p ). 2. For a low performing pilot, (Q πe (s, a p ) ≈ min a Q πe (s, a)), π I is low bounded by π c : J(π I ) ≥ J(π c ).The proof of Theorem 1 is in Appendix A. Intuitively, the copilot will only intervene when the pilot attempts to play actions from the current state that have expected future returns less than that of the copilot's action across all possible goals. The IA policy therefore does not degrade performance of a high-performing pilot, and when the pilot is poor, guarantees performance no worse than the copilot.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss in the introduction the broader impacts of developing shared autonomy systems and the overall need to develop AI that assists humans and does not repalce humans.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental setup: 
 Baselines. In the experiments that follow, we compared three different control methods. The first method is pilot-only control. The second method is copilot control, where the behavior policy is equal to the copilot policy π c (a c |a p , s). Copilot control is the standard practice in SA (Reddy et al., 2018; Yoneda et al., 2023; Schaff and Walter, 2020; Jeon et al., 2020) as it allows a copilot to improve the human action before it is played in the environment. For copilot control, the action played is the action generated by the diffusion copilot using a forward diffusion ratio of γ = 0.2, the setting that obtained the best control in (Yoneda et al., 2023) . Our third method is IDA, which involves dynamically setting the behavior policy based on Equation 8.Environments. The first environment we use is Reacher, a 2D simulation environment that models a two-jointed robotic arm with inertial physics. In this environment, torques are applied to the two joints of the robotic arm to position the arm's fingertip at a randomly spawned goal position within the arm's plane of motion. The state of the environment is an 11 dimensional observation containing information about the position and velocities of the joints and goal location. Rewards are given for making smooth trajectories that move the fingertip close to the goal. In each episode, the arm's position is reset to a starting location and a new goal is sampled uniformly across the range of the arm's reach. Following previous works (( Reddy et al., 2018; Schaff and Walter, 2020; Yoneda et al., 2023; Tan et al., 2022 )), we also use Lunar Lander, a 2D continuous control environment in which a rocket ship must be controlled with three thrusters to land at a desired goal location on the ground. We modify the environment as described in (Yoneda et al., 2023) to make the landing location spawn randomly at different locations along the ground. On each episode the landing zone is indicated by two flags. The states are 9 dimensional observations of the environment containing information about the rocket ship's position, angular velocity, and goal landing zone. We define the success rate as the fraction of episodes that ended with a successful landing between the landing zone flags. We define the crash rate as the fraction of episodes that terminated due to a crash or flying out of bounds.Pilots. We use simulated surrogate pilots (Reacher, Lunar Lander) and eight human pilots (Lunar Lander) to benchmark the performance of pilot-only, copilot, and IDA control (see Appendix D for details about human participants). All human experiments were approved by the IRB and participants were compensated with a gift card. Surrogate control policies are designed to reflect some suboptimalities in human control policies. We consider noisy and laggy surrogate control policies. Surrogate policies are constructed by drawing actions from either an expert policy or a corrupted policy. We use a switch that controls if actions are drawn from the expert or corrupt policies. Actions are initially sampled from the expert policy. At every time step there is a probability of corruption being turned on. Once corruption is turned on, actions are sampled from the corrupt We compared the performance of IDA to pilot-only and the copilot SA method of (Yoneda et al., 2023) in the Reacher environment with targets that could randomly appear anywhere ("Continuous" in Table 1 ). We introduce two additional goal spaces to probe the generalization abilities of IDA: "Linear" and "Quadrant." In the linear goal space, goals spawned uniformly random along a horizontal line located 100cm in front of the arm. In the quadrant goal space, goals spawned uniformly random in the upper right quadrant of the workspace. To use IDA without making the goal space known to the advantage computation (Equation 6) we constructed a "faux goal" space (FG) by assuming a uniform distribution over potential next positions as goals. We then estimated the copilot advantage through Monte-Carlo sampling. Furthermore, we examined IDA's performance when the goal space is unknown during Q function training by using a domain shift (DS) environment where goals appear randomly at one of five locations during training. In lieu of using humans, we employed laggy and noisy surrogate control policies to emulate imperfect pilots in the Reacher environment across these goal spaces.We evaluated performance by quantifying hit rate, the number of targets acquired per minute. In the continuous goal space we found that IDA always achieved performance greater than or equal to pilot-only control and outperformed the copilot (Table 1 ). The expert control policy was optimal, and IDA therefore rarely intervened with a copilot action, leading to similar performance. We also found the laggy pilot performed relatively well because laggy actions do not significantly impair target reaches, although it may delay target acquisition. When the policy was noisy, IDA improved hit rate from 1.6 to 3.3 targets per minute, approximately doubling performance. In contrast, we found the copilot was unable to acquire any targets. This may be because the copilot is unable to infer the precise goal location from surrogate pilot actions. Together, these results demonstrate that IDA is simultaneously capable of preserving optimal control for high performing pilots while improving hit rate for sub-optimal surrogate control policies, consistent with Theorem 1.Additionally, we found that even without knowing the goal space during inference (IDA (FG)) or training (IDA (DS)), IDA still increased or maintained the hit rate of the noisy and laggy pilots in the continuous, linear, and quadrant goal spaces (Table 1 ). We also investigated how the performance of IDA varies with goal space size when goals can appear anywhere (Figure 2a ). For each evaluation, we randomly sampled a specified number of candidate goal positions (ranging from 1 to 10, as well as 100 and 1000) from the continuous goal space and then evaluated IDA with goals restricted to these sampled positions. IDA consistently outperformed copilot and the noisy pilot while maintining the performance of the laggy pilot for all goal space sizes (Figure 2b, c ). Collectively, these results demonstrate that IDA performance is robust even when the true goal space is unknown.We next evaluated the performance of IDA in Lunar Lander with the noisy and laggy surrogate control policies. Consistent with Yoneda et al. (2023) , we modified Lunar Lander so that the landing zone appears randomly in one of nine different locations. IDA always achieved performance greater than or equal to the performance of the pilot-only control policy (Table 2 ). Under the expert policy, which achieves 100% success rate and 0% crash rate, IDA does not degrade its performance, although copilot does. We observed that both copilot and IDA improved the performance of noisy and laggy surrogate control policies, with IDA consistently outperforming copilot in successful landings. Copilots and IDA also reduced crash rates for surrogate pilots. Additionally, we compared the performance of IDA to the penalty-based intervention approach proposed by Tan et al. (2022) . Because the penalty-based intervention in Tan et al. (2022) used an MLP, we compare it to both IDA and IA with an MLP based copilot. We found that IA consistently achieved a higher rate of successful landings for both the noisy and laggy surrogate pilots than penalty based intervention for both copilot architectures. We further found that IA (MLP and IDA) yielded a lower crash rate than penalty-based intervention.Next we examined when and why copilot intervention occurred for surrogate pilots. Because these control policies were constructed by corrupting an expert's control policy, we were able to characterize intervention during periods of expert control versus corruption. We examined the distribution of copilot-human advantage scores, which measures the fraction of the goal space over which the copilot's action has a higher expected return than the pilot's action. For both the noisy and laggy pilots, we found the distribution of copilot advantage scores were different during expert actions vs corrupted actions (Figure 3a, b ). When corrupted actions were played, there were a greater number of states where the copilot advantage was equal to 1, indicating the copilot's action had a greater expected return over the entire goal space. Consistent with this, we see that intervention was more common during periods of corruption. In the top panel, the copilot prevents flipping. In the bottom panel, the copilot action helps to make a graceful landing.We also visualized example states of copilot intervention for both the noisy and laggy pilots (Figure 3c ). In two representative examples, we see that intervention generally occurs to stabilize and prevent the rocket ship from crashing or flipping over. In Figure 3c (top), we observe that during intervention, the copilot applies lateral thrust to the rocket to prevent it from rotating sideways despite the pilot operator attempting to fire the main rocket, which would result in a destabilizing rotation of the rocket ship. In Figure 3c (bottom), we observe that intervention occurred during landing where the copilot increased the amount of thrust on the main thruster to soften the touch down while simultaneously applying thrust on the right rocket to level the rocket ship for landing. In both instances, the copilot action prevented the rocket ship from entering a universally low-value state.Given IDA improved the performance of surrogate pilots in Lunar Lander, we performed experiments with eight human participants. Participants played Lunar Lander using pilot-only, copilot, or IDA. Participants used a Logitech game controller with two joysticks to control the rocket ship. The left joystick controlled the lateral thrusters and the right joystick controlled the main thruster. Each participant performed three sequences of 3 experimental blocks (pilot, copilot, IDA) for a total of 9 blocks (see Appendix D for experiment block details). Each block consisted of 30 trials (episodes). Participants were blind to what block they were playing. The game was rendered at 50 fps.Human pilots experienced considerable difficulty playing Lunar Lander, successfully landing at the goal locations only 14% of the time (Table 3 ). Copilot control allowed humans participants to successfully land the rocket ship at the goal locations 68.2% of the time. However, the copilot also frequently prevented any landing, resulting in a timeout in 13.8% of trials (0% in pilot only and 0.1% in IDA). While this led to a lower crash rate (3.6%), it also reduced user autonomy and overall lowered the success rate. In contrast, IDA achieved the highest performance, enabling participants to successfully land at the goal location 91.7% of the time which was significantly higher than pilot (p < 0.01, Wilcoxon signed-rank) and copilot (p < 0.01, Wilcoxon signed-rank) control. IDA also resulted in a significant reduction of crash rate when compared to human pilot control (p < 0.01, Wilcoxon signed-rank).To quantify the level of ease, control, and autonomy human participants felt in each block (pilot only, copilot, IDA), we asked participants to provide a subjective rating in response to multiple questions. We asked participants to rate which blocks felt easiest, most controllable, and most autonomous.To assess ease of control, we asked: "How easy was the task of landing the rocket ship at the goal location?" To assess how much control the participants had, we asked: "How in control did you feel when performing this task?" However, participants may not have felt much control even in pilot-only mode, so to assess autonomy, we asked: "How much do you believe your inputs affected the trajectory of the rocketship?" All questions were answered on a scale of 1 to 5 with 5 indicating the easiest, most in control, or most autonomy, respectively. We found that humans subjectively prefer IDA to baseline copilot assistance in terms of ease of use, controllability, and preserving autonomy (p < 0.01, Wilcoxon signed-rank). Participants subjectively rated IDA as achieving a similar level of autonomy to pilot only control but significantly better than copilot control. 
conclusion and discussion: 
 Our primary contribution is Interventional Assistance (IA): a hyperparameter-free and modular framework that plays a copilot action when it is better than the pilot action across all possible goals. We find that IA outperforms previous methods for intervention based shared autonomy proposed by Tan et al. (2022) as well as traditional copilot-only based methods for control sharing (Yoneda et al., 2023; Reddy et al., 2018) . Furthermore, we empirically demonstrated IDA (IA with a Diffusion copilot) improves both objective task performance and subjective satisfaction with real human pilots in Lunar Lander (Figure 4 ). While prior SA systems may degrade pilot performance, particularly when copilots incorrectly infer the pilot's goal (Tan et al., 2022; Du et al., 2021) , IA does not degrade human performance (Theorem 1) and often improves it.One limitation of our approach is that we must train an autonomous agent in a simulated environment to obtain an expert Q function. However, this is not a fundamental requirement to learn an intervention function.A straightforward extension of the current work may use an offline dataset of expert demonstrations to train an ensemble of Q-networks (Chen et al., 2021) . In general, while IA requires access to an expert Q function it makes no assumptions about how that Q function is obtained and we leave various methods of obtaining a Q function as directions for future work.Additionaly, IDA demonstrated resilience across changes in goal space and can be easily adapted to real world settings where the goal space is unknown by construction of these faux goal spaces. Of course, in many settings task structure can be leveraged to further constrain the goal space and improve the assistance IA is able to provide. In these settings, another direction for future work is an implementation of IA that leverages a belief system to differentially weight candidate goals. Future work could potentially improve IA by removing unlikely goals from the advantage computation. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Or paper leverages publically available models and datasets and builds a model for performing shared human-AI control. No new foundation models or datasets are released with our research.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the methods we adapt for training various parts of IDA are cited in section 3.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we will prove the theoretical guarantees on the performance of the interventional assistance (IA) in Theorem 1. We first introduce several useful lemmas.First, we introduce a lemma for later use by following the Theorem 3.2 in (Xue et al., 2023) . Lemma 1. For any behavior policy π I deduced by a copilot policy π c , a pilot policy π p , and an intervention function T(s, a p , a c ), the state distribution discrepancy between π I and π c is bounded by the policy discrepancy and intervention rate:EQUATIONwhere β = Es∼τ π I ,ac ∼πc ,ap∼πp ∥T(s,ap,ac) [πc(ac|s)-πp(ap|s)]∥ 1 Es∼τ π I ,ac ∼πc ,ap∼πp ∥πc(ac|s)-πp(ap|s)∥ 1is the weighted expected intervention rate. τ π I and τ πc are the corresponding state visitation distributions following π I and π c , respectively.Proof. We begin with the result of Theorem 3.2 in (Xue et al., 2023) ,∥τ π I -τ πc ∥ 1 ≤ γ 1 -γ E s∼τπ I ∥π I (•|s) -π c (•|s)∥ 1 = γ 1 -γ E s∼τπ I ,ac∼πc,ap∼πp ∥Tπ c (a c |s) + (1 -T)π p (a p |s) -π c (a c |s)∥ 1 = γ 1 -γ E s∼τπ I ,ac∼πc(•|s),ap∼πp(•|s) ∥(1 -T(s, a p , a c )) [π p (•|s) -π c (•|s)]∥ 1 = (1 -β)γ 1 -γ E s∼τπ I ∥π c (•|s) -π p (•|s)∥ 1 .To prove the theorem, the key lemma we use is the policy difference lemma in (Schulman et al., 2015) introduced below. It introduces one policy's advantage function computed on states and actions sampled from another policy's generated trajectory. Here, the advantage function is defined asA π ′ (s, a) = Q π ′ (s, a) -V π ′ (s) and V π ′ (s) = E a∼π ′ Q(s, a) is the state value function. J(π) = E s0∼d0,at∼π(•|st),st+1∼P (•|st,at) [ ∞ t=0 γ t r(s t , a t )]is the expected return following policy π. Lemma 2 (Policy difference lemma). Let π and π ′ be two policies. The difference in expected returns can be represented as follows:EQUATIONWe introduce two lemmas that exploits the intervention function we proposed in Section 3.4. Lemma 3. The Q value of the behavior action under the expert's Q estimate is greater or equal to the Q value of the pilot action.EQUATIONProof. According to the intervention function T in Equation 8, intervention happens when the copilot advantage function A(s, a c , a p ) = 1. If we consider F to be the sign function, then according to Equation 6, A(s, a c , a p ) = 1 means for all goals we will always have I(s, a c |ĝ) > I(s, a p |ĝ), ∀ĝ.Recall I( st , ãt |ĝ) = Q πe (s t , ãt |ĝ), when intervention happens, we will have:EQUATION) where the equality holds when intervention does not happen and T = 0. Now we introduce the expectation over the behavior policy.EQUATIONThe above equation holds for arbitrary ĝ. ThereforeE a∼π I (•|s) Q πe (s, a) ≥ E ap∼πp Q πe (s, a p ).Similar to Lemma 3, we have: Lemma 4. The Q value of the behavior action under the expert's Q estimate is greater or equal to the Q value of the copilot action.E a∼π I (•|s) Q πe (s, a) ≥ E ac∼πc Q πe (s, a c ) (16)Proof. According to the intervention function T in Equation 8, intervention happens when the copilot advantage function A(s, a c , a p ) = 1. If we consider F to be the sign function, then according to Equation 6, A(s, a c , a p ) = 1 means for all goals we will always have I(s, a c |ĝ) > I(s, a p |ĝ), ∀ĝ.Recall I( st , ãt |ĝ) = Q πe (s t , ãt |ĝ), when the intervention does not happen, we will have:EQUATIONTherefore,EQUATIONNow we introduce the expectation over the behavior policy.EQUATIONThe above equation holds for arbitrary ĝ. ThereforeE a∼π I (•|s) Q πe (s, a) ≥ E ac∼πc Q πe (s, a c ).We introduce another useful lemma.Lemma 5 (State Distribution Difference Bound). Let π and π ′ be two policies, and let τ π : S → [0, 1] and τ π ′ : S → [0, 1] be the corresponding state visitation distributions. For any state-dependent function f (s) that is bounded by M (i.e., ∥f (s)∥ 1 ≤ M for all s), the difference in expectations of f (s) under these two distributions is bounded as follows:EQUATIONwhereEQUATION)is the total variation distance between two distributions τ π and τ π ′ .Proof. The expectation of f (s) under the state distribution τ π can be written as:EQUATIONSimilarly, for policy π ′ :EQUATIONThe difference in these expectations is:EQUATION)Given that ∥f (s)∥ 1 ≤ M , we have:EQUATIONThus, combining these bounds, we have:EQUATIONThis completes the proof.In this section, we will lower bound the performance of the IDA policy J(π I ) by the performance of the pilot policy J(π p ) under certain conditions. Theorem 2. Let J(π) = E s0∼d0,at∼π(•|st),st+1∼P (•|st,at) [ ∞ t=0 γ t r(s t , a t )] be the expected discounted return of following a policy π. Then for any behavior policy π I deduced by a copilot policy π c , a pilot policy π p , and an intervention function T(s, a p , a c ),EQUATIONwherein R = R max -R min is the range of the reward, β =Es∼τ π I ∥T(πc(•|s)-πp(•|s))∥ 1 Es∼τ π I ∥πc(•|s)-πp(•|s)∥ 1is the weighted expected intervention rate, and theL 1 -norm of output difference ∥π c (•|s) -π p (•|s)∥ 1 =A |π c (a|s) -π p (a|s)| da is the discrepancy between π c and π p on state s.Proof. By following Lemma 2, we haveEQUATIONEQUATIONBy following Lemma 3, we continue to have:EQUATIONAt the same time, we haveEQUATIONWe want to relate Equation 35 with Equation 36, which has the expectation over state distribution τ πp , instead of τ π I .To solve the mismatch, we apply Lemma 5. Here, we letEQUATIONWe have ∥f (s)∥ ≤ γ 1 -γ R = M , where R = R max -R min is the range of the reward. Lemma 5 bounds the difference in expectations of f (s) under the state distributions τ πp and τ π I as follows:EQUATIONSubstituting f (s) into this inequality, we get:EQUATIONwhere ϵ = M ∥τ πp -τ π I ∥ 1 .Thus, we can write:EQUATIONFrom the deduction above, we haveJ(π I ) -J(π e ) ≥ E s∼τπ I ∞ t=0 γ t E ap∼πp(•|s) A πe (s, a p ) and the right hand side except ϵ c is J(π c ) -J(π e ).Therefore,EQUATIONAccording to the Theorem 3.2 in (Xue et al., 2023) (Lemma 1), we haveEQUATIONwhere β =Es∼τ π I ∥T(πc(•|s)-πp(•|s))∥ 1 Es∼τ π I ∥πc(•|s)-πp(•|s)∥ 1is the weighted expected intervention rate. Therefore, the ϵ can be upper bounded by:EQUATIONTherefore, we have:J(π I ) ≥ J(π p ) -βR γ 1 -γ 2 E s∼dπ I ∥π c (• | s) -π p (• | s)∥ 1 .(44) This is the lower limit for IDA in terms of the pilot.In this section, we will lower-bound the performance of the IDA policy J(π I ) by the performance of the copilot policy J(π c ).Theorem 3. Let J(π) = E s0∼d0,at∼π(•|st),st+1∼P (•|st,at) [ ∞ t=0 γ t r(s t , a t )] be the expected discounted return of following a policy π. Then for any behavior policy π I deduced by a copilot policy π c , a pilot policy π p , and an intervention function T(s, a p , a c ),EQUATIONProof. According to Lemma 4,EQUATIONEQUATIONWith Lemma 2, we haveEQUATIONNow we will use Lemma 5, lettingEQUATIONWe will have ∥f c (s)∥ ≤ γ 1 -γ R = M , where R = R max -R min is the range of the reward. Lemma 5 bounds the difference in expectations of f (s) under the state distributions τ πc and τ π I as follows:EQUATIONSubstituting f (s) into this inequality, we get:EQUATIONwhereϵ c = M ∥τ πc -τ π I ∥ 1 .Thus, we can write:EQUATIONBy substituting Equation 49and Equation 50 into Equation 54, we have:EQUATIONBy using Lemma 1, we have ϵ c = M ∥τ πc -τ π I ∥ 1 upper-bounded by:ϵ c ≤ (1 -β)R γ 1 -γ 2 E s∼dπ I ∥π c (• | s) -π p (• | s)∥ 1 .(56)Therefore, we have:J(π I ) ≥ J(π c ) -(1 -β)R γ 1 -γ 2 E s∼dπ I ∥π c (• | s) -π p (• | s)∥ 1 .(57) This is the lower bound of IA in terms of the copilot.We now prove our main theorem on the performance guarantees of following the IA policy, restated below:Theorem 1. Let J(π) = E s0∼d0,at∼π(•|st),st+1∼P (•|st,at) [ ∞ t=0γ t r(s t , a t )] be the expected discounted return of following a policy π. Then, the performance following the Interventional Assistance policy (or behavior policy) π I has the following guarantees:Proof. From equation 44 and equation 57 we have obtained the following lower bounds on the performance of IA,J(π I ) ≥ J(π c ) -(1 -β)R γ 1 -γ 2 E s∼dπ I ∥π c (• | s) -π p (• | s)∥ 1 (58) J(π I ) ≥ J(π p ) -βR γ 1 -γ 2 E s∼dπ I ∥π c (• | s) -π p (• | s)∥ 1 (59)For a near optimal pilot,EQUATIONand therefore,EQUATIONThis means that T (s, a p , a c ) ≈ 0 and therefore β ≈ 0. When we take the limit of the pilot lower bound as β → 0, we have:EQUATIONSimilarly for a pilot that is far from optimal,EQUATIONand therefore Q(s, a p ) ≤ Q(s, a c ).This means that T (s, a p , a c ) ≈ 1 and therefore β ≈ 1. When we take the limit of the copilot lower bound as β → 1, we have:EQUATIONWe have therefore shown that for a near optimal pilot, IA is guaranteed to not degrade performance, and for a very poor pilot, IA is guaranteed to do at least as good as the copilot.We found that the computation time for IDA inference only slightly increase as the size of the goal space increases (Table B.1) . In all our experiments, we approximate continuous goal spaces by sampling 1,000 candidate goals which costs about 3 ms. This is sufficiently fast for most real-time control applications. Performance could further be improved by using Monte-Carlo estimates as done in the Faux-Goal experiments. We found that, for human experiments in Lunar Lander, interventions by IDA are most common at the start of episodes and when touching down (Figure C.1) . Each episode starts with a random disruptive force applied to the rocket so it makes sense that intervention should occur initially to ensure the human maintains a stable flight. Additionally, the rocket ship is very sensitive to the speed and orientation when touching down. Interventions near touch down likely serve to prevent collisions with the ground. Justification: We rigorously develop the IDA SA framework as described in the introduction in sections 3.1-3.6. We further empirically demonstrate the utility of our framework in Section 4.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper introduces a novel and effective method for the value-based intervention system.
The experimental validation confirms the main claims. The submission is clearly written and organized. 
Moreover, the human study shows that human subjects prefer the proposed method.

Reviews: review: paper_summary: paper_summary: This paper presents an intervention assistance (IA) method, IDA, that dynamically decides whether the co-pilot should take over the control. The decision is determined by comparing the expected values of the co-pilot’s action versus the pilot’s action. The experiments with human surrogates showed that the proposed IA could improve task performance in the Reacher and Lunar Lander. The human study shows that human subjects prefer IDA.
strength_weakness: strength_weakness: - The proposed method still seems to rely on offline datasets to estimate the Q-value. So, the quality of the estimated Q-value is important to the intervention strategy. There is no evidence or discussion on how the performance or impact on users when the expected Q-value is not good enough.
- In the experiments with simulated human surrogates and real humans, the baseline is using the pilot-only or always with a copilot. These two are the most basic baselines. However, since the paper proposes intervention strategies, a better comparison should be the paper such as Tan et al. 2022 where an intervention budget is set. Without such comparison, it is unclear how the proposed intervention advantage is compared with an existing or naive strategy to limit intervention numbers.
   - Weihao Tan, David Koleczek, Siddhant Pradhan, Nicholas Perello, Vivek Chettiar, Vishal Rohra, Aaslesha Rajaram, Soundararajan Srinivasan, H M Sajjad Hossain, and Yash Chandak. On optimizing interventions in shared autonomy, 2022.
questions: questions: - How is the estimation of Q-value learned in the experiment? Using the same SAC expert or a separate Q-network on the collected data? It is unclear how it is trained and what data it requires.
- If the estimated Q-value is too optimistic or too pessimistic, how does it affect the performance of shared autonomy?
- In Reacher, the laggy action doesn’t affect much about the performance, one explanation can be the domain doesn’t require high-precision control, thus less useful when considering shared control.
limitations: limitations: - The experiments are done with simulated experts, even in real human experiments. In the real world, some domains do not have such simulated/RL experts. How the method transfers or applies to those domains when no simulated/RL is available is not clear.
paper_summary: paper_summary: This work presents the *intervention diffusion assistance* (IDA) framework for shared autonomy between a human "pilot" and an AI "co-pilot".  The IDA framework is designed to be goal agnostic, and does not attempt to infer the pilot's current goal.  This work extends Yoneda et al. (2023), using the same diffusion-based denoising policy, but now only applying the policy in states where the underlying expert policy is believed to be superior to the human pilot's policy.  Their experimental results (with both learned and human pilots) show that this selective denoising policy dominates both the pilot policy and the pilot's policy when corrected using the denoising policy at each step.

References:
1. Yoneda, Takuma, et al. "To the noise and back: Diffusion for shared autonomy." arXiv preprint arXiv:2302.12244 (2023).
strength_weakness: strength_weakness: One potential weakness with this work is that the intervention selection mechanism depends on knowledge of the goal distribution for the task (as well as a function for "masking" goal information contained in the observations).  Thes would need to be implemented separately for each environment we wish to apply IDA to, and require significant domain expertise.

Another issue is Theorem 1.  The assumptions under which Theorem 1 holds are unclear.  In particular, there appears to be no assumption that the intervention policy is superior to the human policy.  This would seem to be difficult to ensure in general, as the intervention policy is trained over a distribution of goals.  It seems possible that the bounds in Equation 7 would not hold for "low probability" goals.

Additionally, as it was explicitly mentioned that the proposed approach is compatible with any shared autonomy architecture, it would have been nice to see evaluations with the selection mechanism applied to other architectures besides the diffusion-based policy of Yoneda et al.
questions: questions: 1. It was somewhat surprising that in the reacher environment (figure 2) the original denoising policy performs extremely poorly when applied to every state, particularly given its success in the aparently similar box-pushing task in the original work (Yondea et al. 2023).
2. Equation 6 suggests that the advantage must be exactly equal to 1 in order for an intervention to occur, which would seem to be a very rare occurence.  Is this correct, or just a typo?
limitations: limitations: A key limitation of this work that deserves more attention is the need to provide a prior goal distribution, and a mechanism for "masking" goal information in the observations.  Both the prior and the masking function would seem to require significant domain knowledge, and might be difficult to implement in real-world tasks.
paper_summary: paper_summary: In the Shared Autonomy context, the authors propose a value-based intervention assistance method that aims to only have a copilot intervene only when the value of the assistant’s actions exceed that of the human. The proposed method first trains an expert policy for the task using privileged goal information, whose actions are then used to train a diffusion copilot for shared autonomy with the goal information removed. To determine if the copilot's actions should be used instead of the pilot, the method proposes a copilot advantage score, which computes the difference in q-value between the copilot and pilot actions marginalized across all possible goals in the environment. The efficacy of the method is demonstrated with both simulated human pilots and real human-in-the-loop experiments.
strength_weakness: strength_weakness: - While Section 2 (Related Works) currently covers more recent relevant work, it would benefit from also providing a more general overview of the shared autonomy literature (e.g. providing an introduction to the shared autonomy and deep RL setting, such as described in "Shared autonomy via deep reinforcement learning" (Reddy et al., 2018) and relevant robotic teleoperation work, such as "A policy-blending formalism for shared control" (Dragan & Srinivasa, 2013)).
- While the proposed method is intuitive, practically it seems difficult to implement since (1) an expert policy needs to be trained beforehand, and (2) the advantage computation at every timestep requires knowledge of the full goal space. A general challenge of the shared autonomy setting is that it is often difficult to capture the full goal space a human has in mind, and it's not clear how the proposed method will work if the agent does not know the full goal space beforehand. 
- Having to marginalize out all goals at every timestep for the advantage computation seems quite expensive, especially for higher dimensional goal spaces.
- Figure 3c is quite interesting! it would be strengthened by adding more quantitative evaluation of the findings in Figure 3c (i.e. that the IDA copilot actively intervenes against negative behaviours). For example, computing the dot product between the copilot action and the pilot action vectors to see how much the copilot action differs from the pilot during intervention actions.
- While the user study results show the improvement provided by the method, the sample size of five participants is quite small. I would encourage expanding this sample size and reporting the significance (e.g. p-values) of each of the corresponding metrics (see the similar comparison done in prior work, https://arxiv.org/pdf/1802.01744) .
- Minor comment: renaming $Q^{\pi_e}$ to $\mathbf{I}$ seems unecessary, the former is more clear.
questions: questions: - Can you clarify what "exploring starts" means for training the SAC expert?
- Have the authors studied how increasing or decreasing the goal space size affects the usability of the method? For example, if we decrease / increase the number of total goals in the reacher environment, are there measurable impacts on the performance of the method? It would be insightful to see a plot showing change in computational efficiency as the number of goals change.
limitations: limitations: The authors do address the limitations of the work. Further discussion on potential societal impact would be helpful (e.g. what are the implications if agents are unable to predict real human goals accurately?)

Comments: title: Thank-you
Thank-you
Thank-you
Response to Rebuttals
comment: Thank you again for your attention and thoughtful review of our manuscript. We are grateful for the opportunity to strengthen our manuscript with additional experiments with faux goal space and to more deeply probe when intervention occurs with real human pilots.
Thank you again for your careful review of our manuscript and comments. We are grateful for the opportunity to further discuss goal space considerations and believe additional experiments with a faux goal space has further strengthened our manuscript.
Thank you again for your time and attention in reviewing our manuscript. We are grateful for your comments and the suggestion to compare to Tan et al., 2022 and believe this has lead to a strengthened manuscript.
I thank the authors for the response and the additional experiments. They addressed my concerns, I'll raise my score accordingly.
I thank the authors for the additional experiments and for answering my questions. In particular, the additional "Faux Goal" experiments and intervention statistics help strengthen the paper. I will be raising my score accordingly.
I believe the authors' responses have addressed my main concerns.  Goal masking is still a potential limitation, but not one that is unique to this work.  I have raised my score accordingly.