Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racing

Authors: jonathan  decastro
andrew  silva
deepak  gopinath
emily  sumner
thomas m balch
laporsha  dees
guy  rosman
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 03:29:39

Abstract: Tight coordination is required for effective human-robot teams in domains involving fast dynamics and tactical decisions, such as multi-car racing.  In such settings, robot teammates must react to cues of a human teammate's tactical objective to assist in a way that is consistent with the objective (e.g., navigating left or right around an obstacle). To address this challenge, we present _Dream2Assist_,  a framework that combines a rich world model able to infer human objectives and value functions, and an assistive agent that provides appropriate expert assistance to a given human teammate. Our approach builds on a recurrent state space model to explicitly infer human intents, enabling the assistive agent to select actions that align with the human and enabling a fluid teaming interaction. We demonstrate our approach in a high-speed racing domain with a population of synthetic human drivers pursuing mutually exclusive objectives, such as "stay-behind" and "overtake". We show that the combined human-robot team, when blending its actions with those of the human, outperforms synthetic humans alone and several baseline assistance strategies, and that intent-conditioning enables adherence to human preferences during task execution, leading to improved performance while satisfying the human's objective.

Main paper: 
 introduction: 
 In high-stakes situations where members of a team must coordinate their physical actions in the world for the team to succeed, early coordination on tactical objectives is crucial. In a rapidly-evolving task, such as in high-speed competitive sports, agents must find a way to attain such coordination without explicit communication. A robotic assistive agent equipped with an ability to reason using theory of mind [1] has been shown to be critical to successful collaboration without the burden and latency of explicit communication. In such settings, agents must maintain a rich-enough model of the world to cover a common set of concepts that each agent needs to plan. This includes dynamics of the physical world, the objectives of the team, and the current intent of the other members of the team. Such considerations are prevalent in sports [2, 3] , manufacturing [4] , healthcare [5] , and traffic modeling settings [6] , among others.High-speed performance driving presents a domain where accurate and expressive models are required to have effective human-robot teams. The dynamics in racing evolve quickly, preventing team members from communicating their goals or objectives before taking an action [7] . Because of this constraint, existing approaches in shared control or advanced driver assistance often split authority on predefined boundaries (e.g. steering vs. throttle and brake) [8, 9] . The driving domain also requires us to tackle the multimodal nature of the human decision-making problem (e.g. rules or maneuvers), where discrete decisions are required to be made given knowledge of the situation [10, 11, 12, 13] . We hypothesize that a theory-of-mind-inspired model, coupled with recent advances in flexible planning and reinforcement learning, can enable new ways for machines to help humans drive more efficiently and effectively. We view our work as complementary to existing works in game-solving for racing [14] and planning in racing [15] .In this paper, we present a modeling paradigm to explicitly infer and support a human teammate's intent, enabling our agent to learn how to interact with diverse human strategies in a high-speed continuous control task. We consider the problem of a robotic assistive agent in a race car that helps a human driver perform tactical overtaking maneuvers more safely and more optimally in a high-speed competitive domain where explicit communication is difficult. While the driver remains in control of the vehicle at all times, the assistive agent can augment the driver's lateral (steering) and longitudinal (throttle and brake) actions. The assistant must infer the driver's intention (e.g., whether or not the driver is going to attempt an overtake) in order to provide optimal control augmentations and modifications that help the driver execute the inferred intention.We address planning under the unobservable dynamics involved with human decisions acting in the world via a novel model-based reinforcement learning (MBRL) paradigm that jointly learns the dynamics of the physical world and the human's intentions, enabling fluid shared control with continuous assistance during a race. We contribute the learning of a recurrent state-space model within an MBRL setup in light of cognitive and neuroscience findings indicating human decision-making may be modeled as such [16] . We infuse a world model with a richer world-representation by using fictitious humans with multiple, mutually exclusive, objectives with the aim of encoding diverse human intents into the model's world representation. Specifically, we contribute:1. A novel approach to MBRL for shared control in a tactical setting, leveraging driver intent modeling using fictitious co-play, conditioned on a fixed set of human objectives.A means for expert human value alignment, which conditions an assistive agent's rewards based on inferred intent to reason jointly over the physical world and the human's behavior.3. An evaluation of our approach on a shared control racing domain, demonstrating our model's utility over a diverse set of imperfect fictitious humans with different personal objectives.Significant work has taken place in sharing control [8] and human-robot interaction [17, 18] , exploring topics from ergonomics and physiology to language-based strategic teaming. Prior research has presented learning-based approaches to human-robot interactions that target joint-representation learning [19, 20, 21, 22, 23, 24, 25, 26] . These works focus on a range of topics, including dataefficient representation learning [27] , intent or plan modeling [28, 29, 30, 31] , hypothesis-space specification and explanation [32, 24, 33] , or other cognitive and social motivations [34, 35, 36] . We go beyond prior work by inferring the human's discrete intent and modifying our robotic assistant's objective to encourage support of the inferred intent.In the context of driving, shared control has been explored for planning approaches [37, 38, 39, 40, 41, 42, 43, 44, 45] and learning-based approaches [46, 47, 48, 49, 50] . Prior work has also considered incorporating game-theory into shared control [50, 51] , as well as explicit human-centric design considerations [52, 53] . We refer readers to [54, 55] for comprehensive reviews. Beyond shared control, significant literature explored intent prediction in driving, see [56, 57] , and references therein.Recent work has proposed a shared-control model using model-predictive control that considers predicted trajectory information [39] , therefore implicitly capturing driver intent. Additional recent work has augmented a model-predictive controller with the ability to explicitly infer driver intent [9] , enabling the controller to share steering and control actions with a human. In our approach, we capture discrete driver intent in a way that is conducive to semantically-meaningful, multi-modal continuous behaviors (e.g., going left or right around an obstacle). Further, by framing our task as a multi-agent reinforcement-learning problem, our proposed solution extends to multiple agents much more naturally than prior work. Finally, we frame our decision making approach within a recurrent state-space model [58] which is extended to infer the objective of the human, building on recent work that has explored hierarchical or hybrid state abstractions [59, 60] . 
background and problem statement: 
 We target the problem of shared control in the highly-dynamic setting of high-performance racing against other racing opponents. We aim to build an assistive agent that is capable of reasoning over well-defined task objectives, as well as more general, harder-to-define, and harder-to-observe human objectives. The assistive agent is given a map of the track, states of the ego vehicle and opponent vehicle, the ego driver's steering and throttle controls. The agent's task is to assist the ego driver through modifications to the steering and throttle of the ego car, as depicted in Fig. 1 (lower right).To achieve optimal performance, the agent must provide continuous control adjustments as the driver progresses along the track, helping the driver to stay on course, maintain proper speeds, and avoid collisions. Further, the agent must accurately infer the intentions of the driver (such as "stay to the opponent's left") and provide control augmentations that help the driver to accomplish their immediate task objective more optimally. Note that this problem is different from a conventional autonomous driving problem, as the agent's actions are conditioned explicitly on the human driver's control input, and the control signal that actuates the car is a linear blend of the agent and the driver's control signals. This problem also differs from conventional human-robot teaming, in that the agent must infer a human's intent (i.e., the human objective) early and as consistently as possible, as the efficacy on its assistance (in terms of safety and performance) depends strongly to how early, accurately, and reliably the human objective is captured. 
approach: 
 Our approach learns a common latent representation in the structure of a recurrent state space model (RSSM) [61, 62] to build a hybrid discrete / continuous state of the human partner's behavior, their rewards, and intent for achieving certain goals. RSSMs have been shown to be suitable for many domains (e.g. locomotion, Atari, Minecraft), and we are the first to apply this to modeling human behavior for assistance. To accommodate both a human and assistive agent operating in a collaborative setting, we train the assistive agent alongside human agents, with both agents sharing actions taken in an environment. Each agent receives a reward for each action taken, and each are trained to (1) build an accurate RSSM and (2) learn how to act in order to maximize its own expected returns [63] . One challenge in collaboration is that part of the assistive agent's world model includes Overview of value alignment for the assistive agent. We start with a set of frozen human policies whose values are annotated with predetermined outcomes. We blend human actions linearly with the actions from an unfrozen assistive agent. For both human and assistive agents, a multi-head RSSM architecture is then used to predict the observations, reward, and human intent (assistive only), trained to maximize their log-likelihoods against samples taken from the environment. The intent head is trained to match frozen human intents, which are fixed a priori. The assistive agent's rewards are shaped based on the optimal policy π * ŷA and optimal predicted reward r * ŷA for inferred intent ŷA .aspects of the future human's plan such as preferences, desires, and goals of the human (these are often collectively referred to as objectives [32] or intents [30] in the literature, we will use these terms interchangeably in the paper), which are often unknown and only weakly observed by their behavior [32] . We extend the RSSM formulation to allow it to be supervised on a diverse set of humans, whose objectives (intents) are known and labeled, to force the representation to be jointly aware of the behavior and intent across a variety of human types and physical environments. We then devise a scheme that feeds in inference of the human's objective into the assistant's reward function.In order to allow the system to maintain an ability to reason over the preferences of the human independent of that of the joint human-robot system, our approach considers both the human driver and the assistive agent planner as separate models. The training process is outlined in Fig. 2 .In the cooperative setting, the model of the environment follows a structure in which there are certain task-specific rewards which are available to both human and assistive agents, with the key distinctions being human-objective (specific to the human), and intervention penalties (specific to the assistant).Task-Specific Rewards We assume standard task-specific rewards for high-performance driving from prior work [64, 15] , including an out-of-bounds penalty, passing reward, and collision penalty.r task t = r collision t + r bounds t + r f inish t (1)where r collision t is a negative reward for collisions, r bounds t is a negative reward for driving too far off the track, and r f inish t is a positive reward for reaching the finish line. Intent-Aware World Model The objective of the world model is to provide a representation that the agent can use to interact with the driver and the world, and we posit that this representation can support the agent to reason jointly about both the task and human objectives. We build off of the recurrent state-space model (RSSM) of DreamerV2/V3 [65, 61] , according to the architecture in Figure 2 . For agent κ ∈ {H, A} (respectively, the Human and Assistive agent), the RSSM model, parameterized by ϕ and denoted W κ ϕ , includes: Encoder for discrete representation z κ t :EQUATIONwhere x t denotes the input observation. The output heads are similar to the DreamerV2/V3 architecture, with the addition of an intent predictor for the assistant agent (in red), and are all bottlenecked on the hidden statesEQUATIONTo train the intent predictor, we assume at least partial access to the intent labels for a given episode, which takes the form of an integer-valued target function. We normalize the output by predicting the symlog (sign(y) ln(|y| + 1)) of the output, and use a discrete distribution to predict y using the two-hot encoding of [61] . The parameters ϕ of the world model W κ ϕ are trained to minimize the lossL κ (ϕ) = E q κ ϕ β pred L κ pred + β KL L κ KLThe prediction loss L κ pred minimizes the likelihood under the predictor distributions in (3), L κ KL minimizes the KL divergence between the prior p κ ϕ and the approximate posterior q κ ϕ , and β pred and β KL are scalar weighting values. Given an intent label y t , the portion of loss L A pred for the intent predictor is taken as the negative log-likelihood of the label under p A ϕ . The world model training is alternated with training for the behavior model governing actions a κ t , with the behavior model learned via an actor-critic policy training over the estimates s t , rt , Actor:EQUATIONThe critic is trained to minimize the temporal-difference loss on the value function v κ ψ . The actor attempts to maximize the critic-predicted value. For further details, we refer the reader to [65] .To learn alignment with human drivers, we expose the assistive agent to different humans at training time. Training with human partners in a fictitious co-play setting [27] allows the assistant to become robust to different possible human behaviors, but does not teach the assistant to distinguish between the discrete modes of human behavior, which is necessary for assistance in our driving task.Similar to recent work describing human preferences for a task using reward shaping [66, 67] , we generate a population of humans, but different from these works, we group labeled sub-populations according to certain inherent objectives or preferences. We train several sub-populations using a separate human reward r H t = r task t + r y t , which is composed of the base task rewards in (1) , with the addition of several distinct human objectives, spanning different multimodal behaviors of the human. These are then formed into a collection of humans {π H 1 , π N }, which we can consider as the collection of expert policies of each enumerated objective. We extend this notion to that of intent at runtime. At any given moment, the human may adopt an intent y to abide by policy π H y . To simplify training, we consider drivers and rollouts with fixed intents, y, though our problem setup generalizes to variable-intent scenarios (e.g., a human switching from "following" to "passing" during a race).Humans are error-prone and pursue their objectives irrationally [68] . To capture this, we partially train human partners to pair with the assistive agent in a process known as fictitious co-play (FCP) [27] , yielding a population of humans {π H y,k } k ∪ {π * y } for each intent sub-population Π H y , where k represents the policy checkpoint for some amount of completed training. Using the FCP framework, each checkpoint k ∈ [1, K] is an agent trained to k% completion and the final checkpoint k = K is an optimal policy. Each (sub)optimal human is endowed with a (sub)optimal RSSM world model. While we apply FCP in this work, we note that our approach is agnostic to how human behaviors are generated and future work may consider other approaches to synthesizing human policies [69, 70] .For continuous actions a H t , a A t ∈ R m for the human and assistant, respectively, we encourage the assistive agent to minimize its action intervention by adopting a reward of the form r interv t = -∥a A t ∥ 2 which penalizes the magnitude of the intervention according to an L 2 -norm. Aggregated over time, the intervention cost forms a mixed-norm L 1 -L 2 [71] , encouraging time sparsity of interventions.In addition to the intervention norm penalty, the agent is rewarded for task completion and penalized for driving out of bounds or for collisions. These sparse negative rewards serve to encourage the assistant to act as a "racing guardian", keeping the driver on the track and avoiding collisions while making progress towards the finish line. Note that these rewards align with the task objective (Sec 3.1), but do not favor any particular human objective or intent.Finally, the assistive agent receives reward for aligning with the inferred objective of the human driver. First, the agent receives a reward equal to the inferred reward under the optimal human's world model. In other words, at every time step t, the agent predicts the human's intent, ŷA t and then queries the world model corresponding to the optimal human policy W * ϕ(ŷ A t ) for its reward prediction r * (ŷ A t ), for the current state-action pair. Second, the agent receives reward for maximizing the likelihood of the human-robot's combined action under the optimal human model for the predicted human intent:EQUATIONNote that a H t is the current sub-optimal human action. Intuitively, these rewards encourage the assistant to: (1) select actions for which the optimal human assigns high value and (2) select actions that bring the sub-optimal human closer to the optimal human. Under this setup, although we infer ŷA t from the joint human-robot trajectory, we consider it to be more aligned with the human's intent, due to: (1) the presence of the intervention sparsity term r interv t , and (2) the fact that the inferred expert alignment term tilts the joint behavior toward the inferred expert human's behavior.The complete reward for the assistant is a combination of rewards that promote successful task execution, minimize the magnitude of intervention, and account for human preferences:r A t = r task + r interv ttask performance and intervention sparsityEQUATIONHere, scalars α r and α a weight the contributions of the human terms. We capture both the inferred optimal reward value, r * (ŷ A t ), as well as the similarity between the combined human-robot actions and an action from the inferred optimal policy a * (ŷ A t ) ∼ π * ŷA t (x t ). Note that, though we empirically compare different values of α r and α a in Appendix E, in the next section, we only examine results for the reward-only case (α r = 1, α a = 0). 
experiments and results: 
 We examine the performance of our approach, dubbed DREAM2ASSIST, on different racing tasks with a fictitious human driver. We examine two racing settings, each derived from portions of a two-mile race track, and implemented in the CARLA Simulator [72] using a rear-wheel race vehicle physics model. For each task, opponents are randomly instantiated as replays of real human trajectories from the track, meaning that they do not react to the ego vehicle. In each setting, we run two sets of experiments-one with pass vs. stay fictitious human partners (i.e., the human is trying to overtake or stay-behind their opponent), and one with left vs. right partners (i.e., the human wants to stay on the left or right side of their opponent). We also further examine out-of-distribution intent inference (with intent changing over time), and further ablations in Appendix F and G.In each experiment, we train a population of humans following each objective (e.g., "left" or "right"), and then train a DREAM2ASSIST agent over the combined population. We then evaluate the degree to which the assistive agent can improve fictitious human performance on the track, where performance is measured by total progress, average speed, and no collisions. To measure the contributions of each assistive agent for the fictitious human population, we sample checkpoints at every 20% performance increment for agents up to at least 75% of maximum, or from the bottom five performers if none are under this threshold. We report the mean change in performance (track progress and collisions) when an assistive agent is deployed, as well as the return under each human objective in Table 1 . Means and standard deviations are computed over all five sub-optimal fictitious drivers.The two settings we consider include Straightaway Driving and Hairpin Driving. The straightaway is a flat 370-meter portion of a track with a concrete barrier on the right-hand side. Because the section of road is straight and flat, drivers need only avoid colliding with the concrete wall or their opponent while racing to the finish. The hairpin section, however, is a 960-meter portion of a track involving several sharp turns and hills. Drivers must carefully manage their speed to avoid spin-outs or collisions, and passing is much more challenging in this section of track.Human Decision Characteristics Based on observed behaviors from a human study (see Appendix B), we propose two different sets of human characteristics (ground truth intent): pass vs. stay and left vs. right. We train fictitious human agents to satisfy each of these objectives. For the pass vs. stay behaviors, the fictitious human agents are trained to either overtake or stay-behind the opponent vehicle. For the left vs. right behaviors, the fictitious humans are trained with a preference to stay on either the left or right side of the opponent for as much of the race as possible. Attempting to provide the wrong type of assistance with such distinct behaviors (e.g., offering "right" assistance to a "left" human) will result in fighting between the assistant and human, and will likely lead to collisions or spin-outs. We provide further details on the rewards for each agent in Appendix C.The environment itself consists of the rewards in Sec. 3.1 and an observation and action space for each agent. The human and assistive agent's actions both adopt steering and acceleration values in the range of [-1, 1]. The assistant and human actions are summed together and clipped on [-1, 1] before being passed to the vehicle. The observation space contains the ego vehicle state (position, velocity, tire slip angles, yaw rate, heading), current distance traveled, and an array of forward-looking track edge points. Finally, the human and assistive agents are each able to observe the other's actions.Baselines The claim of our work is that an assistive agent will better support a human partner if the assistant can infer the human's objective and help to satisfy that objective. To test this claim, we compare DREAM2ASSIST against a baseline version of DREAMER, which makes no intent inference and is not rewarded according to a human objective. We also compare to a non-RSSM baseline, GAIL [73] , obtained with the BeTAIL [74] framework. BeTAIL uses a behavior transformer as the human policy, coupled with an assistive agent trained via adversarial imitation learning to correct for distribution shift. Our DREAMER-AIL baseline challenges whether our RSSM approach is necessary at all, or whether a purely data-driven approach using behavior cloning and inverse reinforcement learning could satisfy the multimodal behavioral assistance task that we consider. For this baseline, a fictitious human using a DREAMER policy is paired with the AIL assistive agent from BeTAIL.We observe a consistent trend for all straightaway results -DREAMER offers very low-magnitude intervention, leading to higher performance with expert drivers but poor performance with novice drivers (as the assistant is not helping). We show track progress and human-objective return for assistants deployed to the pass vs. stay problem in Fig. E .1 (bottom). We report additional results in Appendix E. Conversely, DREAM2ASSIST offers higher-magnitude intervention, leading to higher performance with novice drivers, but lower performance gains with expert drivers (as the assistant is not needed). As shown in Table 1 , DREAM2ASSIST generally offers the highest performance over other methods. DREAMER-AIL suffers from mode collapse, trying to turn all drivers into either a "stay" driver or getting caught between "left" and "right" and therefore not moving. This behavior means that the DREAMER-AIL agent consistently underperforms a fictitious human with no assistance at all, as the AIL assistance keeps the driver far behind the opponent. The hairpin domain is more challenging and yields fictitious humans that are not always consistently able to solve the task, thereby leaving greater scope for assistance from our trained agents. In our pass vs. stay experiment, DREAM2ASSIST significantly improves the drivers' abilities to solve the task while still satisfying the human objectives and not leading to an increase in collisions. Similarly, in the left vs. right experiment, DREAM2ASSIST leads to significant increases in track progress, reduction in collisions, and improvements in human-objective alignment. Baseline approaches fail to disentangle the "left" and "right" modes of driving. The DREAMER baseline instead opts to push the driver to stay behind the opponent vehicle in an effort to reduce collisions, thereby making it farther down the track but failing to overtake. The DREAMER-AIL baseline drives aggressively off the track, leading to a significant drop in track progress, collisions, and human-objective alignment. An illustrative example of DREAM2ASSIST is in Fig. 3 , and videos are at https://youtu.be/PVugoxqX5Co. Visualizing the assistance provided to each driver, we can more easily compare the scale of assistance provided by DREAM2ASSIST compared to baselines, showing that DREAM2ASSIST provides marked improvements relative to baseline assistive agents. 
conclusions and limitations: 
 We introduce an assistive paradigm, DREAM2ASSIST, that learns to interact with humans to help them perform more optimally while supporting their personal objectives for the task. We evaluate DREAM2ASSIST in a dynamic and challenging task of high-speed racing, and we show that our approach is able to disentangle and accommodate distinct human objectives more effectively than baseline methods. We show that DREAM2ASSIST results in higher human-robot team performance than baseline methods, suggesting that explicit intent-conditioning and reward-inference can provide crucial performance gains in settings with multimodal, mutually-exclusive, human objectives. 
 
 Appendix: We frame the model-based reinforcement learning (MBRL) problem as a two-player (human ego driver, assistive agent ego driver) partially-observable Markov decision process (POMDP) defined by the tuple M = ⟨X κ , A κ , T κ , R κ , γ⟩ κ=H,A , where, for agent κ ∈ {H, A} (for the human, and AI agent, respectively), X κ denotes the imagined states of the world, A κ denotes the agent's (continuous or discrete) actions, T κ :X κ × A κ → [0, 1] is the transition probability, R κ : X κ × A κ × X κ → Ris a reward function, and γ ∈ [0, 1] is a discount factor. We aim to train both agents such that they maximize their expected returns R κ = E T t=1 r κ t . Crucially, in the semi-cooperative shared control setting, each reward r κ t is factored into subcomponents, with both sharing the same task (driving) rewards, but where r H t contains an additional term for a human's objective, and r A t contains additional terms to weaken its contribution in relation to the human's and enforces alignment to the human.We briefly discuss a study conducted for gathering human subject behavior data in the racing domain we use in the paper. The purpose of the study was to gather qualitative and statistical data on individuals' behavior and objectives in a racing context, and to use that to inform what criteria are important for building models of human objectives. We recruited 48 participants to drive a simulator with the hairpin and straightaway segments of the two-mile track, the same domains for the computational results in this paper. The scenarios were chosen so as to present overtake opportunities in portions of the track of varying levels of difficulty, while keeping the overall task short enough to ensure there is a rich interaction between the ego and opponent. Participants completed a series of warm-up trials in each domain, with three trials devoted to the straightaway segment and eight trials in the hairpin segment, each featuring different opponents of varying difficulty (fixed trajectories) to race against. Again, these were the same trajectories used in our domains.At the conclusion of each trial, participants answered the question: "Did you attempt to pass the other vehicle?" on an iPad. We also gathered, from trajectory data, whether or not the participant actually completed an overtake without collisions or spin-outs. These results are reported in Table B .1. We conclude that even in a simulated setting, there were a lower number of actual overtakes that occurred than were attempted. This suggests that there is room to assist those wishing to overtake, but unable to do so. We consider additional statistics, including statistics on left-and right-hand passing, as well as collisions with the other vehicle or objects, and spin-outs. We include these results in Table B .2. We note that there is a nearly-equal number of overtakes on the right versus left. On an individual level, we ran chi-square tests of test for given probabilities to look for side preferences. We found that only 6 of the 48 participants showed a statistically significant (p < .05) passing side bias, with two participants having a bias for the right side and four participants having a bias for the left side B.2. We also note that participants were, in general, imperfect in their driving, with nearly 50% of trials having a collision and 8% having at least one spin-out.In this section, we discuss the reward terms used to generate the explicit decision-making tendencies of the fictitious human drivers, via r H t = r task t + r y t . Each use the task-specific reward terms outlined in (1) , in combination with objective-specific rewards. Many of the task rewards are borrowed from [15] . We focus here on the human objective term r y t . Pass We adopt a dense reward that provides a penalty when the vehicle is behind the opponent vehicle, and a reward bonus when in front of that vehicle, up to a threshold, to incentivize passing. That is,r y t = c pass (∆s t -∆s t-1 ) I ((s low ≤ ∆s t ≤ s high ) ∨ (s low ≤ ∆s t-1 ≤ s high ))Where I(•) is the indicator function, and ∆s t is the difference in longitudinal positions, relative to track coordinates, between the ego and opponent vehicles, ∆s t = s ego t -s opp t . We take the scalar c pass = 10. In other words, if the difference between s ego t (the ego position) and s opp t (the opponent position) is between s low and s high , the passing reward is equal to 10 * (s ego t -s opp t ). This means there is a high positive reward for getting far ahead of the opponent, and a high negative reward for falling behind the opponent. For both the pass reward, we set s high = -s low = 800, which ensures that the pass reward is active for the entire trial. Note that we do not impose a progress reward with the pass objective.Stay-Behind Due to the non-symmetry of the problem, the stay-behind reward cannot be the complement of the pass reward (otherwise, the stay-behind agent would drive backwards to get away from the opponent). Because the task reward does not consider making progress, we add that here to the human-specific reward. The stay-behind reward is then:r y t = r prog t + c stay (∆s t -∆s t-1 ) I ((s low ≤ ∆s t ≤ s high ) ∨ (s low ≤ ∆s t-1 ≤ s high ))Where c stay = -2, and we impose a progress reward similar to [15] , i.e., r prog t = s ego t -s ego t-1 , with s ego t being the ego's longitudinal position in track coordinates. For the stay-behind reward, we set s high = -s low = 50. In practice, this means that the stay-behind agent is encouraged to make progress along the track (r prog ), but to stay at least 50 meters behind the opponent.Both the left-and right-biased passing agents are passing agents with an additional reward term that encourages a bias to the left or right. Note that these additional treatments do not guarantee passing on one side or the other.Left-Biased We adopt a reward bonus for driving on the opponent's left; i.e. rWe provide a summary of the DREAM2ASSIST training procedure in Algorithm 1. The procedure is split into two phases: the first is a human population generation phase in which we use the rewards in Sec. C to generate a population of humans included in the tuple ⟨W H i , Π H i ⟩ of world models and policies, respectively, and the expert human models denoted by the tuple ⟨W * i , π * i ⟩ for each human objective y i . The second phase entails drawing from samples of ⟨{y j } N j=0 , {W H j } N j=0 , {Π H j } N j=0 ⟩ using fictitious co-play (FCP) [27] in order to train the assistant's world model W A and policy π A . At runtime, both the trained policy π A and world model W A are executed, with W A being additionally useful as a means to interpret the decisions made by π A ; e.g. the intent estimate ŷA t , the estimated reward rA t or the latent variables ẑt . a much greater benefit than the standard DREAMER agent. Immediately after a random intent transition, the human agents perform very poorly and need significant assistance to get "back on track" to their known state distributions. DREAM2ASSIST consistently outperforms DREAMER for worse-performing human partners, as the intervention magnitudes for the DREAM2ASSIST agent consistently show more intervention and assistance. While the DREAMER agent pairs well with highly performant human partners (due to the low intervention magnitude), DREAMER is not able to offer as much useful assistance in this experimental setting.Finally, we show the intent-classification accuracy during a stay-to-pass transition episode with a medium-performance synthetic human (Figure G.3), and during a pass-to-stay transition episode with a high-performance synthetic human (Figure G.4). In these figures, we see that intent-classification accuracy is quite high before the random transition (> 90%), and drops sharply when the transition occurs (as expected). In the seconds that follow, the agent slowly recovers the new ground-truth intent as it observes different behavior from the human in the environment. Within 4-8 seconds after the random transition, we observe that the DREAM2ASSIST agent has recovered > 90% intent-classification performance with the new human partner. 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: Summary of the Paper

The paper introduces DREAM2ASSIST, a novel framework designed to improve shared control in high-speed racing scenarios by leveraging a modification of the Dreamer world model, which is enhanced to predict human intents. The approach involves training a recurrent state space model (RSSM) that combines both the dynamics of the physical environment and the intentions of human drivers. The assistive agent, based on this model, learns to provide minimal and targeted interventions that align with the predicted human intent, thus optimizing the joint performance of human-robot teams. The effectiveness of this method is demonstrated through simulations in a racing environment using the CARLA simulator, where the model shows improved performance over existing baseline methods.

Strengths

- The paper presents a novel application of the Dreamer world model to shared control scenarios, specifically in high-speed racing. The extension of the model to predict human intents and adapt its policy accordingly is a notable contribution.
- The overall structure of the paper is logical, with well-crafted figures that effectively illustrate the concepts and methodologies proposed. The visual aids, particularly Figures 1 and 2, enhance understanding.
- The proposed method demonstrates superior performance compared to baseline approaches, including naive Dreamer and imitation learning-based models, in simulated environments. The results show that the assistive agent can improve human-robot team performance by aligning interventions with human objectives.
- The work is relevant to the field of robotics, particularly in the context of human-robot interaction and shared autonomy, addressing a challenging domain of high-speed racing.
- The paper clearly addresses the limitations of the current work, providing a foundation for future exploration and improvements.

Weaknesses

- The evaluation of the method is restricted to a specific domain (racing) with a fixed set of human distributions. The lack of generalization to other domains and more complex human intent scenarios is a significant limitation.
- The experimental settings involve simplified scenarios with static human intents, which do not adequately capture the dynamic and interactive nature of real-world racing scenarios where human intents can change over time.
- The necessity of using the Dreamer model in this context is questioned by reviewers. Alternatives such as simpler intent predictors might achieve similar results, and the complexity of the Dreamer-based approach needs better justification.
- Some sections, particularly the experimental setup and results, require reorganization for clarity. Terms such as "intent" and "characteristics" need clearer definitions and earlier introduction to enhance comprehension.
- The reliance on a fixed set of fictitious human drivers during training raises concerns about the model's ability to handle out-of-distribution behaviors and real-world variability in human actions.
- While the approach is innovative, the reviewers perceive the contribution as incremental, extending existing work rather than providing a substantial breakthrough in the field.
- The current work does not include experiments with real human subjects, limiting the applicability and impact of the findings to real-world scenarios.

Summary of the rebuttal phase

While the authors' revisions appear to have improved the quality of the paper, the rebuttal did not change the reviewers' opinions. Although not all reviewers agree, acceptance for a poster presentation is judged to be reasonable.

Reviews: review: title: title: Review of submission 417
paper_summary: paper_summary: This paper proposes a novel method for shared control in high-speed racing scenarios using a modification of the Dreamer world model augmented to predict human intents. The world model and an assistive agent is trained on a suite of simulated humans demonstrating a different human intent (overtaking vs staying behind, right vs left) at varying levels of task proficiency. The authors show in two different simulated racing scenarios that the proposed method outperforms naive Dreamer and imitation learning-based approaches in providing assistance to suboptimal simulated human racers.
main_review: main_review: The overall flow of the paper is straightforward to understand, and is well summarized by Figures 1 and 2. The authors ground their work in the setting of shared autonomy for high-speed racing. The method extends the Dreamer RSSM model include a human-intent head that predicts one of several discrete human intents. Given the predicted human intent, the assistive agent is trained to provide a minimal change to the user’s control action to guide the linear combination of the human and agent control to closely match the optimal control for the given intent. 

The overall method seems sound, however the authors choose to train and evaluate in simplified settings with easy-to-specify intents that do not change over the interaction. I believe additional experimentation could help understand how the method adapts in closed-loop interaction, as competitive driving scenarios will have interaction effects that cause agents to mutually influence each other.  One potential way to accomplish this would be to simulate opposing agents using a simple ILQ Game solver like in [1] which also investigates the racing setting. In terms of writing, some notations are not defined before use. For example, the symbol x_t is not defined so it isn’t immediately clear whether this means state or observation without prior knowledge of the Dreamer papers. Defining these mathematical symbols could easily be added in Section 2. 

The work seems to make an assumption that the human intents (derived from hand-tuned reward functions) used during training sufficiently covers the space of possible human behaviors. The experiments are constructed such that this is easy to do (overtake vs stay behind, right vs left), however I am could imagine the method may struggle in scenarios with a wide range of possible human intents/the policy’s hypothesis space of intents do not cover the full range of potential human behaviors (the hypothesis space misspecification problem [2]). 


[1] Hu, Haimin, et al. "Think Deep and Fast: Learning Neural Nonlinear Opinion Dynamics from Inverse Dynamic Games for Split-Second Interactions."
[2] A. Bobu, A. Bajcsy, J. F. Fisac, S. Deglurkar and A. D. Dragan, "Quantifying Hypothesis Space Misspecification in Learning From Human–Robot Demonstrations and Physical Corrections,"


**Update post rebuttal** I thank the authors for taking the time to prepare a clear and concise rebuttal. My clarification questions have been addressed. While I still believe the space of human intents considered is rather narrow, the authors provide sufficient evidence that the assumptions made in this work are fair. The added experiments where human intents change during the interaction makes the method much more compelling.
questions: questions: - What happens when the human’s intent changes over time? If the RSSM and intent-predictor head is only trained on humans that maintain fixed intents over time, would this result in the assistive agent struggling to adapt as human intents change? I could see the human’s intent changing in scenarios where the opponent reacts to the human (the opponent isn’t giving me enough space to pass on the left, so I’ll pass on the right instead).
- What was the motivation for using Dreamer as a backbone? The observation space is the privileged state setting with no images. To my knowledge, Dreamer is typically used in settings where you do not have privileged state information and instead only have images (+proprioception). Couldn't you train an intent predictor from a history of past states/actions and ignore the RSSM entirely? Vehicle dynamics are fairly well understood so it is not clear to me why the method needs a latent dynamics model to train the policy either. Is it because high-speed racing induces more complex dynamics (i.e., drifting) that are hard to model?
limitations: limitations: 3
review_summary: review_summary: The paper's evaluation is relatively narrow. The application of high-speed racing is a highly interactive task, yet the evaluation is performed by competing against log-replay data of a human driver. Furthermore, they consider a setting where human intents are fixed when it is quite common for intents (passing on the left vs right) to change over the course of an interaction depending on the behavior of other agents. As it stands, I am not convinced that this method can actually be applied with or around real people.
title: title: Minor but solid contribution of a shared control algorithm
paper_summary: paper_summary: The paper presents a new formulation for an assistive agent in a shared control environment, tested on the CARLA simulator. The new agent is enhanced with an ability to represent the intent of the human teammate, based on several potential pre-learned human types. The paper is clear in most parts, and the empirical evaluation supports the superiority of the proposed approach compared to previous work.
main_review: main_review: Strengths: 
+ Mostly clear
+ Good, informative figures
+ Novel formulation for learning how to assist in a shared control environment
+ Solid empirical results

Weaknesses: 
- Some concepts could be better clarified (e.g., intent, objective, characteristics)
- Limited novelty (mainly, a new reward formulation that includes an ``intent’’)
- Limited evaluation (tested on only one domain, with one set of human distributions)
- Limited discussion over some decisions (representation, hyperparameters)

First, this paper presents a clear contribution: a new formulation of reward with which an assistive agent can learn to perform. It is an incremental work, as it extends an existing approach (DREAM) but clearly explains how it differs from it.
The figures are nicely done and provide good visual aids to better understand the proposed model. The empirical evaluation, while fairly modest, show clear superiority of the proposed approach.

Some concepts could be better explained: intent is the most prominent one. For example, this concept is referred to as a synonym for many different words, but never formally defined. Line 112 states, ``...includes the preferences, desires, and goals of the human (i.e., the human objective),’’ and then ``whose objectives (intents)’’, yet these concepts are not formalized properly anywhere in the texts. Eventually, in terms of the algorithmic representation, “intent” is translated into a portion of the reward that is dedicated to enforcing a certain preference (e.g. to stay in the right lane), which is not a goal. 

Similarly, the different “characteristics” that represent the different human agents could be explained and detailed better to help the reader understand how such human agents are expected to differ from one another. Moreover, the conditions ``pass vs. no-pass’’ and ``left vs. right’’ should be explained before they are used.

In terms of related work, there are several papers about shared control and shared tasks with goal monitoring (where intention in these papers is ``goal’’ in the BDI-literature sense):

1. Jiang, Y. S., Warnell, G., & Stone, P. (2021, May). Goal blending for responsive shared autonomy in a navigating vehicle. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 7, pp. 5939-5947).
2. Macke, W., Mirsky, R., & Stone, P. (2021, May). Expected value of communication for planning in ad hoc teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 13, pp. 11290-11298).
3. Dann, M., Yao, Y., Logan, B., & Thangarajah, J. (2021). Multi-Agent Intention Progression with Black-Box Agents. In IJCAI (pp. 132-138).

Also, using ToM to reason about other vehicles has much longer history than the one covered in this work:
4. Koller, D., Weber, J., Huang, T., Malik, J., Ogasawara, G., Rao, B., & Russell, S. (1994, October). Towards robust automatic traffic scene analysis in real-time. In Proceedings of 12th International Conference on Pattern Recognition (Vol. 1, pp. 126-131). IEEE.

Lastly, there is other work on shared control with unknown dynamics:
5. Broad, A., Murphey, T. D., & Argall, B. D. (2017). Learning models for shared control of human-machine systems with unknown dynamics. In 2017 Robotics: Science and Systems, RSS 2017. MIT Press Journals.

In terms of the evaluation, it was unclear what fictitious human drivers were used (were they the same ones used for training?). Also, this portion of the description could be revised: Line 216, ``To measure the contributions of each assistive agent for the fictitious human population, we sample checkpoints at every 20% performance increment for agents up to at least 75% of maximum, or from the bottom five performers if none are under this threshold.’’

Lastly, please go over the citations again to make sure they are all valid. E.g., citation [24] is missing a venue and [62] is missing a year.

*** Post-rebuttal:
I have read the reviews and the author's feedback. In the rebuttal, the authors have answered my questions and have specified changes that address most of my concerns.
questions: questions: Q1. Could you explain why the performance of DREAM2ASSIST “left/right” behavior in the straightaway case was not as good as it was in the other cases? 
Q2. Figure 3: Are the humans ordered from novice to expert? If not, that might help to see if the claim in Section 4.1 also holds here.
Q3. Have you looked at out-of-distribution behavior of agents? If so, where is it reported? If not, why?
Q4. How were the fictitious human drivers used in training vs. testing? Were they sampled from the same distribution?
limitations: limitations: 3
review_summary: review_summary: There is clear progress in the paper, with both a clear explanation of the algorithm and solid results on two use-cases. However, the contribution is limited.
title: title: Model-based RL approach for an assistive agent in racing scenarios
paper_summary: paper_summary: This paper proposes a Dreamer-based assistive agent for racing scenarios. The dynamics model is additionally trained to predict intent labels and condition the policy function to assist human drivers towards the aligned objective.
main_review: main_review: It was fairly easy to follow the proposed approach. But, the manuscript needs some improvement.

- It was not straightforward to follow Section 4. Some terminologies such as `pass vs no-pass` appear earlier, but the descriptions follow much later. And, Table 1 is presented too early because all other details are described much later as well. 
- Are there any plots to show "We observe a consistent trend for all straightaway results – DREAMER offers very low-magnitude intervention, leading to higher performance with expert drivers but poor performance with novice drivers (as the assistant is not helping)."? I believe this is something to show quantitatively. `Figure 3` seems implicitly showing this. Maybe, it's supposed to be in Section 4.1?
- There is inconsistency between `Fig. 4` and `Figure 3`. And, `Fig. 4` appears earlier than `Figure 3`.
- I wasn't conviced to use Dreamer for this work. For example, we can simply train a separate intent predictor to predict user's intention and use the prediction to condition the policy. In this way, the entire architecture could be simplified.  Authors need to explain necessity of the use of model-based RL approach for this problem.
questions: questions: - I suggest authors to rewrite Section 4 to re-oreder texts for the reason I described above.
- Authors need to justify the necessity of the use of Dreamer for this work. This feels unnecessary too complicated approach for the given task. It's possible that the simpler approach performs poorly if the racing task is too difficult for such methods, which could convince readers
limitations: limitations: 3
review_summary: review_summary: The manuscript needs to be improved and authors need to justify the use of Dreamer for this work.