Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racing
Authors: Jonathan DeCastro
Andrew Silva
Deepak Gopinath
Emily Sumner
Thomas Matrai Balch
Laporsha Dees
Guy Rosman
Venue: CoRL 2024 (2024)
Keywords: Recurrent State-Space Models
Human-Robot Interactions
Shared-Control
TL;DR: We propose a recurrent state space model capable of reasoning over human intents, enabling an assistive agent to select actions that align with the human and enabling a fluid teaming interaction, and demonstrate it in a high-speed racing domain.

Abstract: Tight coordination is required for effective human-robot teams in domains involving fast dynamics and tactical decisions, such as multi-car racing.  In such settings, robot teammates must react to cues of a human teammate's tactical objective to assist in a way that is consistent with the objective (e.g., navigating left or right around an obstacle). To address this challenge, we present _Dream2Assist_,  a framework that combines a rich world model able to infer human objectives and value functions, and an assistive agent that provides appropriate expert assistance to a given human teammate. Our approach builds on a recurrent state space model to explicitly infer human intents, enabling the assistive agent to select actions that align with the human and enabling a fluid teaming interaction. We demonstrate our approach in a high-speed racing domain with a population of synthetic human drivers pursuing mutually exclusive objectives, such as "stay-behind" and "overtake". We show that the combined human-robot team, when blending its actions with those of the human, outperforms synthetic humans alone and several baseline assistance strategies, and that intent-conditioning enables adherence to human preferences during task execution, leading to improved performance while satisfying the human's objective.

Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racing
Jonathan DeCastro", Andrew Silva", Deepak Gopinath, Emily Sumner, Thomas M. Balch,
Laporsha Dees, Guy Rosman
Toyota Research Institute, Cambridge, MA, USA, last@rirs global Contributed equally
Abstract:
Tight coordination is required for effective human-robot teams domains involving fast dynamics and tactical decisions, such as multi-car racing. In such settings, of human teammate's 's tactical objective
robot teammates must react assist in way that is consistent with the objective (e.g.. navigating left right around an obstacle). To address this challenge, we present DRFAM2ASSIST a framework that combines a rich world model able to infer human objectives and value functions, and an assistive agent that provides appropriate expert assistance to given human teammate. Our approach builds on a recurrent state space model explicitly infer human intents, enabling the assistive agent to select actions that align with the human and enabling a fluid teaming interaction. We demonstrate approach in a high-speed racing domain with population of synthetic human drivers pursuing mutually exclusive objectives, such as "stay- behind" and "overtake". We show that the combined human-robot team, when blending its actions with those of the human, outperforms the synthetic humans alone as well as several baseline assistance strategies, and that coneittondiiininn enables adherence to human preferences during task execution, leading to improved performance while satisfying the human's 's objective.
Keywords: Recurrent State-Space Models, Hlmaan-Robo Interactions, Shared- Control
Introduction
In high-stakes situations where members of team must coordinate their physical actions in the world for the team to succeed, early coordination on tactical objectives crucial. In a rapidly-evolving task, such as in high-speed competitive sports, agents must find way to attain such coordination without explicit communication. A robotic assistive agent equipped with an ability to reason using theory of mind [1] has been shown be critical successful collaboration without the burden and latency of explicit communication. In such settings, agents must maintain rich-enough model of the world to cover common set of concepts that each agent needs to plan. This includes dynamics of the physical world, the objectives of the team, and the current intent of the other members of the team. Such considerations are prevalent in sports [2, 3], manufacturing [4], healthcare [5], and traffic modeling settings [6], among others.
High-speed performance driving presents a domain where accurate and expressive models are required to have effective human-robot teams. The dynamics in racing evolve quickly, preventing team members from communicating their goals or objectives before taking an action [7]. Because of this constraint, existing approaches in shared control or advanced driver assistance often split authority on predefined boundaries (c.g. steering vs. throttle and brake) [8, 9]. The driving domain also requires tackle the multimodal nature of the human decision-making problem (e.g. rules or maneuvers),
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Generate diverse human behaviors
Train the Al assistant over population
World Madel 1 oz
IH
Environment
Infer
m
P
Assistive policy
CARLA
Poicy m!!
Stay Behind
Overtake
Deploy for real-time assistance
"The dr vor avertaking, so help them that better."
II
Figure 1: We mimic human preferences on discrete decisions via population clusters during world model formation. Our assistive models then learn decisively help on the overall control task while taking into account multiple possible human preferences.
where discrete decisions are required to made given knowledge of the situation [10, 11, 12, 13]. We hypothesize that theory-of- mindd-inspiieddd model, coupled with recent advances in flexible planning and reinforcement learning, can enable new ways for machines help humans drive more efficiently and effectively. We view work as complementary existing works in game-solving for racing [14] and planning in racing [15].
In this paper, we present a modeling paradigm to explicitly infer and support a human teammate's 's intent, enabling our agent to learn how interact with diverse human strategies in high-speed that
continuous control task. We consider the problem of robotic assistive agent in helps a human driver perform tactical overtaking maneuvers more safely and more optimally in high-speed competitive domain where explicit communication is difficult. While the driver remains in control of the vehicle at all times, the assistive agent can augment the driver's lateral (steering) and longitudinal (throttle and brake) actions. The assistant must infer the driver's intention (e.g., whether or not the driver is going to attempt an overtake) in order to provide optimal control augmentations and modifications that help the driver execute the inferred intention.
We address planning under the unobservable dynamics involved with human decisions acting in the world via a novel model-based reinforcement learning (MBRL) paradigm that jointly learns the dynamics of the physical world and the human's 's intentions, enabling fluid shared control with continuous assistance during a race. We contribute the learning of a recurrent state-space model within an MBRL setup in light of cognitive and neuroscience findings indicating human decision-making may be modeled such [16]. We infuse world model with richer rerddtrepresnntttion by using fictitious humans with multiple, mutually exclusive, objectives with the aim of encoding diverse human intents into the model's world representation. Specifically, we contribute:
1. A novel approach to MBRL for shared control in a tactical setting, leveraging driver intent modeling using fictitious co-play, conditioned on fixed set of human objectives.
2. A means for expert human value alignment, which conditions an assistive agent's rewards based on inferred intent to reason jointly over the physical world and the human's behavior.
3. An evaluation of our approach on a shared control racing domain, demonstrating our model's utility over a diverse set of imperfect fictitious humans with different personal objectives.
1.1 Related Works
Significant work has taken place in sharing control [8] and human-robot interaction [17, 18], exploring topics from ergonomics and physiology to language-basee strategic teaming. Prior research has

---

presented learning-base based approaches to human- interactions that target joint- repreeetrrennttinnn
learning [19, 20, 21, 22, 23, 24, 25, 26]. These works focus on a range of topics, including data- efficient representation learning [27], intent or plan modeling [28, 29, 30, 311, hypothesis-space specification and explanation [32, 24, 33), or other cognitive and social motivations [34, 35, 36]. We go beyond prior work by inferring the human's discrete intent and modifying our robotic assistant's objective to encourage support of the inferred intent.
In the context of driving, shared control has been explored for planning approaches [37, 38, 39, 40, 41, 42, 43, 44, 45] and learning-basse approaches [46, 47, 48, 49, 50). Prior work has also considered incorporating theory into shared control [50, 51], as well as explicit human-ccnnric design considerations [52, 53]. We refer readers to [54, 55] for comprehensive reviews. Beyond shared control, significant literature explored intent prediction in driving. see [56, 57], and references therein.
Recent work has proposed shared-control model using model-predictive control that considers predicted trajectory information [39], therefore implicitly capturing driver intent. Additional recent work has augmented
prredccteee controller with the ability to explicitly infer driver intent [9]. enabling the controller to share stecring and control actions with human. In our approach, we capture discrete driver intent in way that is conducive to aaaantiiaally-minnnfiii multi-modal continuous behaviors (e.g., going left or right around an obstacle). Further, by framing our task as a multi-agent reinforeement-lerming problem, our proposed solution extends to multiple agents much more naturally than prior work. Finally, we frame our decision making approach within a recurrent state-space model [58] which is extended infer the objective of the human, building on recent work that has explored hierarchical hybrid state abstractions [59, 60].
Background and Problem Statement
We target the problem of shared control in the highly-dynamic setting of high-performance racing against other racing opponents. We aim to build an assistive agent that is capable of reasoning over well-defined task objectives, as well as more general, harder-to-define, and harder-to-observe human objectives. The assistive agent is given a map of the track, states of the ego vehicle and opponent vehicle, the ego driver's steering and throttle controls. The agent's task to assist the ego driver through modifications to the steering and throttle of the ego car, depicted in Fig. (lower right).
To achieve optimal performance, the agent must provide continuous control adjustments as the driver progresses along the track, helping the driver to stay on course, maintain proper speeds, and avoid collisions. Further, the agent must accurately infer the intentions of the driver (such as "stay to the opponent's 's left") and provide control augmentations that help the driver to accomplish their immediate task objective more optimally. Note that this problem is different from a conventional autonomous driving problem, the agent's 's actions are conditioned explicitly on the human driver's s control input, and the control signal that actuates the car is a linear blend of the agent and the driver's s control signals. This problem also differs from conventional human-robot teaming, in that the agent must infer a human's 's intent (i.e.. the human objective) early and as consistently as possible, as the efficacy on its assistance (in terms of safety and performance) depends strongly to how early, accurately, and reliably the human objective is captured.
3 Approach
Our approach learns common latent representation in the structure of a recurrent state space model (RSSM) [61, 62) to build a hybrid discrete / continuous state of the human partner's behavior, their rewards, and intent for achieving certain goals. RSSMs have been shown to be suitable for many domains (e.g. locomotion, Atari, Minecraft), and we are the first to apply this to modeling human behavior for assistance. To accommodate both a human and assistive agent operating in a collaborative setting, we train the assistive agent alongside human agents, with both agents sharing actions taken in an environment. Each agent receives reward for each action taken, and each are trained to (1) build accurate RSSM and (2) learn how to act in order to maximize its own expected returns [63]. One challenge in collaboration is that part of the assistive agent's 's world model includes

---

P(
g4
Mooe
Solert
PM(3)
Figure 2: Overview of value alignment for the assistive agent. We start with a set of frozen human policies whose values are annotated with predetermined outcomes. We blend human actions linearly with the actions from an unfrozen assistive agent. For both human and assistive agents, a multi-head RSSM architecture is then used to predict the observations, reward, and human intent (assistive only), trained to maximize their log-likelihoowe head is trained to match frozen human intents, which are fixed priori. The assistive agent's rewards are shaped based on the optimal policy T and optimal predicted reward rg+ for inferred intent 1²
against samples taken from the environment. The intent
aspects of the future human's 's plan such as preferences, desires, and goals of the human (these are often collectively referred to objectives [32] or intents [30] the literature, we will use these terms interchangeably in the paper), which often unknown and only weakly observed by their behavior [32]. We extend the RSSM formulation to allow it to be supervised on a diverse set of humans, whose objectives (intents) are known and labeled, to force the representation to be jointly aware of the behavior and intent across variety of human types and physical environments. We then devise a scheme that feeds in inference of the human's objective into the assistant's 's reward function.
3.1 Building a World Model over the Human and Physical Environment
In order allow the system to maintain an ability reason the preferences of the human independent of that of the joint human-robot system, our approach considers both the human driver and the assistive agent planner as separate models. The training process is outlined in Fig. 2.
In the cooperative setting, the model of the environment follows a structure which there are certain task-specific rewards which are available to both human and assistive agents, with the key distinctions being human-objective (specific the human), and intervention penalties (specitic to the assistant). Task-Specific Rewards We assume standard task-specific rewards for high-performance driving from prior work [64, 15], including an out-of- -oounds penalty, passing reward, and collision penalty.
tosk It
collision + bounds + rt finish
(1)
where prooliiion is a negative reward for collisions, r is negative reward for driving too far off the track, and rf finish is positive reward for reaching the finish line.
Intent-Aware World Model The objective of the world model is to provide representation that the agent can use to interact with the driver and the world, and we posit that this representation can support the agent to reason jointly about both the task and human objectives. We build off of the recurrent state-space model (RSSM) of DreamerV2/V/ [65, 61], according to the architecture in Figure 2. For agent λ (H, A} (respectively, the Human and Assistive agent), the RSSM model, parameterized by and denoted Wy, includes:
Encoder for discrete representation 2f:
z (3(2fihi (h,, xx))
Sequence model for recurrent state h": h (hi a Dynamics predictor:
2 (3(()))))
where denotes the input observation. The output heads are similar to the DreamerV2/V3 architec- ture, with the addition of an intent predictor for the assistant agent (in red), and are all bottlenecked
on the hidden states s = {h, 2 }.
Decoder: a- ~ p (ā sf)
Continue predictor: 2 ~ p3(Cf117)
Reward: Pr ~ 2 (?!1s))
Intent predictor: af ~ pC

---

To train the intent predictor, we assume at least partial access to the intent labels for a given episode, which takes the form of an integer-valued target function. We normalize the output by predicting the symlog (sign(y) In(lvl + 1)) of the output, and use a discrete distribution to predict y using the two-hot encoding of [61]. The parameters & of the world model W are trained to minimize the loss
L (6) Eag [SpredLipred 1
The prediction loss Spred minimizes the likelihood under the predictor distributions in (3), KrT. minimizes the KL divergence between the prior P and the approximate posterior g5. and Spred and 3xL are scalar weighting values. Given an intent label 1/+, the portion of loss Lired for the intent predictor is taken as the negative log-likelihood of the label under pg.
The world model training is alternated with training for the behavior model governing actions af with the behavior model learned via an actor-critic policy training over the estimates 8t, ft.
Actor: of ~ u (a (
Critic:
V (8t) R Epg.rt [[]
The critic is trained to minimize the temporal-difference loss on the value function u. The actor attempts to maximize the hittcepredicced value. For further details, we refer the reader to [65].
3.2 Alignment with the Human
To learn alignment with human drivers, we expose the assistive agent to different humans at training time. Training with human partners in fictitious co-play setting [27] allows the assistant become robust different possible human behaviors, but does not teach the assistant to distinguish between the discrete modes of human behavior, which is necessary for assistance in driving task.
Similar to recent work describing human preferences for a task using reward shaping [66, 67]. we generate a population of humans, but different from these works, we group labeled uub-populations according certain inherent objectives or preferences. We train several uub-populations using separate human reward r raas + r²., which is composed of the base task rewards in (I), with the addition of several distinct human objectives, spanning different multimodal behaviors of the human. These are then formed into a collection of humans { _² ..... TW } E {II"" .I......II!! for N different objectives. Examples of different settings for r are given in Appendix C.
If trained optimality, such rewards induce optimal behaviors for each human objective T2..,..... N J. which we can consider as the collection of expert policies of each enumerated objective. We extend this notion to that of intent at runtime. At any given moment, the human may adopt an intent y to abide by policy n To simplify training, we consider drivers and rollouts with fixed intents, y, though our problem setup generalizes to variable-intent scenarios (e.g., a human switching from "following" to "passing" during race).
Humans are error-prone and pursue their objectives irrationally [68]. To capture this, we partially train human partners to pair with the assistive agent in a process known as fictitious co-play (FCP) [27], yielding a population of humans { U {m for each intent sub-population I1 where represents the policy checkpoint for some amount completed training. Using the FCP framework. each checkpoint i E [1, K] is an agent trained to A% completion and the final checkpoint h: K is an optimal policy. Each (sub)optimal human is endowed with (sub)optimal RSSM world model. While we apply FCP in this work, we note that our approach is agnostic to how human behaviors are generated and future work may consider other approaches to synthesizing human policies [69, 70].
3.3 Assistive Agent Objectives
For continuous actions a. a² E T for the human and assistant, respectively, we encourage the assistive agent to minimize its action intervention by adopting a reward of the form
interv
=||
which penalizes the magnitude of the intervention according L2-norm Aggregated over time, the intervention cost forms mixed-norm L1 L2 [71], encouraging time sparsity of interventions.

---

In addition to the intervention norm penalty, the agent is rewarded for task completion and penalized for driving out of bounds or for collisions. These sparse negative rewards serve to encourage the assistant to act as a "racing guardian", keeping the driver on the track and avoiding collisions while making progress towards the finish line. Note that these rewards align with the task objective (Sec 3.1), but do not favor any particular human objective or intent.
Finally, the assistive agent receives reward for aligning with the inferred objective of the human driver. First, the agent receives a reward equal the inferred reward under the optimal human's world model. In other words, at every time step t. the agent predicts the human's 's intent, gi and then queries the world model corresponding to the optimal human policy W d(i,)) for its reward prediction f (it") for the current state-action pair. Second, the agent receives reward for maximizing the likelihood of the human-robot's combined action under the optimal human model for the predicted human intent:
Te (a + a. (xx)) == x =a(aa" + ag [rr D12)
(5)
Note that of the current sub-optimal human action. Intuitively, these rewards encourage the assistant to: (1) select actions for which the optimal human assigns high value and (2) select actions that bring the sub-optimal human closer the optimal human. Under this setup, although we infer D/ from the joint human-roke trajectory, we consider it to be more aligned with the human's 's intent, due to: (1) the presence of the intervention sparsity term r riittrr, and (2) the fact that the inferred expert alignment term tilts the joint behavior toward the inferred expert human's 's behavior.
The complete reward for the assistant is a combination of rewards that promote successful task execution, minimize the magnitude of intervention, and account for human preferences:
Josk
interv
+ + (Dte) aa "allaa" + + aj (0t Ill
(6)
task performance intervention sparsity
expert alignment
Here, scalars O₂ and ag weight the contributions of the human terms. We capture both the inferred optimal reward value, f* (ūr"), ), as well as the similarity between the combined human-robot actions and an action from the inferred optimal policy a" a"(t") ~ The (aze). Note that, though we empirically compare different values of ar and oa in Appendix E, in the next section, we only examine results for the reward-only case (a,, = 1, aa = O).
Experiments and Results
We examine the performance of our approach, dubbed DREAM2ASSIST, on different racing tasks with a fictitious human driver. We examine two racing settings, each derived from portions of a two-mile race track, and implemented in the CARLA Simulator [72] using a rear-wheel race vehicle physics model. For each task, opponents are randomly instantiated as replays of real human trajectories from the track, meaning that they do not react to the ego vehicle. In each setting, we run two sets of experiments-onne one with pass vs. stay fictitious human partners (i.e., the human is trying overtake or stay-behind their opponent), and one with left right partners (i.e., the human wants to stay on the left or right side of their opponent). We also further examine out-of- distriiitibttoonn intent inference (with intent changing over time), and further ablations in Appendix F and G.
In each experiment, we train a population of humans following each objective (e.g., "left" or "right"), "), and then train a DREAM2ASSIS! agent over the combined population. We then evaluate the degree to which the assistive agent can improve fictitious human performance on the track, where performance is measured by total progress, average speed, and no collisions. To measure the contributions of cach assistive agent for the fictitious human population, we sample checkpoints at every 20% performance increment for agents up to least 75% of maximum, or from the bottom five performers if none under this threshold. We report the mean change in performance (track progress and collisions) when an assistive agent is deployed, as well as the return under each human objective in Table 1. Means and standard deviations are computed over all five sub-optimal fictitious drivers.
The two settings we consider include Straightaway Driving and Hairpin Driving. The straightaway is flat 370-meter portion of track with concrete barrier the right-hand side. Because the

---

Table 1: Results on straightaway and hairpin experiments. Blue indicates improvement, bold is best.
Pors tople Say theseno
Lek Eckotep//itte (iige (wsses)
Streghoicay
Pragce
Collicers
Prppoet
Colbaton
Prapce
Rosrs
Dalisaee-
Propax
Rctan
Cilliino
DAGAMEER
.1.35 13.6
8:58:25 2.
000 011
21B8 2007
-2 2.5
2:14
UAS T:3
001 EJ
Bansvers All
3692 46.3
2165t 645.6
4
0.1202
-15.64 56.
.2552449 453
-185
412 8.1
83
14660 183
-1911
-3.. 9.6
-452 11.3
842 0.3
-34448 15.3
135 392
-1448
235
45= (J
$055 467
088-94 94
0.1201
6388 364
40.2
011 01
11.38 23.
232:33
4028
-M1 F1.90
2.5-24
022 62
section of road is straight and flat, drivers need only avoid colliding with the concrete wall or their opponent while racing the finish. The hairpin section, however, is 960-meter portion of track involving several sharp turns and hills. Drivers must carefully manage their speed to avoid spin-outs or collisions, and passing is much more challenging in this section of track.
Human Decision Characteristics Based on observed behaviors from a human study (see Ap- pendix B), we propose two different sets of human characteristics (ground truth intent): pass vs. stay and left vs. right. We train fictitious human agents to satisfy each of these objectives. For the pass vs. stay behaviors, the fictitious human agents are trained to either overtake or stay-behind the opponent vehicle. For the left vs. right behaviors, the fictitious humans are trained with preference to stay on either the left or right side of the opponent for as much of the race as possible. Attempting to provide the wrong type of assistance with such distinct behaviors (e.g., offering "right" assistance "left" human) will result in fighting between the assistant and human, and will likely lead to collisions or spin-outs. We provide further details on the rewards for each agent in Appendix C.
Action and Observation Spaces The environment itself consists of the rewards in Sec. 3.1 and an observation and action space for each agent. The human and assistive agent's actions both adopt steering and acceleration values the range of [-1, 1]. The assistant and human actions are summed together and clipped on [-1, 1] before being passed the vehicle. The observation space contains the ego vehicle state (position, velocity, tire slip angles, yaw rate, heading), current distance traveled, and array of forward- barokid-looking track edge points. Finally, the human and assistive agents each able to observe the other's actions.
The claim of our work is that an assistive agent will better support a human partner if the assistant can infer the human's objective and help to satisfy that objective. To test this claim, we compare DREAM2ASSIST against a baseline version of DREAMER, which makes no intent inference and is not rewarded according to a human objective. We also compare to a non-RSSM baseline, GAIL [73], obtained with the BeTAIL [74] framework. BeTAIL uses behavior transformer as the human policy, coupled with an assistive agent trained via adversarial imitation learning to correct for bascline challenges whether our RSSM approach is necessary at all, or whether a purely data-driver approach using behavior cloning and inverse reinforcement learning could satisfy the multimodal behavioral assistance task that we consider. For this baseline, a fictitious human using a DREAMER policy is paired with the AIL assistive agent from BeTAIL.
Baselines
distribution shift. Our DREAMER-AN
4.1 Straightaway Results
We observe a consistent trend for all straightaway results DREAMER offers very low-magnitude intervention, leading higher performance with expert drivers but poor performance with novice drivers (as the assistant is not helping). We show track progress and human- return for assistants deployed to the pass vs. stay problem in Fig. F. (bottom). We report additional results in Appendix E. Conversely. DREAM2ASSIST offers higher-magnitude intervention, leading to higher performance with novice drivers, but lower performance gains with expert drivers (as the assistant is not needed). As shown in Table 1, DREAM2ASSIST generally offers the highest performance over other methods. DREAMER-ALE suffers from mode collapse, trying to turn all drivers into either "stay" driver or getting caught between "left" and "right" and therefore not moving. This behavior means that the DREAMER-AIL agent consistently underperforms fictitious human with no assistance at all, the AIL assistance keeps the driver far behind the opponent.

---

Ground Truth Intent: Pass
Ground Truth Intent: Stay
54.2
Pass complet
Dousts driver's speed to
Boosts driverss speed before turn
Nudiges di ver to the right
Boosts driver's speed poss
speed
Ego
Opponent
Accelaration Modriontion
Steering
Figure 3: Examples of the DREAM2ASSIS! agent's 's actions when paired with a human intending to pass and a human intending stay. DREAM2ASSIST recognizes the driver's ' intent, making lateral corrections for a safer overtake (left) or throttle adjustments to stay behind the opponent while still progressing towards the finish (right), thereby helping to satisfy task and human objectives.
4.2 Hairpin Results
The hairpin domain is more challenging and yields fictitious humans that are not always consistently able solve the task, thereby leaving greater scope for assistance from trained agents. In our pass stay experiment, DREAM2ASSIS, significantly improves the drivers' abilities to solve the task while still satisfying the human objectives and not leading to an increase in collisions. Similarly, in the left vs. right experiment, DREAM2ASSISS leads to significant increases in track progress, reduction in collisions, and improvements in human-objective alignment. Baseline approaches fail to disentangle the "left" and "right" modes of driving. The DREAMER baseline instead opts to push the driver to stay behind the opponent vehicle in an effort to reduce collisions, thereby making it farther down the track but failing to overtake. The DREAMER-AIL baseline drives aggressively off the track, leading to significant drop in track progress, collisions, and human-objective alignment. An illustrative example of DREAM2ASSIST is in Fig. 3, and videos are at https / //yuttu.uuuuu// be be/VuugoxgX5C..
Figure E. (top) provides overview of the amount of assistance provided to drivers for both track progress and human-objective return for assistants deployed to the pass stay problem. Visualizing the assistance provided to each driver, we can more easily compare the scale of assistance provided by DREAM2ASSIS9 compared to baselines, showing that DREAM2ASSIST provides marked improvements relative to baseline assistive agents.
Conclusions and Limitations
We introduce an assistive paradigm, DREAM2ASSIST, that learns to interact with humans to help them perform more optimally while supporting their personal objectives for the task. We evaluate DREAM2ASSIs! in dynamic and challenging task of high-speed racing, and we show that our approach is able to disentangle and accommodate distinct human objectives more effectively than baseline methods. We show that DREAM2ASS5 ASSIST results higher human-rohhe team performance than baseline methods, suggesting that explicit cineitccndiiioning and crucial performance gains in settings with multimodal, -exclusie, human objectives.
can provide
While PREAM2ASSIES represents a state-of-the-arr improvement human-robot teaming, there are limitations in our work that we hope address future work. First, our approach has been tested with fictitious humans, but have not yet evaluated generalization to real human-robot teams. In future work, we intend to deploy DREAM2ASSIST in human-subjects experiment to test how effectively our framework generalizes to real, sub-optimal human drivers. DREAM2ASSIST also relies on privileged access to the inferred reward values from the RSSM of an optimal policy; future work may consider how to estimate optimal policy rewards without such a model. Finally, future work may consider how to provide assistance via multiple modalities (e.g., providing sparse language guidance on when to overtake vs. dense control-level assistance during the overtake).

---

References
[1] M. Romeo, P. E. McKenna, D. A. Robb, G. Rajendran, B. Nesset, A. Cangelosi, and Hastie. Exploring theory of mind for human-robot collaboration. In International Conference on Robot and Human Interactive Communication (RO-MAN), pages 461-468. IEEE, 2022.
[2] T. Reimer, E. S. Park, and V. B. Hinsz. Shared and coordinated cognition in competitive and dy- namic task environments: An information-processine perspective for team sports. International Journal of Sport and Exercise Psychology, 4(4):376-400, 2006.
[3] R. Giske, J. Gjerde, R. Hoigaard, and B. T. Johansen. Importance, characteristics, and develop- ment. Journal of Expertise, 5(1):12-28, 2022.
[4] M. Demir, N. J. McNeese, and N. J. Cooke. Understanding human-robot teams in light of
all-human teams: Aspects of team interaction and shared cognition. Int. J. Hum. Compul. Stud., 140: 102436, Aug. 2020.
[5] J. Braithwaite, R. A. ledema, and C. Jorm. Trust, communication, theory of mind and the social brain hypothesis: deep explanations for what goes wrong in health care. Journal of Health Organization and Management, (445))3353-367, 2007.
[6] D. Koller, J. Weber, Huang, Malik, G. Ogasawara, B. Rao, and Russell. Towards robust automatic traffic scene analysis in real-time. In Proceedings of 12th International Conference on Pattern Recognition, volume 1, pages 22-1331. IEEE, 1994.
[7] H. Summala. Brake reaction times and driver behavior analysis. Transportation Human Factors, 2(3):217-226, 2000.
[8] D. A. Abbink, T. Carlson, M. Mulder, J. C. F. de Winter, F. Aminravan, T. L. Gibo, and E. R. Boer. A topology of shared control systems-finding common ground in diversity. IEEE Transactions on Human-Machine Systems, 48(5):509-525, Oct. 2018.
[9] E. Weiss and J. Christian Gerdes. Follow my lead: Designing an ADAS that shares decision making and control with the driver. In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 1482-1487. IEEE, Oct. 2023.
[10] N. Deo and M. M. Trivedi. Multi-modal trajectory prediction of surrounding vehicles with maneuver based Istms. 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1179-1184. IEEE, 2018.
[11] A. Censi, K. Slutsky, T. Wongpiromsarn, D. Yershov, Pendleton, J. Fu, and E. Frazzoli. Liability, ethics, and culture-aware behavior specification using rulebooks. In 2019 International Conference Robotics and Automation (ICRA), pages 8536-8542. IEEE, 2019.
[12] R. Karagulle, N. Aréchiga, DeCastro, and N. Ozay. Classification of driving behaviors using stl formulas: A comparative study. In International Conference Formal Modeling and Analysis of Timed Systems, pages 153-162. Springer, 2022.
[13] X. Li, G. Rosman, I. Gilitschenski, C.-I. Vasile, J. A. DeCastro, S. Karaman, and D. Rus. Vehicle trajectory prediction using generative adversarial network with temporal logic syntax tree features. IEEE Robotics and Automation Letters, 6(2)::459-3466) 2021.
[14] H. Hu, J. DeCastro, D. Gopinath, G. Rosman, N. E. Leonard, and J. F. Fisac. Think deep and fast: Learning neural nonlinear opinion dynamics from inverse dynamic games for split-second interactions, 2024. URL
wwt//oo//////aaoa//ooo 09810.
[15] R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlie, Eckert, Fuchs, L. Gilpin, Khandelwal, V. Kompella, H. Lin, MacAlpine, D. Oller, T. Seno, C. Sherstan, M. D. Thomure, H. Aghabozorgi, L. Barrett, R. Douglas, D. Whitehead, P. Dürr, P. Stone, M. Spranger, and H. Kitano. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature, 602((896):223- -228, Feb, 2022.

---

[16] D. Zoltowski, J. Pillow, and S. Linderman. A general recurrent state space framework for modeling neural dynamics during decision-making. In International Conference on Machine Learning, pages 11680-1169) PMLR, 2020.
[17] T. B. Sheridan. Human-robot interaction: status and challenges. Human Factors, 58(4): 525-532, 2016.
[18] L.. Onnasch and E. Roesler. A taxonomy structure and analyze human-robot interaction. International Journal of Social Robotics, 13(4):833-849 July 2021.
[19] A. Xie, D. Losey, R. Tolsma, C. Finn, and D. Sadigh. Learning latent representations to influence multi-agent interaction. In J. Kober, Ramos, and C. Tomlin, editors, Conference on Robot Learning, pages 575-588. PMLR, 2021.
[20] Reddy, A. D. Dragan, and Levine. Shared autonomy via deep reinforcement learning. Feb. 2018.
[21] M. Tucker, Y. Zhou, and J. Shah. Latent space alignment using adversarially guided self-play. International Journal of Moaman-Computte Interaction, pages 1753-1771, 2022.
[22] S. Karamcheti, M. Srivastava, P. Liang, and D. Sadigh. LILA: Language-Informed latent actions. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1379-1390. PMLR, 2022.
[23] A. Bobu, Y. Liu, R. Shah, D. S. Brown, and A. D. Dragan. SIRL: Similarity-basee implicit representation learning. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Roue Robot Interaction, pages 565-574 665-574, 2023.
[24] I. Sucholutsky, L. Muttenthaler, A. Weller, A. Peng, A. Bobu, B. Kim, B. C. Love, E. Grant, J. Achterberg, J. B. Tenenbaum, et al. Getting aligned on representational alignment. arXiv preprint arXiv:2310. 13018, 2023.
[25] S. Parekh and D. P. Losey. Learning latent representations co-adapt to humans. Autonomous Robots, 4776):77, 2023.
[26] H. Jeon. D. P. Losey, and D. Sadigh. Shared autonomy with learned latent actions. ArXiv, abs/2005.03210, May 2020.
[27] D. Strouse, K. McKee, M. Botvinick, E. Hughes, and R. Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 14:14502-1457 2021.
[28] R. C. Kelley. Models of intention for human-robot interaction. 2013.
[29] S. Thill and T. Ziemke. The role of intentions in human-robot interaction. In International Conference on Human- Robot Interaction, HRI '17, pages 427-428, New York, NY, USA, Mar. 2017. Association for Computing Machinery.
[30] D. P. Losey, C. G. McDonald, E. Battaglia, and M. K. O'Malley. A review of intent detection, arbitration, and communication aspects of shared control for physical human-robot interaction. Applied Mechanics Reviews, 70(1):010804, 2018.
[31] Y. Cheng, I.. Sun, C. Liu, and M. Tomizuka. Towards efficient human--oboccolaahortioo with robust plan recognition and trajectory prediction. IEEE Robotics Automation and Letters, 5(2): 2602 2609, Apr. 2020.
[32] A. Bobu, A. Bajesy, J. F. Fisac, S. Deglurkar, and A. D. Dragan. Quantifying hypothesis space misspecification in learning from human demonstrations and physical corrections. IEEE Transactions Robotics, 36(3):835-855) 2020.

---

[33] R. Setchi, M. B. Dehkordi, and J. S. Khan. Explainable robotics in human-robot interactions. Procedia Computer Science, 176:3057-3066, 2020.
[34] C. Lebiere, F. Jentsch, and S. Ososky. Cognitive models of decision making processes for human-robot interaction. In Virtual Augmented and Mixed Reality. Designing and Developing Augmented and Virtual Environments: 5th International Conference, VAMR 2013, pages 285- 294. Springer, 2013.
[35] B. Mutlu, N. Roy, and S. Šabanović. Cognitive human-robot interaction. Springer Handbook of Robotics, pages 1907-1934, 2016.
[36] H. Admoni and B. Scassellati. Social eye gaze in human-robot interaction: review. Journal of Human-Robot Interaction, 6(1):25-63, 2017.
[37] C. G. Bobier, S. Joe, and C. Gerdes. Sliding surface envelope control: Keeping the vehicle within a safe state-space boundary. In Dynamic Systems and Control Conference, volume 44175, pages 149-156, 2010.
[38] C. G. Bobier and J. C. Gerdes. Staying within the nullcline boundary for vehicle envelope control using a sliding surface. Veh. Syst. Dyn., 51(2):199-217, Feb. 2013.
[39] J. Talbot, M. Brown, and J. C. Gerdes. Shared control up to the limits of vehicle handling. IEEE Transactions on Intelligent Vehicles, pages 2977 2987, 2023. doi: (0.1100//TV.2223333009999
[40] W. Schwarting. J. Alonso-Mora, L. Paull, S. Karaman, and D. Rus. Parallel autonomy in auto- mated vehicles: Safe motion generation with minimal intervention. In 20/7 IEEE International Conference on Robotics and Automation (ICRA), pages 1928-1935, May 2017.
[41] A. Benloucif, A.-T. Nguyen, C. Sentouh, and J.-C. Popieul. Cooperative trajectory planning for haptic shared control between driver and automation in highway driving. IEEE Transactions Industrial Electronics, (6(12):9846-9 9857, 2019.
[42] Y. Liang, Z. Yin, and L. Nie. Shared steering control for lane keeping and obstaele avoidance based on multi-objectivv mpc. Sensors, 21(14))467,, 2021. 2021.
[43] M. Dann, Y. Yao, B. Logan, and Thangarajah. Multi-agent intention progression with black-box agents. IJCAL, pages 132-138. 2021.
[44] Y.S. Jiang, G. Warnell, and P. Stone. Goal blending for responsive shared autonomy in navigating vehicle. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5939-59491 2021.
[45] W. Macke, R. Mirsky, and P. Stone. Expected value of communication for planning in ad hoc teamwork. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11290-11298, 2021.
[46] P. G. Griffiths and R. B. Gillespie. Sharing control between humans and automation using haptic interface:
Primary and secondary task performance benefits. Human Factors: The Journal of Human Factors and Ergonomics Society, 47:574-590, Sept. 2005.
[47] M. Mulder, D. A. Abbink, and F. R. Boer. Sharing control with haptics: seamless driver support from manual to automatic control. Human Factors, 54(5):786-798, Oct. 2012.
[48] A. Balachandran, M. Brown, S. M. Erlien, and J. Christian Gerdes. Predictive haptic feedback for obstacle avoidance based model predictive control. IEEE Trans. Autom. Sci. Eng.. 13(1):
26 Jan. 2016.
[49] A. Broad, T. Murphey, and B. Argall. Learning models for shared control of human-machine systems with unknown dynamics. In Proceedings of Robotics: Science and Systems, Cambridge, Massachusetts, July 2017, 105.00.. 2011 XIIL.037.

---

[50] A. H. Ghasemi. Game theoretic modeling of steering operation in a haptic shared control framework. ASME 2018 Dynamic Systems and Control Conference, Nov. 2018.
[51] S. C. Jugade, A. C. Victorino, and V. B. Cherfaoui. Shared driving control between human and autonomous driving system via conflict resolution using non-cooperative game theory. In ITSC, pages 2141-2147 IEEE, Oct. 2019.
[52] H. Deng, Y. Zhao, S. Feng, Q. Wang, and F. Lin. Shared control for intelligent vehicle based on handling inverse dynamics and driving intention. IEEE Trans. Veh. Technol., 71(3):270-22720, Mar. 2022.
[53] J. Xie, X. Xu, F. Wang, Z. Liu, and L. Chen. Coordination control strategy for Sluman-Maching cooperative steering of intelligent vehicles: A reinforcement learning approach. IEEE Trans. Intell. Transp. Syst., 13111772116--1117,,, Nov. 2022.
[54] M. Marcano, S. Díaz, J. Pérez, and E. Irigoyen. A review of shared control for automated vehicles: Theory and applications. IEEE Transactions on Human-Machine Systems, 50(6):
475-491, Dec. 2020.
[55] J. Sarabia, M. Marcano, J. Pérez, A. Zubizarreta, and S. Diaz. A review of shared control in
automated vehicles: System evaluation. Frontiers in Control Engineering, 3, 2023.
[56] S. Lefèvre, D. Vasquez, and C. Laugger. A survey on motion prediction and risk assessment for intelligent vehicles. ROBOMECH Journal, 1(1):1-14, July 2014.
[57] A. Rudenko, L. Palmieri, M. Herman, K. M. Kitani, D. M. Gavrila, and K. O. Arras. Human motion trajectory prediction: a survey. Int. J. Rob. Res., 39(8):895-935, July 2020.
[58) D. Hafner, T. Lillicrap, J. Вa, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representationss 2019.
[59] A. Zholus, Y. Ivehenkov, and A. Panov. Factorized world models for learning causal relation- ships. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, Mar. 2022.
[60] C. Gumbsch, N. Sajid, G. Martius, and M. V. Butz. Learning hierarchical world models with adaptive temporal abstractions from discrete latent dynamics. In The Twelfth International Conference on Learning Representations, 2023.
[61] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. Jan. 2023.
[62] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555-2565. PMLR, 2019.
[63] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018
[64] I.. Chen, S. Manuel, Delgado, Subosits, and P. Tylkin. Learn thy enemy: Online, task-aware opponent modeling in autonomous racing. 2023.
[65] D. Hafner, Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02/93, 2020.
[66] Z. Tang, C. Yu, B. Chen, H. Xu, X. Wang, F. Fang, S. Du, Y. Wang, and Y. Wu. Discovering di- verse multi-agent strategic behavior via reward randomization. arXiv preprint arXiv: a00iv:2003.0455550 003.04564, 2021.
[67] C. Yu, J. Gao, W. Liu, B. Xu, H. Tang. J. Yang, Wang, and Y. Wu. Learning zero-shot cooperation with humans, assuming humans are biased. arXiv preprint 200::23002.06055. 2023.

---

[68] S. E. Chipman. The Oxford Handbook of Cognitive Science. Oxford University Press, 2016.
[69] Park, J. O'Bren, C. J. Cai, M. R. Morris, Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. In ACM Symposium User Interface Software and Technology, pages 1-22, 2023.
[70] D. Venuto, S. N. Islam, M. Klissarov, D. Precup, S. Yang, and Anand. Code as reward: Empowering reinforcement learning with VLMs. arXiv preprint arXiv:2402.04764, 2024.
[71] M. Elad. Sparse and Redundant Representations. Springer New York.
[72] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban driving simulator. In Conference on Lobot learning, pages 1-16. PMLR, 2017.
[73] J. Ho and Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 29, 2016.
[74] C. Weaver, C. Tang, C. Hao, K. Kawamoto, M. Tomizuka, and W. Zhan. BeTAIL: Behav- ior transformer adversarial imitation learning from human racing gameplay. arXiv preprint arXiv:2402.14194 2024.

---

A MBRL Preliminaries
We frame the model-based reinforcement learning (MBRL) problem as a two-player (human ego driver, assistive agent ego driver) partially-osservable Markov decision process (POMDP) defined where, for agent K E {H, A} (for the human, and AI
by the tuple . = (X"", ,A", T", R² RR",)"" , agent, respectively), A² denotes the imagined states of the world, AA denotes the agent's (continuous or discrete) actions, 7 A R A + [0, is the transition probability, R X A X I reward function, and y [0, is discount factor. We aim train both agents such that they maximize their expected returns It Σー
Crucially, in the semi-cooperative shared control setting, each reward rf is factored into sub- components, with both sharing the same task (driving) rewards, but where r? contains an additional term for a human's 's objective, and r contains additional terms to weaken its contribution relation to the human's 's and enforces alignment to the human.
B Human Subject Data Collection
We briefly discuss a study conducted for gathering human subject behavior data the racing domain we use in the paper. The purpose of the study was to gather qualitative and statistical data on individuals" behavior and objectives in racing context, and to that to inform what criteria important for building models of human objectives. We recruited 48 participants to drive simulator with the hairpin and straightaway segments of the two-mile track, the same domains for the computational results in this paper. The scenarios were chosen so as to present overtake opportunities in portions of the track of varying levels of difficulty, while keeping the overall task short enough to ensure there is a rich interaction between the ego and opponent. Participants completed a series of warm-up trials in each domain, with three trials devoted to the straightaway segment and eight trials in the hairpin segment, each featuring different opponents of varying difficulty (fixed trajectories) to race against. Again, these were the same trajectories used in our domains.
At the conclusion of each trial, participants answered the question: "Did you attempt to pass the other vehicle?" on an iPad. We also gathered, from trajectory data, whether or not the participant actually completed overtake without collisions or spin-outs. These results are reported in Table B. We conclude that even in simulated setting, there were a lower number of actual overtakes that occurred than were attempted. This suggests that there is room to assist those wishing to overtake, but unable to do so.
Table B.1: Number and percentage of overtakes occurred and attempted. Note the diversity in intent and in tvertakin-ddifficlle for the subjects, motivating the need for assistive shared autonomy.
Overtake occurred?
Frequency
Percentage
Overtake atlempted?
Frequency
Percentage
"No" "Yes"
178
30.07
"No" "Yes"
65
11.02
414
69.93
525
88.98
We consider additional statistics, including statistics on left- and right-hand passing, well as collisions with the other vehicle objects, and spin-outs. We include these results in Table B.2. We note that there is a nearly-equal number of overtakes on the right versus left. On an individual level, we ran chi-square tests of test for given probabilities to look for side preferences. We found that only 6 of the 48 participants showed a statistically significant (p < .05) passing side bias, with two participants having a bias for the right side and four participants having a bias for the left side B.2. We also note that participants were, in general, imperfect in their driving, with nearly 50% of trials having collision and 8% having at least one spin-out.
C Human Objectives
In this section, discuss the reward terms used to generate the explicit dicision-making tendencies of the fictitious human drivers, via r! rask + r.. Each use the task-specific reward terms outlined

---

Overlake Percentage Participant
Partsopann TD
Percentage Overtales
Figure B.1: Consistency of overtake versus non-overtakes.
Lefl Overlake Percenlage per Participant
701
75/75
01.5%
Parto parie
E01 18.5%
140%
50.6
lago Ovartakes
Figure B.2: Consistency of left-handed versus right-handed overtakes.
in (1), in combination with objective-specific rewards. Many of the task rewards are borrowed from [15]. We focus here on the human objective term r..
Pass We adopt a dense reward that provides a penalty when the vehicle is behind the opponent vehicle, and a reward bonus when in front of that vehicle, up to threshold, to incentivize passing. That is,
P = Cpass (Ast T S Ast S 8high) V (Stow S AA0 S Shigh))
Where T(.) is the indicator function, and As; is the difference in longitudinal positions, relative to track coordinates, between the ego and opponent vehicles, As, s dpp We take the scalar Cpass = 10. In other words, if the difference between 5,90 (the ego position) and SPPP (the opponent position) is between Stow and Shigh, the passing reward is equal to 10 (x.90 8
= This means
there is high positive reward for getting far ahead of the opponent, and a high negative reward for falling behind the opponent. For both the pass reward, we set Shigh = -Stow 800, which ensures that the pass reward is active for the entire trial. Note that we do not impose a progress reward with the pass objective.
Stay-Behind Due to the non-symmetry of the problem, the -thy-d reward cannot be the complement of the pass reward (otherwise, the stay-behind agent would drive backwards to get away

---

Table B.2: Other effects.
Percentages are percent of trials with the listed event.
Observed event
Percentage
Left-handed overtakes
50.82%
Right-handed overtakes
49.1 49.18%
Collisions Spin-outs
48.66%
8.36%
from the opponent). Because the task reward does not consider making progress, we add that here to the numan-specifee reward. The stay-behind reward is then:
= .r.oog + Cstay (As: As;-1) (Ssoww I As S Shigh) V (Stow I As S Shigh))
Where Catay = 2, and we impose progress reward similar to [15], i.e., r: 00 8 with s: being the ego's longitudinal position in track coordinates. For the stay-behind reward, we set Shigh =
ago
Slow 50. In practice, this means that the stay-behind agent is encouraged to make progress along the track (rrrog), but to stay at least 50 meters behind the opponent.
Both the left- and eight-biased passing agents are passing agents with an additional reward term that encourages bias to the left or right. Note that these additional treatments do not guarantee passing on one side or the other.
Left-Biased We adopt a reward bonus for driving on the opponent's left; i.e.
r = (As; As ((siow I As S Shigh) V (Stow I As S Shigh)) + (Der + Cmargin)
where De; is the difference in lateral positions of the two vehicles, in the track coordinate frame; i.e. Ae; ego opp and Cmargin is margin (which we set to Cmargin 0.3).
Right-Biased Right-biased reward is the complement of the left-biased reward:
r (Ast As ((stow I As, S Shigh) V (Stow S Asc-1 shigh)) (Der Cmargin)
D Additional Model Details
We provide a summary of the DREAM2ASSIST training procedure in Algorithm 1. The procedure is split into two phases: the first a human population generation phase in which we use the rewards in Sec. C to generate a population of humans included in the tuple (W _1 of world models and policies, respectively, and the expert human models denoted by the tuple (W?", mT) for each human objective ys- The second phase entails drawing from samples of (1y jjo [W/' Jo=0- (1I (1 1 1
using fictitious co-play (FCP) [27] order to train the assistant's world model W and policy A At runtime, both the trained policy A and world model W4 are executed, with WA being additionally useful means interpret the decisions made by <,; e.g. the intent estimate oAA the estimated reward PA or the latent variables 2,
Table C.1: Training Hyperparameters.
Hyperparameter
Value
Hyperparameter
Value
Encoder decoder MI.P layers
Steps
2e6
Encoder decoder MLP units
512
Batch size
16
Predictor head layers
Batch length
64
Predictor head units
512
Training ratio
512
Discount factor Discount À
0.997
Model learning rate
le-4
0.95
Value learning rate
3e-5
Imagined horizon
15
Actor learning rate
3e-5
Actor entropy
3e-4
Dataset max size
le6
Dynamics hidden units
512
Steps between evaluations
lc4
Dynamics discrete dimension
32
# Episodes to evaluate
10

---

Algorithm DREAM2ASSIST using FCP
Given: diverse intents, Di {1,2..... 3 Given: reward functions for each intent y/ for V/, & E (1, 2, ..... M) do Initialize ;;;; W:"
D Generate human population
while not converged do Sample an opponent policy Topp from Hopp Initialize Zg. with t = while not done do
Perform gradient step; update T." W." Sample action at from nr
Step the environment with action a and a a TP Shape rewards according to ith agent reward
+-
end while
Append checkpoint (W... m./) (WH,, )
end while
Append final m?" (W!,, I end for
Freeze agents and world models (W Jo=01 ( 3ر Initialize πΛ, W
while not converged do
Train assistant agent
Sample intent i from (Lu))
I.M
Sample checkpoint for intent y from ((W#) LN Jz=0: ( }
Sample opponent policy Topp from Hopp Initialize Vo. with - while not done do
Perform gradient step: update vA W using ground truth label
Sample action at from (x), a from u.. (xx)
Step the environment using shared action a." + aa, and a. uP Evaluate intent úf using WAA
Shape rewards as per (6), using r" (i(A) from Wj. ((A) ~ Tzg. (3x))
f + ++1
end while end while
D.1 Training Hyperparameters and Environment Specifies
We provide the DREAM2ASSIST hyperparameters in Table C.1. We train using the Adam optimizer for 2 10 steps.
The CARLA simulator is used for our environment, and is executed with step size of 0.1 sec. We terminate episodes if: (a) the ego collides with the opponent or other collidable objects (c.g. static barriers), (b) the ego vehicle veers too far off course, or (c) predefined finish line reached. The map used geospatially-calibaae representation of the Thunderhill Raceway in Willows, CA.
E
Additional Experimental Results
E.1 Summarized Performance
We summarize the discussion in Sec. 4.2 and 4.1 here in Fig. E.1.
E.2
Performance Across Different Humans
We provide more complete comparison the results, showing additional metrics in an evaluation that extends Fig. 3 from imperfect to near-perfect humans (1-11, ordered according to unassisted track progress performance) for DREAMER, DREAMER-AIL. BETAIL, and DREAM2ASSIST. We show these results in Figs. E.2-E.5. Note that the additional fictitious humans (6-11) achieve

---

Truth
Stay
Soenario: Halrpin
100
200
Scanarto: Strolghtoway Propee
pon
425-
400
450
Figure E.1: Change in track progress and return when adding assistance to five imperfect pass (left) and stay (right) fictitious humans the hairpin and straightaway problem settings. The addition of DREAM2ASSIST leads to higher gains in task performance and greater adherence to human objectives than baselines.
nearly-identical bascline performance across all metrics, indicated by Figs. E.3 and E.5. Hence, the changes across humans 6-11 in Figs. E.2 and E.4 are likewise similar.
From Fig. E.2, we observe that DREAM2AsSIS5 when applied the imperfect pass humans (1-5) generally yield improvements in progress, collisions, and speed, with slight overall decrease in reward, and an overall moderate intervention level compared to the baselines. For humans 6-10, progress, reward, collisions, and speed are all negatively impacted, hinting at room for improvement in handling near-perfect humans. Similar trends for imperfect humans (1 (1-5) tending to stay can be seen in Fig. E.4, where all metrics except collision see improvement. For near-perfect humans (6-11), the results generally indicate marginal improvement across all metrics, except collision.
We also complement Table with additional metrics, including the magnitude of intervention, and speed, both averaged over time. Tables E. and E.2, we compare Dreamer, DREAM2ASSIS and DREAM2A DREAM2ASSIST-AL SSIST-AI II, and further include results for variant of DREAM2ASSIST with the action- based reward of DREAM2ASSIST with the action-based reward term and reward, i.e. 0₂ Or, 0, which For the pass stay case in Table E. DREAM2ASSIS, achieves
(6) using a, = a₂ = which we call DREAM2ASSIST-aR as well variant
we call DREAM2ASSIST best performance in categories. while improving over unassisted humans in 10 categories. The performance of DREAM2ASSIST-A and DREAM2ASSIST+an were mixed. DREAM2ASSIST+a was able to improve over unassisted humans, but with generally lower progress than DREAM2ASSIST, while DREAM2ASSIST-
almost completely hindered human's progress, due to the fact that the reward term no longer explicitly captures the dense progress sub-reward. and are not implicitly reflected in the actions of the optimal human. In E.2, we similar trends, with DREAM2ASSIST outperforming baseline approaches in 5 categories, and performing better than unassisted humans in 8 categories. DREAM2ASSIST - - --r is unable to make progress, and DREAM2ASSIST- also reveals lower progress than the DREAM2ASSIST in the hairpin and straightaway domains.
E.3 Intent Classification Performance
We next probe the performance of the intent classification. Fj scores achieved on training data yields high performance, as shown in Table E.3.
We provide two example time traces illustrate stability of inferring the human's intent by the assistant's world model in Figs. E.7 and E.8.
To uncover whether intent inference is due to world model training, we evaluate the t-SNE embeddings of the logits of the assistant's discrete latent state 2A We see that in the Dreamer case, t-SNE is unable find strong separations between the ground truth intent classes without intent inference in the latents, while in DREAM2ASSIST, there is stronger separation, and the ground truth intent

---

Change Prggee
50
100
-200
FiceBouua Harminn
HcmmnousH.mon
Ge
04
Change Coliion
02
Change
DREAMER-AIL
0.8
DREANZA8BEST
Figure E.2: Changes in various metrics in the hairpin scenario when adding assistance to various imperfect (1-5) and near-perfect (6-11) humans tending to pass, averaged over four random seeds. Due to the fact that BeTAIL uses its own internal human model, compare only one instance / human, as denoted by the red line.
400
300
-10
30:
.40
144
12
30-
10
25
0.8
02
40
Pictious Humonn
Figure E.3: Absolute metrics in the hairpin scenario evaluated for the unassisted imperfect (1-5) and near-perfect (6-11) humans tending to pass, averaged over four random seeds. error bars are shown.
classes are more strongly clustered in the embedding space, allowing intent to be inferred with much higher accuracy.
E.4 Learning Curves
We provide learning curves for the DREAMER. DREAM2ASSIST and DREAM2ASSIST-aL assistance schemes the two domains across all the human objectives in Fig. E.9. We compare these across track progress, and observe a general trend of stability in training.
E.5 Visuals of Assistance
We show, in Fig. E. CARLA and bird's s-ee-vveew snapshots of driving with assistance, and compare that to human without assistance in Fig. E.10.

---

Chenge Progros
800
200
-000
Intevction Intencton Magetudo
Fictiious
VECEROus/HLLreE Paamin
60
0.4
04
ChargeinCatoos
03
Change Spocd
02-
FRcStious Murmam
01
DREAMER-AIL
0
DREAM2ASSIST
BETAIL
Figure E.4: Changes in various metrics in the hairpin scenario when adding assistance to various imperfect (1-5) and near-perfect (6-11) humans tending to stay, averaged over four random seeds. Due to the fact that BeTAIL uses its own internal human model, compare only instance / human, as denoted by the red line.
500
400
100
6
0.5
05
02-
10
HD
Focticus
Figure E.5: Absolute metrics in the hairpin scenario evaluated for the unassisted imperfect (1-5) and near-perfect (6-11) humans tending to stay, averaged over four random seeds. 1-o error bars are shown.
F Comparisons Without a Recurrent State Space Model
For completeness, we compare to a simple windowed-observatinee baseline agent that does not use recurrent state space model, yet still learns to perform intent classification. We keep our training pipeline and synthetic human partners the same as that used for DREAM2ASSIST for these experiments. This agent (No-RSSM) is given 16 frames of prior history (corresponding about 1.5 seconds of observations) and is trained with the same rewards as DREAM2ASSIST in addition to an auxiliary thent-classification objective. We train this agent to assist pass and stay-behind human partners on the hairpin section of the track.
The results of this experiment are in Figures F.I & F.2. We observe that the No-RSSM baseline performs significantly worse than DREAM2ASSIST and worse than most RSSM-based methods, despite having high intent classification accuracy (98% Fl, equal to the DREAM2ASSIST agent during

---

Table E.1: Improvement unassisted humans for the pass-stay humans straightaway and hairpin experiments, with statistics aggregated across four random seeds. Blue indicates improvement over unassisted humans, bold is best.
Pasa (Sops/ Say (bstion))
Harpa
Strighrowyy
Spasi
Intervations
DREAMER
21.12 65.6
11.32 13.6
0.34 0.4
6.5 29
0,02 0.0
01222
0.1 1..0..0
0.22::2.2
0.
29 16
0.75 4,6
6.1 17.5
8:82 0.1
.0.1 0.9
0.1:022
0.02:00
0.1 1:0.0
0.1 2:00
[03
40.4 19
DREAMER-AIL
-18.9
172004
01
54
-116 38.7
4a4 8.3
1.:1:00
-124
105
DREAMEASSEET
60.5 69.7
0.:
18
KBB
$7.82
4018 0.2
01:01
04:
1:00
25.3 11.0
2.1 7.
03
072-012
08:
6.9 2: MM
5.0 244
101:0:02
088 0.3
147
24.3 66.6
02:88:05
0.0 04
0.7 8.0
5.1 3.
9.2.1.75 7.8
-6.62.0.
0.10
0.9 2.0.
-9.0 55.2
11.5=33
0.05
86-
-1 5.6
400 20.3
-2.9
00 00:00:1
8-0.
.174112 1.2
228.9
-25
4.
0.7.0.0 0.0
-14-22..
104
-1.0
0.
18
training). While the No-RSSM agent is trained with the same rewards as the DREAM2ASSIST agent, it regularly terminates episodes early by steering the human driver out of bounds, which could be the result of a poor understanding of when it will receive positive or negative rewards (as the two intents result in opposite reward signals). As the No-RSSM agent is not trained to predict reward, this information may not be as explicitly captured in its internal representation as would be in an RSSM based agent, such as DREAMER or DREAM2ASSIST.
G Evaluations With Random Intent Transitions
While our primary experiments focus on humans with a static intent (i.c., the human is attempting to pass or stay-behind for the entire episode), here we include additional experiments in which our synthetic humans randomly change intents in the middle of an episode. For these experiments, we run 10 trials and, after randomly sampled number of episode steps, we swap out the synthetic human for a comparably-traiee trained synthetic human with the opposite intent. For example, we might start an evaluation trial with pass human that is trained for 10K steps and, after 10 seconds of the trial, suddenly swap the synthetic-human policy to a stay-behind human that has been trained for 10K steps.
We run these experiments with a standard DREAMER baseline and our DREAM2ASSIST agent, evaluating the impact our iewerded-reward and intent classification objective. These experiments are repeated with 5 random seeds for total of 50 episodes for each randomized transition. Random seeds are kept consistent for the DREAMER and DREAM2ASSIST experiments, the 50 randomly-selected transition times are consistent for the two agents.
We note that our assistants have never been trained under such dynamics, and so these transitions highly outttf-diitributiin for both the assistants and the synthetic humans. Our synthetic human agents are therefore very likely to crash, spin-out, or otherwise perform far worse than usual. To evaluate our assistive agents in this domain, we compare the collision rate, average change in speed, average track progress (i.e., task-completion) and intervention norm from the assistive agents for randomized pass-to-stay transitions (Figure G.I) and randomized stay-to-pas to-ssssssssssseeo transitions (Figure G.2).
While the DREAMER and DREAM2ASSIS! agents can both help to mitigate the problems inherent to a random intent transition (collisions and spin-outs), we observe that DREAM2ASSIST offers
Table E.2:
Improvement over unassisted humans for the left-rght humans on straightaway and hairpin experiments, with statistics aggregated across four random seeds. Blue indicates improvement over unassisted humans, bold best.
Lefi clop) Right (wamm)
Harpa Collisons
Stagittawy Collsicts
Progresu (m)
Retura
Entevvctions
Speed (m/))
Progres (os)
Retue
Interventons
Speed (as))
DECAMIR
160a 17.9
26/7
2204 2.2
-133 1.5
011:011
0,0c
0.2 0.1
0114 6.1.1
0.1 1.5
-.1:622
4100 0:5
6.28 1,3
0.2 0.1
0.1 2.00
0:24:00
0.92 1.0
DREomur-AI.
.119 70/4
.144.02 89:9
-7.18 8.6
17.3
-842 B.2
122 12:2:01
0.
-144
1.
134.42 13.0
1136
-114-70
-352
+8.5 6.1
144-00
14:100
-311-11
-1142 7.5
DREAMASSIST
27.26 23.1
2.3
3.:24.7
-L1
285 2.4
2.1
9.3 32.0
33:2433
0.1 0.1
0.9 00:2:11
19:258
18.3 12.9
02:8:12
022
0.5 0.1
49
-1.2 40.5
2.1.1:2.
.02 1.3
09 0.1
-14111
-82 38.6
JU1 0.3
1.0
-14.0 1.0
DREASEASSHSS-
114.3
24.241111
.44
.04
0.5 0.1
-6
61
126
13
011000
00
0.:000
04410.1
61
$2 39

---

Table E.3: F1 F1-Scores the training set.
Pass vs. Stay Hairpin
Pass vs. Stay Straightaway
Left vs. Right Hairpin Left vs. Right Straightaway
0.99 0.006
1.00 0.000
0.95 + 0.00
0.98 + 0.00
pass
Pass ground truth
60
Stay ground truth
Inference
-60
-60
-40
-20
20
40
60
(a) Dreamer
40
Poss ground truth
pass
Stay ground truth
20
10
Inference
-10
200
-30
-40
stay
(b) DREAM2ASSIST
Figure E.6: t-SNE embeddings 2 for: (a) the non-intent-awart world model of Dreamer versus (b) the intent-supervised world model of DREAM2ASSIST. The consistency of the clusterings present in the DREAM2ASSIST world model states indicates that the world model has learned to identify the human's intent.
much greater benefit than the standard DREAMER agent. Immediately after random intent transition, the human agents perform very poorly and need significant assistance to get "back on track" to their known state distributions. DREAM2ASSIS, consistently outperforms DREAMER for worse-perforne perse-perforiig human partners, as the intervention magnitudes for the DREAM2ASSIS, agent consistently show more intervention and assistance. While the DREAMER agent pairs well with highly performant human partners (due to the low intervention magnitude), DRFAMER is not able to offer as much useful assistance in this experimental setting.
Finally, show the intent-classifcattio accuracy during stay-to to-y-to-sss transition episode with cediumm-pefformance synthetic human (Figure G.3), and during pass-to-stay transition episode with

---

a perfforrror synthetic human (Figure G.4). In these figures, we see that intent-classificatt accuracy is quite high hefore the random transition (> 90%), and drops sharply when the transition occurs (as expected). In the seconds that follow, the agent slowly recovers the new ground-truth intent as observes different behavior from the human in the environment. Within 4-8 seconds after the random transition, we observe that the DREAM2ASSIST agent has recovered 90%% catstt-cassiification performance with the new human partner.

---

Intervention over time for a passing agent
inntt Suy
80
Retadive staanee of Ego Osp (m) (rah))
Assistant provides high-magnitude acceleration nudge driver into overtake when they close.
Steer ng intervetion (450)
60
(.00)
40
Arann
CoanAAAN MA
-20
-40
5
10
15
20
25
30
Time (s)
Figure E.7: Time traces in the hairpin scenario showing the intent inference (denoted by background color), along with ego-opponent distance, speed, and DREAM2ASSI6 steering and acceleration modifications for a human tending to pass. Notice that DREAM2 ASSIST maintains an accurate estimate of the driver's intent, and provides high-magnitude acceleration intervention to assist as the ego begins to overtake the opponent.
Intervention time for stay-behind agent
50
25
-25
Helaaviee Datance of Egg Copp (n))
Figo (mmh))
Staer rg Intervaniion (c50)
Acze wendon Inservention (:201)
-75
-10
-125
10
20
30
40
Time (s)
Figure E.8: Time traces in the hairpin scenario showing the intent inference (denoted by background color), along with ego-opponent distance, speed, and DREAM2A ASSIST steering and acceleration modifications for human tending stay.

---

300
140
120
100
(m)
Wwm
Progress (ml)
80
Dreom2Assist
Dream2AAssit
Dregm2Aseestro
Dream2Asssatta
Drcomer
Dreamer
20
Process time (hr)
Process time (hr)
Pass-Stay Humans in the Hairpin Domain
(b) Pass-Stay Humans in the Straightaway Domain
150
www. Mo un www
250
Progress (m)
Progressa (rmil
100
100
Drsom2Assistt
Dream221ssit
Dreon2Aesist-o
Dream2Assistto
Dreomer
Dreamer
10
20
30
40
30
20
Process time (hr)
Process time (hr)
(c) Left Right Humans in the Hairpin Domain.
(d) Legh-Right Humans Straightaway Domain.
Figure E.9: Learning curves for each domain, averaged over four random seeds.

---

Ego vehicle speed going turn
Human driver early
Ego vehicle spins
Figure E.10: Example time sequence an imperfect passing human driving in the hairpin domain.
Assistant helps with steering braking to avoid collision
Assistant gives steering nudge help the driver
Assistant acceleration boost to avoid
Assistant slow the vehicle avoid spin-out
Figure E.11: Example of a time sequence of DREAM2ASSIST assistance to help an imperfect passing human in the hairpin domain.

---

100
cherge PiTrquee
Chonge Rotu
100
-00
Intuecee Wagtbull
2000
20
04-
02-
chongo1 80o00
<02
DREAMER-AIL
-36-
No.RSBM
DREAMER
DREAM2ASDIET (REWARD)
REWARD)
DIEAM2As91s (ACTion)
Figure F.1: Changes in various metrics in the hairpin scenario when adding assistance various imperfect (1-5) humans tending to pass, averaged over five random seeds. Note that our two No- RSSM baseline is trained to perform intent classification and achieves high accuracy, yet performs far worse when paired with sub-optimal synthetic human partners than our RSSM basclines.
Charree Natam
10
-900
Ga
Fictllooo Humion
10
0.
0.
Change Spmne
01
DREAMER-AI AIL
-01
No RSSM
-00
DELAMUI
DeeAM2As9le (eewARо)
Piztttous Hamnon
DReAM2Asoist (ACTION REwARD)
DREAM2Assist (ACTON)
Figure F.2: Changes in various metrics in the hairpin scenario when adding assistance to various imperfect (1-5) humans tending stay-behind, averaged over five random seeds. We again note that the No-RS NO-RSSM baseline is trained to perform intent classification and achieves high accuracy, yet performs far worse when paired with sub-optimal synthetic human partners than our RSSM baselines.

---

DREAMER
DREAM2ASSIST (REWARD)
120
100
Change in Prggess
80
60
Change
2-
40
20
2
Fictitious Humar ID
Flatit oua Humar ID
C8
0.05
Intervention Magnitude
Change in Collsion
.005
03
0.
-01
0015
3 Fictit ous Humar ID
5
2 3
Fictitious Humon
Figure G.I: Changes in various metrics in the hairpin scenario when adding assistance to imperfect (1-5) humans whose intent randomly swaps from stay-behind to passing, averaged over five random seeds and ten trials each. We observe that our inferred-reward objective and our mnssttcllasificattion objective enable the DREAM2ASSIST agent to generalize much better this highly out-of-distrrbution behavior, particularly with significant reductions in collisions and improvements in track progress (i.e., task completion).

---

DREAMER
DREAM2ASSIST (REWARD)
80
8
70
60
Change in Speed
Change in Pragress
4
40
20
10
0
5
2
5
Fictitious Humar D
Fictitious Human ID
0.12
01-
0.5
Interventonn Magnitude
0.07
Crange in Celision
0.4
0.05
0.02
0.3
0.2
0.02
01
-0.05
2
3
Fictitious
Fictitious Humon
Figure G.2: Changes in various metrics in the hairpin scenario when adding assistance to imperfect (1-5) humans whose intent randomly swaps from pass stay-behind, averaged over five random seeds and ten trials each. Again, our inferred-rewant objective and our intent-classifican objective enable the DREAM2ASSIST agent to generalize much better this highly particularly with respect to track progress.
behavior,

---

1.2
Mean Accuracy
1.0
Intent Prediction Accuracy
0.8
0.6
0.4
02
[-8, -4]
[-4, -2]
[-2, -1
[1,
[0, +1]
[+1, +2]
[+2, +4]
[+4, +8]
[+15, -]
Bucketed Seconds From Change-Point
Figure G.3: Intent-classification accuracy before and after a synthetic human partner is randomly swapped from medium- stay-behind observe that the DREAM2AsSIST agent able to accurately classify its partner's ' intent the seconds leading up to the change, which point intent-classification accuracy drops sharply. In the following seconds,
pecdium-performant pass partner. We
classiflasiii accuracy climbs back up to about 90%, indicating that DREAM2ASSIS) is able to recover human partner's intent if it dynamic. The performance dip at the end is likely due to sub-optimalities in the pass partner's policy, leading to collisions or spin-outs.
1.2
Mean Accuracy
1.0
Intent Prediction Accuracy
0.8
0.6
0.4
0.
0
[-8, [-4,
[-2,
[-1. 0]
[+], +2]
[+2, +4]
[+4, +8]
[+15, -]
Bucketed Seconds From Change-Point
Figure G.4: Intent-classification accuracy before and after a synthetic human partner is randomly swapped from a high- -erfforefnrn pass to nigh-performing stay-behind partner. We observe that the DREAM2A ASIST agent is able to accurately classify its partner's intent in the seconds leading to the change, at which point massnt-llassfiication accuracy drops sharply. intent-classffication accuracy climbs monotonically after the change, suggesting that DREAM2ASSIST can perfectly recover intent for high- pethh-rerfomiggggg agents, even under unexpected and previously-unsee unseen transitions.


Reviews:
[{"review_id": "UfareyhQVa", "reviewer": "Reviewer_jTCE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel method for shared control in high-speed racing scenarios using a modification of the Dreamer world model augmented to predict human intents. The world model and an assistive agent is trained on a suite of simulated humans demonstrating a different human intent (overtaking vs staying behind, right vs left) at varying levels of task proficiency. The authors show in two different simulated racing scenarios that the proposed method outperforms naive Dreamer and imitation learning-based approaches in providing assistance to suboptimal simulated human racers.", "summary_of_recommendation": "The paper's evaluation is relatively narrow. The application of high-speed racing is a highly interactive task, yet the evaluation is performed by competing against log-replay data of a human driver. Furthermore, they consider a setting where human intents are fixed when it is quite common for intents (passing on the left vs right) to change over the course of an interaction depending on the behavior of other agents. As it stands, I am not convinced that this method can actually be applied with or around real people.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "- What happens when the human’s intent changes over time? If the RSSM and intent-predictor head is only trained on humans that maintain fixed intents over time, would this result in the assistive agent struggling to adapt as human intents change? I could see the human’s intent changing in scenarios where the opponent reacts to the human (the opponent isn’t giving me enough space to pass on the left, so I’ll pass on the right instead).\n- What was the motivation for using Dreamer as a backbone? The observation space is the privileged state setting with no images. To my knowledge, Dreamer is typically used in settings where you do not have privileged state information and instead only have images (+proprioception). Couldn't you train an intent predictor from a history of past states/actions and ignore the RSSM entirely? Vehicle dynamics are fairly well understood so it is not clear to me why the method needs a latent dynamics model to train the policy either. Is it because high-speed racing induces more complex dynamics (i.e., drifting) that are hard to model?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uWv0CPArZ5", "reviewer": "Reviewer_5oAj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a new formulation for an assistive agent in a shared control environment, tested on the CARLA simulator. The new agent is enhanced with an ability to represent the intent of the human teammate, based on several potential pre-learned human types. The paper is clear in most parts, and the empirical evaluation supports the superiority of the proposed approach compared to previous work.", "summary_of_recommendation": "There is clear progress in the paper, with both a clear explanation of the algorithm and solid results on two use-cases. However, the contribution is limited.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "Q1. Could you explain why the performance of DREAM2ASSIST “left/right” behavior in the straightaway case was not as good as it was in the other cases? \nQ2. Figure 3: Are the humans ordered from novice to expert? If not, that might help to see if the claim in Section 4.1 also holds here.\nQ3. Have you looked at out-of-distribution behavior of agents? If so, where is it reported? If not, why?\nQ4. How were the fictitious human drivers used in training vs. testing? Were they sampled from the same distribution?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "44SBMHcKBs", "reviewer": "Reviewer_TZys", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a Dreamer-based assistive agent for racing scenarios. The dynamics model is additionally trained to predict intent labels and condition the policy function to assist human drivers towards the aligned objective.", "summary_of_recommendation": "The manuscript needs to be improved and authors need to justify the use of Dreamer for this work.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- I suggest authors to rewrite Section 4 to re-oreder texts for the reason I described above.\n- Authors need to justify the necessity of the use of Dreamer for this work. This feels unnecessary too complicated approach for the given task. It's possible that the simpler approach performs poorly if the racing task is too difficult for such methods, which could convince readers", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary of the Paper

The paper introduces DREAM2ASSIST, a novel framework designed to improve shared control in high-speed racing scenarios by leveraging a modification of the Dreamer world model, which is enhanced to predict human intents. The approach involves training a recurrent state space model (RSSM) that combines both the dynamics of the physical environment and the intentions of human drivers. The assistive agent, based on this model, learns to provide minimal and targeted interventions that align with the predicted human intent, thus optimizing the joint performance of human-robot teams. The effectiveness of this method is demonstrated through simulations in a racing environment using the CARLA simulator, where the model shows improved performance over existing baseline methods.

Strengths

- The paper presents a novel application of the Dreamer world model to shared control scenarios, specifically in high-speed racing. The extension of the model to predict human intents and adapt its policy accordingly is a notable contribution.
- The overall structure of the paper is logical, with well-crafted figures that effectively illustrate the concepts and methodologies proposed. The visual aids, particularly Figures 1 and 2, enhance understanding.
- The proposed method demonstrates superior performance compared to baseline approaches, including naive Dreamer and imitation learning-based models, in simulated environments. The results show that the assistive agent can improve human-robot team performance by aligning interventions with human objectives.
- The work is relevant to the field of robotics, particularly in the context of human-robot interaction and shared autonomy, addressing a challenging domain of high-speed racing.
- The paper clearly addresses the limitations of the current work, providing a foundation for future exploration and improvements.

Weaknesses

- The evaluation of the method is restricted to a specific domain (racing) with a fixed set of human distributions. The lack of generalization to other domains and more complex human intent scenarios is a significant limitation.
- The experimental settings involve simplified scenarios with static human intents, which do not adequately capture the dynamic and interactive nature of real-world racing scenarios where human intents can change over time.
- The necessity of using the Dreamer model in this context is questioned by reviewers. Alternatives such as simpler intent predictors might achieve similar results, and the complexity of the Dreamer-based approach needs better justification.
- Some sections, particularly the experimental setup and results, require reorganization for clarity. Terms such as "intent" and "characteristics" need clearer definitions and earlier introduction to enhance comprehension.
- The reliance on a fixed set of fictitious human drivers during training raises concerns about the model's ability to handle out-of-distribution behaviors and real-world variability in human actions.
- While the approach is innovative, the reviewers perceive the contribution as incremental, extending existing work rather than providing a substantial breakthrough in the field.
- The current work does not include experiments with real human subjects, limiting the applicability and impact of the findings to real-world scenarios.

Summary of the rebuttal phase

While the authors' revisions appear to have improved the quality of the paper, the rebuttal did not change the reviewers' opinions. Although not all reviewers agree, acceptance for a poster presentation is judged to be reasonable.
Decision: Accept