Dynamic Multi-Team Racing: Competitive Driving on 1/10-th Scale Vehicles via Learning in Simulation
Authors: Peter Werner
Tim Seyde
Paul Drews
Thomas Matrai Balch
Igor Gilitschenski
Wilko Schwarting
Guy Rosman
Sertac Karaman
Daniela Rus
Venue: CoRL 2023 Poster (2023)
Keywords: Multi-Agent
Reinforcement Learning
Sim-to-Real Transfer
Autonomous Racing

Abstract: Autonomous racing is a challenging task that requires vehicle handling at the dynamic limits of friction. While single-agent scenarios like Time Trials are solved competitively with classical model-based or model-free feedback control, multi-agent wheel-to-wheel racing poses several challenges including planning over unknown opponent intentions as well as negotiating interactions under dynamic constraints. We propose to address these challenges via a learning-based approach that effectively combines model-based techniques, massively parallel simulation, and self-play reinforcement learning to enable zero-shot sim-to-real transfer of highly dynamic policies. We deploy our algorithm in wheel-to-wheel multi-agent races on scale hardware to demonstrate the efficacy of our approach. Further details and videos can be found on the project website: https://sites.google.com/view/dynmutr/home.

Dynamic Multi-Team Racing: Competitive Driving on 1 10-tth Scale Vehicles via Learning in Simulation
Peter Werner" Tim Seyde" Paul Drews", Thomas Balch", Wilko Schwarting, Igor Gilitschenski", Guy Rossman', Sertac Karaman", Daniela Rus
MIT CSAIL, Toyota Research Institute, 3 UofT RI, MIT LIDS, cqual contribution (wernerpe, tseyde @mit.edu
Autonomous racing is challenging task that requires vehicle han- dling at the dynamic limits of friction. While single-agent scenarios like Time Trials are solved competitively with classical model-based or model-free feed- back control, agget wheel-to-wheel racing poses several challenges includ- ing planning over unknown opponent intentions as well as negotiating interac- tions under dynamic constraints. We propose address these challenges via a learning- based approach that effectively combines model-based techniques, mas- sively parallel simulation, and self-play reinforcement learning to enable zero- shot sim-to-real transfer of highly dynamic policies. We deploy our algorithm in wheel-to-wheel multi-agent races on scale hardware to demonstrate the efficacy our approach. Further details and videos can be found on the project website: https://sites.geee google.......ooooooooooo.. com/viewiiiiie wwt.:vveee/yyymtr////////////
Abstract:
Keywords: Multi-Agent RL, Autonomous Racing, Sim-to-Real Transfer
Opeck
O
Arggel
Observations
Opock
Figure 1: We train multi-team racing agents simulation and transfer them directly to hardware. Our approach leverages a hierarchical policy as a layer of abstraction between strategic planning and continuous control. A high-level categorical policy predicts goal distributions for a low-level policy achieve, effectively projecting long-term decision making into limited preview horizon.
Introduction
Autonomous racing in multi-vehicle team setting is a highly challenging task. requires safely handling vehicle its dynamic limits while simultaneously reasoning about opponents and team strategy. These challenges push the limits of autonomous navigation and have, thus, attracted consid- erable academic interest. At the same time, solving autonomous racing promises to impact decision- making for real-world autonomous vehicles. The frequent interactions and driving at the limits of controllability are reminiscent certain near-accident scenarios which are absent from common autonomous vehicle research datasets [1].
The multi-agent learning community has focused largely simulated challenges such [2, 3, 4J. We see multi-team racing as step towards simulation and hardware benchmark that is both
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

accessible and realizable without the use of proprietary hardware, due to the vibrant open-source community revolving around scaled-car racing [5].
In the considered multi-team racing task, two teams of two agents compete for team win. A team's rank is determined by its best-ranked team member the end of race. Solving this task effectively is challenging because it requires both long-term strategic planning (e.g. coordination with teammates setting up overtaking maneuvers over multiple comers of the track) and fast, responsive short-term planning for swift and precise handling of the vehicle (e.g. to avoid collisions or to counteract unexpected slippage).
Most autonomous racing systems follow a highly structured pipeline similar traditional driving pipelines. This allows for fine-graine mne-ggained design of each component within perception, planning, and control. In contrast, purely learning-basee bbatd approaches to autonomous racing allow addressing fun- damental questions of embodied Al research such as the emergence of social behavior, embodied learning setup, and safe real-world policy transfer. However, these have mostly been tested in simu- lated environments, due to the difficulty of large-scale experience data collection on real hardware. Simulations present many ways to diversify and extend data collection. In particular, simulating competitive multi-agent scenarios enables self-play with large agent populations and avoids poten- tial damage to hardware systems. Yet this day, sim-to-real transfer in domains where neither driving at the limits nor wheel-to-wheel contact dynamicc are well modeled remains challenge.
In this work, we demonstrate a sim-to-real approach to multi-team racing. We expand on successful hierarchical policy architectures and leverage parallelized self-play in simulation to learn competi- tive agents with variety of emergent behaviors, that transfer onto real hardware. Toward this aim, our work makes the following contributions:
A light-weight GPU-accelerated simulation environment for multi-team racing, augment- ing first principles-based with data-driven models to facilitate highly parallelized learning.
hierarchical policy structure that serves as a layer of abstraction to disentangle long-term team-centric strategic planning from short-term ego-centric continuous control.
Training in simulation with zero-shot transfer to scale race cars, demonstrating dynamic maneuvers in highly interactive wheel-to-wheel multi-team racing scenarios on hardware.
Specifically, we review related work in Section followed by our approach for environment mod- cling and control in Sections and 4, respectively. We describe details of our race car platform in Section 3.1, followed by results of transfer for soeeel-tt-heed racing on hardware in Section 5.
2 Related Work
This work presents a platform that advances motion planning for autonomous racing, multi-agent reinforcement learning (MARL) applications, and practical approaches for dynamics modeling and sim-to-real transfer. Pushing the limits of autonomous vehicles through autonomous racing has a recent surge of interest [6]. Investigated systems include both reduced scale platforms [5, 7, 8), full scale platforms [9], as well as arcade-style simulated systems [10, 11, 12].
Motion Plamning Motion planners can be grouped into three main categories. First, obstacle avoidance planners treat other agents (henceforth referred as ado-agents, [13]) moving obsta- cles without explicit consideration of interactions between agents and their objectives. This includes collision-free path identification via dynamic programming [14], filter-based prediction [15], rein- forcement learning (RL) [16], graph-based planning
[17], and adapting a low-level optimization either via learned reward specifications [18] or trajectory warm-starts [19]. Second, game-theorene planners explicitly represent interaction strategies ado agents and iterate until an equilibrium is reached. This includes formulations as sequential bi-matrix games [20] or iterative dynamic pro- gramming in belief space [21], with analysis commonly limited to 1-vs-1 scenarios. Iterative best response for 6 vehicles is showcased in [22], with the prevalent drawback that game theoretic plan- ners introduce a trade-off between solution quality and real-time capabilities. Finally, barning-based

---

planners only implicitly reason about interactions from prior data and do not compute explicit ado plans. In [23], hand-crafted scenarios and rewards are combined with model-free RI. to beat human players in the game of Grand Turismo, while [11] learns a latent world-model that reasons about opponent behavior for competitive 1-vs-1 racing via self-play. The hierarchical approach in [12] enables a high-level policy propose lane change sequences low-level controller in response to ado behavior. Our approach leverages hierarchical policy structure as a layer of abstraction be- tween team-centric strategic planning and ego-centric continuous control, learning efficient multi- agent interactions from experience and displaying emergent behavior via self-play. This further enables real-time inference when scaling to many-vs-many races on hardware.
Multi-Agent Reinforcement Learning This work considers a mixed, noperative-competiiie racing scenario, where teams receive joint reward for their overall performance, with only very limited regularization terms signaling individual contribution. Similar problems extensively studied in the MARL literature [24, 25, 26, 27]. In particular, we consider the centralized training decentralized execution (CTDE) paradigm with factored (action) value functions as described in [28, 29, 30J. These approaches attempt to learn individual payoff assignments between cooperating agents. This enables decentralized exccution by greedy action with respect to the individual payoff. Alternatively factored policies can be directly trained using actor-critic approaches such as Proximal Policy Optimization [3 We make use of CTDE to train the high-level policy with DecSARSA, an on-policy adaptation of decoupled Q-Networks [30), learning credit assignment for team members.
Dynamics Modeling and Sim-to-Reat Transfer To accelerate learning, we leverage massively parallel simulation and training running directly on the GPU [32]. A wide variety of dynamics mod- els are used in practice for autonomous vehicles. For example, the authors in [33] use dynamic single-track bicycle model with Pacejka tire models [34] and model randomization for policy learn- ing. The same model formulation was used in [35] containing system identification of parameters for the FeTth platform. In [36], physics-informee informed basis functions are combined with Bayesian regression to fit dynamics model used for predictive control. For comprehensive overview of dy- (sim-to-real) transfer. See [37, 38] for select surveys on sim-to-real approaches and [27] for MARL transfer onto hardware.
namics models for racing, see [6]. Finally, our work uses simulation-to-rean
Simulation and Dynamics Modeling
Our approach leverages a highly parallelized, light-weight simulator for time-efficient behavior learning on GPUs. The simulator merges data-driven dynamics obtained through system identi- fication on hardware with simplified analytic models to generate high-fidelity interactive scenarios.
3.1 Hardware Platform
We employ the TRIKart hardware platform developed by the Toyota Research Institute [8], a vari- ation of the Fl vehicle design [5]. Each vehicle has an Nvidia Jetson onboard computer to handle communication with the VESC motor driver, steering servo, and the base station computer. The base station provides real-time control commands together with state estimates obtained from an OptiTrack motion capture system. We further outfit each vehicle with a I Lexan cover augmented with ABS bars to mitigate damage during vehicle-to-vehicle contact scenarios.
3.2 Dynamics Model
The simulator leverages a switched dynamics model conditioned on the contact state of the system. Out of collision, we employ the kinematic bicycle model [39] augmented with data-driven models. During collisions, we employ the dynamic bicycle model [14]. Both variations are implemented as elementary tensor operations in PyTorch for rapid data generation [40], and described below.
3

---

Kiramanii Bieycle
Cullision
Ageni
Dynamic Ricyclo Model
Figure 2: Left scale TRIKart platform. Right Block diagram of the switched dynamics. The simulator augments kinematic bicycle model with data driven front slip angle and drive train dynamics learned from the real vehicle. If a collision event is detected, the dynamics are switched temporarily a lower fidelity dynamic model that directly models the effects of external wrenches.
Oversteer
Understcer
2000
2000
7
5500
400
200
500
500
-0.4
-0.
0.0
0.2
0.1
0.6
-03
-0.1
0.0
0.1
0.
an
Cysigaa (Mane)) [rad]
06 [c/r
Figure 3: A linear model with odd steering angle features is fit on hardware data using BLR to capture model mismatches based on understeer. Left - Model predictions with epistemic uncertainty used for automatic domain randomization. Right Histograms of front, Ky, and rear slip angle, a,- The front slip angle is scaled such that positive values correspond to understeer at the front axle, which covers around 87% of the recorded data.
Out-of-contact model: We combine the kinematic bicycle model from [39] with feature-based predictors of the front tire slip angle well as data-driven model of the longitudinal dynamics (see Section 3.3). The dynamics of the kinematic model are then given by
vcos + B) vsin 0 + +B)
x (x,
vtan(Wsteer (t))
Manteeet +ay 1
Plang (V. Msteer: Macc):
where x = V,, is the vehicle state, [itsteer Maco] the controls, the center of mass slip angle is B arctan (tan(ustecr ac) ) ayy is the predicted front slip angle, and Dhong represents the longitudinal acceleration model. The data-driven augmentation yields a sufficiently high-fidelity representation of the hardware dynamics, but does not explicitly model reaction forces. We iden- tify collisions based on rectangular hit boxes and simulate contact wrenches with spring model accounting for penetration depth.
In-contact model: When a collision event is detected, the simulation switches to the dynamic bicycle model from [14, 35). The resulting contact wrenches are then explicitly propagated into the state dynamics, switching back to the kinematic model after a pre-specified cool-down or if the rear slip angle falls below a threshold. A schematic of the switched dynamics is provided in Figure 2.
3.3 System Identification
The parameter identification and data-driven model estimation proceed directly the hardware system. The steering commands calibrated based trajectories recorded with motion capture

---

system under the condition of minimal tire slip, allowing for the steering angle to be inferred from the center of mass slip angle. While the resulting kinematic bicycle model provides a good initial approximation of the dynamics, it does not capture higher-order effects relating to e.g., variable drive-train acceleration or tire traction. To better model these effects, we collected dataset of trajectories exciting the full range of throttle and steering inputs, while avoiding the oversteer regime due to the vehicles' tendency to understeer when running at the dynamic limits on our track. See Figure 3. A longitudinal acceleration model is then trained to minimize the one-step prediction error based on velocity, steering, and throttle inputs. To model lateral deviations at higher speeds we estimate the slip angle of the front tire via Bayesian linear regression (BLR), Hereby we use simple hand crafted features that are odd in the steering angle to fit our data. We assume our measured slip stems from a linear model in the features with i.i.d. Gaussian measurement noise ~ (00, of and a spherical Gaussian prior on the model weights E, ~ . (0, o0 During training the dynamics randomized by drawing slip models from the posterior distribution and injecting noise into the received commands. See Appendix A and C.4 for details.
4 Agent Structure and Training Process
We define multi-team racing as a finite-horizon decentralized MDP (decMDP) [41, 42] (brief review Appendix D). We leverage a GPU-accelerated implementation the dynamics described in the previous section to train competitive multi-team racing agents. Our approach combines coarse high- level planning with fine-grained low-level control within hierarchical policy structure trained via bilevel optimization. An overview of the bilevel learning procedure is illustrated in Figure 4.
4.1 Hierarchical Control Structure
Competitive control in interactive settings requires both long-term strategic decision decision-making well as short-term responsive action selection. These objectives occur at different layers of abstraction, with low-level control conditioned on high-level decisions, thus lending themselves to bilevel op- timization procedures [43, 44, 45, 46]. Here, we leverage a hierarchical policy design [47, 48] to effectively guide ego-centric low-level decision-making through team-centric high-level goals. To this end, we train a discrete high-level policy suggest goal states, and a low-level policy to resolve local velocity and heading commands in order to achieve these goals, illustrated in Figure
High-level control: Our formulation enables the high-level policy to propose goal states to the low- level controller. This introduces a layer of abstraction that helps to disentangle long-term strategic planning from short-term control. Goal states are encoded by their relative position along the track as well as their associated uncertainty. The uncertainty encodes goal importance with respect to the high-level strategy and enables the low-level policy to weight short-term gains against long-term considerations. We further leverage coarse high-level representations via categorical distributions to encode mode-switching behavior, e.g., "slow down and keep to the right". This encourages diverse high-level strategies and can provide favorable exploration characteristics [49]. The high- level policy is obtained as an 6-greedy evaluation of a state action value function trained with De- coupled SARSA, an on-policy variation of Decoupled Q-Networks [30]. This leverages value de- composition across team members and enables Centralized Training with Decentralized Execution (CTDE) [28, 50]. We further adopt Munchausen Value Iteration [51] for regularization.
Low-level control: The low-level controller receives high-level goals with associated time budgets, predicting steering and velocity commands. This effectively augments short-horizon local reasoning with long-horizon strategic planning. However, the goal states only act as soft constraints, enabling the low-level controller to retain ficxibility and prioritize local decision-makin when necessary. The low-level policy is trained via PPO [52], with temporal difference (TD) learning of the values and Generalized Advantage Estimation (GAB) of the policy, under the primary objective of clear- ing high-level goals, while considering behavior smoothness, obstacle avoidance, and local track geometry. The strategic team decisions thereby implicitly provided through high-level goals.
5

---

HL high-level
02
12
observation
Qh
QHL
reward
Multi-agent Environment
HHL
policy
value function
Joint TD
TD
GAE
state-action value updating
Figure 4: The hierarchical policy structure abstracts team-centric strategy from ego-centric con- trol. The high-level predicts discrete goal distributions and is trained jointly across team members leveraging value decomposition with DecSARSA. The low-level learns continuous actions to satisfy individual goal states with PPO. Training proceeds in parallel simulation via bilevel optimization.
The steering and velocity commands sampled from the low-level policy serve as references to hardware-level PD tracking controller. The PD controller generates steering and acceleration commands, running at faster rate to mitigate the effects of unmodeled dynamics.
Hardware-level control:
4.2 Training in Simulation
The hierarchical policy is trained via bilevel optimization in parallel simulation. Training consists of two phases, learning basic driving skills during pre-trane pre-training and effective racing during fine-tuning.
Initially, the low-level policy will have to learn rudimentary driving skills that en- able goal-reaching behavior. During this stage, the agent will only generate low quality learning signals for the high-level policy that may induce representational collapse [53]. We therefore pre- train the low-level policy for 500 iterations with PPO on goals sampled from the initially uniform high-level policy, competing against Pure Pursuit opponents to simulate capable ado behavior.
Pre-training:
The bilevel optimization then alternates every 50 iterations. High-level training leverages DecSARSA to refine strategic behavior via goal prediction, while low-level training em- ploys PPO align local controls with the adapted high-level goal distribution. During this stage, sample opponents a mixture from prior policy checkpoints to encourage competitive behavior to emerge through self-play, and opponents steered by pure-pursuit control to prevent strategic col- lapse [54]. We consider both joint and independent high-level Q-functions for fine-tuning, referring to the two approaches as RL Team and RL Independent.
Fine-tuning:
Observations and rewards: The high-level policy observes ego state, rank information, and track boundaries
preview horizon. It further receives noisy ado odometry data within view- limited neighborhood, which is processed by multi-head attention encoder to extract ado relevance with respect to the current ego state into latent representation [55]. The low-level observations additionally contain the high-level goals, as in Figure 1. The high-level policy is primarily rewarded based on team rank information, encouraging strategic behavior that improves positioning of the ego team, and receives zero-sum overtaking reward and small term encouraging ego progress. Hereby, the dominating term is chosen be sparse end-of-episode team-rank reward. The low- level policy is primarily rewarded for accomplishing high-level goals their respective timings, with regularization objectives that modulate vehicle collisions, off-track track driving, and control smoothness. Further details regarding the observation and reward structure are provided in Appendices B and C.
5 Experiments
We evaluate the efficacy our approach by first training policies in simulation and then transferring them directly to the scale race car hardware. Policy learning is conducted on single Nvidia RTX 2080Ti with 2048 parallel environments taking roughly 8 hours. Policy inference is conducted on the CPU for all experiments with the high-level policy rumning 0.5 Hz and the low-level policy running at 20 Hz. The employed learning parameters and training details are summarized in Appendix C.

---

Win tate Hf Iarn for loning Senulatonn
Behavior vs Pure Pursit:
Behavior Independent
pursucy
leader
pursuer
loader
60
1.0
Dyvitakos
Laptine
Dyartakos
Loptime
690
700
500
1000
Opooment: RL Than [1kg
RL Indepandent
Pure Puruutt
Densttnn:
000
800
1000
Serstion: 600
500
1030
Figure 5: Evaluation in simulation shows consistent improvement of winrate and driving skill over the course of training. We compare our RL Team agent to an egoistic variation with independent high-level learners and pure pursuit baseline, achieving winrate of 79% and 69%, respectively. The behavior profiles indicate how increases winrate yield complex trade-offs between lower lap- times, resk-seeking, and risk- behavior (e.g. blocking increases laptime and may risk crashes).
The results from finentunng in simulation are summarized in Figure We evaluate winrates of the RL Team agent over the course of training against itself at convergence, variation that uses independent learning, and the pure pursuit baseline. We further quantify behavior characteristics based on races against the latter two agents. The winrate of RL Team steadily improves to reach around 80% against pure pursuit and 70% against RL Independent. In particular, winrate from behind approaches 60% against RL Independent, while laptime and time spent off-track are reduced. The interplay of other factors more complex, as e.g. emergence blocking will initially induce larger deviation from the racing line in high-risk maneuvers that other agents may exploit, while skillful blocking will reduce the likelihood of crashes and the need re-overtake later on. Generally, find high-level training significantly improve performance, yield better positioning the track, and enable smoother driving.
We further evaluate our approach against two model predictive path integral control (MPPI) [56] agents, running collision avoidance based on an MPPI motion predictions for all 3 ado cars (see [57]), and against two RL Independent agents on hardware for 20 five-lap races each. The 20 races split into 3 starting configurations: Front RL. Team starting in 1 and 2nd (for races), Back RL Team starting in 3rd and 4hh place (for 5 races), and Random randomized starting configurations (for 10 races). The maximum velocity is set to 3.5 [T] for all hardware experiments. The results from the hardware evaluation are summarized in Figure 6. On hardware, we observed single-agent average laptimes across 10 laps of 6.8s for RL Team, 7.6s for RL Independent and 6.7s for MPPI. We found that RL Team was great at keeping its position once in front, for example by using the rear team member fend off opponents, resulting in a 92% winrate against MPPI and a 100% winrate against RL Independent, when starting in 1 place aggregated over Front and Random scenarios. We further observe that RL Independent agents display more reckless driving style reflected in the number of collisions, crashes, and offtrack time, see Figure 6, which forces the RL Team agents to negotiate an increased number of (near- )collision events as reflected by the behavioral differences in the radar charts. We refer the reader to Appendix for the un-normailzed data.
Emergent behavior observed on hardware is showcased in Figure 7. The hierarchical policies learn multi-car overtaking maneuvers, altruistic blocking where the rear team-memher significantly slows down and attempts to block attacking opponents, and pitting maneuvers to jostle opponents out of the way, yielding successful sim-to-real transfer of competitive behaviors. The emergent teaming be- havior is facilitated by the centralized high-level controller that sets distinct goals for and distributes team success to individual members, allowing for altruistic agents that sacrifice own performance by blocking opponents to profit from the resulting increase in win-chance of their team member (Figure 7, middle). We refer the reader Appendix F for further simulated examples.
6 Limitations
We briefly discuss limitations our current approach, which point towards avenues for future work. Ado observation sensitivity: Agents retain sensitivity to their ado observations even when strategic interactions are not likely. This was more pronounced during hardware experiments when coupled
7

---

Win Ratt of RL Tuamn for Evaluaaioon on hurdiware
Dehavior MPP
Dehavior vs RL Independen:
purueer
Insder
pursuer
leader
W0t 120 (5)
GA
1.0
Overtalies
Laps
Overtakes
Lopaime
Offerack
Collsions
offirrcc
RL. Toam
V55
MPPC
independent mn Cree? [Lnneier)
Palest
Team
MPP1
Policy:
IL laam
Incepender;
Figure 6: Transfer to hardware shows strong performance of our RL Team agent compared to the RL Independent and MPPI baselines, achieving winrates above 80% and 70%, respectively. We observe consistent wins when starting in front and a strong ability to win from the back of the starting grid. Stripes indicate crash scenarios in which a pursuing agent did not finish the race, colored by winner.
Overtaking
Altruistic Blocking
Pit Maneuver
Figure 7: Three observed emergent behaviors. Overtaking: Agent O performs double overtake two ado agents. Altrusitic Blocking: Agent stays back to block agents 2 and 3, allowing for agent O pull away. Pit Maneuver: Agent pulls up behind agent 3 and performs a pitting maneuver. See also supplementary material for video sequences.
with system delays and the sim-to-real gap of the dynamics model, sometimes causing exaggerated deviations from the race line. Collisions between teammates and corner cutting: Teammates occasionally jostle for positions, and trade penalties for potential position gains by cutting corners, which could be refined by adapting the problem formulation. Dynamic limits: While this work brings the vehicles to the limits of friction and demands corrections for understeer, the presented pipeline does not allow for agents to act at the torque limits of the scaled cars. Future work could investigate this dynamics regime while exploring methods further reduce hardware system delays.
Conclusion
We propose a novel liight simulator for learning dynamic multi-team wheel-to-whed racing for zero-shot sim-to-real transfer. The simulator combines classical analytic models with data-driven components and is GPU-accelerated to enable massively parallel simulation. We further introduce hierarchical control structure that disentangles high-level strategic planning from low-level continu- control. The high-level policy suggests goals to the low-level controller and is represented with categorical variables to encode mode-switching behavior. The low-level policy predicts continuous controls and incorporates high-level guidance into its local decision making, but retains the ability to deviate if required. The high-level policy is obtained from centralized training across team members and therefore provides strategic information beyond single-agent control.
We train multi-agent racing policies in simulation and deploy onto the TRIKart scale race plat- form without further adaptation. The transferred policies retain their efficacy and are able generate dynamic behavior on hardware when tested with team of MPPI ado agents, including overtaking, blocking, and even pit maneuvers. Future work will be directed at further reducing the sim-to-real gap, while tackling hardware system engineering challenges in supporting more ado agents. Gener- ally, our results indicate great promise for combining parallelized simulation with hierarchical policy abstraction for achieving zero-shot sim-to-real transfer of dynamic multi-team racing behaviors.

---

Acknowledgments
This work was supported in part by Toyota Research Institute (TRT). This article solely reflects the opinions and conclusions of its authors and not TRI, Toyota, or any other entity. We thank them for their support. We further thank Velin Dimitrov for assistance with hardware deployment and Markus Wulfmeier for fruitful discussions on DecSARSA.
References
[1] M. Zhou, J. Luo, J. Villella, Y. Yang. D. Rusu, J. Miao, W. Zhang, M. Alban, I. Fadakar, Z. Chen, et al. Smarts: Scalable multi-agent reinforcement learning training school for au- tonomous driving. arXiv preprint arXiv:2010.09776, 2020.
[2] M. Samvelyan, T. Rashid, C. S. De Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung,
P. H. Torr, J. Foerster, and S. Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 2019.
[3] K. Kurach, A. Raichuk, P. Stanczyk, M. Zajkac, O. Bachem, L. Espeholt, C. Riquelme, D. Vin- cent, M. Michalski, O. Bousquet, et al. Google research football: A novel reinforcement learn- ing environment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4501-4510, 2020.
[4] G. Papoudakis, F. Christianos, L. Schäfer, and S. V. Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint arXiv:2006.07886 2020.
[5] M. O'Kelly, H. Zheng, D. Karthik, and R. Mangharam. Fltenth: An open-source evalua- tion environment for continuous control and reinforcement learning. Proceedings of Machine Learning Research, 123, 2020.
[6] Betz, H. Zheng, A. Liniger, U. Rosolia, P. Karle, M. Behl, V. Krovi, and R. Mangharam. Au- tonomous vehicles the edge: A survey autonomous vehicle racing. IEEE Open Journal of Intelligent Transportation Systems, 3:458 2022.
[7] O. So, P. Drews, T. Balch, V. Dimitrov, G. Rosman, and E. A. Theodorou. Mpogames: Efficient multimodal partially observable dynamic games, 2022.
[8] V. Dimitrov, P. Drews, T. Balch, X. Cui, A. A. Allaban, G. Rosman, and S. G. McGil. Trikart: Enabling human-centered and rapid design of shared control adas. In H-MRS workshop, 2022.
[9] J. Kabzan, M. de Iglesia Valls, V. Reijgwart, H. F. C. Hendrikx, C. Ehmke, M. Prajapat, A. Bühler, N. Gosala, M. Gupta, R. Sivanesan, A. Dhall, E. Chisari, N. Kamnchanachari, Brits, Dangel, Sa, R. Dubé, A. Gawel, M. Pfeiffer, Liniger, Lygeros, and R. Sieg- wart. AMZ driverless: The full autonomous racing system. May 2019.
[10] T. Seyde, W. Schwarting, S. Karaman, and D. Rus. Learning to plan optimistically: Uncertainty-guided deep exploration via latent model ensembles. arXiv preprimt:2100.1460 14641, 2020.
[11] W. Schwarting, T. Seyde, Gilitschenski, L. Liebenwein, R. Sander, S. Karaman, and D. Rus. Deep latent competition: Learning to race using visual control policies latent space. arXiv preprint arXiv:2 02.10/2..99...... 2021.
[12] R. S. Thakkar, A. S. Samyal, D. Fridovich- Keil, Z. Xu, and U. Topeu. Hierarchical control for cooperative teams competitive autonomous racing. arXiv preprint arXiv:2204. 2022.
[13] T.-N. dot com. Noh terminology, ado. html. 2023.
the noh. cha.-/oo..ccm/eeiie///0//////2/2

---

[14] A. Liniger, A. Domahidi, and M. Morari. Optimization-base based autonomous racing of 1:43 scale RC cars. Optimal Control Applications and Methods, 36(5):628-647, sep 2015. ISSN 01432087.
[15] A. Raji, A. Liniger, A. Giove, A. Toschi, N. Musiu, D. Morra, M. Verucchi, D. Caporale, and M. Bertogna. Motion planning and control for multi vehicle autonomous racing high speeds. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (FTSC), pages 2775-2782 IEEE, 2022.
[16] B. Evans, H. A. Engelbrecht, and H. W. Jordaan. Learning the subsystem of local planning for autonomous racing. In 2021 20th International Conference Advanced Robotics (ICAR), pages 601-606. IEEE, 2021.
[17] T. Stahl, A. Wischnewski, J. Betz, and M. Lienkamp. Multilayer graph-based trajectory plan- ning for race vehicles in dynamic scenarios. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 3149-3154. IEEE, 2019.
[18] R. Reiter, J. Hoffmmann, J. Boedecker, and M. Dichl. A hierarchical approach for strategic motion planning in autonomous racing. arXiv preprint arXiv:2212.01607, 2022.
[19] S. He, J. Zeng, and K. Sreenath. Autonomous racing with multiple vehicles using a parallelized optimization with safety guarantee using control barrier functions. In 2022 International Con- ference on Robotics and Automation (ICRA), pages 3444-3451. IEEF, 2022.
[20] A. Liniger and J. Lygeros. noncooperative game approach to autonomous racing. IEEE Transactions on Control Systems Technology, 2883)888--977, 2019.
[21] W. Schwarting, A. Pierson, S. Karaman, and D. Rus. Stochastic dynamic games in belief space. IEEE Transactions on Robotics, 37
[22] Z. Wang, T. Taubner, and M. Schwager. Multi-agent sensitivity enhanced iterative best sponse: A real-time game theoretic planner for drone racing in 3d environments. Robotics and Autonomous Systems, 125: 15:103411 003410, 2020.
[23] R. Wurman, S. Barrett, K. Kawamoto, MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlic, F. Eckert, Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896):223- 228, 2022.
[24] L. Busoniu, R. Babuska, and B. De Schutter. A comprehensive survey of multiagent reinforce- ment learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38-2):156-172, 2008
[25] K. Zhang, 7. Yang, and T. Başar. Multi-Agent reinforcement learning: A selective overview of theories and algorithms. Nov. 2019.
[26] S. Gronauer and K. Diepold. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, pages 1-49, 2022.
[27] T. Haarojja, B. Moran, G. Lever, S. H. Huang, D. Tirumala, M. Wulfmeier, J. Humplik, S. Tun- yasuvunakool, N. Y. Siegel, R. Hafner, et al. Learning agile soccer skills for bipedal robot with deep reinforcement learning. arXiv preprint orXiv:2304. 13653, 2023.
[28] P. Sunchag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi. M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, et al. Value-decomposition networks for cooperative multi- agent learning, arXiv preprint arXiv:1706. 1706.0529, 2017.
[29] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson. Mono- tonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234-22) 7284, 2020.
10

---

[30] T. Seyde, P. Werner, W. Schwarting, I. Gilitschenski, M. Riedmiller, D. Rus, and M. Wulfmeier. Solving continuous control via q-learning. In The Eleventh International Conference on Learn- ing Representations, 2022.
[31] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of PPO in cooperative Multi-Agent games, Oct. 2022.
[32] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[33] E. Chisari, A. Liniger, A. Rupenyan, L. Van Gool, and J. Lygeros. Learning from simulation, racing in reality. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 8046-8052. IEEE, 2021.
[34] H. B. Pacejka and E. Bakker. The magic formula tyre model. Vehicle system dynamics, 21 ($1):1-18, 1992.
[35] D. Zahrádka. Optimization- control of the f1/10 autonomous racing car. Czech Techni- cal University in Prague, Master's Thesis, 2020. URL https: ciirc. cvut. cz/ allapajarsstudente Settudenti/F--DD. 200-202000 Zahradka Saatradka-Davii---
pdf.
[36] G. Williams, P. Drews, B. Goldfain, J. M. Rehg. and E. A. Theodorou. Aggressive driving with model predictive path integral control. In 2016 IEEE International Conference Robotics and Automation (ICRA), pages 1433-1440. IEEE, 2016.
[37] W. Zhao, J. P. Queralta, and T. Westerlund. Sim-to-Ree Real transfer in deep reinforcement learning for robotics: a survey. Sept. 2020.
[38] X. Hu, S. Li, Huang, B. Tang, and L. Chen. Sim2real and digital twins in autonomous driving: A survey. May 2023.
[39] Kong, Pfeiffer, Schildbach, and F. Borrelli. Kinematic and dynamic vehicle models for autonomous driving control design. In 2015 IEEE intelligent vehicles symposium (IV), pages 1094-1099 IEEE, 2015.
[40] A. Paszke, Gross, Massa, Lerer, J. Bradbury, G. Chanan, Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. De- Vito. M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performanee deep learning li- brary. In Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019. URL. http://papers... www.ips.cc/papppaaaa 9015-pytorch- an- htw.ii/eeeett.ettiiiaaerrrroeeoe -ewrnneee----n----eeee-- library..... pdf.
[41] F. A. Oliehoek and C. Amato. Concise Introduction to Decentralized POMDPs. Springer International Publishing.
[42] A. Beynier, F. Charpillet, D. Szer, and A.-I. Mouaddib. DEC-MDP/OMMD. In Markov De- cision Processes in Artificial Intelligence, pages 277-318. John Wiley & Sons, Inc., Hoboken, NJ USA, Mar. 2013.
[43] H. Stackelberg et al. Theory of the market economy. 1952.
[44] J. Bracken and J. T. McGill. Mathematical programs with optimization problems in the straints. Operations research, 21(1):37-44, 1973.
[45] A. Sinha, P. Malo, and K. Deb. A review bilevel optimization: From classical to evolu- tionary approaches and applications. IEEE Transactions Evolutionary Computation, 22(2): 276-290 295, 2017.

---

[46] Т. Seyde, J. Carius, R. Grandia, F. Farshidian, and M. Hutter. Locomotion planning through a hybrid bayesian trajectory optimization. In 2019 International Conference on Robotics and Automation (ICRA), pages 5544-5550. IEEE, 2019.
[47] M. Wulfmeier, A. Abdolmaleki, R. Hafner, J. T. Springenberg, M. Neunert, T. Hertweck, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller. Compositional transfer in hierarchical reinforcement learning. arXiv preprint arXiv: /228, 2019.
[48] T. Seyde, W. Schwarting, I. Gilitschenski, M. Wulfmeier, and D. Rus. Strength through diver- sity: Robust behavior learning via mixture policies. In Conference Robot Learning, pages 1144-1155 PMLR, 2022.
[49] Seyde, I. Gilitschenski, W. Schwarting, B. Stellato, M. Riedmiller, M. Wulfmeeer, and D. Rus. bang-bang control all you need? solving continuous control with bernoulli poli- cies. Advances in Neural Information Processing Systems, 01227209-2722) 2021.
[50] R. Lowe, I. Wu, A. Tamar, Harb, Pieter Abbeel, and I. Mordatch. Multi-agent actor- critic for mixed cooperative-compente cessing systems, 30, 2017.
environments. Advances in neural information pro-
[51] N. Vieillard, O. Pietquin, and M. Geist. Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 33:4235-4246, 2020.
[52] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.0000
2017.
[53] E. Nikishin, M. Schwarzer, P. D'Oro, P.-L. Bacon, and A. Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, pages 16828- 16847. PMLR, 2022.
[54] Bansal, J. Pachocki, Sidor, Sutskever, and Mordatch, Emergent complexity via multi- agent competition. arXiv preprint arXiv:1710.037453 2017.
[55] S. Iqbal and F. Sha. Actor-attention-critiic for multi-agent reinforcement learning. Interna- tional conference on machine learning, pages 2961-2970. PMLR, 2019.
[56] G. Williams, A. Aldrich, and E. Theodorou. Model predictive path integral control using 149, 2015.
covariance variable importance sampling. arXiv preprint
[57] G. Williams, N. Wagener, B. Goldfain, Drews, M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic mpe for model-based reinforcement learning. In 2017 IEEE Interna- tional Conference Robotics and Automation (ICRA), pages 1714-1721, IEEE, 2017.
[58] C. M. Bishop and N. M. Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[59] A. Tavakoli, M. Fatemi, and P. Kormushev. Learning to represent action values as hypergraph on the action vertices. In International Conference on Learning Representations, 2021.
[60] H. Van Seijen, M. Fatemi, J. Romoff, R. Laroche, T. Barnes, and J. Tsang. Hybrid reward architecture for reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
[61] T. Haarojj,, H. Tang. P. Abbeel, and S. Levine. Reinforcement learning with deep energy- based policies. In International conference machine learning, pages 1352-1361. PMLR, 2017.
12

---

A System Identification Details
In order to identify the slip model we use Bayesian Linear regression. Analogously to [58), Chapter
3, we compute the posterior and inference distributions at & as
p( X,Y = N Noesswwwwwe (Wslipill
(2)
1
x X
+
X
(3)
1)
x x
P (f: &, &' +
(5)
epistemic
aleatoric
where the resulting uncertainty splits into an epistemic and an aleatoric component. In order to fit the model we use the following features and targets,
X =
Msser
Wisteer
"steer
3
Y =
(6)
along with hand-tuned noise parameters in Table 1.
[rad]
G [rad]
1.0
0.01
Table 1: Employed BLR noise parameters.
B
Observation Details
Otargettt
Symbol
Meaning
Otarget.st
targett
Ouack.
Evenly spaced samples on left track boundary
Otrack,R
Orrack11
Evenly spaced samples on right track boundary Longitudinal coordinate of target position
Oargett,
Otarget,sl
Otrack,
Transversal coordinate of target position
Otrack.sl
Longitudinal shape of target Transversal shape of target Longitudinal coord. of relative position to ado Transversal coord. relative position to ado
OuackLL
Otruck.st
Oado.t
Oado, I
0
Oudo.1
Oado,
Figure 8: Track and position-basee ado observation components employed to train the hierarchical policies.
The high-level observations consist of the ego observations and ado observations
Oego
OHL.
(7)
Oado
13

---

where is the concatenation of all individual, view-limited, ado observations Oado, The individ- ual components read
Oudo,1
Otrack.R
Oado,t sin( Orel) cos Ozet)
Otrack,L
Oego
Repisode
Ondo,
Owt
Viel
Oprggrsss
drel
2
2
where Tepisode the fraction of the high-level episode remaining. Owot is the fraction wheels off track, Oprogress is the amount progress along the center line of the track over the last time step, 2_ and 2 are the ego and team rank respectively.
The low-level observations are the concatenation of the high-level observations with the target po- sition provided by the high-level policy, where distances are expressed in track coordinates, such that
OHL
Otarget,
OLL
Otarget, Ourget, sl
(9)
Ourrget, st
C Learning Parameters and Domain Randomization
elaborate on aspects of the training setup below, discussing the reward functions and domain randomization used. Throughout, we consider high-level steps that consist of Til 20 low-level steps. At each high-level step, goal state is sampled and then held fixed for the duration of the low-level substeps.
C.1 PPO hyperparameters
In order to train the low-level controller we leverage Proximal Policy Optimization (PPO) with Generalized Advantage Estimation (GAE) similar to the parallelized agent employed [32]. The associated hyperparameters are provided in Table 2. The low-level controller generates single agent driving behavior based on local observations, while receiving guidance from the high-level policy through goal-conditioning.
Parameter
Value 10
Learning rate
Minibatch size
512
Minibatches
4
Learning epochs
2
Clip value
0.2
Discount (y)
0.99
GAE parameter (2)
0.95
Entropy coefficienct
0.01
Desired KI.
0.01
Max grad norm
10.0
MVI
0.9
MVI
0.03
Table 2: Algorithm hyperparameters
14

---

C.2 Decoupled Expected SARSA
In order to train the high-level controller we build on recent results at the intersection of parallel optimization and multi-agent control and employ Decoupled Expected SARSA, an on on-policy vari- ation of Decoupled Q-learning [30] within the family of Hypergraph Q-Network algorithms [59]. This approach leverages value decomposition [28] across action dimensions and team members to represent the state-action value function as
(g(s..a)) M
Qo(s, a.) = Σ
(10)
where coordination is facilitated by individual utility functions being conditioned on the global robot state and high-degree of parameter sharing within unified critic [60]. The linear combination of univariate utility functions allows for efficient decomposition of the argmax operator and global op- timization over a, simplifies into parallel local optimizations over a/. Training of the action value function remains centralized, while online action selection is decoupled. Inserting this decompo- sition into the Bellman equation yields decoupled target representation with expectation over the underlying policy (e.g. E-greedy)
Co (6lsmm14(44) y1 = r(s + E n a())))
M
(11)
M
We can then insert the factorized value function of Eq. 10 and the decoupled target Eq. 11 into the loss function (()) Σ Lg (y: 1y-2an Qs ()). We furthermore explore the effects of soft Q- learning [61] within the context of Munchausen Value Iteration [51] to yield the adapted target
Fr.VI r(s,, orrnn + + n a(s Co(s:t1.
+ cIn To(a s5+1)
(12)
M
where na sm 20) is now taken as the softmax policy with temperature and scaling factor a. We observed improved training using Munchausen Value Iteration and trained our final policies based on this formulation. The associated hyperparameters are provided in Table 2.
C.3 Reward functions
We provide details on the reward function used to train the high-level (HL.) and low-level (LL.) policies. For ease notation, we introduce the following variables: 2 and 2 represent team and ego rank, respectively, p and p denote the vehicle position and velocity in track frame, respectively, 4 and Σx correspond the mean and standard deviation of the multivariate goal state distributions, respectively, w; is indicator for whether tire i is off off-track, represents the actions, and is collision indicator. The wheel-on-tracl term features a slight asymmetry that up-weights penalty terms when either the front or rear axle are off-track, while the collision term up-weights rear- ending collisions we omitted these features from the reward equations for visual clarity. We further introduce index denoting time and ELL as the fraction of time left until the next high-level step.
A,_ 27
HL = c1 = 0) + C2 277 +
+c4 27" + +cse" 2
(13)
min
(4_1
24)
3
1. = 2n 2 e b(p (1)
( Hf)
+ +07
Σ
w/ +cdA tcg 11 +c1ola - 2- +1i
(14)
To provide intuition for the effect of individual reward terms, we discuss the high-level reward function in more detail. The high-level episode duration set to 16 seconds and low-level episodes terminate after two high-level targets have been received. The first place reward (c1) provides sparse signal about team performance to each team member each timestep and serves as the primary indicator for winning a race. The starting rank reward (cz) acts as small regularizer that
15

---

Symbol
Value
Term explanation
C1
4.0e-1
Team is in first place
C2
5.0e-2
Team improved over starting rank
HL
1.0e-1
Zero-sum overtaking reward
C4
1.0e-3
Regularize with team rank
1.0e-2
Regularize with ego rank
C6
li(tu < 0.1) (1.0c+0)
Clear goal at end of low-level episode
C7
-1.0e-3
Penalize wheels off track
LL
08
[(Ap;
> 2.5) * (-2.5e-1)
Penalize large progress jumps (cutting)
C9
-2e-3
Encourage high velocity
C10
-6e4
Regularize action smoothness
C11
-5e-3
Penalize collisions
Table 3: Reward function components
up/down-weighs trajectories based their team rank change, as winning from behind requires more skill than winning from the front. The overtaking reward (cz) explicitly highlights important sparse interactions that resulted in team rank change to pinpoint the influence of particular short-term maneuvers on long-term race strategy. The team rank reward (ca) complements the first place reward (c1) by encouraging lower team rank even if the team is not in first place, providing dense guidance during the learning process. The ego rank reward (cs) further encourages agents to improve their individual rank even not impacting overall team rank. The latter term primarily act as regularizers that counteracts potential lazy agents e.g- encourage team member in last place to remain competitive even when their team is in first place and they do not foresee short-term individual impact. Note that given in Table 4, the sparse high-level reward outweighs the highest possible accumulated dense reward components, which are weighted by and C5.
C.4 Domain Randomization
To improve the sim-to-real transfer of the learned policies we randomize the dynamics simulation during training time. The simulated slip is randomized by sampling new weight vector from the posterior distribution in Equation 2 at the beginning of every training episode. We further randomize select model parameters with additive noise ö summarized in Table 4.
Symbol
Distribution
Explanation
Vmax 8 Sj,
2 (3.5,4.5) (-0.02,,0002 m
Additive noise velocity cap Additive noise to front length
(-0.02, 0.02) m
Additive noise to rear bicycle length Additive noise to all steering commands Additive noise to simulated throttle commands Table 4: Ramdomized model parameters
8 Suster
2 (70.033.0.03 rad] 2/ (-0011,.0055)
S
In order to improve robustness of the policies add uniform random noise, drawn independently from / (-0.005,0.005 to all observations except the rank and team rank observations.
D Decentralized Markov Decision Process Formulation
We formulate multi-team racing as a finite horizon decentralized Markov Decision Process (DecMDP), that is characterised by the tuple (Y, where y is the discounting fac- tor, is the set of joint states, T is the probability densitiy fucntion p(s'|s,a) of transitioning from s to s', when agents take the joint action a, A is the set of action sets available to all agents, Ω the set of all observation spaces, is the set of all observation functions, and R is the of reward functions across agents, where each reward function returns the reward for taking joint action state
16

---

In practice, we define multi-team racing in a symmetric fashion across agents such that their action spaces are identical. The observation functions are defined as outlined in Section 4, such that each agent only observes their velocity, nearby track boundaries, number of wheels that are off track, rank, remaining time the episode, last actions, and respective ado agents.
E Hardware Experiments
The radar plots in Figure are intended to give the reader an intuitive understanding the behavior of the various agents, For clarity the axis are normalized with the maximum value of the respective statistic aggregated during both the races against of RL Team (RL T.) against MPPI agents and RL Independent (RL Ind.). In this appendix the raw data is listed in Table 5, along with brief explanations. All statistics are averaged over the 20 respective races. As a guideline, the average race duration is around 40s.
Statistic
RL T. MPPI
RL vs RL Ind.
Explanation
Laptime [s]
(8,44, 8.62)
(8.36, 8.26)
Avg. Laptime of fastest team member Win w/ team member starting in 1st
Wins as leader
(0,92, 0.38)
(1.00, 0.2)
Wins as pursuer
(0.63, 0.08)
(0.80, 0.00)
Win rate w/ opponent starting in 1st Number of overtakes per team
Overtakes
(2.85, 0.95)
(5.00, 4.70)
Off-track [s]
Time off track per team number of collisions per team member Table 5: Un-normalized data from the radar charts in Figure 6.
(1.57, 1.54)
(3,89, 3.104)
Collisions
(1.0, 0.25)
(1.6, 1.8)
17

---

Emergent Behavior in Simulation
We observe similar emergent behavior in simulation hardware. This is shown in Figure 9. The simulator itself features viewer with visualizations of the command distributions of the high and low-level policies, markers for the observations, and miscellaneous additional information, see Figure 10.
Overtaking
Pit Maneuver
S
Altruistic Blocking
Figure 9: Three observed emergent behaviors in simulation. Overtaking: The marked agent per- forms an overtake. Pit Maneuver: The marked agent pulls up behind the red agent and performs pitting maneuver. Altrusitic Blocking: The marked agent overtakes opponent and stays back to block both opponents allowing the blue teammate to pull away.
Sppex221
1. episode statistics
+0.00
3.
1.
+3.00
2. low-level action distribution
Va
G
3. high-level target
4.
2.100
Ty
4. track observations
5.
5. normalized Q values
6. wheel off track indicator
2.
6.
Figure 10: The viewer of the simulator displays individual environments along with episode infor- mation and information about the distributions of both the high and the low-level policy.
18

---

G Tracks used for Training
In Figure 11 the tracks employed for training and evaluation are shown. During training and evalu- ation, races take place both the clockwise and counterclockwise direction. Track b) a re-scaled version of the track used in [14].
(a)
(b)
(c)
(d)
Figure 11: Tracks used for training hierarchical racing policies. evaluation.
Track a) is used for the hardware
19


Reviews:
[{"review_id": "t0BKavVfI1", "reviewer": "Reviewer_NBPG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a new robotics challenge in the form of a team-based scale care race, where two teams of two cars each race against each other. They implement a zero-shot sim2real strategy, where they implement a GPU-accelerated simulation combining kinematic, dynamic and data-driven models to minimise the transfer gap. Racing policies are trained using self-play and in a hierarchical fashion, where a (team-shared) high-level planner sets position subgoals for a low-level throttle and steering controller to reach. The resulting controllers are compared to a \"teamless\" RL and MPPI baselines.", "summary_of_recommendation": "I vote to accept this paper as it presents both an interesting robotics challenge as well as a well-designed and executed system towards solving it, resulting in interesting emergent behaviour. However as none of the individual parts of the proposed approach are significantly novel, and the merit of the work lies largely in the integration, I struggle to motivate a strong accept.\n\nEdit after rebuttal: following the authors' rebuttal I have increased my recommendation. It is clear from the discussion that the authors have thoroughly investigated the problem, and the additional details in the paper and appendix are very helpful and appreciated. I would even urge the authors to add further details based on the discussions in appendix, e.g. regarding the double model based on contacts.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Please define \"ado\" on l61.\n- Please elaborate on the difference with [12] based on the results (RL Team vs. RL Independent).\n- Section 3.2: please explain why there is a need for a separate in- and out-of-contact models.\n- Section 4:\n  - Please make more explicit, symbolically, what the exact inputs and outputs of the various components are, and refer to the Appendix for the reward definitions.\n  - Please provide more details, possibly in Appendix, about the training regime (batch size, learning rate, etc.) and specifically about opponent sampling.\n- Figure 4:\n  - Please define \"TD\".\n  - If my understanding is correct, $V_{HL}$ and $\\pi_{HL}$ should be $V_{LL}$ and $\\pi_{LL}$\n- Figure 5: on the radar plots, it is not always immediately clear for which metric which way (higher or lower) is better. Ideally, higher would be better across all metrics.\n- Figure 6: the first pair of bars seem to indicate that the win rate of RL Team against MPPI is 100%, which contradicts the 91% mentioned on l230. Please consolidate.\n- Figure 7: the top row is very dense and difficult to interpret. If the goal is to show the visualisation of the simulation rather than the behaviour, it might be better to use a single, bigger frame, and add annotations.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an interesting challenge in the form of team-based scale car racing. This is relevant for the autonomous driving and wider robotics communities as it combines dynamics and control at various timescales, from team-level strategy to split-second corrections. This pushes the boundaries of what seems easily achievable with existing approaches, e.g. zero-shot sim2real.\nYet the authors also describe a methodology that yields impressive results with interesting emerging behaviour, by using custom accelerated and partially data-driven simulation, self-play, hierarchical control and bilevel optimisation, not to mention actually successfully deploying on hardware. While any single part of the approach in itself is perhaps not incredibly novel, this work gets its value from the well-executed integration and evaluation, and the interesting emerging results.\nThe main weakness of this submission would have to be the presentation and clarity, which could be improved. Specifically, some figures could use additional annotation and in general the text could use more symbolic notation to describe the ins and outs of various components. E.g. it is not clear how much information agents get from teammates compared to adversaries. Is it just the high-level that sees both? Ideally more details about the training regime would be added to the appendix. More specifics in the question section. The related work could also be improved with the inclusion of recent multi-agent domains in robotics, such as https://arxiv.org/abs/2304.13653.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "lRW55ZYuXXN", "reviewer": "Reviewer_x7zC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper is focusing on autonomous driving with high-speed autonomous ground vehicles. The researchers use an in the community well-known 1:10 scale autonomous vehicle called F1TENTH to demonstrate their research. The goal of the research is to demonstrate wheel-to-wheel racing which means overtaking maneuvers at the limits of handling + close proximity overtaking maneuvers. To achieve that kind of behavior, the authors suggest a learning-based approach which is using self-play reinforcement learning  in simulation to learn compete tive agents with a variety of emergent behaviors. After successful training, the authors demonstrate the results ins simulation and in real-world experiments.", "summary_of_recommendation": "In summary, the presented paper shows an interesting approach for autonomous racing and multi-vehicle overtaking and wheel-to wheel maneuvers. The paper has some clear contributions and shows, that their method can be transferred from simulation to real-world autonomous small-scale vehicles. While the manuscript does possess certain strengths and valuable contributions, some areas require further improvements and clarifications. The paper can be improved in several ways e.g. shorten some paragraphs and extend the method section and additional discuss the results more in detail. In summary I think this is a solid conference paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Why is a team approach something that is worth researching on or better said: Where does the foundation and justification of multi teams is coming from? All-well-known racing series have multiple teams, but they do not cooperate normally (e.g. formula 1). I like the idea but I miss the real-world application ultimately. Furthermore, I would like the authors to explain: What happens if we only train one car without the partner.\n- The collision avoidance with the contact wrenches is currently hard to understand and difficult to see, where this is coming from. There are some well-know collision checking models with different fidelity, why have you not chosen those collision checkers e.g. circles or volumes (Sphere, OBB)?\n- I suggest to revise Figure 7 completely. I can not see anything in any of those images, both the simulation and real world examples are hard to deciver. It is furthermore unclear how the simulation and the three different types (Overtaking, blocking, Pit) fit together and why we need to see a pit maneuver here. Focus only on the most important parts and elements", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality of the Research Paper\n\nOriginality:\n- The originality of the paper lies in the difficulty of the high-speed/ high-acceleration and proximity overtaking maneuvers. Racing itself is a difficult task and many authors try to created modular software pipelines in order to create autonomous racing algorithm. With the self-play reinforcement learning approach it was shown that multi-agent competitive driving and overtaking can be demonstrated.\n- By focusing on teams of cars, the chance of multi-team learning is given. This is a aspect that could help to transform DRL further to real-world autonomous cars e.g. in overtaking or merging scenarios\n\nSignificance of the work:\n- Although autonomous racing is a niche field,  the study addresses an important and relevant research problem and therefore making a valuable contribution to the existing autonomous racing community. Especially the task of multi-vehicle overtaking was rather underrepresented so far.\n\nStrength of the work:\n- the paper addresses an important and timely research question and contributes to the existing body of knowledge in the field of field.\n- Well-organized, structured and cited state of the art\n- Clear and Coherent Structure: The paper exhibits a well-organized structure, making it easy for readers to follow the flow of ideas and understand the research process.\n- Comprehensive Approach: The paper effectively combines simulation and real-world experimentation, providing a holistic and well-rounded analysis of the research problem.\n- Comparison with SoTA algorithms: The authors show two additional algorithms (Pure Pursuit and MPPI) as a comparison and demonstrate that their RL approach can outperform those in different sections\n\nWeaknesses: \n- Currently, the switching of the dynamic models in collisions is unclear \"Out of collision, we employ the kinematic bicycle model [37] augmented with data-driven models.113 During collisions, we employ the dynamic bicycle model [13].\" -> I do understand that we need correct tyre forces calculations in critical overtaking maneuver, but since the dynamic bycicle model also only has cornering stiffness as a tyre force calculation is should not be to complex. Additionall the authors are writing \"If a collision event is detected, the dynamics are switched temporarily to a lower fidelity dynamic model that directly models the effects of external wrenches.\" --> The dynamic bycicle model is of higher fidelity isn't it? So this sentence needs to be switched?\n- The methodological section is quite short and does not capture the concept of the RL with PPO quite good. I can recommend to shortend the Sys-Id Part and Vehicle dynamics model part in order to focus more on the exact method that allows for dynamic overtaking \n- Currently, the discussion section of the results is quite poor and does not show in detail", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "G3CRhqHTwPV", "reviewer": "Reviewer_sWPm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors demonstrate emergent team behaviors in the multi-agent autonomous racing task on real hardware. The contributions can be summarized as:\n\n- A novel, GPU accelerated simulator with various classical and learned dynamics models, and domain randomization to facilitate sim2real transfer\n\n- A hierarchical policy formulation trained with multi-agent reinforcement learning, where the high-level policy attends to long-horizon decision making and goal setting tasks, and the low-level policy attends to low level control \n\n- Demonstrated emergent team behaviors on real hardware ", "summary_of_recommendation": "The paper is very well written with some surprising results, such as the emergence of certain team behaviors like altruistic blocking. The results are convincing and the demonstrations on hardware are enjoyable to watch. However, I am concerned about the lack of attention and analysis given to the reward function and the impact of the design decision there on the results. I am also concerned about the high collision rate.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "The velocity limit of 3.5 m/s seems to be limiting, to the point that I'm not sure if the intricate modeling of the front and rear slip angle is relevant. Have the authors already explored the impact of modeling these parameters? An ablation on this would be welcome. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strengths\n\n- The agents learn impressive and, as far as I’m aware, novel emergent behavior. Altruistic blocking is particularly impressive given that it is not explicitly rewarded in the reward function and demonstrates an awareness of the importance of team rank, even at the expense of ego-rank and potentially other penalties such as collisions, etc. This suggests that the hierarchical policy formulation and/or reward formulation as designed by the authors is an important contribution and factor of the reported results. \n\n- Domain randomization techniques for facilitating zero-shot transfer are thorough, methodical, and seem to work quite well based off of the reported success rates. The zero-shot transfer and real robot results in impressive and interesting to watch.\n\n# Weaknesses\n\n- The quite complex reward function, despite being arguably the single most important factor of the success rate and emergent behaviors, is only briefly and qualitatively discussed in the main paper and expanded in equation form with little explanation or analysis in the appendix. There are quite a few components in the reward function and my guess is that these were carefully designed and tuned in order to get the sort of emergent interactions we see in the experiments section. The reward function and the design decisions behind it should be analyzed in greater depth. For example, in $r^{HL}$, how important is the $c_1$ term, especially since it is a sparse signal only available at the last timestep? Why is the team rank weighted lower than the ego rank, despite the team rank deciding the outcome of the race? etc.\n\n\n- **Collisions**. While I understand that certain beneficial team behaviors such as altruistic blocking and cutting-off result in (near) collisions, the reported collision rates are unusually high and seems to be a rather weak spot of the paper. The aforementioned behaviors should still be learnable without necessitating a collision, for example, by inducing a negative reward potential field within a small, fixed radius around each car. The authors could have spent more time investigating and tuning the reward function to address the collision rate as this is important in real F1 and even F1/10 races.  \n\n- Figure 3 is difficult to interpret. For one, the y-axis on right-side diagrams are unlabelled. Is this supposed to be a histogram of the frequency rates of various slip angles? Furthermore, it is unclear what information the authors are trying to convey with the figure on the left. Is this supposed to show that the authors have minimized epistemic uncertainty with their model? Why does it seem (at least from this angle) that front steering slip $a_f$ goes down with velocity and left steering? Further analysis and/or a more interpretable diagram would be welcome here. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "5xxdLRGg1m", "reviewer": "Reviewer_Edzc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper takes a multi-agent RL approach to multi-team car racing. While existing approaches to this problem rely on game theory (which results in slow computation) or \"vanilla\" multi-agent RL (MARL) (which struggles to learn meaningful team-level behavior), the proposed approach takes a hierarchical strategy, wherein a high-level RL agent takes actions to set team-level goals to be pursued by the low-level agents. The authors develop a lightweight simulator for training that models a number of important dynamic effects (e.g., tire friction and slipping) to enhance sim-to-real transfer. In both simulation and hardware experiments (on 1/10th scale cars), the authors find their method outperforms an MPPI baseline, as well as an ablation where the high-level value function only takes in single-agent state information (a typical CTDE approach).", "summary_of_recommendation": "I'd characterize this paper as a weak accept. There are some nice MARL insights here, but this is a crowded area, and, despite a quick lit review, I'm unsure if the work is particularly novel (and would defer to the expertise of other reviewers with more MARL experience). The paper is well-done but I could see it being rejected if the novelty claims are not sufficiently strong. \n\nEdit after rebuttal: After reading the authors' responses, I feel more confident that the approach is novel and would be of interest to the broader robotics community. My score remains a \"weak accept\" (since this work still skews a bit incremental) but think the paper is of pretty good quality in general.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Were the track geometries varied at all during training? Or did agents race repeatedly on the same track? Was the track used in hardware seen during training?\n\nIs it correct to say that the RL-Individual baseline uses the same hierarchical policy archictecture, but the high-level critic doesn't see team-level information when evaluating the Q-function?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I found this paper to be interesting, novel, and clearly written. The problem the authors study is of considerable interest to the robotics community since mixed cooperative-competitive games are quite difficult to solve in general, and have important implications for broader topics such as HRI, autonomous driving, etc.\n\nThe hierarchical approach to MARL taken here, while straightforward, does seem to be novel, and I think provides a nice insight for other practitioners seeking to take RL approaches to similar games. Further, the modeling considerations studied by the authors when designing their simulator seem insightful for other researchers considering competitive or high-performance driving problems. \n\nPossible limitations include the limited track geometry shown in the paper, as there are only two track layouts shown in Fig. 7, one simulation, one hardware (will ask a clarifying question about this in the rebuttal and edit as needed). Thus it's unclear how much the resulting policies learn good driving behavior for a specific track, or if the policies generalize to arbitrary track layouts. I will note the observation space is not track-dependent. Similarly, while the agents see a few different driving policies during training (either a heuristic policy or policies from previous checkpoints) it's unclear if this is sufficient to truly \"solve\" the game in a game theoretic sense (e.g., converge to a Nash equilibrium with new players).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the original submission, the rebuttal by the authors and the discussion that followed, the reviewers agree that this paper should be accepted to CoRL. 
To the authors: great job on writing a strong rebuttal and engaging with the reviewers during the discussion period! Please address the remaining comments in the camera-ready version of the paper.
Decision: Accept (Poster)