IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learning
Authors: Vindula Jayawardana
Baptiste Freydt
Ao Qu
Cameron Hickert
Zhongxia Yan
Cathy Wu
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
generalization
benchmarking
eco-driving
Primary area: datasets and benchmarks
TL;DR: A multi-factorial cooperative eco-driving environment based on data-driven simulations for benchmarking multi-agent contextual reinforcement learning algorithms.

Abstract: Despite the popularity of multi-agent reinforcement learning (RL) in simulated and two-player applications, its success in messy real-world applications has been limited. A key challenge lies in its generalizability across problem variations, a common necessity for many real-world problems. Contextual reinforcement learning (CRL) formalizes learning policies that generalize across problem variations. However, the lack of standardized benchmarks for multi-agent CRL has hindered progress in the field. Such benchmarks are desired to be based on real-world applications to naturally capture the many open challenges of real-world problems that affect generalization. To bridge this gap, we propose IntersectionZoo, a comprehensive benchmark suite for multi-agent CRL through the real-world application of cooperative eco-driving in urban road networks. The task of cooperative eco-driving is to control a fleet of vehicles to reduce fleet-level vehicular emissions. By grounding IntersectionZoo in a real-world application, we naturally capture real-world problem characteristics, such as partial observability and multiple competing objectives. IntersectionZoo is built on data-informed simulations of 16,334 signalized intersections derived from 10 major US cities, modeled in an open-source industry-grade microscopic traffic simulator. By modeling factors affecting vehicular exhaust emissions (e.g., temperature, road conditions, travel demand), IntersectionZoo provides one million data-driven traffic scenarios. Using these traffic scenarios, we benchmark popular multi-agent RL and human-like driving algorithms and demonstrate that the popular multi-agent RL algorithms struggle to generalize in CRL settings.

Published as a conference paper at ICLR 2025
INTERSECTIONZO0: ECO-DRIVING FOR BENCHMARK- ING MULTI-AGENT CONTEXTUAL REINTORCEMENT LEARNING
Vindula Jayawardana'; Baptiste Freydt"; Ao Qu', Cameron Hickert), Zhongxia Yan', Cathy Wu!
MIT (vindula, qua, chickert, zxyan, cathywu) @mit edu EETH Zurich, @student. .ttzt.ccccc..
ABSTRACT
Despite the popularity of multi-agent reinforcement learning (RL) in simulated and two-player applications, its success in messy real-world applications has been limited. key challenge lies in its generalizability across problem variations, a common necessity for many real-world problems. Contextual reinforcement learn- ing (CRL) formalizes learning policies that generalize across problem variations. However, the lack of standardized benchmarks for multi-agent CRL has hindered progress the field. Such benchmarks are desired to be based on real-world applications to naturally capture the many open challenges of real-world prob- lems that affect generalization. To bridge this gap, we propose IntersectionZoo, comprehensive benchmark suite for multi-agent CRL through the real-world application of cooperative eco-driving in urban road networks. The task of coop- erative eco-driving to control fleet of vehicles reduce fleet-level vehicular emissions. By grounding IntersectionZoo in a real-world application, we naturally capture real-world problem characteristics, such as partial observability and multi- ple competing objectives. IntersectionZoo is built on data-informed simulations of 16.334 signalized intersections derived from 10 major US cities, modeled in open-source industry- grade microscopic traffic simulator. By modeling factors affecting vehicular exhaust emissions (e.g., temperature, road conditions, travel demand), IntersectionZoo provides one million data-driven traffic scenarios. Using these traffic scenarios, we benchmark popular multi-agent RL and human-like driving algorithms and demonstrate that the popular multi-agent RL algorithms struggle to generalize in CRL settings. Code and documentation are available at http:///hhhbbbooommmmmwww//////e
INTRODUCTION
Having demonstrated impressive performance in simulated multi-agent applications such as Star- craft (Samvelyan et 2019),, RI. holds potential for various multi-agent real-world applications including autonomous driving (Kiran et al. 202I), robotic warchousing (Bahrpeyma & Reichelt 2022), and traffic control (Wu et | 2021). However, compared to simulated applications, the success of RL in real-world applications has been rather limited (Dulac-Arnold et al. (021). A key challenge lies in making RL algorithms generalize across problem variations, such when weather conditions change in autonomous driving. Problem variations are common in real- world applications but may not be designed to be explicitly assessed in simulated applications (Kirk et al. 2021).
Moreover, there are many open challenges that affect the generalization RL. algorithms in real-world multi-agent problems, such the effect of complex multi-agent dynamics with aleatory uncertainty and partial observability of states and problem variations, optimizing multiple objectives over long horizons and strict physical constraints. This makes the generalization in multi-agent RL a class of problems rather than one problem as it encapsulates many open challenges (Kirk et al. 2021)
"Equal contribution.

---

Published as a conference paper at ICLR 2025
From algorithmic point of view, CRL formalizes learning policies that generalize across problem variations, aiming to learn robust, transferable, and adaptable policies (Benjamins et al.] 2022). CRL aims to find policy that can solve set of (not too different) Markov Decision Processes (MDPs), each posing a problem variation and defined by a variation context vector drawn from a context distribution. This MDP set is referred to Contextual MDP (CMDP) (Hallak at. 2015).
Thus, benchmarking CRL algorithms with real-world applications and the corresponding context distributions could move the needle on the successful use of RL for real-world applications by identifying the many open challenges that affect generalization and measuring progress. However, we observe such efforts are hindered by the lack of standardized multi-agent CRL benchmarks, especially those that are based on real-world applications and that model realistic context distribu- tions. Simulated benchmarks often address only a subset of real-world challenges and lack realistic complexity. Benchmarks based on real-world applications but not actual context distributions still remain partially simulated, adopting a subset of the same limitations. While some multi-agent RL benchmarks can be improvised for CRL benchmarking, this only superficially addresses the underlying problem (Cobbe et al., 2020), lacks desired properties of a CRL benchmark (Kirk et al. 2021) and may lead confliciing evaluation protocols (Whiteson et al. 2011; Jayawardana et al., 2022). Benchmarks that neglect these properties risk a ceiling effect and fail to incentivize further algorithmic improvements (Ellis et al. 2024 Yu al. 2022
Hu al. 2021
To fill this gap, we present Intersection/oo, a compre- hensive multi-agent CRL benchmark suite based the real-world application of cooperative eco-driving signalized intersections (Figure []. Cooperative eco-driving encodes several open challenges that af- feet generalization in multi-agent CRL, making it well suited for the purpose. IntersectionZoo is built on data- informed context distributions defined by 16,334 signal- ized intersections in 10 major cities across the United States. By identifying factors affecting vehicular emis- sion (e.g., temperature), IntersectionZoo models nearly one million traffic MDPs on these intersections. De- signed with input from domain experts, these MDPs are structured as CMDPs that interface with standard multi-agent interfaces in highly configurable, fast-to- simulate open-source framework.
Controlled Vehicle (CV)
Human Driven Vehicle (HDV)
Control via car following dynamics
Local cooperation team
Figure 1: Cooperative eco-driving at sig- nalized intersections where the controlled vehicles (CVs) are operated by an RL pol- icy (or policies) to minimize the fleet-wise emissions that include both CVs and human- driven vehicles (HDVs). CVs implicitly control HDVs through car-following dy- namics and form locally cooperative teams for better system control.
Intersection differs from existing CRL benchmarks on multiple fronts. Contrary to standard autonomous driving benchmarks centered on ego-vehicle control (Li et al., 2022), eco-driving tackles a mixed- multi- agent control problem. It is designed based on well- studied real-world application with known factors that affect generalization (Mintsis et al. 2020 Chen et al.) 2022). IntersectionZoo is built on ten CMDPs with data-driven context distributions and also provides the capability to procedurally generate CMDPs. The con-
text distributions of CMDPs are comprehensive, covering variations in states, observations, rewards (with multiple objective terms), and transition dynamics. Last, IntersectionZoo by design supports both independent and identically distributed (IID) and out-of-distribution (OOD) evaluation protocols. We also interface IntersectionZoo with RLLib (Liang et al., 2018) for ease of benchmarking.
Broader Societal Impact: Limited familiarity among RI. researchers with real-world physical systems has inhibited the development of RL algorithms addressing challenges in these domains. IntersectionZoo bridges this gap by decoupling the modeling complexity of real-world tasks from the experimental process, allowing researchers to focus on algorithmic advancements. This approach, in turn, holds the potential to improve eco eco-driving, which is known for its impact on climate change mitigation goals (Barkenbus] 22100), with the automotive industry actively pursuing robust eco-driving controllers. This aligns with the growing interest in application-driven research in machine learning, promising a mutually beneficial outcome for all communities involved (Rolnick et al. 2024).
2

---

Published as a conference paper at ICLR 2025
Remark 1: With CRL, focus on intra-task generalization, which means train and test on environments stemming from the same task (e.g., eco-driving). A related parallel direction is inter-task generalization, training generally capable agents. In this work, do not focus that.
2 PRELIMINARIES
2.1 CONTEXTUAL REINFORCEMENT LEARNING
CRL formalizes the notion of solving a collection of tasks, each stemming from the same problem (e.g., eco-driving). A collection of tasks is formalized using CMDPs (Hallak et al. 2015). We utilize the formalism from Ghosh ct al. (202I). Accordingly, a CMDP M is an MDP with state space the form (s', c) where is the state and defines the context for the state s'. The context c is fixed within an episode (e.g., a fixed random seed). Given an initial context distribution p(c), the initial state distribution of CMDP is defined by p(s) p(c) Then, given context the CMDP M is restricted to an MDP M. and is called context-MDP In other words, the CMDP manifests as collection of Context-MDPs Note that context is not always visible the agents (e.g., random seed). Then, a CMDP becomes a Contextual Partially Observable MDP (CPOMDP) (Kirk et al. 2021).
We seek to find a policy ग- that maximizes the overall expected return across all context-MDPs where R(T,, Me) is the expected return of policy æ on context-MDP Me.
max Bemp(e) R (T, Mc)l]
(I)
In aulti-agent CRL, each context-MDP Me manifests as multi-agen control problem. Further, these multi-agent control problems are often partially observable. This is also the case with coop- erative eco-driving. Then, we leverage a Decentralized Partial Observable MDP (Dec-POMDP) formulation (Bernstein et al. 2002) following previous work context-MDB?
(aan et al. 2022) to define each
2.2 COOPERATIVE MULIT-AGENT ECO-DRIVING
Optimizing eco eco-driving across full-traffic network is ideal but is impractical in large cities like Los Angeles, with nearly 5000 signalized intersections, and remains an open optimization challenge (Qadri al. 2020). A common approach is to decompose the network into individual intersections for
separate optimizations (Yang et al.) 2016 Jayawardana et al.) 2024) Yang et al., (0200) while reg- ulating intersection throughput Therefore, it is often assumed that vehicle flow is not at saturation, allowing for reasonable throughput
prevent traffic spill-back due to possible increased throughput.
improvements. We adopt the same modeling assumption.
The default objective of cooperative eco-driving at individual signalized intersections is to minimize the total exhaust emissions of a fleet of vehicles (both CVs and HDVs) while having a minimal impact on individual travel time. At given time t, the number of CVs is kev. and HDVs is DDD such that k + & n² where n² the total number of vehicles in the fleet. Then, we control the longitudinal accelerations of all CVs decentrally using a learned policy to optimize,
T Σ
11
min J
E (a,()), + XT;-
Here, T; denotes the travel time of vehicle i and time is discretized time with increments of 8 (usually 0.5 seconds). The vehicular exhaust emission function denoted by F((). which takes speed v)(() and acceleration a, (t) of vehicle i time 1. and outputs vehieular emission amount (usually the amount of carbon dioxide). is the trade-off hyperparameter. We seek to optimize J subject to hard constraints of ensuring vehicle safety, connectivity via vehicle-to-vehicle and vehicle-to-traffic signal communication, and soft constraints of vehicle kinematics, control realism, traffic safety at the fleet level (e.g., minimum time to collision across all vehicles), and passenger comfort.
Here, by MDP we generally refer any form of MDPs such as Partially Observable MDPs, etc.
3

---

Published as a conference paper at ICLR 2025
What affects generalization in cooperative eco-driving: Eco-driving is complex multi-agent control problem involving both CVs and uncontrolled HDVs where HDVs can introduce uncertainty in CV planning. Adding to the complexity is the required CV coordination. Based the environment, there can be hundreds of vehicles at an intersection and each vehicle affects the flow of traffic. Further, generalization is naturally required across many factors such as intersection topology, atmospheric conditions, eco-driving adoption level, and traffic flow rates- - defining high-dimensional context distributions. Each CV control relies on a partially observed state of its surroundings and partially observed context vector. Optimization objectives are multi-fold, with many competing objectives optimized over long horizons. Hard constraints such as vehicle safety have be satisfied. These challenges individually and collectively shape the generalization capacity of CRL algorithms.
RELATED WORK
In Table D we present our assessment of several known singlc-agent, multi-agent, and CRL bench- marks, focusing on eleven key properties desired for CRL benchmarking (Kirk et al. 2021 Cobbe et al. 2020 Benjamins et al. 2022). We aim to identify whether they satisfy the given properties or whether repurposing or improvising the current benchmark could be used to satisfy the properties.
Realistic task column assesses if a benchmark is based on real-world tasks. Many benchmarks focus on video games (Cobbe et al.] 2020; Machado et al. 2018), strategy games (Wang et al. (02Tb), grid worlds (Chevalier-Be Boisvert ct al. 2023), or simple control tasks (Benjamins et al. 20022). which
lack real-world complexity. This limitation can lead to exploitable structures (Mohan et al. 2022) or
hinder algorithmic advancements (Ellis et al., 2024 Yu et al. 2022 Hu et al. 2021). For instance, in the SMAC benchmark, a policy based only on the timestep can achieve notable win rates (Ellis et al.
2024). Some work involves more realistic robotics tasks (James et al. [2200] Yu et al. 2020 but they often use tightly constrained context features like limited friction levels.
Data-driven context distribution checks if benchmarks provide real-world context distributions, while native CRL support assesses if a benchmark is primarily designed to support CRL. Except for MetaDrive (Li et al.) (2022) and SMACv2 (Ellis al., (202)), most native CRL benchmarks focus on single-agent CRL. MetaDrive is close second to IntersectionZoo but lacks data-driven context distributions for CMDPs, limiting its ability to capture weal-wrld complexity. Moreover, the initializable context distributions column checks if a benchmark facilitates initializing arbitrary user-defined context distributions. IntersectionZoo facilitates using both additional real-world context distributions (e.g.. intersections of other cities) as well as procedurally generating contexts based on user-specified context feature distributions.
Typically, there are two types of features defining the context distribution and thereby describing the context of a Context-MDP: observed context features and unobserved context features. The key difference is whether these features are explicitly visible to the agent. Observed features are directly visible factors of variations, such as lane length in eco-driving. Unobserved features are mostly used for random variations such those that arise from procedural content generation (PCG) (Cobbe et al., 2020) and are not explicitly visible to the agent (e.g., HDV aggressiveness). Both observed and unobserved features are desired for CRL benchmarking (Kirk al. (02T). This enables systematic targeted evaluations of generalization such as systematicity (generalization using systematic recombination of known knowledge) and productivity (generalization beyond seen training
data) Hupkes et al. 2020) with non-trivial tasks.
Varying S, O,O,T, R refers variations in states (S), observations (O), transitions (7). and rewards (R). In IntersectionZoo, the variations in environments stem from states (e.g.. single vs. multiple lane driving), observations (e.g.. diverse vehicle sensor capabilities), rewards (e.g.. the trade-off between travel time and emission reduction), and dynamics (e.g., vehicle behavior changes due to varying traffic signal timings). Multiple forms of variations enable diversity and provide more avenues for targeted evaluations of generalizations (Li et al., 2022). While benchmarks like CARL (Benjamins) ct al.) 2022) and MDP Playground (Rajan et al.| [023) are categorized to consist of all forms of variations, they lack evaluation protocols spanning all these variations consist of multiple tasks without including all variations in each task.
Multiple objectives can bring another form of variation in CMDPs and are common in real-world problems. thus manifests as another axis of targeted evaluation of generalization. While existing multi-objective RL benchmarks overlook generalization challenges (Felten et al. 2024), addressing
4

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of Intersection2oo related benchmarks. Benchmarks are rated based on whether they satisfy (V), somewhat satisfy ) or do not satisfy (x) given desired property.
"Context" is abbreviated as 'ctxt.' for formatting.
Observed ctxt. features
Initializable ctxt. dist.
Data-driven ctxt. dist.
Varying S, S,O
Native CRJ support
Multi objectivee
Realistic task
Multi-agen aggnt
Unobserved
ctxt. features
OOD testing
IID testing
Benchmark
MDP Playground
x x
x
x
Rajan al.
2023)
bsuite (Osband al.
2020
X
X
x x
x
Revisiting ALE
x
x
(Machado et al. 2018
x
CARL [Benjamins et al.
2022
x
X
*
<
Procgen (Cobbe al.
2020)
Alchemy Wang
2021
Meta-World Yuu et al.
2020)
RT. BBBech (James ct al.
2020)
Minigrid
X
Thevalier-Boisvert et al.
2023
NetHack Kiitller al. 2020)
LKY
MiniHack Samvellynn ct all
2021
Particle Eny (Lowe et al. 20
SMACv2 Samvelyan
2019
X
EElls 2025
Google Football
x
(Kurach et al. |020
Flow [Wu el af. 2022
Nocturne /iiniskyy
2022
MetaDrive [Li et al. (022))
SMARTS (Zhou et al. 2021
RESCO (uult & Sharon
2021
IntersectionZoo (ours)
this gap acknowledged as needed (Kirk et al. 2021). Nocturne (Vinitsky al, 2022). Flow (Wu
al. 2021) and SMAC (Ellis et al. 2023 Samvelyan et al. 2019) can be improvised for this purpose but lack native support. IntersectionZoo naturally frames a multi-objective optimization problem, prioritizing emission reduction with less impact vehicle travel time. Further sub-objectives incorporate passenger comfort, kinematic realism, and traffic level safety (Appendix (A.).
IID and OOD testing commonly used evaluation protocols of generalization. IID evaluation assumes training and testing context sets are drawn from the same distribution. Benchmarks such as Procgen (Cobbe al. 2020) NetHack (Küttler al. 2020) employ this scheme, where the training seeds can be sampled uniformly at random from the full distribution, and the full distribution can be used for testing. OOD evaluation does not assume the same distribution for training and testing and requires domain generalization (Wang et al, 2022). IntersectionZoo models 10 different CMDPs based on 10 US cities, each with different intersection distribution. Hence, OOD evaluation can be performed by training on one CMDP and testing on another. Similarly, IID testing can be performed by train/test split within CMDP.
4 INTERSECTIONZO00
Designing a CRL benchmark suite based on real-world task is challenging. Identifying the factors of variations requires domain knowledge and expert opinion, and data-driven modeling is required
5

---

Published as a conference paper at ICLR 2025
Traffic scenario modeling layer
Signalized intersection configurations (Intersection lopologies, road grades, speed limits)
Vehicle mix distribution (cars, busses, trucks)
Vehicle inflow rates incoming approach
Human models varying driving aggressiveness (cars, busses, trucks)
Vehicle fuel mix distriuttinn (gasoline, diesel)
Temperature and humidity (weather and seasonal changes)
Vehicle age distribution (1, 2,.... 10, 10+ years)
16,334 aignalized intersections metropolitan cities
Eco-driving adoption level (0%. 100%)
pef-
traffic hours
1M+ Traffic sosnarios
CMDP modeling layer
Emission model suite
MOVES
MDP Builder (CMDP Builder)
SUMO
Evalustion metrics
Defining
User configuration layer
RLiNb
User configurations
Figure 2: A schematic overview of IntersectionZoo divided into three architectural layers.
to model them realistically without artificially making the problem easy or hard. For example, it known that eco-driving short lanes hard (Jayawardana et al. 2024). and hence, a context distribution that may have superficially many short lanes could make the problem harder than it is. In the following sections, we provide details of how we construct IntersectionZoo with the help of domain experts. We refer the reader Appendix A.9 for more detials release notes of Intersectioniooo including license details.
4.1 DEFINING INTERSECTIONZO0 SCENARIOS FOR CONSTRUCTING CMDPs
In Intersection/ooo eco-driving CMDPs are formulated as a collection of traffic context-MDPs. Each traffic scenario is a basis for context-MDP A traffic scenario manifests as combination of a set of eco-driving factor values that have known effect emission benefits at signalized intersections. These factors are related to intersection topology, human driver behavior, vehicle characteristics, traffic flow, and atmospheric conditions.
Concretely, an intersection first defined by factors such lane lengths, lane counts, road grades, turn lane config- uration, and speed limit each approach. Then, vehicle type, age, and fuel type distributions are used with ap- propriate traffic flow rates and HDV behaviors to define a realistic traffic flow. Each intersection scenario fur- ther assigned representative atmospheric temperature and humidity values based on the season. Further scenario variations can be achieved by changing the eco-driving adoption level (0%-100%) We follow this procedure for every intersection, and the resultant traffic scenarios are the basis for the context-MDD MDDs each city.
Figure 3: Left: The signalized inter- section in the intersection of Bosworth Street and Diamond Street in Salt Lake City, Utah. Right: The reconstructed intersection in simulation.
Figure illustratee the overall process: sider and how they are leveraged to build intersections, traffic scenarios, and, subsequently, CMDPs. In the traffic
the factors we con-
scenario modeling layer, we first build data-driven simula- tion environments of signalized intersections. We Open Street Maps (OSM) (Haklay Weber

---

Published as a conference paper at ICLR 2025
2008) data and follow guidelines provided by Qu et al. 2022)) Intersection lane lengths, lane counts, turn lane configurations, and speed limits are extracted from OSM. Road grades are taken from US geological surveys (Survey). An example instance of reconstructed intersection is given in Figure B More details are given in Appendix A.1 regarding all ten cities and the feature distributions.
To model the vehicle arrival process, (AADT) [Huntsinger 2022) released by the Departments of Transportation of each state/city. To differentiate the inflows between peak and off-peak hours, we use recommended conversion rates (Pre- cisiontraffic 2014). To model realistic vehicle arrival processes to intersections, we also model one-hop nearby intersections as illustrated in Figure 4
the Annual Average Daily Traffic data
For traffic signal timing, we exhaustively search through the fixed-time traffic signal timing plans (Thu nig et al. 2019) and find the optimal plan. We source vehicle age, fuel type, and vehicle type distributions from the openly available MOVES databases (epa) and data from US National Centers for Environmental Information (for Environmental Information) is used for atmospheric condition modeling with temperature and humidity changes.
To model HDV behavior, we use the Intelligent Driver Model (IDM) (Treiber al. 2000) and calibrate it using the guidelines from /Zang & Sun (2022) with real-world arterial driving data from CitySim (Zheng et al. 2022) (See Appendix [A.2 for details). While IDM controls the longitudi- nal acceleration of HDVs, lane changes are performed using standard rule-based lane changing
model (Erdmann 2014).
In the CMDP modeling layer, we use the modeled traffic scenarios and organize them based on the city to create 10 CMDPs. All traffic scenarios configured for use in the open- agent-bass based traffic simulator SUMO (oope et al. 2018). A key requirement for capturing the effect of traffic scenarios on vehicle exhaust emission is a rich emission function. For this purpose, we integrate suite of comprehensive and fast neural emission models (Sanchez et al.) 2022) that replicate the industry-standar Motor Vehicle Emission Simulator (MOVES) (epa). Finally, the user configuration layer exposes the CMDPs that can be used by researchers for training CRL algorithms. We provide RLLib-based training by default and also enable flexibility with custom implementations.
One-hop intersection Eco-driving intersection
Figure 4: For each intersec- tion, default nearby intersec- tions are added for realistic vehicle arrival processes sub- jected to nearby traffic signals.
Autonomous driving benchmarks often prioritize the realistic rendering of traffic scenarios, aiming to train closed-loop policies from sensory inputs like perception inputs to navigate the ego-vehicle. However, in cooperative criviriving, we use 2D simulator that generates vectorized representations of vehicle states, isolating the challenge of generalization in CRL over feature extraction. Given that CRL often requires many samples during training, fast simulations are also an essential requirement that we achieve through simplified rendering.
A note on simulation realism:
4.2 DEFINING CONTEXT-MDDD
Here, we provide an overview of the context-MDP definition we use in IntersectionZoo and refer the reader to Appendix A.4 for more specific details. In cooperative eco-driving, each traffic scenario manifests as a partially observable multi-agent control problem. Then, we leverage Decentralized Partial Observable MDP (Dec-POMDP) formulation (Bernstein et al.] (2022) following previous work (Yan et al. 2022) as follows.
to define each context-MDP. For each CV, state, action, and reward are defined
State Space: The design of the observed state of a vehicle is mainly based the capabilities of existing sensor technologies. The observed state of a CV includes its own status, status of neighboring vehicles (leader and follower all immediate nearby lanes), and status of the immediate traffic signal timing. Further, we provide a selected set of observed features based on the feasibility of obtaining them in the real world by the CV as observed context of the underlying environment.
Action Space: Longitudinal acceleration of each CV. Standard rule-based controller is used (Erdmann 2014) for lane changing, focusing Intersection/ooo the continuous control aspect of eco-driving.
7

---

Published as a conference paper at ICLR 2025
Multi-objective Reward Function: The reward r for each CV at time t is defined in Equation 10 inspired by Jayawardana et al. (2024). Here, n, is the vehicle fleet size, of is the velocity, and ej is the CO2 emissions of vehicle at time t. Hyperparameters include, 17 [0, 1]. a. 3, and т. The indicator function < indicates whether the vehicle is stopped, while the term ej encourages low emissions. The velocity term captures the effect on travel time. Users can configure the parameter 7j to either get a fleet-based reward, agent- agent-based reward, combination of both. All such formulations acceptable.
The Σ 11 n1
1g + al o/<r + Bet) + (1 ()( + al ofter + Be,)
Appendix A.66 details additional sub-objectives in IntersectionZoo to enhance fleet-level traffic safety, passenger comfort, and kinematic realism. IntersectionZoo ensures vehicle safety and actuator limits via pre-defined rule-based checks when used with the default objective.
EVALUATIONS IN INTERSECTIONZ00
5.1 EVALUATION PROTOCOLS
By default, IntersectionZoo provide interfaces for train/test split evaluations to measure generalization, which is often used with zero-shot policy transfer (Harrison et al., 2019 Higgins al. 2017 Kirk et al. 202I). This means we train policies on one subset of context MDPs and test on another subset of context MDPs. This includes both IID and OOD evaluation protocols. Hence, OOD evaluation can be performed by training in one city (train CMDP) and testing in another city (test CMDP). Similarly, IID testing can be performed by train/test split of context-MDPs within given city.
5.2 BASELINES
IDM: IntersectionZoo provides calibrated Intelligent Driver Models using real-world human driving data (Appendix (A.2) a human driving baseline for benchmarking the performance of an eco-driving CRL policy. The goal here to capture how much emission reduction can be obtained from driving differently than humans usually do (Jayawardana & Wu 2022 Jayawardana al., 2024
GLOSA: IntersectionZoo provides an implementation of commercially used state-of-the-an art GLOSA
eco-driving controller (Katsaros al.) (011)) for performance comparison. This is heuristical approach for eco-driving that has been implemented by popular car companies such as Audi (aud). The core idea of GLOSA is simple- avoid idling by gliding the intersection while the signal is red. The basic GLOSA controllers do not account for the impact of nearby vehicles when deciding the gliding deceleration for each ego agent, which may make sub-optimal when vehicle inflow is high.
5.3 EVALUATION METRICS
We assess the performance of a policy using the average vehicular exhaust emission per vehicle and the intersection throughput of each intersection. Refer to Appendix A.7 for more details on the definitions of these metrics.
Individual metrics:
We introduce effective emission benefits as a composite metric that aggregates the individual metrics across multiple intersection approaches for method ranking. A crucial cri- terion for evaluating the performance of an eco-driving policy is ensure that it does not reduce intersection throughput, as detailed in Section 2.2 Effective emission benefits quantify the emis- sion improvements across intersection approaches while considering only approaches where both emissions and throughput show positive gains. For any approach where throughput or emissions deteriorate compared the IDM controller, the policy reverts to human-like driving at that approach using IDM, resulting in zero benefits for both metrics. Formally, we define the effective emission per vehicle ea for an approach E under policy π as,
Composite metric:
N. 2 20& E. 0 otherwise
eq
(4)
Gddm
8

---

Published as a conference paper at ICLR 2025
where E. and Na denote the two individual metries: emission benefit percentage and throughput benefit percentage as described in Appendix [A.7 respectively. Then, the metric effective emission benefit is defined percentage improvement,
(%) 100
Σ aEA
(5)
idm
idm
INTERSECTIONZOO BENCHMARKING
In this section, we benchmark popular RI. algorithms used in multi-agent control with three objectives. First, we aim to show current MARL methods do not appear to easily solve the CRL problem in IntersectionZoo CMDPs and hence indicate that IntersectionZoo provides reasonable room for future CRL algorithmic improvements. Second, while each CMDP is not fully solved, we show that some context-MDPs in both Atlanta and SLC CMDPs are solved by these algorithms, indicating that IntersectionZoo design is reasonably tuned and MARL methods can learn and perform up to a certain extent. Third, we aim show example use cases of IntersectionZoo and demonstrate how it can be used to perform targeted evaluations of generalizations an aspect that has not been focused on by previous benchmarks.
Considering their success in many cooperative multi-agent problems Yu et al. 2022), we employ PPO (Schulman al., 2017), DDPG (Silver al. 2014), multi-agent PPO (MAPPO) and DDPG (MADDPG) with a centralized critic Yuu et al. [022] Lowe et al. 2017), and graph convolution networks (GCRL) for explicit cooperation modeling multi-agent RL (Jiang et al. 2020). GLOSA baseline controller is also used for comparison. All performance benefits are given relative to the IDM controller. According to our benchmarking goals, we mainly assess their IID generalization capacity but briefly test two more forms of targeted evaluations of generalizations: systematicity and productivity, to demonstrate the use of IntersectionZoo, Due to the space limitation, the systematicity and productivity analysis is given in Appendix B.4 We illustrate the emission benefits at the level of intersection incoming approaches instead of the intersection for a better understanding and visualization of the benefit distributions.
Method
Atlanta effective emission benefits (f) SLC effeettve emission benefits (1)
GLOSA
2.30%
0.95%
PPO
0.47%
0.05%
DDPG
2.84%
2.95 %
MAPPO
1.11% 11%
0.61% 1
GCRL
0.0%
0.0%
MADDPG
4.47 %
0.0%
Table 2: Effective emission benefits of different methods across Atlanta and Salt Lake City(SLC)
CMDPs. All percentages are given relative to human-like driving baselines given by IDM policy, Bolded values indicate performances better than the GLOSA baseline.
We assess how well the MARL algorithms can generalize when the training and testing CMDPs are the same (in-distribution generalization). For this, we leverage Atlanta CMDP (621 intersections) and Salt Lake City (SLC) CMDP (282 intersections) under summer temperature and humidity, with 1/3rd of vehicles being CVs. Following the IID evaluation protocol, train and evaluate
the same CMDP. The results are given in Table effective emission benefits. As can be seen, most of the algorithms achieve positive emission benefits overal DDPG and MADDPG in Atlanta and DDPG in SL.C perform better than the GLOOSA controller, indicating that MARL can, in fact, learn better policies than simple rule-based methods. These results serve validate one of our objectives of benchmarking the IntersectionZoo context-MDP definitions (e.g., reward and observation functions) are reasonable, and MARL methods can learn improve the emissions.
However, while the effective emission benefits are positive for most MARL methods, some context- MDPs are not solved in each CMDP. Emission benefit histograms given in Figure Atlanta further
We typically do not expect significantly high effective emission benefits percentages based on eco-driving literature. However, when done large scale, these emission reductions still result in large CO2 reductions.

---

Published as a conference paper at ICLR 2025
Atlanta + Atlanta
Atlanta - Atlanta
80
80
Approach count
70
133
1582
(ount
70
859
60
Total count: 2152
50
Total count: 2152
40
40
PPO IID
DDPG IID
30
20
20
10
10
-1.0-
-0.5
0.0
0.5
1.0+
-1.0-
-0.5
0.0
0.5
1.0+
Emission benefit percentage (%)
Emission benefit percentage (%)
(a) Benefit histogram of PPO policy
(b) Benefit histogram of DDPG policy
Atlanta Atlanta
Atlanta Atlanta
80
80
Approach count
113
1330
8 di count
2069
60
60
50
Total count: 2152
50
Total count: 2152
MAPPO
GCRL IID
30
30
20
20
10
10
-1.0-
-0.5
0.0
0.5
1.0+
-1.0-
-0.5
0.0
0.5
+.0+
Emission benefit percentage (%)
Emission benefit percentage (%)
(c) Benefit histogram of MAPPO policy
(d) Benefit histogram of GCRL policy
Atlanta + Atlanta
Atlanta + Atlanta
80
Approach count
70
889
5 count
70
72
1046
60
Total count: 2152
50
Total count: 2152
40
40
MADDPG IID
GLOSA
30
30
20
20
10
10
-1.0-
-0.5
0.0
0.5
1.0+
-1.0-
-0.5
0.0
0.5
1.0+
Emission benefit percentage (%)
Emission benefit percentage (%)
(e) Benefit histogram of MADDPG policy
Benefit histogram of GLOSA policy
Figure 5: Emission benefit histograms of Atlanta under different RL algorithms with in-distribution testing. Percentages are relative to the IDM baseline. Large y-axis counts are truncated for clarity. The total approach count refers to the total number of intersection approaches. The spikes at 0% are part due to the zeroing of emissions benefits for any scenarios where throughput is reduced.
illustrate the benefit distributions. The SLC histogram is given in Appendix Section B.3 This serves as evidence of the fact that current MARL methods do not appear to easily solve the CRL problem in IntersectionZoo CMDPs and hence indicate that IntersectionZoo provides reasonable room for future CRL algorithmic improvements.
We also show example use cases of IntersectionZoo when used for targeted evaluations of general- izations an aspect neglected by most existing benchmarks. We test systematicity and productivity of generalization (Kirk et al. 2021 Hupkes et al., (020). Due the space limitations, we refer the reader to Appendix Section B.4 for more details of this analysis and results.
CONCLUSION
In this work, we propose IntersectionZoo, comprehensive multi-agent CRL benchmark suite based on the real-world application of cooperative eco-driving. Using IntersectionZoo, we benchmark multi-agent RI,, rule-baed, and human-like driving algorithms and demonstrate that the popular multi-agent RL algorithms struggle to effectively generalize in CRL settings. A current limitation of the benchmark is can be primarily used to benchmark only continuous control algorithms. However, IntersectionZoo provides discrete lane-changing control for interested users. Further, despite our best efforts to create realistic traffic scenarios, the provided scenarios may have variations from their real-world counterparts due inevitable data errors and missing data. Overall, IntersectionZoo aims to advance generalization in multi-agen RL research by providing a rich benchmark suite that naturally captures many of the real-world problem characteristics that affect generalization.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGMENTS
The authors thank Blaine Lenoard, Mark Taylor, Michael Sheffield, Christopher Siavrakas, and Kelly Njord at Utah Department of Transportation for their constructive feedback on eco-driving scenarios. The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing computational resources to conduct experiments reported this paper.
REFERENCES
Audi expands traffic light information now includes speed recommendations to minimize stops. URL httthtt//mmmii audiusa, www..e/wwweee/ccccootcecceae.eee 301 Accessed: 2024-11-18.
Moves and other mobile source emissions models. EPA. URL https:/ www. epa. gov moves
James Ault and Guni Sharon. Reinforcement learning benchmarks for traffic signal control. In Thirly-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 1), 2021.
Fouad Bahrpeyma and Dirk Reichelt. A review of the applications of multi-agent reinforcement learning in smart factories. Frontiers in Robotics and Al, 9:1027340, 2022.
Jack N Barkenbus. Eco-driving: An overlooked climate change initiative. Energy policy, 38(2): 762-769, 2010.
Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, André Biedenkapp. Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Contextualize nh-the case for context rein- forcement learning. Transactions on Machine Learning Research, 2022.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 819-840, 2002.
Zhijun Chen, Shengguang Xiong, Qiushi Chen, Yishi Zhang, Jinqiu Yu, Junfeng Jiang, and Chaozhong Wu. Eco-driving: seientometric and bibliometric analysis. IEEE Transactions on Intelligent Transportation Systems, 23(12):22716 2022.
Maxime Chevalier-Boisvert,, Bolun Dai, Mark Towers. Rodrigo Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306.13831, 2023.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference machine learning, pp. 2048-2056. PMLR, 2020.
Gabriel Dulac-Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 11009):449-2466 2021.
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Ma- hajan, Jakob Nicolaus Foerster, and Shimon Whiteson. SMACv2: An improved benchmark for cooperative multi-agent reinforcement learning. In Thirty-sevennt Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2023. URL https
//
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi- agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Jakob Erdmann. Lane-changing model in sumo. Proceedings of the SUM02014 modeling mobility with open data, 24:77-88, 2014.
11

---

Published as a conference paper at ICLR 2025
Florian Felten, Lucas N Alegre, Ann Nowe, Ana Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno C da Silva. A toolkit for reliable benchmarking and research in multi-objectiie reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
National Centers for Environmental Information. Local climatological data (led). National Centers for Environmental Information. URL
http:/wwwwccii.... nww.. gov wwwaascceseeeeeeee http///cees///ercc//./ search/ data- sea ratarch/lle www.rrhholccacc Local imato og capicccllaaaae -asaalddaaaaaa publ iiatiiiionnn
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability. Advances in neural information processing systems, 34:25502-25515, 2021.
Mordechai Haklay and Patrick Weber. Openstreetmap: generrted street maps. IEEE Pervasive Computing, 7(4):12-18, 2008. doi: 10. 10.0PP00022200
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint /502.0 2015.
James Harrison, Animesh Garg, Boris Ivanovic. Yuke Zhu, Silvio Savarese, Li Fei-Fei, and Marco Pavone. Adapt: zero-shot adaptive policy transfer for stochastic dynamical systems. In Robotics Research: The 18th International Symposium ISRR, pp. 437-453. Springer, 2019.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference on Machine Learning, pp. 1480-1490. PMLR, 2017.
Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih-wei Liao. Rethinking the imple- mentation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2102.03479, 2021.
Leta F. Huntsinger. Part 2 transportation systems planning. In Daniel Findley, Christo- pher M. Cunningham, Thomas H. Brown, Lorraine M. Cahill, Guangchuan Yang, and Leta F. Huntsinger (eds.), Highway Engineering (Second Edition), pp. 17-81. Butterworthh-Heinemann, second edition edition, 2022. ISBN 978-0-12-822188-333
doi: http:///oi.org/10..001 B978-0 12- 11188-0-1--2211553.000000 3.00000 X. URL https: www www.scciineiiieeeccm wtt.:rrettt.c.cceeeeeeeeeeeeeeee aaticcee/11//9760122221888888
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2): 3026, 2020.
Vindula Jayawardana and Cathy Wu. Learning eco-driving strategies at signalized intersections. In 2022 European Control Conference (ECC), pp. 383-390. IEEE, 2022.
Vindula Jayawardana, Catherine Tang, Sirui Li. Dajiang Suo, and Cathy Wu. The impact of task underspecificationn in evaluating deep reinforcement learning. Advances in Neural Information Processing Systems, 35:23881-23899, 2022.
Vindula Jayawardana, Sirui Li, Cathy Wu, Yashar Farid, and Kentaro Oguchi. Generalizing coopera- tive eco-driving via multi-residual task learning. IEEE International Conference on Robotics and Automation (ICRA), 2024.
Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongging Lu. Graph convolutional reinforcement learning. In International Conference on Learning Representations, 2020.
Konstantinos Katsaros, Ralf Kernchen, Mehrdad Dianati, and David Rieck. Performance study of green light optimized speed advisory (glosa) application using integrated cooperative its simulation platform. In 201 7th International Wireless Communications and Mobile Computing Conference, pp. 918-923 IEEE, 2011.
12

---

Published as a conference paper at ICLR 2025
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions Intelligent Transportation Systems, 23(6):4909 2021.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. survey of generalisation in deep reinforcement learning. arXiv preprint arXiv:2111.0979: 1:16, 2021.
Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 4501-4510, 2020.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684, 2020.
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on paltern analysis and machine intelligence, 45(3):3461-3475, 2022.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. International conference on machine learning, pp. 3053-3062. PMLR, 2018
Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker Bilk,r-Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamaric Wießner. Microscopic traffic simulation using sumo. 2018
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiive environments. Advances in neural information processing systems, 30, 2017.
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Evangelos Mintsis, Eleni I Vlahogianni, and Evangelos Mitsakis. Dynamic eco-driving near signal- ized intersections: Systematic review and future research directions. Journal of Transportation Engineering, Part A: Systems, 146(4):04200188, 2020.
Aditya Mohan, Amy Zhang, and Marius Lindauer. Structure in reinforcement learning: A survey and open problems. Journal of Artificial Intelligence Researe, 2024.
Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvári. Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. In International Conference Learning Representations, 2020. URLhttps: hettps://openreete
net. forum? //www.prect innt ftt. www..////ooo.ossoooooooecooooooe htt.://ooooooostffiiiieoe/ studiestttssss//ss/
id=rygf-kSYwHl
Precisiontraffic. Traffic studies. Precision Traffic and Safety Systems, Aug 2014. URL http:
Syed Shah Sultan Mohiuddin Qadri, Mahmut Ali Gökge, and Erdinç Öner. State-of-an art review of traffic signal control methods: challenges and opportunities. European transport research review, 12:1-23, 2020.
Ao Qu, Anirudh Valiveru, Catherine Tang, Vindula Jayawardana, Baptiste Freydt, and Cathy Wu. What is a typical signalized intersection city? pipeline for intersection data imputation from openstreetmap. Transportation Research Board, 2022.
Raghu Rajan, Jessica Lizeth Borja Diaz, Suresh Guttikonda, Fabio Ferrera, André Biedenkapp. Jan Ole von Hartz, and Frank Hutter. Mdp playground: An analysis and debug testbed for reinforcement learning. Journal of Artificial Intelligence Research, 77:821-890 2023.
13

---

Published as a conference paper at ICLR 2025
David Rolnick, Alan Aspuru- Guzik, Sara Beery, Bistra Dilkina, Priya L Donti, Marzyeh Ghassemi, Hannah Kerner, Claire Monteleoni, Esther Rolf, Milind Tambe, et al. Application- innovation in machine learning. arXiv preprint
2024.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: /902.0404,, 2019.
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holdes Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. Advances in Neural Information Processing Systems, 2021.
Edgar Ramirez Sanchez, Catherine Tang, Vindula Jayawardana, and Cathy Wu. Learning surrogates for diverse emission models. Tackling Climate Change with Machine Learning, NewrIPS, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387-395. Pmlr, 2014.
U.S. Geological Survey. The national map elevation point query service. Elevation Point Query Service. httptttttt/tttt
nat ona 1 1mp. gov/ g
Theresa Thunig, Robert Scheffler, Martin Strehler, and Kai Nagel. Optimization and simulation of fixed-time traffic signal control in real-world applications. Procedia Computer Science, 151: 826-833, 2019.
Treiber, Hennecke, and Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical review. E, 2000.
Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang, Brandon Amos, and Jakob Foerster. Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. Advances in Neural Information Processing Systems, 35:3962-3974, 2022.
Chen Wang, Yuanchang Xie, Helai Huang, and Pan Liu. A review of surrogate safety measures and their applications connected and automated vehicles safety modeling. Accident Analysis & Prevention, 157:106157 10157,,61.1101 2021a.
Jane X Wang. Michael King, Nicolas Porcel, Zeb Kurth-Nelson, Tina Zhu, Charlie Deck, Peter Choy, Mary Cassin, Malcolm Reynolds, Francis Song, et al. Alchemy: A benchmark and analysis toolkit for meta-reiinorcemment learning agents. Advances in Neural Information Processing Systems Datasets and Benchmarky Track, 2021b.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and S Yu Philip. Generalizing unseen domains: survey on domain generalization. IEEE transactions on knowledge and data engineering, 35(8):8052 8072, 2022.
Marius Wegener, Lucas Koch, Markus Eisenbarth, and Jakob Andert. Automated eco-driving in urban scenarios using deep reinforcement learning. Transportation research part C: emerging technologies, 126: 102967, 2021.
Shimon Whiteson, Brian Tanner, Matthew E Tayloo, and Peter Stone. Protecting against evaluation overfitting in empirical reinforcement learning. In 2011 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pp. 120-127. IEEE, 201
Cathy Wu, Abdul Rahman Kreidieh, Kanaad Parvate. Eugene Vinitsky, and Alexandre M Bayen. Flow: A modular learning framework for mixed autonomy traffic. IEÉE Transactions on Robotics, 38(2):1200-1286 2021.
14

---

Published as a conference paper at ICLR 2025
Zhongxia Yan, Abdul Rahman Kreidieh, Eugene Vinitsky, Alexandre M Bayen, and Cathy Wu. Unified automatic control of vehicular systems with reinforcement learning. IEEE Transactions on Automation Science and Engineering, 20(2):789-804, 2022.
Hao Yang, Hesham Rakha, and Mani Venkat Ala. Eco-cooperative adaptive cruise control at signalized intersections considering queue effects. IEEE Transactions on Intelligent Transportation Systems, 18(6):1575-1585 2016.
Hao Yang, Fawaz Almutairi, and Hesham Rakha. Eco-driving at signalized intersections: A multiple signal optimization approach. IEEE Transactions Intelligent Transportation Systems, 22(5): 2943-2955, 2020.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang. Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611 2022.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Chengyuan Zhang and Lijun Sun. Bayesian calibration of the intelligent driver model. Transportation Research Board, 2022.
Ou Zheng, Mohamed Abdel-Aty, Lishengsa Yue, Amr Abdelraouf, Zijin Wang, and Nada Mahmoud. Citysim: A drone-based vehicle trajectory dataset for safety oriented research and digital twins. arXiv preprint arXiv:2208. 11036, 2022.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, et Smarts: An open-source scalable multi- agent rl training school for autonomous driving. In Conference Robot Learning, pp. 264-285 PMLR, 2021.
Zhaoxuan Zhu, Shobhit Gupta, Abhishek Gupta, and Marcello Canova. A deep reinforcement learning framework for eco-driving in connected and automated hybrid electric vehicles. IEEE Transactions on Vehicular Technology, 2023.
15

---

Published as a conference paper at ICLR 2025
Appendices
Appendix A Additional details
17
Details of IntersectionZoo CMDP context distributions
17
A.2
Modeling (luman-drivet vehicles
17
A.3
Emission models and their behavior
20
A.4
Defining Context MDPs
21
A.5
Design rationale behind reward function and observation function
.........
22
A.6
Additional objectives terms and their encoding schemes
22
A.7
Evaluation metrics)
23
A.8
Guidelines for using IntersectionZo0
23
A.9
License details and accessibility
24
Appendix B Expeermmental setup and results
25
B.I
Experimental setup
25
2
Notes on evaluations
25
B.3
Salt Lake City in-distribution generalization analysis
25
B.4
Systematicity and productivity in generalization
25
16

---

Published as a conference paper at ICLR 2025
A APPENDIX A ADDITIONAL DETAILS
A.1 DETAILS OF INTERSECTIONZ00 CMDP CONTEXT DISTRIBUTIONS
IntersectionZoo provides 10 CMDPs based on 10 major metropolitan cities across the United States. In Table 3, we list the ten cities we consider and the number of intersections considered for building these CMDPs in each city. In Figure we present the spatial distributions of the intersections considered in building the Intersection/2oo across the ten cities.
City
Intersection Count
Atlanta
621
Boston
917
Chicago
2864
Dallas Fort Worth
1570
Houston
1014
Los Angeles
4276
Salt Lake City
282
New York Manhattan
2586
San Francisco
1209
Seattle
995
Total
16,334
Table 3: Intersection count distribution within the ten cities.
In Figure 6 we provide comparison of feature distributions of the incoming approaches of intersec- tions of the ten cities. The y-axis is the percentage, while each x-axis is the respective feature range. Lane length is measured in meters, speed limit in meters per second, vehicle inflows are vehicles hour, and road grade is in percent grade. Phase count denotes how many different traffic signal phases are applicable for given incoming approach. The signal time ratio denotes the ratio between approach-related phase times and total cycle time.
Based on our traffic scenario modeling efforts, IntersectionZooo provides more than one million traffic scenarios. As a high-level breakdown, for all 16,334 intersections, we enable modeling four seasons as changes in temperature and humidity, four main eco-driving adoption levels ( (10% 20%, 50%, 75%, and 100%), two traffic hours (peak and off-peak) two-vehicle engine technologies (electric vs. internal combustion engine) summing up to more than I million traffic scenarios.
In Figure illustrate representative intersection topologies reconstructed in IntersectionZoo.
A.2 MODELING DUMAN-DRIVEL VEHICLES
For the foreseeable future, human-driven vehicles will remain prevalent. To model human drivers
in our simulations, we use the Intelligent Driver Model (IDM) (Treiber et al. 2000). IDM is widely accepted car-following model that can produce realistic traffic waves. IDM calculates vehicle's 's acceleration using Equation 6 with desired velocity vo. space headway so. time headway T. maximum acceleration a, and comfortable braking deceleration 3. The velocity difference with the leading vehicle is denoted as Ac(t), and ó is constant.
((())
8" (o((), 27 )] s(1)
a(t) =a
(6)
60
s" (v((t, Ao(t)) = so + max 0, 0 + v(t) 2/0B
For simulation accuracy, precise calibration of parameters Up, So. T, a, and B is crucial. Different regions may exhibit varying driving behaviors, such as American drivers versus British drivers. Thus, calibrating the IDM parameters is critical for accurate human driver modeling.
17

---

Published as a conference paper at ICLR 2025
Speed
Weiile iritw (peak) uelicle
coutt
Road
Phase
rats
S441
20%
20%
61%
25%
25%
30%
20%
10%
20%
250
500
750
00
30
1000
2007
0%
1630
2003
-22
Dd
0%
C0
0%
1.0
1000
6015
201
90%
50%
20%
25%
0%
U030
04
40%
100%
60%
300
50%
40%
40%
2015
50%
20%
100%
220
1000 2002
1020 2002
20%
20%
40%-
35%
25%
50%
20%
1015
59%
20%
250
500
7500
10
20
1000 2002
1030 2003
2
--2
0.0
00
05
1.0
100%
60%
65%
20%
90%
201
50%
23%
10%
09
0%
0%
5030
0%
09
40%
100%
60%
30%
40%
40%
20%
10%6
20%
10%
250
1000 2002
1000 2002
-32
1.0
San Poreiie
20%
200%
40%
37%
30%
75%
2015
59%
20%
250
500
770
10
20
30
1000 2002
0%
1030 2003
2
0%
-2
00
00
05
1.0
1004
80%
65%
30%
D302
90%
20%
20%
20%
25%
09
0%
0%
1000
2
0%
0%
100%
60%
20%
40%
40%
20%
59%
20%
10%
250
3500
1030
2002
50%%
20%
27%
25%
50%
20%
10%
20%
250
500
750
10
1000
2002
1030
2003
-02
0.0
00
05
3.0
Figure 6: Comparison of context feature distributions of the incoming approaches of the ten cities. The y-axis is the percentage, while each x-axis is the respective feature range.
Our IDM model calibration goals threefold. We aim to align it with real- world human driving behavior at US signalized intersections, tailoring the five IDM parameters for human-like trajectories. We also need separate IDM models for distinct vehicle types (e.g., cars, buses, trucks) due their influence on driving style. Lastly, we aim to establish a range of IDM models reflecting diverse driver behaviors and aggressiveness observed in real-world driving.
For this purpose, we employ Bayesian inference and Gaussian process-based approach proposed by Zhang & Sun (2022) and leverage real-world arterial driving data from CitySim (Zheng et al., 2022).
(T--)AA, where represents the index for each driver d E (1, D) and For Vtt, d) E (t, d) ft=tod==1
represents the timestamp, we have,
In(0) N (No, 2o) g In (ax) N (He: 01) E IR
(8a)
(8b)
(++At)
lid. N FIDM (1))0) (o,At) ER
(8c)
d
Here, 0 = [No So. T, a, 3] E R the IDM model parameters where [(u, σ) represents a Gaussian distribution with mean of j and standard deviation of a. Independent and identically distributed is
18

---

Published as a conference paper at ICLR 2025
Figure 7: Left: snapshot of the real-world intersection. Middle: the reconstructed intersection topology SUMO. Right: the full network around the ego-intersection with one-hop nearby intersec- tions added for realistic vehicle arrival processes subjected nearby traffic signals. IntersectionZoo covers many of the diverse intersection topologies as briefly illustrated here.
indicated by i.i.d. Furthemore, we denote the inputs at time a vector (/ [s ud A vt
(t) {fo. to + (T 1)At} where (t) is the headway, 2/ is the velocity and Aud is the relative velocity driver and its leading vehicle at time t. We further define a function FIDM(() that updates the ego vehicle's speed at + At using Equation 6 Equation where At is the step size.
(2)
(((+1)) ( A0
The formulation in Equations Jasioonssggeenne generatee population-level joint distribution of IDM parameters. As vehicle type influences driving behavior, we build distinct joint distributions for each vehicle type. This requires separate human-driving trajectory datasets for every vehicle type.
CitySim (Zheng et al.) 2022) is a drone-based vehicle trajectory dataset that provides vehicle trajec- tories along arterial roads, but it lacks explicit vehicle type labels. Therefore, we use the bounding box length of vehicles distinguish cars from buses and trucks. While this identifies cars, it does not allow us to differentiate between trucks and buses. Thus, we assume a common IDM parameter distribution for trucks and buses. This is reasonable both are heavh- vehicles with similar behavior at signalized intersections. After modeling these joint distributions, we utilize them to sample human drivers for the micro-simulations.
19

---

Published as a conference paper at ICLR 2025
San Francisco
Los Angeles
Atlanta
Houston
Manhattan, New York
Chicago
Seattle
Dallas
Boston
Salt Lake City
Figure 8: The distributions of the intersections considered in building the IntersectionZoo across the ten cities.
A.3 EMISSION MODELS AND THEIR BEHAVIOR
key requirement for capturing the effect of traffic scenarios vehicle exhaust emission is rich emission function. For this purpose, we integrate a suite of comprehensive and fast neural emission models (Sanchez et 2022) that replicate the industry-standard Motor Vehicle Emission Simulator (MOVES) (epa). For more details on how these neural surrogate emission models are built, we refer the readers Sanchez al. (Sanchez et al. 2022).
we present a visual illustration of the emission landscape of some of these emission models when the vehicle's instantaneous acceleration and velocity are changed. We present these details for interested users the objective of the CRL problem is specified by these emission models' behaviors. While different vehicle models under different conditions demonstrate varying emission quantities for given velocity and acceleration, few observations are generally common among them. First, decelerating under any velocity is preferred. This is consistent with the design of the GLOSA controller, which adopts gliding behavior when the signal is red. Second, the higher the acceleration, the more the emission with higher velocity and higher acceleration pairs may result in the highest emissions.
In Figure 9.
20

---

Published as a conference paper at ICLR 2025
2009/Diesel/Passenger car/44/70/1 2000/Gasliin///asseeeer... car/80/75/0
car8////0/
2010/Diesel/Posseneer car/50/60/0
velocity. mis
velocity m/s
velocity mis
10
10
10
4
-
-4
-2
0
-4
-2
acceleration m/s?
acceleration m/s?
acceleration m/s²
2010/Gasoinne//sssenner car/66/50/2
2012/Dieeel/Bus/233447-.
2012/Gasolire/Passenngr truck/41/25/-
velocity mnls
15
velooh m/s
15
velocity mls
15
10
10
10
-2
-4
-2
-4
-2
Emission
acceleration m/s?
acceleration m/s?
acceleration m/s?
2016/Diesee//assengee truck/75550/0
2016/Gasolime/Bss//0
2018/Diesel/Bus/22/25/00
velocity mis
15
velocity mis
15
velocity mis
15
-2
-2
acceleration m/s?
acceleration m/s?
acceleration m/s
201s/eiesel/Passengge car/56/68/-0 2019//asoliin//asseeeeeee car/32/48/1
velocity mis
velocity mis
velocity mis
10
10
10
5
44
2
-4
2
0
-4
22
0
acceleration
acceleration m/s?
acceleration m/s?
Figure 9: Emission landscapes of neural surrogate emission models. The figure depicts several randomly chosen contour plots depicting emission landscapes from various vehicle models operating under different conditions. Each plot title should be interpreted as vehicle manufacture year/ fuel type/ vehicle type/ lemperature/ humidity/ road grade. Temperature is in Celsius, and humidity the relative percentage, and the road grade is in degrees.
A.4 DEFINING CONTEXT MDPs
State Space: The design of the observed state of a vehicle is mainly based on the capabilities of existing sensor technologies. The observed state includes the speed of the ego-vehicle, relative distance to the traffic signal, traffic signal state (red, green, or yellow) for the current phase, time remaining in the current phase, time remaining until the traffic signal turns green for the second and third cycle, vehicle location (i.e., flag indicating whether the vehicle is approaching the intersection, the intersection, or exiting the intersection). index of the vehicle's current lane, vehicle's intention to turn right, left, go straight at the upcoming intersection, and for the follower and the leader vehicles on the same lane, adjacent right lane, and left lane of the ego-vehicle: speed, relative distance, turn signals status (turning right, left, none).
As mentioned earlier. IntersectionZoo has both observed features and unobserved PCG-based features defining the context of context-MDP, For users interested in conditioning the policies based on the context, we provide observable context features that include eco-driving adoption level, signal timing plan for the traffic signal phase relevant to the vehicle, atmospheric conditions such temperature and humidity, the fuel type (electric or internal combustion engine), and information about the ego-vehicle's s current approach (number of lanes, lane length, speed limit). The decision which
21

---

Published as a conference paper at ICLR 2025
features are available for conditioning is also based on the feasibility of implementing them in the real world.
Action Space:
Longitudinal acceleration of each CV. For lane changing, standard rule-based controller is used (Erdmann 2014). This focuses IntersectionZoo on the continuous control aspect of eco-driving.
Multi-objective Reward Function: The reward r for each CV at time t is defined in Equation 10 inspired by Jayawardana et al. (2024). Here, n, is the vehicle fleet size, vi is the velocity, and e, the CO2 emissions of vehicle at time t. Hyperparameters include, a, 3, and r. The indicator function 1, < indicates whether the vehicle is stopped, while the term ej encourages low emissions. The velocity term captures the effect on travel time. Users can configure the parameter 1j to either get a fleet-based reward, acceptable.
reward, or a combination of both. All such formulations are
1 nh
Rt (ns + al ofer + Bel) + (1 -)((t + allerrr + Bee)
(10)
Users can extend the above default objective with additional reward terms as explained in Section A.6
A.5 DESIGN RATIONALE BEIHNN REWARD FUNCTION AND OBSERVATION FUNCTION
Our decisions on reward function design and observation design have been shaped by existing eco- driving literature and the existing sensor technologies to capture the state of a vehicle. Below we detail some of the decision choices and reasons behind them.
Reward function: As often discussed in eco-driving literature, many studies use emission and vehicle speed as the subterms of the objective function of the optimization methods (Jayawardana & Wu 2022; Wegener et al. 2021 Zhu et al., (2023] Jayawardana et al.) (024). The emission term natural choice as the goal is to reduce the emission. The speed term is used as shaped proxy reward for throughput maximization, as throughput itself would be a sparse reward (that can only be calculated at the end of the episode). Then, we introduce a term that penalizes stopping. This has also previously been used in eco-driving literature Jaaywarddana & Wu (2022). In our case, the choice to do this serves three purposes. First, based on the common emission model behaviors as explained in Section [ decelerating is often preferred action to reduce emission with stopping, and idling often emits more than decelcrating. Therefore, it is preferred to encourage gliding behavior. This has been the core design principle of the GLOSA controller, and our penalty term encourages gliding to avoid idling. Second, if stopping is not penalized, methods could reward cheat by making the controlled vehicles stop at the beginning of an incoming approach, blocking the other vehicles and hence reducing emissions. With the stopping penalty term, we discourage these undesired behaviors. Third, the gliding behavior encouraged by the stopping penalty term also results in smoother vehicle jerk, which could be more comfortable for the passengers. Therefore, we use these three terms in the reward. Following previous eco-driving work (Jayawardana & Wu 2022 Wegener al.] 2021 Zhu et al. 2023: Jayawardana et al. 2024). we use a linear combination of these terms as the final objective, where weight coefficients are found by setting the weights based on each term's 's relative scale and doing a hyperparameter search around the starting confi configuration.
Observation function: Our observation function features are designed with the requirement that they should be available to the vehicle with the existing sensor technologies both onboard the vehicle and based on roadside units (RSU). By design, we provide all features that are considered in previous eco-driving literature. This is often the strategy adopted by existing benchmarking work based on real-world applications AAult & Sharon (202]); Vinitsky et al. (2022). The rationale behind this missing from the observation, and if some features are less important for solving the problem, a given method could learn to ignore them. If required, as explained in Section A.8 users are allowed to perform observation shaping.
decision is make sure that none of the essential features
A.6 ADDITIONAL OBJECTIVES TERMS AND THEIR ENCODING SCHEMES
IntersectionZoo provides additional objective terms for users who wish to assess the effect of multiple objectives on generalization.
22

---

Published as a conference paper at ICLR 2025
Passenger comfort: To accommodate passenger comfort, vehicles should maintain low accelerations and decelerations. To encourage this behavior, a reward term defined as the (ay where a is the acceleration (or deceleration) of the vehicle at time t. When used with shared fleet-wise reward, the mean of |ax] across all vehicles is used.
Kinematic realism: Vehicles often cannot have high jerks (changes in accelerations in unit time) as actuators have jerk limits. To account for this, IntersectionZoo provides jerk control as |a, 9--1 where ar is the acceleration (or deceleration) of the vehicle at time When used with shared fleet-wise reward, the mean jerk across all vehicles is used.
Fleet-level safety: While individual vehicle safety is ensured using pre-defined rule-based checks, IntersectionZoo provide surrogate safety measures such as Time To Collision (TTC) to improve traffic flow level safety. These surrogate safety measures are commonly used by traffic engineers to measure the impact of new roadway interventions (aag et al. 2021a).
Time to Collision (TTC) for a vehicle measured as the time it would take for the vehicle collide it were to continue moving along its current paths without any changes in speed or direction. Formally, TTC = Ze where Ad is the relative distance and Áv the relative velocity. Both distance and velocity are measured relative to the leading vehicle of the ego-vehicle. In using TTC for fleet-level safety, we take the minimum TTC value across all vehicles a given time step and share it with all vehicles.
A.7 EVALUATION METRICS
To measure the generalization performance of a learned CRL policy, we leverage two metrics.
Average emission benefits: This measures the per vehicle per time step CO2 emissions in grams based on the CRL polic and compares that with the human driving baseline. The lower the emissions, the better. Percentage emission reduction is given as the benefit.
Average intersection throughput benefits: This measures the number of vehicles that cross the intersection during an episode. We present as the average throughput change percentage compared to the human driving baseline.
In addition to the task-specific metrics, users may present the training stability, sample complexity, and qualitative evaluations of the behavior of the agents through visualization either supportive metrics or main evaluation metrics depending on the goals of the benchmarking. Furthermore, additional objective terms such as passenger comfort and kinematic realism of vehicles used in the objective as explained in Section A.6] it is important to present relevant metrics that can capture the effects of these objectives. As these metrics are often experimental purpose-specie specific, do provide a default implementation and leave it to the users to define their custom metrics.
A.8 GUIDELINES FOR USING INTERSECTIONZ00
To standardize the use of IntersectionZoo, we provide the following guidelines.
The environment comes with a default reward and observation function definitions. However, users are allowed to perform reward and observation shaping.
A key requirement for assessing the performance is to ensure intersection throughput per intersection is never reduced by the learned eco-driving policy as explained in Section 2.2 If, for any intersection, the throughput is reduced (even there are low emissions), its emission and throughput benefits must be set zero.
We recommend presenting the performance histograms (as given in Section 6 apart from presenting the average emission benefits over the intersection approaches of a given CMDP to visualize the generalization capacity of algorithms. This would facilitate identifying methods that overfit to few specific intersections with dominating performances.
The bounds the controlled vehicle acceleration, minimum possible time to the vehicle in front, minimum headway, and willingness to obey the speed limit rules should be respected. These set constraint the actions based on vehicle actuator limits and to ensure safety.
23

---

Published as a conference paper at ICLR 2025
A.9 LICENSE DETAILS AND ACCESSIBILITY
Our code and the IntersectionZoo are released under the MIT License and are available at httpr///oomiiiwuuaaanntteeeeeeee
The intersection datasets also released under the Liceice All data used for creating traffic scenarios are based on publicly available open data.
SUMO traffic simulator licensed under the EPL-2.0 with GPL v2 later as secondary license option (refer to SUMO website for more details).
24

---

Published as a conference paper at ICLR 2025
B APPENDIX B EXPERIMENTAL SETUP AND RESULTS
B.1 EXPERIMENTAL SETUP
All experiments are carried out using RLLib (Liang et al. 2018) with the default hyperparameter configuration. All policies are trained as multi-task learning policies where the context to condition the policy is defined Section A..4 We leverage 10 multiple workers in training the multi-task learning policies. Experiments were carried out in a computing cluster with 20 CPUs and an NVidia Volta V100 GPU with 32GB RAM. Each benchmarking run took roughly hours in RLLib, with 5000 episodes (each with horizon of 1000 steps with 50 warmups). We purposely ran each run for large number of iterations to ensure no further training can improve the policies. Benchmarking runs can be run for shorter number of iterations, reducing computation times further.
B.2 NOTES ON EVALUATIONS
For the reported results in Section 6 for each algorithm, we train with four random seeds. We train for 500 training iterations to ensure policies are well-converged. During the evaluation, we select the best-performing policy based on the rewards, vehicle throughout, and emission reductions.
B.3 SALT LAKE CITY IN- R--ISTTIBUTTON GENERALIZATION ANALYSIS
In this section, we demonstrate the emission benefit histograms of Salt Lake City CMDP with different MARL methods. Figure 10 illustrates the results. As discussed in Section 6 for Atlanta CMDP, SLC CMDP behaves similarly. While the effective emission benefits are positive for most MARL methods, some context-MDDPs are not solved, can be seen from the histograms. This serves as evidence of the fact that current MARL methods do not appear to easily solve the CRL problem in IntersectionZoo CMDPs and hence indicate that IntersectionZoo provides reasonable room for future CRL algorithmic improvements.
B.4 SYSTEMATICITY AND PRODUCTIVITY IN GENERALIZATION
Below, as an example use of IntersectionZoo, we assess two targeted evaluations of generalization using IntersectionZoo, namely systematicity and productivity (Kirk et al.) 2021).
Systematicity in generalization: Systematicity generalization using systematic recombination of known knowledge (Kirk et al. 2021 Hupkes al. 2020). To test this ability of DDPG and PPO, leverage Intersection/oo' 's (2021) we first define capability to procedurally generate context-MDPS.. Following Kirk et al. of context features and their corresponding values set of uniform distributions (per feature). Then, we train policies by sampling feature values from each distribution create context vectors. However, certain feature value combinations are never used during training. During testing, we only use the feature value combinations that were not used in training. This tests the algorithms' ability to systematically combine known knowledge to generalize.
We define in Table 4 the feature distributions for training, and in Table the feature distribution for evaluation. During training, we sample intersections from Table 4 defined distribution but also not in Table defined distribution. During evaluations, we only sample intersections from Table 5 defined distribution.
The resultant performance histogram is given in Figure Ila While exact performance results are not the core focus of this experiment but rather to show the use IntersectionZooo performing an analysis like this, we observe both DDPG and PPO fail to systematically generalize; baseline performs better in most cases.
Productivity in generalization. Productivity is when the learned policies generalization beyond seen training data (Kirk et al. [222 Hupkes et al., 2020). To test this in PPO and DDPG, we perform an OOD evaluation by using policy trained The resultant performance histogram is given in Figure [ib
CMDP with zero-shot transfer to Atlanta CMDP.
We would like to note that the primary focus of this experiment is not the exact performance results, but rather demonstrating how IntersectionZoo can be effectively used to conduct such an analysis. That said, though the in-distribution performance analysis in Section shows the DDPG policy
25

---

Published as a conference paper at ICLR 2025
Salt Lake City + Salt Lake City
Salt Lake City + Salt Lake City
Approach count
50
913
Approach count
50
248
40
40
30
Total count: 1048
30
Total count: 1048
PPO IID
DDPG
20
20
10
10
-1.0-
-0.5
0.0
0.5
1.0+
-1.0-
-0.5
0.0
0.5
1.0+
Emission benefit percentage (%)
Emission benefit percentage (%)
(a) Benefit histogram of SLC with PPO policy
(b) Benefit histogram of SLC with DDPG policy
Salt Lake City Salt Lake City
Salt Lake City Salt Lake City
Approach count
50
Reproach count
50
1033
Total count: 1048
30
Total count: 1048
MAPPO
GCRL IID
20
20
10
10
-1.0-
-0.5
0.0
0.5
1.0+
-1.0-
-0.5
0.0
0.5
+.0+
Emission benefit percentage (%)
Emission benefit percentage (%)
(c) Benefit histogram of SLC with MAPPO policy
(d) Benefit histogram of SLC with GCRL policy
Salt Lake City + Salt Lake City
Salt Lake City - Salt Lake city
Approach count
50
1048
Approach count
50
447
40
40
30
Total count: 1048
30
Total count: 1048
MADDPG IID
GLOSA
20
20
10
10
-1.0-
-0.5
0.0
0.5
1.0+
-1.0-
-0.5
0.0
0.5
1.0+
Emission benefit percentage (%)
Emission benefit percentage (%)
(e) Benefit histogram of SLC with MADDPG policy
(l) Benefit histogram of SLC with GLOSA policy
Figure 10: Emission benefit histograms of Salt Lake City (SLC) under different RL algorithms with in-distribution testing. Percentages are relative to the IDM baseline. Large y-axis counts are truncated for clarity. The total approach count refers to the total number of intersection approaches. The spikes 0% are part due to the zeroing of emissions benefits for any scenarios where throughput reduced.
PCG
PCG
Salt Lake City Atlanta
Approach count
16
40
119
count
140
945
1791
120
12
Total count: 128
100
Total count: 2152
8
DDPG
80
DDPG OOD
PPO
60
PPO OOD
4
40
20
0
-1.0-
-0.5
0.0
0.5
1.0+
-1.0-
-0.5
0.0
0.5
1.0+
Emission benefit percentage (%)
Emission benefit percentage (%)
(a) Systematicity evaluation using procedurally gener- ated intersections
(b) Productivity evaluation using policy transfer from Salt Lake City to Atlanta
Figure 11: Performance histograms PPO and DDPG in assessing systematicity and productivity in generalization. All emission benefit percentages are measured relative to the human-driving baseline. For y-axis counts that are large, we truncate them for better visualization and indicate the count on the plot. The spikes 0% are in part due to the aforementioned zeroing of emissions benefits for any scenarios where throughput is reduced. The total approach count is also given in each plot for reference, and the title indicates train CMDP test CMDP.
trained on SL-C CMDP seems perform slightly better than the policy trained on Atlanta CMDP,
26

---

Published as a conference paper at ICLR 2025
Feature
Value range
Lane setup
(1,),
(3,1), (3,2) Format: (lane count, phase count)
Vehicle inflow
[100, 600] vehicles per hour
Green phase time
[20, 32] seconds
Red phase time
[20, 32] seconds
Lane length
[100, 775] meters
Speed limit
[16, 20] m/s
Signal offset
[1.3] seconds
Table 4: Feature distribution for training for systematicity
Feature
Value range
Lane setup
(2,2), - Format:
(lane count, phase count)
Vehicle inflow
[400, 500] vehicles per hour
Green phase time
[26, 29] seconds
Red phase time
[26, 29] seconds
Lane length
[325, 500) meters
Speed limit
[17. 18] m/s
Signal offset
[2, 3] seconds
Table 5: Feature distribution for evaluation systematicity
after the transfer, both DDPG and PPO seem to perform poorly, further indicating the limitations of existing RL algorithms when it comes to generalization across problem variations.
27


Reviews:
[{"review_id": "RDItKJxmnv", "reviewer": "Reviewer_dDfH", "summary": "**Overview**  \nThis work proposes a new benchmark for testing the generalizability of MARL algorithms across different MDP settings. With the background of reducing $CO_2$ emission by controlling vehicle fleets, progress on this benchmark may potentially generate real-world impact and advance the development of MRL. \n\n**Method**  \nSpecifically, intersections were collected for 10 major cities in the US and reconstructed in the SUMO simulator with a set of configurable parameters (context) like vehicle throughput, speed limit, temperature, the ratio of autonomous cars, and so on. The parameter distribution is provided by referring to literature from the transportation department. Thus, sampling parameters from these distributions allow the simulator to build intersection scenarios similar to its real-world counterpart. Each parameter combination forms a special MDP, and thus training agents with scenarios sampled from the same distribution allows benchmarking in-distribution performance. Testing agents in scenarios built with another parameter distribution can reflect the algorithm's OOD performance.\n\n**Experiment**   \nThe experiment shows that the current SOTA MARL method struggles to do even in-distribution generalization, indicating there is a large room for improvement.", "questions": "1. How would the temperature and humidity affect the policy behavior? For example, is it better to drive at low speed in summer to reduce the CO2 emission? But it would reduce the throughput right?\n2. As far as I know, the OSM data only provides the location of the traffic light and doesn't tell which lane it controlls. How does this information be completed in IntersectionZoo?\n3. Type: Appendix, table 3. evaluation -> training", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. It is a good application for MARL researchers to study how to build algorithms to further reduce exhaust emissions and make a real-world impact. \n2. The scenarios database is large. With realistic context distribution, IntersectionZoo is a good place to study the generalizability of algorithms. Also, the whole system including the intersection data, simulator, metric, and training framework is built reasonably and solidly.\n3. As an open-sourced benchmark, the code is of good quality with documentation included. Seems it is ready for use.", "weaknesses": "1. The writing is poor. The experiment figures are hard to understand. Specifically, what does the total count mean? Though I can roughly infer it, it would be better to explain it in the paper. For the benefit percentage, does 0.5 means 0.5% or 50%. If it is 0.5%, it seems the emission doesn't change a lot by switching to MARL control, and the key is that MARL agents can not achieve the same level of throughput as human drivers. However, it seems GCRL agent is close to human drivers' performance. I suggest giving more analysis and explanation to these figures. It can be done in the captions since I found the captions of Figure 5 and Figure 6 are simple copy-paste. Also, is it possible to provide more visualizations in the appendix or even provide a video? So readers can know how different the map topologies and vehicle distributions are for these cities. I can not imagine a large dataset from pure text.\n2. More experiments or results are needed. Though the experiments show that MARL can not generalize well, we don't know if it is due to the algorithm itself or if the task is too difficult to learn for RL agents, i.e. improper reward function and observation or MDP definition. One way to clarify this is to provide training performance as well. It needs to be proved first that the MARL can overfit the training scenarios and exceed human driver performance. Then we can confirm that the simulation design is appropriate, so MARL agents have the chance to surpass humans, and the failure on the test set is indeed due to poor generalizability. Also, parameters are fixed for the experiments like the adoption rate, which is 1/3. Did the authors conduct experiments to see which factor affects the performance most? I believe the adoption rate is an important factor since the MARL performance would be largely influenced by the population size. Also, a baseline can be added to use the IDM vehicle control these CVs to see if the MARL can outperform this simple rule-based baseline at least at training time. With such a trend, we can conclude it shows some signs that MARL can achieve better performance, and thus work on this benchmark is promising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yWTiTci77G", "reviewer": "Reviewer_Ld56", "summary": "In this paper the authors introduce a new benchmark for multi-agent reinforcement learning (MARL) called IntersectionZoo, designed to address the cooperative eco-driving task within urban networks. The objective of cooperative eco-driving is to control a fleet of vehicles at signalized intersections to minimize fleet-wide emissions with minimal impact on individual travel times.\n\nThe authors discuss the benchmark in the context of Multi-Agent Contextual Reinforcement Learning (MA-CRL) and highlight how IntersectionZoo is purposefully tailored to generate multiple scenario variations (i.e., contexts) and evaluate the generalization capabilities of MA-CRL agents across such variations. Specifically, IntersectionZoo addresses a range of contextual factors, including intersection configuration, vehicle type distribution, fuel type distribution, human driver models, weather conditions, eco-driving adoption rates, peak and off-peak traffic patterns, among others. The benchmark leverages data across 10 cities and over 16,000 signalized intersections and is built on open-source software, e.g. SUMO for microscopic traffic modeling, and RLLib for RL training.", "questions": "- Can the authors elaborate on how IntersectionZoo is expected to benefit the broader MA-RL community? What features of the cooperative eco-driving task are common to other MA-RL tasks? To what extent the insights obtained on IntersectionZoo can be extended to broader MA-RL settings? More specifically, could the authors suggest particular MA-RL domains where insights from IntersectionZoo are likely, or unlikely, to transfer? Providing both positive and negative examples would help the readers with assessing whether IntersectionZoo could serve as a relevant benchmark for their own purposes.\n\n- Can the authors provide a more extensive experimental analysis of MA-RL algorithms on this benchmark? If no algorithm performs well on the benchmark, what are the underlying causes for such a performance gap? For example, IntersectionZoo focuses on two performance metrics: average emission benefits and average intersection throughput benefits. While these metrics are interesting from a domain-driven perspective, the paper would benefit from considering additional algorithm-driven metrics relating to, e.g., sample efficiency, stability of training, etc. Moreover, a qualitative visualization of the resulting policy would further improve the experimental assessment.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I resonate with the overall purpose of this work: bridging the gap between RL research (focused on benchmarks and generality of algorithms) and impactful, real-world applications. The paper is generally well-written and straightforward to follow. The authors thoroughly compare their benchmark with a comprehensive body of literature in the field. Based on my understanding, this work entails significant data collection and processing, which could provide substantial benefits to the broader transportation research community.", "weaknesses": "In my opinion, the primary limitation of this work is its relatively narrow contribution to the broader RL, or specifically multi-agent RL, community. Although the authors claim that IntersectionZoo aims to serve as a novel standardized benchmark for MA-CRL research, many of the benchmark’s design decisions and characteristics are highly tailored to cooperative eco-driving and to the specific problem formulation adopted within this work. While evaluating generalization performance across different scenario characteristics is important for this application, it seems questionable to assert that results and insights on IntersectionZoo would extend to MA-RL research more broadly. This, to me, somewhat undercuts the fundamental purpose of a benchmark.\n\nFor instance, while scenario factors vary, the overall application and formulation remain fixed. This impacts algorithmic choices as well, such as the mandatory choice of a continuous action space (acceleration of the controlled vehicles), a specific reward function design, limited environment scale (restricted to single intersections), predefined driver models, and no significant uncertainty sources. These are just a few examples.\n\nTo the best of my understanding, this work would have greater impact if it concentrated more on the application side—namely, contributing novel algorithms to address a critical problem—rather than positioning this benchmark to generate deep insights into MA-RL algorithm development. \n\nFurthermore, the experimental analyses in this work need _considerable_ improvement. The authors provide a brief comparison of PPO, DDPG, MAPPO, and GCRL, concluding that these algorithms fail to generalize and perform poorly across all tested conditions. While experimental evaluation may not be the core focus of this paper, a thorough and insightful assessment of established MA-RL algorithms is essential in order to evaluate the usefulness of a benchmark. Is it the case that no algorithm performs adequately in any of the examined scenarios? The authors should consider expanding the experimental section by exhaustively benchmarking popular RL algorithms on the proposed benchmark, with in-depth evaluation and analyses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pC3kteriTx", "reviewer": "Reviewer_r4Ef", "summary": "The paper proposes a benchmark suite, IntersectionZoo, for contextual reinforcement learning evaluation. The suite is designed based on a complex real-world task, i.e., cooperative eco-driving at signalized intersections. This paper well summarizes the recent work related to RL benchmark. Several experiments are conducted to test some popular multi-agent RL algorithms and show that all the adopted algorithms perform poorly on generalization. The open-source code is also provided to facilitate the community.", "questions": "What does the “baseline” refer to in “Both DDPG and PPO fail to systematically generalize; baseline performs better in almost all cases” in Line 497?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The originality is good. This paper addresses the problem of contextual reinforcement learning by creating a comprehensive benchmark for eco-driving simulation.\n\nThe problem formulation is presented clearly and in detail. \n\nThe literature review offers an extensive and in-depth overview of related work.\n\nThe proposed IntersectionZoo benchmark can be a boost for contexture RL research.\n\nOpen-source code is provided to support the research community.", "weaknesses": "* The main content seems to exceed the page limit – the 8. The Reproducibility Statement is on the 11th page.\n\n* PPO and DDPG are not specifically for multi-agent RL but are only used with independent agents. It seems not reasonable to choose these two algorithms for testing. Besides, for systematicity and productivity in generalization tests, the authors only use PPO and DDPG for evaluation, which is confusing.\n\n* From your experiments, it’s hard to get the conclusion in Line 532 that “the popular multi- agent RL algorithms struggle to generalize in CRL settings” since only two MARL-based methods, i.e., MAPPO and GCRL, are adopted for evaluation, and just on the in-distribution generalization.\n\n* Given the unsatisfactory results of the chosen multi-agent RL algorithms on this benchmark, it would be better to briefly review the latest multi-agent RL to justify why you chose PPO, DDPG, MAPPO, and GCRL for evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SPJntw8R5L", "reviewer": "Reviewer_giey", "summary": "This paper proposes IntersectionZoo, a comprehensive benchmark suite for multi-agent CRL through the real-world application of cooperative eco-driving in urban road networks. The authors ground IntersectionZoo in real-world applications, and capture real-world problem characteristics that are neglected in previous datasets, including partial observability, and multiple competing objectives. Also, IntersectionZoo covers 16,334 signalized intersections derived from 10 major US cities, enabling the testing of policy transfer between different cities. Using traffic scenarios in the proposed dataset, the authors benchmark various popular multi-agent RL and human-like driving algorithms and demonstrate their performance in Eco-driving.", "questions": "1. Could you please provide some experimental results to verify that IntersectionZoo can reflect the real-word vehicular exhaust emissions precisely?\n2. Could you please explain the rationality of your benchmarking results? How do they strengthen the contribution of the proposed dataset?\n3. Could you please provide some major testing results of the popular algorithms on other existing benchmarks? How do the results on other comparable benchmarks differ from the results on IntersectionZoo? Will the differences illustrate the advantage of IntersectionZoo?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper focus on benchmarking the task of cooperative Eco-driving. The idea of this task is to control a fleet of vehicles to reduce fleet-level vehicular emissions. The task sounds interesting and is absolutely beneficial to reduce air pollution in the era of global climate change.\n2. The authors ground IntersectionZoo in real-world applications, and capture real-world problem characteristics that are neglected in previous datasets, including partial observability, and multiple competing objectives.\n3. The author model a wide range of major factors affecting vehicular exhaust emissions, including temperature, road conditions, travel demand, making the traffic scenarios provided in IntersectionZoo more realistic and reliable.\n4. The dataset is of substantial volume, including one million scenarios simulated in 16,334 signalized intersections derived from 10 major US cities. This enable robust testing results of a given algorithm.", "weaknesses": "1. In this paper, the framework of IntersectionZoo looks sophisticated, but there lacks experimental results that illustrate how precise the simulation results are. Can they reflect the real-word vehicular exhaust emissions precisely?\n2. When benchmarking popular algorithms with IntersectionZoo, the authors show the performance of some algorithms from various aspects. However, what is the rationality of these results? How do they strengthen the contribution of the proposed dataset?\n3. The authors test the performance of some algorithms from various aspects with IntersectionZoo, but there lacks the testing results with other comparable benchmarks (like the ones in Table 1). How do the results on other benchmarks differ from the results on IntersectionZoo? Will the differences illustrate the advantage of IntersectionZoo?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)