IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learning

## Abstract

Abstract Despite the popularity of multi-agent reinforcement learning (RL) in simulated and two-player applications, its success in messy real-world applications has been limited. A key challenge lies in its generalizability across problem variations, a common necessity for many real-world problems. Contextual reinforcement learning (CRL) formalizes learning policies that generalize across problem variations. However, the lack of standardized benchmarks for multi-agent CRL has hindered progress in the field. Such benchmarks are desired to be based on real-world applications to naturally capture the many open challenges of real-world problems that affect generalization. To bridge this gap, we propose IntersectionZoo , a comprehensive benchmark suite for multi-agent CRL through the real-world application of cooperative eco-driving in urban road networks. The task of cooperative eco-driving is to control a fleet of vehicles to reduce fleet-level vehicular emissions. By grounding IntersectionZoo in a real-world application, we naturally capture real-world problem characteristics, such as partial observability and multiple competing objectives. IntersectionZoo is built on data-informed simulations of 16,334 signalized intersections derived from 10 major US cities, modeled in an open-source industry-grade microscopic traffic simulator. By modeling factors affecting vehicular exhaust emissions (e.g., temperature, road conditions, travel demand), IntersectionZoo provides one million data-driven traffic scenarios. Using these traffic scenarios, we benchmark popular multi-agent RL and human-like driving algorithms and demonstrate that the popular multi-agent RL algorithms struggle to generalize in CRL settings. Code and documentation are available at https://github.com/mit-wu-lab/IntersectionZoo/ .

## 1 Introduction

Having demonstrated impressive performance in simulated multi-agent applications such as Starcraft (Samvelyan et al., ), RL holds potential for various multi-agent real-world applications including autonomous driving (Kiran et al., ), robotic warehousing (Bahrpeyma & Reichelt, ), and traffic control (Wu et al., ). However, compared to simulated applications, the success of RL in real-world applications has been rather limited (Dulac-Arnold et al., ). A key challenge lies in making RL algorithms generalize across problem variations, such as when weather conditions change in autonomous driving. Problem variations are common in real-world applications but may not be designed to be explicitly assessed in simulated applications (Kirk et al., ).

Moreover, there are many open challenges that affect the generalization of RL algorithms in real-world multi-agent applications, such as the effect of complex multi-agent dynamics under aleatory uncertainty and partial observability of states and problem variations, optimizing multiple objectives over long horizons, and physical constraints. This makes the generalization in multi-agent RL a class of problems rather than one problem as it encapsulates many open challenges (Kirk et al., ).

From an algorithmic point of view, CRL formalizes learning policies that generalize across problem variations, aiming to learn robust, transferable, and adaptable policies (Benjamins et al., ). CRL aims to find a policy that can solve a set of (not too different) Markov Decision Processes (MDPs), each posing a problem variation and defined by a variation context vector drawn from a context distribution. This MDP set is referred to as a Contextual MDP (CMDP) (Hallak et al., ).

Thus, benchmarking CRL algorithms with real-world applications and the corresponding context distributions could move the needle on the successful use of RL for real-world applications by identifying the many open challenges that affect generalization and measuring progress. However, we observe such efforts are hindered by the lack of standardized multi-agent CRL benchmarks, especially those that are based on real-world applications and that model realistic context distributions. Simulated benchmarks often address only a subset of real-world challenges and lack realistic complexity. Benchmarks based on real-world applications but not actual context distributions still remain partially simulated, adopting a subset of the same limitations. While some multi-agent RL benchmarks can be improvised for CRL benchmarking, this only superficially addresses the underlying problem (Cobbe et al., ), lacks desired properties of a CRL benchmark (Kirk et al., ) and may lead to conflicting evaluation protocols (Whiteson et al., ; Jayawardana et al., ). Benchmarks that neglect these properties risk a ceiling effect and fail to incentivize further algorithmic improvements (Ellis et al., ; Yu et al., ; Hu et al., ).

Figure: Figure 1: Cooperative eco-driving at signalized intersections where the controlled vehicles (CVs) are operated by an RL policy (or policies) to minimize the fleet-wise emissions that include both CVs and human-driven vehicles (HDVs). CVs implicitly control HDVs through car-following dynamics and form locally cooperative teams for better system control.
Refer to caption: extracted/5939961/images/eco-drive-v5.png

To fill this gap, we present IntersectionZoo, a comprehensive multi-agent CRL benchmark suite based on the real-world application of cooperative eco-driving at signalized intersections (Figure 1). Cooperative eco-driving encodes several open challenges that affect generalization in multi-agent CRL, making it well suited for the purpose. IntersectionZoo is built on data-informed context distributions defined by 16,334 signalized intersections in 10 major cities across the United States. By identifying factors affecting vehicular emission (e.g., temperature), IntersectionZoo models nearly one million traffic MDPs on these intersections. Designed with input from domain experts, these MDPs are structured as CMDPs that interface with standard multi-agent interfaces in a highly configurable, fast-to-simulate open-source framework.

IntersectionZoo differs from existing CRL benchmarks on multiple fronts. Contrary to standard autonomous driving benchmarks centered on ego-vehicle control (Li et al., ), eco-driving tackles a mixed-motive multi-agent control problem. It is designed based on a well-studied real-world application with known factors that affect generalization (Mintsis et al., ; Chen et al., ). IntersectionZoo is built on ten CMDPs with data-driven context distributions and also provides the capability to procedurally generate CMDPs. The context distributions of CMDPs are comprehensive, covering variations in states, observations, rewards (with multiple objective terms), and transition dynamics. Last, IntersectionZoo by design supports both independent and identically distributed (IID) and out-of-distribution (OOD) evaluation protocols. We also interface IntersectionZoo with RLLib (Liang et al., ) for ease of benchmarking.

Broader Societal Impact: Limited familiarity among RL researchers with real-world physical systems has inhibited the development of RL algorithms addressing challenges in these domains. IntersectionZoo bridges this gap by decoupling the modeling complexity of real-world tasks from the experimental process, allowing researchers to focus on algorithmic advancements. This approach, in turn, holds the potential to improve eco-driving, which is known for its impact on climate change mitigation goals (Barkenbus, ), with the automotive industry actively pursuing robust eco-driving controllers. This aligns with the growing interest in application-driven research in machine learning, promising a mutually beneficial outcome for all communities involved (Rolnick et al., ).

Remark 1: With CRL, we focus on intra-task generalization, which means we train and test on environments stemming from the same task (e.g., eco-driving). A related parallel direction is inter-task generalization, training generally capable agents. In this work, we do not focus on that.

## 2 Preliminaries

### 2.1 Contextual Reinforcement Learning

CRL formalizes the notion of solving a collection of tasks, each stemming from the same problem (e.g., eco-driving). A collection of tasks is formalized using CMDPs (Hallak et al., ). We utilize the formalism from Ghosh et al. (). Accordingly, a CMDP $\mathcal{M}$ is an MDP with a state space of the form $s=(s^{\prime},c)$ where $s^{\prime}$ is the state and $c$ defines the context for the state $s^{\prime}$. The context $c$ is fixed within an episode (e.g., a fixed random seed). Given an initial context distribution $\rho(c)$, the initial state distribution of a CMDP is defined by $\rho(s)=\rho(c)\rho(s^{\prime}|c)$. Then, given a context $c$, the CMDP $\mathcal{M}$ is restricted to an MDP $\mathcal{M}_{c}$ and is called a context-MDP. In other words, the CMDP manifests as a collection of context-MDPs(^1^11Here, by MDP we generally refer to any form of MDPs such as Partially Observable MDPs, etc.). Note that context $c$ is not always visible to the agents (e.g., a random seed). Then, a CMDP becomes a Contextual Partially Observable MDP (CPOMDP) (Kirk et al., ).

We seek to find a policy $\pi^{*}$ that maximizes the overall expected return across all context-MDPs where $\mathcal{R}(\pi,\mathcal{M}_{c})$ is the expected return of policy $\pi$ on context-MDP $\mathcal{M}_{c}$,

$$ $\pi^{*}=\max_{\pi}\Bigl{[}\mathbb{E}_{c\sim\rho(c)}[\mathcal{R}\left(\pi,\left .\mathcal{M}_{c}\right)\right]\Bigl{]}.$ (1) $$

In multi-agent CRL, each context-MDP $\mathcal{M}_{c}$ manifests as a multi-agent control problem. Further, these multi-agent control problems are often partially observable. This is also the case with cooperative eco-driving. Then, we leverage a Decentralized Partial Observable MDP (Dec-POMDP) formulation (Bernstein et al., ) following previous work (Yan et al., ) to define each context-MDP.

### 2.2 Cooperative Mulit-agent Eco-driving

Optimizing eco-driving across a full-traffic network is ideal but is impractical in large cities like Los Angeles, with nearly 5000 signalized intersections, and remains an open optimization challenge (Qadri et al., ). A common approach is to decompose the network into individual intersections for separate optimizations (Yang et al., ; Jayawardana et al., ; Yang et al., ) while regulating intersection throughput to prevent traffic spill-back due to possible increased throughput. Therefore, it is often assumed that vehicle flow is not at saturation, allowing for reasonable throughput improvements. We adopt the same modeling assumption.

The default objective of cooperative eco-driving at individual signalized intersections is to minimize the total exhaust emissions of a fleet of vehicles (both CVs and HDVs) while having a minimal impact on individual travel time. At a given time $t$, the number of CVs is $k_{CV}^{t}$, and HDVs is $k_{HDV}^{t}$ such that $k_{CV}^{t}+k_{HDV}^{t}=n^{t}$ where $n^{t}$ is the total number of vehicles in the fleet. Then, we control the longitudinal accelerations of all CVs decentrally using a learned policy to optimize,

$$ $\min J=\sum_{i=1}^{n}\sum_{t=0}^{T_{i}}E\left(a_{i}(t),v_{i}(t)\right)+\lambda T _{i}.$ (2) $$

Here, $T_{i}$ denotes the travel time of vehicle $i$ and time $t$ is a discretized time with increments of $\delta$ (usually 0.5 seconds). The vehicular exhaust emission function is denoted by $E(\cdot)$, which takes speed $v_{i}(t)$ and acceleration $a_{i}(t)$ of vehicle $i$ at time $t$ and outputs a vehicular emission amount (usually the amount of carbon dioxide). $\lambda$ is the trade-off hyperparameter. We seek to optimize $J$ subject to hard constraints of ensuring vehicle safety, connectivity via vehicle-to-vehicle and vehicle-to-traffic signal communication, and soft constraints of vehicle kinematics, control realism, traffic safety at the fleet level (e.g., minimum time to collision across all vehicles), and passenger comfort.

What affects generalization in cooperative eco-driving: Eco-driving is a complex multi-agent control problem involving both CVs and uncontrolled HDVs where HDVs can introduce uncertainty in CV planning. Adding to the complexity is the required CV coordination. Based on the environment, there can be hundreds of vehicles at an intersection and each vehicle affects the flow of traffic. Further, generalization is naturally required across many factors — such as intersection topology, atmospheric conditions, eco-driving adoption level, and traffic flow rates — defining high-dimensional context distributions. Each CV control relies on a partially observed state of its surroundings and a partially observed context vector. Optimization objectives are multi-fold, with many competing objectives optimized over long horizons. Hard constraints such as vehicle safety have to be satisfied. These challenges individually and collectively shape the generalization capacity of CRL algorithms.

## 3 Related Work

In Table 1, we present our assessment of several known single-agent, multi-agent, and CRL benchmarks, focusing on eleven key properties desired for CRL benchmarking (Kirk et al., ; Cobbe et al., ; Benjamins et al., ). We aim to identify whether they satisfy the given properties or whether repurposing or improvising the current benchmark could be used to satisfy the properties.

Realistic task column assesses if a benchmark is based on real-world tasks. Many benchmarks focus on video games (Cobbe et al., ; Machado et al., ), strategy games (Wang et al., ), grid worlds (Chevalier-Boisvert et al., ), or simple control tasks (Benjamins et al., ), which lack real-world complexity. This limitation can lead to exploitable structures (Mohan et al., ) or hinder algorithmic advancements (Ellis et al., ; Yu et al., ; Hu et al., ). For instance, in the SMAC benchmark, a policy based only on the timestep can achieve notable win rates (Ellis et al., ). Some work involves more realistic robotics tasks (James et al., ; Yu et al., ), but they often use tightly constrained context features like limited friction levels.

**Table 1: Comparison of IntersectionZoo to related benchmarks. Benchmarks are rated based on whether they satisfy (✔), somewhat satisfy (⚫), or do not satisfy (✘) a given desired property. ‘Context’ is abbreviated as ‘ctxt.’ for formatting.**
| Benchmark | Realistic task | Multi-agent | Observed ctxt. features | Unobserved<br>ctxt. features | Initializable ctxt. dist. | Varying $\mathcal{S},\mathcal{O},\mathcal{T},\mathcal{R}$ | Multi-objective | IID testing | OOD testing | Data-driven ctxt. dist. | Native CRL support |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MDP Playground |  |  |  |  |  |  |  |  |  |  |  |
| (Rajan et al., ) | ✘ | ✘ | ✔ | ✔ | ✘ | ✔ | ✘ | ✘ | ✘ | ✘ | ✘ |
| bsuite  (Osband et al., ) | ✘ | ✘ | ✔ | ✘ | ⚫ | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ |
| Revisiting ALE |  |  |  |  |  |  |  |  |  |  |  |
| (Machado et al., ) | ✘ | ✘ | ⚫ | ⚫ | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ |
| CARL (Benjamins et al., ) | ✘ | ✘ | ✔ | ✘ | ✔ | ✔ | ✘ | ✔ | ✔ | ✘ | ✔ |
| Procgen (Cobbe et al., ) | ✘ | ✘ | ✘ | ✔ | ✘ | ⚫ | ✘ | ✔ | ✘ | ✘ | ✔ |
| Alchemy (Wang et al., ) | ✘ | ✘ | ✘ | ✔ | ✘ | ✔ | ✘ | ✔ | ⚫ | ✘ | ✔ |
| Meta-World (Yu et al., ) | ⚫ | ✘ | ✔ | ⚫ | ✔ | ✔ | ✔ | ✔ | ⚫ | ✘ | ✔ |
| RL Bench (James et al., ) | ⚫ | ✘ | ✔ | ✔ | ✔ | ✔ | ✘ | ✔ | ✔ | ✘ | ✔ |
| Minigrid |  |  |  |  |  |  |  |  |  |  |  |
| (Chevalier-Boisvert et al., ) | ✘ | ✘ | ✔ | ✔ | ✘ | ⚫ | ✘ | ✘ | ⚫ | ✘ | ✔ |
| NetHack (Küttler et al., ) | ✘ | ✘ | ✘ | ✔ | ✘ | ✔ | ⚫ | ✔ | ✔ | ✘ | ✔ |
| MiniHack (Samvelyan et al., ) | ✘ | ✘ | ✔ | ✔ | ✔ | ✔ | ⚫ | ✔ | ✔ | ✘ | ✔ |
| Particle Env (Lowe et al., ) | ✘ | ✔ | ⚫ | ✔ | ✘ | ✔ | ✘ | ✔ | ⚫ | ✘ | ✘ |
| SMACv2 (Samvelyan et al., ) |  |  |  |  |  |  |  |  |  |  |  |
| (Ellis et al., ) | ✘ | ✔ | ✔ | ✔ | ✔ | ⚫ | ✘ | ✔ | ⚫ | ✘ | ✔ |
| Google Football |  |  |  |  |  |  |  |  |  |  |  |
| (Kurach et al., ) | ✔ | ✔ | ✔ | ✔ | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ |
| Flow (Wu et al., ) | ⚫ | ✔ | ✔ | ✔ | ✔ | ✔ | ✘ | ✔ | ⚫ | ✘ | ✘ |
| Nocturne (Vinitsky et al., ) | ✔ | ✔ | ✔ | ⚫ | ✘ | ✔ | ⚫ | ✔ | ✔ | ✘ | ✘ |
| MetaDrive (Li et al., ) | ✔ | ✔ | ✔ | ✔ | ✔ | ⚫ | ✘ | ✔ | ✔ | ✘ | ✔ |
| SMARTS (Zhou et al., ) | ✔ | ✔ | ✔ | ⚫ | ⚫ | ✔ | ⚫ | ✔ | ⚫ | ✘ | ✘ |
| RESCO (Ault & Sharon, ) | ✔ | ✔ | ✔ | ✘ | ✘ | ⚫ | ⚫ | ⚫ | ⚫ | ✔ | ✘ |
| IntersectionZoo (ours) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |

Data-driven context distribution checks if benchmarks provide real-world context distributions, while native CRL support assesses if a benchmark is primarily designed to support CRL. Except for MetaDrive (Li et al., ) and SMACv2 (Ellis et al., ), most native CRL benchmarks focus on single-agent CRL. MetaDrive is a close second to IntersectionZoo but lacks data-driven context distributions for CMDPs, limiting its ability to capture real-world complexity. Moreover, the initializable context distributions column checks if a benchmark facilitates initializing arbitrary user-defined context distributions. IntersectionZoo facilitates using both additional real-world context distributions (e.g., intersections of other cities) as well as procedurally generating contexts based on user-specified context feature distributions.

Typically, there are two types of features defining the context distribution and thereby describing the context of a context-MDP: observed context features and unobserved context features. The key difference is whether these features are explicitly visible to the agent. Observed features are directly visible factors of variations, such as lane length in eco-driving. Unobserved features are mostly used for random variations such as those that arise from procedural content generation (PCG) (Cobbe et al., ) and are not explicitly visible to the agent (e.g., HDV aggressiveness). Both observed and unobserved features are desired for CRL benchmarking (Kirk et al., ). This enables systematic targeted evaluations of generalization such as systematicity (generalization using systematic recombination of known knowledge) and productivity (generalization beyond seen training data) (Hupkes et al., ) with non-trivial tasks.

Varying $\mathcal{S},\mathcal{O},\mathcal{T},\mathcal{R}$ refers to variations in states ($\mathcal{S}$), observations ($\mathcal{O}$) , transitions ($\mathcal{T}$) , and rewards ($\mathcal{R}$). In IntersectionZoo, the variations in environments stem from states (e.g., single vs. multiple lane driving), observations (e.g., diverse vehicle sensor capabilities), rewards (e.g., the trade-off between travel time and emission reduction), and dynamics (e.g., vehicle behavior changes due to varying traffic signal timings). Multiple forms of variations enable diversity and provide more avenues for targeted evaluations of generalizations (Li et al., ). While benchmarks like CARL (Benjamins et al., ) and MDP Playground (Rajan et al., ) are categorized to consist of all forms of variations, they lack evaluation protocols spanning all these variations or consist of multiple tasks without including all variations in each task.

Multiple objectives can bring another form of variation in CMDPs and are common in real-world problems. It thus manifests as another axis of targeted evaluation of generalization. While existing multi-objective RL benchmarks overlook generalization challenges (Felten et al., ), addressing this gap is acknowledged as needed (Kirk et al., ). Nocturne (Vinitsky et al., ), Flow (Wu et al., ) and SMAC (Ellis et al., ; Samvelyan et al., ) can be improvised for this purpose but lack native support. IntersectionZoo naturally frames a multi-objective optimization problem, prioritizing emission reduction with less impact on vehicle travel time. Further sub-objectives can incorporate passenger comfort, kinematic realism, and traffic level safety (Appendix A.4).

Figure: Figure 2: A schematic overview of IntersectionZoo divided into three architectural layers.
Refer to caption: extracted/5939961/images/greenwave_architecture_v4.png

IID and OOD testing are commonly used as evaluation protocols of generalization. IID evaluation assumes training and testing context sets are drawn from the same distribution. Benchmarks such as Procgen (Cobbe et al., ) or NetHack (Küttler et al., ) employ this scheme, where the training seeds can be sampled uniformly at random from the full distribution, and the full distribution can be used for testing. OOD evaluation does not assume the same distribution for training and testing and requires domain generalization (Wang et al., ). IntersectionZoo models 10 different CMDPs based on 10 US cities, each with a different intersection distribution. Hence, OOD evaluation can be performed by training on one CMDP and testing on another. Similarly, IID testing can be performed by train/test split within a CMDP.

## 4 IntersectionZoo

Designing a CRL benchmark suite based on a real-world task is challenging. Identifying the factors of variations requires domain knowledge and expert opinion, and data-driven modeling is required to model them realistically without artificially making the problem easy or hard. For example, it is known that eco-driving in short lanes is hard (Jayawardana et al., ), and hence, a context distribution that may have superficially many short lanes could make the problem harder than it is. In the following sections, we provide details of how we construct IntersectionZoo with the help of domain experts. We refer the reader to Appendix A.9 for more detials on release notes of IntersectionZoo including license details.

### 4.1 Defining IntersectionZoo scenarios for constructing CMDPs

Figure: Figure 3: Left: The signalized intersection in the intersection of Bosworth Street and Diamond Street in Salt Lake City, Utah. Right: The reconstructed intersection in simulation.
Refer to caption: extracted/5939961/images/intersection_recreation.png

In IntersectionZoo eco-driving CMDPs are formulated as a collection of traffic context-MDPs. Each traffic scenario is a basis for a context-MDP. A traffic scenario manifests as a combination of a set of eco-driving factor values that have a known effect on emission benefits at signalized intersections. These factors are related to intersection topology, human driver behavior, vehicle characteristics, traffic flow, and atmospheric conditions.

Concretely, an intersection is first defined by factors such as lane lengths, lane counts, road grades, turn lane configuration, and speed limit of each approach. Then, vehicle type, age, and fuel type distributions are used with appropriate traffic flow rates and HDV behaviors to define a realistic traffic flow. Each intersection scenario is further assigned representative atmospheric temperature and humidity values based on the season. Further scenario variations can be achieved by changing the eco-driving adoption level (0%-100%). We follow this procedure for every intersection, and the resultant traffic scenarios are the basis for the context-MDPs of each city.

Figure 2 illustrates the overall process: the factors we consider and how they are leveraged to build intersections, traffic scenarios, and, subsequently, CMDPs. In the traffic scenario modeling layer, we first build data-driven simulation environments of signalized intersections. We use Open Street Maps (OSM) (Haklay & Weber, ) data and follow guidelines provided by Qu et al. (). Intersection lane lengths, lane counts, turn lane configurations, and speed limits are extracted from OSM. Road grades are taken from US geological surveys (). An example instance of a reconstructed intersection is given in Figure 3. More details are given in Appendix A.1 regarding all ten cities and the feature distributions.

To model the vehicle arrival process, we use the Annual Average Daily Traffic data (AADT) (Huntsinger, ) released by the Departments of Transportation of each state/city. To differentiate the inflows between peak and off-peak hours, we use recommended conversion rates (Precisiontraffic, ). To model realistic vehicle arrival processes to intersections, we also model one-hop nearby intersections as illustrated in Figure 4.

For traffic signal timing, we exhaustively search through the fixed-time traffic signal timing plans (Thunig et al., ) and find the optimal plan. We source vehicle age, fuel type, and vehicle type distributions from the openly available MOVES databases () and data from US National Centers for Environmental Information () is used for atmospheric condition modeling with temperature and humidity changes.

To model HDV behavior, we use the Intelligent Driver Model (IDM) (Treiber et al., ) and calibrate it using the guidelines from Zhang & Sun () with real-world arterial driving data from CitySim (Zheng et al., ) (See Appendix A.2 for details). While IDM controls the longitudinal acceleration of HDVs, lane changes are performed using standard rule-based lane changing model (Erdmann, ).

Figure: Figure 4: For each intersection, default nearby intersections are added for realistic vehicle arrival processes subjected to nearby traffic signals.
Refer to caption: extracted/5939961/images/ghost-intersections-v2.png

In the CMDP modeling layer, we use the modeled traffic scenarios and organize them based on the city to create 10 CMDPs. All traffic scenarios are configured for use in the open-source agent-based traffic simulator SUMO (Lopez et al., ). A key requirement for capturing the effect of traffic scenarios on vehicle exhaust emission is a rich emission function. For this prupose, we integrate a suite of comprehensive and fast neural emission models (Sanchez et al., ) that replicate the industry-standard Motor Vehicle Emission Simulator (MOVES) (). Finally, the user configuration layer exposes the CMDPs that can be used by researchers for training CRL algorithms. We provide RLLib-based training by default and also enable flexibility with custom implementations.

A note on simulation realism: Autonomous driving benchmarks often prioritize the realistic rendering of traffic scenarios, aiming to train closed-loop policies from sensory inputs like perception inputs to navigate the ego-vehicle. However, in cooperative eco-driving, we use a 2D simulator that generates vectorized representations of vehicle states, isolating the challenge of generalization in CRL over feature extraction. Given that CRL often requires many samples during training, fast simulations are also an essential requirement that we achieve through simplified rendering.

### 4.2 Defining context-MDPs

Here, we provide an overview of the context-MDP definition we use in IntersectionZoo and refer the reader to Appendix A.3 for more specific details. In cooperative eco-driving, each traffic scenario manifests as a partially observable multi-agent control problem. Then, we leverage Decentralized Partial Observable MDP (Dec-POMDP) formulation (Bernstein et al., ) following previous work (Yan et al., ) to define each context-MDP. For each CV, state, action, and reward are defined as follows.

State Space: The design of the observed state of a vehicle is mainly based on the capabilities of existing sensor technologies. The observed state of a CV includes its own status, status of neighboring vehicles (leader and follower on all immediate nearby lanes), and status of the immediate traffic signal timing. Further, we provide a selected set of observed features based on the feasibility of obtaining them in the real world by the CV as observed context of the underlying environment.

Action Space: Longitudinal acceleration of each CV. Standard rule-based controller is used (Erdmann, ) for lane changing, focusing IntersectionZoo on the continuous control aspect of eco-driving.

Multi-objective Reward Function: The reward $r_{i}^{t}$ for each CV $i$ at time $t$ is defined in Equation 8 inspired by Jayawardana et al. (). Here, $n_{t}$ is the vehicle fleet size, $v_{t}^{i}$ is the velocity, and $e_{t}^{i}$ is the CO2 emissions of vehicle $i$ at time $t$. Hyperparameters include, $\eta=[0,1]$, $\alpha$, $\beta$, and $\tau$. The indicator function $\mathbbm{1}_{v^{i}_{t}<\tau}$ indicates whether the vehicle is stopped, while the term $e_{t}^{i}$ encourages low emissions. The velocity term captures the effect on travel time. Users can configure the parameter $\eta$ to either get a fleet-based reward, agent-based reward, or a combination of both. All such formulations are acceptable.

$$ $r_{t}^{i}=\eta\frac{1}{n_{t}}\sum_{j=1}^{n_{t}}(v_{t}^{j}+\alpha\mathbbm{1}_{v ^{j}_{t}<\tau}+\beta e_{t}^{j})+(1-\eta)(v_{t}^{i}+\alpha\mathbbm{1}_{v^{i}_{t }<\tau}+\beta e_{t}^{i}).$ (3) $$

Appendix A.4 details additional sub-objectives in IntersectionZoo to enhance fleet-level traffic safety, passenger comfort, and kinematic realism. IntersectionZoo ensures vehicle safety and actuator limits via pre-defined rule-based checks when used with the default objective.

## 5 Evaluations in IntersectionZoo

### 5.1 Evaluation protocols

By default, IntersectionZoo provide interfaces for train/test split evaluations to measure generalization, which is often used with zero-shot policy transfer (Harrison et al., ; Higgins et al., ; Kirk et al., ). This means we train policies on one subset of context MDPs and test on another subset of context MDPs. This includes both IID and OOD evaluation protocols. Hence, OOD evaluation can be performed by training in one city (train CMDP) and testing in another city (test CMDP). Similarly, IID testing can be performed by train/test split of context-MDPs within a given city.

### 5.2 Evaluation metrics

The performance of a given CRL policy trained to eco-drive is often benchmarked against human-like driving baselines (how much improvement can be obtained from driving differently than humans do) (Jayawardana & Wu, ; Jayawardana et al., ). IntersectionZoo provides calibrated Intelligent Driver Models using real-world human driving data (Appendix A.2) as human driving baseline for this purpose. We assess the performance of a policy using the average vehicular exhaust emission per vehicle and the intersection throughput of each intersection. Refer to Appendix A.5 for more details on the definitions of these metrics.

A key requirement for assessing the performance is to ensure intersection throughput per intersection is never reduced by the learned eco-driving policy as explained in Section 2.2. If, for any intersection, the throughput is reduced (even if there are low emissions), its emission benefits are set to zero.

## 6 IntersectionZoo Benchmarking

We benchmark popular RL algorithms used in multi-agent control to assess their generalization capacity using IntersectionZoo CMDPs. Considering their success in many cooperative multi-agent problems (Yu et al., ), we employ PPO (Schulman et al., ), DDPG (Silver et al., ), multi-agent PPO (MAPPO) with a centralized critic (Yu et al., ), and graph convolution networks (GCRL) for explicit cooperation modeling in multi-agent RL (Jiang et al., ). In assessing the performance of the algorithms, we assess their generalization capacity along three axes (three forms of targeted evaluations of generalizations) that are often considered as desired generalization properties. In measuring performance, we illustrate the emission benefits at the level of intersection incoming approaches instead of the whole intersection for a better understanding and visualization of the benefit distributions.

Figure: (a) Benefit histogram of SLC with PPO policy
Refer to caption: extracted/5939961/images/results/slc/slc_ppo.png

Figure: (a) Benefit histogram of Atlanta with PPO policy
Refer to caption: extracted/5939961/images/results/atlanta/alt-ppo.png

In-distribution performance: First, we assess how well these algorithms can generalize when the training and testing CMDPs are the same. For this, we leverage Salt Lake City (SLC) CMDP (282 intersections) and Atlanta CMDP (621 intersections) under summer temperature and humidity, with 1/3rd of vehicles being CVs. Following the IID evaluation protocol, we train and evaluate on the same CMDP. The emission benefit histograms are given in Figure 5 for SLC and in Figure 6 for Atlanta.

As evident, all four algorithms fail to generalize, having many failure cases. Failures include having higher emissions and/or lower throughput than the human-driving baseline (0% benefits case). While DDPG has a better success rate than the other three algorithms, it is still far from being successful. Further, generalization performance in Atlanta is different than in SLC for each algorithm, indicating that different CMDPs may pose different challenges in generalization. Given all four algorithms struggle to generalize, for the next set of experiments, we use only PPO and DDPG, which show relatively better generalization than the MAPPO and GCRL.

Systematicity in generalization: Systematicity is generalization using systematic recombination of known knowledge (Kirk et al., ; Hupkes et al., ). To test this ability of DDPG and PPO, we leverage IntersectionZoo’s capability to procedurally generate context-MDPs. Following Kirk et al. (), we first define a set of context features and their corresponding values as a set of uniform distributions (per feature). Then we train policies by sampling feature values from each distribution to create context vectors. However, certain feature value combinations are never used during training. During testing, we only use the feature value combinations that were not used in training. This tests the algorithms’ ability to systematically combine known knowledge to generalize. The resultant performance histogram is given in Figure 7(a). Both DDPG and PPO fail to systematically generalize; baseline performs better in almost all cases. More details are given in Appendix A.8.

Productivity in generalization: Productivity is when the learned policies generalization beyond seen training data (Kirk et al., ; Hupkes et al., ). To test this in PPO and DDPG, we perform an OOD evaluation by using a policy trained on SLC CMDP with zero-shot transfer to Atlanta CMDP. The resultant performance histogram is given in Figure 7(b). Note that from the in-distribution performance analysis, it is visible that the DDPG policy trained on SLC CMDP seems to perform slightly better than the policy trained on Atlanta CMDP. However, even after the transfer, both DDPG and PPO seem to perform poorly, further indicating the limitations of existing RL algorithms when it comes to generalization across problem variations.

Figure: (a) Systematicity evaluation using procedurally generated intersections
Refer to caption: extracted/5939961/images/results/pcg_pcg.png

## 7 Conclusion

In this work, we propose IntersectionZoo, a comprehensive multi-agent CRL benchmark suite based on the real-world application of cooperative eco-driving. Using IntersectionZoo, we benchmark popular multi-agent RL and human-like driving algorithms and demonstrate that the popular multi-agent RL algorithms struggle to generalize in CRL settings. Specifically, we show popular multi-agent RL algorithms perform poorly when tested for in-distribution generalization, systematicity in generalization, and productivity in generalization with real-world data-driven eco-driving. A current limitation of the benchmark is it can be primarily used to benchmark only continuous control algorithms. However, IntersectionZoo provides discrete lane-changing control for interested users. Further, despite our best efforts to create realistic traffic scenarios, the provided scenarios may have variations from their real-world counterparts due to inevitable data errors and missing data. Overall, IntersectionZoo aims to advance generalization in multi-agent RL research by providing a rich benchmark suite that naturally captures many of the real-world problem characteristics that affect generalization.

## 8 Acknowledgments

The authors thank Blaine Lenoard, Mark Taylor, Michael Sheffield, Christopher Siavrakas, and Kelly Njord at Utah Department of Transportation for their constructive feedback on eco-driving scenarios. The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing computational resources to conduct experiments reported in this paper.

## Appendix A Appendix

### A.1 Details of IntersectionZoo CMDP context distributions

IntersectionZoo provides 10 CMDPs based on 10 major metropolitan cities across the United States. In Table 2, we list the ten cities we consider and the number of intersections considered for building these CMDPs in each city.

**Table 2: Intersection count distribution within the ten cities.**
| City | Intersection Count |
| --- | --- |
| Atlanta | 621 |
| Boston | 917 |
| Chicago | 2864 |
| Dallas Fort Worth | 1570 |
| Houston | 1014 |
| Los Angeles | 4276 |
| Salt Lake City | 282 |
| New York Manhattan | 2586 |
| San Francisco | 1209 |
| Seattle | 995 |
| Total | 16,334 |

In Figure 8, we provide a comparison of feature distributions of the incoming approaches of intersections of the ten cities. The y-axis is the percentage, while each x-axis is the respective feature range. Lane length is measured in meters, speed limit is in meters per second, vehicle inflows are vehicles per hour, and road grade is in percent grade. Phase count denotes how many different traffic signal phases are applicable for a given incoming approach. The signal time ratio denotes the ratio between approach-related phase times and total cycle time.

Figure: Figure 8: Comparison of context feature distributions of the incoming approaches of the ten cities. The y-axis is the percentage, while each x-axis is the respective feature range.
Refer to caption: extracted/5939961/images/city_features_summary.png

Based on our traffic scenario modeling efforts, IntersectionZoo provides more than one million traffic scenarios. As a high-level breakdown, for the all 16,334 intersections, we enable modeling four seasons as changes in temperature and humidity, four main eco-driving adoption levels (10%, 20%, 50%, 75% and 100%), two traffic hours (peak and off-peak), two vehicle engine technologies (electric vs internal combustion engine) summing up to more than 1 million traffic scenarios.

### A.2 Modeling human-driven vehicles

For the foreseeable future, human-driven vehicles will remain prevalent. To model human drivers in our simulations, we use the Intelligent Driver Model (IDM) (Treiber et al., ). IDM is a widely accepted car-following model that can produce realistic traffic waves. IDM calculates a vehicle’s acceleration using Equation 4, with desired velocity $v_{0}$, space headway $s_{0}$, time headway $T$, maximum acceleration $\alpha$, and comfortable braking deceleration $\beta$. The velocity difference with the leading vehicle is denoted as $\Delta v(t)$, and $\delta$ is a constant.

$$ $a(t)=\alpha\left[1-\left(\frac{v(t)}{v_{0}}\right)^{\delta}-\left(\frac{s^{*}( v(t),\Delta v(t)}{s(t)}\right)^{2}\right]$ (4) $$

$$ $s^{*}(v(t),\Delta v(t))=s_{0}+\max\left(0,v(t)T+\frac{v(t)\Delta v(t)}{2\sqrt{ \alpha\beta}}\right)$ (5) $$

For simulation accuracy, precise calibration of parameters $v_{0}$, $s_{0}$, $T$, $\alpha$, and $\beta$ is crucial. Different regions may exhibit varying driving behaviors, such as American drivers versus British drivers. Thus, calibrating the IDM parameters is critical for accurate human driver modeling.

Our IDM model calibration goals are threefold. We aim to align it with real-world human driving behavior at US signalized intersections, tailoring the five IDM parameters for human-like trajectories. We also need separate IDM models for distinct vehicle types (e.g., cars, buses, trucks) due to their influence on driving style. Lastly, we aim to establish a range of IDM models reflecting diverse driver behaviors and aggressiveness as observed in real-world driving.

For this purpose, we employ Bayesian inference and Gaussian process-based approach proposed by Zhang & Sun () and leverage real-world arterial driving data from CitySim (Zheng et al., ). For $\forall(t,d)\in\{(t,d)\}_{t=t_{0},d=1}^{t_{0}+(T-1)\Delta t,D}$ where $d$ represents the index for each driver $d\in\{1,\cdots,D\}$ and $t$ represents the timestamp, we have,

$$ $\displaystyle\ln({\theta})\sim\mathcal{N}\left({\mu}_{0},{\Sigma}_{0}\right) \in\mathbb{R}^{5}$ (6a) $\displaystyle\ln\left(\sigma_{\epsilon}\right)\sim\mathcal{N}\left(\mu_{ \epsilon},\sigma_{1}\right)\in\mathbb{R}$ (6b) $\displaystyle v_{d}^{(t+\Delta t)}\mid{i}_{d}^{(t)},{\theta}\stackrel{{ \scriptstyle\text{i.i.d.}}}{{\sim}}\mathcal{N}\left(\mathcal{F}_{\mathrm{IDM}} \left({i}_{d}^{(t)};{\theta}\right),\left(\sigma_{\epsilon}\Delta t\right)^{2} \right)\in\mathbb{R}$ (6c) $$

Here, ${\theta}=\left[v_{0},s_{0},T,\alpha,\beta\right]\in\mathbb{R}^{5}$ is the IDM model parameters where $\mathcal{N}(\mu,\sigma)$ represents a Gaussian distribution with a mean of $\mu$ and standard deviation of $\sigma$. Independent and identically distributed is indicated by $i.i.d$.
Furthemore, we denote the inputs at time $t$ as a vector $i_{d}^{(t)}=[s^{(t)}_{d},v^{(t)}_{d},\Delta v^{(t)}_{d}],\forall t\in\{t_{0},
\cdots,t_{0}+(T-1)\Delta t\}$ where $s^{(t)}_{d}$ is the headway, $v^{(t)}_{d}$ is the velocity and $\Delta v^{(t)}_{d}$ is the relative velocity of driver $d$ and its leading vehicle at time $t$. We further define a function $\mathcal{F}_{IDM}(\cdot)$ that updates the ego vehicle’s speed at $t+\Delta t$ using Equation 4 and Equation 7 where $\Delta t$ is the step size.

$$ $v^{(t+\Delta t)}=v^{(t)}+a^{(t)}\Delta t$ (7) $$

The formulation in Equations 6 generates a population-level joint distribution of IDM parameters. As vehicle type influences driving behavior, we build distinct joint distributions for each vehicle type. This requires separate human-driving trajectory datasets for every vehicle type.

CitySim (Zheng et al., ) is a drone-based vehicle trajectory dataset that provides vehicle trajectories along arterial roads, but it lacks explicit vehicle type labels. Therefore, we use the bounding box length of vehicles to distinguish cars from buses and trucks. While this identifies cars, it does not allow us to differentiate between trucks and buses. Thus, we assume a common IDM parameter distribution for trucks and buses. This is reasonable as both are heavy-weight vehicles with similar behavior at signalized intersections. After modeling these joint distributions, we utilize them to sample human drivers for the micro-simulations.

### A.3 Defining Context MDPs

State Space: The design of the observed state of a vehicle is mainly based on the capabilities of existing sensor technologies. The observed state includes the speed of the ego-vehicle, relative distance to the traffic signal, traffic signal state (red, green, or yellow) for the current phase, time remaining in the current phase, time remaining until the traffic signal turns green for the second and third cycle, vehicle location (i.e., a flag indicating whether the vehicle is approaching the intersection, at the intersection, or exiting the intersection), index of the vehicle’s current lane, vehicle’s intention to turn right, left, or go straight at the upcoming intersection, and for the follower and the leader vehicles on the same lane, adjacent right lane, and left lane of the ego-vehicle: speed, relative distance, turn signals status (turning right, left, or none).

As mentioned earlier, IntersectionZoo has both observed features and unobserved PCG-based features defining the context of a context-MDP. For users interested in conditioning the policies based on the context, we provide observable context features that include eco-driving adoption level, signal timing plan for the traffic signal phase relevant to the vehicle, atmospheric conditions such as temperature and humidity, the fuel type (electric or internal combustion engine), and information about the ego-vehicle’s current approach (number of lanes, lane length, speed limit). The decision on which features are available for conditioning is also based on the feasibility of implementing them in the real world.

Action Space: Longitudinal acceleration of each CV. For lane changing, a standard rule-based controller is used (Erdmann, ). This focuses IntersectionZoo on the continuous control aspect of eco-driving.

Multi-objective Reward Function: The reward $r_{i}^{t}$ for each CV $i$ at time $t$ is defined in Equation 8 inspired by Jayawardana et al. (). Here, $n_{t}$ is the vehicle fleet size, $v_{t}^{i}$ is the velocity, and $e_{t}^{i}$ is the CO2 emissions of vehicle $i$ at time $t$. Hyperparameters include, $\eta$, $\alpha$, $\beta$, and $\tau$. The indicator function $\mathbbm{1}_{v^{i}_{t}<\tau}$ indicates whether the vehicle is stopped, while the term $e_{t}^{i}$ encourages low emissions. The velocity term captures the effect on travel time. Users can configure the parameter $\eta$ to either get a fleet-based reward, agent-based reward, or a combination of both. All such formulations are acceptable.

$$ $r_{t}^{i}=\eta\frac{1}{n_{t}}\sum_{j=1}^{n_{t}}(v_{t}^{j}+\alpha\mathbbm{1}_{v ^{j}_{t}<\tau}+\beta e_{t}^{j})+(1-\eta)(v_{t}^{i}+\alpha\mathbbm{1}_{v^{i}_{t }<\tau}+\beta e_{t}^{i})$ (8) $$

Users can extend the above default objective with additional reward terms as explained in Section A.4.

### A.4 Additional objectives terms and their encoding schemes

IntersectionZoo provides additional objective terms for users who wish to assess the effect of multiple objectives on generalization.

Passenger comfort: To accommodate passenger comfort, vehicles should maintain low accelerations and decelerations. To encourage this behavior, a reward term is defined as the $|a_{t}|$ where $a_{t}$ is the acceleration (or deceleration) of the vehicle at time $t$. When used with shared fleet-wise reward, the mean of $|a_{t}|$ across all vehicles is used.

Kinematic realism: Vehicles often cannot have high jerks (changes in accelerations in unit time) as actuators have jerk limits. To account for this, IntersectionZoo provides jerk control as $|a_{t}-a_{t-1}|$ where $a_{t}$ is the acceleration (or deceleration) of the vehicle at time $t$. When used with shared fleet-wise reward, the mean jerk across all vehicles is used.

Fleet-level safety: While individual vehicle safety is ensured using pre-defined rule-based checks, IntersectionZoo provide surrogate safety measures such as Time To Collision (TTC) to improve traffic flow level safety. These surrogate safety measures are commonly used by traffic engineers to measure the impact of new roadway interventions (Wang et al., ).

Time to Collision (TTC) for a vehicle is measured as the time it would take for the vehicle to collide if it were to continue moving along its current paths without any changes in speed or direction. Formally, $TTC=\frac{\Delta d}{\Delta v}$ where $\Delta d$ is the relative distance and $\Delta v$ is the relative velocity. Both distance and velocity are measured relative to the leading vehicle of the ego-vehicle. In using TTC for fleet-level safety, we take the minimum TTC value across all vehicles at a given time step and share it with all vehicles.

### A.5 Evalution metrics

To measure the generalization performance of a learned CRL policy, we leverage two metrics.

Average emission benefits: This measures the per vehicle per time step $CO_{2}$ emissions in grams based on the CRL policy and compares that with the human driving baseline. The lower the emissions, the better. Percentage emission reduction is given as the benefit.

Average intersection throughput benefits: This measures the number of vehicles that cross the intersection during an episode. We present it as the average throughput change percentage compared to the human driving baseline.

### A.6 Experimental setup

All experiments are carried out using RLLib (Liang et al., ) with the default hyperparameter configuration. All policies are trained as multi-task learning policies where the context to condition the policy is as defined in Section A.3. We leverage 10 multiple workers in training the multi-task learning policies. Experiments were carried out in a computing cluster with 20 CPUs and an NVidia Volta V100 GPU with 32GB RAM. Each benchmarking run took roughly 24 hours in RLLib, with 5000 episodes (each with a horizon of 1000 steps with 50 warmups). We purposely ran each run for large number of iterations to ensure no further training can improve the policies. Benchmarking runs can be run for a shorter number of iterations, reducing computation times further.

### A.7 Additional results

For the reported results in Section 6, for each algorithm, we train with four random seeds. We train for 500 training iterations to ensure policies are well-converged. During the evaluation, we select the best-performing policy based on the rewards, vehicle throughout, and emission reductions.

### A.8 Systematicity context distribution

For measuring the systematicity in Section 6, we define the Table 3 feature distributions for training and Table 4 defined feature distribution for evaluation. During training we sample intersections from Table 3 defined distribution but also not in Table 4 defined distribution. During evaluations, we only sample intersections from Table 4 defined distribution.

**Table 3: Feature distribution for evaluation of systematicity**
| Feature | Value range |
| --- | --- |
| Lane setup | (1,1), (1,2), (2,1), (2,2), (3,1), (3,2) - Format: (lane count, phase count) |
| Vehicle inflow | [100, 600] vehicles per hour |
| Green phase time | [20, 32] seconds |
| Red phase time | [20, 32] seconds |
| Lane length | [100, 775] meters |
| Speed limit | [16, 20] m/s |
| Signal offset | [1,3] seconds |

**Table 4: Feature distribution for evaluation of systematicity**
| Feature | Value range |
| --- | --- |
| Lane setup | (2,2), (3,1) - Format: (lane count, phase count) |
| Vehicle inflow | [400, 500] vehicles per hour |
| Green phase time | [26, 29] seconds |
| Red phase time | [26, 29] seconds |
| Lane length | [325, 500] meters |
| Speed limit | [17, 18] m/s |
| Signal offset | [2, 3] seconds |

### A.9 License details and accessibility

Our code and the IntersectionZoo are released under the MIT License. The code, the intersection datasets of CMDPs, and the full documentation on how to use IntersectionZoo are available in the Github repository [https://github.com/mit-wu-lab/IntersectionZoo/](https://github.com/mit-wu-lab/IntersectionZoo/). The intersection datasets are also released under the MIT License. All data used for creating traffic scenarios are based on publicly available open data. SUMO traffic simulator is licensed under the EPL-2.0 with GPL v2 or later as a secondary license option (refer to SUMO website for more details).

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. How would the temperature and humidity affect the policy behavior? For example, is it better to drive at low speed in summer to reduce the CO2 emission? But it would reduce the throughput right?
2. As far as I know, the OSM data only provides the location of the traffic light and doesn't tell which lane it controlls. How does this information be completed in IntersectionZoo?
3. Type: Appendix, table 3. evaluation -> training
soundness: 3
strengths: 1. It is a good application for MARL researchers to study how to build algorithms to further reduce exhaust emissions and make a real-world impact. 
2. The scenarios database is large. With realistic context distribution, IntersectionZoo is a good place to study the generalizability of algorithms. Also, the whole system including the intersection data, simulator, metric, and training framework is built reasonably and solidly.
3. As an open-sourced benchmark, the code is of good quality with documentation included. Seems it is ready for use.
summary: **Overview**  
This work proposes a new benchmark for testing the generalizability of MARL algorithms across different MDP settings. With the background of reducing $CO_2$ emission by controlling vehicle fleets, progress on this benchmark may potentially generate real-world impact and advance the development of MRL. 

**Method**  
Specifically, intersections were collected for 10 major cities in the US and reconstructed in the SUMO simulator with a set of configurable parameters (context) like vehicle throughput, speed limit, temperature, the ratio of autonomous cars, and so on. The parameter distribution is provided by referring to literature from the transportation department. Thus, sampling parameters from these distributions allow the simulator to build intersection scenarios similar to its real-world counterpart. Each parameter combination forms a special MDP, and thus training agents with scenarios sampled from the same distribution allows benchmarking in-distribution performance. Testing agents in scenarios built with another parameter distribution can reflect the algorithm's OOD performance.

**Experiment**   
The experiment shows that the current SOTA MARL method struggles to do even in-distribution generalization, indicating there is a large room for improvement.
weaknesses: 1. The writing is poor. The experiment figures are hard to understand. Specifically, what does the total count mean? Though I can roughly infer it, it would be better to explain it in the paper. For the benefit percentage, does 0.5 means 0.5% or 50%. If it is 0.5%, it seems the emission doesn't change a lot by switching to MARL control, and the key is that MARL agents can not achieve the same level of throughput as human drivers. However, it seems GCRL agent is close to human drivers' performance. I suggest giving more analysis and explanation to these figures. It can be done in the captions since I found the captions of Figure 5 and Figure 6 are simple copy-paste. Also, is it possible to provide more visualizations in the appendix or even provide a video? So readers can know how different the map topologies and vehicle distributions are for these cities. I can not imagine a large dataset from pure text.
2. More experiments or results are needed. Though the experiments show that MARL can not generalize well, we don't know if it is due to the algorithm itself or if the task is too difficult to learn for RL agents, i.e. improper reward function and observation or MDP definition. One way to clarify this is to provide training performance as well. It needs to be proved first that the MARL can overfit the training scenarios and exceed human driver performance. Then we can confirm that the simulation design is appropriate, so MARL agents have the chance to surpass humans, and the failure on the test set is indeed due to poor generalizability. Also, parameters are fixed for the experiments like the adoption rate, which is 1/3. Did the authors conduct experiments to see which factor affects the performance most? I believe the adoption rate is an important factor since the MARL performance would be largely influenced by the population size. Also, a baseline can be added to use the IDM vehicle control these CVs to see if the MARL can outperform this simple rule-based baseline at least at training time. With such a trend, we can conclude it shows some signs that MARL can achieve better performance, and thus work on this benchmark is promising.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Can the authors elaborate on how IntersectionZoo is expected to benefit the broader MA-RL community? What features of the cooperative eco-driving task are common to other MA-RL tasks? To what extent the insights obtained on IntersectionZoo can be extended to broader MA-RL settings? More specifically, could the authors suggest particular MA-RL domains where insights from IntersectionZoo are likely, or unlikely, to transfer? Providing both positive and negative examples would help the readers with assessing whether IntersectionZoo could serve as a relevant benchmark for their own purposes.

- Can the authors provide a more extensive experimental analysis of MA-RL algorithms on this benchmark? If no algorithm performs well on the benchmark, what are the underlying causes for such a performance gap? For example, IntersectionZoo focuses on two performance metrics: average emission benefits and average intersection throughput benefits. While these metrics are interesting from a domain-driven perspective, the paper would benefit from considering additional algorithm-driven metrics relating to, e.g., sample efficiency, stability of training, etc. Moreover, a qualitative visualization of the resulting policy would further improve the experimental assessment.
soundness: 2
strengths: I resonate with the overall purpose of this work: bridging the gap between RL research (focused on benchmarks and generality of algorithms) and impactful, real-world applications. The paper is generally well-written and straightforward to follow. The authors thoroughly compare their benchmark with a comprehensive body of literature in the field. Based on my understanding, this work entails significant data collection and processing, which could provide substantial benefits to the broader transportation research community.
summary: In this paper the authors introduce a new benchmark for multi-agent reinforcement learning (MARL) called IntersectionZoo, designed to address the cooperative eco-driving task within urban networks. The objective of cooperative eco-driving is to control a fleet of vehicles at signalized intersections to minimize fleet-wide emissions with minimal impact on individual travel times.

The authors discuss the benchmark in the context of Multi-Agent Contextual Reinforcement Learning (MA-CRL) and highlight how IntersectionZoo is purposefully tailored to generate multiple scenario variations (i.e., contexts) and evaluate the generalization capabilities of MA-CRL agents across such variations. Specifically, IntersectionZoo addresses a range of contextual factors, including intersection configuration, vehicle type distribution, fuel type distribution, human driver models, weather conditions, eco-driving adoption rates, peak and off-peak traffic patterns, among others. The benchmark leverages data across 10 cities and over 16,000 signalized intersections and is built on open-source software, e.g. SUMO for microscopic traffic modeling, and RLLib for RL training.
weaknesses: In my opinion, the primary limitation of this work is its relatively narrow contribution to the broader RL, or specifically multi-agent RL, community. Although the authors claim that IntersectionZoo aims to serve as a novel standardized benchmark for MA-CRL research, many of the benchmark’s design decisions and characteristics are highly tailored to cooperative eco-driving and to the specific problem formulation adopted within this work. While evaluating generalization performance across different scenario characteristics is important for this application, it seems questionable to assert that results and insights on IntersectionZoo would extend to MA-RL research more broadly. This, to me, somewhat undercuts the fundamental purpose of a benchmark.

For instance, while scenario factors vary, the overall application and formulation remain fixed. This impacts algorithmic choices as well, such as the mandatory choice of a continuous action space (acceleration of the controlled vehicles), a specific reward function design, limited environment scale (restricted to single intersections), predefined driver models, and no significant uncertainty sources. These are just a few examples.

To the best of my understanding, this work would have greater impact if it concentrated more on the application side—namely, contributing novel algorithms to address a critical problem—rather than positioning this benchmark to generate deep insights into MA-RL algorithm development. 

Furthermore, the experimental analyses in this work need _considerable_ improvement. The authors provide a brief comparison of PPO, DDPG, MAPPO, and GCRL, concluding that these algorithms fail to generalize and perform poorly across all tested conditions. While experimental evaluation may not be the core focus of this paper, a thorough and insightful assessment of established MA-RL algorithms is essential in order to evaluate the usefulness of a benchmark. Is it the case that no algorithm performs adequately in any of the examined scenarios? The authors should consider expanding the experimental section by exhaustively benchmarking popular RL algorithms on the proposed benchmark, with in-depth evaluation and analyses.

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 4
questions: What does the “baseline” refer to in “Both DDPG and PPO fail to systematically generalize; baseline performs better in almost all cases” in Line 497?
soundness: 3
strengths: The originality is good. This paper addresses the problem of contextual reinforcement learning by creating a comprehensive benchmark for eco-driving simulation.

The problem formulation is presented clearly and in detail. 

The literature review offers an extensive and in-depth overview of related work.

The proposed IntersectionZoo benchmark can be a boost for contexture RL research.

Open-source code is provided to support the research community.
summary: The paper proposes a benchmark suite, IntersectionZoo, for contextual reinforcement learning evaluation. The suite is designed based on a complex real-world task, i.e., cooperative eco-driving at signalized intersections. This paper well summarizes the recent work related to RL benchmark. Several experiments are conducted to test some popular multi-agent RL algorithms and show that all the adopted algorithms perform poorly on generalization. The open-source code is also provided to facilitate the community.
weaknesses: * The main content seems to exceed the page limit – the 8. The Reproducibility Statement is on the 11th page.

* PPO and DDPG are not specifically for multi-agent RL but are only used with independent agents. It seems not reasonable to choose these two algorithms for testing. Besides, for systematicity and productivity in generalization tests, the authors only use PPO and DDPG for evaluation, which is confusing.

* From your experiments, it’s hard to get the conclusion in Line 532 that “the popular multi- agent RL algorithms struggle to generalize in CRL settings” since only two MARL-based methods, i.e., MAPPO and GCRL, are adopted for evaluation, and just on the in-distribution generalization.

* Given the unsatisfactory results of the chosen multi-agent RL algorithms on this benchmark, it would be better to briefly review the latest multi-agent RL to justify why you chose PPO, DDPG, MAPPO, and GCRL for evaluation.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Could you please provide some experimental results to verify that IntersectionZoo can reflect the real-word vehicular exhaust emissions precisely?
2. Could you please explain the rationality of your benchmarking results? How do they strengthen the contribution of the proposed dataset?
3. Could you please provide some major testing results of the popular algorithms on other existing benchmarks? How do the results on other comparable benchmarks differ from the results on IntersectionZoo? Will the differences illustrate the advantage of IntersectionZoo?
soundness: 3
strengths: 1. This paper focus on benchmarking the task of cooperative Eco-driving. The idea of this task is to control a fleet of vehicles to reduce fleet-level vehicular emissions. The task sounds interesting and is absolutely beneficial to reduce air pollution in the era of global climate change.
2. The authors ground IntersectionZoo in real-world applications, and capture real-world problem characteristics that are neglected in previous datasets, including partial observability, and multiple competing objectives.
3. The author model a wide range of major factors affecting vehicular exhaust emissions, including temperature, road conditions, travel demand, making the traffic scenarios provided in IntersectionZoo more realistic and reliable.
4. The dataset is of substantial volume, including one million scenarios simulated in 16,334 signalized intersections derived from 10 major US cities. This enable robust testing results of a given algorithm.
summary: This paper proposes IntersectionZoo, a comprehensive benchmark suite for multi-agent CRL through the real-world application of cooperative eco-driving in urban road networks. The authors ground IntersectionZoo in real-world applications, and capture real-world problem characteristics that are neglected in previous datasets, including partial observability, and multiple competing objectives. Also, IntersectionZoo covers 16,334 signalized intersections derived from 10 major US cities, enabling the testing of policy transfer between different cities. Using traffic scenarios in the proposed dataset, the authors benchmark various popular multi-agent RL and human-like driving algorithms and demonstrate their performance in Eco-driving.
weaknesses: 1. In this paper, the framework of IntersectionZoo looks sophisticated, but there lacks experimental results that illustrate how precise the simulation results are. Can they reflect the real-word vehicular exhaust emissions precisely?
2. When benchmarking popular algorithms with IntersectionZoo, the authors show the performance of some algorithms from various aspects. However, what is the rationality of these results? How do they strengthen the contribution of the proposed dataset?
3. The authors test the performance of some algorithms from various aspects with IntersectionZoo, but there lacks the testing results with other comparable benchmarks (like the ones in Table 1). How do the results on other benchmarks differ from the results on IntersectionZoo? Will the differences illustrate the advantage of IntersectionZoo?

Meta-review: The paper presents a benchmark for MARL via fleet (CO2) control. 

The primary strength of this paper is the extent to which it builds on real-world data and captures important features of real world deployment. 

The reviewers noted several weaknesses. First, the reviewers rightly question whether this benchmark could be extended to a broader class of problems. There were also several questions around poor/lacking analysis of existing methods, but that has largely been addressed to the satisfaction of the reviewers. 

Reviewers all recommend acceptance, and the major weaknesses have been discussed and largely addressed by the authors. Benchmarking work is often severely underrated by the community and this work involves significant data collection and simulation preparation, and so I lean toward acceptance.

Decision: Accept (Poster)