Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This contribution consists of an OpenAI Gym style wrapper around the SUMO traffic simulation package, a selection of traffic scenarios that can be run using that package, as well as a set of (RL) algorithms that can use this interface.
The authors ran all of the provided algorithms against all provided environments, and report performance figures for those runs, and make the code for doing so available.
strengths: This system seems potentially very useful for making Traffic Signal Control easier to experiment with in an RL setting, since the OpenAI Gym API is ubiquitous, and this system handles featurization of the action and observation spaces in a convenient way.
It is also helpful that there are selected traffic scenarios covering a range of relevant realistic conditions that any algorithm attempting to solve this problem should handle.

The paper also provides a reasonable introduction to the topic, as well as a description of how Traffic Signal Control is modeled as an MDP.
weaknesses: The primary weakness of this paper is that its main contribution, the benchmark, is provided under a no-derivatives license.
This makes it illegal for other researchers to build on this work by modifying the provided code, which drastically decreases the value of the contribution.
I recognize that the individual traffic scenarios are licensed under restrictive licenses (for understandable reasons), but there's no clear reason why this restriction is present for the new contributions made in this paper.

The packaging and documentation would also both benefit from additional work.
The existing documentation consists of this paper, as well as a small readme describing some of the installation requirements and how to run the code, with the expectation that the user works out of the code directory provided.
Working out of another author's code directory is not convenient compared to being able to install the library through python's standard packaging system.
The vast majority of frequently used RL benchmarks provide the ability to install them, and it is not difficult to add support for doing so.

It would also be preferable if the different algorithms presented used a single neural network library, instead of requiring multiple ones with unclear version requirements to be installed. In particular, dependencies on versions of tensorflow that lack officially supported builds are highly inconvenient.

The paper has a few minor formatting issues (mostly unresolved references), and spends too long criticizing CityFlow, but this is not consequential.
correctness: The benchmark appears to have appropriate considerations for experiment design.
clarity: The paper is reasonably clear.
relation to prior work: The relationship to prior works is fairly clear, although I do not have sufficient knowledge to know if any significant prior work has been missed.
additional feedback: None of my objections to this paper should be difficult for the authors to fix, and I look forward to them being resolved.
review_title: A potentially useful set of RL tools for Traffic Signal Control, but needs more permissive licensing and more technical work.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a benchmark for RL-based control of traffic lights. The primary advantages touted over existing work include: use of the SUMO simulator; traffic scenarios based on real-world road layouts and traffic levels; a standard Gym interface; and a set of reference algorithms for the proposed environments. Experiments show that independent PPO/DQN tend to do quite well at the end of training, but take far longer to converge than specialised algorithms like MPLight.
strengths: [S1] Paper makes a good case for the importance of studying the traffic control problem and having accurate simulators. Arguments in favour of this new simulator are clearly laid out (although note first weakness below).

[S2] On execution: Writing was generally clear, choice of experiments seemed reasonable.
weaknesses: [W1] It is unclear to me how much value this new benchmark provides for the community relative to the various existing benchmarks. The strongest argument seems to be in Section 2.3.1, which claims that existing benchmarks are less realistic because they either use simplified or arbitrarily over-complicated versions of real-world traffic grids. I do not know how much this matters for evaluation, though—it may be that the simplifications in question don't affect relative rankings of different methods at all, and the experiments in this paper do not try to evaluate how rankings change when moving from, e.g., the Jinming & Feng simulator to RESCO.

[W2] There are a few important experimental details missing from the paper.
correctness: [CO1] Ideally the paper should list all the hyperparameters for all the algorithms (step sizes, architectures, batch sizes, replay buffer sizes, preprocessing, etc. etc.). Right now Section 3.4 says a lot of things like "we left all the values at defaults in the reference implementation"—this information should be included in the paper as well, ideally in a big table somewhere in the appendix.

[CO2] Relatedly, Sections 3.1 and 3.2 note that there are various choices for the observation space and reward function that the user can select between. I might have missed it, but I didn't see an explanation of which options for sensing and the reward function were selected for the experiments in Section 4. I'm also not sure how much simulated time each state transition represents (e.g. is it running at 100Hz? 0.1Hz? Variable?).
clarity: [CL1] Overall writing quality was good. Section 2.2 was helpful for understanding domain-specific nomenclature.

[CL2] It would be ideal if standard deviations were included in Table 1 (e.g. write μ±σ in each cell, or similar for radii of 95th percentile confidence intervals).

[CL3] It's not totally clear what quantities are being plotted in Table 4 just from the description. e.g. I read the top left cell as "average standard deviation of average delay". I think I understand the intention, but it's a bit confusing.
relation to prior work: [R1] In Section 2.3.1, the paper argues that various existing benchmarks are unjustified in claiming to be representative of "real-world city layout and demand" or having "relevancy to the real world" because they require "deeper analysis of traffic trends". I'm not sure what this deeper analysis would entail, but I didn't see anything like that in Section 3. I can see how RESCO might have an advantage over synthetic benchmarks given that it's based on simulations of real cities, but I'm not sure why it can claim to have greater realism or practical relevancy than the New York-based CityFlow environment.
additional feedback: The basic idea of this paper was well-motivated and seemed well-executed. My main concern was the missing experiment details listed in the clarity & correctness sections. Traffic control is out-of-area for me, so for now I'm only comfortable giving a borderline recommendation, with low confidence.

------

(review last checked/updated 2021-07-19)
review_title: Review of RESCO benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes 1) benchmarking signal control tasks based on well-established traffic scenarios; 2) implementation of various RL algorithms on these signal control problems; 3) comparision and analysis of these RL algorithms under varying sensing assumptions. The benchmarking proposed in this paper could be very helpful for future study on using RL for traffic signal control.
strengths: - A standarized benchmarking on RL for traffic signal could serve as the foundation to help better future algorithm design for such problems, which could bring lots of benefits for real-world traffic control.
- The paper is well written, and the benchmarking tasks are carefully chosen to match the real-world scenarios;
- The experiments are well-conducted. A comprehensive set of baseline controllers and RL controllers are evaluated. The implementation are also validated by check against previous work, and detailed analysis are given for comparing the performance of the benchmarked algorithms.
weaknesses: - It would be interesting to see how the RL algorithms would work under weaker sensing abilities;
- Can the author give any discussion on how much sensing information mentioned in the paper can be accurately measured in the real-world traffic control scenarios. i.e., how realistic are the sensing assumptions are?


correctness: Yes
clarity: The paper is clearly written. 
relation to prior work: The paper gives a very detailed discussion on how it differs from previous contributions.
additional feedback: no
review_title: Review for Reinforcement Learning Benchmarks for Traffic Signal Control 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: - Proposes benchmarks to study the problem of congestion control using realistic traffic situations 
- Presents baselines and  thorough comparative evaluations 
strengths: - Current progress in congestion control algorithms are evaluated using different settings avoiding fair comparisons between them. This is a well motivated paper that positions itself as an effort to consolidate the diverging field 
- Presents a set of baselines and evaluations to outline a fair comparison between various state of art algorithms.
- Baselines are validated before they are used for comparisons.

weaknesses: - line 72-32: Be precise and revise the premise and claim about model-based methods and deep q learning algorithms 
- Provide more explanation wrt yellow light -- (a) is the duration of yellow light subsumed inside a phase? (b) How is wait time etc. affected by the yellow light (c) is yellow light mandatory after every phase transitions? (d) How are "free right turn on red" etc situations handelled?
- Line 166: Missing reference and typo 
- While I'm convinced, please justify why allowing algorithms the choice of state and rewards is a good choice for fair comparison give they are being evaluated on two different MDPs
- Sec 4.1 - the pronounced trend for FMA2C and IDQN isn't well justified. This is too large of a gap too gloss over 

correctness: Authors evaluate the correctness of their baselines before using them for evaluations.
clarity: Easy to follow
relation to prior work: Clearly outlines prior methods and explains the changes made.
additional feedback: - Comparative study and analysis of various methods is a bit underplayed. I'd suggest making this a prominent part of the submission. Extract it out from conclusions into its own section.
review_title: Well positioned paper consolidating a diverging field

Meta-review: All the reviewers appreciate the value of the proposed benchmarks. The remaining concerns seem addressable. I recommend accepting the paper while asking the authors to incorporate the review feedback into the camera-ready paper.
Decision: Accept