Green Wave Coordination For Trac Signal Control Using Deep
Reinforcement Learning
Maxime Trécaa,∗ , Mahdi Zargayounab , Dominique Bartha and Julian Garbisoc
a Université Versailles Saint-Quentin-en-Yvelines, Versailles, France

maxime.treca@gmail.com, dominique.barth@uvsq.fr

b Université Gustave Eiel, Champs-sur-Marne, Francemahdi.zargayouna@univ-eiel.fr

c Institut Védécom, Versailles, Francejulian.garbiso@vedecom.fr∗ Corresponding author

Extended abstract submitted for presentation at the 11th Triennial Symposium onTransportation Analysis conference (TRISTAN XI)
June 19-25, 2022, Mauritius Island
March 16, 2022

Keywords: Trac Signal Control, Green Wave Coordination, Deep Reinforcement Learning

1 INTRODUCTION
Trac signal control (TSC) designates the use of trac lights to achieve two crucial tasks inurban mobility: optimizing trac ows on a road network while ensuring the safety of its users.
While various classical TSC methods have all aimed at solving these two challenges with dierent
levels of success and complexity, the use of reinforcement learning (RL) algorithms for TSC, whichrst appeared in 1992 and has known an exponential rise since the years 2010 (Noaeen et al.,2021), have also provided excellent results in simulated trac conditions.
The reinforcement learning for trac signal control (RL-TSC) literature has developed andimproved in recent years: recent advances in the eld of deep learning has marked a shift from theuse of classical RL algorithms such as Q-learning to more advanced deep reinforcement learningalgorithms such as 3DQN (Greguri¢ et al., 2020) for TSC, notably increasing the performance ofRL-TSC controllers. A common area of research in the RL-TSC literature nowadays is the studyof coordination mechanisms used by the learning agents in multi-intersection road networks,
which can improve the overall performance of RL-TSC systems. These modes of coordination areusually divided into three groups: independent control, in which intersections do not communicate
with each other, indirect coordination methods, such as MARLIN-IC (El-Tantawy and Abdulhai,2012), in which intersections model joint state and joint actions with their neighbors throughindirect state observation and direct coordination methods, such as MARLIN-DC (El-Tantawyand Abdulhai, 2012), in which intersections directly communicate in order to decide on their nexttrac-routing actions jointly. This paper presents a novel fourth mode of coordination directly
inspired by the trac engineering literature known as green wave coordination, in which multiple
intersections maximize the vehicular throughput over an arterial. This extended abstract presents
this novel coordinated RL-TSC method in section 2, before pitting it against a state-of-the-art
RL-TSC method in a simulated setting in section 3. Finally, we conclude and present future
research directions in section 4.
TRISTAN XI Symposium

Original abstract submittalTréca Maxime, Zargayouna Mahdi, Barth Dominique and Garbiso Julian

22 LEARNING AND TRAFFIC SIGNAL CONTROL
2.1 Reinforcement Learning for Trac Signal Control

RL tasks for TSC are usually modeled using a Markov decision process (MDP). At each time
step t, the agent (i.e. the intersection) observes the state of the environment st and chooses anaction at . We dene the state as st = (φt , dt , ct (l1 ), . . . , ct (ln )), where φt represents the active
green phase index, dt the duration for which this phase has been active, and ct (li ) the amount ofvehicles present in lane li at step t. On the basis of this observation, the intersection chooses anaction at inuencing the current trac signal settings in dierent ways depending on the RL-TSC
method (see subsection 2.2). After application of the chosen action at , the trac environmenttransitions to a new state st+1 and returns a reward rt to the agent. The reward of an agentPP
taking two successive actions at steps t and t + k is dened as rt = li ωt+k (li ) − li ωt (li ),where ωt (li ) is the cumulated waiting time of vehicles on lane li at step t,

2.2 Trac Signal Control Methods
This paper features two RL-TSC methods based on deep reinforcement learning. The rstmethod, I-2DQN, does not coordinate between agents and is hence an independent method.
This method is both used as a benchmark and as a building block for the second method,GW-DQN, which features deep reinforcement learning for green wave coordination.

2.2.1 Independent Trac Signal Control
The independent dueling deep Q-network (I-2DQN) method is a state-of-the-art RL-TSC method(Greguri¢ et al., 2020) which leverages modern reinforcement learning techniques. Our I-2DQN
implementation features three fully connected hidden layers of 128 neurons, each associated witha ReLU rectier and batch normalization layer, and a nal double layer used for the computationof the advantage and value functions A(s, a) and V (s) typically found in dueling networks (Wanget al., 2016). This method uses experience replay by sampling mini-batches of 32 observations
from a replay buer D and uses a target network θT , which is updated every 100 learning steps.
The implemented policy is an ε-greedy policy with a decaying search parameter ε which ensures
a good balance between exploration and exploitation of the problem's state space. Intersections
using I-2DQN choose between step-based actions: at each decision step t, the intersection has tochoose whether to extend the current green phase by a single step (up to a parameter maximumduration dmax ) or to switch to the next green phase within the signal cycle (after a parameter
mandatory amber and red safety phase of duration dmin ). Since they do not impose constraints
on the total signal cycle duration, and since they provide a high level of signal control, step-based
actions usually provide, ceteris paribus, superior performances in RL-TSC applications (Trécaet al., 2020).

2.2.2 Green Wave Coordination
Green wave coordination aims at maximizing the throughput over a major arterial by limiting
the number of stops encountered by vehicles driving alongside it. This green wave phenomenonis achieved by computing osets, which indicates the travel time of a vehicle driving between
each intersection of the arterial in non-saturated trac conditions. These osets are usually
computed manually using a time-space diagram, as shown on Figure 1. Interestingly, to the bestof our knowledge, green wave coordination has not been applied in the RL-TSC literature.
The green wave dueling deep Q-network (GW-2DQN) method is a novel RL-TSC methodfeaturing green wave coordination. Each intersection using this method features the same architecture as the I-2DQN method, with the exception that it implements phase-based actions.
Phase-based actions, for which an agent chooses the entire length of a green phase at once, areTRISTAN XI Symposium

Original abstract submittalTréca Maxime, Zargayouna Mahdi, Barth Dominique and Garbiso Julian

3u4
u3
u2
u1
Figure 1  Example Time-Space Diagram on a four intersection arterial. The x-axis represents

time and the y axis distance. The signal cycles of the four intersections are computed so thatgreen waves, represented by black arrows, can occur in both directions of the arterial.
needed to maintain proper osets between signal cycles of the intersections along the arterial
and allow for a green wave to occur. The signal cycle's duration, D, is divided between the green
phase duration over the arterial d1 and perpendicular to the arterial, d2 , and two mandatory
amber and red phases of xed duration dmin . Each intersection implementing the GW-DQN
method hence has to pick a single action per signal cycle: the duration of green phase d1 , which
can range between dmin and D − 3dmin , respectively the minimal green phase duration, and themaximal allowable phase duration which ensures that phase d2 receives the required minimal
phase duration. After choosing duration d1 , duration d2 is automatically computed to ll up theentire signal cycle of total duration D.
The GW-DQN method is straightforward to implement since the only requirement is to maintain osets between intersections of the arterial, which can easily be expressed as an additional
set of constraints on action selection, regardless of the underlying RL method. Furthermore,
the GW-DQN method represents a trade-o due to these additional constraints. On the onehand, using phase-based actions allows for green wave coordination, maximizing throughput
over the arterial. On the other hand, using a phase-based action space reduces the adaptabilityof intersection to changing trac conditions due to longer decision intervals, decreasing agent
performance in the process (Tréca et al., 2020). Trac simulations in section 3 comparing theI-2DQN and GW-2DQN methods should indicate which factors are more prominent.

3 EXPERIMENTAL RESULTS
3.1 Experimental Settings

We run our experiments on the SUMO trac simulator and the carmulator RL-TSC library wehave developed. The two methods are tested on a 4-intersection arterial road network. We use
signal cycle values of D = 60 and dmin = 5. Green wave osets are automatically computed bySUMO. Trac is generated using a Poisson arrival process of parameter λu,v for each entry andexit lane pair (u, v) of the network. The base value of parameter λu,v is doubled if the entry laneu is located on the arterial or if the exit lane v is located on the arterial, and tripled if both uand v are located on the arterial, which promotes larger trac ows on the main arterial while
maintaining trac ows on alternative routes.

3.2 Agent Performance
We measure the performance of both methods after their training process of 500 simulation
episodes of 1000 steps each. These performances are measured by running 20 trac scenarios,
each using distinct trac demand patterns, and by measuring the performance spectrum of eachmethod by plotting the worst and best vehicular cumulated waiting time evolution throughout
the simulation scenario. These performance measurements are plotted in two cases: normal tracTRISTAN XI Symposium

Original abstract submittalTréca Maxime, Zargayouna Mahdi, Barth Dominique and Garbiso Julian

4conditions, using a near-saturating base arrival rate of λ = 0.06 (around 216 vehicles/hour) andsaturated trac conditions using a base arrival rate of λ = 0.08 (around 288 vehicles/hour), inorder to observe how both methods fare in these scenarios.
Best and Worst Agent Performances Over 20 Traffic Simulations, Regular Traffic

Best and Worst Agent Performances Over 20 Traffic Simulations, Saturated Traffic
100000

I2DQN
Adaptive Green Wave

25000

Worst/Best Cumulated Waiting Time (s)Worst/Best Cumulated Waiting Time (s)

30000

20000
15000
10000
5000
00200

400

600

800

1000

1200

1400

Adaptive Green Wave
I2DQN

90000
80000
70000
60000
50000
40000
30000
20000
10000
00200

400

Simulation Step (s)

600

800

1000

1200

1400

1600

Simulation Step (s)Figure 2  Performance spectrum of the I-2DQN and GW-2DQN methods in normal (left) andsaturated (right) trac conditions.
Results show that the GW-2DQN method outperforms the I-2DQN method in terms ofaverage and variance in cumulated waiting time in normal trac conditions, showing that green
wave coordination on arterials using deep reinforcement learning is benecial for TSC. However,this relative superiority is entirely reversed when trac conditions are saturated since the GW2DQN performance drops signicantly in the second experiment.

4 DISCUSSION AND FUTURE WORKS
This paper proposes a novel green-wave-based approach for a deep RL-TSC. The results showthat the method outperforms I-2QN but underline the somewhat fragile nature of green wave
coordination. If the GW-2DQN method allows for increased performance gains with minimal
overhead in terms of complexity, these gains only hold if the green wave phenomenon can occur,
implying that no trac congestion is present along the arterial. Since this condition is not likely
to be constantly veried in real-life scenarios, a hybrid RL-TSC method that could automaticallyswitch between I-2DQN and GW-2DQN depending on congestion levels along an arterial couldoer the best of both worlds. Such a mechanism could easily be constructed using vehicularcongestion values along the arterial lanes, which are already used as a state feature of both RL
algorithms.

References
S. El-Tantawy and B. Abdulhai. Multi-agent reinforcement learning for integrated network of adaptive trac signal controllers (marlin-atsc). In Intelligent Transportation Systems (ITSC), 2012 15th
International IEEE Conference on, pages 319326. IEEE, 2012.
M. Greguri¢, M. Vuji¢, C. Alexopoulos, and M. Mileti¢. Application of deep reinforcement learning intrac signal control: An overview and impact of open trac data. Applied Sciences, 10(11), 2020.
M. Noaeen, A. Naik, L. Goodman, J. Crebo, T. Abrar, B. Far, Z. S. H. Abad, and A. L. Bazzan.
Reinforcement learning in urban network trac signal control: A systematic literature review. 2021.
M. Tréca, J. Garbiso, and D. Barth. Guidelines for action space denition in reinforcement learning-based
trac signal control. In ICAPS Conference, Special Track on Planning and Learning. ICAPS, AAAI,
2020.
Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures
for deep reinforcement learning. In International conference on machine learning, pages 19952003.
PMLR, 2016.

TRISTAN XI Symposium

Original abstract submittal