# An Overview of Reinforcement Learning Methods for Variable Speed Limit Control
## 3 Research and Implementation
### Results of Research Questions
RM, the cycle time of which is controlled by a feedback controller, has been used as a baseline to compare the efficiency of proposed DRL-VSL in controlling the inflow rate into the bottleneck. The results show that 10% penetration rate of controlled AV can improve the throughput of the two-stage bottleneck (four lanes reduce to two and then reduce to one) for \(\approx\)25% but only in the case of high inflows (\(inflow\geq 1600\)\(veh/h\)) compared to the no-control. At lower inflow rate, the DRL-VSL worsens the traffic outflow compared to both no-control and feedback RM. In [37], the Differential Variable Speed Limit (DVSL) system was tested for variable speed control. Depending on the traffic situation on the motorway segment, DVSL can assign different speed limit values for each lane separately. The traffic model for DVSL testing is part of the San Bernardino Highway in California, USA. It consists of five traffic lanes and one on- and off-ramp. The off-ramp is located immediately after the on-ramp, causing a disturbance in the far right traffic lanes as drivers start re-arranging earlier to be able to get off the motorway. Thus, they intertwine with the flow of vehicles entering the on-ramp, creating congestion. The far left lanes often remain used under their capacity. By applying the classic VSL, the same speed limit value is mandatory in each traffic lane, which can unnecessarily disturb the flow of less occupied traffic lanes due to lower limit values. Hence the solution of applying DVSL that would manage the speed limit depending on the load of a particular lane. Traffic demand in the simulated model was taken from the PeMS (Performance Measurement System) traffic database. A Deep Learning (DL) structure based on the policy iteration  
Figure 6: Determination of state set in the RL agent [35]. algorithm known as _actor-critic architecture_[60] was used to train the agent for DVSL. The actor generates a speed limit, and the critic evaluates the executed action of the actor. The architecture makes it possible to avoid evaluating the whole set of actions for each lane. The set of actions consists of six speed limits \(50,55,60,65,70,75\) mph. Given the five traffic lanes, the number of combinations is relatively large (\(6^{5}=7776\)) and is a problem for classical Q-Learning or DQL. Therefore, the Deep Deterministic Policy Gradient (DDPG) algorithm optimizes parameters in the actor-critic architecture since it enables work with large sets of discrete action values. The algorithm first gives a continuous action, and based on it finds a set of the closest discrete speed constraints. To overcome the issue with temporal correlation between samples (\(s_{t}\),\(s_{t+1}\),\(a_{t}\),\(r_{t+1}\)) and rapid forgetting of possibly useful experiences, the prioritized experience replay [61] is used to address these problems by storing experience into a replay memory. The experiences are constantly sampled from the replay memory to update the agent's knowledge, which stabilized the learning of neural networks in DRL. The traffic situation is described by a 15-dimensional vector that describes the occupancy in the area immediately after the flow interference (mainstream-entry ramp), in front of the congestion area (mainstream), and the occupancy at the on-ramp. Four reward functions were tested in which the value evaluates the agent's performance during each control time step (duration of the action is 1 min). The first reward function is proportional to the negative value of the measured \(TTS\), the second function is positive, and describes the mean speed in the bottleneck. The third reward function is proportional to the negative number of sudden braking of the vehicles, and the detection threshold of such events is a deceleration of \(4.5\) m/s\({}^{2}\). The fourth reward function takes the environmental aspect into account, and is proportional to the negative sum of the scaled values of \(CO\), \(HC\), \(NO_{x}\) and \(PM_{x}\) emissions (according to the Euro VI standard).