# An Overview of Reinforcement Learning Methods for Variable Speed Limit Control
## 3 Research and Implementation
### Results of Research Questions
Three baseline cases were taken to compare the results. The first without VSL, the second with QL structure similar to that in [21] with only the reward function being proportional to the negative \(TTS\) and the third case in which the agent is trained using the DQL algorithm. Both starting cases (QL and DQL) have the possibility of setting the same speed limit for all traffic lanes, i.e., they do not have the possibility of assigning a different speed limit for each individual traffic lane as in the case of DVSL controllers. All learning-based VSL controllers were trained through 150 episodes. Each episode is a simulation lasting 18 simulation hours, where the SUMO microscopic traffic simulator was used for simulations. Upon completion of the learning process, the operation of all RL controllers was simulated on an equal set of simulations (using 50 different simulation seeds) to ensure a valid comparison of the obtained results. The analysis of the results included a comparison of the average cumulative values of the reward functions and the average travel time. By applying DVSL based on the DDPG algorithm, a better result was achieved compared to QL, DQL, and the case without VSL. For example, the DDPG with a reward function proportional to \(TTS\) reduced the mean cumulative value of \(TTS\) by approximately 3% compared to the case without VSL. The analysis of the optimal control policy (DDPG approach), depending on the reward function, proved that the most stable learned control policy was the case when the reward function was formulated through \(TTS\). The authors note that such a system is not yet possible in reality because it requires CAV technology. This research and the obtained results provide a good insight into the possibility of using AV as excellent actuators for managing the entire traffic system on motorways as well as the design of the deep learning algorithm structure related to the number of actions and defining the appropriate criterion function (reward function). As shown in [18], predicted traffic parameters can be used as input to the RL-VSL control algorithm to improve the learning and control process. As shown in [62], Convolutional Neural Network (CNN) can be used for speed prediction on a motorway segment. CNN has been proven as suitable for processing \(2D\) data, such as images. Ideally, spatial characteristics of traffic flow on the motorway segment can be represented as an image containing information about traffic speed, density or vehicle position, etc. (see Figure 7). One image represents a two-dimensional matrix. The sequence of traffic environment measurement samples (several images) is most commonly used. In this case, the entrance to the CNN network is a three-dimensional matrix filled with traffic data of several consecutively collected images in time (channels). Images are continuously sent to the input of a deep neural network. However, other DL algorithms, such as LSTM can be used for the prediction of traffic parameters as well. For example, in [63], LSTM was used to process time-series traffic data from the motorway segment, thus adding the spatial-temporal information to capture the long time dependency of a nonlinear traffic dynamic in an effective manner. In general, CNN can output one value, while the LSTM structure enables the forecast of a sequence of new traffic states for several future time steps.