# An Overview of Reinforcement Learning Methods for Variable Speed Limit Control
## 3 Research and Implementation
### Results of Research Questions
A drawback in modeling VSL as MDP on a longer motorway stretch is a large number of needed state-action pairs that lead to the problem of the exponential growth of the solution space. It becomes impossible to search for an optimal solution in real-time. This is known as the _course of dimensionality_[51]. As shown in [18], this can be solved with function approximation techniques. In [20], three different approaches for feature-based state representation (Coarse and Tile coding, and Radial Basis Function (RBF)) having a linear function approximator for \(Q\)-function were compared for solving the QL-VSL problem. The mentioned methods were tested on a synthetic motorway model modeled in the microscopic traffic simulator VISSIM. Results show that function approximation methods outperform QL-based VSL formulated with a Q-table by an average improvement of \(10\%\) regarding the convergence of reward function (\(TTS\) in this case), where feature extraction methods (Coarse and Tile coding) showed a slightly faster learning rate and a more stable control policy. In [19], an algorithm for simultaneously solving VSL and RM by means of Multi-Agent RL (MARL) was proposed in order to reduce congestion in the bottleneck area produced by the interference of entering vehicles from the on-ramp with vehicles in the mainstream. Traffic conditions or \"_states_\" of the agent's environment in case of VSL are described with the densities measured directly downstream \(\rho_{ds}\) of the on-ramp (bottleneck location), the density \(\rho_{app}\) measured directly upstream of the bottleneck \"_applied VSL area_\" (\(VSL_{app}\)), and the third state variable is the density \(\rho_{us}\) measured on the motorway section further upstream from that \"_upstream section_\" comprising the area considered for the second state variable \(\rho_{app}\). In the case of RM, only the first two density values were taken as well as the length of the vehicle queue at the on-ramp. Actions (allowed speed limits) are defined by the set \(\{90,100,110,120\}\) km/h. Learning algorithms based on temporal-difference learning and weighted k-Nearest Neighbors for linear function approximation denoted as kNN-TD [52] were used to approximate the Q-function. Depending on the Euclidean distance between the entry point of the state and the points that represent the centers of each group, the corresponding Q-value is refreshed depending on the severity of the state belonging to each group. Thus, the simplest method tested considering RM and VSL agents learn independently. In the second case, a hierarchical MARL approach was used where an agent with a higher hierarchy is the first to bring an action, and this action is communicated to the second-highest ranked agent who takes this action into account and selects its own action accordingly. The third approach is the so-called maximax MARL based on the principle of locality of intersection among agents [53]. In it, a mapping of the effect of an agent's actions to the global value function (in this case, only the neighboring agents are considered) is being done [54], in which agent \(i\) and neighboring agent \(j\) look for actions that maximize the Q-value gain. Only the agent whose action has the largest contribution to the increase in joint payoff may change the action, while other neighboring agents may not. The process is repeated until the maximum joint payoff is achieved. It can be said that agents cooperate locally and their usefulness is further used in the calculation of the global Q-value. By applying the maximax MARL coordinated control of the VSL and RM using kNN-TD learning algorithm, an additional improvement of \(TTS\) compared to all baselines was achieved. Almost \(11\%\) compared to the no-control case.