# An Overview of Reinforcement Learning Methods for Variable Speed Limit Control
## 2 Application of RL in VSL Control
### Variable Speed Limit Control
While taking actions (e.g., speed limits in the case of VSL agent) by trial-and-error, the agent incrementally learns a \"value function\" over state-action pairs \(Q_{i}(s_{t},a_{t})\), which indicates their utility to that agent. Q-Learning (QL) [47] is one of the most implemented _value iteration_ RL algorithms. The value function updating rule for QL is as follows:  
\[Q_{i}(s_{t},a_{t}):=(1-\alpha_{Q})Q_{i}(s_{t},a_{t})+\alpha_{Q}(r_{t+1}+\gamma \max_{a^{\prime}\in A}Q_{i}(s_{t+1},a^{\prime})), \tag{1}\]  
where \(t\) is the discrete time step, action \(a_{t}\) in state \(s_{t}\) induces a state change to the new state \(s_{t+1}\). Depending on that transition, the agent receives the reward \(r_{t+1}\). The goal is to find an optimal policy \(\pi^{*}\) that maximizes the expected discounted reward for state \(s\). Parameter \(\alpha_{Q}\) is the learning rate that controls how fast the \(Q\)-values are altered. Discount factor \(\gamma\) controls the importance of future rewards. Finally, an agent looks one step ahead at the maximum \(Q_{i}(s_{t+1},a^{\prime})\) value given by optimal action \(a^{\prime}\) in new state \(s_{t+1}\). At any time, the RL methods use one-step lookahead with the current value function to choose the best action in each state by some kind of maximization. The agent can explore optimal actions or exploit its current control policy. Therefore, the policies that RL methods learn are called _greedy_ with respect to their value functions. In addition to such greedy actions, RL methods also take  
Figure 2: Reinforcement learning framework for VSL control. some directed or random (exploratory) actions. This ensures that all states are visited plenty of times so that the learning method does not get stuck in a local optimum. There are several exploration strategies to ensure this. The random exploration strategy takes random action with a fixed probability, giving high probabilities to actions with high values. The counter-based exploration prefers to execute actions that lead to infrequently visited states. Recency-based exploration promotes actions, which have not been executed recently in a given state. A commonly used scheme for action selection is \(\epsilon-greedy\) strategy, which takes a random action with probability \(\epsilon\) and a greedy action with probability \(1-\epsilon\). The Deep Reinforcement Learning (DRL) approach represents the extension of the classical RL used as the latest approach to tackle specific VSL control problems such as differential VSL where each lane receives its own speed limit value [37]. It is based on the learning representations of data. This approach attempts to model high-level abstractions in the data by using multiple processing layers with complex structures or otherwise composed of multiple nonlinear transformations [48]. The model, which contains multiple processing layers with complex structures based on the concept of the human cerebral cortex, is known as Deep Neural Network (DNN). The DNN integrates feature extraction, and classification (or prediction) processes in a single framework by using information-dense input datasets [1]. The DRL can also be used as the framework that integrates several different traffic control methods on motorways such as VSL control, RM, and lane change control [36].