# Real-Time Adjustment Method for Metro Systems with Train Delays Based on Improved Q-Learning
## 4 TTR Using Reinforcement Learning
### Algorithm Process  
An improved Q-learning algorithm is designed to solve the TTR problem and is tailored to real operational scenarios. A matrix is constructed to represent the Q-value state table, as shown in Table 3. The columns represent different states, i.e., all physical stations arranged in the order of operation, while the rows represent all available actions. The \(\varepsilon\)-greedy strategy is a commonly used action selection strategy for Q-value state, where the agent explores new actions with a probability of \(\varepsilon\) and exploits the best action with a probability of \((1-\varepsilon)\). However, excessive exploration in the later stages may affect the convergence speed. To address the balance between exploration and exploitation, the simulated annealing concept is employed to improve the convergence. A temperature parameter is applied to control the degree of exploration. As the iterations progress, the temperature is reduced to decrease the acceptance probability of inferior actions, making it more inclined to exploit advantageous actions. By gradually lowering the temperature, the acceptance probability of suboptimal solutions is reduced, achieving a balance between exploration and exploitation. This accelerates the convergence speed and avoids excessive exploration. When incorporating the concept of simulated annealing into the Q-learning algorithm, the Metropolis criterion is employed to dynamically regulate the trade-off between exploration and exploitation. The essence of the Metropolis criterion lies in controlling the acceptance probability of new actions through the dynamic adjustment of the temperature parameter. The algorithm follows the following steps:  
1. Set the optimal action as the current action \(a_{i}^{best}(k)\). 2. Select an action \(a_{i}^{random}(k)\) randomly. 3. A random number \(\delta\) is generated. According to the Metropolis criterion, the random number \(\delta\) is compared with \(o=\exp\left[\left(Q\left(S_{i}(k),a_{i}^{random}(k)\right)-Q\left(S_{i}(k),a_{ i}^{best}(k)\right)\right)/pT\right]\) to decide whether to accept the new action. If \(\delta\) is larger, accept the action \(a_{k,i}^{random}\) as the current action; otherwise, keep the optimal action \(a_{i}^{best}(k)\) unchanged. Here, \(p\) is the adjusting factor, and \(T\) represents the number of iterations in the algorithm, both equivalent to the temperature control parameter in the simulated annealing algorithm. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Physical Stations**} & \multirow{2}{*}{**State**} & \multicolumn{5}{c}{**Action**} & \\  & & \multicolumn{2}{c}{**Strategy A**} & \multicolumn{2}{c}{**Strategy B**} & \multicolumn{2}{c}{**Strategy C/(s)**} \\  & & **1** & **2** & \(\ldots\) & \(-\)**10** & **10** & \(\ldots\) \\ \hline (1401,1402, \(\ldots\),1417) & (0,0,\(\ldots\),0) & 0 & 0 & 0 & 0 & 0 & \(\ldots\) \\ (1401,1402, \(\ldots\),1417) & (0,0,\(\ldots\),50, \(\ldots\),0) & 0 & 0 & 0 & 35 & 20 & \(\ldots\) \\ (1401,1402, \(\ldots\),1417) & (0,0,\(\ldots\),0) & 20 & 0 & 10 It is obvious that with the increase in \(T\), the value of \(pT\) also increases, reducing the probability of accepting inferior actions. This strategy ensures that as the algorithm iterates, the probability of exploration gradually decreases, relying more on the knowledge already learned, thus finding a balance between exploration and exploitation. This strategy can be considered as an equivalent form of the greedy strategy. The adjusting factor \(p\) directly influences the rate of temperature reduction. The larger \(p\) is, the faster the transition to the greedy strategy. Once an action is selected, the value of the state-action pair is updated at the corresponding position in the Q-value table according to the state transition probability, as shown in Equation (19). Through continuously performing state transitions and updating the Q-value function, the Q-learning algorithm gradually converges to the optimal Q-value function, thereby learning the optimal action strategy.