# Real-Time Adjustment Method for Metro Systems with Train Delays Based on Improved Q-Learning
## 1 Introduction
### Literature Review
Heuristic algorithms are also employed to provide optimization solutions in TTR. Various metaheuristic approaches have been extensively studied, including taboo search [30], genetic algorithms [31], particle swarm algorithms [32], and ant colony algorithms [33]. Within this field, many scholars have designed specific operators to handle problem constraints. Compared to optimization model methods and simulation methods, heuristic algorithms offer greater flexibility, faster computation, and the ability to handle complex problems. AI methods have gained popularity in addressing the TTR problems. Recognizing the limitations of heuristic algorithms, such as their susceptibility to local optima and poor adaptability, researchers are now exploring machine-learning-based methods. Machine-learning-based methods can be further divided into three main types:  
1. The first type involves understanding the search process of existing search algorithms. For example, [PERSON] et al. employed reinforcement learning to solve the process of mixed-integer linear programming, designing a reinforcement-learning-based branching strategy [34]. [PERSON] et al. used reinforcement learning to select appropriate cutting planes for the branch-and-cut process [35]. 2. The second type is based on data-driven approaches. For instance, [PERSON] et al. combined genetic algorithms with artificial neural networks to simulate train dispatchers in conflict resolution [36]. However, these methods may not be suitable for solving the TTR problem efficiently due to their extended response times. 3. The third type involves using reinforcement learning to directly construct solution strategies, which is more suitable for the TTR problem. In the field of railway operations, [PERSON] was among the early adopters of the Q-learning algorithm to tackle the single-track train rescheduling problem. However, challenges arose due to the large state vector and scalability issues to other scenarios [37]. [PERSON] extended the Q-learning algorithm to real-world instances of single-track and multi-track dispatch, reducing the state vector representation and enhancing scalability. Real-time computation remained a challenge [38]. [PERSON] et al. employed Q-learning to solve the railway timetable rescheduling problem and demonstrated its effectiveness in finding high-quality solutions within a limited training set [39]. [PERSON] et al. utilized a multi-agent deep reinforcement learning approach for the TTR problem. However, their method lacked delay information, had limited state generality, and exhibited insufficient scalability [40]. [PERSON] expanded the state representation to include the actual arrival and departure times of trains, providing more information about delays [41]. Some scholars found that learning strategies could only be trained for individual problem instances. For example, [PERSON] et al. trained learning strategies from multiple instances to address delay scenarios at railway intersections [42]. [PERSON] et al. used a policy-based reinforcement learning approach, but their problem only allowed for the delay of one train, which does not capture the simultaneous delays of multiple trains that often occur in reality [43]. In the field of metro systems, [PERSON] et al. used Q-learning to simulate metro train operations and adjust schedules. However, their state definition made it challenging to solve large-scale scenarios, leading to the risk of local optima or difficulties in problem-solving [44]. [PERSON] et al. proposed using deep reinforcement learning (DRLA) to minimize energy consumption in metro train timetable rescheduling [45]. In summary, each method has its strengths and limitations. A comparison between these methods and the proposed approach in this paper is provided in Table 1.  
Upon analyzing the literature, it becomes apparent that previous studies on the TTR problem have certain limitations. When applying certain algorithms to tackle this problem, there are issues related to incomplete model construction. Some specific problems include:  
1.