# Real-Time Adjustment Method for Metro Systems with Train Delays Based on Improved Q-Learning
## 1 Introduction
### Literature Review  
For train timetable adjustments in delay scenarios, scholars such as [PERSON] have summarized it as the train timetable rescheduling (TTR) problem [5]. TTR involves modifying train operations by altering the sequence of trains, departure/arrival times, tracks/stations, etc., with the objective of reducing train delays. This adjustment is crucial for enhancing the quality of train services and improving passenger experiences. However, the TTR problem falls under the category of NP-Hard, indicating that it is a highly challenging problem that cannot be easily solved within affordable time in computational theory. As the problem's scale increases, the number of variables and constraints in TTR grows exponentially, making it exceedingly difficult to find rapid solutions for the model [5; 6; 7]. It is worth noting that compared to other scheduling problems like job shop scheduling, the TTR problem requires higher decision-making efficiency to ensure the safety and operational efficiency of trains [8]. Moreover, train operation adjustments have stringent real-time requirements, implying accurate modifications to be made within a short timeframe. The inherent conflict between these two aspects has made finding rapid solutions to the train timetable adjustment problem a long-standing focus and a significant challenge in research. Currently, researchers have put forward multiple solutions to address the TTR problem. [PERSON] et al. conducted a comprehensive review on this matter and suggested a hybrid approach as a future development direction. They also expressed the need for new methods in future research [9]. In line with this, the present article incorporates novel machine learning techniques and broadly categorizes the existing research methods into the following five categories: manual adjustment methods, optimization model methods, simulation methods, optimization methods based on heuristic algorithms, and AI methods. Manual adjustment methods are currently more commonly used in practice. Existing manual rule-based methods include First-Come-First-Serve (FCFS) [10], First-Schedule-First-Serve (FSFS) [11], and First-In-First-Out (FIFO) [12]. However, designing efficient scheduling remains a challenging task. Dispatchers face the crucial question of how to devise a high-performance and cost-effective method. Optimization model methods generally employ different planning models to describe the TTR problem, including integer programming [13], mixed-integer linear programming [14], constraint programming [15], and quadratic programming methods [16]. Scholars have applied specialized algorithms, such as branch and bound (B and B) [17] and branch and price [18], to tackle these planning models, decomposing the complex problem into simpler ones. Although this approach can provide precise solutions, the response time tends to be relatively long. To address this issue, researchers have proposed novel targeted algorithms, including parallel algorithms [7], decomposition methods [19; 20], rolling time domain algorithms [21], traffic direction multiplier methods [19; 22], and heuristic-assisted methods [23; 24], aiming to reduce computational costs. Simulation methods, such as train movement models based on the following theory, are used to test control algorithms and design solutions for train operations. For instance, [PERSON] designed a train movement model under a fixed block signaling system [25]. [PERSON] et al. further developed a railway traffic cellular automaton model under a moving block signaling system [26]. [PERSON] et al. extended traffic flow models to railways using a stochastic process model [27]. [PERSON] et al. introduced a train operation model under a virtual coupling system [28]. [PERSON] et al. presented a mesoscopic train-following model that accurately captures train interactions and predicts delays based on train spacing [29]. These simulation methods are known for their realism and accuracy, but face challenges related to complex modeling and computational requirements.