## 2 Literature Review
### DRL for Solving JSP  
DRL has become a powerful tool for tackling high-dimensional scheduling and combinatorial optimization problems. With its capabilities in autonomous learning and real-time decision making, DRL has been widely applied to job shop scheduling. To address dynamic events such as machine breakdowns and job insertions, [PERSON] et al. [42] proposed a hierarchical and distributed framework using the Double Deep Q-Network (Double DQN) algorithm to train scheduling agents, capturing the complex relationships between production information and scheduling objectives, thereby enabling real-time scheduling decisions in flexible job shops. [PERSON] et al. [43] developed a hierarchical multi-agent DRL approach based on PPO, which incorporates multiple types of agents, including objective agents, job agents, and machine agents. [PERSON] et al. [44] modeled manufacturing equipment as agents and employed an improved contract net protocol to guide agent cooperation and competition, training the agents via the PPO algorithm. [PERSON] et al. [45] proposed a PPO-based hybrid prioritized experience replay strategy to effectively reduce training time. In static job shop scheduling scenarios, [PERSON] et al. [46] used a disjunctive graph embedding method to learn graph representations containing JSSP features, enhancing the model's generalization ability. They employed a modified Transformer architecture with multi-head attention to solve large-scale scheduling problems efficiently. Similarly, [PERSON] et al. [27] represented solutions to distributed Job Shop Scheduling Problems using disjunctive \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Reference** & **AGV Energy Consumption** & **Objective Function** & **Solution Algorithm** \\ \hline \({}^{*}\) [PERSON] et al. [33] & No & Makespan & Imperialist Competitive Algorithm \\ \({}^{*}\) [PERSON] et al. [4] & Yes (Transport Energy) & Makespan + Energy & Enhanced Genetic Algorithm \\ \({}^{*}\) [PERSON] et al. [34] & Yes (Transport Energy) & Makespan + Energy & Improved Gray Wolf Algorithm \\ \({}^{*}\) [PERSON] et al. [35] & Yes (Transport Energy) & Makespan + Energy & Improved Genetic Algorithm \\ \({}^{*}\) [PERSON] et al. [36] & Yes (Transport Energy) & Makespan + Energy & Hybrid many-objective \\ \({}^{*}\) [PERSON] et al. [36] & Yes (Transport Energy) & Makespan + Energy & evolutionary algorithm \\ \({}^{*}\) [PERSON] et al. [24] & Yes (Transport Energy) & Makespan + Energy & Novel heuristic method \\ \({}^{*}\) [PERSON] et al. [37] & No & Makespan & Multi-agent System \\ [PERSON] et al. [16] & No & Makespan & Multi-agent Hybrid Improved \\ Heuristic Algorithm \\ [PERSON] et al. [12] & No & Makespan & Constraint Programming \\ [PERSON] et al. [13] & No & Makespan & Improved Genetic Algorithm \\ [PERSON] et al. [14] & No & Makespan & Improved Genetic Algorithm \\ [PERSON] et al. [23] & No & Makespan & Improved Genetic Algorithm + \\ [PERSON] et al. [38] & Yes (Transport Energy) & Makespan & Digital Twin \\ [PERSON] et al. [25] & No & Makespan & Learning-based Multi-population \\  & & & Evolutionary Algorithm \\ [PERSON] et al. [17] & No & Makespan + Energy & Hybrid Deep Q-Network \\ [PERSON] et al. [39] & Yes (Energy-aware) & Makespan + Energy & DQN + Memetic Algorithm \\ [PERSON] et al. [40] & No & Makespan & Memetic Algorithm \\ [PERSON] et al. [41] & Yes & Makespan + Energy & Deep Q-Network \\ [PERSON] et al. [11] & Yes & Makespan + Energy & Nested Hierarchical DRL \\ Ours & Yes (Transport Energy With & Makespan & CRGPPO-TKL (Improved PPO) \\ Charging Constraints) & & & \\ \hline \hline \end{tabular}  
* indicates unlimited transport resources.  
\end{table} 
Table 1: Literature comparison. graphs and combined graph neural networks with reinforcement learning to solve the problem. [PERSON] et al. [47] proposed a heterogeneous graph neural network framework to capture intricate relationships between jobs and machines, and designed a novel DRL-based method to learn priority scheduling rules in an end-to-end manner. [PERSON] et al.