## 3 Method
### Markov Decision Process  
This study formulates the scheduling process of the FJSP as an MDP to model and optimize the scheduling decision-making process. The MDP provides a mathematical framework to describe an agent's decision-making in an environment, which is particularly suitable for modeling complex systems with dynamic interactions and state transitions [37; 38]. By modeling the FJSP as an MDP, we define key elements such as states, actions, state transitions, and reward functions. This formalization offers a theoretical foundation for optimizing scheduling decisions by addressing task allocation and resource selection.  
The scheduling process is divided into a series of discrete decision points. At each decision instant \(t\), the agent selects an action \(a_{t}\) based on the current system state \(s_{t}\). This action corresponds to assigning an unscheduled operation to an available machine and determining its processing start time. Subsequently, the system transitions from state \(s_{t}\) to a new state \(s_{t+1}\), and a corresponding reward \(r_{t}\) is generated. This cycle repeats continuously until all operations are completed, signaling the end of the scheduling process.  
State: the state \(s_{t}\) encapsulates all relevant information of the system at time step \(t\), including the operation status--details such as the start time, completion time, status (scheduled or unscheduled), and dependencies between operations; machine status: information about each machine's availability time and utilization rate. These components collectively constitute a comprehensive description of the system's state at time step \(t\), providing the agent with a complete view of the system.  
Action: The action \(a_{t}\) is the scheduling decision made by the agent at decision instant \(t\), defined as an operation-machine pair \(\big{(}O_{ij},M_{k}\big{)}\), where \(M_{k}\) represents an available machine and \(O_{ij}\) denotes an operation pending scheduling. The action set dynamically changes at each decision point, encompassing all feasible operation-machine pairs. By selecting appropriate actions, the agent can effectively allocate resources and optimize the scheduling process.  
State Transition: The state transition describes the change from state \(s_{t}\) to state \(s_{t+1}\), determined by the current state \(s_{t}\) and the chosen action \(a_{t}\). In the context of the FJSP, state transitions are deterministic, which makes the dynamic evolution of the scheduling process clear and predictable, thereby providing a stable environment for the agent's learning and decision-making.  
Reward: The reward \(r_{t}\) measures the contribution of the action \(a_{t}\) chosen by the agent at time step \(t\) towards the scheduling objective. In this study, the reward is defined as the change in the maximum completion time between two consecutive states, i.e., \(r_{t}=C_{max}(s_{t})-C_{max}(s_{t+1})\), where \(C_{max}(s_{t})\) represents the current makespan under the scheduling scheme. This reward mechanism incentivizes the agent to minimize the makespan, thereby enhancing scheduling efficiency and achieving the optimization goal.  
Policy: The policy \(\pi(a_{t}|s_{t})\) defines the probability distribution over actions \(a_{t}\) given the current state \(s_{t}\). Utilizing deep reinforcement learning algorithms, the policy \(\pi\) is parameterized by a neural network. By continuously optimizing the parameters of the policy network, the agent learns to select optimal actions in various states, thereby maximizing the cumulative reward and effectively solving the FJSP.