## 4 DT Workshop Planning Modeling Based on GNN
### Graph Neural Network Structure
#### 4.2.2 Markov Decision Process  
The MDP is a method that transforms the conventional workshop scheduling process into a five-tuple \(\langle\mathcal{S},\mathcal{A},P(\mathcal{s}|\mathcal{s},\mathcal{a}),r, \gamma\rangle\) to describe it. The five elements within the tuple correspond to the state, action, state transfer function, reward, and discount factor of the scheduling calculation, which are important parameters for the realization of the DRL-based scheduling decision process. The decision-making process is performed by discerning whether there is a combination of process \(o_{ij}\) that needs to be scheduled and idle machine \(m_{k}\) at each time point \(t\) within the planning time \(T_{p}\). The \(\left(o_{ij},m_{k}\right)\) is transformed into \(\mathcal{A}(t)\) input to the agent and obtains the reward \(r(t)\) from the environment at the current \(\mathcal{S}(t)\), which based on the makespan value. Finally, a complete plan scheme with the total operation time is obtained after several iterations. The detailed MDP definition is as follows. _(1) State._ State definition is the pivotal aspect of solving scheduling problems by graph representation learning, and it is also the crucial step for feature extraction in a large-scale schedule optimization problem. The scheduling process state encompasses three types: completed operation, operation in progress and operation to be scheduled. Let the operation, device and action correspond to \(\mathcal{O}(t),\mathcal{M}(t)\) and \(\mathcal{A}(t)\), respectively, and determine \(\mathcal{S}(t)\) according to the decision step \(t\) together with the scheduling environment. By extracting the feature values of each element, the state \(\mathcal{S}(t)\) forms a set of multidimensional feature  
Figure 6: The planning framework based on the GAT and DRL. vectors, including but not limited to operation features \(\mathbf{\Lambda}_{o_{ij}}\in\mathbb{R}^{8},o_{ij}\in\mathcal{O}(t)\), machine features \(\mathbf{\Lambda}_{m_{k}}\in\mathbb{R}^{5},m_{k}\in\mathcal{M}(t)\), and action features \(\mathbf{\Lambda}_{(o_{ij}m_{k})}\in\mathbb{R},(o_{ij},m_{k})\in\mathcal{A}(t)\). To be specific, the operation features consist of eight-dimensional parameters, which are the operation status label (Whether to complete scheduling, 0--no, 1--yes), process time \(p_{ij}^{k}\) (the value of an unscheduled operation is the average of the completion times of that operation on all available machines, i.e., \(\overline{p_{ij}^{k}}=\sum_{m_{k}\in\mathcal{M}_{ij}(t)}p_{ij}^{k}/|\mathcal{ M}_{ij}(t)|\)), start time \(st_{ij}(t)\) (the start time is \(st_{ij}(t)=st_{il}+p_{ilk}\) when the preorder operation has been scheduled, otherwise it is estimated recursively according to \(st_{ij}(t)=st_{il}(t)+\overline{p_{ilk}}\)), remaining process time (Unscheduled operation is 0), completion time, latest deadline, operation waiting time and the number of remaining operations \(j_{i}\). The 5-dimensional machine features contain machine status (Available or not, 0--no, 1--yes), machine operating time, number of operations to be processed, waiting time and remaining processing time. The action feature corresponds to the operation-machine combination feature, taking the value of the process time \(p_{ij}^{k}\) for selecting machine \(k\) for the \(o_{ij}\). \(\mathcal{S}(t)\) integrates the static and dynamic attributes of the scheduling process, which can be updated in a timely manner by the data acquired by the DT system, enabling a genuine realization of adaptive scheduling in the actual business of the workshop. _(2) Action_. The action space \(\mathcal{A}(t)\), which is in a dynamic change process at each decision time, contains all feasible combinations of \(\left(o_{ij},m_{k}\right)\). Each action execution process first selects an eligible action \(o_{ij}\) for which the predecessor operation has been scheduled, and then randomly picks a set of compatible combinations with available machines under this operation.