## 4 Methods
### MDP Model  
The MDP model serves as a bridge between DRL and scheduling problems. Resolving the scheduling problem in panel block assembly can be viewed as determining the orientations of the disjunctive graphs. The underlying MDP model we establish is as follows:  
State: Due to the limitations of existing methods in providing a comprehensive and rational representation of the current scheduling state in the planar segment assembly line and considering the inherent characteristics of the scheduling environment, we introduce the concept of disjunctive graphs for representation. Disjunctive graphs encompass all the information related to the scheduling environment, including the processing status of each workstation and the number of segments currently being assembled. The state consists of the processing status of each workstation and the blocks placed on the panel block assembly line. To capture the current scheduling state at step \(t\), we utilize the disjunctive graph \(G(t)=(N,\;C(t),\;D(t))\), where \(C(t)\) comprises the set of all connecting arcs up to step \(t\), and \(D(t)\) represents the remaining disjunctive arcs in the graph. The  
Figure 4: Scheduling instance of dimensions \(5\times 4\). initial state \(s_{0}\) represents the initial state of the panel block assembly line before scheduling begins, while the terminal state \(s_{t}\) corresponds to a feasible solution when scheduling is complete, at which point \(D(t)=\varnothing\), indicating that all disjunctive arcs have been assigned orientations. As the assembly process progresses, the disjunctive graph provides varying state descriptions of the current scheduling environment, and in turn, the changes in the environmental state lead to different disjunctive graphs. Action: The actions of the intelligent agent involve selecting the block to be placed in the first workstation during the assembly process of the panel block assembly line. The design of the action space should thoroughly consider minimizing idle time across workstations and enhancing their utilization rate. State transition: As the scheduling environment of the panel block assembly line constantly evolves, the scheduling state progresses from state \(s_{t}\) to the next state \(s_{t+1}\). When determining the next action \(a_{t}\) to schedule, our first step is to identify the earliest feasible time to allocate \(a_{t}\) on the required workstation. Subsequently, we update the direction of the disjunctive arcs for the workstations based on the current temporal relationships, generating a new disjunctive graph as the new state \(s_{t+1}\). Reward: The rational definition of rewards serves as a prerequisite for successful learning in reinforcement learning. Rewards should be defined in direct relation to the objective of minimizing _makespan_. Therefore, we initially compute the difference between partial solutions at two consecutive steps, denoted as \(U_{t}=C(s_{t+1})-C(s_{t})\), where \(C(s_{t})\) is defined as \(C(s_{t})=max_{ij}\big{\{}LB_{t}(O_{ij})\big{\}}\), representing a lower bound on the _makespan_. We assign the negative value of the difference \(U_{t}\) as the immediate reward for each step \(t\), i.e., \(R(a_{t},s_{t})=-U_{t}\). In other words, the cumulative reward corresponds to the negative makespan when all operations are scheduled. Policy: For state \(s_{t}\), the stochastic policy \(\pi(a_{t}|s_{t})\) generates a probability distribution over actions \(a_{t}\), with action selection prioritized based on the probability distribution. Graph embedding: Graph isomorphic network (GIN) [42] is a kind of deep neural network [43; 44] capable of learning representations of graph-structured data, and it is the latest variant of graph neural network (GNN). The disjunctive graph we have constructed encompasses all the information regarding the scheduling states, including the processing times of blocks at each workstation and the order of block processing.