## 4 Solving FJSPLSP-LTCC Using DRL
### MDP Formulation
#### 4.1.1 State  
At each decision step \(t\), a total of 13 state features are designed to capture the key aspects of the FJSPLSP-LTCC system, including the statuses of machines, jobs, and AGVs. These features form the state vector \(s_{t}\) observed by the agent. The definitions are as follows:  
(1) Average utilization rate of machines, denoted as \(URM_{ave}(t)\). \[URM_{ave}(t)=\frac{\sum_{m=1}^{M}URM_{m}(t)}{M} \tag{15}\]  
where \(URM_{m}(t)\) is the utilization rate of machine \(m\), calculated as the total processing time of all operations on machine \(m\) divided by its total running time up to time \(t\). (2) Standard deviation of machine utilization rate, denoted as \(URM_{std}(t)\). \[URM_{std}(t)=\sqrt{\frac{\sum_{m=1}^{M}(URM_{m}(t)-URM_{ave}(t))^{2}}{m}} \tag{16}\]  
(3) Average completion rate of jobs, denoted as \(CRJ_{ave}(t)\). \[CRJ_{ave}(t)=\frac{\sum_{i=1}^{I}CRJ_{i}(t)}{I} \tag{17}\]  
where \(CRJ_{i}(t)\) denotes the completion rate of job \(i\), defined as the ratio of completed operations to total operations. (4) Standard deviation of job completion rate, denoted as \(CRJ_{std}(t)\). \[CRJ_{std}(t)=\sqrt{\frac{\sum_{i=1}^{I}(CRJ_{i}(t)-CRJ_{ave}(t))^{2}}{I}} \tag{18}\]  
(5) Average remaining processing time of unfinished jobs, denoted as \(RT_{ave}(t)\). \[RT_{ave}(t)=\frac{\sum_{i=1}^{I}RT_{i}(t)}{RJ_{total}} \tag{19}\]where \(RT_{i}(t)\) denotes the remaining processing time of job \(i\), and is set to 0 if the job is already completed; \(RI_{total}\) is the number of unfinished jobs. (6) Total number of operations remaining for unfinished jobs, denoted as \(RO_{total}(t)\). \[RO_{total}(t)=\sum_{i=1}^{n}RO_{i}(t) \tag{20}\]  
where \(RO_{i}\) is the number of remaining operations for job \(i\). (7) Average current power level of AGVs, denoted as \(CPA_{ave}(t)\). \[CPA_{ave}(t)=\frac{\sum_{a=1}^{A}CPA_{a}(t)}{A} \tag{21}\]  
where \(CPA_{a}(t)\) is the current power level of AGV \(a\). (8) Standard deviation of AGVs current power level, denoted as \(CPA_{std}(t)\). \[CPA_{std}(t)=\sqrt{\frac{\sum_{a=1}^{A}(CPA_{k}(t)-CPA_{ave}(t))^{2}}{a}} \tag{22}\]  
(9) Average utilization rate of AGVs, denoted as \(URA_{ave}(t)\). \[URA_{ave}(t)=\frac{\sum_{a=1}^{A}URA_{a}(t)}{A} \tag{23}\]  
where \(URA_{a}(t)\) is the utilization rate of AGV \(a\), calculated as the total material handling time divided by its running time. (10) Standard deviation of AGV utilization rate, denoted as \(URA_{std}(t)\). \[URA_{std}(t)=\sqrt{\frac{\sum_{a=1}^{A}(URA_{z}(t)-URA_{ave}(t))^{2}}{A}} \tag{24}\]  
(11) Average number of charges for AGVs, denoted as \(NCA_{ave}(t)\). \[URM_{std}(t)=\sqrt{\frac{\sum_{m=1}^{M}(URM_{m}(t)-URM_{ave}(t))^{2}}{m}} \tag{16}\]0  
where \(NCA_{a}(t)\) denotes the number of times AGV \(a\) has charged up to time \(t\). (12) Standard deviation of AGV charge counts, denoted as \(NCA_{std}(t)\). \[URM_{std}(t)=\sqrt{\frac{\sum_{m=1}^{M}(URM_{m}(t)-URM_{ave}(t))^{2}}{m}} \tag{16}\]1  
(13) Total number of AGVs with power below \(s_{\text{rated}}\), denoted as \(TAB_{total}(t)\). \[URM_{std}(t)=\sqrt{\frac{\sum_{m=1}^{M}(URM_{m}(t)-URM_{ave}(t))^{2}}{m}} \tag{16}\]2  
\[TAB_{a}(t)= \begin{cases}1,&\text{if }s_{a}(t)<s_{\text{rated}}\\ 0,&\text{else}\end{cases}  \tag{28}\]  
The 13 state features designed in this subsection are based on a systematic approach, aiming to comprehensively characterize the dynamic characteristics of the FJSPLSP-LTCC system from both operational status and energy management perspectives. Metrics such as the mean and variance of machine utilization and job completion rates quantify production progress and load distribution, aiding the agent in identifying potential bottlenecks and resource allocation inefficiencies. Indicators including the average remaining processing time and the total number of unfinished operations reflect workload intensity and scheduling complexity, providing a basis for reasonable task prioritization to enhance overall scheduling efficiency.