# Digital Twin and Reinforcement Learning-Based Resilient Production Control for Micro Smart Factory
## 6 Conclusions  
To improve the CPPS for enhancing the process and systematic efficiency of MSF, the DT and RL-based resilient production control methods are proposed in this paper. This method enables learning of the RL policy network that replaces the dispatching rule in the post-processing station of MSF. To design an efficient method, the technical requirements are defined. Because of the restructuring characteristic of MMS, the robustness needs to be considered. Additionally, the MTO production environment of personalized production increases the complexity of MSF. Moreover, the technical functionalities of CPPS in MSF must be considered in the design to achieve resilience. Furthermore, dynamic information, such as progress production volume, WIP, machine status, and changed situation, needs to be synchronized in the DT. With the technical functionalities in CPPS, this method is implemented based on the coordination between the DT application and policy network construction module. The DT application creates, synchronizes, and utilizes the DT for providing DT simulation as its technical functionality. The DT simulation provides the virtual event logs for supporting the learning process of the RL policy network. In contrast, the proposed policy network construction module learns the RL policy network using the dueling network technique. Based on the action, state, and reward in the virtual event logs, the RL policy network is learned and applied. The creation procedure of DT application reflects the RL policy \begin{table}
\begin{tabular}{c c c c} \hline \hline  & \multicolumn{3}{c}{**Improvement Rate**} \\ \hline
**Experiment** & \(C_{max}\) & \(\sigma(c_{i})\) & \(k\) \\ \hline Restructure of MMS & 2.585 & 6.456 & 13.953 \\ Resilience in type phase & 3.015 & 8.325 & 9.677 \\ Resilience in instance phase & 4.617 & 17.468 & 23.529 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Result of application of the proposed method (unit. %). \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Proposed Method**} & \multicolumn{3}{c}{**Existing Heuristics**} \\ \hline
**Case No.** & \(C_{max}\) & \(\sigma(c_{i})\) & \(k\) & \(C_{max}\) & \(\sigma(c_{i})\) & \(k\) \\ \hline
1 & 42.534 & 0.455 & 10 & 44.743 & 0.561 & 8 \\
2 & 42.326 & 0.436 & 8 & 43.956 & 0.549 & 12 \\
3 & 41.892 & 0.426 & 5 & 44.113 & 0.488 & 10 \\
4 & 43.074 & 0.448 & 9 & 43.278 & 0.524 & 9 \\
5 & 41.846 & 0.431 & 7 & 45.829 & 0.539 & 12 \\ \hline Average & 42.334 & 0.439 & 7.8 & 44.384 & 0.532 & 10.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Result of the experiment for resilience in instance phase (unit: hour). network repeatedly, and the utilization procedure of the DT application evaluates the RL policy network. The proposed method has several aspects of originality, contribution, and findings. This method is an early case of coordination between DT and RL. Using the advanced characteristics of DT, the RL-based production control, which uses the traditional DES, can enhance its robustness and efficiency. The advanced characteristics are vertical integration and horizontal coordination and exhibit the advantage of better representing the environment from a learning perspective. In addition, this study is also an early case of applying priority concepts to decide what-next/where-next with DT and RL. Moreover, the event definition with the CPPS architectural framework can be one of the contributions of the proposed method. The abovementioned aspects were verified and validated in the three experiments. Furthermore, the proposed framework and concept can be extended to an efficient solution in various manufacturing domains because the priority rule concept is frequently applied. As a further study, the event definition in the concept of end-to-end integration needs to be enhanced. This enhancement needs to consider the business and manufacturing process perspectives in the entire supply chain of personalized production.