# Digital Twin and Reinforcement Learning-Based Resilient Production Control for Micro Smart Factory
## 4 Method for Resilient Production Control in Modular Manufacturing System
### Cyber-Physical Production System Architectural Framework for Resilient Production Control  
The proposed method applies DT and RL to satisfy the abovementioned requirements. The DT can provide the evaluation result to the learning process of the policy network. The policy network is an RL-based network model that selects action \(a\) according to state \(s\) to maximize reward \(r\). In addition, the RL policy network is denoted as \(\pi^{R}(a|s)\) and is learned from the initial solution \(\pi^{C}(a|s)\). Through the learning process, the RL policy network \(\pi^{R}(a|s)\) is adjusted to maximize reward \(r\), and the virtual event logs for this network are returned by DT. Moreover, the RL technique enables the estimation of parameters that are suitable for the product diversity in the production operation, the revised plan and schedule, and the current situation of the physical work center.  
In the proposed method, the DT plays a role in providing the virtual event trace and KPI for learning the RL policy network \(\pi^{R}(a|s)\). The virtual event trace is the pair of action \(a\) and state \(s\) during the DT simulation. RL uses state \(s\) as inputs and action \(a\) as an output for indicating the derived entity in the MMS. In addition, the reward \(r\) is also required for DT application to maximize the specific KPIs from the production control perspective. Moreover, the current information from the physical work center needs to be synchronized to minimize the gap between DT and the physical work center. If the current information, such as progressed production volume, WIP, and machine status, is not considered in the DT simulation, the simulation result might support the learning of RL policy network \(\pi^{R}(a|s)\) with the inappropriate solution space.  
To satisfy the abovementioned requirements, the DT application is designed, as shown in Figure 3. The architectural framework follows an AAS model with SOA principles. To enable the interoperability in the heterogeneous development environment, the entire system considers loosely coupled integration based on web services. Following the CPPS architectural framework, which was proposed by [PERSON] et al. [7], the advanced planning and scheduling (APS) application and device control application are included. Moreover, the P4R information model is applied for efficient information management and application of 'type and instance' concept based on the VREDI [9]. The following are the detailed descriptions of elements in this architectural framework:  
* Component manager: This element is a centralized coordination component and takes the role of a service bus in the SOA principle. The component manager is a subject of vertical integration and horizontal coordination and controls the entire service composition and engineering applications.
* DT application: This is a core element in this architectural framework. The operation module creates, synchronizes, and utilizes DT with DT engine. This application provides simulation-related technical functionalities and visualization according to the request from the service composition.
* Policy generation module: This element learns and deploys the RL policy network using the virtual event logs from the DT application. The RL policy network is learned to maximize reward \(r\) and is deployed in the format of a systematic logic library (SLL).  
APS application: This application returns the production plan and schedule alternative that needs validation and objective values. The APS algorithm is necessary to establish alternative and simulation-based optimization, metaheuristics, and heuristics can be an option for the core functional engine.  
Device control application: This element extracts the path, kinematics, and estimation related to the robotics configuration. Based on the locations of the MHRs, the required extraction is operated to use forward and backward functions in the simulation.