# Digital Twin and Reinforcement Learning-Based Resilient Production Control for Micro Smart Factory
## 1 Introduction
It is necessary to enable efficient production even after a restructured product in production is changed, and production plan and schedule are revised [10,14,17,19]. However, the core functional requirements for resilience cannot be achieved by the traditional heuristic rule [19]. Therefore, a novel method for the production control of MMS needs to be proposed. The reinforcement learning (RL) technique enables the adjustment of parameters by repeating episodes to learn the policy network [19,10,21]. This technique is often applied to select a robust action in response to stochastic arrival by replacing the dispatching rule in a work center [19,22]. To support the adjustment process without user intervention, a model that reflects the accurate systematic behavior of a work center is required [19,23,24]. To overcome the limitations of CPPS in MSF, we propose a digital twin (DT) and RL-based design. This method establishes and adjusts the required parameters for production control in MSF when they need to be revised. The RL supports this establishment and adjustment of parameters through the learning process. Moreover, the KPI, which represents the configuration of the physical work center, reflecting functional units, and synchronizing parameters, properties, and current status, can be measured based on the DT simulation. Furthermore, the asset administration shell (AAS) model is applied and inherited by this method for interoperability between DT and RL. The core contents of this study are as follows:  
1. The technical requirements for designing the DT and RL-based production control methods to achieve resilience are defined. To define these requirements, the general process of FaaS platform and the existing research studies on FaaS and MSF are analyzed. 2. The CPPS architectural framework that includes the proposed method is revised and proposed. In this CPPS architectural framework, the essential components for enabling the proposed method are also suggested. These components coordinate with the components for the proposed method. 3. The policy network is designed to provide the appropriate action to the specific state for maximizing the reward. The action is defined with the concept of priority rule for the efficient replacement of the existing dispatching rule in MSF. Further, the dispatching rule is designed to ensure robustness and resilience upon changes to configuration and production operation. 4. Horizontal coordination, which is the service composition between the technical functionality of DT application and RL technique, is designed to enable the RL policy network for MSF. This coordination considers the advanced characteristics of DT that can reflect the current status of MSF. Moreover, the advantage of RL, which includes efficient adjustment for production control, is also reflected in the design. 5. An industrial case study in MSF is performed to verify and validate the proposed method. Three experiments related to the industrial case study are conducted to confirm whether technical requirements are satisfied.