# Digital Twin and Reinforcement Learning-Based Resilient Production Control for Micro Smart Factory
## 4 Method for Resilient Production Control in Modular Manufacturing System
### Policy Network for Production Control in Micro Smart Factory  
The policy network is the result of the proposed method. As described above, the RL policy network \(\pi^{R}(a|s)\) is learned based on the virtual event trace, which is a pair of states \(s\) and action \(a\). The initial virtual event trace is reported by the DT that reflects the current policy function \(\pi^{C}(a|s)\). In this study, the RL technique for learning is selected for the dueling network technique, which was proposed by [PERSON] et al. [50]. The dueling network technique is an advanced Q-learning technique and has the advantage that the policy network and value network are in the same network. Additionally, the Q-learning-based techniques can be controlled in discrete time and coordinated with discrete event simulation [51; 52; 53]. Moreover, the dueling networks separately learn \(V(s)\), which is determined only by the state, and the advantage \(A(s,~{}a)\), which is determined according to actions, to derive \(Q(s,~{}a)\). This approach has the advantage of being able to divide the information  
Figure 3: CPPS architectural framework for resilient production control in MSF. of the Q-function into the portion determined only by the state, and that is determined according to actions. Furthermore, in contrast to a deep Q-network (DQN), it learns the combined weights that lead to \(V(s)\) at every step regardless of action. It also requires fewer episodes to complete learning compared to a DQN, which results in better performance as the number of action types increases [50; 52; 54; 55]. With the dueling network exhibiting the abovementioned advantages that make it suitable for application to this method, the Q-function of the RL policy network is presented in Equation (1). In addition, the RL policy network \(\pi^{R}(a_{t}|s_{t})\) selects the action type with the highest Q-function among the actions in step \(t\) when the decision of the tower handler in MSF is required. This policy network is designed as a single agent, and it is not necessary to consider coordination between multi-agents. \[Q(s,a_{t})=A(s,\ a_{t})+V(s) \tag{1}\]  
\[\pi^{R}(a_{t}|s_{t})=MAX(Q(s_{t},a_{t}))\ (\forall i,\forall j,\forall t) \tag{2}\]  
As described in Equation (3), the action \(a_{t}\) of each neuron indicates the priority \(p_{m,t}\) for what-next, which is for the selection of the part in buffer. Additionally, the configuration of MSF is enabled to restructure, and the number of selectable resource types can be changed. Therefore, because the capacity of all resource types is equal to 1 and the time of material handling operation is not significant, the number of resource instances can be projected to the machine capacity of each resource type \(u_{k}\). Until the entire resource instances are occupied or all feasible actions are finished, the material handling operation from space \(m\) is performed according to the priority \(p_{m,t}\). \[\left\{o_{k,t}^{r}\leftarrow o_{k,t}^{r}+x_{k,t}\Big{|}\Big{(}o_{k,t}^{r} \leq u_{k}\Big{)}\vee(y_{k,m,t}>0)\right\}\ \left\{ \begin{array}{l}x_{k,t}=1\ p_{m,t}=MAX_{k}(p_{m,t})\\ x_{k,t}=0\ p_{m,t}\
eq MAX_{k}(p_{m,t})\end{array} \right. \tag{3}\]  
To meet the requirements of the MSF, the state is selected by considering production and delivery. State \(s\) includes the remaining production volume \(v_{m,t}^{r}\), remaining due date \(d_{i,t}\), the number of WIPs in each resource type \(o_{k,t}^{r}\), the number of WIPs in buffer \(o_{t}^{b}\), machine availability \(y_{k,t}\) that includes machine failure, processing time \(t_{i,j,k}^{p}\), and setup time \(t_{i,j,k}^{s}\). As illustrated in Equation (4), the information indexed by part \(i\) and process \(j\) is pre-processed to information with indexing space \(m\). Thus, the state \(s\) is projected to two dimensions for the efficient representation.