## 3 CPPS with DES-Based DT and DRL Agent Architecture  
The central element of the proposed CPPS architecture is the DT based on discrete event 3D simulation models. To reflect the dynamics of the real system, the simulation model is supported by automatic generators of simulation models used in methods of semi-automatic creation and automatic updating of simulation models. Generators can be based on data mapping and transformation methods using a neutral data format and the XSL and XSL languages [53; 54]. Process data from sensors and programmable logic controllers (PLC) are transferred to the SCADA system and then to MES/ERP systems. From there, through data exchange interfaces and integration modules, the data required in the process of creating and updating the simulation model can be mapped and, depending on the needs, transformed into a neutral data model, and, then, using internal simulation software languages, transformed into code that creates a new or modifies an existing simulation model, as shown in Figure 4. This solution allows for easy practical implementation of the method because it can be used with any management system and any computer simulation system (e.g., Enterprise Dynamics, Arena, FlexSim).  
Of course, as mentioned in Section 1, the transition from a digital model to a digital twin requires that the digital model has mechanisms for a bi-directional data flow between existing physical and virtual/digital objects. For this purpose, a model-control system architecture is proposed using the so-called emulation of industrial automation system components tools directly in the simulation software. It allows to create direct connections between digital, virtual objects of the simulation system and external physical sensors, PLC controllers, or communication clients/servers. The architecture of such a solution is shown in Figure 5. The combination of solutions shown in Figures 1 and 2 allows the model to be updated with data from production management systems as well as directly from sensors or PLC controllers (e.g., using OPC or Modbus protocols). This functionality is provided by, e.g., the emulation tool of the FlexSim simulation system [55]. It also makes it possible to send responses back from the DT to the planning or control system.  
The next stage includes supplementing the cyber layer with modules enabling the use of selected RL algorithms and enabling a two-way connection between the DRL agent and the simulation model. The simulation model serves as the environment for interaction, learning, and verification of the agent's policy. Most of the algorithms mentioned in Section 2 are publicly available in open-source libraries containing their implementations. These include popular Python libraries: OpenAI Baselines, Stable Baselines3, or keras-rl. Additionally, an API must be provided for communication between DRL agents and the environment. In the case discussed, the environment with which the agent communicates is a simulation model, so the simulation software will constitute a non-standard environment for which the design of appropriate communication and configuration modules will be required. Such modules should not only implement two-way communication, but also provide access to model configuration functions that allow, among other things, the definition of the observation space, the space of actions taken by the agent, and the reward function  
Figure 4: Digital model—indirect data exchange/transformation.  
along with its standardization to the values required by individual algorithms. Appropriate tool packages that provide the functionality described above can be implemented directly in commercially available simulation programming using API mechanisms. The architecture of such a solution is shown in Figure 6.  
A practical example of the implementation of this concept is shown in the next section.