## 1 Introduction
Moreover, it is pointed out that classical methods of allocating processes to resources, based on priority dispatching rules or evolutionary algorithms, are not adapted to the dynamic environment in modern production [19; 20]. It becomes necessary to enrich CPPSs with solutions, e.g., implementing advanced artificial intelligence and machine learning methods. In such a combination, the DT reflecting the behavior of the production system can play the role of a virtual environment used in the training phase. It is especially important in relation to today's requirements, where companies have to deal with mass customization, complex technologies, and shortening product life cycles. They must, therefore, be able to operate under highly dynamic market conditions and meet increasing sustainability standards. In recent years, there has been a significant increase in interest in the use of reinforcement learning algorithms, and also in the area of production systems [19]. Figure 2: Development of production management and control strategies. Regardless of the production planning or control problem considered, the architectures of RL agent interaction in the training process proposed in the literature indicate the need to develop an environment (appropriate for a given problem and system) in which the Deep RL agent can interact. Most non-simulation-based approaches to solving production planning problems use analytical models. These models often describe a problem through the objective function, as well as constraints to define the problem's structure [21]. Developing analytical models of real production planning, scheduling, and sequencing problems are complex tasks [21]. Analytical models are now often the basis for modules that determine, for example, the production schedule and production indicators and, as a result, the value of the reward function in the communication process with the RL agent [22; 23; 24; 25]. In many cases, the interaction environment is based on dedicated computational algorithms and mathematical models, where it is necessary to develop formulas that calculate subsequent states and rewards based on the state and action (see, e.g., [26; 27; 28; 29]). However, compared to the DT based on discrete simulation models considered in this paper, modifying analytical models, especially related to adding a new type of constraint or combining models covering different subsystems or production planning areas, is usually also a complex and time-consuming task. The DT, thanks to the direct access to operational data and the use of communication techniques for data acquisition, allows a fast semi-automatic or automatic adaptation of the simulation model. In addition, 3D simulation models also take into account intralogistics subsystems and/or the availability of operators and their travel times, which are usually omitted in dedicated analytical models. They can be quickly adapted to changes resulting from emergencies that require route changes and the use of alternative resources, supply chain disruptions at various levels, and changes in demand. This also meets the requirements of the latest concepts of the so-called Symbiotic Simulation System [30]. Analytical approaches, due to the need to update mathematical models and formulas (the agent's interaction environment) in the event of changes (often even minor ones) in the real production system, may be too inflexible in practical industrial applications. This gap motivated research in the area of using DTs of production systems based on modern computer discrete simulation systems as an environment for the interaction of reinforced machine learning agents, the results of which are presented in this paper. Additionally, many studies provide the parameters of the RL algorithms used (e.g., [31]). Still, there is no analysis or discussion of the impact of their values on the results obtained or the speed of the training process.