## 3 Concept
### Reinforcement Learning-Based Controlling Element  
The ACSN becomes adaptive and learning fusing RL in the controlling element (cf. requirements 3 and 4). Therefore, no strategy is given for deciding on actions, but the ACSN controller derives them from observing its environment, in this case, the effects of its decisions on expected delivery dates in manufacturing. Consequently, the system can learn without labeled data and to adjust even complex manipulated variables with many dependencies automatically. Furthermore, effects can be handled that are only visible after orders have been fully processed in manufacturing. The quality and performance of the ACSN controller's decisions are increasing over time. To realize a RL-based ACSN controller, an initial design, a training phase as well as an application of the trained agent is necessary (cf. Figure 6).  
As a foundation for RL, the MDP must be designed. To fully represent the environment's status, the state space \(S\) needs to contain data on production plan, supply network events as well as manufacturing feedback data. Here, the expected delivery dates are the most important data points of the manufacturing feedback data. Actions in action space \(A\) describe the ACSN controller's scope of action to influence the production plan and consequently create an adapted schedule as input for manufacturing control. The ACSN controller is rewarded for taking actions leading to increased delivery reliability and, thereby, increasing the supply networks resilience. The outcome of this setup is the agent forming the ACSN controller. This includes an initial set of hyper parameters to use within the agent's algorithm. Furthermore, a simulation model of the manufacturing system for the agent to interact with safely in training is needed. Ideally, an existing simulation model of a digital twin can be used to represent the manufacturing system as realistically as possible. If no such model exists, it must be built. Guidelines for this can be found in the literature [65]. An interface between the simulation model and existing manufacturing IT  
Figure 6: Reinforcement learning-based controlling element for an adaptive control loop for an integrated supply network.  
systems is important, in order to work with data in the simulation model that is as close to reality as possible.  
To train the agent, it continuously interacts with the simulation model of the manufacturing system to evaluate various actions and their respective rewards. In doing so, the agent must always tradeoff between exploitation (maximize reward) and exploration (discover environment) when selecting an action. Therefore, the agent must be given a strategy such as \(\epsilon\)-greedy or decaying-\(\epsilon\)-greedy [61]. Consequently, training is an iterative process varying hyper parameters that must be repeated many thousands of times until the reward is maximized. However, in addition to statistical validation, the suitability of the agent for the problem at hand must be checked, in particular before transferring it to its application. Doing so, the simulation model is used to simulate the same scenario with and without intervention by the agent. If the agent performs better regarding delivery reliability, it can be transferred to the application.  
Subsequently, the ACSN controller can be deployed to its application in a real manufacturing environment. Consequently, the agent no longer explores, but selects specific actions for exploitation. Instead of interacting with a simulation model, the agent uses manufacturing control as an interface to interact with the manufacturing system. Actions are performed by adapting control values in manufacturing control, states are constructed by data gathered from various manufacturing IT systems.