## 2 Related Work
### Approaches to Improve the Interaction of Supply Network and Production Planning and Control
#### 2.2.3 Machine Learning  
A system can be described as learning if it is improving its performance on future tasks after making observations about its environment [50]. In such a system, ML aims at generating knowledge from data by developing a complex heuristic model from training data [51]. The automated development of models from data by ML is also the main difference to earlier approaches of artificial intelligence, which were based on manually constructed knowledge bases such as predefined rules in case-based reasoning [51; 52; 53]. Furthermore, ML also differs from mathematical optimization since it allows generalization to unknown data [54]. Therefore, the use of ML is particularly worthwhile when, due to the complexity of the problem, not all possible situations and all changes over time can be anticipated. This is also the case when, due to the complex relationships and interactions between parameters, it is unclear what the algorithmic solution to the problem must look like [50]. In manufacturing systems this is related to the combinatorial complexity and the dynamic complexity [55]. Experience has shown that ML is much more suitable than conventional methods of data analysis, especially for more than 15 dimensions in the data sets [56]. As described by Figure 3, ML methods can be categorized per type of feedback into unsupervised learning, supervised learning and Reinforcement Learning (RL) [50]. ML is especially suitable for tasks in manufacturing, since due to its complexity, manufacturing can behave unexpectedly when manual interventions are made; under- or over-steering is possible if parameters are incorrectly manually adjusted [2]. Each of the methods is suitable for different learning tasks in manufacturing, such as unsupervised learning for fault detection [57], supervised learning for process control [58] or load forecasting [59] and RL for order dispatching [60]. RL is especially suitable if labeled data is not available for a sequential learning problem and agents and their environments as well as reinforcements for actions can  
Figure 3: Methods of machine learning [43]. be modeled [50; 61; 62; 63]. In RL, an agent learns a strategy of how to act to maximize a cumulative reward. This strategy is learned from a series of situations and taken actions, as well as their corresponding reinforcements, which may be either rewards (positive) or punishments (negative). In the case of punishments, reward maximization means that the agent strives to minimize the cumulative punishment. The agent is not told which actions to take, but instead must discover which actions yield the most cumulative reward by trying them. Therefore, RL is suitable, if a solution for a problem can be found by trial and error, meaning learned through receiving feedback from an environment. Additionally, it can handle delayed feedback [61; 62]. As a prerequisite for the application of RL, it is necessary to describe the problem as a Markov Decision Process (MDP). In addition, a simulation model is most often necessary. This is used to enable the agent to perform the trial-and-error learning safely without interfering with a real environment [61; 63]. The MDP describes states \(S_{t}\) in a state space \(S\), actions \(A_{t}\) in an action space \(A\) and rewards \(R_{t}\) for a learning agent interacting with its environment at each of a sequence of discrete time steps \(t\) (cf. Figure 4). At each time step \(t\), the agent receives some representation of the environment's state, \(S_{t}\in S\), and on that basis selects an action, \(A_{t}\in A\). One time step later \(t+1\), and partially caused by its action, the agent receives a numerical reward, \(R_{t+1}\in R\), and finds itself in a new state, \(S_{t+1}\)[61]. A good overview of existing approaches using RL in PPC is given by [PERSON] et al. [64]. Most approaches focus primarily on order dispatching on machine level in rather simple environments and on maximizing resource related key performance indicators such as use.