## 4 Machine Learning Applications for Production Machines Real Time Control
Due to their ability to explicitly consider constraints and predictively optimize the controller output, MPC solutions are already well established in the industry. Important for the applicability of MPC are sufficient slow system dynamics so as to execute the model online during every sampling step. Reinforcement learning for process control appears to be a promising method to reduce online computation times. The m aim differences lie in the modelling approach, where RL algorithms learn the model through trial and error, whereas MPC requires a model developed with first principles or process identifications. Furthermore, the online execution time of model-free RL is faster than MPC, since only a forward run of the policy network is required, whereas the MPC requires solving an optimization problem at each execution. A general comparison between deep RL and MPC for optimal control is given by [PERSON] et al. [70] and is displayed in Table 4. More details about the comparison of MPC and RL can be found in [70; 71]. Due to the possibility of offline training, reinforcement learning is suitable for real-time control. Famous results of reinforcement learning were achieved by DeepMind beating human players in games such as Go [72]. However, these kinds of simulated environments are perfectly known and are therefore particularly suitable for the learning of reinforcement algorithms. In contrast to simulated environments, industrial processes suffer from high levels of imprecision and uncertainty (see sensor fusion; Section 2). The training requires a lot of data and interactions with the environment, which can be performed by interacting with a model. However, accurate models covering the entire manufacturing process over all parameter combinations are extremely complex, and experiments are costly. As discussed \begin{table}
\begin{tabular}{c c c} \hline \hline
**Comparison Criteria** & **Reinforcement Learning** & **MPC** \\ \hline Solution form & Neural networks & Model based optimization \\ \hline Online computation time & Low & High, especially for high-degree \\  & & non-linear systems \\ \hline Optimality-seeking capability & Near-optimal & Near-optimal with long \\  & & prediction horizons \\ \hline Generalization issue of & Reduced optimality & N/A \\ machine learning & & \\ \hline Handling of hard constraints & Under development & Yes \\ \hline Handling of modelling errors, & & \\ control delays, and/ & Better with larger errors & Worse if not robust MPC \\ or disturbances & & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of deep reinforcement learning and model predictive control for optimal control [72]. \begin{table}
\begin{tabular}{c c} \hline \hline
**Type** & **Control Method** \\ \hline Traditional & PID, fractional PID \\ Fuzzy Logic & Fuzzy controller \\ \hline \multirow{4}{*}{Reinforcement Learning} & Temporal difference (TD), \\  & SARSA (State–Action–Reward–State–Action) \\ \cline{1-1}  & Soft Actor–Critic (SAC) or \\ \cline{1-1}  & Asynchronous Advantage Actor–Critic (A3C) \\ \cline{1-1}  & \multicolumn{1}{c}{Q-learning, Double Q-learning} \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Overview of common and new control algorithms. in Section 3, AI-based modelling approaches could therefore be suitable to model the process and serve as an environment for the RL algorithm. In contrast to traditional control approaches, there are no stability guarantees for RL controllers. Compared to traditional control methods, the development of RL control algorithms is still in an early stage. Nevertheless, the general learning capability of RL is promising for the future. Furthermore, research is needed to search for methods to improve stability and convergence of RL algorithms. For further applications as control approaches, safe RL will be, in the future, an important topic to enhance the reliability of RL-based controllers [73,74].