## 3 Traditional Controllers: PID and MPC Closed-Loop Control Strategies
### Deep Learning Neural Network
#### 4.3.1 Generate Reward Function from the Cost and Constraint Specifications Defined in an MPC Object
* Penalty functions: \[y_{min}=[0\;0],\;y_{max}=[10\;52],\;mv_{min}=0,\;mv_{max}=1.1,\;mv_{ ratemin}=-1000,\;\text{and}\;mv_{ratemax}=\text{Inf}\]3 (37) \[y_{min}=[0\;0],\;y_{max}=[10\;52],\;mv_{min}=0,\;mv_{max}=1.1,\;mv_{ ratemin}=-1000,\;\text{and}\;mv_{ratemax}=\text{Inf}\]4 (38) \[y_{min}=[0\;0],\;y_{max}=[10\;52],\;mv_{min}=0,\;mv_{max}=1.1,\;mv_{ ratemin}=-1000,\;\text{and}\;mv_{ratemax}=\text{Inf}\]5 (39) and finally: \[y_{min}=[0\;0],\;y_{max}=[10\;52],\;mv_{min}=0,\;mv_{max}=1.1,\;mv_{ ratemin}=-1000,\;\text{and}\;mv_{ratemax}=\text{Inf}\]6 (40)  
To calculate the reward value as a result of the MATLAB function block illustrated in Figure 9h, the following relationship between all three components--reward, cost, and penalty--is used:  
\[y_{min}=[0\;0],\;y_{max}=[10\;52],\;mv_{min}=0,\;mv_{max}=1.1,\;mv_{ ratemin}=-1000,\;\text{and}\;mv_{ratemax}=\text{Inf}\]7  
For the actual control application under research, the specifications of the RL DLNN environment were the following:  
* The observations reference signals (\(Y_{1 sp}\) and \(Y_{2 sp}\)), output variables (\(T_{clw}\) and \(Level\)), and their integrals, were as shown in Figure 9j;
* The \(T_{clw}\) and \(Level\) signals were normalized by multiplying with the gain [1/10 1/52];
* The action of \(U_{com}\) and \(u_{EXV}\) was limited to between [0 1.1] for \(U_{com}\) and [0 1] for \(u_{EXV}\);
* The sample time and total simulation time were \(T_{s}=0.1\) s. In order to capture the full evolution of the dynamics for both plant outputs, the simulation time was set to \(\text{Tsim}=60\) s.  
The MATLAB Simulink simulations result is presented in \[y_{min}=[0\;0],\;y_{max}=[10\;52],\;mv_{min}=0,\;mv_{max}=1.1,\;mv_{ ratemin}=-1000,\;\text{and}\;mv_{ratemax}=\text{Inf}\]8  
The block parameters and the block diagram for simulation of reinforcement learning of the RL Agent using a Simulink model as a training and simulation environment are depicted in Figure 9f. The same block also generates the RL Agent greedy policy \(\pi_{0}\)(. ), as shown in Figure 9i. Figure 9h shows a screen capture of the MATLAB reward function, and Figure 9j presents a detail of the observation block that appears in Figure 9b,c. The RL agent chosen in the case study was a twin-delayed deep deterministic policy gradient (TD3) agent [20; 21] that used two parametrized Q-value function approximators to estimate the value (that is, the expected cumulative long-term reward) of the policy, as shown in Figure 9f,i. To model the parametrized Q-value function, a neural network that was used had two inputs (the observation and action) and one output corresponding to the value of the policy \(\pi_{0}(\ )\) when taking a given action from the state corresponding to a given observation, as shown in Figure 9i. Before training the RL Agent, each network path of the DLNN was defined as an array of layer objects, and names assigned to the input and output layers of each path in order to connect the paths. Thus, a layer graph object was created and layers added to generate the criticNet, as is shown in Figure 10a. The critic function objects were created using a dedicated MATLAB _rlQValueFunction_ command. To make sure the critics had different initial weights, each network was explicitly initialized before being used create critic 1 and critic 2 [20; 21]. The TD3 agent learns a parametrized deterministic policy over continuous action spaces. The policy is learned by a continuous deterministic actor that takes the current observation as input and returns as output an action that is a deterministic function of the observation. The neural network that is used to model the parametrized policy within the actor has one input layer (which receives the content of the environment observation channel) and one output layer (which returns the action to the environment action channel) [20; 21]. An actor network named actorNet is created; its layer graph is shown in Figure 10b. The deterministic actor function is generated for the purposes of modeling the policy of the RL Agent.