## 3 Traditional Controllers: PID and MPC Closed-Loop Control Strategies
### Model Predictive Control Based on Centrifugal Chiller MIMO State-Space Representation
#### 4.1.1 Reinforcement Learning Process--Description
It is important to note that the RL deep learning method differs from supervised learning because it does not require correct sets of actions and labelled input/output pairs [24; 25; 26; 27]. The general reinforcement learning structure shown in Figure 8a is described by the following Equations [17]:  
\[A_{t}\ \sim\pi(A_{t}|S_{t}) \tag{19}\]  
\[S_{t+1}\ \sim f_{state}(S_{t+1}|S_{t},A_{t}) \tag{20}\]  
\[R_{t+1}=f_{reward}(S_{t},A_{t},S_{t+1}) \tag{21}\]  
\[R=\sum_{t=0}^{t=0}\gamma^{t}R_{t+1} \tag{22}\]  
\[Q^{new}(S_{t},A_{t})=Q^{old}(S_{t},A_{t})+\alpha\left(R_{t}+\max_{d^{\prime} \in\mathcal{A}}Q^{next}\big{(}S_{t+1},\ a^{\prime}\big{)}-Q(S_{t},A_{t}) \right)=Q^{old}(S_{t},A_{t})+\alpha(Target-Prediction) \tag{23}\]  
where \(\pi(A_{t}|S_{t})\) is the policy at time t, \(A_{t}\) denotes the action of time t, \(S_{t}\) is the state (observation) at time t, \(f_{state}\) and \(f_{reward}\) are the transition functions from time \(t\) to \(t+1\), and Q is the state-action value function. The so-called Q function, \(0\leq\gamma<1\) is an important hyperparameter called the discount factor, which determines how much we care about rewards now versus rewards later; more precisely, this is an exponential decay factor, which means longer term planning is harder. Also, \(0<\alpha<1\) represents the learning rate--a value that controls how quickly an agent updates its Q-values based on new information, essentially determining how much weight is given to newly acquired experiences compared with previously learned information; a higher value for \(\alpha\) means faster learning and more significant updates to Q-values with each new interaction. When adopting a Monte Carlo technique to estimate the Q-values, every iteration of the Q-value requires updating Q for every state. Monte  
Figure 8: Reinforcement learning architecture [18; 19]: (**a**) RL process; (**b**)standard feedback closed-loop control schematic. In this figure, the blue color is used to identify the Agent and Environment blocks, the color green to show the direction of the information flow in the closed-loop of traditional system, and the red color is to highlight that the parameters of the controller are adjustable and a particular Reward function, respectively. [PERSON] estimation of the expectation \(\mu=\mathbb{E}[X]\) repeatedly samples X, and the convex relationship [23] is updated as follows:  
\[\mu\leftarrow\mu+\alpha(X-\mu) \tag{24}\]  
In this relationship, the X dynamics is required to evaluate the expectation \(\mu\). The \(\varepsilon\)-greedy policy that appears in Figure 9f for a block of parameters of the TD3 RL agent is defined as a policy that chooses \(argmax_{\alpha\in\mathcal{A}}\text{Q}(\text{S},\ \text{A})\) with probability 1-\(\varepsilon\) and a random action with probability \(\varepsilon\). The typical value for the parameter \(\varepsilon\) is 0.005 [23]. Figure 9: Cont. Figure 9: _Cont._  
Combining all the information on Monte Carlo estimation of the expectation \(\mu\), a helpful pseudocode algorithm for implementers called Q-learning with \(\varepsilon\)-greedy policy is given in [23]. The \(\varepsilon\)-greedy policy is a valuable mechanism for managing the exploration-exploitation balance, as has been stated [23]. It operates according to the following equation:  
\[\pi_{\epsilon}(\mathcal{S};Q)=\left\{ \begin{aligned} & argmax_{a\in\mathcal{A}}Q(S_{t}, \text{ A})\text{ with probability }1-\varepsilon\\ & Uniformly\ random\ action\ in\ A\ with\ probability\ \varepsilon\end{aligned} \right. \tag{25}\]  
Due to its simplicity, the \(\varepsilon\)-greedy policy has become the most widely used method to balance exploitation and exploration. It ensures that most of the time (probability 1-\(\varepsilon\)), the RL agent exploits its incomplete knowledge of its environment by choosing the best action, and occasionally (probability \(\varepsilon\)) explores other actions. There may be some good actions that the agent will never find without exploration [23].