## 3 Traditional Controllers: PID and MPC Closed-Loop Control Strategies
### Model Predictive Control Based on Centrifugal Chiller MIMO State-Space Representation
#### 4.1.1 Reinforcement Learning Process--Description  
Reinforcement learning (RL) involving multi-layered deep learning neural networks is a well-used and well-suited method for research and development in modern artificial intelligence, as is shown in [18]. More precisely, RL is a modeling process in which an agent (controller) learns to make decisions by interacting with an unknown environment through trial and error, as shown in Figure 8a [18,19]. Mathematically speaking, the RL modeling process (algorithm) depicted in Figure 8a is typically based on a Markov decision process (MDP). In this type of process, the RL agent block receives in its current state the observation St as a result of the interaction with the unknown Environment, performing an action that it then sends to the same Environment. The latter reacts by sending a scalar reward Rt to the same RL block for a new transition to the next state St+1, according to the conditional probability of the Environment dynamics; p(St+1 \(|\) St, At). The RL agent attempts to learn a policy \(\pi(\text{At}\,|\,\text{St})\) or map from observations to actions, in order to maximize its returns (expected sum of rewards). In RL (as opposed to optimal control), the algorithm has access to Environment dynamics p(St+1 \(|\) St, At) only during the sampling time. During the training process, the RL blocks learn the unknown environment and \"make a series of decisions to maximize the cumulative reward for the task without human intervention and without being explicitly programmed to achieve the task\" [18]. From the perspective of a system-oriented approach, the diagram shown in Figure 8a shows a general representation of a reinforcement learning (RL) algorithm. Figure 8b reveals its equivalence to a traditional closed-loop control strategy used in system applications [19]. In other words, the key role of the RL policy in Figure 8a is to observe the unknown environment (state St) and generate actions (At) to \"complete a task optimally\", similar to a traditional controller operating in a control system application. Of course, the RL algorithm comes with a valuable improvement to the RL policy, known as policy updating. More specifically, RL can be seen as a mapping of a conventional feedback control system depicted in Figure 8b, and the correspondence between both diagrams is well presented in [19]. The concept is based on _\"rewarding or punishing an agent's performance in a specific environment\"_[24]. According to this definition, a state is a description of the environment that provides the agent with helpful information for taking a decision at each time step. The agent receives observations and a reward from the Environment and the actions generated are sent into the same Environment. Reward is a measure of the achievement of an action on the completion of the task goal; i.e., the reward signal evaluates the outcome of past actions [18]. RL policy is based on cost function to map each state to the optimal action in order to maximize its reward function during the episode [17]. This RL policy can be a deterministic policy, defined as \(A_{t}=\pi(S_{t})\) for \(\pi:\mathcal{S}\rightarrow\mathcal{A}\), or stochastic policy, described as \(A_{t}\ \sim\pi(\,\cdot|S_{t})\) for \(\pi:\mathcal{S}\rightarrow\mathcal{P}(\mathcal{A})\), where \(\mathcal{P}(\mathcal{A})\) is a set of probability distributions over actions. For an MDP, these probability distributions have an interesting feature; they factorize over trajectories [17; 19]:  
\[p(S_{1},A_{1},\ldots,\ S_{n},\ A_{n})=p(S_{1})\pi(A_{1},S_{1})\ \mathcal{P}(S_{2}|S_{1},\ A_{1})\pi(A_{2},S_{2})\ \ldots\ldots\mathcal{P}(S_{n}|S_{n-1},\ A_{n-1})\pi(A_{n},S_{n}) \tag{18}\]  
Thus, this feature is useful in the field of control applications, since RL policies need only to consider the current state, which is a strong consequence of the Markov assumption and full observability [23]. Otherwise, if the RL environment is partially observable, then the RL policy must depend on the history of observations.