## 1 Introduction
### Reinforcement Learning--Q-learning  
Systematic research in what is today called the reinforcement learning (RL) field started in late 1979 [15]. The RL approach is based on the interaction between an agent and his environment (Figure 1). The agent is something that can see the status or configuration (\(s\in S\)) of the environment and that can perform actions (\(a\in A\)) to change it. Moreover, upon application of an action, the agent receives from the environment an evaluative feedback signal. This signal is called reinforcement and can be a reward or a punishment. All the actions that the agent applies have an impact on the environment, and the agent learning process is influenced by the environment through the rewards. Figure 1: Q-learning scheme. The objective of the whole learning process is to find the optimal policy \(\pi^{*}:S\to A\) to perform whatever task the agent is supposed to carry out. The optimal policy is defined as the one that maximizes the cumulative reward (or the expected cumulative reward) given by the environment. This is achieved by mapping states (configurations) to actions and modifying that mapping until the optimum is encountered. The RL technique used in this paper, where an agent learns a policy to maximize an objective function, is based only on the current and next state of the system, \(s_{t}\) and \(s_{t+1}\), respectively, without using prior history. This approach is justified because the reward depends only on the current state and current action. Normally RL problems are modelled in the framework of Markov decision processes (MDP), that is stochastic processes where the probability of transition from one state to the next one depends only on the actual state, although a deterministic formulation of the method can be given [16]. To implement a self-optimizing controller for the reference system, a particular RL algorithm called Q-Learning, originally developed in Reference [17], was applied. This algorithm belongs to the class of unsupervised, model-free RL methods. Here the agent does not have any prior knowledge or model of the system characteristics from which it could estimate the next possible state. Essentially, the agent does not know what the effects on the environment of a certain action are and chooses the next action on the basis of the cumulative effect of the actions performed in the past. All reinforced learning problems are solved by applying the dynamic programming paradigm to the so-called value function, which is usually defined as a weighted sum of the rewards received by the agent. Model-free methods need to find both the value function and the optimal policy from observed data. Q-learning solves this problem by introducing an evaluation function \(Q:S\times A\rightarrow\mathbb{R}\) representing the total cumulative reward obtainable if an action \(a_{t}\) is carried out in a certain state \(s_{t}\) and then the optimal policy is followed afterwards. The value of this function indicates how good is to perform action \(a_{t}\) in state \(s_{t}\). For practical application, it is reasonable to assume that the number of states describing the environment and the number of actions that the agent can perform are both limited. Then, the Q-learning algorithm calculates the \(Q(s_{t},a_{t})\) function iteratively, storing it in a lookup table \(\mathbf{Q}\) (Figure 1). During the learning phase, the agent updates the values of the \(\mathbf{Q}\) matrix (initialised to zero) at each step according to the well-known Equation (1):  
\[\mathbf{Q}_{t+1}(s_{t},a_{t})=\mathbf{Q}_{t}(s_{t},a_{t})+\alpha\Big{(}r_{t}+\ \gamma\max_{a}\mathbf{Q}_{t}(s_{t+1},a)-\mathbf{Q}_{t}(s_{t},a_{t})\Big{)} \tag{1}\]  
where \(\alpha\) is the learning rate and \(\gamma\) is the discount factor and both take values in the [0,1] interval. These two factors weight, respectively, the increment to the present Q-value (learning speed) and the contribution from the future states (decay or discount). The value multiplied by factor \(\alpha\) is also known as the temporal difference error [15].