# Fuzzy Q-Learning Agent for Online Tuning of PID Controller for DC Motor Speed Control
## 1 Introduction  
DC (Direct Current) motors are used in industries extensively. This is due to their dynamic and reliable behavior. The main aspect of a DC motor is to control its speed. There are three main techniques of controlling the speed of a DC motor. The first one relies on changing the armature resistance, the second one relies on changing the field resistance and the last one relies on changing the armature voltage. The disadvantage of the armature resistance and field circuit techniques is the increase of power losses. Consequently, the best method is the change of the armature voltage [1]. The main method of changing the armature voltage in order to perform speed control in DC motors is the PID. The PID is the most well-known industrial controller, due to its robust performance in a wide range of applications and the ease of implementation. The implementation of a PID controller requires the definition of three parameters, which are the proportional, the integral and the derivative gain. For setting these parameters, there are a variety of methods, mostly empirical, such as Z-N [2]. These methods tune the controller according to the control process, and then the parameters are kept constant, no matter what is happening in the process. This characteristic has many times lead to poor controlling results, due to the dynamic nature of the processes. In these systems, intelligent PID controllers turn out to be very helpful [3] by adopting intelligent techniques such as neural network [4; 5; 6; 7], fuzzy logic [8; 9; 10], genetic algorithms [11; 12; 13] etc. Specifically, in terms of speed control of the DC motors, many researchers have been used intelligent PIDs. Supervisor fuzzy controllers have been used for adjusting online the gains of the PID controller in a two-layer controlscheme [1; 14]. Additionally, neural networks and adaptive neuro-fuzzy inference systems have been used in order to tune online the parameters of the PID controller [15; 16]. Genetic algorithms have been used for tuning offline the three terms of the PID controller [17]. These methods require the knowledge of an expert about the process, in order to be embedded in the control scheme. The main disadvantage of these methods lies on cases where there is no information about the process. Additionally, many of them can only optimize the controller offline, and cannot confront changes of the dynamic behavior of the process. This paper proposes a hybrid Z-N reinforcement learning approach for the online tuning of the parameters of the PID. The PID controller is tuned by the Z-N method, and then its three gains are adapted online through the fuzzy Q-Learning algorithm. This way, the gains of the controller are online adjusted via the interaction with the environment. As a result, the knowledge of an expert is not a necessity during the online tuning process. Q-Learning is a model-free approach and does not require a model of the controlled system. The main advantage of this approach is that it does not require prior knowledge, and its performance is not based on expert knowledge. This happens because its learning mechanism is entirely based on the interaction with the controlled system. On the other hand, Q-Learning is time consuming and needs to explore the state-action space in order to demonstrate good performance. Additionally, it cannot be used in a straight forward manner in problems with continuous state-action space. Recently, Q-Learning has been used in order to improve control processes. In [18], a combination of artificial neural networks and Q-Learning were used for controlling a bioreactor. An artificial neural network controller was trained by data, in order to initialize the Q-function. This way, the exploration phase of the Q-Learning was significantly reduced, and the algorithm converged very quickly.