## 4 The Proposed h-POQL MPPT Control
### The Assessment of the MPPT Control Methods
Q-learning is a useful RL method for handling and figuring out the running average values of the reward function. Considering that S is a discrete set of states, where A is a discrete set of actions, the agent will experience every state \(s\in S\) and possible set of actions a \(\in A\) through the learning process. When taking the action \(a_{t}\) the agent will transit from state \(s_{t}\) to state \(s_{t+1}\) and receive a reward \(r_{t+1}\), then the Q-learning update rule is given by equation below [48]:  
\[Q_{t+1}(s_{t},a_{t})=Q_{t}(s_{t},a_{t})+a\Big{[}r_{t+1}+\gamma\ \max_{a_{i}}Q_{t}[s_{t+1},a_{i}]-Q_{t}[s_{t},a_{t}]\Big{]} \tag{4}\]  
in which, \(Q_{t}(s_{t},a_{t})\) is the action value function, \(\alpha\) is the learning rate, \(\gamma\) is the discount factor, and \(\max_{a_{i}}Q_{t}(s_{t+1},a_{i})\) is the maximum expected future reward given the new state s and possible action at the next step. The flowchart of Q-learning algorithm is demonstrated in Figure 13[12]. The output power of PV system can be calculated by the equation below [16]:  
\[P_{pv}=V_{pv}I_{pv}=V_{pv}\Big{\{}I_{ph}-I_{pv}\Big{[}e^{\frac{q}{ART}[V_{pv} +I_{pv}R_{s}]}-1\Big{]}\Big{\}} \tag{5}\]  
where \(I_{ph}\) is the light-generated current, \(R_{s}\) is the series resistance, A is the non-ideality factor, \(k\) is the Boltzmann constant, \(I_{pv}\) is the dark saturation current, T is temperature, and \(q\) is the electron charge. Generally, there are two stages in the MPPT control based on Q-learning: the offline learning process and the online application process [12]. Firstly, the agent will learn a map from state to action and then the learned values of the actions will be stored in the Q-table. Following this Q-table, the relationship between the voltage and power is determined. Secondly, the action value Q-table will be used to control the PV system in the application process. The procedure of initial input configuration for Q-learning is shown in Figure 13 as follows [16]:  
* State-spaces are represented by the voltage-power pair: \[S=\Big{\{}s|s_{kj}=(V_{pv,k},P_{pv,j}),k\in[1,2,\ldots,N],j\in[1,2,\ldots,M] \Big{\}}\] (6)
* Action-spaces are the perturbations of duty cycle \(\Delta D\) to the PV voltage: \[A=\{a|+\Delta D,0,-\Delta D\}\] (7)* Rewards:  
\[r_{t+1}=\left\{ \begin{array}{l}w_{p}\Delta P\;\;if\;\Delta P>\delta_{1}\\ w_{best}\;\;if\;|\Delta P|>\delta_{1}\,and\,a_{i}\
eq 0\\ w_{n}\Delta P\;if\;\Delta P<\delta_{1}\,or\,a_{i}=0\end{array} \right. \tag{8}\]  
where \(\Delta P=P_{t+1}-P_{t}\) and \(\delta_{1}\) is the small number represented as the small area around the maximum power point. Based on the weights \(w_{p}\), \(w_{best}\), and \(w_{n}\) the separation between positive, best, and negative states is clearly defined. Based on the state of art reinforcement learning in the field of MPPT control, the proposed h-POQL method aims to get the advantages of low learning time, low cost, and easy implementation in a practical system. By separating the control regions based on the irradiation and temperature, the state space can be reduced. The agent will spend less time learning the optimal policy in a small control region. In addition, the fixed step size of the duty cycle is the major problem of the P&O method in the response to the fast change of weather conditions. The Q-learning method aims to use a variable step size to define the optimal duty cycle in a specific control region. With the knowledge learned by the Q-learning agent, the P&O can change the reference input of the duty cycle so that the smaller step size of the duty cycle can be applied to track for the maximum power of the PV source. Figure 13: The learning state of the maximum power point tracking (MPPT) based Q-learning algorithm.