## 1 Introduction
### The Overall Q-Learning-Based GMPPT Algorithm  
A flowchart of the overall Q-learning-based PV GMPPT algorithm, which is proposed in this paper, is presented in Figure 4. The PV GMPPT process is performed as follows:  
**Step 1**: Q-table initialization. As analyzed in Section 2.2, a Q-table is created using four dimensions according to (5); three dimensions for the state and one for the action. This table is initially filled with zeros, because there is no prior knowledge. Also, a table that contains the number of visits of each state is defined. The size of this table is equal to the number of states. Lastly, an initial value is given to the duty cycle.  
**Step 2**: After the PV array output voltage and current signals reach the steady-state, the PV array-generated power is calculated. According to the value of the duty cycle in the current time-step, its value during the previous time-step and the PV array-generated power, the state is determined using (5).  
**Step 3**: The learning rate is calculated by (8).  
**Step 4**: The temperature \(\tau\) is calculated by (4) and the probability of every possible action for the current state is also calculated by (3).  
**Step 5**: If the number of visits to the current state is higher than or equal to the predefined value of \(N_{\max}\) and the Q-value for the action corresponding to \"no-change\" of the duty cycle value is the maximum compared to the other actions, then it is concluded that convergence to an operating point close to the GMPP has been achieved and operation at the local MPPs has been avoided. Then, the proposed GMPPT process executes the P&O MPPT algorithm in order to help the agent fine-tune its operation at the GMPP.  
**Step 6**: If the number of visits of the current state is lower than \(N_{\max}\) or if the Q-value for the action corresponding to \"no-change\" of the duty cycle value is not the maximum, a number in the range (0, 1)is randomly selected and compared to the probabilities of every possible action for the specific state (step 4). Each probability encodes a specific duty cycle change, as follows:  
\[\begin{array}{rl}\text{If }0<N_{r}\leq p_{1}:&\text{Action 1}\\ \text{Else if }p_{1}<N_{r}\leq p_{1}+p_{2}:&\text{Action 2}\\ \text{Else if }p_{1}+p_{2}<N_{r}\leq\sum\limits_{k=1}^{3}p_{k}:&\text{Action 3}\\ \text{Else if }\sum\limits_{k=1}^{3}p_{k}<N_{r}\leq\sum\limits_{k=1}^{4}p_{k}:& \text{Action 4}\\ \text{Else if }\sum\limits_{k=1}^{4}p_{k}<N_{r}\leq\sum\limits_{k=1}^{5}p_{k}:& \text{Action 5}\\ \text{Else if }\sum\limits_{k=1}^{5}p_{k}<N_{r}\leq\sum\limits_{k=1}^{6}p_{k}:& \text{Action 6}\\ \text{Else : Action 7}\end{array} \tag{9}\]  
where \(p_{k}\) (\(k=1,2,\dots\) 6) is the probability of each possible action for the current state that is calculated by using (3), while \({}_{r}\) is a random number in the range \((0,\,1)\).  
**Step 7**: After the change of the action, a new duty cycle value is applied to the control signal of the DC/DC power converter and the algorithm waits until the PV array output voltage and the current signals reach the steady state. Then, the proposed GMPPT algorithm calculates the difference between the power produced by the PV array at the previous and the current time-steps, respectively, and assigns the reward according to (7).  
**Step 8**: The future state is determined and the Q-function is updated according to (2).  
**Step 9**: Return to step 2.  
Figure 4: A flowchart of the proposed Q-learning-based PV GMPPT algorithm.