## 1 Introduction
The performance of many metaheuristic optimization techniques implementing the GMPPT process in partially-shaded PV systems has been compared in [22]. This paper presents a new GMPPT method, which is based on a machine-learning algorithm. Compared to the past-proposed GMPPT techniques, the Q-learning-based method proposed in this paper has the following advantages: (a) it does not require knowledge of the operational characteristics of the PV modules and the PV array comprised in the PV system; and (b) due to its inherent learning capability, it is capable of detecting the GMPP in significantly less search steps. Thus, the proposed GMPPT method is suitable for employment in PV applications, where the shading pattern may change quickly (e.g., wearable PV systems, building-integrated PV systems, where shading is caused by people moving in front of the PV array, etc.). The numerical results presented in this paper, validate the capabilities and advantages of the proposed GMPPT technique. This paper is organized as follows: the operating principles and structure of the proposed Q-learning-based GMPPT method are analyzed in Section 2; the numerical results by applying the proposed method, as well as the PSO evolutionary GMPPT algorithm, for various shading patterns of a PV array are presented in Section 3; and, finally, the conclusions are discussed in Section 4. The Proposed Q-Learning-Based Method for Photovoltaic (PV) Global Maximum Power Point Tracking (GMPPT)  
A block diagram of the PV system under consideration is illustrated in Figure 2. The PV array comprises multiple PV modules connected in series and parallel. The GMPPT process is executed by the GMPPT controller, which produces the duty cycle, \(D\), of the pulse width modulation (PWM)control signal of the DC/DC converter using measurements of the PV array output voltage and current. Each value of \(D\) determines the PV array output voltage according to (1):  
\[V_{PV}=(1-D)\cdot V_{o} \tag{1}\]  
where \(V_{o}\) (\(V\)) is the DC/DC converter output voltage and \(V_{PV}\) (\(V\)) is the PV array output voltage. The proposed technique considers the PV GMPPT operation as a Markov decision process (MDP) [23], which constitutes a discrete-time stochastic control process and models the interaction between an agent implemented in the GMPPT control unit and the PV system (i.e., PV array and DC/DC power converter). The MDP consists of: (a) the state-space \(\mathbf{S},\) (b) the set of all possible actions \(\mathbf{A},\) and (c) the reinforcement reward function, which represents the reward, when applying action \(a\) in state \(S\), which leads to a transition to state \(S^{\prime}\)[23]. The Markovian property dictates that each state transition is independent of the history of previous states and actions and depends only on the current state and action; likewise, each reward is independent of the past states, actions, and rewards and depends only on the most recent transition. A temporal-difference Q-learning algorithm is applied for solving the PV GMPPT optimization problem. The Q-learning algorithm's goal is to derive an action selection policy, which will maximize the total expected discounted rewards that the system will receive in the future. A simplified representation of process implemented by the Q-learning algorithm in order to control a PV system for the implementation of the GMPPT process is presented in Figure 3.  
In Q-learning, an agent interacts with the unknown environment (i.e., the PV system) and gains experience through a specific set of states, actions and rewards encountered during this interaction [24; 25; 26; 27]. Q-learning strives to learn the Q-values of state-actions pairs, which represent the expected total discounted reward in the long term.