## 5 MPPT Algorithms
### Deep Q-Network (DQN)  
Figure 12 presents a simplified block diagram of the Deep Q-Network (DQN) algorithm used for MPPT control in PV systems. It illustrates the key components involved in the agent-environment interaction, including state observation, action selection, Q-value estimation, and policy updating.  
The agent receives the system state, comprising the PV voltage, current, duty cycle, and its variation, and evaluates possible actions using a Q-network (a deep neural network). The agent selects the next action using an \(\varepsilon\)-greedy policy, balancing exploration (random actions) and exploitation (choosing the action with the highest Q-value). This policy is formally defined as follows [10; 31]:  
\[a_{t}=\begin{cases}\arg\max_{a\in A}Q(s_{t},a|\theta)&\text{with probability }1- \varepsilon\\ \text{random}(a_{t}\in A)&\text{with probability }\varepsilon\end{cases}\]  
where  
* \(A\) is the set of all available actions;
* \(\varepsilon\in[0,1]\) is the exploration rate.  
Figure 11: Simulink model used under PSC conditions.  
Figure 12: Block diagram of the DQN algorithm, showing interaction between environment, critic network, and policy updating.  
This approach allows the agent to continuously explore new strategies while also exploiting the best-known actions, facilitating learning in complex and high-dimensional environments. The selected action is applied to the PV system by modifying the duty cycle of the buck converter, which changes the operating point of the panel. The environment then moves to a new state and returns a reward that reflects the change in power output resulting from the agent's action.  
In DQN, the traditional Q-table is replaced by a deep neural network (_Q-network_) to approximate the action-value function \(Q(s,a|\theta)\), where \(\theta\) are the network weights. Two networks are used: the _predict Q-network_ (with weights \(\theta\)) and the _target Q-network_ (with weights \(\theta^{\prime}\)) [10]. The loss function minimized during training is the mean squared error (MSE) between the predicted and target Q-values described in Equation (13) [10; 32; 33]:  
\[L(\theta)=\mathbb{E}_{s,a}\bigg{[}\Big{(}Q_{\text{target}}-Q_{\text{predict}} \Big{)}^{2}\bigg{]} \tag{13}\]  
The target Q-value is computed as  
\[Q_{\text{target}}=\underbrace{r}_{\text{reward}}+\underbrace{\gamma}_{ \text{discount}}\max_{a}Q(s_{t+1},a_{t+1}|\theta^{\prime}) \tag{14}\]  
And the predicted Q-value from the online network is  
\[Q_{\text{predict}}=Q(s_{t},a_{t}|\theta) \tag{15}\]  
The critic network evaluates the Q-values and calculates the loss, which is used to update the weights of the predict network through backpropagation. Over time, this allows the agent to improve its decision-making, leading to convergence to an optimal policy that maximizes long-term power extraction from the PV system.  
Being a model-free method, the DQN agent adapts autonomously to varying conditions, including non-uniform irradiance and PSC, demonstrating strong performance in identifying the GMPP without requiring prior knowledge of the system dynamics.  
Additionally, Figure 13 shows the detailed Simulink implementation of the DQN-based MPPT control strategy developed in this study. This model integrates the trained agent within the PV system environment to perform real-time duty cycle adjustments based on learned policies.  
Figure 13: Simulink implementation of the DQN-based MPPT control strategy.