# Hybrid Fuzzy-DDPG Approach for Efficient MPPT in Partially Shaded Photovoltaic Panels
## 2 Materials and Methods
### Reinforcement Learning Fuzzy-DDPG
3: Compute Targets: \[y_{i}=r_{i}+\gamma Q^{\prime}(\phi(s_{i+1}),\mu^{\prime}(\phi(s_{i+1})|\theta^{ \mu^{\prime}})|\theta^{Q^{\prime}})\]
4: Update Critic: Minimize loss \(L\): \[L=\frac{1}{N}\sum_{i=1}^{N}\Bigl{(}y_{i}-Q



This training procedure allows the agent to learn the optimal policy and value function by leveraging the fuzzy feature representation, enabling global Maximum Power Point Tracking under partial shading conditions.

Algorithm 3 in the form of pseudocode could be used to implement the proposed method.
```
1:Initialize actor network \(\mu(\phi(s)|\theta^{\mu})\) and critic network \(Q(\phi(s),a|\theta^{Q})\)
2:Initialize target networks \(\theta^{\mu^{\prime}}\leftarrow\theta^{\mu}\), \(\theta^{Q^{\prime}}\leftarrow\theta^{Q}\)
3:Initialize replay buffer \(\mathcal{D}\)
4:for episode = 1 to max_episodes do
5: Initialize state \(s_{0}\)
6:for\(t=1\) to max_steps do
7: Compute fuzzy features \(\phi(s_{t})\)
8: Select action \(a_{t}=\mu(\phi(s_{t})|\theta^{\mu})+\mathcal{N}_{t}\)
9: Execute action \(a_{t}\), observe reward \(r_{t}\) and next state \(s_{t+1}\)
10: Store transition \((s_{t},a_{t},r_{t},s_{t+1})\) in \(\mathcal{D}\)
11: Sample mini-batch of \(N\) transitions from \(\mathcal{D}\)
12:for each sample \((s_{i},a_{i},r_{i},s_{i+1})\)do
13: Compute target: \[y_{i}=r_{i}+\gamma Q^{\prime}(\phi(s_{i+1}),\mu^{\prime}(\phi(s_{i+1})|\theta^ {\mu^{\prime}})|\theta^{Q^{\prime}})\]
14:endfor
15: Update critic by minimizing: \[a=\mu(\phi(s);\theta_{\mu}) \tag{13}\]0
16: Update actor using policy gradient: \[a=\mu(\phi(s);\theta_{\mu}) \tag{13}\]1
17: Update target networks: \[a=\mu(\phi(s);\theta_{\mu}) \tag{13}\]2 \[a=\mu(\phi(s);\theta_{\mu}) \tag{13}\]3
18:endfor
19:endfor
```

**Algorithm 3** Fuzzy-DDPG Algorithm

Figure 2 illustrates the main steps of the algorithm, depicting how the state observations from the PV environment are processed to update both the critic and actor networks through a gradient-based rule. Convergence conditions are periodically checked to ensure that the control policy remains near-optimal. The final result is an adaptive, data-driven control strategy capable of guaranteeing a stable operating point at or near the global maximum power region of the PV system. ### Proof

We present a formal proof of the convergence of a hybrid approach that combines Fuzzy Q-iteration with the Deep Deterministic Policy Gradient (DDPG) algorithm. The main goal is to show that, under standard assumptions (bounded rewards, proper learning rates, sufficient function approximators, etc. ), the proposed method converges to a near-optimal policy. Our proof proceeds in three main steps:

1. Fuzzy Q-iteration Convergence: We show that the fuzzy Bellman operator is a contraction, thereby ensuring that it converges to a fixed point. 2. DDPG Critic Convergence: We prove that the Critic network (Q-function approximation) converges to a close approximation of the optimal action-value function. 3. DDPG Actor Convergence: We demonstrate that the Actor parameters converge to a near-optimal deterministic policy given that the critic accurately approximates the Q-function. ### Preliminaries and Notation

**Definition 1** (State and Action Spaces). : _Let \(S\subseteq\mathbb{R}^{n}\) be a (continuous) state space and \(A\subseteq\mathbb{R}^{m}\) be a (continuous) action space._

\[a=\mu(\phi(s);\theta_{\mu}) \tag{13}\]4 Flow diagram illustrating the proposed control scheme for Maximum Power Point Tracking in a PV system. **Definition 2** (Reward Function). : _We assume a bounded reward function \(r:S\times A\to\mathbb{R}\), i.e., there exists \(R_{\max}\geq 0\) such that \(\sup_{(s,a)}|r(s,a)|\leq R_{\max}\)._

**Definition 3** (Transition Dynamics). : _We assume a deterministic environment with transition dynamics \(s_{t+1}=f(s_{t},a_{t})\)._

### Theorem

**Theorem 1** (Convergence of fuzzy-DDPG algorithm). : _Consider a continuous state space \(S\subseteq\mathbb{R}^{n}\), a continuous action space \(A\subseteq\mathbb{R}^{m}\), and a discount factor \(\gamma\in[0,1)\).