# Hybrid Fuzzy-DDPG Approach for Efficient MPPT in Partially Shaded Photovoltaic Panels
## 2 Materials and Methods
### Reinforcement Learning Fuzzy-DDPG
Let \(Q(s,a)\) be the action-value function and \(\pi_{\theta}(s)\) be a deterministic policy parameterized by \(\theta\). Assume the reward function \(r(s,a)\) is bounded, the fuzzy membership functions \(\{\phi_{i}(s)\}_{i=1}^{N}\) form a partition of unity, and the neural networks for critic and actor have sufficient representational capacity with appropriate learning rates. Then the combined Fuzzy-DDPG algorithm converges to a near-optimal policy \(\pi^{*}(s)\) under the following conditions:_

1. _Fuzzy Q-iteration satisfies the contraction mapping property via the Bellman operator._
2. _The critic network converges to a good approximation of the optimal_ \(Q\)_-function_ \(Q^{*}(s,a)\)_._
3. _The actor network's policy gradient updates converge to a near-optimal deterministic policy._

Proof. : We prove convergence in three major steps, corresponding to the numbered items in the theorem statement, according to [34; 46; 47]. Step 1: Fuzzy Convergence

We start with the Bellman operator \(\mathcal{T}\) for a Q-function \(Q\):

\[a=\mu(\phi(s);\theta_{\mu}) \tag{13}\]5

where \(s^{\prime}\) is the next state determined by the environment's transition dynamics \(f(s,a)\). In the fuzzy framework, the Q-function is approximated as follows:

\[a=\mu(\phi(s);\theta_{\mu}) \tag{13}\]6

where \(\phi_{i}(s)\) are fuzzy membership functions (satisfying \(\sum_{i=1}^{N}\phi_{i}(s)=1\) for each \(s\)) and \(\theta_{i}\) are the learnable parameters for each fuzzy region \(i\). Contraction Property. For any two Q-functions \(Q\) and \(Q^{\prime}\), the Bellman operator satisfies:

\[a=\mu(\phi(s);\theta_{\mu}) \tag{13}\]7

which implies that \(\mathcal{T}\) is a contraction mapping with factor \(\gamma\in[0,1)\). The fuzzy partition \(\{\phi_{i}\}\) respects this contraction because of the following:

\[a=\mu(\phi(s);\theta_{\mu}) \tag{13}\]8

Consequently, repeated application of \(\mathcal{T}\) converges to a unique fixed point, i.e.,

\[a=\mu(\phi(s);\theta_{\mu}) \tag{13}\]9

Hence, the fuzzy Q-iteration converges to an approximate optimal Q-function \(\hat{Q}^{*}\). Critic Objective. In DDPG, the critic is represented by a neural network \(Q_{\theta_{c}}(s,a)\) and is trained to minimize the mean-squared Bellman error:Step 2: Critic Objective

\[Q(\phi(s),a;\theta_{Q}) \tag{14}\]0

where the target \(y_{t}\) is given by the following:

\[Q(\phi(s),a;\theta_{Q}) \tag{14}\]1

and \(\theta_{c}^{\prime},\theta_{a}^{\prime}\) are the parameters of the slowly updated target networks. Fuzzy Q-function Approximation in the Critic. In the fuzzy context, the critic's Q-function is expressed as follows:

\[Q(\phi(s),a;\theta_{Q}) \tag{14}\]2

where \(\theta_{c,i}\) are the critic network's parameters for each fuzzy set \(i\). The update rule for \(\theta_{c}\) via gradient descent is the following:

\[Q(\phi(s),a;\theta_{Q}) \tag{14}\]3

with

\[Q(\phi(s),a;\theta_{Q}) \tag{14}\]4

Convergence of the Critic Network. Critic convergence relies on:

* Bellman Operator Contraction: The targets \(y_{t}\) come from the contraction mapping \(\mathcal{T}\). * Gradient Descent to Reduce Bellman Error: Iterative updates align \(Q_{\theta_{c}}(s,a)\) with the fixed point \(Q^{*}(s,a)\). Thus, for suitable learning rates and sufficient network capacity, \[Q(\phi(s),a;\theta_{Q}) \tag{14}\]5 (25) where \(\beta\in[0,1)\) is a constant bounded by learning rate and fuzzy approximation smoothness. This iterative improvement implies \(\hat{Q}_{\theta_{c}}^{k}\to Q^{*}\) in the limit. Step 3: Actor Policy Convergence. Policy Gradient Update. The actor learns a deterministic policy \(\pi_{\theta}(s)\) by following the gradient of the expected return:

\[Q(\phi(s),a;\theta_{Q}) \tag{14}\]6

The actor parameters are updated by gradient ascent to maximize this expected return. Convergence of the Actor. Once the critic converges close to \(Q^{*}(s,a)\), the policy gradient points towards actions that maximize \(Q^{*}(s,a)\).