# Hybrid Fuzzy-DDPG Approach for Efficient MPPT in Partially Shaded Photovoltaic Panels
## 2 Materials and Methods
### Reinforcement Learning Fuzzy-DDPG
The Deterministic Policy Gradient algorithm [42] extends the policy gradient theorem to deterministic policies, laying the groundwork for DDPG [43]. The deterministic policy gradient algorithm [42] states that the gradient of the expected return with respect to the policy parameters \(\theta^{\mu}\) is:  
\[\
abla_{\theta^{\mu}}J=\mathbb{E}_{s\sim\rho^{\mu}}\left[\
abla_{\theta^{\mu} }\mu(s|\theta^{\mu})\
abla_{\alpha}Q^{\mu}(s,a)\right]_{a=\mu(s|\theta^{\mu})} \tag{8}\]  
where:  
* \(Q^{\mu}(s,a)\) is the action-value function under policy \(\mu\). * \(\rho^{\mu}(s)\) is the discounted state visitation distribution under \(\mu\). In [44] is introduced a fuzzy Q-iteration algorithm for approximate dynamic programming, addressing the curse of dimensionality by using a fuzzy partition of the state space and discretization of the action space to approximate the Q-function, making it applicable to continuous state and action spaces. This approach allows for handling larger state spaces than traditional dynamic programming, which struggles with the exponential growth of computational complexity as the number of state variables increases. Fuzzy Q-Iteration approximates the Q-function using fuzzy basis functions:  
\[Q(s,a)\approx\sum_{i=1}^{N}w_{i}\phi_{i}(s,a) \tag{9}\]  
where:  
* \(\phi_{i}(s,a)\) are fuzzy basis functions representing the degree of membership of \((s,a)\) in fuzzy sets. * \(w_{i}\) are the weights to be learned. The update rule minimizes the Bellman residual:  
\[w^{(k+1)}=\arg\min_{w}\sum_{(s,a)}\left(\mathcal{R}(s,a)+\gamma\max_{a^{ \prime}}Q^{(k)}(s^{\prime},a^{\prime})-\sum_{i=1}^{N}w_{i}\phi_{i}(s,a)\right)^ {2} \tag{10}\]  
The fuzzy Q-iteration algorithm iteratively improves the Q-function approximation, converging to the optimal Q-function under certain conditions. The proposed method combines the strengths of fuzzy techniques and DDPG to achieve global Maximum Power Point Tracking under partial shading conditions. The fuzzy component handles the action space, while the DDPG component addresses the continuous state space, resulting in a hybrid approach that is both scalable and able to operate in complex environments [28; 45]. By integrating fuzzy basis functions into the actor and critic networks of DDPG, we aim to approximate both the policy and value function using fuzzy features, as follows:  
\[ \begin{split}\text{Actor Network:}&\quad\mu(\phi(s )|\theta^{\mu})\\ \text{Critic Network:}&\quad Q(\phi(s),a|\theta^{Q}) \end{split}  \tag{11}\]  
The fuzzy feature vector \(\phi(s)\) captures the essential characteristics of the state space, facilitating learning in continuous domains. This hybrid approach combines the strengths of fuzzy approximation and deep reinforcement learning, enabling the agent to efficiently navigate the complex power landscape and find the global maximum power point under partial shading conditions. For each state variable \(s_{i}\), we define \(M\) membership functions \(\{\mu_{ij}\}_{j=1}^{\text{M}}\), typically Gaussian functions:  
\[\mu_{ij}(s_{i})=\exp\left(-\frac{(s_{i}-c_{ij})^{2}}{2\sigma_{ij}^{2}}\right) \tag{12}\]  
The parameters are defined as follows:  
* \(c_{ij}\): Center of the \(j\)-th membership function for variable \(s_{i}\). * \(\sigma_{ij}\): Width (standard deviation) of the membership function. The actor and critic networks take this fuzzy feature vector as input, and the output represents the actor policy and critic value function, respectively. The actor network architecture is as follows:  
* Input Layer: Receives fuzzy feature vector \(\phi(s)\). * Hidden Layers: Multiple fully connected layers with nonlinear activations. * Output Layer: Produces action \(a=\mu(\phi(s)|\theta^{\mu})\), using a bounded activation function like tanh. The critic network architecture is as follows:* State Input Layer: Receives fuzzy feature vector \(\phi(s)\). * Action Input Layer: Receives action \(a\). * Hidden Layers: Concatenates \(\phi(s)\) and \(a\), followed by fully connected layers. * Output Layer: Outputs scalar value \(Q(\phi(s),a|\theta^{Q})\).