# Soft Fuzzy Reinforcement Neural Network Proportional-Derivative Controller
## 1 Introduction
While these approaches take advantage of SAC's model-free adaptability, they must also address a major challenge in reinforcement learning (RL), experience replay, which stores and reuses past transitions to improve sample efficiency [18]. More specifically, in PID controllers, the integral term introduces a dependency on historical errors, which conflicts with the Markov property and can lead to unstable learning dynamics when combined with experience replay. While training with PID controllers and experience replay is not inherently incapable of convergence, it tends to exhibit instability, with successful convergence occurring inconsistently across multiple training attempts [19]. In contrast, fuzzy PD controllers do not suffer from this issue, as they rely solely on the current error and its derivative. Existing studies on auto-tuning fuzzy PD controllers using neural networks and reinforcement learning present several limitations. Some approaches utilize Q-learning to adjust fuzzy PD parameters [20], but without a replay buffer, the learning process is inefficient and less effective compared to SAC. Additionally, certain methods employ neural networks as a compensation mechanism for PD controllers [21; 22], requiring a predefined PD controller while still lacking a replay buffer, which limits sample efficiency. Other studies explored the use of neural networks to tune fuzzy rule parameters through supervised learning, necessitating pre-collected training data [23]. However, obtaining high-quality training data can be challenging, as it often requires extensive expert knowledge, precise system modeling, or costly real-world experiments. Additionally, the collected data may not fully represent the system's operating conditions, leading to poor generalization when applied to new scenarios. [PERSON] [24] combined neural networks with fuzzy PD controllers to control the speed of BLDC motors, using the mean square error (MSE) for training without a critic network or experience replay, leading to suboptimal learning. Similarly, [PERSON]. [25] applied model-based reinforcement learning with a State-Buffer State Prediction (SBSP) framework to auto-tune PD parameters for systems with large time delays. However, their approach employed a fully connected neural network within RL to tune the parameters of a fuzzy PD controller, but this is less effective than using a fuzzy neural network (FNN) [26], which is inherently better at capturing nonlinear relationships and adapting to system variations. While fuzzy logic itself can incorporate expert knowledge, relying solely on a standard neural network for parameter tuning limits its effectiveness, leading to poor initial performance and slow convergence. Moreover, FNNs offer a degree of interpretability by explicitly representing fuzzy rules and membership functions, making it easier to understand and analyze the decision-making process compared to standard neural networks [27]. These limitations highlight the need for a more effective approach that integrates fuzzy logic, neural networks, and reinforcement learning within a structured framework, ensuring better adaptability, interpretability, and sample efficiency for fuzzy PD controller optimization. To the best of our knowledge, no studies have leveraged the SAC framework for fuzzy neural network PD controllers. This paper explores this novel application, marking the first known attempt to integrate SAC with fuzzy neural network PD controllers. We propose employing a fuzzy neural network to automatically adjust the PD controller's parameters. Through reinforcement learning, the fuzzy neural network's parameters are learned and adapted autonomously. By leveraging an actor-critic architecture, the parameter adjustments of the fuzzy neural network are guided by the critic network. The backpropagation algorithm is utilized to fine-tune the fuzzy neural network parameters, enabling the PD controller to achieve the desired performance. To enhance exploration efficiency, a stochastic output is employed during training.