# Soft Fuzzy Reinforcement Neural Network Proportional-Derivative Controller
## 1 Introduction  
Proportional-Integral-Derivative (PID) controllers [1] are widely used in the field of control systems. However, their control performance is suboptimal for strongly nonlinear, high-order, and hysteresis systems [2]. Especially in the presence of noise, traditional control systems can face challenges in maintaining optimal performance. To address these issues, various advanced technologies have emerged. Among these, fuzzy PID controllers that integrate fuzzy logic have gained widespread applications (e.g., robotic trajectory tracking [3], brushless DC motors [4]) since their emergence [5, 6]. These model-free controllers provide improved handling of nonlinearities, enhanced robustness, and greaterflexibility compared to traditional PID controllers [7]. However, fuzzy PID controllers rely on fixed parameters derived from human expertise, making them less adaptable to system variations and highly dependent on prior knowledge [8]. While the integral term in PID controllers helps reduce steady-state errors, it also introduces lag and increases computational complexity. Unlike PID controllers, which accumulate past errors through integral action, Proportional-Derivative (PD) controllers [9] focus solely on the current error and its rate of change, allowing for quicker adaptation to dynamic system variations. However, their fixed parameters still limit adaptability in highly nonlinear or uncertain environments. Fuzzy PD controllers present an alternative to traditional PD controllers, providing greater flexibility while preserving the benefits of PD control. Their simpler fuzzy rule design [10] allows for easier implementation, while their rule-based adaptability enables them to deliver performance comparable to fuzzy PID controllers [11]. By leveraging fuzzy logic, they improve responsiveness and adaptability, making them well suited for applications that require fast reaction times without the drawbacks of integral action [12; 13]. However, they still rely on rule-based methods, which can become inadequate in highly complex or uncertain environments. To enhance performance in such challenging scenarios, reinforcement learning (RL) algorithms, such as Soft Actor-Critic (SAC) [14], offer a model-free approach that learns optimal control policies directly through interaction with the environment. Unlike traditional model-based control methods, RL algorithms like SAC do not require an explicit system model, making it more adaptable to complex and dynamic environments. SAC, in particular, employs entropy maximization to promote exploration and prevent premature convergence, effectively balancing exploration and exploitation, which makes it a powerful tool for continuous control tasks. It also aims to maximize the cumulative reward (CR) while maintaining this balance. This differentiates it from classical control approaches, which typically rely on predefined control laws and system models. By leveraging an actor-critic framework and experience replay, SAC enhances sample efficiency and stability, making it well suited for complex, high-dimensional control problems. Several studies have explored the integration of Soft Actor-Critic (SAC) with PID control, leveraging the replay buffer to enhance sample efficiency and improve system performance. [PERSON] et al. [15] applied SAC-based deep reinforcement learning for online PID tuning in hydraulic servo control systems, showcasing its effectiveness in real-time applications by improving the system's responsiveness and stability. [PERSON] et al. [16] proposed a self-adaptive SAC-PID control approach for mobile robots, emphasizing its capability to dynamically adjust to unpredictable environments. [PERSON] et al. [17] investigated SAC-based PID parameter tuning for unmanned surface vehicle (USV) path following, optimizing control performance through automated and real-time tuning.