# Soft Fuzzy Reinforcement Neural Network Proportional-Derivative Controller
## 4 Results and Discussion
### Testing Performance Evaluation
#### 4.2.2 Case 2: CSTR System Training Performance Evaluation  
The training process, as shown in Figure 9, involved 50 epochs for both the SFPD and SAC algorithms. The fuzzy membership initialization parameters for SFPD's FNNs were uniformly distributed within their respective value ranges, with the number of fuzzy memberships set to seven. From Figure 9, the training process on the CSTR system highlighted the significant advantages of SFPD over SAC. In Figure 9a, the cumulative reward (CR) for SFPD remained stable, consistently approaching 1000, while SAC exhibited substantial fluctuations and slower convergence, revealing instability in its learning process. Regarding policy loss, SFPD quickly minimized the loss and maintained stability, whereas SAC experienced higher losses and a slower decline, emphasizing SFPD's superior efficiency in policy optimization. As shown in Figure 9b, the Q-network loss for SFPD was consistently low and stable from the outset, while SAC began with high losses that gradually decreased but still exhibited noticeable errors, underscoring SFPD's accuracy in value function estimation. Similarly,  
Figure 9: Training process on CSTR: (**a**) CR over training epochs. (**b**) Policy loss over training epochs. (**c**) Q Loss over training epochs. (**d**) V loss over training epochs. Figure 9c illustrates that SFPD maintained low and stable V-network loss throughout, while SAC exhibited significant initial fluctuations before stabilizing, further demonstrating SFPD's advantages in state-value function estimation and policy stability. Overall, SFPD outperformed SAC in terms of stability, convergence speed, and loss control, proving to be a more effective solution for complex dynamic system control tasks. Notably, when the exploration space is large, such as with vast action spaces, SAC's convergence becomes significantly slower or may even fail entirely, whereas the proposed method continues to exhibit robust and reliable convergence performance. Testing Performance Evaluation  
Figure 10 presents the testing results, comparing the performance of the SAC, SFPD, and traditional PD algorithms. To evaluate noise robustness, a Gaussian white noise measurement disturbance signal with \(\mu\sim\mathcal{N}(0,1^{2})\) was introduced, as depicted in Figure 10b. Additionally, to assess disturbance rejection capability, an external torque disturbance was applied [45] between steps 1000 and 1005, as shown in Figure 10c. Unlike previous tests that employed a positional PD controller, this experiment utilized a traditional incremental PD controller. The mathematical formulation of the incremental PD controller is given below:  
\[T_{c}(t+1)=T_{c}(t)+K_{p}\times e+K_{d}\times e^{\prime} \tag{25}\]  
where \(T_{c}(t+1)\) denotes the next time step's control signal, while \(T_{c}(t)\) denotes the current time's control signal. The parameters for the PD algorithm were determined using the Ziegler-Nichols method, with \(K_{p}=6.690\) and \(K_{d}=1.842\). Figure 10: Testing results on CSTR: (**a**) without Gaussian noise; (**b**) with Gaussian noise; (**c**) with Gaussian noise and external disturbance. From Figure 10, it can be observed that the SAC algorithm struggled with stability, showing significant oscillations. In contrast, both the SFPD and traditional PD algorithms managed to control the system effectively. Figure 10b illustrates that the SFPD algorithm demonstrated higher robustness compared to the traditional PD algorithm. Table 5 provides the quantitative data. From Figure 10c, it is evident that SFPD (red line) demonstrated superior control stability, maintaining the temperature close to the reference value (yellow line) with minimal fluctuations. In contrast, SAC (blue dashed line) exhibited significant instability, with large oscillations and a pronounced peak around step 1000, exceeding 350 \({}^{\circ}\)C. Traditional PD (green dashed line) performed better than SAC but still showed noticeable deviations and slower recovery.