# Soft Fuzzy Reinforcement Neural Network Proportional-Derivative Controller
## 4 Results and Discussion
### Experimental Setup
#### 4.2.1 Case 1: Pendulum-v1 Training Performance Evaluation  
Figure 6 represents the training processes of SFPD and SAC, evaluated by the cumulative reward (CR) per episode [42]. The definition of cumulative reward (CR) is as follows:  
\[CR=\sum_{t=1}^{T}r(t) \tag{24}\]  
The training process spanned 10 epochs. The policy network parameters for SAC were randomly initialized, while the fuzzy membership initialization parameters for SFPD's FNNs were uniformly distributed within values. \begin{table}
\begin{tabular}{l l l l} \hline
**Param** & **Value** & **Unit** & **Description** \\ \hline \(F\) & \(1\) & m\({}^{3}\)/h & Liquid flow rate in the vessel \\ \hline \(V\) & \(1\) & m\({}^{3}\) & Reactor volume \\ \hline \(R\) & \(1.985875\) & kcal/(kmol\(\cdot\)K) & Boltzmann’s ideal gas constant \\ \hline \(\Delta H\) & \(-5.960\) & kcal/kmol & Heat of chemical reaction \\ \hline \(E\) & \(11,843\) & kcal/kmol & Activation energy \\ \hline \(k_{0}\) & \(34,930,800\) & \(1/\)h & Pre-exponential nonthermal factor \\ \hline \(\rho\)C\({}_{P}\) & \(500\) & kcal/(m\({}^{3}\)\(\cdot\)K) & Density multiplied by heat capacity \\ \hline \(UA\) & \(150\) & kcal/(K\(\cdot\)h) & Overall heat transfer coefficient multiplied by vessel area \\ \hline \end{tabular}
\end{table} 
Table 3: Parameters of CSTR model. Figure 6a shows the variation in cumulative rewards with training episodes for two algorithms (SFPD and SAC) in the pendulum control task. It suggested that the SFPD algorithm improved cumulative rewards rapidly during the early episodes, with a significantly faster convergence speed than the SAC algorithm. Additionally, the cumulative rewards of SFPD exhibited greater stability, with almost no significant regressions. In contrast, the SAC algorithm experienced significant fluctuations during the early stage and showed noticeable negative trends in the mid-training phase (around episodes 50 to 70), indicating potential instability in its policy updates. Overall, SFPD outperformed SAC, demonstrating faster convergence and higher stability. Figure 6b describes the changes in policy gradient loss per episode for both algorithms. It can be observed that the policy loss of SFPD trended toward zero during the early training stage and remained stable throughout the process, indicating a smooth and well-converged policy optimization. In contrast, the SAC algorithm exhibited severe fluctuations in policy loss during the initial training stage, with significant negative gradients observed around episodes 50 to 70, which may reflect instability in its policy updates. While the policy loss of SAC gradually stabilized later, its overall performance remained less stable compared to SFPD. Figure 6c presents the loss trends of the Q networks (Q1 and Q2) for both algorithms across training episodes. Results show that the Q network loss of SFPD decreased rapidly in the early training stage and stabilized thereafter, indicating effective control of value function estimation errors. On the other hand, the Q network loss of SAC showed large fluctuations during the early stages, and dramatic oscillations occurred around episodes 50 to 70. This aligned with the fluctuations observed in SAC's cumulative rewards and policy  
Figure 6: Training process on the pendulum: (**a**) CR over training epochs. (**b**) Policy loss over training epochs. (**c**) Q loss over training epochs. (**d**) V Loss over training epochs. loss, further indicating instability in value function estimation during that phase. Overall, SFPD exhibited superior stability and performance in Q network updates. Figure 6d displays the trends in V network loss across episodes for the two algorithms. The V network loss of SFPD declined rapidly in the early training stage and remained at a low level throughout the process, suggesting a stable and efficient value function learning process. In contrast, the SAC algorithm experienced significant peaks in V network loss around episodes 50 to 70, reflecting instability in value function learning.