# Soft Fuzzy Reinforcement Neural Network Proportional-Derivative Controller
## 3 Proposed Soft Fuzzy Reinforcement Neural Network PD (SFPD) Controller
### Forward Propagation  
The lower part of Figure 4 depicts the interaction between SFPD and the environment, representing the forward propagation. Based on the current state \(S_{t}\), FNNs provide the PD control parameters. Each PD parameter, along with the standard deviation provided by standard neural networks (Std NNs), is then transformed into the final PD controller using the reparameterization technique. The PD controller subsequently calculates the current action \(a\), which is applied to the environment (plant). The plant provides a reward, and the process continues to the next state \(S_{t+1}\). The loop keeps going until the current epoch ends. The tuple (\(S_{t}\), \(a\), \(r\), \(S_{t+1}\)) is stored in the replay buffer for parameter updates. Inputs are processed through tanh before being passed to the neural network to enhance conver  
Figure 4: Proposed framework for SFPD control: The upper part illustrates the backward propagation process, where inputs are passed through the FNNs to generate PD parameters, which are then used by the PD controller to produce the final control signal \(a\). The lower part depicts the forward propagation process, where training of the neural networks is conducted by resampling tuples (\(S_{t}\), \(a\), \(r\), \(S_{t+1}\)) from the replay buffer. gence by normalizing values, maintaining a zero-centered distribution, and improving gradient flow. In FNNs, the first layer is the input layer, which receives the inputs \(x\). The tanh transformation is applied before training, as a preprocessing step to normalize the inputs into the range \([-1,1]\), which facilitates stable and efficient convergence. While tanh is not part of the model architecture and does not affect gradient flow during training, this bounded and smooth transformation helps reduce the impact of outliers and brings the input data into a scale that aligns well with the model's initialization and training dynamics, thereby improving stability and robustness. \[x=\tanh([e,e^{\prime}]) \tag{9}\]  
The second layer is the fuzzification layer and consists of fuzzy memberships. Each neuron represents one membership. The Gaussian function is used to transform the input into the fuzzy sets, as shown below [37]:  
\[\mu_{i}^{j}=e^{-\frac{(x_{i}-c_{ij})^{2}}{\sigma_{ij}^{2}}} \tag{10}\]  
where \(x_{i}\) is the input, and \(c_{ij},\sigma_{ij}\) represent the center and width of the distribution. The notation \(i=2,j=7\) indicates that there are two crisp inputs, each associated with seven fuzzy memberships. The partial interpretability of Fuzzy Neural Networks (FNNs) enables the integration of expert knowledge during the initialization phase. By setting initial parameters such as the centers and widths of the Gaussian membership functions based on domain expertise, the model can achieve a more informed starting point. This approach leads to faster convergence early in the training process, as the network starts with parameters that are more aligned with the underlying system behavior. The third layer is the multiplication of all the input fuzzy sets, which represents the fuzzy inference:  
\[\alpha_{j}=\mu_{1}^{i_{1}}\mu_{1}^{i_{2}}\cdots\mu_{1}^{i_{n}} \tag{11}\]  
where \(n=2\) represents the fuzzy membership functions associated with this rule:  
The fourth layer is the normalization layer. \[\beta_{j}=\frac{\alpha_{j}}{\sum_{j=1}^{m}\alpha_{j}},\,j=1,2,\ldots,m \tag{12}\]  
where \(m=ji=49\) indicates that there are 49 fuzzy rules. The fifth layer is the defuzzification layer:  
\[y_{i}=\sum_{j=1}^{r}\omega_{ij}\beta_{j},\,j=1,2,\ldots,m \tag{13}\]  
where \(y_{i}\) includes the two parameters of the PD controller, \(P^{\prime}\) and \(D^{\prime}\), and \(\omega_{ij}\) indicates the connection weight.