# Soft Fuzzy Reinforcement Neural Network Proportional-Derivative Controller
## 3 Proposed Soft Fuzzy Reinforcement Neural Network PD (SFPD) Controller
### Backward Propagation  
The upper part of Figure 4 represents the update process, specifically the backpropagation phase. Backpropagation is a key algorithm in training neural networks, where the model's error is propagated backward through the network. During this phase, the gradients of the loss function with respect to each weight are calculated, allowing the model to update its weights to minimize the error. This iterative process helps the network learn from the data and improve its performance over time. Initially, data \((s_{t},a_{t},r,s_{t+1})\) of batch size are retrieved from the replay buffer to update various network parameters. For enhanced stability during training, this study employed two state-action value neural networks (QNNs), using the minimum of their outputs, \(\min(Q_{1},Q_{2})\), and incorporated a target state value network (VNNs). The QNNs were updated using the Bellman equation as shown in (5). The loss function for the VNNs is depicted in (6), where the calculation of \(Q_{\theta}^{\pi}(s_{t+1},a_{t+1})\) selects the minimum output between the two QNNs, while \(\log\pi_{\phi}(a_{t}|s_{t})\) represents the sum of the logarithms of the parameters \(P\) and \(D\) output by the Std NNs, as shown below:  
\[\log\pi_{\phi}(a_{t}|s_{t})=\log\pi_{\phi}f_{\phi}^{P}(s_{t})+\log\pi_{\phi}f_ {\phi}^{D}(s_{t}) \tag{16}\]  
The loss function for FNNs and the Std NNs, as shown in (4), similarly involves selecting the minimum Q value and incorporating the aforementioned logarithm.  
The forward and backward propagation are summarized in Algorithm 1.  
```
1: Initialize all the NNs, including \(Q_{1}\), \(Q_{2}\), \(V\), FNNs, Std NNs.
2: Initialize target VNNs, with \(\theta_{v^{\prime}}\leftarrow\theta_{v}\)
3: Initialize the parameters \(\omega_{ij}\), \(c_{ij}\), and \(\sigma_{ij}\) in FNNs.
4: Set a replay buffer \(R\)
5:for episodes \(k=1\) to \(E\)
6:while this episode is not completed or the maximum steps \(>N\)
7: Obtain an action \(a(t)\) from the PD controller, with parameters derived from the FNNs and Std NNs based on the current state \(s(t)\), according to the forward propagation
8: Apply action \(a(t)\) to interact with the environment, obtaining the next state \(s(t+1)\) and reward \(r(t)\)
9: Put the \((s(t),a(t),s(t+1),r(t))\) into the replay buffer
10: Updating part:
11: Sample \(N\) samples from the replay buffer \(R\)
12:for all the \(N\) samples
13: Update two QNNs, \(Q_{1}\) and \(Q_{2}\) based on [PERSON]'s equation according to (5)
14: Based on the sampled states \(s(t)\), calculate the action \(a(t)\) from the FNNs and the log probability from Std NNs
15: Based on the states \(s(t)\) and actions \(a(t)\), calculate \(Q_{1}\), \(Q_{2}\), and \(\min_{q}(\min(Q_{1},Q_{2}))\)
16: Update the VNNs based on the loss function as described in (6)
17: Update all the parameters \(\omega_{ij}\), \(c_{ij}\), and \(\sigma_{ij}\) in FNNs according to the loss function, represented by (4)
18: Update all the parameters \(\varphi\) in Std NNs according to the loss function, represented by (4)
19: Update target VNNs, \(\theta_{v^{\prime}}\leftarrow\theta_{v}\)
20:endfor
21:endfor
```  
**Algorithm 1** SFPD