# Soft Fuzzy Reinforcement Neural Network Proportional-Derivative Controller
## 2 Preliminaries  
### Reinforcement Learning (RL)  
RL is a model-free approach where the controller continuously interacts with the environment to receive feedback and iteratively learns control strategies [28]. The goal of RL is to maximize the cumulative reward (CR). Based on the current state, the RL agent selects an action, which the environment then responds to by providing a reward and transitioning to the next state. Through this cyclic training process, the agent learns to determine the optimal actions that yield the highest CR given the current state. As shown in Figure 1, the agent and the environment together constitute the RL framework. The agent is determined by the policy, \(\pi\), which is a mapping from the state space \(S\) to the action space \(A\) (\(\pi:S\to A\)). At each time step, the policy selects an action (\(a_{t}\)) based on the current state (\(s_{t}\)), which is then applied to the environment to obtain a reward (\(r_{t}\)). The goal of RL is to find an optimal policy, denoted as \(\pi^{*}\), that maximizes the cumulative reward (CR) over an epoch. However, relying solely on the current reward \(\mathbf{r}\) to train the policy makes it challenging to achieve the maximum CR because we need to consider long-term rewards, not just the immediate reward (\(r_{t}\)). This is where the critic network comes into play, evaluating the quality of the current state-action pair. This forms the basis of the actor-critic algorithm. Among these, the Soft Actor-Critic (SAC) [29] algorithm stands out due to its efficient exploration-exploitation trade-off, off-policy learning, and superior stability in continuous action spaces. The SAC algorithm uses an experience replay buffer to store tuples of (state, action, reward, next state). It resamples batches of these tuples from the buffer and trains the neural network using stochastic gradient descent (SGD) [30]. Additionally, the SAC algorithm incorporates entropy into the exploration process to balance exploration and exploitation, which introduces stochasticity into the decision-making process. Consequently, the cost function of the policy in SAC is:  
\[J(\pi)=\mathbb{E}[R(\tau)+\alpha H(\pi)] \tag{1}\]  
where  
* \(J(\pi)\): the objective function of the policy network. * \(\mathbb{E}[R(\tau)]\): the expected return of a trajectory. * \(\alpha\): a hyperparameter used to balance the weights of return and entropy. * \(H(\pi)\): the entropy of the policy network distribution. SAC is an actor-critic architecture with a critic network that consists of a state-action value (\(Q\)-value) network and a state value (\(V\)-value) network. The objective functions for these networks involve maximizing the CR and entropy, leading to what is known as  
Figure 1: Reinforcement learning framework: interaction between the agent and environment, where \(s\) represents the state, \(r\) denotes the reward, and \(a\) is the action taken by the agent. the soft \(Q\)-value function and the soft \(V\)-value function. The objective functions for these networks involve maximizing the CR and entropy, leading to what is known as the soft \(Q\)-value function and the soft \(V\)-value function. \[Q_{\theta}^{\pi}(s,a)=\mathbb{E}_{s_{t},a_{t}\sim\rho_{\pi}}\bigg{[}\sum_{t=0}^ {\infty}\gamma^{t}r(s_{t},a_{t})+\alpha\sum_{t=1}^{\infty}\gamma^{t}H(\pi( \cdot|s_{t}))\Big{|}s_{0}=s,a_{0}=a\bigg{]} \tag{2}\]  
\[V_{\psi}(s_{t})=\mathbb{E}_{s_{t},a_{t}\sim\rho_{\pi}}\big{[}Q_{\theta}^{\pi}( s,a)-\alpha\log\pi_{\varphi}(a_{t}|s_{t})\big{]} \tag{3}\]  
where \(\gamma\) is a hyperparameter known as the discount factor, and \(\alpha\) is the temperature factor.