## 4 Constrained SAC Algorithm
### The Proposed Constrained SAC Algorithm  
Standard SAC is a policy gradient approach which uses the actor network to evaluate the policy and critic network to improve the policy. Additionally, the entropy of the policy is used as a penalty to improve its exploration, which yields fast convergence speed [43]. The maximum objective with entropy in it could be achieved when the optimum policy is found, which can be expressed as:  
\[\pi^{*}=\operatorname*{arg\,max}_{\pi}\sum_{t=0}^{T}\mathbb{E}_{s^{(t)},a^{(t )}}\Big{[}r\Big{(}s^{(t)},a^{(t)}\Big{)}+\mu_{\pi}\mathcal{H}\Big{(}\pi\Big{(} a^{(t)}\Big{|}s^{(t)}\Big{)}\Big{)}\Big{]}, \tag{22}\]  
where \(\mu_{\pi}\) is the temperature parameter that regulates the relative importance of the entropy, which could increase the variance of the policy distribution so that more actions are available to be chosen. The Shannon entropy of the policy is defined as:  
\[\mathcal{H}\Big{(}\pi\Big{(}a^{(t)}\Big{|}s^{(t)}\Big{)}\Big{)}=-\sum_{a^{(t )}}\pi\Big{(}a^{(t)}\Big{|}s^{(t)}\Big{)}\log\pi\Big{(}a^{(t)}\Big{|}s^{(t)} \Big{)}. \tag{23}\]  
The architecture of the proposed constrained SAC is illustrated in Figure 3. As shown in the picture, mainly three different components with five neural networks are used in this scheme, which are elaborated below. To evaluate the soft state value, a deep neural network (DNN) with a target network is adopted, which are represented by parameters \(\psi\) and \(\bar{\psi}\), respectively. The soft state value is used for soft Q-value approximation, both of which are utilized to evaluate received return in the future and are defined as:  
\[V\Big{(}s^{(t)}\Big{)}=\mathbb{E}_{a^{(t)}\sim\pi}\Big{[}Q\Big{(}s^{(t)},a^{(t) }\Big{)}-\log\pi\Big{(}a^{(t)}\Big{|}s^{(t)}\Big{)}\Big{]}, \tag{24}\]  
\[Q\Big{(}s^{(t)},a_{t}\Big{)}=r\Big{(}s^{(t)},a_{t}\Big{)}+\rho\mathbb{E}_{s^{( t+1)}\sim\rho}\Big{[}V\Big{(}s^{(t+1)}\Big{)}\Big{]}, \tag{25}\]  
where \(\rho\) is the discount factor. With the objective and its gradient below, we can update the soft value network using the gradient descent method. This network is trained to minimize the square residual error with a target network to stabilize the training. \[\mathcal{J}_{V}(\psi)=\mathbb{E}_{s^{(t)}}\bigg{[}\frac{1}{2}\Big{(}V_{\psi} \Big{(}s^{(t)}\Big{)}-\mathbb{E}_{a_{t}}\Big{[}Q_{\theta}\Big{(}s^{(t)},a^{(t) }\Big{)}-\log\pi_{\phi}\Big{(}a^{(t)}\Big{|}s^{(t)}\Big{)}\Big{]}\Big{)}^{2} \bigg{]}, \tag{26}\]  
\[\
abla\mathcal{J}_{V}(\psi)=\
abla_{\psi}V_{\psi}\Big{(}s^{(t)}\Big{)}\Big{[} V_{\psi}\Big{(}s^{(t)}\Big{)}-Q_{\theta}\Big{(}s^{(t)},a^{(t)}\Big{)}+\log\pi_{ \phi}\Big{(}a^{(t)}\Big{|}s^{(t)}\Big{)}\Big{]}. \tag{27}\]  
The soft Q-value is estimated by double soft Q-value network. The two-Q-network design is meant to avoid overestimation. Parameters in this network could be updated by minimizing the soft Bellman residual. The objective and its gradient are expressed as:  
\[\mathcal{J}_{\text{Q}}(\theta)=\mathbb{E}_{s^{(t)},a^{(t)}}\bigg{[}\frac{1}{2 }\Big{(}Q_{\theta}\Big{(}s^{(t)},a^{(t)}\Big{)}-\Big{(}r\Big{(}s^{(t)},a^{(t) }\Big{)}+\rho\mathbb{E}_{s^{(t)}\sim\rho}\Big{[}V_{\bar{\psi}}\Big{(}s^{(t+1) }\Big{)}\Big{]}\Big{)}\Big{)}^{2}\bigg{]}, \tag{28}\]  
Figure 3: Architecture of constrained SAC. \[\
abla\mathcal{J}_{Q}(\theta)=\
abla_{\theta}Q_{\theta}\Big{(}s^{(t)},a^{(t)} \Big{)}\Big{[}Q_{\theta}\Big{(}s^{(t)},a^{(t)}\Big{)}-r\Big{(}s^{(t)},a^{(t)} \Big{)}-\rho V_{\bar{\theta}}\Big{(}s^{(t+1)}\Big{)}\Big{]}. \tag{29}\]  
Finally, the policy network provides the action for agent based on the optimal policy it finds. By minimizing the Kullback-Leibler (KL) divergence between the policy and soft Q-value, the parameters could be learned. The objective and its gradient are listed below.