# Augmented Lagrangian-Based Reinforcement Learning for Network Slicing in IIoT
## 3 Proposed Augmented Lagrangian-Based Reinforcement Learning
### Proposed AL-SAC Algorithm
#### 3.3.5 Policy Network \(\pi\)  
This network searches the optimal policy according to the estimated values generated by networks in the critic and constraint part. It generates an action based on policy \(\pi\) for each state, i.e., \(\pi_{\theta}(\times|\mathbf{s})\), where \(\theta\) is the parameter of the policy network. As mentioned earlier, we consider a two-stage action selection. Specifically, the discrete action will be selected first, i.e., the BS connection state will be determined. Then with the corresponding channel state between the device and BS, the bandwidth allocation is determined.  
Then, we can update \(\pi\) by maximizing the following function  
\[J_{\pi}(\theta)=\mathbb{E}\bigg{[}\log\big{(}\pi_{\theta}(\mathbf{a}^{(t)}|\mathbf{s}^ {(t)})\big{)}-Q_{\bar{\varphi}}+\sum_{m\in\mathcal{M}}\Big{(}\lambda_{m}Q_{ \varphi_{m}}(\mathbf{h}_{m}^{(t)},\mathbf{  
The workflow of the proposed AL-SAC algorithm is summarized in Algorithm 1. Specifically, lines 2-6 illustrate the experience collected from the environment with the current policy, and then the update of the networks is presented in lines 8-18.  
```
0: Initialize: Initialize network parameters \(\phi,\bar{\phi},\theta,\psi,\varphi_{m},m=1,\ldots,M\), Lagrange multiplier \(\lambda\), \(\mu\), and replay buffer \(\mathcal{D}\).
0: Initialize: Initialize the time episode \(T\), and batch size \(K\).
1:for each time episode \(t=1,\ldots,T\)do
2: Observe the environment \(\mathbf{s}^{(t)}\).
3: Select the action \(\mathbf{a}^{(t)}\sim\tau_{\theta}(\mathbf{s}^{(t)})\).
4: Calculate total bandwidth allocated in each BS, i.e., \(G_{m}^{(t)}=\sum_{n\in\mathcal{N}}x_{nm}\times b_{nm}\).
5: Calculate the reward \(r^{(t)}\) depending on \(G_{m}^{(t)}\), i.e., (20) or (21).
6: Update the replay memory buffer \(\mathcal{D}\leftarrow\mathcal{D}\cup\{\mathbf{s}^{(t)},\mathbf{a}^{(t)},r^{(t)},\mathbf{s} ^{(t+1)},G_{m}^{(t)}\}\).
7:if\(t>K\)then
8: Randomly select a batch of samples \((\mathbf{s}^{(t)},\mathbf{a}^{(t)},r^{(t)},\mathbf{s}^{(t+1)})\) from the replay memory buffer \(\mathcal{D}\).
9: Update the value network \(\mathcal{V}\) by \(\phi\leftarrow\phi-\xi_{V}\
abla_{\phi}J_{V}(\phi)\).
10: Update the Q-value network \(\mathcal{Q}\) by \(\psi\leftarrow\psi-\xi\
abla_{\phi}J_{Q}(\psi)\).
11: Update the cost networks \(\mathcal{C}\) by \(\varphi_{m}\leftarrow\varphi_{m}-\xi_{m}\
abla_{\phi}J_{Q}(\varphi_{m})\).
12: Update the policy network \(\pi\) by \(\theta\leftarrow\theta-\xi_{\pi}\
abla_{\theta}J_{\pi}(\bar{\theta})\).
13: Update the Lagrangian multiplier networks \(\mathcal{L}\) by \(\lambda_{m}\leftarrow\lambda_{m}-\mu\
abla_{\lambda_{m}}J(\lambda)\).
14:if Constraints not satisfy the requirement then
15: Enlarge the \(\upmu\) with a scalar.
16:endif
17: Update the parameter \(\bar{\phi}\) by \(\bar{\phi}\leftarrow\delta\phi+(1-\delta)\bar{\phi}\).
18:\(\bar{\varphi}_{m}\) in cost networks are periodically updated by copying \(\varphi_{m}\).
19:endif
20:endfor
```  
**Algorithm 1** Augmented Lagrangian-based soft actor-critic (AL-SAC)