# Augmented Lagrangian-Based Reinforcement Learning for Network Slicing in IIoT
## 4 Simulation
### Parameter Setting  
We consider a wireless scenario with multiple BSs and eMBB/URLLC/mMTC devices, where the locations of devices are randomly distributed. The channel fading between a device and BS follows a Rayleigh distribution, varying with time. Combining the distance-based path loss and fading, the channel condition in an episode can be calculated. In the simulation, the number of BSs is \(M=2\), the number of devices in eMBB, URLLC, and mMTC slices are \((3,3,4)\) or \((5,5,5)\). Three different weight designs are considered, that is, \((w_{\text{eM}},w_{\text{UR}},w_{\text{mM}})=(\frac{1}{3},\frac{1}{3},\frac{ 1}{3})\), \((\frac{2}{3},\frac{1}{6},\frac{1}{6})\), or \((\frac{1}{10},\frac{3}{5},\frac{3}{10})\). Furthermore, the transmission power \(P_{m}\), the path loss exponent \(\alpha\) and noise power \(\sigma^{2}\) are set as 2 W, 3.09 and \(10^{-9}\) W, respectively [32]. For mMTC devices, \(B_{0}=0.18\) MHz, for eMBB devices, \(R_{0}=4\) Mbps, for URLLC devices, \(T_{0}=20\) ms. The available bandwidth for each BS is set as \(B_{m}=10,12.5\), or 15 MHz, the neural networks are trained by the Adam optimizer, and batch size is set as \(K=256\). All these simulation parameters are also listed in Table 1 for clarity. In Figure 4, we plot the average reward achieved by the proposed AL-SAC algorithm when the maximum bandwidth \(B_{m}=10,12.5\), and \(15\) MHz available with the number of devices \(N=10\) for each BS, respectively, as well as \(B=10\) MHz, \(N=15\). Firstly, it is observed that in all cases, the proposed AL-SAC algorithm converges, although a slight fluctuation exists when \(B_{m}=15\) MHz, \(N=10\). Secondly, we can see that with the growth of available bandwidth, the curve of achieved reward is a little less stable. The reason behind this is that the bandwidth allocation options for multiple devices also increase, i.e., the action space. Thirdly, it can be seen that with the growth of available bandwidth resources, the reward achieved by the proposed algorithm increases apparently, which implies increasing system utilities due to more radio resources being available. Lastly, compared with \(B_{m}=15\) MHz, \(N=10\), the proposed AL-SAC algorithm with \(B_{m}=15\) MHz, \(N=15\) can reach similar reward but more converge speed because when more devices share the same limited bandwidth resources, the weighted overall utility is not higher. Moreover, the action space for each device reduces when the devices number increases, resulting in more stable performance. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Number of BSs, \(M\) & 2 \\ \hline Number of devices, \(N\) & 10 or 15 \\ \hline Number of devices in different slices, \(N_{\text{eM}}\), \(N_{\text{UR}}\), \(N_{\text{mM}}\) & 3,3\(\mathcal{A}\) or 5,5,5 \\ \hline Transmission power, \(P_{m}\) & 2 W \\ \hline Path loss exponent, \(\alpha\) & 3.09 \\ \hline Noise power, \(\sigma^{2}\) & \(10^{-9}\) W \\ \hline Minimum bandwidth allocated for mMTC devices, \(B_{0}\) & 0.18 W \\ \hline Minimum transmission rate for eMBB devices, \(R_{0}\) & 4 Mbps \\ \hline Maximum delay for URLLC devices, \(T_{0}\) & 20 ms \\ \hline Maximum bandwidth for each BS, \(B_{m}\) & \(10,12.5,15\) MHz \\ \hline Weight, \((w_{\text{eM}},w_{\text{UR}},w_{\text{mM}})\) & \((\frac{1}{3},\frac{1}{3}),(\frac{2}{3},\frac{1}{6}),(\frac{1}{10},\frac{3}{5}, \frac{3}{10})\). \\ \hline Batch size, \(K\) & 256 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Settings of parameters. Figure 4: Performance for AL-SAC algorithm with different maximum available bandwidths. Meanwhile, to verify the proposed algorithm can provide an effective solution to the constrained RL problem, in Figure 5, we also show the bandwidth constraint in the same scenarios as Figure 4. It clearly can be seen that in all the cases, the proposed AL-SAC can meet bandwidth requirements after 100 episodes. This also shows that the proposed algorithm can provide a feasible and effective solution to the constraint optimization problem considered.