## 5 Simulation Results
### Results and Discussion
Therefore, it looks like a straight line compared with the other two algorithms that have a larger reward in Figure 7. Constrained DDPG, however, uses the constraint as penalty in its reward function, showing a similar look to constrained SAC. But it is obvious that its convergence speed is slower. The reason for this is that constrained SAC has entropy in the reward function and noise in its action for more exploration, which will eventually improve the convergence speed. Therefore, CSAC has a better exploration performance than DDPG, which only adds noise in the action. And the reward DDPG achieves is a bit lower than that of our algorithm. Figure 6: Capacity of constrained SAC with different users. The energy efficiency performance comparison is depicted in Figure 8. Original SAC has the lowest performance because its rate requirement cannot be satisfied. DDPG once again presents relatively slow convergence speed. Its energy efficiency is higher than original SAC but lower than constrained SAC. SCA converges after five iterations. Its energy efficiency is even lower than DDPG, demonstrating that DRL could achieve better performance than the model-based optimization approach without any prior knowledge. However, unlike the \"black-box\" characteristic of DRL, SCA has more explicit expression. CSAC outperforms the other three with the highest performance, which validates its effectiveness. We also compare the capacities of four algorithms to check if the constraint satisfied. In Figure 9, the straight line represents the capacity requirement in the three users scenario, which is 12 bps/Hz. CSAC, DDPG, and SCA meet the capacity requirement with their different approaches. Original SAC, on the other hand, fails to satisfy the constraint, even  
Figure 8: Energy efficiency with different algorithms. Figure 7: Average reward with different algorithms. though its fluctuation goes beyond the straight line in rare cases. CSAC achieves the highest capacity among them, which results in the highest energy efficiency in Figure 8.  
We investigate the effect of the on-off mechanism. In Figure 10, we vary thresholds to observe the energy efficiency change of the three algorithms. As the threshold increases, it is more likely that the last action will be taken instead of finding a new one. Consequently, the energy efficiency will decrease as the action does not perfectly apply to the new state. According to [9], when the strength difference of two channels is 0.3, additional 5 dB loss will be introduced. Although less learning process with a bigger threshold will cost less time, the tradeoff between time cost and performance should be considered carefully. Normally, the threshold needs to be less than 0.3, or it will deteriorate the capacity and energy efficiency. Figure 10: Energy efficiency with different on–off mechanism thresholds. Figure 9: Capacity with different algorithms. Figure 11 shows the time cost performance of three algorithms. All of their time consumption is lowered when adopting an on-off mechanism because some of the learning process is substituted by the actions from the last states. Among them, the original SAC has the lowest time consumption because it does not need to compute the Lagrangian multiplier or penalty. DDPG has a simpler architecture and constraint in its reward function than CSAC, thus the lower time cost. Finally, the energy efficiency performance of CSAC and DDPG with different constraint requirements is illustrated in Figure 12. As the rate constraint grows, it is more difficult for CSAC and DDPG to find the suitable precoders, so their energy efficiency declines. However, the penalty design in DDPG is more sensitive to requirement change, and it drop  
Figure 11: Average time cost per execution with different on–off mechanism thresholds. Figure 6:0 Energy efficiency with different constraint requirements.