## 9 Simulation Results and Discussion
It can be observed that as the number of user devices in the cell is increasing, the suggested RL based Q-learning algorithm still shows dominance in accomplishing higher rates with respect to MMSE and DL based LSTM channel estimation methods. Similar to Figure 6, the RL actor-critic procedure applied  
Figure 5: Outage Prob. vs. power (\(Q\)-learning & Conventional NOMA (MMSE)). in [15] is created in our MISO-NOMA environment with the following parameters: the actor and critic networks are both composed of two hidden layers with 400 and 300 nodes, respectively. The learning rate for actor and critic networks are \(10^{-4}\) and \(10^{-3}\) respectively. The discount factor \(\gamma\) is set to be 0.9 and has a buffer size of \(10^{5}\)[15]. As shown in the results, the developed Q-learning scheme is showing an advantage over the actor-critic scheme with up to 6 users in the cell. Then, the hidden layers feature in the actor-critic procedure starts producing some sort of improvement in the sum rates compared to the Q-learning algorithm while the number of user terminals in the cell is increasing. Overall, these outcomes reveal that dependability can be assured by the suggested Q-learning algorithm even when the user devices in the cell are increased. In addition, it is worth saying that while increasing the user devices in the system, the interference will also grow up, thus the sum rate could be degraded. Figure 8 illustrates simulation outcomes for the achievable capacity for every device in the examined MISO-NOMA system when both the developed Q-learning algorithm and MMSE channel estimation procedures are implemented. The attained rate for near devices reveals substantial improvement by 10 b/s/Hz over far and middle users' rates. The superiority of the near user in terms of the achievable rate is anticipated, due to the stable channel situation for the near user compared to other devices in the system. Additionally, the suggested Q-learning algorithm still can deliver few visible improvements compared to the MMSE technique for far and middle users' environments, this slight improvement is associated with the interference and inadequate link conditions for far and middle devices. In addition to the three baselines comparisons implemented in Figures 6 and 7, we also create and implement RL based State-Action-Reward-State-Action (SARSA) algorithm [48; 49; 50] in Figures 9-11 for the purpose of more investigations and benchmark comparisons. The features and parameters of the SARSA algorithm are adapted in order that the SARSA procedure can be used as a channel estimator and compare the results of SARSA algorithm with the results obtained based on our developed Q-learning algorithm. Figure 8: Individual rate vs. power (\(Q\)-learning, Conventional NOMA (MMSE)). Figure 7: Sum rate vs. number of users (MMSE, LSTM, RL actor-critic, RL \(Q\)-learning). Figure 10: Outage Prob. vs. power (Q-learning, SARSA). Figure 9: BER vs. power (Q-learning, SARSA). The Q-learning algorithm and SARSA algorithm are two efficient RL algorithms, they are both table-based procedures with a Q-table to record equivalent Q-values of each state-action pair. However, when the size of state space increases, it will need a considerable amount of memory. Similar to the Q-learning algorithm, the SARSA algorithm also has exploration and exploitation processes, and it also needs a Q-table to record \(Q(s_{\mathbf{t}},a_{\mathbf{t}})\) value corresponding to state \(s_{\mathbf{t}}\) and action \(a_{\mathbf{t}}\). Differently, the running steps of the SARSA algorithm are as follows. First, according to the action selection scheme, the gent at the current state \(s_{\mathbf{t}}\), will select the action \(a_{\mathbf{t}}\). Then, the agent gets an immediate reward \(\mathbf{R}\) based on the corresponding \(Q(s_{\mathbf{t}},a_{\mathbf{t}})\) value. Finally, \(s_{\mathbf{t}}\) will transfer to \(s_{\mathbf{t+1}}\) and the agent will choose the next action \(a_{\mathbf{t+1}}\).