## 6 Channel Estimation Based Q-Learning Algorithm  
In the considered channel prediction scheme, it is assumed that the action spaces are discrete, therefore, we manage to use an RL-based Q-learning procedure as one of the candidates of RL schemes for parameters update in our examined cell [34; 35]. The Q-learning algorithm is categorized as a model-free, and off-policy reinforcement learning procedure, also a Q-learning agent is characterized as a value-based RL agent that has the role of updating a specific critic value function to enhance the future rewards. At a certain state, the agent can inspect and select the action for which the expected reward is maximized. In this section, RL based Q-learning is employed for channel prediction tasks in the MISO-NOMA cells where pilot symbols are also adopted to assist in the channel estimation process [36]. Therefore, it is assumed that there is coordination between BS and user devices such that the pilot symbols can be recognized at the BS and user terminals. In our work, we have considered the BS as the Q-learning agent, and we assume that the BS will start estimating the channel parameters for each user after user devices complete sending the pilot signals [37]. Therefore, in our developed RL based Q-learning algorithm can be utilized to estimate the CSI after the BS receives the pilot signals. The scenario for the channel prediction process based on the developed Q-learning model can be outlined in this way [38]. Firstly, at the start of each transmission time slot, user devices can send pilot symbols to BS across the uplink channel. Secondly, on the basis of the developed RL based Q-learning algorithm and availability of network information such as user's distance and path loss, BS (agent) can predict the downlink CSI for user devices. Thirdly, BS will generate the superposition coding signal and performs downlink data transmission. Finally, the receiver of each user terminal will receive the downlink transmitted data and the estimated channel parameters based on Q-learning algorithm will be utilized to decode the desired signal. In addition, each user device can feedback the signal-to-interference plus noise ratio (SINR) or the achieved rate to the BS to enhance the detection process. In this study, the main objective of the developed RL based Q-learning algorithm is to maximize the downlink sum rate and reduce the estimation loss. Instead of estimating the received signal, we primarily concentrate on incorporating the developed Q-learning model in the NOMA system for the purpose of channel estimation [39]. The RL-based Q-agent is designed to estimate the channel parameters by interacting with the environment, hence strict orthogonal pilot symbols are not required as shown in the standard procedures. Throughout the learning iteration, the Q-learning agent decides on the action that can enhance the approximated state-action value function \(Q(s,a)\) therefore, the expected long-term reward can be also maximized in the neural networks. It is worth  
Figure 2: Reinforcement Learning Framework. mentioning that when increasing the number of learning iterations, updating Q-values becomes more sufficient, and an improved channel approximation and sum rate reward can also be achieved [34; 36; 40]. In the proposed Q-learning scheme, the sum rate is presented at the learning time interval \(t\) as \(R_{t}\), hence, the instantaneous sum rate at time instant \(t\) can be shown as follows [15; 34]  
\[R_{t}=\sum_{i=1}^{M}\log(1+SINR_{it}) \tag{33}\]  
where \(SINR_{it}\) is the signal-to-interference plus noise ratio of user \(i\) at time instant \(t\) and \(M\) is the number of users in the MISO-NOMA cell. In this work, the optimum goal of the developed \(Q\)-learning algorithm is to maximize the total discounted reward \(R^{\gamma}\) starting from time instant \(t\), which can be denoted as  
\[R_{t}^{\gamma}=\sum_{k=t}^{\infty}\gamma^{k-t}R_{k+1} \tag{34}\]  
where \(R_{t}^{\gamma}\) is the discounted reward at time slot \(t\), and \(\gamma\) is the discount factor.