## 7 Q-Learning Network Architecture
### Dataset Preparation  
Essentially, path loss and the distance between every user terminal and the BS need to be specified in the dataset to facilitate the random generation of the channel weights for every user device in the examined MISO-NOMA network [43]. In the beginning, pilot symbols are created, transmitted, and identified at the BS and at the receiver of every device. Additionally, power factors for every device in the cell need to be initially assigned. The channel weights for every device in the cell are initialized to set up the Q-table values, and during the algorithm iterations, the Q-values are modified according to a Q-learning procedure [34; 35; 36]. Throughout the learning process and for the sake of updating the \(Q\)-table, the discount factor \(\mathbf{\gamma}\), learning rate \(\mathbf{\kappa}\), the target rate \(\mathbf{R}_{T}\), current state, and the terminal state should be identified. In our developed Q-learning algorithm, the Q-agent will choose the next state at random and set it as the next state, then the Q-learning agent will inspect all possible actions available to move to the next state. Next, the Q-learning agent will carefully identify the best action \(\mathbf{a}\), that satisfies the maximum value for \(Q(s,a)\) to move to the new state. After moving to the new state, a reward value will be assigned to the agent as a measure of how successful this transition was in order to move to the new state [44]. During the update phase, we compute \(\Delta Q\), which represents the difference between the new generated  
Figure 3: The architecture of the proposed Channel prediction scheme based developed Q-Learning algorithm. value function and the preceding value function of \(Q(s,a)\). Then, update the resultant \(Q(s,a)\) value in the \(Q\)-table according to the following formula. \[Q(s,a)=Q(s,a)+\alpha\cdot\Delta Q \tag{39}\]  
Based on the updated Q-values in Q-table and the updated channel gain, a new achieved rate can be calculated and compared to the target rate for each user device in the cell. In the developed Q-learning algorithm, once the optimum rate or the terminal state is reached, the developed Q-matrix will be employed to compose the channel taps for each user device. The developed Q-learning procedure for channel approximation can be summarized as presented in Algorithm 1. ```
1. Initialize the \(\mathbf{Q}\) table values and initialize the reward matrix \(\mathbf{R}\) with zeroes. Inputs
2. Number of Iterations and the size for the channel parameters for every user device. 3. Initial distance \"\(\mathbf{d}\),\" of every user device from the BS. 4. Path loss parameter \"\(\mathbf{\theta}\)\". 5. Design random pilot symbols. 6. Initialize the random channel parameters for each user \"\(\mathbf{h}_{ij}\)\" based on fading model, \(j\in[1,\ 2,\ \ldots,N]\) and \(i\in[1,\ 2,\ \ldots,M]\). \(N\) is the number of antennas at BS and \(M\) is the number of devices in the cell. 7. Designate the power percentage \"\(\mathbf{\eta}_{i}\)\" for each user. 8. Determine system bandwidth \"\(\mathbf{B}\)\", Total transmit power \"\(\mathbf{P}_{T}\)\", and noise spectral density \"\(\mathbf{N}_{o}\)\"
9. Assign the desired channel parameters \"\(\mathbf{h}_{id}\)\" and the target rate \"\(\mathbf{R}_{T}\)\"
10. Based on the channel gain \(\left|\mathbf{h}_{ij}\right|^{2}\), total transmit power \"\(\mathbf{P}_{T}\)\", and initial power factor for each user \"\(\mathbf{\eta}_{i}\)\", signal to interference noise ratio \"\(\mathbf{SINR}\),\", minimum required rate \"\(\mathbf{R}_{i}\)\" can be calculated for each device. 11. At each iteration, compare the initial generated rate \"\(\mathbf{R}_{i}\)\" with the target rate \"\(\mathbf{R}_{T}\)\". 12. Update the values for the \(\mathbf{Q}\)-table that represent the current state and action pair \(\mathbf{Q}(\mathbf{s},\mathbf{a})\). 13. identify discount factor \"\(\mathbf{\gamma}\)\", learning rate \"\(\mathbf{\alpha}\)\", the current state, and the terminal state. 14. Choose the next state at random and set it as the next new state. 15.