## 4 Performance Analysis and Simulation Outcomes
### Evaluation of the Proposed Solution Schemes  
Figure 4 compares the convergence performance of the RL-based proposed schemes in terms of the total reward achieved per episode during the learning stage. It is observed from Figure 4 that these rewards fluctuate rapidly and are apparently small in the first part of the episodes during the learning period and then become relatively stable and large. The total reward performance of the proposed Q-learning-based scheme (Scheme-Q) is the worst due to the large state-action space. The proposed _deep_ Q-learning-based solution scheme (Scheme-DQ) performs better than scheme-Q. This is because it approximates the complex mapping between the state-action spaces using a deep neural network with experience replay. The proposed actor-critic scheme (Scheme-AC) performs the best. This is because all parameters of this scheme are initialized, and the experiences are stored in the replay buffer. These parameters are then updated with the joint process of policy value learning during the learning stage. Here, the proposed Scheme-AC optimizes the policy with good convergence properties. Therefore, the performance of the proposed Scheme-AC is better than that of Scheme-Q and Scheme-DQ. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Values** \\ \hline Learning rates of the actor/critic & 0.0001/0.001 \\ \hline Discount factor & 0.99 \\ \hline Greedy rate & 0.1 \\ \hline Soft target update parameter & 0.001 \\ \hline Replay buffer size & 10,000 \\ \hline Mini-batch size & 62 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Simulation Parameters for RL model. Figure 4: Convergence of the RL-based proposed solution schemes. Figure 5 shows the SE performance with respect to the number of CUs for different solution schemes. The result implies that the SE performance of the network increases with the number of CUs with a fixed number of D2D pairs. This is because when the number of CUs increases, the D2D pairs have a better opportunity to access the best spectrum resources with less interference. It is also observed that the SE performance of the RL-based proposed schemes outperforms that of the other schemes. The SE performance of Scheme-Q and Scheme-AC remains at the top because they use the experience replay buffers to learn faster and benefit from the potential deployment of the IRS elements. Figure 6 presents the SE performance with respect to the number of D2D pairs for different solution schemes. It is observed that when the SINR requirement of the D2D pairs is less, the SE performance of all the solution schemes is high. However, at a higher SINR requirement for D2D pairs, the SE performance starts decreasing for all the schemes. This is because a higher SINR requires lower interference, which naturally decreases the  
Figure 5: SE versus the number of CUs. Figure 6: SE versus number of D2D pairs. SE. It can also be seen that the RL-based schemes yield better SE performance than the other schemes. Figure 8 shows that the SE performance increases for all four schemes with the number of IRS elements. The SE performance of scheme-2 (without IRS) remains unchanged for obvious reasons. Again, the RL-based proposed schemes outperform the other schemes. The reason behind this increasing trend is that more reflecting elements can further enhance the channel strength by passive beamforming and suppress the undesired channel interference. However, the increase in the SE performance slows down as the number of IRS elements becomes too large, which is caused by the increase in interference signal paths between the CUs and D2D pairs. In Figure 9, it can be observed that the SE performances of the four schemes first increase with the increase of the Rician factor and then remain almost constant. However, the SE performance of scheme-2 (without IRS) remains unchanged. Again, the proposed RL-based schemes outperform the other two schemes.