# RIS-Assisted Multi-Antenna AmBC Signal Detection Using Deep Reinforcement Learning
## 5 Numerical Results and Analysis
### Convergence Analysis  
In this section, the hyperparameters' effect on the RIS-TD3 intelligent signal detection algorithm's performance is investigated. In the RIS-TD3, the main hyperparameters are as follows: the hidden layer size, exploration step size, discount rate, and learning rate. The \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Algorithms** & **RIS-TD3** & **RIS-DDPG** & **SCA** & **SDR** \\ \hline
1 & 0.4188 & 0.3231 & 0.6420 & 10.4801 \\ \hline
2 & 0.3837 & 0.3349 & 0.5787 & 7.0571 \\ \hline
3 & 0.3684 & 0.3371 & 0.4785 & 7.5617 \\ \hline
4 & 0.3647 & 0.3241 & 0.5332 & 9.7516 \\ \hline
5 & 0.3868 & 0.4239 & 0.4490 & 5.9844 \\ \hline
6 & 0.382 & 0.3531 & 0.4675 & 7.6004 \\ \hline
7 & 0.369 & 0.3291 & 0.4493 & 6.6297 \\ \hline
8 & 0.3786 & 0.3142 & 0.4527 & 6.0317 \\ \hline
9 & 0.3526 & 0.3331 & 0.4411 & 7.2425 \\ \hline
10 & 0.361 & 0.3577 & 0.5159 & 7.2832 \\ \hline Average & 0.3766 & 0.3430 & 0.5008 & 7.5622 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Time consumed of various algorithms at N = 36 (unit: second). effect of these hyperparameters on the RIS-TD3 algorithm's performance is simulated. The other parameters and their corresponding values are as follows: the number of reflection units at RIS is 36, the length of batch learning data is 512, the discount rate is 0.99, the learning rate is 15 \(\times\) 10\({}^{-5}\), the hidden layer is 128, and the exploration step is 5000. The exploration noise gradually decreases when the empirical data are greater than two times the exploration step, with a decrease factor of 0.9999 until it decreases to 0.1. The hidden layer size:  
As shown in Figure 5, as the hidden layer gradually increases from 32 to 64, the maximum cumulative reward value of rounds of the RIS-TD3 algorithm keeps increasing, and the average energy ratio of normalized rounds increases to over 1000. However, when the hidden layer is further increased to 128 or 196, RIS-TD3\({}^{\prime}\)s performance gradually decreases. This is because as the hidden layer increases, the DNN network is larger, and its learning and training complexity increase significantly, making it difficult to perform the training in the limited number of rounds. This suggests that the choice of hidden layers is a compromise between the system performance and algorithm complexity. The exploration step:  
As illustrated in Figure 6, when the exploration step length is 4000, RIS-TD3 owes its first peak at around 80 rounds. As the iterations increase, RIS-TD3 owes its maximum peak at around 270 rounds, with a maximum peak of 131,703.35; when the exploration step length is 5000, RIS-TD3 has its maximum peak at around 90 rounds, with a maximum peak of 1,270,947.24. The optimal value can be achieved by RIS-TD3 quickly; when the exploration step length increases to 10,000, RIS-TD3 shows the first peak at 200 rounds and the maximum peak at 260 rounds, with a maximum peak of 463,016.65. According to the simulation results, it shows that as the exploration step length increases and more rounds are used, the later the DNN network starts learning, and the later the peak appears; however, with more exploration data as the initial sample for network learning, the faster the learning, and the higher the maximum peak can be achieved. When the exploration step is too large, the proposed RIS-TD3 algorithm cannot find the maximum value in the limited number of the iteration rounds, and the efficiency of the algorithm decreases. The discount rate:  
Figure 7 illustrates that when the discount rate is 0.95, RIS-TD3 owes the maximum value of 265,010.03 at round 140; when the discount rate increases to 0.99, the algorithm has the maximum value at round 90, and the maximum value reaches 1,270,947.24. This shows that the acquisition of the optimal value can be influenced significantly by the discount rate. The larger the discount rate is, the larger the reward of the algorithm can be achieved.