# The Deep Deterministic Policy Gradient Algorithm Based on RIS Technology in a Coal Mine Tunnel
## 3 Deep Reinforcement Learning Based Optimization
### DDPG Algorithm Elements and Processes
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Height of the tunnel \(2b\) & 6 m \\ Width of the tunnel \(2a\) & 5 m \\ Buffer size for experience replay \(E\) & \(10^{6}\) \\ The number of experiences in the mini-batch \(W\) & 16 \\ The number of episodes & \(5\cdot 10^{3}\) \\ The number of steps in each episode & \(2\cdot 10^{4}\) \\ Discounted rate for future reward \(\gamma\) & 0.999 \\ Learning rate of action/critic networks \(\beta_{a}/\beta_{c}\) & \(10^{-3}\) \\ Targeted action/critic network learning rate \(\lambda_{a}/\lambda_{c}\) & \(10^{-3}\) \\ Action/critic of network decaying rate \(\tau\) & \(10^{-6}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Simulation parameters. Figure 4 shows that the link rate optimization performance of the three algorithms is DDPG > A3C > DQN. Specifically, DDPG achieves the highest average link rate (about 6.5 bps/Hz) with superior convergence speed and stability, followed by A3C with a link rate of about 5.5 bps/Hz and DQN with the worst performance with a link rate of about 3.4 bps/Hz. The DDPG algorithm is superior to DQN and A3C, mainly because its \"actor-critic\" structure is suitable for continuous action space and can more accurately optimize the phase and gain of the RIS reflection unit, thereby increasing the link rate. At the same time, DDPG introduces the target network and experience replay mechanism to enhance the stability of convergence, making communication quality improvement even more significant in complex coal mine environments. ### Effect of Neural Network Parameters on Average Reward

#### 4.2.1 Effect of Learning Rate on Average Reward

The effect of different learning rates on the average reward as a function of the number of steps is shown in Figure 5. When \(lr\) is 0.01, the average reward \(R_{\text{avg}}\) is 2 bps/Hz. For lower values (e.g., 0.0001 and 0.00001), the final reward value is 3 bps/Hz, and when \(lr\) is 0.001, \(R_{\text{avg}}\) reaches 6.6 bps/Hz. It helps the model to achieve the best average reward. #### 4.2.2 Effect of Decaying Rate on Average Reward

The effect of different decaying rates on the average reward as the number of steps changes is shown in Figure 6. As can be seen from the figure, with the accumulation of training steps, \(R_{\text{avg}}\) shows an upward trend at all decay rates. Specifically, when \(dr=0.00001\), the performance is the best in the entire training process, with the most

Figure 4: Comparison of average rewards under different algorithms. Figure 5: The effect of learning rate on average reward. significant upward trend in \(R_{\text{avg}}\), and finally reaches 6.6 bps/Hz. Then, when \(dr\) is 0.001 and 0.00001, the reward level is slightly lower than \(dr=0.000001\), which is about 5.7 bps/Hz. Therefore, choosing an appropriate decay rate is crucial to improving the average reward of the model. ### Effect of Base Station Transmit Power on Average Reward

#### 4.3.1 Comparative Analysis of Average Reward at Different Power

The effect of different base station transmit powers on the average reward as the number of steps changes is shown in Figure 7. The simulation conditions are set to \(M=N=K=8\). The average reward under different powers is shown in Table 2. As the transmission power increases from 36 dBm to 38 dBm, the link rate of the communication system is improved regardless of ideal or non-ideal conditions. This shows that increasing the transmit power can significantly enhance the communication capability of the system. Under the ideal reflection condition with 38 dBm power, the rate increases from 2 bps/Hz to 6.7 bps/Hz with an increase of 4.7 bps/Hz, while under the non-ideal condition, the rate increases from 2 bps/Hz to 4.7 bps/Hz with an increase of only 2.7 bps/Hz. In short, RIS technology can significantly improve the performance of communication systems under appropriate transmit power, especially under ideal reflection conditions. Figure 6: Effect of decay rate on average reward. Figure 7: Effect of power on average reward.