# The Deep Deterministic Policy Gradient Algorithm Based on RIS Technology in a Coal Mine Tunnel
## 3 Deep Reinforcement Learning Based Optimization
### DDPG Algorithm Elements and Processes  
To enhance the stability of training, DDPG incorporates an experience replay mechanism. The experience replay buffer stores samples generated from interactions between the agent and the environment, improving training efficiency by reducing data correlation through random sampling. Each experience usually includes the current state, the action taken, the reward received, the next state, and a termination flag indicating whether the interaction has ended. The specific definitions of state \(s_{t}\), action \(a_{t}\), and reward \(r_{t}\) are as follows:  
1. State \(s_{t}\): Represents the environment, comprising the transmit power \(P_{t}\) at time \(t\), the channel matrix \(\mathbf{H}_{nm}\) from the BS to the RIS, and the signal matrix \(\mathbf{H}_{ru}\) from the RIS to the user; the size of the state space is as follows: \[N_{i}=2K+2K^{2}+2 MK+2N+2 MN+2 NK\] (29)
2. Action \(a_{t}\): Comprises the beamforming matrix \(\mathbf{G}\) and phase shift matrix \(\mathbf{\Phi}\) at time \(t\). The size of the action space is as follows: \[N_{o}=2 MK+2N\] (30)
3. Reward \(r_{t}\): The reward at the time \(t\) corresponds to the value of the objective function defined in Equation (19). \[r_{t}=\log_{2}(1+\beta_{k})\] (31) The flow of the DDPG algorithm proposed in this paper is as follows:```
1:trainst beam forming matrix \(G\), phase shift matrix \(\Phi\), experience replay buffer \(E\), parameters of critic training network \(w_{c}\), parameters of actor training network \(w_{a}\), parameters of the target network \(w_{c^{\prime}}\), \(w_{a^{\prime}}\)Inputs: channel matrix from BS to RIS \(\mathbf{H}_{nm}\), channel matrix from RIS to user \(\mathbf{H}_{mu}\)Outputs: Q-value function, optimal action \(a=\{G,\Phi\}\)
2: Get the initial state \(s_{0}\) from the environment;
3: Calculate the action \(a_{t}\) at each moment in turn, \(a_{t}=\pi(s_{t},w_{a})\);
4: Based on the action \(a_{t}\), interact with the environment to get the next state \(s_{t+1}\) and instant reward \(R_{t+1}\);
5: Store the transfer tuple \((s_{t},a_{t},R_{t+1},s_{t+1})\) in the experience replay buffer \(E\);
6: Randomly draw a mini-batch of size \(W\) from the experience replay buffer \(E\);
7: Calculate the target Q-value according to Equation (25);
8: Update the parameters \(w_{c}\) and \(w_{a}\) of the critic and actor networks according to Equations (24) and (27) and the target network parameters \(w_{c}\), \(w_{a}\), \(r\) according to Equation (28);
9: Update the state to the next state \(s_{t}=s_{t+1}\). ```

**Algorithm 1**DDPG Algorithm

## 4 Results and Discussions

The performance of the RIS-assisted coal mine communication system based on the DDPG algorithm was evaluated, and it was divided into two scenarios: ideal reflection and non-ideal reflection of RIS. In both scenarios, system performance was influenced by factors such as the learning rate \(lr\), fading rate \(dr\), base station power, and the number of RIS units. The key simulation parameters are shown in Table 1.

In the simulation, the randomly generated channel matrices \(\mathbf{H}_{nm}\) and \(\mathbf{H}_{nu}\) obey Rayleigh distribution, with the total effective number of scattering paths being \(R=128\). The gain of the receiving and transmitting antennas was \(5 dBm\), the roughness variance of the coal mine tunnel wall was \(h=0.01\), and the relative permittivity was \(\varepsilon_{r}=5\). In this paper, we used the average reward as a measure of the performance of the system, which was defined as follows:

\[R_{arg}=\frac{\sum_{t=1}^{T}\mathit{reward}(t)}{t},t=1,2,\ldots,T \tag{32}\]

### Performance Comparison of Different DRL Algorithms

This paper selects three typical DRL algorithms--DDPG, DQN, and A3 C--to explore their effects on link rate optimization in RIS-assisted coal mine tunnel communication systems. The simulation conditions are defined as \(P_{t}=38 dBm,M=K=8,N=16\), and RIS ideal reflection.