# Deep Reinforcement Learning-Based Deployment Method for Emergency Communication Network
## 4 Proposed DRL-Based Method
### Introduction of Algorithm  
The basic idea of our algorithm has been introduced in Section 4.1. The training process of the algorithm, as shown in Algorithm 1, can be roughly divided into three stages. Initial stage. The parameters \(w\) of the value network are randomly initialized and then copied to the target network to generate its initial parameters. Subsequently, the training environment is loaded, and an experience replay buffer is established to store interaction data. To enhance algorithmic stability, the update frequency of the target network parameters is constrained by a threshold \(\tau\). Exploration stage. To expand the scale of the neural network training dataset, the agent performs \(D\) element deployments within a single iteration cycle. Each epoch initializes the environment and state \(S_{0}\) by randomly generating an initial point. Then, the agent follows a predefined priority sequence to deploy elements. During each deployment, the agent employs an \(\varepsilon\)-greedy strategy for action decision-making \(a_{t}\): with probability \(\varepsilon\), it explores random actions, and with probability \(1-\varepsilon\), it selects the current optimal policy. The environment returns an immediate reward value \(r_{t}\) based on the action execution result and transitions the state from \(S_{t}\) to a new state \(S_{t+1}\). After completing a single element deployment, the system injects the experience data tuple \(\{S_{t},a_{t},R_{t},S_{t+1}\}\) into the replay buffer for storage. Training stage. The parameters \(w\), \(w^{-}\) of the value network and target network are updated through an experience replay buffer \(B\). Specifically, \(H\) transition samples \(\{S_{t},a_{t},R_{t},S_{t+1}\}\) are randomly sampled from \(B\) to form a mini-batch for neural network training. The Temporal Difference (TD) target is calculated as shown in Equation (8). However, since \(Q_{t+1}\) is an \(n\times n\) matrix, traversing all \(n^{2}\) actions directly would lead to excessive computational complexity, degrading algorithm efficiency. To address this, a two-stage optimization strategy is proposed. 1. Action Space Dimensionality Reduction: A grid sampling strategy with a step size of two reduces the original \(n^{2}\) action space to \(n^{2}/4\) candidate actions. Combining the geographic environment encoding matrix (deployable areas marked as 1, obstacles as 0), only valid deployment regions \(A^{\prime}\) are retained for value evaluation to further reduce redundant computations. 2. Value Matrix Reconstruction: For the next state \(S_{t+1}\), the optimal action \(a_{\max}=\underset{A^{\prime}}{argmax}\ Q_{w}(S_{t+1},a)\) is computed via the target network, and its corresponding value \(q_{t+1}=Q_{w^{-}}(S_{t+1},a_{\max})\) is extracted to construct the reduced-value set \(Q^{\prime}_{t+1}\). Bilinear interpolation then restores \(Q^{\prime}_{t+1}\) to \(n\times n\) resolution \(Q_{t+1}\), balancing efficiency and precision. Finally, the target value matrix \(\hat{Q}_{t}=R_{t}+\gamma Q_{t+1}\) is generated using the TD equation [27]. By Equation (9), the mean squared error between the estimated Q value \(Q_{w}(S_{t},a)\) and target Q value \(\hat{Q}_{t}\) serves as the loss function, updating the value network parameters \(w^{*}\) via stochastic gradient descent. \[q_{t+1} = \begin{cases}Q_{w^{-}}[S_{t+1},\underset{A^{\prime}}{argmax}Q_{w}(S _{t+1},a)],&a\in A^{\prime}\\ 0,&a\
otin A^{\prime}\end{cases}