## 4 Problem Formulation  
In the CBTC scenario, there are different numbers of wayside nodes to assist information transmission depending on the distance between two trains; thus, the link selection for transmission is particularly important. To solve the problem of multi-hop relay selection in wireless ad hoc networks, we propose an optimal transmission model based on a discrete Markov process. The aim is to design a relay node selection decision that satisfies low latency and high throughput, so that information can be transmitted quickly and accurately between trains to each other. Since the next-hop selection depends only on the current state and the next state changes with the current action selection, next hop selection can be considered as an MDP. The transfer probability between states in the MDP is unknown; thus, we can use the DRL approach to better solve our proposed problem. In DRL, the agent finds the optimal policy that maximizes the long-term reward value according to the channel state information (CSI) and the congestion level of the node. In this paper, we use multiagent DRL (MADRL), in which each node acts as an agent. As agents need to make decisions in a shorter period of time, the agent's network is trained offline centrally by collecting information between nodes, and then porting it to each agent for online decision making. Therefore, each agent needs to select the next-hop node independently according to the current state, without additional communication for further training. In DRL, there are several key components, as described below. 1. State Space In each time slot, the agent updates and learns the policy by observing the variation of the state. In particular, the state contains two components: the number of packages queued in each node and the channel throughput of the links between the two nodes. In time slot \(t\), the state space is defined as \[s_{t}=\{Q(t),\;C(t),\;V(t)\},\] (23) where \(Q(t)\) indicates the queue length of each node. \(C(t)\) is the throughput between the transmission node and other nodes. \(V(t)=\{0,1\}\); when \(V(t)=0\), at time slot \(t\), the wayside node sends packets, whereas \(V(t)=1\) means that the train sends the packet. 2. Action Space According to the channel state and queue state of each candidate node, the optimal next-hop node \(A(t)\) is selected. The action space can be given by \[a_{t}=\{A(t)\},\] (24) where \(A(t)=\{0,1\;\dots\;m\;\dots\;M\}\). If \(A(t)=m\), then the next hop is node \(m\). 3. Reward Function In the selection of the next-hop node, the optimization objective is to minimize the delay and maximum the throughput of the entire T2T link while ensuring that the next-hop SNR is greater than threshold value. The packet is successfully transmitted from the initial train to the destination train after \(k\in\{1,2,\;\dots\;,\;\text{K}\}\) hops through wayside nodes \(i,j\in\{R_{1},R_{2},\dots,R_{m},\dots R_{M}\}\). Furthermore, \(T_{n}\) is the packet source train, and \(T^{\prime}{}_{n}\) is the packet destination train. 4. Overall Transmission Time of Whole Link The total transmission time for packet transmission between source train \(T_{n}\) and destination train \(T^{\prime}{}_{n}\) is \[\tau_{T_{n},T^{\prime}{}_{n}=D_{T_{n},i}^{1}}+\sum_{k=2}^{K-1}D_{i,j}^{k}+D_{f ^{\prime},T^{\prime}{}_{n}}^{K}\;.\] (25)
5. Throughput of Whole Link In order to better measure the quality of each hop in the transmission link and considering the packet loss and packet error rate, the throughput of the entire link is defined as follows [35; 36]: \[C_{T_{n},T^{\prime}{}_{n}}=(1-p^{\prime}{}_{f})\;\cdot\;\min\Bigl{\{}E[C_{T_{n },i}],E[C_{i,j}]\dots E\Bigl{[}C_{f^{\prime},T^{\prime}{}_{n}}\Bigr{]}\Bigr{\}},\] (26) where \(p^{\prime}{}_{f}\) is the unsuccessful transmission rate of whole link; \(p_{f}\) for the one-hop unsuccessful rate is calculated using Equation (16). \(C_{i,j}\) denotes the throughput between waysides node \(i\) and \(j\). 6.