# A Hybrid Route Selection Scheme for 5G Network Scenarios: An Experimental Approach
## 4 CenTri: Reinforcement Learning Model and Algorithm
### Reinforcement Learning Models
#### 4.1.2 Distributed Reinforcement Learning Model  
In the data plane, the distributed RL model is embedded in all small-cell BSs and their corresponding nodes in the network. Hence, a source node in the network can select a route towards the destination in a reactive manner. Route selection in the data plane of the small cells follows the priority levels (refer to Section 1.3). The selected route has lower intermediate nodes and traffic intensity for maximizing traffic offload and achieving a higher throughput through increasing packet delivery rate. The channel capacity of a link in a route is determined by \(\Phi_{t_{n}}^{c_{n}k_{n}}\) from Equation (2) and it shows the utilization of links \(l_{n}\), including PU activities, in a D2D route \(k_{n}\). Assume that a packet \(p_{i}^{c^{i}}\in PT\) with size \(c_{i}\) traverses along a link \(l_{n}\) with channel \(C_{n}\) at time instant \(t_{n}\), the utilization \(U_{t_{n}}^{l_{n}k_{n}}\) of links \(l_{n}\) of route \(k_{n}\) is defined as follows [57]:  
\[U_{t_{n}}^{c_{n}l_{n}k_{n}}=\frac{\Sigma p_{i}^{c^{i}}c_{n}l_{n}}{BW_{c_{n}} \in C} \tag{6}\]  
where \(BW_{c_{n}}\) is the available bandwidth of the channel \(c_{n}\) for the link \(l_{n}\) in the femtocell layer. It is noteworthy that all D2D routes in this study have the same bandwidth but different frequencies, which means the link between two nodes uses different channels. The channel utilization of a route is defined as follows:\[\tau_{t_{n}}^{c_{n}k_{n}}=\Sigma(U_{t_{n}}^{c_{n}k_{n}}) \tag{7}\]  
where \(\Sigma(U_{t_{n}}^{c_{n}k_{n}})\) is the sum of the channel utilization by PUs in the links \(l_{n}\) of route \(k_{n}\) with channels \(c_{n}\) at time instant \(t_{n}\). The channel utilization of a route includes the PU activities, and it is important for source nodes in the network to be aware of the traffic intensity of each route to reduce the possibility of interference from SUs to PUs. Traffic intensity is particularly important when multiple routes from a source node have the same number of hops (or intermediate nodes). In the D2D type of communication, including node-to-node, node-to-BS, and BS-to-BS communications, the traffic intensity of a route can be calculated as long as the backbone routes are not utilized. For instance, in Figure 2, the source node \(fc_{s}\) and the destination node \(fc_{d}\) are three hops away in route \(k_{2}\) (\(fc_{s}-fc_{3}-fc_{6}-fc_{d}\)). However, for routes \(k_{3}\) (\(fc_{s}-fc_{1}-fc_{4}-fc_{7}-fc_{d}\)) and \(k_{4}\) (\(fc_{s}-fc_{2}-fc_{5}-fc_{8}-fc_{d}\)), both routes have the same number of intermediate nodes and links (i.e., four hops). Hence, traffic intensity is used to select a route with a lower congestion level. In this example, PUs appear in route \(k_{3}\), then route \(k_{4}\) is used. In such a case, route \(k_{4}\) continues to be used until communication ends or interference occurs. However, the presence of PUs in a route (i.e., \(k_{3}\)) makes the route unavailable and recorded as occupied. For a source node that is required to send a data stream to a destination node, it has the initial roadmap comprised of available routes toward the destination node. These routes include nodes and BSs from both macrocell and femtocell layers. Based on the communication priority levels, the source node rearranges the given list by the CC and prefers routes using the femtocell layer. The higher the number of intermediate nodes, the greater the number of links in the route and the more possibility of PUs appearing in its channels. Hence, the route with a lower number of intermediate nodes tends to be selected. A route with a lower PU activity level in the channels of a route has a higher availability, and so the priority level of the route increases, which makes the route more likely to be selected by the source node with an increased learning rate \(\alpha(s_{t}^{k_{n}},a_{t}^{k_{n}})\). The given routes with a different number of intermediate nodes, links, and channels experience different PU activity levels.