# A Hybrid Route Selection Scheme for 5G Network Scenarios: An Experimental Approach
## 4 CenTri: Reinforcement Learning Model and Algorithm
### Reinforcement Learning Models
#### 4.1.2 Distributed Reinforcement Learning Model
The amount of time that a PU appears in a channel determines the channel capacity at the bottleneck link of a route as follows:  
\[\Phi_{t_{n},\beta,OFF}^{c_{n}k_{n}}=\max_{\forall k_{n}\in K}\biggl{(}\min_{l_ {n}\in L}\Bigl{(}\Phi_{t_{n},OFF}^{c_{n}l_{n}k_{n}}\Bigr{)}\biggr{)} \tag{8}\]  
where \(\beta\) signifies the bottleneck link of a route. The traffic intensity \(\Psi_{t_{n}}^{k_{n}}\) of a route \(k_{n}\) in the femtocell layer at time instant \(t_{n}\) is the cumulative of the bottleneck channel capacity in the absence of PUs, as follows:  
\[\Psi_{t_{n}}^{k_{n}}\leftarrow\Sigma(\Phi_{t_{n},\beta,OFF}^{c_{n}k_{n}}) \tag{9}\]  
The traffic intensity of a route is proportional to its channel capacity. A greater channel capacity in a route has lesser PU interference and vice-versa. The source node learns about the dynamic changes of traffic intensity and adapts the learning rate accordingly. For instance, the greater traffic intensity in route \(k_{1}\) at the time instant \(t_{1}\) causes the source node to reduce the learning rate \(\alpha(s_{t_{1}}^{k_{1}},a_{t_{1}}^{k_{1}})\to 0\) closer to zero, which makes the Q-function more dependent on the previous Q-value. The lower traffic intensity causes the source node to increase the learning rate \(\alpha(s_{t_{1}}^{k_{1}},a_{t_{1}}^{k_{1}})\to 1\) closer to one, which makes the Q-function more dependent on the current Q-value. The source node makes a route selection decision by choosing a route with the maximum Q-value, which has the minimum PU activities as follows:  
\[a_{t}^{k_{n}}=\max_{k_{n}\in A}\Bigl{(}\mathcal{Q}_{t_{n}}^{k_{n}}\Bigl{(}s_{t _{n}}^{k_{n}},f_{t}^{fc}\Bigr{)}\Bigr{)} \tag{10}\]  
The dynamic learning rate \(\alpha_{\text{min}}\leqslant\alpha(s_{t_{n}}^{k_{n}},a_{t_{n}}^{k_{n}}) \leqslant\alpha_{\text{max}}\) is as follows:  
\[\alpha(s_{t_{n}}^{k_{n}},a_{t_{n}}^{k_{n}})\leftarrow\Psi_{t_{n}}^{k_{n}} \tag{11}\]A source node is equipped with the Q-routing model to rank the D2D routes. The best route toward the destination node is based on the number of intermediate hops and traffic intensity of available routes. Table 1 shows the RL model of the reactive mechanism embedded in BSs and nodes. In this model, state \(s_{t}^{k_{n}}\) represents the given routes \(k_{n}\in K\) from the CC towards the destination node \(fc_{d}\), in which both the source and destination nodes are in the data plane of the femtocell. Action \(a_{t}^{k_{n}}\) represents the selection of an available route with the highest priority. If one of the selected routes is blocked by PUs, then the next highest priority route is selected by the source node. Reward \(r_{t}^{k_{n},t+1}\) represents the cost reflecting the traffic intensity of a route from the source node to the destination node. Based on Table 1, three criteria are checked between source and destination nodes prior to communication in the data plane: (a) the type of communication (i.e., D2D); (b) the number of intermediate nodes; and (c) traffic intensity. The first criterion uses the available white spaces in channels and intends to offload traffic from MC BS. The second criterion helps to make route decisions more efficient by looking at routes with a lesser number of intermediate nodes, which helps to reduce the possibility of the PU appearance, leading to a higher successful transmission rate. The third criterion selects the route with fewer PU activities when two routes are identical in terms of the number of hops and channels. The route with fewer PU activities has a lower traffic intensity and a higher Q-value, so it is preferred. In Figure 4, an example of the random appearance of PUs in routes is illustrated for three transmission cycles, namely A, B, C. The PUs can occupy a channel of routes \(k_{2}\), \(k_{3}\), and/ or \(k_{4}\). The presence of PUs in one of the link channels of a D2D route can cause the entire route to be blocked, and the source node must select another route. In this figure, at time \(t\), the source node selects a route to the destination node.