## 5 Routing Algorithm
### Our Proposed Routing Algorithm  
Different routing algorithms have been proposed in the literature to achieve Quality of Services (QoS), energy consumption and link heterogeneity in WSN. Some of the famous algorithms are E-TORA (energy-aware TORA), EBCRP (energy balanced chain-cluster routing protocol), HGMR (hierarchical geographic multicast routing), EADAT (energy-aware data aggregation protocol) and many more, as mentioned above in Table 1. Due to different attributes, such as dynamic topology, distributed nature, high density and resource constraints of the 5G-based smart healthcare network, the above-mentioned cluster-based routing protocols are not suitable for this network. The earlier algorithms do not consider the delay in transmission, energy consumption, and unbalanced energy dissipation. To address these issues, we use a Q-learning-based algorithm. The network is divided into three grids (i.e., pico, femto and macro). In the learning phase, the grids periodically exchange information (i.e., the distance between nodes and the nodes' remaining energy). The value of each node state and topological relation of nodes with each other are stored in a Q-table. The agent makes a decision based on the Q-table for the next-hop selection. In this paper, we classified them based on Energy hop (\(E_{hop}\)) and Distance hop (\(D_{hop}\)), which improve the Quality of Services (QoS) and reduce energy consumption by maintaining energy balance between nodes. Furthermore, the values can vary considerably between the various range. Therefore, it is critical to scale them in a specific range. For the \(Y\) feature, we can scale it between range [0,1] as follows:  
\[f(Y)=\frac{Y-min(Y)}{max(Y)-min(Y)}. \tag{21}\]  
In Q-learning based routing protocol, \(Q_{t}\) and \(R_{t+1}\) at the hop can be calculated with Equations (23) and (22) which are associated with \(E_{hop}\) and \(D_{hop}\). At this point, when the agent changes the state from \(s_{t}\) to \(s_{t+1}\), which means data are transmitted from \(N_{t}\) to \(N_{t+1}\). \[Q_{t}(s_{t},a_{t})=f(E_{hop})\cdot(1-\beta) \tag{22}\]  
\[R_{t+1}=f(D_{hop})\cdot\beta. \tag{23}\]  
Additionally, \(\beta(0\leq\beta\leq 1)\) is an administrative factor that imitates the significance of delay and energy consumption. When the \(\beta\) factor is set to value 1, the algorithm emphasizes decreasing transmission delay. While, when the \(\beta\) value is set to 0, the algorithm tries to balance the remaining energy. Otherwise, both transmission delay and remaining energy are considered instantaneously. In this way, we can alter the value of \(\beta\) according to QoS requirements. By combining Equations (18), (22) and (23), the algorithm for routing is updated as follows:  
\[Q_{t+1}(s_{t},v_{t})=f(E_{hop})\cdot(1-\beta)\cdot(1-\alpha)+\alpha\cdot[ \gamma\cdot maxQ_{t}(s_{t+1},v_{t})+\beta\cdot f(D_{hop})]. \tag{24}\]The sink is the end node of the network, which is the information collection unit within the network. Every sink has the next hopping node, known as BS. Thus, \(maxQ_{t}(s_{t+1},v_{t})=0\). In this situation, the formula is updated as follows:  
\[Q_{t+1}(s_{t},v_{t})=(1-\beta)\cdot(1-\alpha)\cdot f(E_{hop})+\alpha\cdot f(D_{ hop})\cdot\beta \tag{25}\]  
Algorithm 2 presents a detailed explanation of the proposed routing algorithm. Each node keeps information of the network and a Q-value. Before information gathering, the base station allows tasks with broadcasting. All nodes in the network update their Q-value according to Equation (24). Based on the updated table of Q-values, the path with the highest reward in Algorithm 3 is selected. The node determines the next available hopping node that has the highest Q-value from the available subsequent hopping nodes if there are subsequent hop nodes available for a node, implying that the node is in the vicinity of the base station. Otherwise, the node will select a node with the highest energy as the next hopping node from all nodes available in the reachable range. Figure 6 shows a Q-value fluctuation with several rounds.