## 4 System Model
### Node Selection in the Optimal Route Based on \(Q\)-Learning Algorithms:  
Q-Learning is a prevalent paradigm of reinforcement learning algorithms applied in wireless multipath networks for decision making policy to improve network performance. Some salient attributes of Q-Learning that make it the most widely adopted algorithm are its simplicity and model-free nature. Moreover, Q-Learning generates an optimal reward value within environments' state-action pairs that are described by [PERSON] decision processes [60]. Q-learning observes agent behavior through trial-and-error interactions with a dynamic environment through a tuple (S, A, R) such as states, actions, and rewards as shown in Figure 2. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Methods** & **(Source, Destination)** & **Hop Count** & **Nodes in Route** & **Priority** \\ \hline ETX & _(s, d)_ & 3 & \(b,e,f\) & \(e>b>f\) \\ \hline EAX & _(s, d)_ & 2 & \(b,e\) & \(b>e\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Selection of potential intermediate nodes in optimal route based on the Expected Number of Transmission (ETX) and Expected Any-path Count (EAX) metrics. Figure 1: The general scenario for potential node selection. Figure 2: Q-learning approach for route selection. The objective of the agent is to maximize its long term rewards of the selection procedure which are obtained from immediate and discounted rewards for the estimation of the Q-value \(Q_{t}^{i}(s_{t}^{i},a_{t}^{i})\), (s \(\in\) S, a \(\in\) A) by taking a given action \(a_{t}^{i}\) at the given state \(s_{t}^{i}\). The Q-Learning algorithm learns in a nondynamic environment and adopts the best policy to solve routing problems in a distributed manner by utilizing the Q-value equation as shown below. \[Q_{t+1}^{i}(s_{t}^{i},a_{t}^{i})\leftarrow(1-\alpha)\times Q_{t}^{i}(s_{t}^{i },a_{t}^{i})+\alpha\Big{[}r^{i}(s_{t+1}^{i})+\gamma\times\max_{a\in A}Q_{t}^{i} (s_{t+1}^{i},a_{t+1}^{i})\Big{]} \tag{12}\]  
where \(\alpha\) is defined as the learning rate and ranges between \(0\leq\alpha\leq 1\), it depends upon variation of Q-value that changes with dynamic topology of the network. If Q-value changes very fast, learning rate value goes to 1 and the agent takes only new reward value. The parameter \(\gamma\in[0,1]\) is a discount factor, which determines the importance of future rewards. Low value, i.e., \(\gamma=0\), indicates the system is myopic and merely takes results of the current action into account. By contrast, as \(\gamma\) close to 1, future rewards play an important role in determining optimal actions. In addition, \(\max_{a\in A}Q_{t}^{i}(s_{t+1}^{i},a_{t+1}^{i})\) is the model of maximum expected future reward, which selects possible action \(a^{i}\) in the next state \(s_{t+1}^{i}\). Based on different decision factors, i.e., link quality, mobility, and energy consumption of nodes, as described above, a new reward function for node \(i\) updated at each instant \(t+1\) is formulated and is as follows:  
\[R_{t+1}^{i}=w_{1}\times EXT(s,d)+w_{2}\times\text{Mobility}+w_{3}\times\text {Residual Energy} \tag{13}\]  
where \(w_{1},w_{2},\text{and},w_{3}\) are the corresponding weight assigned to each criterion based on the available status of nodes (i.e., mobility, link quality, and residual energy) which range from 0 to 1 and their sum equals to 1. The above criteria's value and weights facilities the sensitivity of decision factors for future reward value such as \(R_{t+1}^{i}\). Therefore, Q-value of node \(i\) is updated with a previously stored value and the new reward \(R_{t+1}^{i}\) which is calculated using the following equation as below:  
\[Q_{t+1}^{i}(s_{t}^{i},a_{t}^{i})\leftarrow(1-\alpha)\times Q_{t}^{i}(s_{t}^{i },a_{t}^{i})+\alpha\Bigg{[}R_{t+1}^{i}(s_{t+1}^{i})+\gamma\times\max_{a_{t+1}^ {i}\in A}Q_{t}^{i}(s_{t+1}^{i},a_{t+1}^{i})\Bigg{]} \tag{14}\]  
Lastly, based on the expression, next-hop node which has higher Q-value is selected for the optimal route.