# Multi-Decision Dynamic Intelligent Routing Protocol for Delay-Tolerant Networks
## 2 Routing Decision in Double Q-Learning Environment
### Double Q-Learning Algorithm in DTNs  
Q-Learning is a value-based algorithm in the field of reinforcement learning, and its core involves constructing a table called the Q-table. In this table, each row corresponds to a state, each column corresponds to an action, and the value in each cell represents the maximum expected future reward for taking a specific action in a particular state. By progressively updating the Q-Table, the algorithm can gradually learn the optimal action strategy for each state. In this way, by selecting the action with the maximum value in the corresponding row for each state, the goal of maximizing cumulative rewards can be achieved. The Q-learning model iteratively trains through various components, including agents, states, actions, rewards, episodes, and Q-values. The update formula for the state-action values in the Q-learning algorithm is shown in (1). \(S\) and \(A\) represent a state and an action, respectively. \(Q\) represents the Q-value for taking action \(A\) in state \(S\), \(S^{\prime}\) represents the next state, \(A^{\prime}\) represents the next action, \(R\) represents the actual reward obtained from taking that action, \(\alpha\) is the learning rate, \(\gamma\) is the discount factor, and \(\max(Q(S^{\prime},A^{\prime}))\) represents the Q-value of the action with the highest Q-value among all possible actions in state \(S\). \[Q(S,A)=Q(S,A)+\alpha(R+\gamma\max(Q(S^{\prime},A^{\prime}))-Q(S,A)) \tag{1}\]  
The significance of (1) is to gradually establish the optimal policy by updating the Q-value in the Q-table for the action that yields the highest reward in the current state \(S\). The discount factor \(\gamma\) takes into account future rewards, making the algorithm more focused on \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Notations/Abbreviations** & **Description** & **Notations/Abbreviations** & **Description** \\ \hline \(Q\) & Future rewards & \(S\) & State \\ \(S^{\prime}\) & The next state & \(A\) & Action \\ \(A^{\prime}\) & The next action & \(\alpha\) & Learning rate \\ \(Q(S,A)\) & Future rewards of taking the & DTN & Delay Tolerance Network \\ \(a,b,c,d,e,j,x,y\) & Node & \(T^{*}\) & The total lifetime of the message \\ \(m_{d}^{i}\) & Message \(i\), whose destination node & \(R_{a}(b,m_{d}^{i})\) & Actual reward for forwarding \(m_{d}^{i}\) \\  & is \(d\) & from node \(a\) to node \(b\) \\ \(\gamma_{a}(m_{d}^{i})\) & Discount factor corresponding to & \(T_{avg}\) & The average delay in the network \\  & \(m_{d}^{i}\) in node \(a\) & & at the moment \\  & The time it takes for \(m_{d}^{i}\) to reach & & The number of hops it takes for \(m_{d}^{i}\) \\ \(t_{b}\) & node \(d\) after being forwarded to & \(Hops_{b}\) & to reach node \(d\) after being forwarded to node \(b\) \\  & Number of hops of the message & & The remaining time to live of the \\ \(Hops\) & from the source node to the & \(TTL\) & message \\  & destination node & & Number of contacts \\ \(\gamma^{*}\) & Discount factor constant & \(CT\) & The number of messages \\ \(ET\) & Duration of encounters & \(MS\) & successfully forwarded each other \\ \(rel\) & Relationship & \(FTh\) & Friend threshold \\ \(CTh\) & Colleague threshold & \(NT\) & Encounter interval \\  & Multi-Decision Dynamic & & The number of periods during \\ MDDI & Intelligent & \(N\) & which the nodes have not \\  & & & encountered \\ SDDQ & Single-Decision based on double & DDNR & Dynamic decision based on node \\  & Q-Learning & & relationship \\ \(N_{a}\) & The set of contact nodes for node \(a\) & \(C(a,b)\) & Contact values for nodes \(a\), \(b\) \\ \(T\) & Time period of recoding CT,ET MS & \(T^{\prime}\) & Time period of recoding NT \\ \hline \hline \end{tabular}
\end{table} 
Table 2: List of notations and abbreviations. long-term benefits rather than short-sighted strategies. It also results in smoother updates of Q-values, avoiding abrupt changes. It can be observed that the Q-learning algorithm's update is a typical application of temporal difference methods.