# Multi-Decision Dynamic Intelligent Routing Protocol for Delay-Tolerant Networks
## 2 Routing Decision in Double Q-Learning Environment
### Q-Value Update  
In DTNs, the dynamic movement of nodes allows for opportunistic connections between nodes. Messages are transmitted hop by hop towards the destination node, starting from the source node, following routing protocols, and leveraging opportunistic contacts between nodes. During this process, the Q-value update formula is used to establish and update the corresponding state-action values. In this reinforcement learning environment, the learning task is assigned to each node, and as a result, the learning process involves updating the Q-tables; \(Q^{A}\) and \(Q^{B}\) learn from each other, and they represent future reward values for the same action and state with the same update formula but different values. Their update formulas are shown in (2) and (3). \[Q^{A}_{a}(b,m^{i}_{d})=(1-\alpha)Q^{A}_{a}(b,m^{i}_{d})+ \tag{2}\] \[\alpha(R_{a}(b,m^{i}_{d})+\gamma_{a}(m^{i}_{d})\max(Q^{B}_{a}(x,m^ {i}_{d})))\]  
\[Q^{B}_{a}(b,m^{i}_{d})=(1-\alpha)Q^{B}_{a}(b,m^{i}_{d})+ \tag{3}\] \[\alpha(R_{a}(b,m^{i}_{d})+\gamma_{a}(m^{i}_{d})\max(Q^{A}_{a}(x,m^ {i}_{d})))\]  
\(Q^{A}_{a}\) and \(Q^{B}_{a}\) represent node \(a\) selecting node \(b\) as the destination node for \(m^{i}_{d}\), it represents the expected reward value for the next-hop node, which is the Q-value of node \(a\) selecting node \(b\) for relaying. \(\alpha\) represents the learning rate of a node in the network, which is used to control the extent to which Q-values change with dynamic network variations. \(R_{a}(b,m^{i}_{d})\) represents the actual reward obtained by node \(b\) after forwarding \(m^{i}_{d}\) to node \(b\). \(\gamma_{a}(m^{i}_{d})\) is the discount factor associated with node \(a\) of forwarding of \(m^{i}_{d}\). By adjusting \(\gamma\), we can control the degree to which the node considers short-term and long-term consequences when selecting the next hop. In extreme cases, when \(\gamma\) = 0, it only considers the current outcome of the action, whether the next hop is the message's destination node. When \(\gamma\) approaches 1, it places more emphasis on previous learning results. \(\max(Q^{A}_{a}(x,m^{i}_{d}))\) and \(\max(Q^{B}_{a}(y,m^{i}_{d}))\) refer to the highest expected reward values when node \(a\) forwards to all possible nodes, with \(x\) and \(y\) representing the corresponding relay nodes at that time. \(x,y\in\)\(N_{a}\) and \(N_{a}\) represent the set of contact nodes for node \(a\), indicating all nodes that node \(a\) encounters during its movement. From the equation, it can be observed that Q-value updates primarily depend on the learning coefficient, the actual reward value, and the discount factor. When \(m^{i}_{d}\) reaches the destination node \(d\), the \(R\) values of all nodes along its forwarding path are updated. The actual reward value \(R\) reflects the advantages and disadvantages of one-time forwarding. End-to-end delay represents the time it takes from when a data packet is sent from the source node until it is received by the destination node. The magnitude of delay directly impacts the availability range of DTN and is an important metric to consider in the design of routing protocols. The average hop count is the total sum of hops experienced by copies of all messages in DTNs, divided by the total number of messages generated in the network. The average hop count, along with end-to-end delay, reflects the overall performance of routing protocols, with fewer hops and a lower delay indicating more efficient routing protocols. Therefore, in this paper, when calculating reward values, consideration is given to both hop count and delay to control energy consumption. The update formula for the reward value \(R\) is defined as follows in (4). \[R_{a}(b,m^{i}_{d})=e^{\left(1+\frac{T_{avg}}{T_{b}}+\frac{3}{H_{avg}T_{b}}\right)} \tag{4}\]  
\(T_{avg}\) represents the average delay in the network at the moment. \(t_{b}\) signifies the time it takes for \(m^{i}_{d}\) to reach the destination node after being forwarded from node \(a\) to node \(b\).