## 3 A Resilient Routing Protocol
### Routing Algorithm
#### 3.2.1 Algorithm Explanation  
The Algorithm 1 begins with an initialization phase (Lines 3-6), where the critical components for the DQN process are set up. This includes initializing the DQN parameters \(\theta\) randomly, creating a target network \(\theta^{-}\) that is synchronized with \(\theta\) for training stability, setting up an experience replay memory \(D\) to store past routing experiences, and initializing a clustering timer to manage periodic clustering updates. Following this, the routing request handling phase (Lines 7-12) collects network parameters from all nodes and normalizes them. The algorithm checks if the clustering update interval \(T\) has been reached. If so, it re-executes the K-means clustering algorithm to update the cluster formations and recalculates the AHP-based weights. This ensures that clustering results remain relevant to the current network state without being updated too frequently, thereby maintaining stability. In the action selection and execution phase (Lines 13-24), the algorithm enters a loop that continues until the destination is reached. An \(\epsilon\)-greedy policy is used to select actions: with probability \(\epsilon\), the algorithm explores new paths by choosing a random action within the higher-level clusters; otherwise, it selects the action with the highest Q-value. The selected action forwards the packet to the next node, after which the system receives an acknowledgment and observes the reward \(R\) and the new state \(s^{\prime}\). These experiences are stored in the replay memory for future training. ```
1:Input: Set of nodes \(\mathcal{N}\), mobility-resilient parameters, learning rate \(\alpha\), discount factor \(\gamma\), exploration rate \(\epsilon\), number of clusters \(K\), clustering update interval \(T\)
2:Output: Optimal routing paths
3:Initialize:
4: Initialize DQN parameters \(\theta\) randomly
5: Initialize target network parameters \(\theta^{-}\leftarrow\theta\)
6: Initialize experience replay memory \(D\)
7: Initialize clustering timer \(t\gets 0\)
8:for each routing request do
9: parameters \(\leftarrow\) collect_parameters(\(\mathcal{N}\))
10: normalized_parameters \(\leftarrow\) normalize(parameters)
11:if\(t\geq T\)then
12: clusters \(\leftarrow\) K_means(normalized_parameters, \(K\))
13: weights \(\leftarrow\) AHP(parameters)
14:\(t\gets 0\)
15:endif
16:\(s\leftarrow\) get_current_state(source node, clusters)
17:while destination not reached do
18:if\(\text{random}()<\epsilon\)then
19:\(a\leftarrow\) select_random_action(clusters)
20:else
21:\(a\leftarrow\arg\max_{a^{\prime}}Q(s,a^{\prime};\theta)\)
22:endif
23: forward_packet(\(a\))
24:\((R,s^{\prime})\leftarrow\) receive_acknowledgment()
25: store_experience(\(D,s,a,R,s^{\prime}\))
26: sample_mini_batch \(\leftarrow\) sample_mini_batch(\(D\))
27:for each experience \((s_{i},a_{j},R_{j},s^{\prime}_{j})\) in mini_batch do
28:\(y_{j}\gets R_{j}+\gamma\max_{a^{\prime}}Q(s^{\prime}_{j},a^{\prime};\theta^ {-})\)
29:endfor
30: Update \(\theta\) by minimizing the loss \(L(\theta)\)
31:\(s\gets s^{\prime}\)
32:endwhile
33: Update clustering timer \(t\gets t+1\)
34: Update target network parameters \(\theta^{-}\leftarrow\theta\)
35:endfor
```  
**Algorithm 1** Resilient Routing Protocol  
The learning phase (Lines 25-30) trains the DQN using experiences stored in the memory. A mini-batch of experiences is sampled, and for each experience, the target value is calculated using the Bellman equation:  
\[y_{j}=R_{j}+\gamma\max_{a^{\prime}}Q(s^{\prime}_{j},a^{\prime};\theta^{-}).\]  
This target value guides the update of the DQN parameters \(\theta\) by minimizing the loss function \(L(\theta)\), thereby improving the routing policy over time. Finally, in the synchronization phase (Lines 31-32), the clustering timer is incremented, and the target network parameters \(\theta^{-}\) are updated to match the DQN parameters \(\theta\). This step ensures the stability of the learning process, preventing oscillations and divergence in the estimated Q-values.