## 3 A Resilient Routing Protocol
### Protocol Design
#### 3.1.3 Routing Path Selection with DQN  
In the routing path selection phase, the SDN controller leverages DQN to determine optimal routing paths using the clustered nodes and the weights obtained from AHP. The DQN is trained to make intelligent routing decisions based on the clustered node information and the parameter weights. The state space for the DQN includes features such as residual energy (\(E_{i}\)), queue length, cluster level of the current node, and cluster levels of neighboring nodes. By excluding RSSI from the state space, we avoid redundancy with the clustering parameters and focus on the most critical real-time metrics that influence routing efficiency and reliability.  
To maintain clustering stability and network adaptability, the clustering algorithm is executed periodically at defined intervals, ensuring that cluster formations reflect the current network state without being overly sensitive to transient fluctuations. This periodic update mechanism prevents the clustering from changing too frequently, which could destabilize the DQN process and lead to inconsistent routing decisions.  
The reward function for the DQN is designed to balance multiple objectives, including minimizing energy consumption, reducing end-to-end delay, and maximizing PDR. Specifically, the reward function is formulated as:  
\[R(s,a)=w_{1}R_{\text{energy}}+w_{2}R_{\text{delay}}+w_{3}R_{\text{delivery}}\]  
where \(w_{1}+w_{2}+w_{3}=1\), and each \(R\) component represents energy consumption, transmission delay, and PDR, respectively. By optimizing this reward function, the DQN learnsto maximize overall network performance through efficient routing decisions that favor higher-cluster-level nodes.  
The cluster level information derived from the clustering phase is seamlessly integrated into the DQN's state space. This integration allows the DQN agent to assess the reliability and efficiency of potential next-hop nodes based on their cluster levels. During action selection, the DQN primarily prioritizes actions that lead to higher-cluster-level nodes, ensuring that data packets traverse through more reliable and energy-efficient paths. However, to enhance flexibility and prevent potential bottlenecks, the action space is designed to allow the selection of lower-cluster-level nodes under specific conditions, such as when higher-level clusters are insufficient or when network demands necessitate a more balanced energy consumption across nodes. This flexibility ensures that the routing protocol can adapt to varying network conditions without compromising efficiency or reliability.  
To ensure the stability and convergence of the learning process, the DQN employs a target network that is periodically synchronized with the main network. This synchronization prevents oscillations and divergence in Q-value estimates, thereby maintaining the integrity of the learning process. Additionally, by controlling the clustering update interval, we ensure that the DQN has sufficient time to learn from the current clustering configuration before any significant changes occur, further enhancing the stability of the routing decisions.  
Overall, the integration of DQN with periodic clustering updates and flexible action space design enables the routing protocol to dynamically adapt to the evolving conditions of MANETs. This holistic approach ensures that routing decisions are both reliable and energy-efficient, thereby enhancing the overall performance and resilience of the network.