## 3 Methodology
### Correlated Multi-Objective Energy-Efficient Routing Scheme for SDWSN-IoT
#### 3.2.4 DOS-RL Routing Algorithm  
The proposed energy-efficient routing scheme employs the DOS-RL scheme with shaped rewards to acquire knowledge about the network environment. Algorithm 2 outlines the procedure employed in the proposed approach. The algorithm aims to find optimized routing paths for all source-destination pairs between the current state and the goal state for all valid link states. ```
1:Input:
2:Learning rate: \(\alpha\)
3:Exploration and exploitation parameter: \(\varepsilon\)
4:Number of learning episodes: \(n\)
5:Potential function for each state: \(\phi\)
6:All link pairs \((\text{src},\text{dst})\): VLinksPair
7:Link-states, Set of learning objectives: \(\mathbf{N}\)
8:Initialize \(Q\): \(Q(S,A)=0\), \(\forall s\in S\), \(\forall a\in A\)
9:procedure\(S_{j}(G)\leftarrow\text{STP}(\text{VLinksPair})\triangleright\) Set of loop-free paths from the initial state to the end state \(G\)
10:for each objective \(o\) in \(\mathbf{N}\), compute Q-values for all \((\text{src},\text{dst})\) in \(S_{j}(G)\)do
11:for episode \(\gets 1\) to \(n\)do
12: Start in state \(s_{t}=\text{src}\in S\);
13:while\(s_{t+1}\) is not dst do
14: Select \(A_{t}\) for \(s_{t}\) using the \(\varepsilon\)-greedy method according to Equation (10)
15:\(R_{t}^{{}^{\prime}}\gets R_{t}+F_{t+1}\)\(\triangleright\) Get the shaped reward \(R_{t}^{{}^{\prime}}\) and observe new state \(s_{t+1}\)
16: Update \(Q^{n}(s_{t},a_{t})\)\(\triangleright\) Update according to Equation (8)
17:\(s_{t}\gets s_{t+1}\)\(\triangleright\) Set the current state \(s_{t}\) as the new state \(s_{t+1}\)
18:endwhile
19:endfor
20: Take \(Q\)-table and find the path between src and dst with the maximum Q-values
21:endfor
22: Store the set of computed paths in the route repository
23:endprocedure
```  
**Algorithm 2** The DOS-RL with Shaped Rewards Routing Scheme  
Stages followed by this algorithm can be summarized as follows:  
1. _Initialization and Setup_: The intelligent controller is initialized and assigned an IP address. Partial topologies received from the controllers are used to create a global topology containing the eligible RNs. Set the learning rate \(\alpha\), discount factor \(\gamma\), and exploration rate \(\varepsilon\), and define the potential function \(\phi\) to provide hints on states for shaping rewards (Lines 2 to 7). 2. _Loop-Free Route Calculation_: Loop-free routes are computed using the Spanning Tree Protocol (STP) algorithm for all given source-sink pairs. This step ensures that the Q-learning algorithm focuses only on exploring states that lead to the end goal. Here, a source node is defined as any RN that receives data from the SN(s) at the sensing layer (Line 8). 3. _Q-Routing Exploration_: The set of computed loop-free routes serves as input for the SDN intelligent controller, which uses the Q-routing algorithm to find the best path for routing. The Q-learning process runs for a specified number of episodes or until the final state is reached. For each state \(S_{t}\), compute Q-values for every action \(a_{t}\) for each objective \(o\) simultaneously, based on the current Q-table values (Lines 9 to 13). 4. _Q-Learning Process_: During the learning process, the RL agent dynamically chooses the learning objective from the current state, takes the next action, and moves to the next state. Regarding the randomly generated number \(x\) ranging between 0 and 1, according to the exploration and exploitation \(\varepsilon\)-greedy method, if \(x<\varepsilon\), then exploit with the probability \(pr=\varepsilon\) by selecting the action with the highest Q-value for the most confident objective \(o_{best}\). Otherwise, explore with the probability \(pr=1-\varepsilon\) by randomly selecting an action for the most confident objective \(o_{best}\) (Lines 14 to 19). 5. _Optimal Route Determination_: The RL agent uses the generated Q-table to determine the most rewarding route between the source and sink nodes. This is based on the state-action combinations that received the highest Q-values after completing a transition.