## 3 Methodology
### Correlated Multi-Objective Energy-Efficient Routing Scheme for SDWSN-IoT
#### 3.2.3 The DOS-RL Routing Scheme
The second term subtracts the load of node \(i\) from the load balancing reward value, allowing for a more comprehensive estimation that takes into account both node \(i\) and node \(j\) in the load balancing process. A higher value of \(R\_Q_{i,j}\) indicates a better load balancing situation for the given state-action pair \((s_{i},s_{j})\). 3. Link quality: A wireless link can be measured by retrieving useful information from either the sender or receiver side. To achieve \(o_{3}\), we use simple measurements to estimate the link quality based on the parameter packet reception ratio (PRR), measured as the ratio of the total number of packets successfully received to the total number of packets transmitted through a specific wireless link between two nodes. Unlike other sophisticated techniques, this approach involves a low computation and communication overhead. Instead of using an instant value of the PRR, we calculate an average-over-time using an Exponentially-Weighted Moving Average (EWMA) filter. Suppose node \(i\) is forwarding data packets to node \(j\), the reward \(R\_LQ_{i,j}\) received by node \(j\) based on PRR using EWMA is estimated as follows: \[R\_LQ_{i,j}=PRR\_est_{i,j}(k-1)\cdot\alpha_{EWMA}+PRR\_sample_{i,j}(k)\cdot( 1-\alpha_{EWMA}),\] (6) whereby \(PRR\_est_{i,j}(k-1)\) is the previously estimated average, \(PRR\_sample_{i,j}(k)\) is the most recent measured value of the packet reception ratio calculated, and \(\alpha_{EWMA}\) is the filter parameter. 2. _The State Space (S)_: We define the state space as a graph corresponding to the global topology created by the RNs on the data plane, as seen by the intelligent RL controller. Each state in the state space corresponds to an RN, and a state transition refers to a link connecting two RNs. The intelligent RL controller uses the partial maps created by the _topology discovery module_ to create a global topology. Therefore, the cardinality of the set of states depends on the number of nodes that can actively participate in routing. 3. _The Action Space (A)_: The action space, denoted as A, includes all possible actions that an agent can undertake from a given state of the RL environment. It defines the choices available to the agent at each time step, presenting the range of options to the agent. In our specific problem, the discrete action space comprises a finite number of actions that the RL agent can select when in a particular state \(s_{i}\in S\). The cardinality of \(A\) at state \(i\) is determined by the number of nodes eligible to participate in the routing process from that specific state. 4. _The Optimal Policy (P)_: The policy determines how the learning agent should behave when it is at a given state with the purpose of maximizing the reward value in the learning process. Our proposed scheme estimates the Q-function of every objective \(o\) simultaneously and decides, before every action selection decision, which objective estimate \(o_{best}\) an agent will consider in its decision-making process. We use the concept of _confidence_ on computed Q-values by representing each action as a distribution and using a normal distribution of Q-values and the mean values to keep track of the variance as shown in Equation (2). The agent approximates the optimal Q-function by visiting all pairs of action-states and stores the updated Q-values in the Q-table. In our proposed scheme, the approximated Q-value \(Q(s_{t},a_{t})\) represents the expected cumulative reward when the RL agent is in the state \(s_{t}\) and takes action \(a_{t}\), transitioning to a new state \(s_{t+1}\) while maximizing the cumulative rewards for an objective \(o_{n}\), where \(n\in N\) and \(N\) denote the set of all objectives.