## 3 Methodology
### Correlated Multi-Objective Energy-Efficient Routing Scheme for SDWSN-IoT
#### 3.2.1 Dynamic Objective Selection RL with Shaped Rewards  
Dynamic Objective Selection (DOS)-RL is an approach that combines the principles of Q-learning, a model-free reinforcement learning technique, with the concept of dynamic objective selection. In DOS-RL, the traditional Q-learning algorithm is extended to dynamically select objectives during the learning process based on the current state of the system. The DOS-RL learning agent dynamically selects an objective to optimize from a predefined set of objectives based on its confidence in the estimated Q-values. This means that only the estimated Q-values of the most confident objective are considered when taking an action, allowing the agent to adaptively choose the objective it believes is the most promising at that given time. The Q-learning technique operates by iteratively estimating the action-value (Q-value). Q-learning is easy to implement and applicable to solving many problems since the estimation of the Q-value is independent of the policy that the learning agents follow. In Q-learning, the agent learns sequentially through a series of stages known as epochs. For example, in epoch \(n\), the agent is in state \(s_{n}\), performs action \(a_{n}\), and receives an immediate reward \(r_{n}\) as it transitions to the next state \(s_{n+1}\). The action-value is updated at discrete time \(t\) according to the equation below:  
\[Q(s_{t},a_{t})\leftarrow(1-\alpha)\cdot Q(s_{t},a_{t})+\alpha\cdot[R_{t+1}+ \gamma\cdot\max_{\forall a\in A}\{Q(s_{t+1},a)\}], \tag{1}\]  
whereby \(\alpha\) is the learning rate to determine how fast the agent learns. The discount factor \(\gamma\) determines how much the learning agents take into consideration rewards in the distant future, relative to those in the immediate future, and \(R\) is the reward received for performing an action \(a\) at time \(t\) while in the state \(s_{t}\) resulting in the new state \(s_{t+1}\). As previously mentioned, the Dynamic Objective Selection (DOS) scheme operates by the parallel estimation of the action-value for each objective \(o\), based on the measured current state \(s\) and a potential future state \(s^{\prime}\). The action selection decision in DOS Q-learning is based on the objective that the learning agent determines to have the highest level of certainty in its estimated Q-values. This dynamic approach highlights the adaptive nature of objective selection, as the agent prioritizes objectives based on its confidence in the estimated Q-values. The foundation of this idea can be traced back to the work by [PERSON] et al. [38], where they introduced the Adaptive Objective Selection (AOS) method for handling correlated objectives in multi-objective RL for pathfinding problems. To quantify the confidence in Q-values, they assumed a normal distribution and utilize the mean of the estimated Q-values to track the variance of the distribution using the TD-error \(\delta_{o}\) for objective \(o\) as follows:  
\[VAR(s,a,o)\leftarrow(1-\psi)\cdot VAR(s,a,o)+\psi\cdot\delta_{o}^{2}. \tag{2}\]  
Algorithm 1 presents the pseudocode for the DOS approach. When the agent intends to select an action, it first determines which objective it has the most confidence in, based on a confidence metric \(C_{k}\). It then makes an action selection decision, using the \(e\)-greedy strategy, based solely on the estimates for that particular objective. The variance reflects the dispersion of the action-value distribution for each objective. The confident estimation of an objective will produce uniform values, which leads to a small variance \(\sigma\). In contrast, a larger variance indicates that the policy is more unstable for objective \(k\).