# Research on High-Efficiency Routing Protocols for HWSNs Based on Deep Reinforcement Learning
## 3 Optimization Problem Analysis
### Algorithm Design  
#### 3.3.1 Overview of Q-Learning  
Reinforcement learning (RL) is one of the mainstream intelligent methods used to address data transmission and routing optimization issues in WSNs. Reinforcement learning involves an Agent interacting with the environment to take actions and obtain the maximum cumulative reward, thereby continuously optimizing its decision-making capability. Reinforcement learning methods are defined by a quadruple \((S,A,P,r)\), where \(S\) represents the current state of the Agent; \(A\) denotes the action taken by the Agent in the current state; \(P\) indicates the probability of transitioning from the current state to other states after executing an action; and \(r\) signifies the reward received by the Agent after performing the corresponding action. Typically, the reinforcement learning process can be described using a Markov decision process (MDP).  
Q-learning is a model-free reinforcement learning algorithm used to solve optimization problems by learning the optimal action selection policy for a given environment. It is a type of temporal difference learning aimed at finding the optimal policy by estimating the value of actions in various states of the environment. The Q-value update algorithm is as follows:  
\[Q(S,A)\gets Q(S,A)+\alpha\Big{[}R+\gamma\underset{a}{\max}Q\big{(}S^{ \prime},a\big{)}-Q(S,A)\Big{]} \tag{13}\]  
In Equation (13), \(\alpha\) represents the learning rate, \(\gamma\) denotes the reward decay factor, and \(R\) is the reward for action \(A\). Equation (13) signifies that the Agent uses the past Q-values from the Q-table as experience to estimate the updated Q-values for each possible action in the next state \(S^{\prime}\). It then selects the maximum Q-value, multiplied by the decay factor \(\gamma\), and adds the actual reward value to compute the current Q-value update. When \(\gamma\) is close to 0, the Agent relies predominantly on existing experience and does not learn new information, which is known as a greedy strategy. Conversely, when \(\gamma\) is close to 1, the Agent relies more on new information rather than prior experience, leading to a more exploratory approach, though this can potentially cause instability in the learning process. When \(\gamma\) is close to 0, only the immediate reward is considered, whereas a value close to 1 emphasizes long-term high returns. If the Q-values reach or exceed 1, there is a risk of divergence. The detailed initialization and update mechanism of the Q-function (Q-Matrix) can be described in detail as follows.  
The Q-function uses the Bellman equation, accepting two inputs, namely the state and the action, described as follows:  
\[Q_{\pi}(s,a)=E_{\pi}\big{[}r^{\prime}+\gamma r^{\prime\prime}+\gamma r^{\prime \prime\prime}+\cdots\big{|}s,a\big{]} \tag{14}\]  
where \(Q_{\pi}(s,a)\) represents the Q-value of the given state and \(E_{\pi}[r^{\prime}+\gamma r^{\prime\prime}+\gamma r^{\prime\prime\prime}+ \cdots\big{|}s,a\big{]}\) represents the expected discounted cumulative reward of the given state and action. The update method for Q-value based on the principle of the Bellman equation is as follows:  
\[Q^{*}(s,a)=(1-\delta)Q(s,a)+\delta r+\gamma\underset{a}{\max}Q(s^{\prime},a) \tag{15}\]  
where \(Q^{*}(s,a)\) represents the Q-value of the next state and \(\delta\) represents the learning rate. \(Q(s,a)\) represents the current Q-value. \(\underset{a}{\max}Q(s^{\prime},a)\) represents the estimation of the future optimal value. By iteratively updating the Q-values, the DQN algorithm approximates the Q-values, using DNNs to learn the optimal action policy, thereby facilitating effective decision-making in reinforcement learning tasks.