# A Failure Risk-Aware Multi-Hop Routing Protocol in LPWANs Using Deep Q-Network
## 3 Brief Review of DQN  
An RL framework is typically modeled as a Markov Decision Process (MDP), characterized by a tuple \(\langle S,A,P,R\rangle\), where \(S\) represents the state space, \(A\) denotes the action space, \(P\) is the state-transition probability, and \(R\) signifies the rewards. At each time step \(t\), the agent executes the action \(a_{t}\in A\) determined by the policy \(\pi\) based on the current state \(s_{t}\in S\). Subsequently, the environment provides the agent with an immediate reward \(r_{t}\in R\) contingent upon \(a_{t}\) and transitions to the next state \(s_{t+1}\). This process generates an experience \((s_{t},a_{t},r_{t},s_{t+1})\). The overarching goal of the agent is to derive an optimal policy \(\pi^{*}\) that maximizes the expected cumulative reward, thereby optimizing long-term performance within the given MDP framework [15].  
Q-learning is a value-based RL algorithm that iteratively refines policies to approximate \(\pi^{*}\). The action-value function \(Q(s,a)\) estimates the expected return of taking action \(a\) in state \(s\), which is updated iteratively using the following formula:  
\[Q(s,a)\gets Q(s,a)+\alpha\bigg{\{}r+\gamma\underset{a^{\prime}\in A}{ \text{max}}Q\big{(}s^{\prime},a^{\prime}\big{)}-Q(s,a)\bigg{\}}, \tag{1}\]  
where \(\underset{a^{\prime}\in A}{\text{max}}Q(s^{\prime},a^{\prime})\) is the maximum Q-value over all possible actions \(a^{\prime}\) in the subsequent state \(s^{\prime}\), \(\gamma\) is the discount factor, and \(\alpha\) is the learning rate. \(\pi^{*}\) directs the agent toward actions that yield the highest \(Q(s,a)\) in each state.  
When the state space is large, exhaustively computing \(Q(s,a)\) becomes infeasible. Consequently, DQN is adopted to approximate \(Q(s,a)\), where the output is \(Q(s,a;\omega)\ \approx\ Q(s,a)\). Here, \(\omega\) represents the weights of the DNN, and the stochastic gradient descent (SGD) algorithm is used to update parameters.  
However, the neural network can become unstable owing to correlations between Q-value and target value, or small updates to Q-value at each step. To address this instability, experience replay and a quasi-static target network are employed in DQN [18]. In experience replay, at each time step \(t\), an experience sample \(e_{t}=(s_{t},a_{t},r_{t},s_{t+1})\) is stored in a replay memory \(\boldsymbol{M}=\{e_{1},e_{2},\cdots,e_{t}\}\). During training, the agent randomly samples a minibatch of experiences from \(\boldsymbol{M}\), thus removing the correlations between continuous samples and improving the stability and efficiency of learning. Additionally, an independent target neural network with weights \(\omega^{-}\) is used for the quasi-static target network. The loss function \(L(\omega)\) is calculated as follows:  
\[L(\omega)=E_{(s,a,r,s^{\prime})\in M}\Big{[}\big{(}y\big{(}s^{\prime},r\big{)}-Q (s,a;\omega)\big{)}^{2}\Big{]}, \tag{2}\]  
where \(y(s^{\prime},r)\) is the output of the target neural network:  
\[y\big{(}s^{\prime},r\big{)}=r+\gamma\underset{a^{\prime}\in A}{\max}Q\big{(}s ^{\prime},a^{\prime};\omega^{-}\big{)}, \tag{3}\]  
where \(\omega^{-}\) is synchronized with \(\omega\) every \(C\) steps. This approach decouples the target value computation from the Q-network weights, thereby reducing the likelihood of divergence and ensuring more stable learning.