## 3 Multi-Hop State-Aware Routing Strategy Based on Traffic Flow Forecasting
### Traffic Flow Prediction Strategy Based on RNN Combined with the DDPG Method  
According to the research results in [20,21], it can be assumed that the size of traffic between two nodes is related to the geographical location between the two nodes. In this section, the traffic model between two nodes is proposed. It is assumed that each node can obtain its own average traffic after running for a period of time, which is defined as \(j_{i}\) and belongs to node \(v_{i}\). Therefore, the traffic between node \(v_{i}\) and \(v_{j}\) can be calculated as (1). \[j_{ij}=j_{i}\times\frac{j_{j}/l_{ij}}{\sum_{v_{i}\in V_{k}\
eq i}j_{k}/l_{ik}} \tag{1}\]  
where \(l_{ij}\) is the spatial distance between node \(v_{i}\) and \(v_{j}\). In addition, the network traffic of each node is composed of two components: the periodical flow and the random flow [22]. Therefore, traffic merits using neural networks in order to discover the periodic feature. After the network has been running for some time, the enhancement node can obtain the topology of the network. Then, each enhanced node takes the ordinary nodes closest to them as a node-set and collects the position information and traffic information of the ordinary node. After the collection of enhanced nodes is completed, the spatiotemporal distribution of traffic can be calculated according to Equation (1). In this section, the traffic flow prediction strategy based on RNN combined with the DDPG method is shown (Figure 2). \begin{table}
\begin{tabular}{c c} \hline \hline
**Symbol** & **Definition** \\ \hline \(G\) & The direct graph of the WSN \\ \(V\) & The set of nodes \\ \(E\) & The set of links \\ \(V^{E}\) & The set of enhanced nodes \\ \(V^{O}\) & The set of ordinary nodes \\ \(E^{EE}\) & The communication link between two enhanced nodes \\ \(E^{EO}\) & The communication link between enhanced node and ordinary node \\ \(E^{OO}\) & The communication link between two ordinary nodes \\ \(\Gamma_{i}(v)\) & The set of \(i\)-hop neighbors of node \(v\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Definition of the symbols. Figure 2: The traffic flow prediction strategy based on RNN combined with the DDPG method. The interaction process between the environment and the agent is considered as a Markov Decision Process (MDP) by the reinforcement model. The element in MDP is defined as \(M=(S,A,R,P,\lambda)\), where the space of state, action, reward is \(S\), \(A\), \(R\), respectively. The transition probability function is \(P\), and the discount factor is \(\lambda\) (\(\lambda\in[0,1]\)). According to the policy, the agent will decide an action under a different state [22]. The Q value is used in this model to evaluate the performance of the policy. Therefore, the Q value under \((s_{t},a_{t})\) is  
\[Q(s_{t},a_{t})=E\Big{[}\Sigma_{n=0}^{\infty}\lambda^{k}R(s_{t+n},a_{t+n})\Big{]} \tag{2}\]  
In online network usage, the Q value is more commonly expressed as  
\[Q(s_{t},a_{t})=E[R_{t}+\lambda Q(s_{t+1},a_{t+1})] \tag{3}\]  
In the proposed model, the loss function in DDPG is used to process the back propagation, which can be expressed as  
\[L(\varphi)=E\big{[}\left(Q(s_{t},a_{t}|\varphi)-\psi_{t}\right)^{2}\big{]} \tag{4}\]  
where the \(Q(s_{t},a_{t}|\varphi)\) represents the Q value expressed by a set of parameters \(\varphi\) in the neural network. \(\psi_{t}\) is  
\[\psi_{t}=R(s_{t},a_{t})+\lambda Q(s_{t+1},\mu(s_{t+1})|\varphi) \tag{5}\]  
The policy gradient of DDPG can be computed as  
\[\
abla_{\varphi Q}\int\approx E\big{[}\
abla_{\varphi}Q(s,\mu(s\Big{|}\varphi ^{\text{\tiny H}})\Big{|}\varphi^{\text{\tiny Q}})\big{]}=E[\
abla_{\varphi} Q(s,a\Big{|}\varphi^{\text{\tiny Q}})\
abla_{\varphi^{\text{\tiny H}}} \mu(s\Big{|}\varphi^{\text{\tiny H}})] \tag{6}\]  
Algorithm 1 shows the detailed DDPG method used in this section.