## 5 QDAR Algorithm  
In this section, we describe the proposed QDAR algorithm in details. The important notations are listed in Table 1.  
In order to design a Q-learning-based delay-aware routing protocol to extend the network lifetime of UWSNs, we define an action-utility function whose value is \(Q\). The sink node has a matrix in which \(Q\) values of all the nodes are stored. These \(Q\) values are future rewards of packet forwarding and are used in the routing decisions. In our protocol, node \(i\) and packet forwarding from node \(i\) to node \(j\) are seen as state \(i\) and action \(j\) of Q-learning technique, respectively. \begin{table}
\begin{tabular}{c c} \hline \hline
**Name** & **Description** \\ \hline \(a_{j}\) & Packet forwarding from node \(i\) to node \(j\) \\ \(\beta_{0}\) & Constant cost \\ \(\beta_{1}\), \(\beta_{2}\) & Weight of two costs \\ \(N_{i}\) & The neighbor node set of \(i\) \\ \(\varphi_{ij}\) & Delay sensitivity of communication between node \(i\) and \(j\) \\ \(\varphi^{\prime}_{ij}\) & The modified \(\varphi_{ij}\) \\ \(t_{ij}\) & Delay of communication between node \(i\) and \(j\) \\ \(t^{\prime}_{ij}\) & The time of failed communication between node \(i\) and \(j\) \\ \(ce\Big{(}e^{i}_{res}\Big{)}\) & The energy-related cost of node \(i\) \\ \(ct\Big{(}t_{ij}\Big{)}\) & The delay-related cost of node \(i\) \\ \(\gamma\) & The discount factor of future reward \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Notations. Figure 2: Packet structures of DATA_READY packet in data_ready phase and INTEREST packet in interest phase and their relationship. Firstly, we define a reward function \(R_{s_{ij}}^{a_{j}}\) for action \(j\), which is related to both propagation delay and residual energy. If this transmission is successful, the reward of the action for node \(i\) is:  
\[R_{x_{ij}}^{a_{j}}=-\beta_{0}-\beta_{1}C_{ij}=-\beta_{0}-\beta_{1}\Big{[}ce \Big{(}e_{res}^{i}\Big{)}+{\varphi_{ij}}^{*}\;ct\big{(}t_{ij}\big{)}+ce\Big{(}e _{res}^{j}\Big{)}\Big{]} \tag{9}\]  
where \(a_{j}\) is action \(j\). Because forwarding packet occupies channel bandwidth and disturbs other nodes, a constant cost \(\beta_{0}\) is added into the function. \(\beta_{1}\) is the weight of the sum of delay-related cost and energy-related cost. \(\varphi_{ij}\) is the delay sensitivity. A higher \(\varphi_{ij}\) means the delay is more repellent. If the transmission fails, node \(i\) will resend the packet, which means the node should pay double energy-related cost and more delay-related cost. Thus, the reward function becomes:  
\[R_{x_{ii}}^{a_{j}}=-\beta_{0}-\beta_{2}C_{ii}=-\beta_{0}-\beta_{2}\Big{[}2 \times ce\Big{(}e_{res}^{i}\Big{)}+{\varphi_{ij}}^{*}\;ct\Big{(}t_{ij}^{ \prime}+t_{ij}\Big{)}+ce\Big{(}e_{res}^{j}\Big{)}\Big{]} \tag{10}\]  
where \(t_{ij}^{\prime}\) is the time that node \(i\) spends in the failed transmission. \(\beta_{2}\) is the weight of the sum of delay-related cost and energy-related cost, in the same position as \(\beta_{1}\). To further prolong the network lifetime, we design an adaptive detouring path strategy. We define a set \(N_{i}\) for node \(i\) and an energy warning threshold \(\mathrm{e_{wht}}\). The elements of \(N_{i}\) are neighbor nodes of node \(i\). When the residual energy of the next hop of node \(i\), for example, node \(j\), is lower than \(\mathrm{e_{wht}}\), or the energy-related cost \(ce\Big{(}e_{res}^{j}\Big{)}>ce(\mathrm{e_{wht}})\), the sink node modifies the \({\varphi_{ii}}_{\|_{h\in N,\;h\
eq j}}\) values to \({\varphi_{ih}^{\prime}}_{\|_{h\in N,\;h\
eq j}}\), where \({\varphi_{ih}^{\prime}}_{\|_{h\in N,\;h\
eq j}}={\varphi_{0}\varphi_{ih}}_{\|_ {h\in N,\;h\
eq j}},0<\varphi_{0}<1\). In this way, the weight of delay-related cost of communication with node \(j\) is higher than those of the other nodes. With \({\varphi_{ih}^{\prime}}\), the sink node can determine a detouring path by choosing nodes with more residual energy as the next hop of node \(i\).