# SROR: A Secure and Reliable Opportunistic Routing for VANETs
## 5 Deep Reinforcement Learning--Opportunistic Routing
### Selection of Relay Node  
The DRL that is employed in the SROR algorithm consists of an evaluation network and a target network, and their structure is same. Therefore, a DRL-based method is designed to find the relay node. Figure 5 depicts the overall SROR framework. The pseudo code of the SROR is illustrated in Algorithm 1. Next, we analyze the computational complexity of the Algorithm 1. For the SROR, computational complexity of each time slot is denoted by \(\tau_{1}=(O(|\mathcal{A}|\times|\mathcal{S}|\times N))\). Here, the \(\mathcal{A}\) and \(\mathcal{S}\) represent the size of the action space and state space, respectively. The \(N\) is the number of vehicles. Considering the total number of episodes and the number of time slots in each episode, the total computational complexity of Algorithm 1 is denoted by \(K_{max}\times T_{max}\times\tau_{1}\), where \(K_{max}\) and \(T_{max}\) represent the total number of episodes and the number of time slots in each episode, respectively. ```
1 Input: mini-batch size \(B\), learning rate \(\alpha\), exploratory factor \(\varepsilon_{\text{init}}\), discount factor \(\gamma\), and time interval \(\varpi\);
2 Output: relay nodes ;
3 Initialize the evaluation network parameter \(\theta\) and an empty replay buffer \(\mathcal{R}\);
4 for each episode \(k=1,2,\cdots,K_{max}\)do
5 Reset simulation parameters and set an initial state \(s_{0}\) randomly;
6 for each time slot \(t=1,2,\cdots,T_{max}\)do
7 Observe \(s_{t}\) and randomly select a number \(p_{\varepsilon}\in[0,1]\);
8 if\(p_{\varepsilon}\geq\varepsilon\)then
9 Randomly select the \(a_{t}\);
10 end if
11 if\(p_{\varepsilon}<\varepsilon\)then
12 Select the \(a_{t}=arg\max_{\alpha\in\mathcal{A}}Q(s_{t},a_{t};\theta_{t})\);
13 end if
14 Execute the \(a_{t}\) and compute the \(r_{t}\) according to Equation (10);
15 Observe the state \(s_{t+1}\) ;
16 Store \((s_{t},a_{t},r_{t},s_{t+1})\) into \(\mathcal{R}\);
17
18 end if
19 Update the exploratory factor \(\varepsilon\): \(\varepsilon=\varepsilon_{\text{init}}\times 0.99^{k}\);
20 for each gradient stepdo
21 Randomly select a mini-batch data from \(\mathcal{R}\);
22 Compute the loss function according to Equation (17) and its gradient according to Equation (18);
23 Update \(\theta_{t+1}\) according to Equation (19);
24 Update \(\theta_{t+1}^{\prime}\) every \(\varpi\) time steps
25 end if
26 Update the learning rate according to OneCycleLR policy
27
28 end for
```  
**Algorithm 1**SROR  
Figure 5: Structure of SROR. Specifically, a current state \(s_{t}\) is generated randomly by the environment at time step \(t\), and the state \(s_{t}\) is inputted into the DRL. Then, an action \(a_{t}\) is selected according to a \(\varepsilon\)-greedy policy. Finally, \(Q\) value of the action \(a_{t}\) is evaluated by the evaluation network:  
\[Q(s_{t},a_{t};\theta_{t})=\mathbb{E}[R_{t}]=\mathbb{E}\left[\sum_{k=0}^{ \infty}\gamma^{k}r_{t+k}|s_{t},a_{t}\right], \tag{15}\]  
where \(\mathbb{E}[\cdot]\) represents the expectation operation and \(\theta_{t}\) represents the network parameter of evaluation network. Then, the state \(s_{t}\) is transferred to the state \(s_{t+1}\). The quadruples \((s_{t},a_{t},r_{t},s_{t+1})\) are stored into a replay buffer. To revisit the (17), \(Q(s_{t},a_{t};\theta_{t})\) is a prediction value, and it is evaluated by the evaluation network. Correspondingly, the target value is denoted by \(y_{t}\), which is evaluated by the target network. The \(y_{t}\) is given by:  
\[y_{t}=r_{t+1}+\gamma Q\bigg{(}s_{t+1},\arg\max_{a\in\mathcal{A}}Q(s_{t+1},a; \theta_{t});\theta_{t}^{\prime}\bigg{)}, \tag{16}\]  
where \(\theta_{t}^{\prime}\) is the target network parameter and \(\arg\max_{a\in\mathcal{A}}Q(s_{t+1},a;\theta_{t})\) represents the action with the greatest \(Q\) value in evaluation network. In addition, the target network parameter \(\theta_{t}^{\prime}\) is copied from the evaluation network parameter \(\theta_{t}\) every \(\varpi\) time steps due to the same network structure.