# A Hybrid Handover Scheme for Vehicular VLC/RF Communication Networks
## 3 Hybrid Handover Scheme
### Value Iteration Algorithm  
Our goal is to minimize the expected value of the total cost by finding the optimal policy. A value function for state \(i\) with action \(p(i)\) is defined as the immediate cost of this state plus the expected sum of the cost of all future states [25],  
\[Q(i,p(i))=\hat{g}(i,p(i))+\alpha\sum_{i^{\prime}\in\Omega}\hat{\mathsf{Fr}}_{i \to i^{\prime}}\{i,p(i)\}V(i^{\prime}), \tag{17}\]  
where \(\alpha\) is the discount factor and \(V(i^{\prime})\) is the value function of the next state. The value function for state \(i\) is defined as  
\[V(i)=\sum_{p(i)\in\mathcal{A}}\Pr(p(i)|i)Q(i,p(i)), \tag{18}\]  
where \(\Pr(p(i)|i)\) is the probability of taking action \(p(i)\) in state \(i\). If a deterministic policy is adopted, only one certain action is performed.  
According to [PERSON]'s function, the optimal value function is defined as  
\[V(i) =\min_{p(i)}\biggl{\{}\hat{g}(i,p(i))+\alpha\sum_{i^{\prime}\in \Omega}\hat{\mathsf{Fr}}_{i\to i^{\prime}}\{i,p(i)\}V(i^{\prime}) \biggr{\}} \tag{19}\] \[=\min_{p(i)}Q(i,p(i)) \tag{20}\]  
Therefore, the dynamic programming of value iteration can be applied to find the optimal value function for each state as illustrated in Algorithm 1. With the optimal value function \(V^{*}\), the optimal policy is derived by \(p^{*}(i)=\min_{p(i)}\{Q^{*}(i,p(i))\}\).  
```
Input:\(\Omega\); \(\mathcal{A}\); \(\alpha\); \(\hat{\mathsf{Fr}}_{i\to i^{\prime}}\{i,p(i)\}\); \(\hat{g}\); Convergence criterion parameter \(\epsilon\); Output: Optimal Policy \(\mathbf{p}^{*}=(\cdots,p^{*}(i),\cdots)\); Initialize:\(V_{0}(i)=0\) for each \(i\in\Omega\); \(k=0\); \(\Delta=\infty\); While\(\Delta>\epsilon\) For each \(i\in\Omega\) For each \(p(i)\in\{1,0,(-1)(h-1)\}\) \(Q_{k}(i,p(i))=\hat{g}(i,p(i))+\alpha\sum_{i^{\prime}\in\Omega}\hat{\mathsf{ Fr}}_{i\to i^{\prime}}\{i,p(i)\}V_{k-1}(i^{\prime})\); End For \(p^{*}_{k}(i)=\min_{p(i)}\{Q_{k}(i,p(i))\}\); \(V^{*}_{k}(i)=Q_{k}(i,p^{*}_{k}(i))\); End For \(\Delta=\|V_{k}-V_{k-1}\|\); End While
```  
**Algorithm 1**Value iteration algorithm  
It is noted that the cost function \(\hat{g}(i,g(i))\) for each state is bounded. Hence, this iteration algorithm can be proved convergent [26]. The final handover decision for the considered vehicle VLC/RF communication system is made in terms of the optimal policy \(\mathbf{p}^{*}\)and the current transmission mode \(w\).  
With an optimal policy \(p^{*}(i)\) and the state transition probability \(\hat{\mathsf{Fr}}_{i\to i^{\prime}}\{i,p^{*}(i)\}\), the transition rate matrix can be written as \(\mathbf{P}\) for all states \((s,s^{\prime},h,b,w)\in\Omega\). Let \(\pi^{*}(s,s^{\prime},h,b,w)\) be the stationary probability for the state \((s,s^{\prime},h,b,w)\) and \(\boldsymbol{\pi}^{*}=(\cdots,\pi^{*}(s,s^{\prime},h,b,w),\cdots)^{T}\) the stationary probability vector. \(\boldsymbol{\pi}^{*}\) can be obtained by solving the following group of linear equations  
\[\boldsymbol{\pi}^{*}\mathbf{P} =\boldsymbol{\pi}^{*} \tag{21}\] \[\boldsymbol{\pi}^{*}\mathbf{1} =1 \tag{22}\]where **1** is a column vector of ones.  
Packet blocking occurs when a new packet arrives at states of \(b=B\), which results in buffer-overflowing. Therefore, the sum probability of all states with the number of packets in the queue being \(B\) is the blocking probability of the packets, which can be calculated by  
\[P_{b}=\sum_{s,s^{\prime}\in\mathcal{S}}\sum_{h\in\mathcal{H}}\sum_{w\in \mathcal{W}}\pi^{*}(s,s^{\prime},h,B,w). \tag{23}\]  
When there are \(B\) packets waiting in the queue, the newly arrived packet will not be dropped if at least one packet can be delivered before the arrival time. Therefore, \(P_{b}\) is the upper bound of the packet loss rate.