## 5 Simulation Results  
### Simulation Environment  
We evaluate the performance of the proposed scheme in a V2I downlink scenario with a single RSU and \(K=4\) vehicles, where vehicles move at a constant speed \(v\), on a 2 km road. The vehicles are randomly dropped on the different lanes between 0\(\sim\)50 m on the road. The simulation parameters are summarized in Table 1. For simplicity of the RL model, we use the quantized antenna beamwidth and beam alignment interval time as follows: \(\varphi\in\{5,6,\cdots,15\}\) [deg] and \(\tau\in\{1,2,\cdots,5\}\) [s].  
For the performance comparison, we consider five schemes: the random selection scheme, the beamwidth optimization scheme, the beam alignment interval optimization scheme, and two joint beamwidth/interval optimization schemes. In the random selection scheme, both the beamwidth and the beam alignment interval are randomly selected for each time slot. In the beam alignment interval optimization scheme, the beam alignment interval is dynamically selected to maximize the amount of received data by using the RL for each time slot, but the antenna beamwidth is fixed to 7 degrees. Here, the value of the antenna beamwidth was chosen to maximize the performance of the beam alignment interval optimization scheme. In the beamwidth optimization scheme, the antenna beamwidth is dynamically selected to maximize the amount of received data by using the RL for each time slot, but the beam alignment interval time is fixed to 2 s [26]. Here, the beam alignment interval time was chosen to maximize the performance of the beamwidth optimization scheme. In joint beamwidth/interval optimization schemes, both the beamwidth and the beam alignment interval are dynamically selected for each time slot by using the RL. However, depending on whether the future reward in the RL model is taken into account, we classify the joint beamwidth/interval optimization schemes into the joint beamwidth/interval optimization scheme without future reward (i.e., \(\beta=0\)) and the proposed joint beamwidth/interval optimization scheme with future reward (i.e., \(\beta>0\)).  
We first consider the performance metrics of the cumulative received data and the cumulative link stability time while the vehicle moves along the road. As long as the transmitter and receiver stay aligned in the main-lobe beamwidth, the link is stable or reliable, but while the transmitter and receiver stay aligned in the side-lobe beamwidth, the link may be in a low-quality state or may be disconnected. Hence, to satisfy the stable and high-quality service, we define the link stability time as the sojourn time in the antenna main lobe, i.e., \(t_{\rm tx}^{k}(\varphi_{j}^{k})-t_{\rm sl}^{k}(\varphi_{j}^{k})\), at time slot \(j\) in the \(k\)th V2I link, where \(t_{\rm tx}^{k}(\cdot)\) is the data transmission time and \(t_{\rm sl}^{k}(\cdot)\) is the sojourn time in the antenna side lobe. The average link stability ratio of the \(k\)th V2I link can then be expressed as follows:  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Carrier frequency & 5.9 GHz \\ Channel bandwidth & 1 MHz \\ Noise power spectral density & \(-174\) dBm/Hz \\ Slot duration & 0.01 s \\ Pilot transmission time & 0.001 s \\ Path loss model & WINNER+B1, LOS \\ V2I shadowing model & Log-normal with \(\sigma^{2}=3\) dB \\ Antenna height of the RSU & 10 m \\ Antenna height of each vehicle & 1.5 m \\ Learning rate & 0.99 \\ Discount factor & 0.01 \\ Greedy rate & 0.1 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Simulation parameters.  
\[\Gamma^{k}=\frac{1}{\sum_{j=1}^{N}\lambda_{j}^{k}}\sum_{j=1}^{N}\lambda_{j}^{k} \frac{t_{\text{x}}^{k}(\varphi_{j}^{k})-t_{\text{sl}}^{k}(\varphi_{j}^{k})}{\tau_ {j}^{k}}, \tag{12}\]  
where \(\lambda_{j}^{k}\) is the beam alignment indicator and \(\tau_{j}^{k}\) is the beam alignment interval time at time slot \(j\), in the \(k\)th V2I link.