## 6 Conclusions  
In vehicle communications, the directional antenna combined with beamforming is the key technology to satisfy the quality-of-service of advanced autonomous vehicles. To maximize the throughput due to beamforming, it is important to manage the signaling overhead required to control the beamwidth and beam alignment. In this paper, we proposed a joint antenna beamwidth and beam alignment interval optimization scheme in highway V2I communication. We designed an aperiodic beam alignment strategy to over  
Figure 11: Average link stability ratio vs. transmit power. come the signaling overhead caused by the periodic alignment procedure and formulated the optimization problem that jointly determines the beamwidth and the beam alignment interval. Moreover, to overcome the noncausal problem, we approximated the future reward and used a Q-learning model to solve the optimization problem. The developed Q-learning model jointly determines two parameters of the proposed scheme, namely antenna beamwidth and beam alignment interval, where the reward is defined as the weighted sum of the past and expected future rewards. The proposed RL-based joint beamwidth/interval optimization scheme outperforms the other schemes. When the vehicle speed was 90 km/h, the proposed scheme increased the average throughput by about 15.9% and 98.8%, and increased the average link stability by 8.6% and 24.8% in comparison with the conventional beamwidth optimization scheme and the joint beamwidth/interval optimization scheme without future reward, respectively. The proposed aperiodic beam alignment scheme for directional beamforming increases the average throughput and link stability; moreover, it can decrease the beam alignment overhead. In this paper, for simplicity of the environment, we assumed that the vehicles are located at random distances in different lanes of the road and move at a constant speed. Additionally, there was assumed to be no interference among V2I links. Hence, if the initial positions of the vehicles are sufficiently far away, the beamwidth and beam alignment interval of the V2I link can be determined without considering the positions of other vehicles. However, in a practical environment, when the RSU steers the beam in a particular direction and determines the beamwidth, it should consider the positions of other vehicles to avoid the inter-beam interference [49; 50]. For example, if a serving vehicle is close to subsequent vehicles or vehicles in other lanes, the RSU would select a narrow beamwidth to avoid the inter-beam interference, even though the beam alignment overhead is increased. Unfortunately, the optimization problem of determining the beamwidth and beam alignment interval of all vehicles on the road, taking into account their locations and inter-beam interference as well as the channel state of all vehicles, is an NP-hard problem. Second, the variable speed of the vehicle makes uncertainties in the practical environment. When a vehicle varies its speed while moving on the road, it is difficult to predict the amount of data it will receive in the future until the vehicle hands off to other RSU, and it is also difficult to predict the inter-beam interference with other vehicles. For example, if a subsequent vehicle suddenly accelerates, it leads to the increased beam interference with the preceding vehicle. Moreover, the uncertainty increases the complexity and convergence time of the machine learning model. This paper assumed no inter-beam interference and that vehicles move at a constant speed, but the trends in performance results are expected to be similar even in the practical environment. For further study, this work can be extended to a V2I network with inter-beam interference. If the received SNR is modified to take the inter-beam interference into consideration and deep reinforcement learning is applied, the proposed approach can be extended to find a global solution in a practical environment. Conceptualization, J.L. and J.S. ; software, J.L. and H.K. ; validation, J.L., H.K. and J.S.